Back

Audit of IT Systems Development

Pritom Phookun Joint Director NAAA, Shimla

Audit of IT Systems Development

Overview

A proper approach to systems development constitutes an important general control. In this session we will discuss the need for the audit of systems development followed by a detailed study of one well-known approach. Finally we will discuss the audit issues that should be considered while conducting the audit of systems development. In so doing, we will focus on the System Development Life Cycle (SDLC) approach and only make a passing reference to two other approaches to systems development.

This paper will discuss the need for the audit of systems development, the various stages of the SDLC and, finally, the audit issues relating to each stage of systems development.

Need for the Audit of Systems Development

Systems development, like any other project, is exposed to various risks of failure. It is the responsibility of the management to take adequate controls such that the risks of project failure are minimized. One of these controls is to adopt an acceptable IT system development approach. On the other hand, by providing an independent assessment of an IT systems development project, audit can contribute to the process of minimizing the risk of IT project failures. First, audit acts as a deterrent: In anticipation of adverse findings by audit, audited organizations often take care to minimize mismanagement. Second, audit observations and recommendations relating a particular audit can help audited organizations to avoid repeating earlier mistakes in future systems development projects. Third, based on similar audit findings across different systems development projects in various organizations, audit can produce very useful best practice guides.

The audit of various systems development projects has revealed the common types of failures in such projects. These include the following:

• Project is abandoned (not delivered)
• Project is delivered late.
• Project cost exceeds approved budget.
• The delivered system lacks required functions.
• The delivered system does not meet performance standards.
• The system is not user friendly.
• The system is unreliable / fails frequently during operation.
• The system is not integrated with other systems.
• The system is difficult and costly to maintain.
• The system is not flexible enough to take care of future requirements.
• The system is costly to enhance

Some of the common reasons for such failures have been found to be:

• Failure to follow an acceptable approach to system development
• Inadequate commitment of top management
• Inadequate involvement of users
• Inexperienced project manager
• Unclear objectives
• Failure to identify project risks
• Inadequate control system to manage project risks
• Mismanagement of suppliers and consultants

System Development Life Cycle

If the auditor plans to conduct a comprehensive audit of IT systems development, then it is very useful for him to be knowledgeable about the best practice approaches to systems development. This knowledge provides him/her with a framework against which to evaluate (i) whether the audited organizations did actually follow a structured approach, and (ii) if they did, whether it was followed adequately.

The best-known and most comprehensive approach to IT systems development is the Systems Development Life Cycle (SDLC) approach. In recent times, two other approaches have emerged, namely Prototyping and Rapid Application Development (RAD). The latter approaches are not completely different from the SDLC approach. In fact, they have emerged from modifications of the SDLC approach as a response to clients' need for quicker delivery.

Literature on SDLC conceptualizes the various stages in slightly varying ways, from one author to another. For this session, the SDLC may be conceptualized into seven sequenced stages: (i) project need definition, (ii) feasibility study, (iii) system analysis, (iv) system design, (v) system development, (vi) implementation and maintenance, and (vii) post implementation review. Let us look at each of these stages in more detail.

1.    Project Need Definition

This is the first stage of the process. The objective of this step is to clearly define the need for the project. The primary questions asked at this stage are (a) Is the project required? and (b) if yes, what are the broad expectations?

Project need definition involves the following:

Recognize the problem. There has to be an awareness that a problem(s) exists in the existing systems which should not be ignored.

Define the problem. This involves determining the exact nature of the problem, its location and its likely causes. The objective is to understand the problem to enable finding an appropriate solution.

Set system objectives. At this point the objectives are set in broad, general terms. They will be made specific later.

Identify system constraints. Some constraints are imposed by the environment, such as the government's demand for tax reports and the customers' need for billing information. Others are imposed by the firm's management, such as the requirement to use existing hardware or to have the system up and running by a certain date. If it is finally decided that there appears to be a need for the project, a feasibility study is conducted.

2.    Feasibility Study

This is the analysis of the possibility and the worthiness of undertaking the project. There are several dimensions of feasibility:

• Technical: Is there hardware and software available to perform the necessary processing?
• Financial: Can the proposed system be justified monetarily by comparing its benefits and costs?
• Non-financial: Can the proposed system be justified based on benefits that cannot be measured in monetary terms? Example, more comfortable working environment for computer staff is a non financial benefit, while health problems arising from extended use of computers is a non financial cost
• Operational: Is the system design such that it will receive the support of the people who must make it work?
• Schedule: Will it be possible to implement the system within the imposed time constraints?

The study is documented and a feasibility study report is prepared. The report would also recommend whether to undertake the project or not to do so. The report is submitted to top management for a decision to go ahead or not to. If the decision is to go ahead, then the next step is undertaken.

3.    System Analysis

The objective of this step is to thoroughly understand the existing system and determine the users' information and performance requirements. At this stage a project team should be formed to establish clear responsibilities and accountability for the project. The main issues considered by the project team at this stage are:

Define information needs: What information is needed? Who needs it? When is it needed? Where is it needed? In what form is it needed?

Define performance criteria: What are the acceptable performance standards? For example, a marketing manager who needs a monthly expense report may insist that the report must be available no later than three days after the end of the month.

Analysts learn about the users' information needs by engaging in a variety of information-gathering activities: personal interviews, observations, record searches, and surveys.

This stage concludes with the preparation of a User Requirements Specification (URS) document. This has to be approved by the user representative and by top management.

4.    System Design

Once the URS has been approved, the project team starts designing the new system. The system design is meant to be a blueprint of the new IT system. The project team considers and evaluates alternative designs and selects the one that is expected to meet the user requirements most satisfactorily within the given constraints. The output of this stage is the system design document (SDD). The SDD includes the following:

1. Data flow in the information system
2. Database structure
3. Hardware and software configurations
4. User interface: That is, how the users are expected to interact with the system.
5. Physical facilities required

The SDD is submitted to top management for approval.

5.    System Development

This stage is also called system construction: The objective of this step is to convert the design into an actual working system. The main issues considered in this stage are:

1. Procurement of hardware and software.
2. Preparation of the database.
3. Coding of programs that form part of the application.
4. Acceptance testing: This had three parts - (a) testing of individual programs, (b) testing of the whole application, and (c) acceptance testing by the user (a very important issue).
5. Preparing training plans and conducting preliminary training for various levels of users.

By the end of this stage, the new system is ready for use.

6.    Implementation and maintenance

The objective of this step is to bring the new system into actual production and ensure that it functions adequately. The main issue in implementation is the manner of cutover to the new system. There are four alternatives to cutover:

1. Pilot: A pilot is a trial system implemented in a subset of the overall operation, such as an office or geographic area. For example, the Air Force might try out a new inventory system on a single air base. If the pilot succeeds, the system is implemented in the remainder of the operation, using one of the three other cutover approaches.
2. Immediate: The simplest approach is to convert from the old system to the new one on a given day. However, this approach is feasible only for small firms or small systems, since the timing problems become greater as the scale of operation increases.
3. Phased: In a phased cutover, the new system is put into use one part at a time. For example, the firm can cutover to the order entry system, followed by the inventory system, and so on. Or, cutover for all of the systems can be accomplished at one geographic location, followed by another location, and so on. Phased cutover is popular for large-scale systems.
4. Parallel: A parallel cutover requires that the old system be maintained until the new one is fully checked out. This approach offers the greatest security against failure but is the most expensive, as two sets of resources must be maintained.

Once the system has been put to use, modifications can be made so that the system continues to provide the needed support. These modifications are called systems maintenance. Systems maintenance is performed for three reasons:

1. To correct errors: System use uncovers bugs in the programs or weaknesses in the design that were not detected in system testing. These errors are corrected.
2. To keep systems current: Over time, changes occur in the system's environment that requires modifications in the design or software. For example, the government changes the formula for computing income tax.
3. To improve the systems: As managers use the systems, they see ways to make improvements. These suggestions are passed along to the information specialists, who modify the systems accordingly.

7.    Post Implementation Review

After the new system has had a chance to settle down, a formal study is conducted to determine how well it is meeting the user needs and system objectives. Such a study is called a post implementation review and is undertaken to assess success of the project and learn lessons for the future. If this review reveals serious deficiencies in the system or need to substantially enhance it, then the modifications may constitute a project in itself. In that case, the system development cycle returns to the first stage of project need identification.

Audit Considerations

We discussed earlier how audit of systems development can make significant contributions to IT project success. In conducting such an audit, the auditor should consider various issues to ensure a comprehensive audit. A checklist of audit considerations are indicated below, starting with general audit issues, followed by considerations in each stage of the SDLC.

General audit issues

• Was a formal systems development approach followed ?
• Was documentation adequate at each stage ?
• Was the steering committee involved at every stage ?
• Was there a specific project team ?
• Did the project team have adequate range of skills represented ?

Project Need Identification

• Was the problem clearly defined ?
• Were alternative solutions considered ?
• Why was the IT solution accepted ?
• Were broad objectives of proposed IT solution clearly defined ?
• Were major constraints identified ?

Feasibility Study

• Were alternatives considered ?
• Was each alternative evaluated in terms of technical, financial non- financial, operational and scheduling feasibility ?
• Were decisions based on reliable and adequate information ?
• Were the assumptions reasonable ?
• What were the recommendations ?

3.    System Analysis

• Were potential users at all levels consulted ?
• Were system manuals and working papers studied ?
• Were information needs clearly identified ?
• Were measurable performance criteria identified ?
• Did user sign off the URS ?

4.    System Design

• Were users adequately consulted again ?
• Were alternative designs considered ?
• Did selected design meet URS ?
• Was adequate financial audit trail provided for ?
• Were adequate controls provided for ?
• Was design flexible to cope with change ?
• Were hardware and software configurations specified ?
• Did system security designs meet user needs ?
• Did users sign off the SDD ?

• Did hardware and software meet specified configurations ?
• Were well-documented user, operational and technical manuals prepared ?
• Was there an acceptance-testing plan ?
• Was acceptance testing done as per plan at all levels - unit programs, system and user acceptance ?
• Were test results and action taken well documented ?
• Were program library standards followed ?
• Did users sign off after user acceptance testing ?
• Were training plans finalized and delivered ?

• hat cutover method was followed ?
• If "immediate" cutover was made, were reasons justified ?
• Was the training program completed ?
• Was there a qualified system administrator ?
• Was data transferred to new system well controlled ?
• Was master data file integrity verified ?
• Was the backup plan followed ?
• Were user responsibilities and access rights clearly documented ?
• Were there violations of system security ? If yes, what was the impact and what subsequent actions were taken ?
• What kinds of problems occurred? What were the impact and what subsequent actions were taken ?
• What were the maintenance measures ?
• Were users satisfied with maintenance ?
• Was there a comprehensive business continuity plan ?

7.    Post Implementation Review

• Did system meet URS ?
• Were there any project failures, such as :
  • time over run
  • cost over run
  • system lacked certain functionalities
  • performance standards not met
  • not user friendly
  • system failed frequently during operation
  • was not integrated with other systems
  • proved difficult and costly to maintain
  • could not incorporate future needs
  • proved costly to expand system
  • project was abandoned.

Note: In many cases, third parties may be substantially engaged during various stages of the SDLC. If that is so, the auditor should assess the adequacy of, and compliance with, outsourcing controls.

Summary

In this paper we discussed various aspects of the audit of systems development. Because SDLC is the most comprehensive approach to systems development, the session focused on this approach in IT systems development.

(From: Pritom Phookun, Joint Director, NAAA, Shimla)

References

1. Raymond McLeod, Jr. Management Information Systems Sixth Edition (New Jersey: Prentice Hall, 1995)

2. Laudon, C. Kenneth and Laudon, P. Jane. Management Information Systems, Fourth Edition (New Jersey: Prentice-Hall, Inc. 1996)

3. Weber, Ron. EDP Auditing - Conceptual Foundations and Practice, Second edition (Singapore: McGraw-Hill, 1988)

4. INTOSAI Standing Committee on EDP Audit. "INTOSAI IT Audit Training Courseware", 1998.

5. Johnson, John H. Micro Projects Causes Constant Change. (The Standish Group International Inc., 2001)