Back

ARTICLES

1. Introduction

Over the last few years, SAI-Oman has been using IT extensively as a driver in its critical performance audits. We felt that these experiences needed to be disseminated widely in the INTOSAI community, especially as e-governance is spreading world-wide. In this context, SAI-Oman is also leading a project on "CAATs for Non-Financial Audits" on behalf of the INTOSAI Standing Committee on IT Audit.

In this article, we attempt to explain our approach to IT -enabling performance audit, with the help of suitable examples from our experience in field audit

2. Need for IT support

SAI-Oman is a relatively new entrant to performance audits, and therefore had the advantage of not carrying the baggage of tradition. Expectations of the auditees from the SAI was quite high, especially in view of a significant expansion of the SAI’s mandate. Top management in the auditees were looking forward to a fresh perspective on their operations, which was not available to them. An "out-of the box" approach was thus imperative. SAI was expected to provide management advice and recommendations, rather than fault-finding on individual transactions. In our view, the traditional performance audit approach of reviewing only the auditee's MIS reports or test-check of a small sample after due risk assessment, would not be adequate.

Many of the key auditees were corporate government entities, where the core business activities were fully computerized. This provided an opportunity for the SAI to conduct such IT-enabled analytical reviews. Also, we were short of skilled manpower for such performance audits, and therefore needed a "force-multiplier". IT provided just such a multiplier.

3. IT-enabled Performance Audit Strategy

Our IT-enabled performance audit strategy had two broad components:

• Reconciliation of management's representations / reports on performance by cross-checking with other alternative sources of data, and
• Free-form analytical review of business data -like a data mining exercise-to identify undiscovered trends as well as to gain a different insight on known trends.

3.1 Reconciliation of Management Information

Often, management reports do not entirely reflect the correct state of affairs on the grounds. This is often due to a variety of reasons like:

• Multiplicity of reporting levels and filters;
• Rigid departmental boundaries and inconsistencies between priorities of different departments;
• Lack of quality control in data entry
• Reliance on third-party information

Besides, of course, lack of integrity in the reporting process.

Therefore, the first task, while analyzing data, is to (a) check for internal consistency within the database and (b) search for alternative sources of data to make suitable comparisons, e.g. between financial/billing data and operational data. The focus is two-fold -(a) to attempt a broad reconciliation between multiple sources (b) and keep looking out for discrepant data elements.

3.2 Free-form Analytical Review
Analytical review of performance data is essentially a free-form, unstructured exercise. It is difficult to prepare a comprehensive plan of the exact lines of analysis in advance of the testing. However, the nature of the exercise is such that without a time budget, the audit team is likely to get lost in data analysis, without any meaningful audit results. It is therefore important to have a "guillotine" time schedule to close out the analytical review.

In our experience, there are two aspects to free-form analytical review

• Data Profiling
• Hypotheses Generation and Testing

3.2.1 Data Profiling

Normally, the first stage in the process would be to profile the data in different key tables on various dimensions -singly and jointly -to give a 360-degree view of the data. This profile would also involve extensive statistical analysis –including measures of central tendency, variance as well as skewness and frequency distribution.

For example, operational hydrocarbon drilling data by a fleet of rigs would be profiled by:

• Rig and Rig contractor
• Type of well
• Period of drilling and time taken for drilling
• Productive and non-productive time
• Productivity in terms of depth drilled
• Time taken on key activities like maintenance, rig move etc.
• Status of drilling and drilling completion

5.3.2 Hypotheses Generation and Testing

Large numbers of hypotheses (in hundreds) would need to be generated for electronic analysis, out of which only a very small number would be taken for detailed analysis.

The generation of hypotheses for data analysis is based largely on our collective past experiences applied on the data profiling undertaken.

The knowledge of business processes forms a key driver in this regard. Application of past experiences in similar business processes, helps to cut down the time taken to acquire enough domain expertise for such hypotheses generation.

Those hypothesis, which throw up discrepancies or unusual trends, are then prioritized for detailed checking with reference to other records. This is done in order to eliminate errors on account of data errors, or incorrect / incomplete understanding of data. An alternative approach is to ask the auditor to account for the discrepancy, indicating whether it is a data error or an organizational non-compliance. This forms the core of the audit quality control.

4. Examples
Some examples of IT -enabled performance auditing drawn from our experiences, are described below in brief:

4.1 Case 1

A development finance institution was providing subsidised long-term credit to local corporate clients, with a view to promoting local entrepreneurs. The organisation had computerised its credit function using a bespoke system developed on Oracle. Our analysis revealed that there were multiple tables for storing data relating to principal and interest amounts for borrowers, which were internally inconsistent and also not consistent with borrower records. The organisation was unable to assure itself as to the correctness of calculation of amounts due (principal as well as interest) from borrowers, nor to the accuracy of the statements of outstanding loan amounts intimated to borrowers.
In view of these gross deficiencies, we could not proceed further. We recommended that the existing IT system be replaced, and data cleaned and re-entered. This was accepted by the auditee.

4.2 Case 2

We conducted a performance audit of a long-term contract for chemical injection into pipeline facilities. The primary objective of the contract was to optimise the rate of injection of chemicals into a pipeline network, without affecting performance. There was a bonus / penalty system for short / excess chemical consumption. The contract was won by the successful contractor, on his promise to achieve a 15% reduction in chemical consumption from existing levels.
We noticed that while the contractor was submitting hard copy monthly reports, detailing the chemical deliveries at various tanks, and all adjustments to injection rates for each pipeline, there was no evidence of due consideration or analysis of the monthly reports by the auditee. We consequently created a Microsoft Access 2000 Database, where we re-entered details of:

• All pipeline facilities, and the contractual norms for chemical concentration
• Original injection rates and adjustments thereto over a five month period over 16 pipelines
• Details of fluid flow, which would affect the injection rates
• Chemical deliveries and tank level measurements over the same period.

Using this data, we were able to recomputed the actual injection rates, based on fluid flow, deliveries and opening and closing tank levels. Our analysis revealed that:

• The promised reduction in chemical consumption was not being achieved
• Measurements systems for delivery as well as injection were flawed
• Monthly reports submitted by the contractor were unreliable
• The basis for payments to the contractor was unreliable.

Management broadly agreed with our findings, and agreed to take remedial action for (a) levy of penalty for non-optimization of chemical consumption (b) improving measurement systems, and (c) quality check of all monthly reports, including past reports.
 

4.3 Case 3

We were conducting a performance audit of services contracts, which involved both turnkey model as well as a time charter model. In order to ensure productivity, the time charter model involved fixing of daily targets as well as a bonus scheme for extra achievement over target. This scheme was devised, based on an earlier analysis of targets over three years.

We revisited the earlier analysis, and conducted a fresh analysis of achievements vis-a-vis targets on multiple dimensions, namely:

• Day-wise target achievement
• Month-wise target achievement, as the billing is on a monthly basis
• Prospect-wise target achievement

The frequency distribution and statistics generated by the above analysis, revealed significant skewness in target setting, tending to favour the contractor. Management agreed with our findings, and revised the targets upwards, with significant productivity gains.

5. Lessons

5.1 Audit Findings

The audit findings, as a result of IT -enabled performance audits, far exceeded even our initial expectations, let alone those of the auditees. We were clearly able to highlight trends, not discernible to auditee management, and come up with specific and clear-cut recommendations for improvement. These findings were invariably accepted, simply on the strength of the data and SAI's analysis thereof, and not subject to substantial differences of opinion. In addition to substantive procedural changes, all the auditees also accepted the need to clean up data.

5.2 Skills Required

We are firmly convinced that it is much easier to teach the auditor IT skills, rather than teach the IT specialist audit skills. With the advent of simple desktop suites, IT skills are relatively easy to teach. The domain expertise of the auditors can be leveraged substantially if such IT skills are added on. Our experiences show that an IT skilled auditor is a much better option than adding an IT specialist to the audit team.

In any case, the modem auditor has to necessarily use personal productivity software in his daily work. Adding querying skills is a relatively small additional investment, with the potential for substantial returns.