1. This appendix contains a broad framework of evidence gathering techniques. The evidence gathering techniques and the introduction to the statistical sampling are not exhaustive list of such techniques. These are basically in the nature of introduction to some of the more important techniques, which are mentioned below.
2. A survey is a technique for gathering specific information, through a questionnaire, from a group of people, representative of the larger population. The responses to the questionnaire are analysed and projected to the whole population. Surveys can be used both at the planning (for identifying issues or key concerns in an issue) as well as execution phases of performance audit (for providing necessary evidence).
3. Surveys can collect quantitative information in order to estimate output (performance indicators) or evaluate processes in a project, e.g., assess how well a project is being monitored. Surveys often seek opinion, which is of use in assessing the beneficiary satisfaction/ quality of service provided in the project. Surveys provide a framework for gathering evidence when information required for addressing the audit objective cannot be gained from files and administrative records.
4. The survey process would involve the following stages:
- Deciding the objective of the survey and its target population – At the start, the reasons for conducting the surveys and the information needs to be gathered.
- Deciding the size of the sample and a method of drawing a sample from the target population: For the results of the sample survey to be applicable to the entire target population, the sample size needs to be derived through statistical sampling.
- Deciding the nature of survey to be done, direct interviewing or through post, telephone, e-mail: After selecting the sample, it is essential to establish contact with the members of the sample to collect the desired information. This could be done through personal interviews, questionnaires sent by post or through e-mail, or interviews on the telephone.
- Framing the questionnaire: Questionnaire is the set of structured questions the answers to which are being sought in the survey. Designing the questionnaire is an important element, which determines the success of the survey project.
- A way to test the validity of the response would be to introduce counter questions within the questionnaire at appropriate stages. This would verify the correctness of the response.
- Administering the survey and collecting information: The survey needs to be administered to the chosen sample as per the technique decided on. It is necessary to specify a suitable time-limit for providing the responses, especially in a survey through mail (electronic or postal).
- Completed questionnaires need to be checked to ensure that the respondents have answered all the appropriate questions and there are no unexpected answers suggesting that the respondents misinterpreted the questions.
- Analysing the results of the survey: Analysing the results of the survey would involve understanding its nature and projecting it to the population. Several qualitative and quantitative techniques can be used to analyse the results of a survey. Small surveys can be analysed manually while for larger surveys software packages can be used.
- Need for an expert: Experts can provide the necessary skills and input necessary for the auditor. However, it is the auditor’s responsibility to provide competent, relevant and sufficient evidence.
5. Success of a survey depends both on skills and the environment in which the survey is done. If the sample is chosen incorrectly or there are a large number of non-responses or wrong responses, the results projected for the population may be erroneous.
6. Sampling means testing less than 100% of the cases in the population for some characteristic and then drawing a conclusion about that characteristic for the entire population. When using judgmental sampling or non-statistical sampling, a pre-determined proportion of the cases is checked by the auditor on the basis of his/her judgment. This sampling technique can be effective if properly designed. However, it does not have the ability to measure sampling risk and thus audit conclusions reached becomes rather difficult to defend. For statistical sampling techniques, there is a measurable relationship between the size of the sample and the degree of risk. Statistical sampling procedure uses the laws of probability and provides a measurable degree of sampling risk. Accepting this level of risk, (or conversely at a definite assurance level) the auditor can state his conclusions for the entire population. In sum, statistical sampling provides greater objectivity in the sample selection and in the audit conclusion.
7. The basic hypotheses of statistical sampling theory are:
8. Statistical sampling may be used in different auditing situations. The auditor may wish to estimate how many departures have occurred from the prescribed procedures; or estimate a parameter in the population. Based on whether the audit objective is to determine a qualitative characteristic or a quantitative estimate of the population, the sampling is called an attribute or variable sampling.
9. Attributes sampling estimates the proportion of items in a population having a certain attribute or characteristic. In an audit situation, attribute sampling could estimate the existence or otherwise of a error. Attribute sampling could be used when drawing assurance that prescribed procedures are being followed properly.
10. Variables sampling estimates a quantity, e.g., amount of sundry debtors shown in the balance sheet or the underassessment in a tax circle. Variables sampling has certain drawbacks which can be overcome through monetary unit sampling, which is an attribute sampling which provides quantitative results and is suited to most audit situations.
11. There are different ways in which a statistical sample can be selected. The most frequently used method is random selection where each item in the population has a equal chance of selection. Simple random sampling ensures that every member of the population has an equal chance of selection. Though simple to administer, the underlying assumption is that the population is homogeneous. In cases where the population is non-homogeneous, a stratified sampling would be a better option. Here the population is sub-divided into homogeneous groups and then a random sampling is done on the groups, ensuring a better representative sample. Each sampling method has its practical use and limitation. The auditor uses his judgment in determining which kind of sampling is best suited to his audit job. It is advisable to take expert advice in judging the most suitable method.
12. Some random sampling methods that are commonly used are
At each stage of sampling a suitable method of selecting the sample could be used. Once the method of sampling is decided, it is essential to design the actual sample.
13. This is used when audit desires to estimate an attribute in a population. It is useful for testing internal controls. For example, the auditor may decide that if there are errors above a certain threshold the control systems are inefficient. The attribute, which the auditor is interested here are errors/ aberrations from processes. The basic stages that are involved here are mentioned below:
14. Benchmarking is a process for measuring an organization’s performance or process against such organizations that consistently distinguish themselves in the same categories of performance. In the context of performance audits, benchmarking helps identify opportunities of achieving better economy, efficiency and effectiveness. Benchmarking can be done with other external organizations or with internal units of the same organization having different levels of performance. For example, processes and performances of a province/ district in a project can be benchmarked against a high performing province/ district to efficiently highlight problems and their causes.
15. Benchmarking can be of use both during the planning and execution phase of performance audit. In the planning phase, benchmarking can be an important tool while setting the audit criteria. In the execution phase, comparing detailed processes of the auditee with its benchmarking partner will reveal the gaps and areas of mismatch. It would help the performance audit team in substantiating audit findings and conclusions.
16. The benchmarking process will involve the following stages:
17. While benchmarking helps the performance audit in providing objective data on methods of operation, supporting recommendations for making changes, carrying out objective review of processes, practices and systems and presenting a target for improvement in the auditee, it also requires high degree of skill and the acceptability of findings of benchmarking with the audited entity is an area which needs attention.
18. A focus group usually comprises between 8-12 people that are judgmentally selected as representatives of a larger population, for example members of the public, senior officials of the audited entity, decision makers, experts, beneficiaries, etc. A facilitator is used to direct the group discussion and to focus attention on the specific aims of the session (which may involve developing practical recommendations). The evidence produced through focus groups is qualitative in nature and has the major advantage that it gives depth and understanding to a given topic. Focus groups are particularly useful in identifying issues, understanding the reason behind taking decisions, testing emerging findings to generate survey questions and to develop practical recommendations. A typical focus group session can last from 90 to 180 minutes.
19. Focus groups provide a means to obtain a broad understanding of the issues with which performance audit is concerned like attempts to assess the impact of the programmes on people and understanding causes and effects of problems to achievement of economy, efficiency and effectiveness by obtaining the opinion, perception of individuals actually associated with the activity being examined.
20. Focus groups can be used both during the planning and execution stages of performance audit. Focus groups can be useful in determining the quality of service provided in a programme. Focus group of beneficiaries of a programme will give an insight into its effectiveness.
21. Evidence obtained from focus groups alone is rarely sufficient. It is used in combination/ collaboration with other evidence.
Steps to using this technique
22. An interview is a question-answer session to elicit specific information. Interviews could be structured or individual (un-structured). Structured interviews aim to gather the same information from many people. Questions in a structured interview have specific wording and are asked in a set order. Data from structured interviews can be quantified. In an individual interview, the interviewer can change direction of the interview, probe on certain issues. Interviews can be used both in the planning and execution phases of performance audit. Individual interviews form an important data collecting technique in performance audit.
Steps to using this technique
23. Interviews, however, often suffer from problems of confirming what is said. This reduces their value as evidence. Interviews generally are weak evidence and need to be corroborated through information from other independent sources such as documents, observations or structured interviews
24. Case study is the examination of a selection of incidents, events, transactions or items in order to understand or examine a programme or activity. It is an in-depth study of individual cases to explore the audit issues. Case studies can be used in the planning and execution stages of performance audit. Case studies can be used to highlight good practices. A comparison of cases that have performed well with those of average performance can help to identify the good practice. Case studies can show why an activity does not provide value for money and to demonstrate the impact of a specific action or event.
Steps to using this technique
25. The results from the cases can be more accurate as greater in-depth investigation is possible. Case studies make it easier to determine problems, their cause and effect, alongside good practices and can thus produce pragmatic recommendations. However, case studies involve judgment in selection and could be subjected to bias. While the finding from the individual case may be sustainable, it is difficult to determine whether the issue highlighted is a generic problem or an aberration.