Measuring INTOSAI regions’ performance for the delivery of capacity development programs
When I first took over my new position as ARABOSAI General Secretariat adviser I was keen to get going and make a difference. I wanted to find out how an INTOSAI region could maximize the value of its interventions to support capacity development programs. Perhaps I was little bit naïve, but things turned out to be more complicated than I was expecting.
I learned quickly that ARABOSAI’s role is more than just the sum of the capacity development initiatives underway. Like any other INTOSAI region, ARABOSAI is tasked with facilitating learning and skills development through a wide range of activities, including cooperative audits, expanding INTOSAI competency framework to reflect and accommodate regional requirements and providing input to INTOSAI’s development of audit methodologies based on the needs of the member SAIs. These activities flow from the region’s statutes, its strategic plan and the INTOSAI CBC framework for regional professionalism.
However, while ARABOSAI is responsible for achieving its own strategic objectives, responsibility for capacity development within an individual SAI belongs to that SAI. This might conflict with what SAIs expect from the region, so to avoid misunderstanding perhaps we need to set out the roles and responsibilities of the SAI and the ARABOSAI region. I do not suggest this for the fun of it. When I, together with a few of my colleagues in the ARABOSAI general secretariat, recently started to think about how to measure, and therefore differentiate, the ARABOSAI region’s contribution to the success of capacity development programs, two things surfaced.
- Understanding a region’s contribution
The SAI-PMF framework is the most commonly recognized tool to measuring an SAI’s performance as a modern, ISSAI compliant, external auditor. If capacity development programs are being undertaken, a SAI-PMF can also be a way for an SAI and its development partners to understand if any interventions being undertaken areas of activity are improving the SAI’s capability. However, the SAI-PMF does not attribute causality or ownership to each intervention, but delivers a judgment on standard performance characteristics.
A good evaluation tries to attribute outcomes to the rightful body. There are a range of factors which influence the capacity development of public auditors. Some factors relate to regional roles, such as the development of guidance, the increasing skills of regional working groups and the assessment of SAI’s needs. Other factors, such as attitude to training, participants’ selection process and the translation of capacity building products into audit, are less easily influenced by the region as compared to SAIs.
If regions need to measure the scale of progress made by their own programs or interventions, perhaps assessing the relative impact of all these factors on the achievement of aims and targets requires an overall logic map which details both the region’s and SAI’s roles and activities. The logic map helps regions consider how results can best be achieved, to articulate a clear narrative for the choice of initiative and will later help with monitoring, evaluation and the reporting on progress. This logic map could be developed in association with a scorecard to monitor progress against the region’s main tasks. A scorecard approach would have the merit of bringing together assessments of progress with tasks. This should cover both the region and SAIs in a way that might make it easy for any evaluator to clearly dissociate between the region’s performance and SAI’s performance. This approach could also help answer the question of if capacity building programs can be delivered more cost-effectively on the regional level compared to on the country level.
- Understanding a region’s organizational maturity
I asked myself, if the SAI PMF was the approach INTOSAI chose to help understand an SAI’s maturity, why not develop such a model for use by INTOSAI regions on themselves? INTOSAI regions operate through a wide range of business models. Despite some similarities, regional activities, and their contribution to delivering capacity development programs are unique. Some regions provide services directly, while others focus on promoting the expansion of capacity building programs. As I have begun to gain more exposure to these differences, I have found that this uniqueness is driven by a regions resources, statutes, strategy and even maturity level. So rather than assess against an absolute standard, perhaps a maturity model would enable assessment of capability while respecting this uniqueness.
Overall, the goal of the above ideas is not to facilitate benchmarking, or ranking of regions. The goal is to help the next people trying to understand the value their region is adding to capacity development. I think that for ARABOSAI, capacity development is not all about raising funds and allocating them to training delivery; it should have a fine balance to it. When such an approach is to be assessed against any benchmark, this should be done with reference to a capability-maturity model, rather than a specific business model.
As a conclusion, I think that measuring the INTOSAI region’s performance for the delivery of capacity development programs, requires mapping out their contribution so that we can see if they are of value versus SAI-led development. For that purpose, an overall logic map and a regions’ capability-maturity model should be developed.