Dear colleagues, Dear attendants the seminar please find in the text below the answers to the questions you pose to the presenters after concerned presentation. Answers are to question they were not answered due to the time constraints during the seminar.
SAO Slovakia/Subcommittee after collecting the questions distributed them to the individual presenters who kindly elaborated the answers which are summarized in this paper. We hope hereby you may find at least some of the hints useful for your possible engagement in Peer Review and other tasks performed by the SAOs you are employed in.
We would like to encourage you to continue in this dialogue. Should you have further questions or need to clarify some elements in your assignments connected to the Peer Review, please feel free to write it to Subcommittee contact email: peerreview@nku.gov.sk. Our team with the assistance of the subcommittee members and colleagues among the Intosai Community will try to answer your inquire. We believe that is useful to gain as much as possible from the experience acquired in different SAOs in this and other matters. The Subcommittee have ambition to facilitate this effort for mutual benefit of the all involved colleagues.
Question after the Presentation SAO Slovakia :
Answers to the question by SAI Slovakia
Sainem SAO Indonesia :
How Slovakia came up to the tree focus? from the result of the SAI PMF self assessment?
SAI Slovakia selected the three focus (areas), from the result of the SAI PMF self-assessment. The management of SAO considering the several deficiencies in Planning, Quality assurance and Communication as indicated in the PMF reflecting the situation in the y 2017 and before introduced several operational measures to improve the situation in these three domains. Therefore, for these domains (areas) the Peer Review was ordered to obtain the external independent evaluation and possible recommendation to the issues concerned. The selection is up to the SAO intends to undergone the Peer Review. In this aspect the instruction is provided in chapter 3 “Strategic Consideration” by INTOSAI GUID 1900 – Peer Review Guidelines.
Mohamad Iqbal Aruzzi SAO Indonesia:
How you choose/determine your auditee for hearing/interview session? Is it random sampling, or purposive based on some criteria? Thanks
The determination of auditee was carried out according to the professional judgement of the reviewer.
The criteria were type of audit, materiality of themes, and areas of audited problems/organisations.
Semih Zencirkıran SAO Turkey :
As far as I understand, Peer report includes three focus. Why you did not think to prepare separate reports for these there focus since it may be thought that they do not so close relationship. Thanks.
In this aspect, we think that it is more appropriate to have just one report, (as given in Chapter 8 „Reporting” by INTOSAI GUID 1900 – Peer Review Guidelines) because report was elaborated within one Peer Review assignment in the line with Memorandum of Understanding and Terms of Reference. Report was not prepared by SAO Slovakia, however by Reviewer team. SAO Slovakia consider the report received is compliant with the requirements given in INTOSAI GUID 1900 – Peer Review Guidelines.
Nuno Lopes SAO Portugal :
Did your self-assessment of 2017 include all sections/chapters of SAI PMF? Did the different peer reviewers worked simultaneously? And did they work with autonomy in the correspondent area to be reviewed or jointly for all subjects?
Did your self-assessment of 2017 include all sections/chapters of SAI PMF?
All except:
SAI-18 Jurisdictional Control Standards and Quality
SAI-19 Jurisdictional Control Process
SAI-20 Results of Jurisdictional Controls (for SAIs with Jurisdictional Functions)
SAO Slovakia does not have a Jurisdictional Functions
Did the different peer reviewers worked simultaneously?
After the joint meetings on general issues of the different reviewers teams worked each according their own time schedule, depending on Different peer reviewers theme and needs. However in mid term and in the final phase the mutual exchange of experience and findings was discussed. On an ongoing basis, the whole team communicated its partial conclusions with each other as well as with the SAO SR and requested new documents in case they were missing.
And did they work with autonomy in the correspondent area to be reviewed or jointly for all subjects?
The teams worked with substantial degree of autonomy for each areas. Details of areas, volume of work, composition of focus group and assessed topic was different in each areas, the coordinated autonomy was important Teams and focus groups did act flexible way , several modification of time schedule and tasks redistributions were necessary to cope with time availability of involved staff of SAO Slovakia and Stake holders / focus groups as well. However in mid term and in the final phase the mutual exchange of experience and findings was discussed.
Sainem SAO Indonesia :
During the online peer review, what the biggest challenges you found? Do you think the online peer review is much more effective or less effective than the offline peer review?
We are not able to provide unequivocal answer to that. We assume, that optimal mode for carrying out the Peer Review (PR) would be to have a balanced weight of “on site” and “on line” PR. We believe, that initial meeting between the SAOs and reviewer ́s team would be useful to have on site, to know each other and clarify the essential points of process and expectation, as well as other at least one meeting in mid-term and final phase.
It seems that in the operational phase of PR many tasks, like assessing/exchanging the SAO ́s documents, including translations, clarification of details, clarification meetings etc. is appropriate to carry out electronically via emails and on line video applications. It should be added that online work saves time and expenses.
Hema SAI Malaysia:
To SAI Slovakia – If a SAI which has never conducted peer review before is keen to conduct peer review now, what should the SAI do? What are the procedures? Should the SAI write in to Subcommittee on Peer Review? Any guidance on selecting the peer for peer review? Can the SAI select the peer or will the peer be assigned by Committee. What are types of costs that are generally be borne by SAI which is keen for peer review.
If a SAI which has never conducted peer review before is keen to conduct peer review now, what should
the SAI do?
First to identify needs which means knowing what SAI expect of peer review. It is necessary to determine domains (areas) to be subject to peer review, to find suitable peer reviewers (SAIs with similar legal background, competences, experience etc.) and to adopt Memorandum of Understanding (or other kind of agreement) signed by peer reviewers and the SAI undergoing the peer review. Examples of these documents are published on the CBC website here: https://www.intosaicbc.org/peer-review-materials/. Well, the experience of selected reviewer in carrying out Peer Review (PR) before is one of the criteria for reviewers team, however in our eyes it is not the most important one. PR is very complex exercise which requires from reviewer team many characteristics as described in Chapter 4 „Partners and resources“ by INTOSAI GUID 1900 – Peer Review Guidelines.
In our opinion the essential, even inevitable of successful PR are :
- Knowledge and experience of the peer reviewer team in the environment in which the reviewed SAI operate, another words the rules, principles and procedures described in the INTOSAI Professional Pronouncements;
- Members of team are professionally and personally mature, possess analytical way of thinking and working, comparative analysis, time management, project management, communication skills and other necessary soft skills;
- Properly planed and designed PR, should answer among others why, what, when. It is important to pay attention to the Chapter 5 “Planning” and Chapter 3 “Strategic Consideration” of INTOSAI GUID 1900 – Peer Review Guidelines and understand/know the Focus, Purpose, and Scope of The Peer Review.
What are the procedures?
Essential for obtaining the essence of the PR procedures is to acquire a necessary theoretical background. It is essential in this aspect to read thoroughly and „with understanding“ the INTOSAI GUID 1900 – Peer Review Guidelines. This material provides in our opinion the understandable instructions and information for both “planning” and “carry out phases” of PR giving inevitable insight to the Planning, Focus and Purpose, Reviewer ́s qualification, PR Report, Quality Assurance, Areas to be assessed, Questions to areas and others.
Other recommended material for thorough reading to learn about necessary items important to cope in the PR process is “Supreme Audit Institutions Performance Measurement Framework”. This way you may benefit among others from the holistic approach given in this material, from the complementarity and synergies between the items of “Supreme Audit Institutions Performance Measurement Framework” and INTOSAI GUID 1900 – Peer Review Guidelines and other useful information.
Should the SAI write in to Subcommittee on Peer Review?
We believe that it is fine to inform the Subcommittee on the intent to carry out the PR with indication the reasons to carry out the PR, and its purpose and scope. Subcommittee intents any way to collect among others the info on planned PR by SAIs of INTOSAI community in the forms of questionnaire sent in the 2years interval, the questionnaire will be sent in the end of y 2022. Should the SAIs deemed the undergone the PR or to be Reviewer even after the adequate preparation phase face the concrete particular problems, SAIs may write to Subcommittee of course. Subcommittee in such cases would try with the assistance of other members provide necessary instruction to handle the problematic situation.
Any guidance on selecting the peer for peer review?
The guidance is given in the chapter 4 „Partners and Resources“ of the INTOSAI GUID 1900 – Peer Review Guidelines. In addition, it could be helpful to have in the Reviewer ́s team some person from the same language area as reviewed SAI, eventually someone familiar with cultural, institutional, legal etc. insight from the country of reviewed SAI.
Can the SAI select the peer or will the peer be assigned by Committee?
Selection of peer is in full responsibility of SAI as described in the chapter 4 „Partners and Resources“ of the INTOSAI GUID 1900 – Peer Review Guidelines.
What are types of costs that are generally be borne by SAI which is keen for peer review?
The typical cost are connected with the translation documents, interpretation (if needed) travel and accommodation, however it is important to define entire cost connected with Peer Review. The structure of cost is described in part “Resources“ of the chapter 4 „Partners and Resources“ and part „How much cost implication“ of the chapter „Financial Memorandum“ by INTOSAI GUID 1900 – Peer Review Guidelines. The cost arrangement can be agreed many ways, the INTOSAI GUID 1900 – Peer Review Guidelines gives liberty of voluntariness in each of its elements, i tis not restricted only to the content provided and it is not binding. The terms and conditions for conducting a peer review, including costs, are usually contained in the relevant agreement between the peer reviewers and the SAI undergoing the peer review. Examples of such agreements are published on the CBC website here: https://www.intosaicbc.org/peer-review-materials/.
Questions after Presentation of GAO USA,
Answers by GAO USA to its presentation and to the general questions:
Eruma SAO Philippines
1)What assessment tool was used in the peer review of US GAO?
As required by Government Auditing Standards, the objectives of GAO’s last peer review were to determine whether, for the period under review, (1) GAO’s system of quality control was suitably designed and (2) GAO was complying with its quality control system in order to provide it with reasonably assurance of conforming with applicable professional standards. The peer review team developed its own tools to complete this assessment.
Chanda SAO India:
2)How are the themes or domains or focus areas determined before undertaking peer review?
The primary objectives of GAO’s last peer review were established in Government Auditing Standards: to determine whether, for the period under review, (1) GAO’s system of quality control was suitably designed and (2) GAO was complying with its quality control system in order to provide it with reasonably assurance of conforming with applicable professional standards. GAO’s peer review also included a review of GAO’s responses to suggestions from prior peer reviews. Any other focus areas to be included in the scope of the peer review would be incorporated into the memorandum of understanding between GAO and the peer review partner countries.
Mirtha Piscoya SAO PERU:
3) What did your SAI do with the outcomes of the peer review?
GAO established task teams to study the suggestions from the peer review team and to develop action plans for management to consider. Once approved by management, the action plans are then implemented.
General question related to the peer review scope:
Sainem SAO Indonesia
1) is it possible during the peer review, the peer scope is changed (reduced/ extended) according to the development of the peer review process and the need of the reviewed SAI?
In general, it is a good practice for the memorandum of understanding governing the peer review to establish the scope that is agreed upon by the reviewed SAI and its peer review partners, which can help clarify expectations about the conduct of the review. Any subsequent changes to the scope would then be agreed upon by all parties and documented in a revised memorandum of understanding.
2) the other challenge of peer review is the different “SAI context” between the reviewing SAI dan the reviewed SAI, that sometimes give rise to the different understanding and lead to the irrelevant recommendations. How to deal with this issue?
This question highlights the importance of good two-way communications between the reviewed entity and the peer review team throughout the peer review and during the development of the draft peer review report. First, the reviewed SAI can proactively communicate to the peer review partner countries important aspects of its own context that the peer review team should be aware of—for example, by comparing and contrasting the reviewed SAI’s context with the contexts of the partner SAIs. This can help the peer review team identify further differences that need to be understood and then tailor its work to be useful and constructive in view of the reviewed SAI’s particular context. Second, ongoing open two-way communications between the reviewed SAI and the peer review team throughout the process can help to illuminate different understandings earlier in the process and can provide a forum for addressing any potential disagreements. Third, the reviewed SAI can work with the partner SAIs from the beginning to ensure that the selected members of the peer review team have the requisite judgment, depth of knowledge, and breadth of experience to perform a contextually sophisticated peer review.
Questions after Joint Presentation SAO Austria, SAO Poland on Peer Review to SAO Romania and Presentation of SAO Romania to the same Peer Review
Aurelija Brukstute SAO Lituania
What the measurement framework has been used? Has this been agreed between your SAI and SAIs that performed peer review?
Answer by SAO Austria:
The SAI PMF 2016) has been used only in those points relating to quality in financial, compliance and performance audits. This was agreed in the Memorandum of Understanding.
Answer by SAO Romania:
As it was agreed between RCoA (reviewed) and the 3 SAI (reviewers), during the peer review, the Supreme Audit Institution Performance Measurement Framework (“SAI PMF”) dated October 24, 2016 was used as an inspiration, with its individual elements (indicators, domains and dimensions) selected by the peer review team according to the scope of the peer review exercise. The peer review was also be guided by Peer Review Guidelines (GUID 1900) of the INTOSAI Framework of Professional Pronouncements (IFPP).
Chanda SAO INDIA
How was the challenge of non digitalised audit information/files overcome ?
Answer by SAO Austria:
Most audit information was digitized. The reviewed SAI did digitize and translate relevant information.
Answer by SAO Romania:
Due to local conditions, an intense effort was needed to present audit files in a digitized format in order to be able to downloaded in the share point or delivered to the reviewing team. For example, many relevant documents from audit files needed to be scanned.
General question in the end of seminar by Ashraf A Hajjaj SAI Jordan :
Is it possible to publish the peer review Practical experiences on IDI web, especially the peer review of strategic plans, in addition can I inquire about the logistic sectors in the SAIs, are peer review applied to its policies and procedures and processes?
Answer by SAO Slovakia:
There are several Peer Review related to diverse areas published on: https://www.intosaicbc.org/peer- review-materials/ . It might happen that also those on strategic planning. We intend to increase the content and composition of different outcomes and material on Peer Review. Related the logistic sector, the Peer review may by applied to any operational aspects of SAIs, including policies, procedures and processed as described in the Peer Review Guidelines (GUID 1900). As explained in this guidelines, the reviewed areas mentioned in are not exhaustive and may vary and be increased depends on SAIs needs. Of course lot of useful inspiration for Peer Review could be obtain via IDI information sources, definitely we consider the SAI PMF by IDI as one of the instruments favorable for the Peer Review.