Case Study
Kosovo National Audit Office
The challenge
Since declaring independence from Serbia in 2008, Kosovo, a country of fewer than two million people, has made progress in a number of areas. It has held four peaceful elections, and voters are becoming more sophisticated, requiring political parties to develop clearer ideologies and plans to implement needed economic and governance reforms. For its part, the Government of Kosovo is making and enforcing its own policies, most notably in macroeconomic management and public financial management.
However, all is not well in Kosovo - the second poorest country in Europe - with around 17 percent of its people living below the poverty line, and unemployment rate having grown rapidly to stand at 52.4 percent (2019). Ref: https://www.cia.gov/library/publications/resources/the-world-factbook/geos/kv.html
The widespread poverty and inequality is especially clear in public education, public health, and other public services, contributing to general unemployment and a lack of economic opportunity to reduce the growing socio-economic gaps.
The lack of progress to achieve international recognition of Kosovo as a sovereign state is a concern that threatens to cause dissatisfaction. The UNDP’s Public Pulse (2018) publication “Challenges and Perspectives of Youth in Kosovo” pointed to corruption leading to low public confidence in many key government institutions.
The same report noted that healthcare is among the top issues facing Kosovo. The widening income gap between the rich and poor has highlighted the need to construct suitable policies to reduce poverty and narrow the income gap. One of the key ways to achieve this is by improving public policies and creating a better system for service delivery. The Kosovo Health Sector Strategy 2017-2021 considers mandatory health insurance premiums as a new opportunity to improve public service for those who are not served by private medical systems.
An audit of readiness and capabilities to implement such a system was required. A performance audit is quite different from a financial audit for example. Firstly, the institution needs to be innovation ready, then it needs to understand the policy systems, and finally, there needs to be ‘buy-in’ from other organisations, for example, to provide the required information.
With an innovative Auditor General, the NAO knew it needed to modernise, but it was a struggle to get the message downstream. Another concern was that the formal mechanisms for entities audited by NAO to provide feedback were not regarded well. As an example, EC funders’ performance auditors were not getting the right information to fund the proposed insurance scheme.
According to Besnik Osmani, Auditor General at KNAO: “The fundamental principle of the Kosovo National Office (KNAO) needed to change. The traditional and conservative approach must be left behind and people encouraged to think more about the organisational evolution of KNAO, and how the existing framework of performance auditing practice can be improved. New potential approaches should be identified and tested.”
The solution
There are however signs of optimism for the younger generation. To build on this, the Centre for Resilience and Sustainable Development (CRSD) at the University of Cambridge was asked in 2019 to partner with the National Audit Office of Kosovo and the University of Business and Technology (UBT) on a special initiative. The programme consisted of a pilot phase project aimed to promote systems thinking and systems practice to help decision makers in the country to make better and more inclusive decisions for Kosovo.
Besnik Osmani takes up the story:
“I asked CRSD and UBT for a special initiative to establish and lead a process, not to develop actual policy suggestions, but to provide a ‘safe’ place to practise the process of creating and delivering innovations. Through this initiative, we planned to offer participants a way of thinking about system dynamics and design to catalyze engagements between stakeholders, to create solutions that are sustainable, socially relevant and economically viable. This was a unique experience thinking out of the box and establishing a foundation for the future behaviour of KNAO people.”
The proposed Kosovo Health Insurance Fund was selected as the focus for the work. Using a methodology developed over many years at the Institute for Manufacturing, University of Cambridge, CRSD set about deploying two tools in particular.
The purpose of the Innovation Readiness Survey (IRS) is to explore the current state of critical thinking, disruptive forces, and the overall ability of the organisation to innovate. It does this by collecting perceptions of internal stakeholders at the participating organisation about the organisation itself.
IRS aims to unpack complexity and supports systems thinking later on when developing solutions to address the needs of innovation within the public sector, e.g. health systems.
The IRS then acts to inform the subsequent Cambridge Policy Simulation Lab (CPSL) stage, where a systems thinking approach is introduced to explore innovative ideas to address the existing challenges.
The Cambridge Policy Simulation Lab is designed to offer a ‘lifelike experience’ to leaders from public and private sectors and civil society to learn powerful new strategies for creating high impact policies to address key challenges.
The lab uses thinking about system dynamics and design to catalyse engagements between stakeholders, to create solutions that are sustainable, socially relevant, and economically viable.
The approach
Innovation Readiness Survey
In the IRS, there are four factors of innovation mindset - Readiness, Evaluation, Diffusion and Effects. These are explored using a mixed-methods survey tool.
Readiness - the relationship between people, processes, systems and performance measurement. A public sector organisation should have processes and people and relationships in place to coordinate the efforts and communicate changes.
The Evaluation stage weighs an idea’s pros and cons in the context of the existing systems and future problems that it needs to address. It narrows down potential ideas by allowing stakeholders to reject ideas and a public sector organisation can play an important role to find good policy ideas.
Diffusion explores how ideas are spread among groups of people. No innovations are adopted by all individuals in a social system at the same time, therefore, it is very important for a leader (change agent) to be able to identify adopter categories (e.g. early adopters, laggards) that can facilitate sustainable innovation over time.
Innovation only has Effects if its negative perceptions and impacts are considered as well as the history and context in which innovations are implemented. To generate a sustainable improvement and success, it is important to acknowledge and work collectively with stakeholders to overcome the sources of resistance to change by the public system, and not just reduce barriers to innovation.
CRSD believes that the necessary conditions for successful innovation are as shown in the figure below, based on:
(a) the innovation capability of the individual: a combination of their skills and other innate characteristics, together with their position and resources in their organisation, and
(b) the innovation capability of the hosting organisation: the combined effect of the people, processes and tools, plus
(c) interaction effects of Innovation Capabilities: between the person and the organisation.
Over a period of ten days in June 2019, this methodology was applied through the use of a custom-created set of questions. The survey was designed to take not more than 30 minutes to complete and was delivered electronically through a Qualtrics online platform.
Cambridge Policy Simulation Lab
The purpose of the Cambridge Policy Simulation Lab is not to develop actual policy suggestions, but rather to provide a ‘safe’ place to practise the process of creating and delivering innovations. Over three days in June 2019, presentations were made and well received by the stakeholder representatives. Both groups asked difficult questions about the sustainability of the Health Insurance Fund, which is not yet known, and there was an appetite for considering more detail about the topics discussed.
Many years of development have resulted in the CPSL methodology, made up of three key stages as described and shown in the figure below.
Sharing Understanding: the Creative Ideas Studio delivers an outcome where difficult issues are redefined, where different ideas are discussed and developed and where potential new approaches are identified and tested. A key feature of this process is that it encourages participants to connect apparently disparate ideas and facts and bring them together to map new possibilities and new directions.
Co-Creating Analysis: using the Cambridge Value Mapping Tool, this stage uses a structured and visual approach to identify ‘value uncaptured’ in the form of failed value exchanges: value missed, destroyed, surplus, and absence.
Co-Testing Innovation: the Institutional Feasibility Study framework combines a political economic framework with systems thinking to analyse how institutional capacities either align with or conflict with the socio-economic system to deliver outcomes. The IRS does not select from among competing or alternative theories, but offers a framework for critical systems thinking.
Using highly interactive sessions, key stakeholders worked together over three days in teams to develop creative ideas and to organise them into themes. The focus was on ensuring a common vision or goals that the participants could form a consensus around, to motivate the next session. Using the Cambridge Policy Value Mapping exercise, the participants created diagrams showing stakeholders for the Health Insurance Fund (HIF)
The conclusions
The Innovation Readiness Survey was circulated by the Auditor General’s Office among employees of the Kosovo National Audit Office, canvassing a wide range of topics. For example, the results showed good agreement about the goals for NAO and shared motivations for individual employees. This level of alignment about goals is a good precursor to organisational creativity and innovation.
An important part of the method was to gain stakeholders’ commitment and to give them the information and capability they needed to conduct a performance audit. The CPSL invited people from the NAO office and took them through the exercise to improve the performance audit capability.
The baseline information gathered here suggests that an Innovation Readiness Score may be gradually built up across different organisations in Kosovo, by making incremental changes and improvements to the prototype scoring methodology that has now been created.
The next step was to identify successful value exchanges among the entities in the system outlined with the stakeholder maps. Diagrams were created with a focus on the more important successful value exchanges. These included flows of information about prevention from private hospitals to citizens, exchange of funds between the Health Insurance Fund and citizens and exchange of data with the Data Protection Agency for validation.
In the Institutional Feasibility Study, the participants were encouraged to think more about the organisational evolution of NAO and how the existing framework of performance auditing practice can be improved. For more information on these methods, see:
https://www.crsd.landecon.cam.ac.uk/research-method
The impacts
‘Systems Thinking’ implies thinking about the world outside ourselves. ‘Systems Practice’ then uses analytical tools for working creatively and collectively with an internal and external network of stakeholders to initiate and guide actions. This research is about both systems thinking and systems practice, and about the relationship between the two as it plays out. The Innovation Readiness Survey (IRS) and Policy Simulation Lab (CPSL) bring together systems thinking and systems practice to co-create and co-learn powerful new strategies.
The intention is that the scoring methodology and the initial data collected in this pilot will serve as a datum at the organisational level that can be compared with other organisations in future, and guide potential improvements to the scoring process. This can then offer unique insight for Kosovo so that the comparative advantages of each country are understood within their own domain of expertise and culture.
The purpose of the Policy Simulation Lab was not to develop actual policy suggestions, but rather to provide a ‘safe’ place to practise the process of creating and delivering innovations in policy in general.
As a result of the performance audit, employees now have a better idea how they can implement such audits. The NAO can show the EC that it is working to improve their procedures. This has contributed to their acceptance into the International Organization of Supreme Audit Institutions (INTOSAI) and the European Organization of Supreme Audit Institutions (EUROSAI). European International Office of Auditors 2021.
1. Challenge the norm: Systems thinking and policy systems inquiry methods were introduced to improve performance audits, an exercise which at the time of the workshop was new to the KNAO.
2. Deliberative space: The ease of using CPSL in a practical way was demonstrated by taking on a complex problem of the country, i.e. access to health care insurance.
3. Leadership mindset change: Confidence was improved among the new leadership and the management leading them to apply to become a member of the International Organization of Supreme Audit Institutions (INTOSAI) and the European Organization of Supreme Audit Institutions (EUROSAI). INTOSAI and EUROSAI provide platforms for Supreme Audit Institutions to exchange expertise and best practices that contribute to increased accountability and transparency and add value and benefits to the lives of citizens worldwide.
4. Future research: Insights were gained into how both the IRS and the CPSL could be improved through the use of an improved scoring methodology and participant selection respectively.
