Data as Medicine: Four Core Practices of Decolonising Evaluation

Nish Doshi – 5th June 2024

[W]hile evaluation reports largely tell stories of success, on the ground there is minimal change, communities remain impoverished, interventions cause harm to the environment, and evaluation allows that to happen.

Evaluation has the power to transform ideas into impactful projects. When conducted with cultural competence, and designed with and by the communities impacted, evaluation can help guide project managers and funders into developing and funding projects, which create real transformative change.

However, so often evaluation works as a means to market a project rather than provide opportunities to grow and learn from them. Evaluation criteria is shaped by Western values and measurements of success, and evaluation methods often fail to engage with the ways in which the project developers and participants understand and perceive themselves and the world around them.

How can we change this? We must decolonise evaluation practice.

Decolonising evaluation is a growing field of research and practice across the world, and this publication aims to show what, where and how that works in practise. We’ll be sharing case studies, interviews with practitioners, summaries of webinars, and some international context and resources to inspire and guide you.

While we can’t cover every aspect of decolonising evaluation in a short blog, we suggest starting by taking into consideration four core practices to transform and grow your evaluation process: cultural safety, epistemic justice, the politics of translation, and Indigenous data governance and sovereignty.

Cultural Safety

Observing that Maori communities often did not face adequate care in Aotearoa / New Zealand’s medical system, Irihapeti Ramsden and other Maori nurses pushed for institutional change, asking that all nurses be trained in Kawa Whakaruruhau or ‘cultural safety’.

As Ruth de Souza defines it, cultural safety “demands intentional self and organisational critique to look at how our behaviours, opinions and actions as people who work within institutions can negatively affect the cultural identity and wellbeing of the people we work with. Cultural safety offers arts, cultural and community workers the opportunity to examine how unquestioned ways of doing things can be barriers for groups to engage with us. Individuals and organisations can also use the idea to reflect on their work and address imbalances of power through building equal partnerships.”

Evaluators who are not from the communities of the project they are evaluating, or funders who are demanding evaluation could build in cultural safety into all of their work.

 

The First Nations Cultural Safety Framework

The Australian Evaluation Society has developed their own ‘First Nations Cultural Safety Framework’ – a collaborative process that engaged with and was shaped by the AES Indigenous Culture and Diversity Committee.

Developing on their principles, the framework provides evaluators and commissioning bodies to explore a series of questions about themselves and the evaluation process, centring that cultural safety is not simply on a check-box activity, but an on-going embodied process.

Epistemic Justice

When thinking about evaluation processes, it’s always important to ask “whose knowledge systems are being valued here?” Within evaluation systems, these are expressed through criteria of success, communication methods, and predefined concepts of what actually constitutes ‘knowledge’. Within academia, this has been explored through the concept of ‘epistemic injustice’ – which explores how identity and systems shape inequity in how knowledge is defined and valued.

It is important to recognise that evaluation is not a Western concept, and that all people and all cultures have had ways of reviewing, reflecting and learning from their actions and activities. How that is expressed and practised, however, can differ. If the body commissioning the evaluation or the evaluator is external to the community that is being supported, it is important that they explore the ways in which that community might already be practising and support an evaluation process that is shaped and defined by that community.

The danger in ignoring those systems of knowledge is that the resulting evaluation is false because the language and criteria used within the evaluation process do not align with the ways in which project participants perceive the world around them. Thus, communities can begin to see evaluation processes as surveillance, where the process works to police their project rather than support it in achieving the goals the participants actually need from it.

 

Visioning B.E.A.R. Circle Intertribal Coalition Inc

After finding that traditional models only advanced the harm against Indigenous communities, the Visioning B.E.A.R. Circle Intertribal Coalition developed a new method of evaluation. First they acknowledge how their training courses were designed alongside Indigenous and local values. Then they adapted the Principles-Focused Evaluation method to suit their ways of communicating, their power structures and their local knowledge.

Listen to the podcast above to hear Strong Oak explore how coalition came to this process, the reasons for it, and the challenges along the way.

Politics of Translation - Who Is Heard?

Participatory methods of evaluation are integral to decolonising practice, but it’s also important to ask who is telling the story of the impact of the project or programme – and who is actually being listened to and heard.

Power shapes who is heard and who is not. This is no different in evaluation processes, where the knowledge provided faces repeated practices of translation – first by the design of the process, then by the way in which the responses are understood, then how they are analysed, then how they are publicised or reported, and finally how those outside of the process understand that report. How much is lost in that process of translation?

Towards Embodied Evaluation Processes

As Ruby Quantson Davis argues, one method of decolonising evaluation processes is using an embodied practice. This centre the lived experience as a way of allowing people and participants of projects to determine and shape the evaluation experience.

Embodied knowledge understands that the way in which we learn and exist in the world is through our own bodies, and the connection those bodies have with the world around us. Embodied knowledge centres interdependence and relationship as way of analysis and interpretation.

In relation to evaluation, embodied evaluation processes are designed around understanding context, needs and power relationships. It allows for a deeper understanding of impact (and what impact actually matters to the people involved), one which gives space for the voices of the oppressed and place through which they can actually be heard and understood.

Indigenous Data Sovereignty and Governance

Indigenous data sovereignty is “indigenous peoples’ right to maintain, control, protect and develop their cultural heritage, traditional knowledge and traditional cultural expressions, as well as their right to maintain, control, protect and develop their intellectual property over these”.

Integral to providing data sovereignty is Indigenous data governance, which can be defined as:

Within the evaluation context, this is about power over the development of the process, data collection, data analysis and reporting. Instead of evaluation processes being extractive processes where an external body takes information and learning knowledge away from a community to report back to a grant-giving authority, the data is held by the community and they make the decisions on how and what to share.

Indigenous Innovation Initiative

Indigenous Knowledges are plural, and the ways in which evaluation, knowledge and data is collected and understood varies. That’s why the Indigenous Innovation Initiative spent a year slowly developing an evaluation framework – the Inquiry and Learning Bundle that worked for the communities they work with.

Understanding that Indigenous data has often been extracted and and used against the communities they work with, they also developed an Nindokiikayencikewin or the Indigenous Knowledges and Data Governance Protocol.

The image above is a simplification of the Protocol, which includes guidance on all stages of any evaluation processes, with practical guidance on values and ways of taking part in any evaluation process with the First Nation, Metis and Inuit peoples.