NWT evaluators ‘have been telling the wrong stories’

The way we decide if northern programs are working needs to change, experts said at a conference in Yellowknife last week.

Larry Bremner, former president of the Canadian Evaluation Society, urged Indigenous communities to take ownership of evaluating the programs delivered to their residents.

For years, funders and agencies have had trouble working out the best way to evaluate programs delivered to, and in partnership with, northern Canada’s small and remote communities.

With small numbers of participants and little with which to compare the programs, how do you measure their success or identify improvements?

“It’s time for Indigenous communities to start setting the evaluation agenda that suits them,” Bremner told Cabin Radio at the Canadian Evaluation Society’s Yellowknife symposium, held at the Chateau Nova Hotel.

“For too long, in Canada, you’ve had transient evaluators come into communities. There have been numerous studies and projects done, and little change has happened, because it was meaningless and not relevant to the community. It didn’t take into consideration the context of the community, or the different realities of different communities.

“Given the growing strength of nations, at some point in the near future, First Nations are going to realize they should be setting the evaluation agenda that suits their communities and their people.”

For Bremner, that also means funders developing the courage to allow communities to conduct their own evaluations.

“In some ways, it could be done in partnership, but my experience is there is always a stronger partner and it’s usually not the community,” he said.

“It’s time for evaluators to let go of some of the things they do. I also think it’s time for funders to realize that the communities have the knowledge. They are the experts in their own experiences, and we should be building on their knowledge as opposed to have a funder come in.

“I really believe evaluators are storytellers, but we have been telling the wrong stories because we have been basing it on the wrong things. Evaluators used to be completely fascinated with numbers, not stories. We based our stories on numbers as opposed to our lived experiences.”

‘Evaluation is medicine’

Dr Nicole Bowman, of the Mohican and Lunaape Nations, delivered the opening day’s keynote speech at the symposium.

Dr Bowman, an expert in culturally responsive evaluation, told Cabin Radio Indigenous communities have to embrace the concept of evaluation in order to conduct the work meaningfully themselves.

“Evaluation is medicine. Sometimes, when they hear the evaluator is coming, the Jaws theme plays,” she said.

“I’m sure some people take their evaluation punitively, but we are learning together; the process, as well as the product, is important. If we want the outcomes of fill-in-the-blank to change, we have to build it differently on the front end.”

Bowman added she believes Canada is “light years” ahead of her home, the United States, in its approach to Indigenous programming.

“Do you know what the treaties are? Could you name the First Nations? I can’t even talk about treaties because it’s not even acknowledged that we were here,” she said, talking about her experience in the US.

“Most mainstream folks know about First Nations and treaties up in Canada. I can’t even start there [in the US] with academics, let alone the general public. You know what sovereignty is, right? People down south don’t. We are starting with almost nothing.”

Bremner now represents EvalIndigenous, a group focusing on Indigenous approaches to evaluation of projects.

“As evaluators, we have to recognize that communities are not only experts in their own realities but they have evidence that’s meaningful to them that might not be meaningful to the funder,” he said.

“Instead of using clipboards, maybe we should be looking at Indigenous ways of knowing and looking at the world around us, and using some of the evidence they see as being legitimate for them, and incorporating that into our work or learning from them.”