Submitted by: Joy Kaufman, Michelle Abraczinskas, Ida Salusky
Consumer researcher-led evaluation is a tool for professional evaluators that can lead to authentic and actionable findings.
The benefits and relevance of CBPR outweigh the difficulties of using the approach.
Community-based Participatory Research (CBPR), where consumers participate in the design and execution of a research study or an evaluation, holds promise for increasing the validity and usefulness of evaluations of services. However, there is no literature comparing methods and outcomes of studies conducted by professional evaluators with those conducted through a consumer-driven evaluation process.
While there are struggles inherent in engaging in CBPR, we believe that including consumer researchers as partners in the evaluation of services they receive is essential. Inclusion of consumers is necessary to understand the experience of receiving services and the impact of the services to guide system improvements that authentically represent the consumers’ needs and desires. Specifically, the presentation of the actionable findings by the consumer researcher team can motivate change agents to improve the service system in a way that directly benefits consumers. Consumers can also see tangible evidence that their concerns were heard. In addition to system level changes, consumer participation in a CBPR based evaluation can lead to personal growth, such as a recommitment to academic or career goals. We believe that the benefits of community-based participatory evaluation strategies far outweigh difficulties, especially since the changes are being driven by consumers, for consumers.
“Data generated by the consumer research team was unique, directly relevant to the community, actionable, and invigorated stakeholders at the systems level to take action. In contrast, data generated by the traditional evaluators was a replication of what was found in previous work and was broader, which may have contributed to the inaction at the system level.”
We present the methods and results from a qualitative evaluation conducted by professional evaluators along with one conducted by a team of consumer researchers who engaged in a CBPR process. Each was led by the same external evaluator within a system of care. Each of these evaluations sought to identify the barriers and facilitators to families accessing social services within a resource poor community. In the paper, we present: 1) the convergent and divergent methods and findings that emerged from these two evaluations; 2) the technical aspects of training consumer researchers, who came from different economic and educational backgrounds than the professional evaluators, to conduct focus groups; and, 3) lessons learned about how to prepare for, and work with, common barriers to CBPR implementation.
How Did A Community Psychology Perspective Inform Your Work?
As Community Psychologists we value: 1) the essential knowledge that our community partners bring to the process, 2) providing training so that our community partners have the skills to engage in this work, 3) the importance of context and lived experience in coding and analyzing the data, 4) the importance of relinquishing power so that our community partners own the process and 5) that engaging in CPBR promotes social justice within under-resourced and disenfranchised communities.
There were differences in results obtained by the professional evaluators and the community researchers. These could be explained by the following:
- The way that the questions were worded by the consumer researchers was more accessible to the community members when compared to the questions posed by the evaluators.
- Interacting with familiar individuals may have provided a level of comfort not present when interacting with the professional evaluation team.
- Consumer researchers’ lived experience allowed for fewer researcher perspective biases in analyzing the data. This brought a contextualized perspective resulting in a more complete and nuanced understanding of the data.
While CBPR yields data that is authentic, useful, and can directly impact service delivery systems, the process is time intensive and can be challenging. Including consumer researchers as partners in the evaluation of services is essential to understand the experience of receiving services and the impact of those services when considering improvements to service provision.
What Does This Mean For?
Research and Evaluation: The comparison presented in this paper is descriptive; however, we believe that it is a first step in identifying the potential differences in the methods and outcomes between a professional-led and a consumer-led evaluation process. Future research is needed to examine the unique benefits and drawbacks of the two approaches.
Social Action: Our team has successfully trained multiple groups of consumer researchers including parents of children receiving social services and adult consumers of publicly funded psychiatric services. Each of these groups have utilized the data they collected to successfully advocate for changes in how services are delivered. The process of service recipients gathering and analyzing input from their peers and using the data for system changes reduces the power imbalance between those who deliver and those who receive publicly funded social services.
Original Citation: Kaufman, JS; Abraczinskas, M; Salusky, I. (2020). Tell it to me straight: The benefits (and struggles) of a consumer driven evaluation process. American Journal of Community Psychology, 65(1-2), 125-135.