When you commission an impact evaluation, you're paying $15,000–$100,000+ for someone to measure whether your program actually worked. If that evaluator doesn't understand your community's culture, assets, and communication norms, your findings will be skewed, your stakeholders won't trust the results, and your organization loses credibility where it matters most.
Why Cultural Competency Matters in Impact Evaluation
Impact evaluators operate in a unique position: they're outsiders tasked with measuring deeply contextual human change. A culturally incompetent evaluator might mistake silence in focus groups for agreement (when it's actually respect for authority), misinterpret kinship networks as nepotism, or rely on Western metrics that don't capture the values your community actually prioritizes. The result is an evaluation that looks methodologically sound but measures the wrong things.
This isn't about performative diversity statements. It's about whether an evaluator can design and execute an assessment that actually reflects what success means to the people you serve.
Red Flags When Vetting Evaluators
Watch for these warning signs before signing a contract:
- One-size-fits-all methodology. If an evaluator proposes the same survey instrument and timeline for a rural agricultural program and an urban youth initiative, they're not adapting to context. Ask how they'll customize their approach for your specific community.
- No prior work with your demographic. Request case studies or references from programs serving similar populations. An evaluator with expertise in immigrant-led organizations shouldn't be your first choice if you run a Native American health initiative.
- Vague staffing plans. Will community members be part of the evaluation team? How many staff members will actually spend time in the field versus analyzing data remotely? Culturally competent evaluation requires presence and relationship-building.
- Dismissing local knowledge. If an evaluator resists input from your staff and program participants on what outcomes matter, that's a dealbreaker. The best evaluations are collaborative, not extractive.
- No plan for language access. Even if your primary program language is English, are surveys, interviews, and reports available in the languages your participants speak at home? This costs 8–12% more but is non-negotiable.
What to Ask (and Listen For)
During the proposal process, ask these specific questions:
On cultural engagement: "Walk me through how you'd adapt your data collection method for our specific community. What would you do differently than with a mainstream audience?" Listen for whether they ask clarifying questions about your community first, or jump straight to describing their standard process.
On team composition: "What percentage of your evaluation team will have lived experience or professional background in [your sector/demographic]? Will any team members be from the communities we serve?" Aim for at least 40–50% of the core team having direct cultural connection.
On validity: "How will you ensure your findings are culturally valid, not just statistically valid?" They should discuss member checking (showing draft findings back to community members for feedback), using community advisory boards, or employing culturally-grounded frameworks like social determinants or strengths-based approaches.
On adaptation: "If our program participants tell you that your survey questions don't make sense, how will you handle that?" Strong evaluators build in flexibility and won't treat the protocol as untouchable scripture.
Practical Vetting Checklist
Before hiring, verify these concrete elements:
- Review their last 3 evaluation reports. Are case studies and participant quotes included, or is it all tables and regression analysis? Culturally competent reports make community voice visible.
- Request references from at least two programs serving similar populations. Call those references and ask: "Did they ask you what success looked like before designing their metrics?"
- Check whether they're affiliated with university partnerships or community-based organizations in your sector. This signals both rigor and relatedness.
- Get a detailed budget breakdown. Culturally competent evaluation costs more because it requires longer field time, translation, and collaborative analysis. If the price seems too low (under $12,000 for a meaningful evaluation), ask why.
Frequently Asked Questions
Q: Does the evaluator have to be from the same cultural background as our participants? No, but they need demonstrated experience working across difference, willingness to learn, and accountability to community voices—often formalized through a community advisory committee that reviews findings.
Q: How do we know if the evaluation uses Western-centric metrics that miss what matters to us? Ask for the logic model or outcomes framework early in the process; workshop it with program staff and community members to ensure the metrics align with your values, not just funder requirements.
Q: What's a realistic timeline for a culturally competent evaluation? Expect 6–12 months for a solid evaluation that includes relationship-building, iterative design, and community feedback—longer than a hit-and-run assessment, but worth the investment.
Use Mercoly to compare impact measurement providers who explicitly list cultural competency experience and can show you their actual community-engaged work.