Hiring the wrong evaluation firm can waste months and thousands of dollars while leaving your nonprofit with unusable data. Before you sign a contract, knowing what to watch for separates solid partners from firms that overpromise and underdeliver. Here's what actually matters when vetting impact evaluation companies.
They Can't Clearly Explain Their Methodology
Red flag: A firm talks around how they'll actually measure your outcomes. Vague language like "we use a holistic approach" or "mixed-method analysis" without specifics means they haven't thought through your program's logic model yet.
Ask them to walk you through a comparable past project. What metrics did they track? Did they use pre/post surveys, control groups, observational data, or administrative records? What was their sample size, and how did they handle missing data? If they struggle to answer these questions or hand you templated case studies that don't match your sector, move on.
A legitimate firm should spend 2–3 weeks on discovery before proposing any evaluation design. They'll ask about your program's actual inputs, activities, and intended outcomes—not assume a one-size-fits-all approach works.
Their Pricing is Mysteriously Low (or Wildly Unclear)
Impact evaluations typically cost $15,000–$75,000+ depending on scope, sample size, and timeline. If a firm quotes $5,000 for a rigorous three-year evaluation, they're either cutting corners or planning to upsell you later.
Request a detailed line-item budget that separates staff time, data collection, analysis, and reporting. Watch for vague categories like "consulting fees" without breakdown. Legitimate firms itemize everything: survey administration costs per respondent, database licensing, statistical analysis hours, and revision rounds included in the base price.
Also ask what happens if your program reaches fewer participants than projected. Some firms lock in prices regardless; others charge per-respondent and scale. Know the difference before you commit.
They Don't Have Direct Experience in Your Sector
A firm that's strong in workforce development may struggle with education or health outcomes. Sectoral knowledge matters because outcome proxies, standard metrics, and funder expectations differ widely.
Check their portfolio specifically for nonprofits or programs similar to yours in mission and scale. A boutique youth mentorship program shouldn't hire a firm whose experience is primarily large-scale government contracts. Conversely, a national network needs firms comfortable with multi-site complexity and comparative analysis.
Ask for client references from similar-sized organizations in your field—and actually call them. Ask how responsive the firm was during the evaluation, whether they delivered on time, and if the final report was actually useful for decision-making.
They Treat Data Collection as Your Job
Some firms design beautiful evaluation plans, then expect your staff to administer surveys, track data, and manage databases. This creates massive friction and often results in incomplete or inconsistent data.
Your team should focus on program delivery. A reputable evaluation firm either:
- Handles primary data collection themselves (surveys, interviews, focus groups)
- Provides clear, simple tools your staff uses with minimal training
- Offers ongoing support if staff-administered tools are necessary
- Includes quality assurance to catch data errors early
If a proposal says "your organization will be responsible for survey administration," ask why they're not doing it and what happens if response rates tank.
They Promise Perfect Attribution
No evaluation can definitively prove your program caused outcomes without a randomized controlled trial—and even those have limitations. Firms that claim they'll show 100% impact attribution are overselling.
Realistic firms discuss what's achievable: pre/post comparisons, matched comparison groups, regression analysis, or qualitative evidence of change pathways. They'll be honest about internal validity threats and explain why their chosen design fits your budget and timeline.
They Don't Talk About Sustainability
A good evaluation firm should help you build capacity to continue measuring impact beyond their contract. That means documenting data collection procedures, training staff on tools, and ensuring your systems work without them.
Ask whether they'll provide training materials, data templates, or ongoing technical support. Will they license you to use their tools independently, or are you locked into paying them annually? A firm invested in your long-term success discusses sustainability from the start.
Frequently Asked Questions
Q: What's a realistic timeline for a baseline evaluation? A solid baseline typically takes 3–4 months from contract signature through final reporting, including stakeholder interviews, survey design, data collection, and analysis.
Q: Should I use the same firm for baseline and endline evaluation? Continuity is valuable—the firm already knows your program—but competitive bidding at endline ensures you're getting fair pricing and fresh perspective.
Q: How do I know if a firm's proposed sample size is adequate? A reputable evaluator justifies sample size with statistical power calculations and explains trade-offs between rigor and cost; they won't just pick a number arbitrarily.
Ready to find an evaluation partner? Mercoly helps you compare and vet impact measurement firms in one place—see their methodologies, pricing, and past work side by side.