When you hire an evaluator to measure your nonprofit's long-term impact, you're often investing $15,000–$75,000+ for a multi-year engagement. Asking the right questions upfront determines whether you get defensible insights or a report that gathers dust on a shelf. Here's what separates a strong evaluator from one who'll leave you guessing.
Know Their Measurement Framework
Ask potential evaluators exactly which outcome model they'll use: theory of change, logic model, results-based accountability, or something else. A credible evaluator should be able to explain in plain language how they'll connect your activities to long-term change. For example, if you run a youth employment program, they should articulate how they'll measure not just job placement (short-term) but wage progression and job retention at year three (long-term).
Push back if their answer is vague. Weak answers sound like "we'll measure what matters to your stakeholders"—that's a starting point, not a methodology.
Ask About Attribution vs. Contribution
Long-term impact evaluation lives in murky territory: Did your program cause the outcome, or did other factors contribute? Ask evaluators explicitly how they'll handle attribution.
Key questions:
- Will they use a comparison group (control or quasi-experimental design)?
- Are they comfortable with contribution analysis for complex programs?
- What's their honest assessment of what you can confidently claim vs. what's speculative?
A strong evaluator will say something like: "We can't prove causation without a comparison group, so we'll triangulate survey data, beneficiary narratives, and administrative records to build a credible contribution story." Weak evaluators oversell certainty.
Clarify Data Collection Over Time
Tracking people for three to five years costs money and effort. Ask how they'll manage attrition—the reality that some participants disappear from your data.
Expect them to discuss:
- Follow-up survey response rates (realistic targets: 60–75% for year two, 40–60% for year three)
- Methods to re-engage lost participants (tracing, incentives, multiple contact attempts)
- How they'll handle missing data statistically
- Whether they'll combine quantitative surveys with qualitative case studies to offset low response rates
If they gloss over attrition, they're not planning realistically.
Probe Their Data Infrastructure
Ask what systems they'll use to store, track, and link participant data across years. Your program probably has a database; their evaluation needs to integrate with it—or they'll track data separately, which doubles the workload.
Questions to ask:
- Do they use secure cloud platforms (Salesforce, REDCap, Qualtrics)?
- How will they match baseline and follow-up data?
- What's their protocol for protecting personally identifiable information?
- Who owns the data—you or them?
A mature evaluator has this conversation early and gives you copies of your own data.
Budget for Realistic Costs
Long-term impact evaluation isn't cheap. A multi-year study for a program serving 200+ participants typically runs:
- Small scope (quantitative surveys only): $20,000–$40,000
- Medium scope (surveys + some qualitative interviews): $40,000–$60,000
- Complex scope (comparison group, longitudinal tracking, mixed methods): $60,000–$100,000+
Ask evaluators to break down costs by year and activity. Cheaper isn't better—a $10,000 evaluation will cut corners on follow-up and rigor.
Request a Realistic Timeline
Long-term doesn't mean indefinitely long. Ask evaluators:
- When will baseline data collection start relative to program launch?
- What's the minimum follow-up period to claim "long-term" impact (most funders want at least 12 months post-program)?
- When will preliminary findings be available vs. final reports?
- How will they communicate mid-stream learnings?
A credible timeline includes built-in buffer for participant attrition and data cleaning. If someone promises final results exactly on schedule, they're not accounting for real-world messiness.
Check Their Experience with Your Sector
Finally, ask for references from nonprofits similar to yours—not just by size, but by program type. An evaluator strong in education outcomes might struggle with workforce development or health behavior change. Ask directly: "How many evaluations have you completed in [your sector]?" Aim for at least three relevant examples.
If you're comparing evaluators, Mercoly helps you side-by-side evaluate Impact Measurement & Evaluation providers, making it easier to spot differences in methodology and experience.
Frequently Asked Questions
Q: What's the difference between formative and summative evaluation for long-term impact? Formative evaluation happens during program delivery and tells you what's working (for course correction), while summative evaluation happens at the end and measures whether you achieved long-term outcomes. Many strong evaluators do both.
Q: Can I measure long-term impact without hiring an external evaluator? You can conduct internal tracking, but external evaluators bring credibility to funders and the public because they're independent; for rigorous long-term claims (especially with comparison groups), external expertise typically costs less than the reputational risk of weak data.
Q: How do I know if my follow-up period is long enough to claim "long-term" impact? It depends on your outcome: six months is long-term for behavior change, but two to three years is standard for employment and education outcomes; ask your evaluator what peer-reviewed research suggests for your specific area.
Start your evaluator search with clear criteria in hand, and you'll invest in measurement that actually informs strategy rather than just checks a funder box.