Choosing a data annotation vendor is a high-stakes decision—poor labeling quality tanks model performance, while a reliable partner accelerates your AI roadmap. Before signing a contract, talking to existing customers of that vendor reveals what you can't learn from a sales call alone. Here's how to vet references strategically and ask the right questions.
Why References Matter for Annotation Services
A vendor's reference list shows real-world performance across different annotation types, dataset sizes, and quality standards. Unlike generic software demos, data labeling is hands-on work where consistency, accuracy, and domain expertise directly impact your model's ability to learn. References give you visibility into turnaround times, how the vendor handles edge cases, and whether they scale smoothly as your workload grows.
What Type of References to Request
Ask for references in your specific domain and use case—a vendor strong in medical image labeling may struggle with fine-grained object detection or NLP tasks. Request at least three references that match your project scope (dataset size, annotation complexity, timeline). If possible, ask for one reference from a company similar to yours in industry or stage. A Fortune 500 company's experience with bulk annotation won't necessarily predict how the vendor handles a mid-market startup's iterative labeling needs.
Key Questions to Ask References
On Quality and Accuracy
Ask: "What inter-annotator agreement score did you achieve, and was it consistent throughout the project?" Most professional annotation vendors target IAA scores of 0.75–0.85+ depending on task complexity, but how they maintain it matters. Ask whether they had to reject batches or go through multiple rounds of refinement, and how transparent the vendor was about quality issues as they emerged.
On Scalability and Timeline
Ask: "Did they deliver on schedule, and how did they handle scope changes?" A vendor can promise 10,000 labeled images in two weeks, but what happens when you discover halfway through that your annotation guidelines need clarification? References reveal whether a vendor builds in buffer time, communicates proactively, or leaves you scrambling. Also ask if they had to spin up additional annotators mid-project—that's normal, but you want to know if quality held steady.
On Communication and Process
Ask: "How often did you get status updates, and could you track progress in real time?" Some vendors batch-deliver results; others provide weekly dashboards. If you need to iterate or pivot your labels, you want a vendor who responds quickly, not one where feedback takes weeks to filter through project management layers.
On Cost and Budget
Ask: "Did the final cost match the quote, and were there surprises?" Clarify what's included in the per-unit price. A vendor charging $0.15 per image might seem cheaper than one at $0.22, but if the first vendor has higher rejection rates or slower revision cycles, you're actually paying more overall.
On Edge Cases and Problem-Solving
Ask: "What happened when you ran into ambiguous labels or data quality issues?" This reveals whether the vendor has experience in your specific domain and can make judgment calls without constant escalation to you.
Red Flags in Reference Conversations
Listen for hesitation or vague answers about quality metrics, turnaround, or cost overruns. If a reference says "the vendor was fine but communication could've been better," that's meaningful—fine isn't good for AI labeling. Also notice if references mention a high employee turnover at the annotation vendor (a common problem in high-volume labeling shops), which often correlates with inconsistent output quality.
Making the Most of Your Diligence
Take notes on each reference call and score them across three dimensions: quality consistency, responsiveness, and cost predictability. If you're comparing multiple vendors, this gives you a repeatable framework. Tools like Mercoly help you compare and find trusted data annotation providers in one place, streamlining your reference and vendor evaluation process.
Frequently Asked Questions
Q: What's a realistic inter-annotator agreement score I should target? For most classification and detection tasks, 0.75–0.85 is standard; medical or legal domains often require 0.85+. Ask references what score they achieved and whether the vendor had to relaunch or revise guidelines to hit it.
Q: How long should a typical annotation project take? A 10,000-image object detection project usually takes 3–6 weeks depending on complexity and annotator availability; NLP tasks vary widely by token count and task difficulty. References can tell you what's realistic for your specific workload.
Q: What happens if I need to pause or cancel mid-project? Ask references whether the vendor had flexible terms and how they handled partial-project payments. Most professional vendors charge for completed work but allow cancellation without penalty on unstarted batches.
Use these conversations to find a vendor you can trust with your training data—it's an investment that compounds over time.