Poor annotation quality can tank your AI model before it leaves the lab. Bad labels propagate through training, creating models that fail in production—and no amount of fine-tuning fixes garbage inputs. Knowing which red flags signal low-quality annotation services saves you weeks of rework and thousands in wasted spend.
Vendor Claims Accuracy Without Methodology
Any annotation provider guaranteeing 95%+ accuracy without explaining how they measure it is dodging accountability. Legitimate vendors detail their inter-annotator agreement (IAA) scores, explain their quality assurance process, and break down accuracy per annotation type. They'll tell you whether they use Cohen's kappa, Fleiss' kappa, or F1 scores—metrics specific to your task.
Ask vendors directly: "What's your measured IAA score for tasks like mine, and how do you calculate it?" If they deflect or cite generic percentages, move on.
Turnaround Times That Seem Unrealistic
A vendor promising 10,000 labeled images in 3 days for object detection should raise suspicion. At typical annotation speeds—8–12 objects per image, plus review—that's physically unsustainable without cutting corners. Quality annotation takes time.
Realistic timelines depend on task complexity:
- Simple classification (10 classes): 500–1,000 items/annotator/day
- Bounding boxes (5–10 objects per image): 100–300 items/annotator/day
- Segmentation masks: 20–80 items/annotator/day
- Entity extraction (documents): 5–20 documents/annotator/day
If quoted timelines significantly beat these ranges, the vendor is either understaffed or skipping review cycles.
No Clear Documentation or Style Guides
Good annotation hinges on consistency, which requires detailed guidelines. Before signing a contract, request the annotation guidelines the vendor will use for your project. A genuine provider has:
- Definitions for each label with edge-case examples
- Sample annotations showing correct vs. incorrect work
- Rules for ambiguous cases (e.g., "partially visible object = label or skip?")
- Team-specific training materials
If they offer a generic template or say "we'll figure it out after kick-off," they haven't planned properly.
Unclear Team Composition and Training
You should know who's annotating. Reputable services provide:
- Annotator profiles: background, experience level, languages spoken
- Training process: how new annotators ramp up on your specific task
- Specialization: whether annotators focus on your domain (medical, autonomous driving, e-commerce)
A team of general annotators with no domain knowledge will produce inconsistent labels. A linguistics expert annotating drone footage is misaligned. Ask whether the vendor trains annotators specifically for your project or reuses generic templates.
Weak Review and Iteration Practices
Single-pass annotation is a red flag. Quality control requires multiple checkpoints:
- Annotator self-review (5% of work)
- Lead reviewer QA (15–25% of work)
- Reconciliation of disagreements between annotators
- Client feedback loop (iterative refinement based on your feedback)
Ask: "What percentage of work gets reviewed before delivery?" Anything under 10% suggests minimal QA. Also ask about their revision policy—good vendors absorb the cost of rework if quality falls below agreed thresholds (typically 90–95% accuracy on sample batches).
Pricing With No Breakdown or Accountability
Vague per-unit pricing ("$0.50 per image") without context invites surprises. Understand:
- Does complexity affect pricing? A labeled image with 2 objects shouldn't cost the same as one with 20.
- Are revisions included or extra? Quality providers usually bundle one round; additional rounds cost more.
- What's the minimum project size? Some vendors require 1,000+ units.
Typical ranges: $0.10–$1.00 per simple classification; $0.50–$3.00 per bounding box image; $2.00–$10.00 per segmentation mask. Anything significantly cheaper likely cuts corners; anything more expensive should justify specialization (rare domains, languages).
Missing Contractual Protections
Before engaging, secure:
- SLA on accuracy (e.g., "95% of labels pass validation, or we rework free")
- Data security clauses (encryption, NDAs, IP ownership)
- Dispute resolution (how disagreements over quality get resolved)
- Timeline penalties (what happens if they miss deadlines)
If they push back on accountability language, they're not confident in their work.
When comparing vendors, platforms like Mercoly let you filter by methodology, team expertise, and past client feedback—cutting the research time significantly.
Frequently Asked Questions
Q: What inter-annotator agreement score should I demand? A: For most tasks, aim for 0.80+ (Cohen's or Fleiss' kappa). Medical, legal, or safety-critical work should target 0.85+. Below 0.75 signals the task definition is unclear or the team isn't aligned.
Q: How much should I budget for QA review on top of annotation costs? A: Plan 20–30% overhead. If annotation costs $5,000, reserve $1,000–$1,500 for lead-reviewer QA, reconciliation, and revisions to ensure quality.
Q: Should I annotate a small pilot batch first? A: Yes. Always run 100–500 items through a vendor before scaling. It reveals methodology gaps, clarifies edge cases, and lets you validate quality before bulk commitment.
Use Mercoly to compare annotation vendors by methodology, pricing transparency, and client reviews in one place.