Data annotation quotes can vary wildly—sometimes by 10x—making it tempting to grab the cheapest option and move on. But cutting corners on labeling quality will haunt your model's performance later, so knowing what to watch for upfront saves thousands in rework.
Price That Seems Too Good to Be True
If a vendor quotes $0.05 per image when competitors are at $0.15–$0.30, that's a red flag. Rock-bottom pricing often indicates corners are being cut: rushed annotators, minimal QA checks, or offshoring to regions where turnaround isn't properly supervised. For complex tasks like 3D bounding box annotation or medical image segmentation, expect to pay $0.50–$2.00+ per sample depending on complexity.
Ask vendors directly: What does your QA process look like, and how many review cycles are included? A legitimate provider will detail their inter-annotator agreement thresholds (typically 85%+ for most tasks) and won't dodge the question.
Vague Scope and Task Definition
Some vendors quote without asking detailed questions about your annotation requirements. This is dangerous. They might assume simple image classification when you actually need multi-class instance segmentation. By the time you realize the mismatch, you've paid for unusable work.
Red flags include:
- Quotes that don't itemize complexity (e.g., "images $0.10 each" with no mention of label count, polygon vs. box, or confidence levels needed)
- No mention of your specific domain (medical, autonomous vehicle, retail) even though complexity varies dramatically across industries
- Refusal to do a small pilot before committing to 10,000+ samples
Get specific upfront. Provide a representative sample of 10–20 items and ask the vendor to annotate and price that precisely—not a generic estimate.
No Quality Guarantees or SLAs
Reputable annotation vendors commit to measurable quality targets. If they won't specify an inter-annotator agreement percentage, accuracy baseline, or timeline, that's a major warning sign. You need contractual language that covers what happens if delivered work doesn't meet agreed standards—do they re-annotate at no cost, or do you eat the loss?
Ask for their Service Level Agreement (SLA) in writing, covering:
- Minimum accuracy or agreement thresholds
- Turnaround time (e.g., 5 business days for 1,000 images)
- Rework policy if quality drops below spec
- Data retention and security commitments
Lack of Annotator Training Details
The quality of your labels depends entirely on who's doing the work and how they're trained. Vendors who can't or won't explain their onboarding process are cutting corners. Proper training for complex tasks—like 3D point cloud labeling or fine-grained object detection—takes days, not hours.
Ask:
- Do you require domain expertise or certifications (e.g., for medical imaging)?
- How long is the typical training period for this task?
- What's your annotator retention rate? (High turnover = inconsistent quality.)
A vendor confident in their process will have clear answers.
Hidden Fees and Scope Creep
A $5,000 quote can balloon to $8,000 once "revision requests" or "additional annotations" are added. Some vendors charge separately for quality assurance, metadata extraction, or format conversion—costs that should be transparent upfront.
Review the quote carefully for:
- Setup or project management fees (typically 5–15% of the total)
- QA costs (should be built in, not added later)
- Revision rounds (clarify: are 2 rounds included, or is each round extra?)
- Export and delivery format charges (some vendors charge for JSON, COCO format, etc.)
Request a detailed line-item breakdown before signing anything.
Inability to Scale or Adapt
Your initial batch might be 5,000 images, but if the model works, you'll need 50,000. Ask vendors whether they can scale without sacrificing quality or timeline. Vendors with fixed teams often can't; those with flexible crowd-sourced pools usually can.
Also check their flexibility on changes. If you realize mid-project that you need a third label class or stricter guidelines, can they adapt? Inflexible vendors will charge change fees or reject modifications outright.
Frequently Asked Questions
Q: How do I know if an annotator's quality is actually good? Request inter-annotator agreement metrics (Cohen's kappa or Fleiss's kappa scores of 0.80+) and ask for a sample of their previous work with benchmark accuracy, or run a small validation set yourself against their output before committing to the full volume.
Q: What's a realistic timeline for annotating 10,000 images? For simple classification, 5–10 business days; for bounding boxes or polygons, 2–3 weeks; for 3D annotation or medical imaging, 4+ weeks—timelines scale with complexity and team size, so confirm your vendor's capacity upfront.
Q: Should I use one vendor or split across multiple? Splitting across 2–3 vendors reduces dependency risk and lets you compare quality, but adds coordination overhead; use one vendor for small pilots (<1,000 samples) and consider splitting only if your project is large (>50,000 samples) or mission-critical.
Compare vetted data annotation providers side-by-side on Mercoly to find the right fit for your project requirements and budget.