Data Annotator | $30/hr Remote

Position: Search Generalists

Type: Hourly contract

Compensation: $10-$30 per hour

Location: Remote

Commitment: 10–40 hours/week

Role Responsibilities

• Evaluate AI-generated search responses for factual accuracy, helpfulness, clarity, completeness, and overall quality
• Assess whether models use search appropriately and whether search queries are well-formed and effective
• Compare model responses side by side and provide concise, defensible rationales
• Write and refine prompts, golden answers, rubric criteria, and edge cases for search-related evaluations
• Apply project guidelines consistently across ambiguous, multi-step, and real-world search tasks
• Identify recurring failure modes and escalate unclear cases or rubric gaps to project leads
• Participate in calibration, QA, and feedback loops to maintain strong agreement and quality standards
• Contribute to structured evaluation workflows to train, benchmark, and refine AI systems

Requirements

• Excellent written English and strong online research skills
• Strong judgment when synthesizing information from multiple sources
• Ability to distinguish factual accuracy from fluency, confidence, or style
• High attention to detail and comfort following structured guidelines
• Reliable, self-directed, and responsive in an asynchronous remote environment
• Experience in search quality, fact-checking, content evaluation, trust and safety, annotation, QA, or prompt/rubric writing preferred
• Familiarity with search evaluation concepts such as factuality, helpfulness, severity, side-by-side comparisons, or tool-use assessment
• Experience working with LLM evaluation workflows or human data projects
• Multilingual skills are a plus
• Bachelor’s degree preferred; advanced degree or strong professional background is a plus

Application Process (Takes 20 Mins)

• Upload resume
• Interview (15 min)
• Submit form