All Jobs Vacancy

Senior AI Evaluation Rubric Auditor (LLM Evals) - $60-$85/hr

Posted 3 days ago by Talent Wall

Role Overview

Mercor is hiring experienced AI Evaluation Rubric Auditors to oversee quality control on a project testing AI agents across realistic shopping and delivery workflows.

In this role, you will audit completed evaluation tasks (prompts and grading rubrics) produced by Project Reviewers to ensure fairness, structural accuracy, and rigorous data verification.

STRICT PREREQUISITE (DO NOT APPLY WITHOUT THIS)

You MUST have substantial prior experience writing or auditing grading rubrics for LLM evaluation / RLHF projects (e.g., Scale AI, Outlier, DataAnnotation, Mercor, etc.).

Generalist applicants without direct LLM rubric experience will not be considered.

You will be required to list your past evaluation projects in your application.

What You’ll Do

  • Rubric & Task Auditing: Review complete tasks (prompt + grading rubric) to determine if rubric failures are fair and defensible.
  • Root-Cause Analysis: Read long AI agent transcripts from end to end to pinpoint exactly where the model went wrong.
  • Fact-Checking: Verify information availability and check underlying data directly rather than blindly trusting reference answers.
  • Quality Calibration: Ensure rubrics accurately grade what the prompt asked for, follow structural guidelines, and would not mark a valid answer wrong.
  • Reviewer Feedback: Deliver clear, constructive, written feedback to prompt writers and reviewers to fix rubric flaws.

Candidate Profile

  • Proven Eval Experience: Substantial hands-on track record writing or reviewing grading rubrics for LLM evals.
  • Coaching & Feedback: Experience reviewing peers' work and providing actionable feedback to writers/reviewers.
  • Attention to Detail: Sharp analytical eye for reading long agent logs, spotting edge cases, and writing with extreme precision.
  • Availability: Consistent weekly availability as an independent contractor.

Eligible Locations

Must reside in and be authorized to work in one of the following countries: United States, United Kingdom, Canada, Argentina, Bolivia, Brazil, Chile, Colombia, Ecuador, Guyana, Paraguay, Peru, or Uruguay.

(Note: Unable to support H1-B or STEM OPT candidates in the US).

Compensation & Terms

  • Work Structure: 100% remote, independent contractor position on your own schedule.
  • Compensation: $60.00 to $85.00 per hour, depending on evaluation experience.
  • Payments: Processed weekly via Stripe or Wise based on services rendered.
Rate:
£85/hour
Location:
United Kingdom
IR35 Status:
Outside
Remote Status:
Remote
Industry:
AI & Machine Learning
Seniority Level:
Senior

Take-Home Pay

£12,000 per month

Visit calculators for additional details

Share job