All Jobs Vacancy

AI Evaluation Rubric Auditor | $60-$85/Hour

Posted 4 days ago by Hirely

Responsibilities

  • Audit complete AI evaluation tasks, including prompts, grading rubrics, and model attempts.
  • Assess whether identified model failures are genuinely justified.
  • Verify whether the required information was available to the AI agent.
  • Identify inaccurate, ambiguous, or unfair grading criteria.
  • Determine whether valid model responses could incorrectly receive failing grades.
  • Check rubric structure and compliance with evaluation guidelines.
  • Identify tasks that cannot be reasonably corrected.
  • Provide clear, constructive feedback to reviewers on their decisions.
  • Maintain consistent auditing quality and availability.

Requirements

  • Substantial experience writing, reviewing, or auditing LLM evaluation rubrics.
  • Previous experience in AI evaluation, RLHF, model training, or AI quality assurance.
  • Experience providing feedback to writers, reviewers, or evaluators.
  • Strong analytical and critical-thinking abilities.
  • Excellent written English and attention to detail.
  • Ability to analyze lengthy AI agent transcripts and identify specific errors.
  • Ability to independently verify information rather than blindly trusting reference answers.
  • Consistent availability and ability to meet project expectations.

Relevant Experience

  • Candidates with experience in the following areas are encouraged to apply:
  • LLM Evaluation and Benchmarking
  • Rubric Design and Auditing
  • RLHF and Human Feedback
  • AI Quality Assurance
  • AI Model Assessment
  • Reviewer Quality Control
  • Prompt and Response Evaluation
  • Technical Content Review

Contract Details

  • Compensation: $60-$85 per hour
  • Work arrangement: Fully Remote
  • Schedule: Flexible, with consistent availability expected
  • Payments: Weekly via Stripe or Wise

About Hirely

Hirely connects qualified professionals with remote opportunities in AI research, business, software engineering, and technology.

We are recruiting candidates on behalf of a hiring partner, which manages final candidate selection, onboarding, project assignments, and engagement terms.

Rate:
£85/hour
Location:
United Kingdom
IR35 Status:
Not specified
Remote Status:
Remote
Industry:
AI & Machine Learning
Seniority Level:
Mid-Level

Take-Home Pay

£10,500 per month

Visit calculators for additional details

Share job