Responsibilities
- Audit complete AI evaluation tasks, including prompts, grading rubrics, and model attempts.
- Assess whether identified model failures are genuinely justified.
- Verify whether the required information was available to the AI agent.
- Identify inaccurate, ambiguous, or unfair grading criteria.
- Determine whether valid model responses could incorrectly receive failing grades.
- Check rubric structure and compliance with evaluation guidelines.
- Identify tasks that cannot be reasonably corrected.
- Provide clear, constructive feedback to reviewers on their decisions.
- Maintain consistent auditing quality and availability.
Requirements
- Substantial experience writing, reviewing, or auditing LLM evaluation rubrics.
- Previous experience in AI evaluation, RLHF, model training, or AI quality assurance.
- Experience providing feedback to writers, reviewers, or evaluators.
- Strong analytical and critical-thinking abilities.
- Excellent written English and attention to detail.
- Ability to analyze lengthy AI agent transcripts and identify specific errors.
- Ability to independently verify information rather than blindly trusting reference answers.
- Consistent availability and ability to meet project expectations.
Relevant Experience
- Candidates with experience in the following areas are encouraged to apply:
- LLM Evaluation and Benchmarking
- Rubric Design and Auditing
- RLHF and Human Feedback
- AI Quality Assurance
- AI Model Assessment
- Reviewer Quality Control
- Prompt and Response Evaluation
- Technical Content Review
Contract Details
- Compensation: $60-$85 per hour
- Work arrangement: Fully Remote
- Schedule: Flexible, with consistent availability expected
- Payments: Weekly via Stripe or Wise
About Hirely
Hirely connects qualified professionals with remote opportunities in AI research, business, software engineering, and technology.
We are recruiting candidates on behalf of a hiring partner, which manages final candidate selection, onboarding, project assignments, and engagement terms.