AI Quality Analyst (LLM) | $30/hr Remote

Summary: The AI Model Evaluator role involves evaluating outputs from large language models and autonomous agent systems, providing structured feedback to enhance model performance. The position requires a commitment of 10 to 40 hours per week and is conducted remotely. Candidates must possess strong analytical skills and experience in AI evaluation. The role emphasizes attention to detail and effective communication of insights to stakeholders.

Key Responsibilities:

• Evaluate outputs from large language models and autonomous agent systems using defined rubrics and quality standards.
• Review multi-step agent workflows, including screenshots and reasoning traces, to assess accuracy and completeness.
• Apply benchmarking criteria consistently while identifying edge cases and recurring failure patterns.
• Provide structured, actionable feedback to support model refinement and product improvements.
• Participate in calibration sessions to ensure consistent evaluation alignment across reviewers.
• Adapt to evolving guidelines and ambiguous scenarios with sound judgment.
• Document findings clearly and communicate insights to relevant stakeholders.

Key Skills:

• Strong experience in LLM evaluation, AI output analysis, QA/testing, UX research, or similar analytical roles.
• Proficiency in rubric-based scoring, benchmarking frameworks, and AI quality assessment.
• Excellent attention to detail with strong decision-making skills in ambiguous cases.
• Proficient English communication skills (written and verbal).
• Ability to work independently in a remote environment.
• Comfortable committing to structured evaluation workflows and evolving guidelines.

Salary (Rate): £30.00/hr

City: undetermined

Country: undetermined

Working Arrangements: remote

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT