AI Quality Assurance Lead - SC Cleared

Role Overview

We are seeking an experienced AI Test & Evaluation Engineer to design, develop, and execute evaluation frameworks for AI platforms, including M365 Copilot agents.

This role focuses on building automated test suites and validation pipelines to evaluate AI-specific quality attributes, performance, and reliability within complex technical environments.

Key Responsibilities

• Framework Design & Execution: Lead the end-to-end design, development, planning, and execution of test and evaluation frameworks tailored for AI solutions, including work on the Cyber RIIO tech stack and M365 Copilot agents.
• Automated Testing & Pipelines: Utilize robust Python skills to build scalable automated test suites and validation pipelines using PyTest.
• AI Quality Attribute Analysis: Rigorously test and evaluate AI-specific criteria including accuracy, relevance, faithfulness, hallucination rates, robustness, bias, fairness, security, privacy, and explainability.
• Data Science Project Delivery: Apply proven data science project delivery experience to ensure rigorous testing standards align with broader system architecture and delivery goals.

Key Requirements

• Security Clearance: Must hold valid, active SC Clearance.
• AI Evaluation Expertise: Proven experience designing and executing evaluation frameworks for AI, specifically M365 Copilot agents.
• Technical Skills: Strong Python programming skills combined with hands-on, practical experience using PyTest to build automated testing framework components.
• Analytical Skills: In-depth understanding of AI metrics and failure modes (hallucinations, bias, drift, security vulnerabilities, and model explainability).
• Delivery Track Record: Solid background in data science project delivery.

Location & Working Pattern

This is a hybrid position requiring 1 day per week in the office, with location options in London, Cardiff, or Glasgow.

The remainder of the week can be worked remotely.