QA Automation Engineer

Role Summary

We are building agentic AI systems that can interpret complex data sources, documentation, and structured information to perform analysis, validation, and decision-support tasks. This role exists to make those agents trustworthy enough to act on.

This is a senior, hands-on quality role weighted toward testing non-deterministic AI and agentic systems. That is where most of your time sits and where the hiring bar is highest. You will also own the broader quality surface: integration, API, and performance testing are part of the remit, not out of scope.

The differentiator for this role is the ability to define what "good" looks like when a system reasons, calls tools, and can be wrong in subtle ways. You will scope, build, and run the frameworks yourself, with the independence of a senior engineer, and decide what to build first.

Agentic System Testing (Primary Focus)

• Tool-Use & Trajectory Evaluation: Test whether agents select the right tools for the right reasons and follow sound multi-step trajectories, not just whether the final answer looks plausible. Evaluate planning, intermediate steps, and recovery when a tool fails or returns nothing.
• Grounding & Citation Verification: Verify that agent claims are backed by cited evidence in source data, documents, or knowledge bases, and that referenced locations actually support the answer, catching confident but unsupported outputs.
• Honest-Failure & Refusal Calibration: Assert that agents ask for clarification or respond appropriately when information cannot be confidently determined, rather than inventing answers.
• Guardrail & Adversarial Testing: Probe prompt injection, jailbreaks, and instructions hidden within ingested content, ensuring the agent treats source content as untrusted data.
• Multi-Turn & State: Validate follow-ups, references to prior turns, and that conversational state carries correctly across a session.

AI & LLM Validation

• Non-Deterministic Testing: Architect automated frameworks that score generative AI outputs for hallucination, consistency, and factual accuracy against gold-standard datasets using LLM-as-judge methods calibrated against human judgement.
• Prompt & Model Regression: Design regression suites that catch prompt drift and model-version drift, ensuring changes to models or system instructions do not quietly degrade quality. Own the ground-truth and evaluation datasets these depend on.

Live Production Quality

• Continuous Evaluation: Extend evaluation beyond pre-release into production, continuously scoring live agent outputs so quality is measured on real usage, not only in test environments.
• Monitoring & Alerting: Build quality monitoring that flags regressions, drift, and anomalous agent behaviour before users discover them.
• Quality Incident Response: Triage quality incidents and trace failures back to specific model versions, prompts, or datasets, feeding fixes into the development cycle.

Integration, API & Performance

• Backend, UI & API Testing: Build robust integration tests that validate API integration across services and key user-facing flows.
• Secure Gateway Validation: Automate testing of secure API gateways, verifying that Role-Based Access Control (RBAC) and PII-redaction logic work correctly before data reaches AI models.
• Performance & Load: Own performance test plans and implementation (for example, Locust, JMeter, k6), validating latency, throughput, and stability under realistic load.

Data, Traceability & Quality Gates

• Data Validation: Use SQL and data-validation tooling (for example, Great Expectations) to verify data quality across data platforms and vector databases, including the ground-truth and retrieval corpora that agents depend on.
• Requirements Traceability: Map test and evaluation cases to system requirements and user needs, producing verification-and-validation evidence and quality reports needed to ship with confidence.
• Quality Gates: Enforce quality gates in GitLab CI/CD that prevent non-compliant models or code from merging and prepare readiness evidence for release reviews.

Technical Requirements

• AI Evaluation (Core): Hands-on experience with LLM/agent evaluation frameworks (eg, DeepEval, TruLens, RAGAS, or custom Python evaluators) and LLM-as-judge techniques.
• Agent Observability: Experience tracing and debugging agent runs, including tool calls, intermediate steps, token usage, and latency, using tools such as LangSmith, Langfuse, or OpenTelemetry-based tracing.
• Core Automation: Expert Python for custom test harnesses and evaluation tooling (Pytest), plus standard automation libraries (Selenium/Playwright for UI, Requests for API).
• Performance Testing: Proven ability to design and implement performance test plans (eg, Locust, JMeter, k6).
• Data Validation: Proficiency with SQL and data-validation tools, and familiarity with vector databases and retrieval corpora.
• CI/CD Integration: Integrating automated tests and evaluations into GitLab CI/CD pipelines and enforcing quality gates.
• Test Management & Reporting: Managing test reports and artifacts (eg, TestRail, Allure) and communicating results clearly.
• Version Control & QE Practices: Maintaining code-based frameworks in Git/GitLab and applying modern quality-engineering practices.
• Traceability Tools: Familiarity with requirements-management tools (eg, Jira, Linear, Jama, Polarion) and linking results to requirement IDs.

Professional Qualifications

• Experience: 5+ years in QA automation or quality engineering, with at least 2 years focused on testing ML models, LLM applications, or AI agents.
• Probabilistic Systems Judgement: Able to define pass/fail criteria for systems whose outputs are not identical every run and communicate confidence levels clearly to engineering leadership.
• Independent Operator: A senior individual contributor who scopes and builds testing and evaluation frameworks with minimal direction and prioritises what matters most.
• Collaboration: Works closely with engineering and product teams, understands existing systems quickly, and moves efficiently.

Nice to Have

• Domain Experience: Familiarity with data-intensive workflows, complex documentation, compliance-focused environments, or highly regulated industries is advantageous.
• Formal V&V Exposure: Exposure to structured systems-engineering governance, stage-gate reviews, or formal verification and validation practices is beneficial but not required. We value the discipline more than the certification.