Senior QA Automation Engineer / SDET - GenAI & LLM Testing

Key Responsibilities

• Design and develop scalable automated test frameworks using Python.
• Create test strategies from architecture and solution design documentation.
• Test RESTful APIs and microservice-based applications.
• Define and execute testing approaches for event-driven architectures and Kafka-based systems.
• Validate GenAI and LLM solutions through functional, non-functional and behavioural testing.
• Develop automated prompt testing and evaluation frameworks.
• Verify AI guardrails, safety controls and compliance requirements.
• Implement validation approaches for non-deterministic LLM outputs.
• Build and maintain contract tests using Pact Flow.
• Collaborate closely with Engineering, Architecture, Data Science and Product teams.
• Support CI/CD pipelines and quality engineering best practices.

Required Skills & Experience

• Core Automation
• Strong Python 3.x development skills with experience building testing frameworks from the ground up.
• Extensive experience with Pytest.
• Strong API testing experience.
• Experience creating test strategies and quality approaches from architectural diagrams and solution designs.
• Cloud & Platform
• Working knowledge of AWS services including: Lambda Bedrock S3 EKS
• Event-Driven Systems
• Experience testing Kafka-based architectures.
• Strong understanding of asynchronous messaging and event-driven systems.
• GenAI / LLM Testing (Essential)
• Proven experience testing LLM and GenAI applications.
• Prompt testing and prompt evaluation.
• Non-deterministic output validation.
• AI guardrail verification and safety testing.
• Hallucination and response quality assessment.
• Experience establishing quality standards for AI solutions.
• LLM Evaluation Tooling
• Experience with one or more of: DeepEval Promptfoo RAGAS LangSmith Similar LLM evaluation frameworks
• Contract Testing
• Hands-on experience with Pact Flow.
• Consumer-driven contract testing within distributed systems.

Desirable

• Banking or Financial Services experience.
• Experience working in highly regulated environments.
• Exposure to RAG architectures.
• Knowledge of LLM observability and monitoring.
• Kubernetes experience.
• CI/CD pipeline integration.