Key Responsibilities
- Design and develop scalable automated test frameworks using Python.
- Create test strategies from architecture and solution design documentation.
- Test RESTful APIs and microservice-based applications.
- Define and execute testing approaches for event-driven architectures and Kafka-based systems.
- Validate GenAI and LLM solutions through functional, non-functional and behavioural testing.
- Develop automated prompt testing and evaluation frameworks.
- Verify AI guardrails, safety controls and compliance requirements.
- Implement validation approaches for non-deterministic LLM outputs.
- Build and maintain contract tests using Pact Flow.
- Collaborate closely with Engineering, Architecture, Data Science and Product teams.
- Support CI/CD pipelines and quality engineering best practices.
Required Skills & Experience
Core Automation
Strong Python 3.x development skills with experience building testing frameworks from the ground up.
Extensive experience with Pytest.
Strong API testing experience.
Experience creating test strategies and quality approaches from architectural diagrams and solution designs.
Cloud & Platform
Working knowledge of AWS services including:
Lambda
Bedrock
S3
EKS
Event-Driven Systems
Experience testing Kafka-based architectures.
Strong understanding of asynchronous messaging and event-driven systems.
GenAI/LLM Testing (Essential)
Proven experience testing LLM and GenAI applications.
Prompt testing and prompt evaluation.
Non-deterministic output validation.
AI guardrail verification and safety testing.
Hallucination and response quality assessment.
Experience establishing quality standards for AI solutions.
LLM Evaluation Tooling
Experience with one or more of:
DeepEval
Promptfoo
RAGAS
LangSmith
Similar LLM evaluation frameworks
Contract Testing
Hands-on experience with Pact Flow.
Consumer-driven contract testing within distributed systems.
Desirable
Banking or Financial Services experience.
Experience working in highly regulated environments.
Exposure to RAG architectures.
Knowledge of LLM observability and monitoring.
Kubernetes experience.
CI/CD pipeline integration.