Job Summary
We are looking for a Senior QA Engineer to test an enterprise Generative AI and Agentic AI platform built using AWS Bedrock and Databricks. The role will focus on AI response validation, agent workflows, Natural Language-to-SQL, data accuracy, KPI assessments, surveillance, security, and end-to-end testing.
The ideal candidate should have strong QA, API, SQL, and automation experience, along with a working understanding of Generative AI/LLM applications and basic US Healthcare/Medicaid knowledge.
Key Responsibilities
- Design and execute functional, integration, regression, API, data, and end-to-end test cases.
- Validate AI-generated responses against source-of-truth data and business rules.
- Test Natural Language-to-SQL, query generation, execution, validation, and result accuracy.
- Validate AI agent workflows, tool/function calls, routing, retries, fallbacks, and error handling.
- Test KPI assessments, alerts, surveillance, anomaly detection, and recommended actions.
- Validate AI behavior for ambiguous questions, unsupported requests, hallucinations, and negative scenarios.
- Perform data validation using SQL and Databricks.
- Validate PHI/PII handling, RBAC, entitlements, and security controls.
- Support automation, performance testing, UAT, defect triage, and production-readiness testing.
- Work with AI Engineers, Developers, Data Engineers, and Healthcare SMEs to identify and troubleshoot issues.
Required Skills
- Hand on exprience years in QA/testing experience with enterprise applications.
- Strong API testing, SQL, data validation, and test automation experience.
- Experience with functional, integration, regression, and E2E testing.
- Understanding of Generative AI, LLMs, prompts, RAG/grounding, and AI evaluation.
- Experience validating AI-generated results against expected business outcomes.
- Basic knowledge of US Healthcare/Medicaid, MMIS, claims, eligibility, providers, and pharmacy data.
- Understanding of HIPAA, PHI/PII, RBAC, and data security.
Preferred Skills
- Generative AI / LLM application testing.
- Agentic AI / AI-agent testing.
- AWS Bedrock and Databricks.
- Natural Language-to-SQL testing.
- RAG and grounding validation.
- AI evaluation frameworks and automated scoring.
- Prompt/model regression testing.
- Healthcare, Medicaid, Pharmacy, PBM, or claims experience.
Rate:
Not specified
Location:
Remote
IR35 Status:
Outside
Remote Status:
Remote
Industry:
IT
Seniority Level:
Senior