Responsibilities
We are seeking an AI Quality Assurance and Security Testing Specialist to evaluate, test, and validate enterprise AI systems, agentic workflows, and LLM based applications.
This role ensures AI security controls, detection logic, and agent behavior meet quality, reliability, and safety standards across the organization.
The successful candidate should have strong knowledge of AI security testing, LLM evaluation, prompt-based QA, and adversarial testing techniques.
Hands on experience with Python, test automation frameworks, AI evaluation tools, and cloud-based AI platforms is essential. The candidate will design test plans, simulate adversarial scenarios, validate detection effectiveness, and collaborate with engineering and security teams to improve AI system resilience.
Experience with Microsoft Copilot, Azure AI Foundry, Azure OpenAI Service, Amazon Connect, and enterprise AI security technologies is a plus. Familiarity with red-team testing, model robustness evaluation, and AI safety methodologies is an asset.
Key Responsibilities
- Develop and execute AI QA test plans for agentic AI systems, LLM applications, and autonomous workflows.
- Perform security testing including prompt injection attempts, jailbreak simulations, model manipulation, and adversarial input testing.
- Validate AI detection logic for accuracy, reliability, and operational value across SIEM and XDR platforms.
- Evaluate AI model behavior under stress, edge cases, and adversarial conditions to identify vulnerabilities.
- Test AI system integrations across cloud services, APIs, identity systems, endpoints, and enterprise applications.
- Automate AI QA workflows using Python, test automation frameworks, and AI evaluation tools.
- Document test results, methodologies, and recommendations in a clear and maintainable format.
- Collaborate with SOC, engineering, and AI platform teams to improve AI security posture and detection quality.
- Research emerging AI threats and testing techniques to ensure continuous improvement of QA processes.
Required Qualifications
- Bachelor s or Master s degree in Computer Science, Cybersecurity, AI, Data Science, or a related discipline.
- 3 5+ years of experience in QA engineering, security testing, AI evaluation, or cybersecurity.
- Strong understanding of LLMs, agentic AI, RAG, and modern AI architectures.
- Hands on experience with test automation, Python, and AI evaluation frameworks.
- Experience with Microsoft Sentinel, Microsoft Defender, and enterprise security monitoring tools.
- Knowledge of identity, endpoint, network, and API security principles.
- Experience conducting security tests and validating detection logic.
- Strong analytical, documentation, and communication skills.
Preferred Qualifications
- Experience with Microsoft Copilot, Azure AI Foundry, Azure OpenAI Service, or enterprise AI platforms.
- Familiarity with AI security frameworks such as OWASP LLM Top 10, MITRE ATLAS, and NIST AI RMF.
- Experience performing AI red-team testing, model robustness evaluation, or adversarial QA.
- Certifications such as SC-200, SC-100, GIAC, AI related certifications, or QA/security testing credentials.