Lead GenAI Quality Assurance Analyst - ONLY W2

------------------------ONLY W2---------------------ONLY W2--------------------ONLY W2--------------------------

Title: Lead GenAI Quality Assurance Analyst

Location: Reston, VA 20191.

Duration: 6 Months Contract + Extension

Job Description:

Terms of Employment

• Contract, 6 Months (Likely Extension)
• This position is remote. Candidates who are local to the DMV area and willing to attend onsite quarterly PI planning sessions in Reston, VA will be prioritized. However, candidates can be based anywhere in the United States.
• The selected candidate must be comfortable working standard Eastern time zone hours.

Overview & Responsibilities

• Our client is looking for a Lead GenAI Quality Assurance Analyst to join a forward-thinking team at a large health insurance provider pioneering new solutions in the exciting field of Generative AI! We are seeking a highly motivated and experienced professional play a crucial role in ensuring the quality and accuracy of cutting-edge GenAI projects. This is a unique opportunity to work with Large Language Models (LLMs), advanced AI tools, and a talented team dedicated to "breaking boundaries" in a fast-paced, innovative environment. You'll be instrumental in shaping how we test and validate complex AI systems that learn and evolve.

You will

• Develop and execute test strategies for Generative AI applications, focusing on validating outputs from Large Language Models (LLMs) and similar AI systems.
• Utilize and gain expertise in GenAI tools such as Compass, Amazon Kendra, and Amazon SageMaker.
• Employ contextual expression comparison tools and methodologies to assess the accuracy and relevance of AI-generated content, working towards defined accuracy thresholds.
• Design and implement Python-based automation scripts for testing GenAI functionalities.
• Perform back-end validation as needed, though this is a lower priority.
• Collaborate within an agile-like environment on "speeding bullet" projects requiring minimal supervision and a proactive approach.
• Participate in understanding and mitigating issues like AI hallucination and ethical biases in GenAI outputs.

Required Qualifications

• A minimum of 1 year of direct experience in testing and validating Generative AI systems or components, including Large Language Models (LLMs).
• A minimum of 8 years of overall experience in software quality assurance and testing.
• Strong proficiency in Python for test automation.
• Demonstrable ability to write SQL code on the fly to define data populations for testing.
• Proven experience in triaging and fixing Java code on the fly.
• Excellent problem-solving skills and the ability to work independently and take initiative on fast-moving projects.
• Familiarity with testing methodologies for non-deterministic systems where outputs can vary.

Preferred Qualifications

• Hands-on experience with the GenAI tool "Compass."
• Experience with Amazon Web Services (AWS) AI tools, particularly Amazon Kendra and Amazon SageMaker.
• Experience with various contextual expression comparison tools and techniques.
• Experience working on projects involving knowledge bases and querying/feedback mechanisms for AI models.
• Located in an area allowing for occasional (e.g., quarterly) travel to an office for PI planning.

Sincerely,

Preetam Raj

Lead Technical Recruiter

nTech Workforce Inc.

D: EXT: 726

E: