ABOUT THE ROLE
Own prompt design and evaluation for LLM-powered chatbots, agents, and RAG systems. You will turn business requirements into reliable prompt and agent workflows and prove they work with measurable results.
RESPONSIBILITIES
- Craft, test, and refine prompts that maximize accuracy, relevance, and consistency
- Build evaluation frameworks: A/B testing, benchmarking, golden datasets, and rubric-based scoring
- Apply deep knowledge of LLM behavior: context windows, tokenization, and fine-tuning versus prompting trade-offs
- Partner with product, engineering, and data science to design prompt and agent workflows
- Implement prompt security and safety: jailbreak prevention, bias mitigation, and guardrails
- Maintain prompt versioning and regression testing
REQUIRED QUALIFICATIONS:
- 5+ years in software, data, or applied AI roles, with minimum 1+ year of hands-on LLM prompt work in production in 5+ yrs of experience
- Strong Python scripting and API integration
- Hands-on experience with GPT and Claude, and orchestration tools such as LangChain
- Demonstrated experience designing evaluation methods and iterating on results
- Working knowledge of RAG and agent patterns, including tool calling
- Data analysis skills to interpret evaluation results
Rate:
Not specified
Location:
Remote
IR35 Status:
Not specified
Remote Status:
Remote
Industry:
AI & Machine Learning
Seniority Level:
Senior