Job Description
Design the conversation: dialogue flow, intents, prompts, and how the experience is grounded in the right source of truth.
Direct the build of the agent with engineering, conversation design, and data science partners, making the calls on what to build, what to reuse, and what to cut.
Decide where the AI hands off to a person, and make sure that handoff works for the member, provider, or associate.
Build and run the evals that gate a release: golden test sets, adversarial and edge-case testing, and regression checks.
Read transcripts once a workstream is live and use what you find to improve the conversation and the eval set.
Hold the bar on accuracy, tone, latency, and safety a conversation must clear before and after launch.
Monitor progress against plan, and flag or adjust scope, schedule, or resourcing when something is off track.
Provide timely, accurate status updates to CAI product leadership and stakeholders across the program.
Maintain clear documentation of conversation design decisions, requirements, and test results for the workstream.
Required Qualifications
- Bachelor's degree or equivalent.
- 7+ years of industry experience, with at least 3 years in product management or a closely related role.
- Experience owning a product or workstream end to end, from discovery through launch, with limited oversight.
- You understand how conversational AI is built: dialogue design, intent taxonomies, retrieval and grounding, and agent frameworks, well enough to design it yourself and judge what engineering builds.
- You can build and run evals for conversation quality and safety and read transcripts to find what to fix.
- Strong written and verbal communication, including the ability to translate between technical and non-technical audiences, up to senior stakeholders.
- Must be passionate about contributing to an organization focused on continuously improving consumer experiences.
Preferred Qualifications
- Experience designing for voice specifically: turn-taking, latency, speech recognition and text-to-speech, and how voice differs from chat.
- Experience with GenAI platforms, prompt engineering, and agent frameworks.
- Experience in a regulated industry, especially healthcare, including handling PHI under HIPAA.
- Experience managing a vendor or platform partner.