About the Role:
How do we make advanced AI/ML not just powerful but trustworthy enough to run in a regulated healthcare environment? We're looking for a Senior Staff engineer who can own that question end to end: the data pipelines that feed our models and evaluations, and the evaluation infrastructure - judges, scorers, statistical regression gates, red-team testing - that decides whether AI models are effective and safe to ship to patients.
This is a leadership role for someone who works comfortably across disciplines. You'll set technical direction for how we build, version, and trust the data and judgments that our AI products are evaluated against. Your scope will expand from raw data ingestion and feature/dataset pipelines, through evaluation methodology and statistical rigor, to the reporting surfaces that let clinical and product teams act on what we learn.
You'll spend most of your time on problems that don't have an existing playbook: ambiguous, cross-team, and genuinely hard to reason about. The job is to bring clarity to that ambiguity, chart a path the rest of the team and organization can follow, and see it through from idea to production, building the relationships and buy-in along the way to make it stick.
You Will:
Own the evaluation as a whole, not just a slice of it
Lead projects that span teams and quarters
Turn hard, ambiguous problems into solutions other teams can build on
Grow the people and the network around you
You Have:
- 10+ years of experience in ML infrastructure, data engineering, or evaluation/testing systems, with a track record of impact that reaches beyond a single team or project.
- Hands-on depth in evaluation systems: designing and calibrating LLM judges/scorers, building statistically sound regression-testing methodology (e.g., paired significance testing with proper correction for multiple comparisons), measuring agreement against human labels, and designing adversarial/red-team evaluation approaches.
- Hands-on depth in data pipeline engineering: dataset versioning, feature and benchmark pipelines, labeling and calibration workflows, and high-throughput ingestion and transformation systems.
- A history of building things that became the standard approach for others - not just solving your own problem, but changing how a broader group of people tackle a category of problem.
- Experience leading multi-team projects to completion, including navigating and resolving genuine technical disagreement along the way.
- A track record of mentoring other engineers, including senior ones, and visibly raising the bar for the teams around you.
- Excellent communication - comfortable adapting the same idea for different audiences, and confident building support for it well before launch.
- Strong Python, and enough statistical fluency to design and defend a testing framework that real production decisions ride on.
Nice to Have:
- Experience with Databricks, MLflow, Unity Catalog, or similar data/eval platforms.
- Experience building reporting tools for people without direct engineering access (e.g., automated Slack digests, spreadsheet reports for non-technical teams).
- Prior experience in a regulated industry (healthcare, fintech, life sciences).
- A track record of company-wide talks or write-ups that changed how other teams approached a problem.
Our Benefits (there are more but here are some highlights):
- Competitive salary & equity compensation for full-time roles
- Unlimited PTO, company holidays, and quarterly mental health days
- Comprehensive health benefits including medical, dental & vision, and parental leave
- Employee Stock Purchase Program (ESPP)
- 401k benefits with employer matching contribution
- Offsite team retreats