Job Description: Principal AI Data Engineer
Contract Duration: 6 months (with view to extend)
Inside IR35
Location: Central London Hybrid (3 days onsite/2 days remote)
Client Sector: Global Technology Consultancy/Enterprise Commercial Delivery
A global technology consultancy is seeking a Principal AI Data Engineer to spearhead the design, architecture, and production delivery of advanced Generative and Agentic AI solutions for a major enterprise client.
In this role, you will bridge the gap between rapid frontier AI experimentation and robust, large-scale enterprise data engineering. Working closely with commercial stakeholders, cross-functional squads, and technical leadership, you will translate high-value operational use cases into reliable, production-grade AI pipelines, intelligent agentic workflows, and governed data platforms.
Key Responsibilities
- Agentic AI and Prototype Delivery: Architect, build, and deploy multi-agent workflows and task-driven AI systems using frameworks such as LangGraph, LangChain, or AutoGen, turning early proof-of-concepts into resilient production applications.
- Retrieval-Augmented Generation (RAG): Design and tune enterprise-grade RAG pipelines, including chunking strategy, dense/hybrid retrieval, reranking, and semantic search.
- Data Platform Engineering: Build scalable, streaming, and batch data processing pipelines across modern cloud lakehouse architectures (Delta Lake, streaming architectures, and unified governance layers).
- Systematic Evaluation and Benchmarking: Establish continuous model evaluation, reproducibility checks, tracing, and benchmarking frameworks (MLflow, LLM-as-a-judge, golden test suites) to ensure performance, safety, and auditability.
- Architecture and Backlog Leadership: Set technical standards, define architectural blueprints, shape the product backlog, and lead agile delivery squads while communicating technical trade-offs to commercial business stakeholders.
- Production MLOps and Governance: Implement modern software engineering and CI/CD practices (OOP in Python, Pydantic, containerisation, automated testing, and cloud gateway governance).
Key Technical Stack and Experience
- Experience: 8-12+ years of hands-on experience in data engineering, data science, or production ML/AI environments.
- Agentic Frameworks: Proven delivery with LangGraph, AutoGen, CrewAI, smolagents, or Model Context Protocol (MCP) integrations.
- Lakehouse and Data Engineering: Strong background with enterprise Databricks environments (Delta Lake, Unity Catalog, Delta Live Tables/DLT, Spark Structured Streaming, or Autoloader).
- Cloud and Model Ecosystem: Broad exposure to Azure cloud environments (Azure AI Foundry, Azure OpenAI) or Databricks Mosaic AI ecosystem.
- Code Craftsmanship: Advanced Python (modular OOP, Pydantic, design patterns) and strong SQL capabilities.
- Education: Degree or advanced degree (BSc/MSc) in Mathematics, Statistics, Computer Science, Engineering, Econometrics, or related quantitative discipline.
£500.00 - £600.00/day