All Jobs Vacancy

Principal AI Data Engineer

Posted 1 week ago by Certain Advantage

You’ll be responsible for:

  • Developing AI, GenAI and Agentic AI prototypes using Azure AI Foundry, Copilot Studio, Databricks Mosaic AI and MLflow.
  • Building, optimising and evaluating Retrieval-Augmented Generation (RAG) solutions using prompt engineering and embedding models.
  • Designing and deploying AI agents using frameworks including LangChain, LangGraph and AutoGen.
  • Delivering production-ready GenAI applications for commercial teams within Trading & Supply.
  • Building scalable AI workflows on Databricks using Mosaic AI, MLflow, Genie and AgentBricks.
  • Applying DevOps best practices through CI/CD pipelines, GitHub workflows and automated testing.
  • Designing evaluation frameworks to benchmark, test and improve GenAI system performance and reliability.

The Individual

  • Strong expertise with Databricks, including Delta Lake, Unity Catalog, DLT and MLflow.
  • Deep knowledge of Large Language Models including GPT, Llama, Claude and Mistral.
  • Experience building and deploying GenAI, RAG and Agentic AI solutions in enterprise environments.
  • Strong programming skills in Python, SQL and/or Scala.
  • Experience with Azure cloud technologies including Azure OpenAI and Azure AI Foundry.
  • Knowledge of Spark Structured Streaming, Autoloader and modern data engineering practices.
  • Experience with ETL/ELT, data modelling and scalable data architectures.
  • Familiarity with Azure Data Factory, Qlik Replicate and data ingestion pipelines.
Rate:
£800/day
Location:
London
IR35 Status:
Not specified
Remote Status:
Hybrid
Industry:
AI & Machine Learning
Seniority Level:
Not Specified

Take-Home Pay

£11,200 per month

Visit calculators for additional details

Create a free account to view the take-home pay for this contract

Share job