Lead Data Architect (AI & Cloud Infrastructure)

Role: Lead Data Architect (AI & Cloud Infrastructure)

Location: Leeds, Manchester, Halifax (Hybrid)

Job Type: Contract

Position Overview

We are seeking a visionary Lead Data Architect to spearhead the evolution of our enterprise data platform. In this role, you will bridge the gap between traditional data engineering and cutting-edge artificial intelligence. You will not build AI models from scratch; instead, you will architect the scalable frameworks, high-performance pipelines, and secure storage systems that power our Generative AI (GenAI) and Predictive Machine Learning (ML) initiatives. The ideal candidate will design enterprise-grade Blueprints for Vector databases, RAG (Retrieval-Augmented Generation) infrastructure, and unified data lakes that ensure our AI assets are secure, governed, and highly available.

Key Responsibilities

• AI & Generative AI Infrastructure Design
• Architect RAG Pipelines: Design scalable end-to-end Retrieval-Augmented Generation infrastructure to inject real-time enterprise context into Large Language Models (LLMs).
• Vector Storage Management: Select, implement, and optimise enterprise vector databases (e.g., Pinecone, Milvus, pgvector) for high-performance embedding storage and semantic search.
• Agentic AI Enablement: Build high-throughput, low-latency data loops required to support autonomous AI agents in production.
• Core Data Architecture & MLOps Integration
• Unified Data Foundations: Scale our modern data stack utilizing Lakehouse architectures (e.g., Delta Lake, Apache Iceberg) to handle both unstructured AI data and structured analytics.
• Feature Engineering Infrastructure: Design and maintain enterprise Feature Stores (e.g., Feast, Tecton) to serve unified data features consistently across offline training and online real-time inference.
• Streamline MLOps Pipelines: Partner with ML Engineers to integrate data pipelines seamlessly with lifecycle tracking frameworks like MLflow or Kubeflow.
• AI Data Governance, Privacy & Quality
• Data Lineage Automation: Implement comprehensive data lineage tracking to audit the source datasets used for AI training, fine-tuning, and prompt context.
• Security & Compliance: Architect automated data masking, anonymisation, and PII-filtering pipelines to prevent sensitive data from leaking into foundational models.
• AI Data Cataloguing: Curate metadata structures within platform catalogs (e.g., Collibra, Atlan) to explicitly map physical data assets to corresponding AI applications.

Required Skills & Qualifications

Experience: Minimum of 7+ years of experience in data architecture, data engineering, or enterprise infrastructure design.

Cloud Mastery: Deep architectural expertise in at least one major cloud platform and its AI ecosystem: AWS: SageMaker, Bedrock, Glue, Redshift. Azure: Azure OpenAI Service, Azure Machine Learning, Synapse. GCP: Vertex AI, BigQuery ML, Dataflow.

Advanced Data Modeling: Proven success modeling for both traditional relational/NoSQL analytical engines and high-dimensional vector spaces.

Data Pipeline Frameworks: Hands-on experience with streaming and batch tooling including Apache Spark, Kafka, Flink, and orchestration tools like Apache Airflow or Prefect.

Programming Literacy: Strong proficiency in Python, SQL, and database internals.

Preferred Qualifications

Experience with unified data clouds such as Databricks or Snowflake . Relevant cloud certifications (e.g., AWS Certified Data Engineer, Azure Solutions Architect, Google Cloud Professional Data Engineer). Familiarity with Docker, Kubernetes, and infrastructure-as-code (Terraform)