Data Engineer (Databricks) - SC Clearance Required
Posted 6 days ago by Korn Ferry
£550 Per day
Inside
Remote
London, UK
Summary: The role of Data Engineer focuses on developing and optimizing data pipelines using Databricks and cloud technologies, requiring active UK Security Check (SC) clearance. The position is remote and involves collaboration with cross-functional teams to deliver high-performance data solutions. Candidates must possess expertise in data engineering and strong communication skills for client interactions. This role combines technical depth with client-facing responsibilities in an agile environment.
Key Responsibilities:
- Design, develop and optimize end-to-end data pipelines using Databricks, Spark/PySpark and Python
- Lead the full pipeline life cycle: analysis, architecture, development, testing, deployment and monitoring
- Implement ingestion, normalization and standardization workflows for high-volume and diverse data sources
- Ensure data platform architecture supports performance, security, and compliance requirements
- Collaborate cross-functionally with engineering, product teams and client stakeholders during project delivery
- Support ML readiness for ingestion and transformation pipelines where required
- Promote data platform best practices across security, governance and cloud adoption strategies
Key Skills:
- Active SC Clearance
- Expertise in Databricks, including Delta Lake and Unity Catalog
- Strong background in Python, PySpark and advanced SQL
- Proven experience in AWS cloud platforms for data engineering
- Track record of building high-performance, large-scale data solutions in production environments
- Ability to lead pipeline life cycle (design, deploy, monitor) autonomously
- Strong communication skills for stakeholder collaboration and client presentations
Salary (Rate): £550 per day
City: London
Country: UK
Working Arrangements: remote
IR35 Status: inside IR35
Seniority Level: Senior
Industry: IT
Detailed Description From Employer:
Data Engineer - Databricks & Big Data Pipelines (SC Cleared is required for this role)
Location: UK only remote is possible
Working Mode: remote
Rate: £450-550 per day inside IR35
Description:
We're looking for a Senior Data Engineer - Databricks & Big Data Pipelines to join our team, working in our UK remote working division. This position requires active UK Security Check (SC) clearance and offers the opportunity to build advanced, large-scale data engineering solutions for an innovative analytics platform. You will design, develop and optimize distributed data pipelines on Databricks and cloud ecosystems while ensuring performance, scalability, and compliance. The role combines technical depth with client-facing delivery as part of an agile and collaborative environment.
Responsibilities
- Design, develop and optimize end-to-end data pipelines using Databricks, Spark/PySpark and Python
- Lead the full pipeline life cycle: analysis, architecture, development, testing, deployment and monitoring
- Implement ingestion, normalization and standardization workflows for high-volume and diverse data sources
- Ensure data platform architecture supports performance, security, and compliance requirements
- Collaborate cross-functionally with engineering, product teams and client stakeholders during project delivery
- Support ML readiness for ingestion and transformation pipelines where required
- Promote data platform best practices across security, governance and cloud adoption strategies
Mandatory Requirements
- Active SC Clearance
- Expertise in Databricks, including Delta Lake and Unity Catalog
- Strong background in Python, PySpark and advanced SQL
- Proven experience in AWS cloud platforms for data engineering
- Track record of building high-performance, large-scale data solutions in production environments
- Ability to lead pipeline life cycle (design, deploy, monitor) autonomously
- Strong communication skills for stakeholder collaboration and client presentations
SkillSet Overview
Must Have:
- Databricks (Delta Lake, Unity Catalog, workflows)
- Python Data Engineering Technologies (including PySpark)
- SQL
- AWS
Nice to Have:
- AI and ML Ops
- PostgreSQL
- OpenSearch
- Microsoft Azure
- Claude Code (AI productivity tooling)
- Docker/Kubernetes
- REST APIs
- Cloud-native ML pipeline deployment
- Familiarity with managing data platform releases, monitoring and cost governance
- Domain understanding in supply chain analytics, compliance or public sector data
- Experience integrating AI tools into engineering workflows
About Korn Ferry
Korn Ferry unleashes potential in people, teams, and organizations. We work with our clients to design optimal organization structures, roles, and responsibilities. We help them hire the right people and advise them on how to reward and motivate their workforce while developing professionals as they navigate and advance their careers. To learn more, please visit our website
