Platform Engineer – AWS, Databricks and Unity Catalog

Platform Engineer – AWS, Databricks and Unity Catalog

Posted 7 days ago by Totaljobs

Apply
£525 Per day
Inside
Hybrid
London; City of London; East London; Central London; South East London; West London; Canary Wharf; South West London; North London; Greenwich; Stratford
Apply
Access Controls Amazon S3 Amazon Web Services (AWS) Apache Hive Apache Parquet AWS Identity And Access Management (IAM) AWS Key Management Service (KMS) Azure Databricks Azure DevOps Cloud Computing Cloud Technology Data Access Databricks Data Governance Data Integration DevOps Github Gitlab Infrastructure as Code (IaC) JSON Metadata Microsoft Azure Pyspark Runbook SQL (Programming Language) Stakeholder Communications Stakeholder Management Terraform

Summary: The Platform Engineer role focuses on integrating AWS-hosted data sources with Databricks and Unity Catalog, ensuring secure access and governance of data. The position requires hands-on experience with AWS services, Databricks, and Terraform, along with strong troubleshooting and documentation skills. The role is contract-based and classified as inside IR35, with a hybrid working arrangement in London. Candidates should be prepared to support various data integration patterns and configurations.

Key Responsibilities:

  • Design and support secure integration across AWS S3, Databricks and downstream data platforms.
  • Configure least-privilege access to S3 using IAM roles, bucket policies, KMS permissions and approved access controls.
  • Support Databricks access patterns where S3-backed data is registered through Hive Metastore and exposed or migrated into Unity Catalog.
  • Configure and validate Unity Catalog objects, including catalogs, schemas, tables, views, grants, storage credentials and external locations.
  • Support downstream connectivity to Databricks using SQL Warehouse, tables, views or connector-based integration patterns.
  • Develop Terraform/IaC for AWS IAM, S3, KMS and Databricks-related configuration.
  • Troubleshoot access, metadata, schema, query and connectivity issues across AWS, Databricks and downstream integrations.
  • Document lineage across AWS S3, Hive Metastore, Unity Catalog and downstream consumption layers.

Key Skills:

  • Strong hands-on experience with AWS IAM, S3, KMS and secure cloud access patterns.
  • Hands-on Databricks on AWS experience, including Hive Metastore, Delta tables, SQL Warehouse and workspace access controls.
  • Practical Unity Catalog experience, including grants, external locations, storage credentials and governed data access.
  • Terraform or similar Infrastructure as Code experience.
  • Good understanding of Delta, Parquet, CSV and JSON data formats.
  • Strong troubleshooting, documentation and stakeholder communication skills.

Salary (Rate): £525 per day

City: London

Country: United Kingdom

Working Arrangements: hybrid

IR35 Status: inside IR35

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

Job Title: Platform Engineer AWS, Databricks and Unity Catalog

Job Type: - Contract - Inside IR35

Location: Leeds and London - Hybrid

Job Description

Platform Engineer AWS, Databricks and Unity Catalog SC Eligible | Foundry experience: Desirable | Core skills: AWS, Databricks, Unity Catalog, Terraform

Role Overview

Support secure integration between AWS-hosted data sources, Databricks workspaces and downstream enterprise data platforms. The role covers data held in S3, registered through Databricks Hive Metastore, governed or exposed through Unity Catalog, and consumed through approved connector-based patterns.

Key Responsibilities

  • Design and support secure integration across AWS S3, Databricks and downstream data platforms.
  • Configure least-privilege access to S3 using IAM roles, bucket policies, KMS permissions and approved access controls.
  • Support Databricks access patterns where S3-backed data is registered through Hive Metastore and exposed or migrated into Unity Catalog.
  • Configure and validate Unity Catalog objects, including catalogs, schemas, tables, views, grants, storage credentials and external locations.
  • Support downstream connectivity to Databricks using SQL Warehouse, tables, views or connector-based integration patterns.
  • Develop Terraform/IaC for AWS IAM, S3, KMS and Databricks-related configuration.
  • Troubleshoot access, metadata, schema, query and connectivity issues across AWS, Databricks and downstream integrations.
  • Document lineage across AWS S3, Hive Metastore, Unity Catalog and downstream consumption layers.

Required Skills and Experience

  • Strong hands-on experience with AWS IAM, S3, KMS and secure cloud access patterns.
  • Hands-on Databricks on AWS experience, including Hive Metastore, Delta tables, SQL Warehouse and workspace access controls.
  • Practical Unity Catalog experience, including grants, external locations, storage credentials and governed data access.
  • Terraform or similar Infrastructure as Code experience.
  • Good understanding of Delta, Parquet, CSV and JSON data formats.
  • Strong troubleshooting, documentation and stakeholder communication skills.

Desirable Skills

  • Palantir Foundry experience, especially Databricks connector, Data Connection, datasets, syncs, projects and permissions.
  • Secure data-sharing, data governance, lineage and access approval experience.
  • Python, PySpark or SQL development experience.
  • CI/CD experience using GitLab, GitHub Actions, Azure DevOps or similar.

Key Deliverables

  • Secure AWS S3 and Databricks access design.
  • Hive Metastore to Unity Catalog integration approach.
  • Terraform/IaC modules or scripts for repeatable configuration.
  • Source-to-target lineage and access-control documentation.
  • Operational runbook for access, schema, sync and connectivity issues.
Apply
Similar Jobs
Loading data...