AWS Data Engineer (Palantir Foundry) - Contract Inside IR35 - London or Leeds, UK

AWS Data Engineer (Palantir Foundry) - Contract Inside IR35 - London or Leeds, UK

Posted 2 days ago by Cactus IT Solutions UK Ltd

Apply
Negotiable
Inside
Undetermined
London/Leeds, United Kingdom
Apply
Access Controls Amazon S3 Amazon Web Services (AWS) Apache Hive Apache Parquet Authentications AWS Identity And Access Management (IAM) AWS Key Management Service (KMS) Azure Databricks Azure DevOps Databricks Data Engineering Data Management DevOps Github Gitlab Infrastructure as Code (IaC) JSON Management Microsoft Azure Ontologies Pyspark SQL (Programming Language)

Summary: The AWS Data Engineer role focuses on configuring and supporting secure integrations between Palantir Foundry, Databricks on AWS, and various AWS-hosted data sources. The position requires hands-on experience with data management and governance, ensuring seamless access and functionality across multiple platforms. The engineer will collaborate with various teams to troubleshoot issues and document data lineage effectively. This contract position is classified as inside IR35 and is based in either London or Leeds.

Key Responsibilities:

  • Configure and support Palantir Foundry integration with Databricks on AWS.
  • Set up and validate the Foundry Databricks connector for approved SQL Warehouses, tables, views and Unity Catalog-governed objects.
  • Support end-to-end access from AWS S3 sources through Hive Metastore and Unity Catalog into Foundry datasets or products.
  • Configure Foundry Data Connection, datasets, syncs, projects, permissions and lineage as required.
  • Work with AWS teams to define IAM roles, S3 bucket policies, KMS permissions and approved cross-account access patterns.
  • Troubleshoot Foundry connector failures, Databricks authentication issues, Unity Catalog grants, schema mismatches and query failures.
  • Document lineage across AWS S3, Hive Metastore, Unity Catalog and Foundry consumption layers.
  • Work with platform, data engineering, architecture, security and governance teams to deliver a secure and supportable integration.
  • Hands-on Palantir Foundry experience, including connectors, Data Connection, datasets, syncs, projects and permissions.
  • Experience configuring or supporting the Palantir Foundry Databricks connector.
  • Strong Databricks on AWS experience, including SQL Warehouse, clusters, tables, views and access controls.
  • Good understanding of Hive Metastore and Unity Catalog, including catalogs, schemas, grants, storage credentials and external locations.
  • Strong AWS IAM, S3, KMS, bucket policy and cross-account access experience.
  • Terraform or similar Infrastructure as Code experience.
  • Strong ability to troubleshoot access, schema, query, sync and connectivity issues across Foundry, Databricks and AWS.
  • Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories.
  • Experience with Delta Lake, Parquet, Iceberg, CSV and JSON.
  • Data governance, lineage, secure data-sharing and access approval experience.
  • Python, PySpark or SQL development experience.
  • CI/CD experience using GitLab, GitHub Actions, Azure DevOps or similar.
  • Configured and validated Foundry Databricks connector.
  • Integration design covering AWS S3, Hive Metastore, Unity Catalog and Foundry consumption.
  • AWS IAM, S3 and KMS access design for source data.
  • Validated access to approved Databricks tables, views or SQL endpoints.

Key Skills:

  • Hands-on Palantir Foundry experience.
  • Experience configuring or supporting the Palantir Foundry Databricks connector.
  • Strong Databricks on AWS experience.
  • Good understanding of Hive Metastore and Unity Catalog.
  • Strong AWS IAM, S3, KMS experience.
  • Terraform or similar Infrastructure as Code experience.
  • Strong troubleshooting skills across Foundry, Databricks and AWS.
  • Experience with Foundry Virtual Tables and related tools.
  • Experience with Delta Lake, Parquet, Iceberg, CSV and JSON.
  • Data governance and lineage experience.
  • Python, PySpark or SQL development experience.
  • CI/CD experience using GitLab, GitHub Actions, Azure DevOps or similar.
  • Integration design experience.
  • Validated access to Databricks tables and views.

Salary (Rate): undetermined

City: London

Country: United Kingdom

Working Arrangements: undetermined

IR35 Status: inside IR35

Seniority Level: undetermined

Industry: IT

Role Overview

Configure and support secure integration between Palantir Foundry, Databricks on AWS and AWS-hosted data sources. The role covers data held in S3, registered through Databricks Hive Metastore, governed or exposed through Unity Catalog, and consumed by Foundry using the Databricks connector or approved ingestion patterns.

Key Responsibilities

  • Configure and support Palantir Foundry integration with Databricks on AWS.
  • Set up and validate the Foundry Databricks connector for approved SQL Warehouses, tables, views and Unity Catalog-governed objects.
  • Support end-to-end access from AWS S3 sources through Hive Metastore and Unity Catalog into Foundry datasets or products.
  • Configure Foundry Data Connection, datasets, syncs, projects, permissions and lineage as required.
  • Work with AWS teams to define IAM roles, S3 bucket policies, KMS permissions and approved cross-account access patterns.
  • Troubleshoot Foundry connector failures, Databricks authentication issues, Unity Catalog grants, schema mismatches and query failures.
  • Document lineage across AWS S3, Hive Metastore, Unity Catalog and Foundry consumption layers.
  • Work with platform, data engineering, architecture, security and governance teams to deliver a secure and supportable integration.
  • Hands-on Palantir Foundry experience, including connectors, Data Connection, datasets, syncs, projects and permissions.
  • Experience configuring or supporting the Palantir Foundry Databricks connector.
  • Strong Databricks on AWS experience, including SQL Warehouse, clusters, tables, views and access controls.
  • Good understanding of Hive Metastore and Unity Catalog, including catalogs, schemas, grants, storage credentials and external locations.
  • Strong AWS IAM, S3, KMS, bucket policy and cross-account access experience.
  • Terraform or similar Infrastructure as Code experience.
  • Strong ability to troubleshoot access, schema, query, sync and connectivity issues across Foundry, Databricks and AWS.
  • Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories.
  • Experience with Delta Lake, Parquet, Iceberg, CSV and JSON.
  • Data governance, lineage, secure data-sharing and access approval experience.
  • Python, PySpark or SQL development experience.
  • CI/CD experience using GitLab, GitHub Actions, Azure DevOps or similar.
  • Configured and validated Foundry Databricks connector.
  • Integration design covering AWS S3, Hive Metastore, Unity Catalog and Foundry consumption.
  • AWS IAM, S3 and KMS access design for source data.
  • Validated access to approved Databricks tables, views or SQL endpoints.

Required Skills and Experience Desirable Skills Key Deliverables Lineage, access-control and operational support documentation

Apply
Similar Jobs
Loading data...