Senior Data Engineer - Databricks and Palantir Foundry

You will migrate and modernise production data pipelines from Databricks to Palantir Foundry, optimise workloads that remain on Databricks and safely retire obsolete processing.

Note: Must have Active SC

Key Responsibilities

• Assess Databricks notebooks, SQL, PySpark code, schedules and dependencies.
• Rebuild pipelines as Python/PySpark transforms within Palantir Foundry Code Repositories and Pipeline Builder.
• Rework Databricks-specific dependencies, including dbutils, notebook orchestration, storage paths and Delta Lake merge/update logic.
• Develop incremental processing covering late-arriving data, deletes, backfills and safe reruns.
• Use Palantir AI FDE, where approved, to support code conversion, testing and build-error investigation.
• Optimise Spark workloads by analysing query plans, job metrics, joins, data skew, shuffles, partitioning and compute utilisation.
• Build automated QA covering reconciliation, schemas, data types, row counts, duplicates, nulls, key integrity and business rules.
• Manage releases through Git and CI/CD, including reconciliation, parallel runs, cutover and operational handover.
• Identify and safely decommission unused or duplicated pipelines with appropriate approvals, monitoring and rollback plans.

Essential Skills

• Strong Python, PySpark and SQL.
• Hands-on Palantir Foundry experience, including Data Connection, Code Repositories, transforms, dataset builds, scheduling, lineage and access controls.
• Strong Databricks experience covering notebooks, scheduled jobs, Delta Lake, incremental processing and Spark performance tuning.
• Experience developing and supporting production ETL/ELT pipelines.
• Automated data testing and cross-platform reconciliation experience.
• Git, code reviews and CI/CD.
• AWS or Azure object storage, identity and secrets-management experience.
• Ability to independently review and correct AI-generated code.

Desirable

• Previous Databricks-to-Palantir Foundry migration experience.
• Palantir AI FDE.
• Pytest or similar reusable Python testing frameworks.
• Experience handling regulated or sensitive data.