Lead Data Engineer

Role Overview

We are hiring a hands-on Databricks Engineer that has experience delivering modern data platforms on Databricks. This role requires a minimum of 2 years of Databricks experience gained recently (i.e., current/recent projects using modern platform capabilities).

Key Responsibilities

• Engineering Delivery of Databricks lakehouse solutions from ingestion to curated serving layers.
• Define and implement Medallion Architecture (Bronze/Silver/Gold) and reusable engineering patterns.
• Build scalable ingestion pipelines using AutoLoader, Lakeflow Connect, batch/streaming, and incremental patterns.
• Develop Declarative Pipelines with Expectations (DLT) to enforce and monitor data quality. Implement and operate Unity Catalog for governance, access control, lineage, and secure data sharing patterns.
• Drive code quality and operational excellence (CI/CD approach, testing strategy, monitoring, incident triage).
• Partner with architects, platform teams, and stakeholders to align delivery with enterprise standards.
• Mentor engineers and act as the technical escalation point during delivery.

Minimum Experience (Filter Criteria)

• Handson Databricks experience, in recent years (e.g., within the last 2–3 years), demonstrating usage of modern Databricks capabilities and patterns.
• Evidence of production delivery (not trainingonly or lab-only exposure).

Must Have (Non-Negotiable)

• Databricks Certification, at least one of: Databricks Certified Data Engineer Associate/Professional OR Databricks Certified Machine Learning Associate/Professional OR Databricks Certified Generative AI Engineer (Associate)
• Unity Catalog handson experience: Metastore/catalog design, grants, lineage, and secure access patterns.

Declarative Pipelines with Expectations (DLT):

• Building pipelines, defining expectations, handling failures/quarantines, observability.

Ingestion engineering using Databricksnative approaches:

• AutoLoader and/or Lakeflow Connect, streaming and incremental ingestion patterns.
• Medallion Architecture implementation and best practices: Designing and implementing Bronze/Silver/Gold with practical decisions (schema evolution, CDC/upserts, SCD patterns, performance strategy).

Should Have

• Demonstrable use of latest Databricks capabilities (candidate can explain what they used recently and why).
• Strong Databricks engineering fundamentals:
• Delta Lake (MERGE, schema enforcement/evolution, OPTIMIZE/ZORDER, VACUUM)
• Databricks Workflows / job orchestration
• Productiongrade PySpark/SQL
• Clear understanding of pipeline reliability: Observability, alerting, replay/backfill strategies, and operational runbooks.

Nice to Have

• Lakeflow ingestion connectors (specific connector experience is a plus).
• RBAC / masking implementations (rowlevel security, column masking, sensitive data handling) using Unity Catalog.
• GenAI on Databricks: Mosaic AI, Vector Search, model serving, RAG pipelines, AI Functions. Lakebase awareness or handson experience.
• Workload/query optimisation: Photon usage, cluster sizing, shuffle/skew mitigation, caching strategy, partitioning, file sizing.
• Cost awareness and controls: Understanding DBU drivers, job vs allpurpose compute, cluster policies, monitoring and chargeback/showback patterns.