Databricks Architect (Resident Solution Architect)

Position Overview

We are seeking experienced Databricks Solution Architects with strong banking industry experience and expertise in enterprise data engineering, cloud data platforms, and modern data warehouse solutions. The ideal candidates will have successfully delivered multiple end-to-end Databricks implementations and possess hands-on architecture and development experience. This role requires deep knowledge of the Databricks Lakehouse Platform, Apache Spark, Delta Lake, cloud-native data solutions, and enterprise data platform modernization.

Primary Skills

• Databricks Architecture
• Databricks Lakehouse Platform
• Apache Spark
• Data Engineering
• Delta Lake
• Cloud Data Platforms

Secondary Skills

• Structured Streaming
• ETL/ELT Pipelines
• Unity Catalog
• Databricks Workflows
• MLflow
• Delta Live Tables
• MLOps
• DevOps and CI/CD
• AWS
• Microsoft Azure
• Google Cloud Platform (Google Cloud Platform)

Key Responsibilities

• Design and implement scalable, cloud-native data platforms using Databricks.
• Lead end-to-end Databricks implementations, from architecture and design through development and production deployment.
• Develop and optimize large-scale ETL/ELT pipelines using Databricks, Delta Lake, Structured Streaming, and modern data engineering frameworks.
• Design enterprise data architecture aligned with banking business requirements and technical standards.
• Optimize Apache Spark workloads through partitioning strategies, execution plan analysis, memory management, and performance tuning.
• Implement and manage Databricks Lakehouse capabilities, including Unity Catalog, Workflows, MLflow, and Delta Live Tables.
• Establish DevOps and CI/CD practices for production-grade data engineering solutions.
• Apply MLOps principles to support machine learning lifecycle management, model deployment, and monitoring.
• Troubleshoot complex data platform challenges and recommend scalable, reliable, and cost-efficient architecture solutions.
• Provide technical leadership and architectural guidance to engineering and delivery teams.

Required Qualifications

• 12–15+ years of experience in data engineering, data platforms, data analytics, and modern data warehouse solutions.
• 10+ years of consulting and client-facing delivery experience.
• Demonstrated success delivering at least 6–8 end-to-end Databricks implementations.
• Hands-on experience as a Databricks developer, technical lead, or solution architect.
• Strong expertise in cloud-native data platform architecture across AWS, Azure, or Google Cloud Platform.
• Deep hands-on proficiency in at least one major cloud platform.
• Advanced knowledge of Apache Spark, including performance optimization, partitioning, execution plans, memory management, and runtime internals.
• Extensive experience with Databricks, Delta Lake, Structured Streaming, and scalable ETL/ELT pipelines.
• Experience implementing CI/CD and DevOps practices using Azure DevOps, GitHub Actions, GitLab CI/CD, or Jenkins.
• Working knowledge of MLOps, machine learning lifecycle management, model deployment, and monitoring.
• Strong understanding of enterprise data architecture, distributed processing, and performance optimization.
• Relevant experience in the banking or financial services industry.
• Strong problem-solving and solution architecture capabilities.

Preferred Qualifications

• Databricks Data Engineering Professional certification or an equivalent advanced Databricks certification.
• Completion of recommended Databricks learning paths and coursework.
• Databricks Solutions Architect Champion program participation.
• Experience with advanced Databricks Lakehouse Platform capabilities and enterprise implementations.