Databricks Engineer(Remote)(12+ Years profiles)

Job Description

As a Senior Engineer in the Databricks practice, you will bring a curious mindset and a passion for exploring innovative solutions to address our clients’ most pressing data challenges.

You are a self-starter who excels at breaking down complex problems and eagerly shares insights with your team and the broader Builder community.

Key Responsibilities

• Develop and implement data solutions using Databricks, with hands-on experience in specific Databricks platform features such as Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
• Collaborate with cross-functional teams to design and optimize data pipelines for both batch and streaming data, ensuring data quality and efficiency.
• Stay updated with emerging technologies, latest Databricks platform features and continuously improve your skills in Databricks and other relevant tools.

Requirements

• 5+ years of data engineering experience, with at least 2 years of hands-on data pipeline design and development experience with Databricks, including specific platform features like Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative pipelines), and Unity Catalog.
• Proficiency in designing and building robust, scalable, YAML configuration driven data pipelines with batch, micro-batch, and streaming data ingestion and processing patterns using tools like Auto Loader demonstrating a strong understanding of modern data engineering practices.
• Experienced in building complex job/workflow orchestration patterns using either Databricks jobs/workflows or with other orchestration tools like Airflow, dbt etc.
• Exposure in building robust data quality, pipeline audit/observability solutions with Databricks native features and/or other data quality tools/frameworks like Great Expectations, Collibra, dbt etc.
• Proficiency in Big Data Platforms: Apache Spark, Presto, Amazon EMR.
• Experience with Cloud Data Warehouses: Amazon Redshift, Snowflake, Google BigQuery.
• Strong programming skills using SQL, Stored Procedures and Object-Oriented Programming languages (like Java, Python, PySpark etc.).
• Familiarity with building DevOPs, CI/CD pipelines using Databricks Asset