Senior Databricks/Pyspark consultant

Summary: The Senior Databricks/Pyspark consultant role involves creating technical designs for data platforms and solutions, primarily utilizing Databricks and PySpark. The consultant will be responsible for writing and optimizing complex code for data processing, implementing CI/CD pipelines, and providing technical leadership while mentoring junior engineers. Collaboration with stakeholders to align technical solutions with business needs is also a key aspect of the position.

Key Responsibilities:

• Create technical designs for data platforms, pipelines, and solutions following medallion architecture principles.
• Write and optimize complex PySpark/Scala code for data ingestion, transformation, and loading (ETL/ELT) on Databricks.
• Implement CI/CD pipelines for Databricks, automate tasks, and manage deployment processes.
• Provide technical leadership, mentor junior engineers, and collaborate with stakeholders to translate business needs into efficient, scalable, and high-quality data architectures.

Key Skills:

• Expertise in Databricks and PySpark/Scala.
• Experience in creating technical designs and implementing data platforms.
• Strong knowledge of CI/CD pipelines and automation.
• Ability to mentor and lead junior engineers.
• Excellent collaboration and communication skills.

Salary (Rate): undetermined

City: undetermined

Country: undetermined

Working Arrangements: remote

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT