Summary: The role of Data Engineer / Scientist involves designing, building, and optimizing scalable data pipelines and solutions on Azure. The position requires a strong background in data engineering practices, particularly with tools such as Spark and Python, and emphasizes data governance and security. The ideal candidate will work collaboratively with cross-functional teams to deliver reliable data solutions while ensuring data quality and compliance. This is a contract position based in the City of London.

Key Responsibilities:

• Design, develop, and optimise robust ETL/ELT pipelines using Azure Data Factory (ADF) and Spark.
• Work with Databricks to build and maintain scalable data processing solutions.
• Implement best practices for data governance using Unity Catalog.
• Manage data storage and access within Azure Data Lake Storage (ADLS) and blob storage.
• Write clean, efficient, and scalable code using Python and SQL.
• Integrate and secure access to sensitive data using Azure Key Vault.
• Collaborate with cross-functional teams including Data Scientists, Analysts, and Business stakeholders to understand data needs and deliver reliable data solutions.
• Ensure data quality, consistency, and reliability across all data systems and pipelines.
• Contribute to the continuous improvement of data engineering processes, standards, and tools.

Key Skills:

• Strong hands-on experience with Apache Spark; experience with Databricks is highly desirable.
• Proficiency in Python for data engineering and scripting.
• Solid understanding of ETL/ELT concepts, with hands-on experience using Azure Data Factory (ADF).
• Advanced SQL skills for data transformation, manipulation, and performance tuning.
• Experience with Azure Data Lake Storage (ADLS) and Blob Storage.
• Knowledge of Unity Catalog for managing data access, lineage, and compliance.
• Familiarity with Azure Key Vault for managing secrets and secure configuration.
• Proven experience working in a modern cloud-based data platform environment.
• Ability to work independently as well as part of a team in a fast-paced, agile environment.
• Strong problem-solving skills, attention to detail, and communication abilities.

Salary (Rate): £500

City: London

Country: United Kingdom

Working Arrangements: undetermined

IR35 Status: inside IR35

Seniority Level: Mid-Level

Industry: IT