Azure Data Engineer

Location: Reading and 2 days on site per week
Contract Type: 6=18 months

We are seeking an experienced Azure Data Engineer to join our dynamic team. As part of our data engineering division, you will work with cutting-edge tools and technologies to support data ingestion, integration, and transformation within the Azure ecosystem. The ideal candidate will have a strong background in data engineering, specifically with Azure Databricks, PySpark, and SparkSQL, as well as experience in testing and Azure DevOps.

Key Responsibilities:

• Data Ingestion & Transformation: Utilize PySpark and SparkSQL in Azure Databricks to ingest, process, and transform large datasets efficiently.
• Testing & Validation: Implement automated testing using Pytest to ensure the quality and integrity of the data pipelines.
• Data Integration: Work with Azure SQL Database to develop and optimize T-SQL queries and stored procedures for data integration and reporting.
• CI/CD Pipeline Management: Leverage Azure DevOps pipelines for automation of deployment, testing, and monitoring of data engineering workflows.
• Collaboration: Work closely with data analysts, data scientists, and other engineering teams to understand data needs and deliver data solutions.
• Optimization & Maintenance: Continuously improve data processes, ensuring high performance and reliability in the data ingestion and transformation workflows.

Primary Skills:

• Data Ingestion & Processing: Expertise in PySpark and SparkSQL for efficient data processing and transformation within Azure Databricks.
• Automated Testing: Hands-on experience with Pytest for testing data pipelines and ensuring their reliability and performance.

Secondary Skills:

• Azure SQL Database: Experience in developing T-SQL queries and stored procedures for data processing, integration, and reporting.
• DevOps Automation: Proficient in Azure DevOps pipelines to automate workflows, testing, and deployment of data solutions.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent work experience).
• Proven experience in Azure Databricks, PySpark, SparkSQL, and automated testing.
• Familiarity with Azure SQL Database, T-SQL, and stored procedures.
• Solid understanding of Azure DevOps and CI/CD pipelines.
• Strong problem-solving skills and ability to work collaboratively in an agile environment.