Data Engineer
Posted Today by Insight Global
Negotiable
Undetermined
Undetermined
London Area, United Kingdom
Summary: The Data Engineer role at Insight Global involves designing and building data foundations and end-to-end solutions to maximize data value across the organization. This position requires the engineer to be a subject matter expert, mentoring junior engineers while driving the conversion of vision and data strategy into actionable IT solutions. The role emphasizes collaboration with cross-functional teams and effective communication with stakeholders to ensure high-quality data solutions.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines using Azure Data Factory and Azure Databricks.
- Implement data transformation workflows using PySpark and Delta Live Tables (DLT).
- Manage and govern data assets using Unity Catalog.
- Write efficient and optimized SQL queries for data extraction, transformation, and analysis.
- Collaborate with cross-functional teams to understand data requirements and deliver high-quality solutions.
- Demonstrate strong ownership and accountability in delivering end-to-end data solutions.
- Communicate effectively with stakeholders to gather requirements, provide updates, and manage expectations.
Key Skills:
- Proven hands-on experience with Azure Data Factory, Databricks, DLT, and Unity Catalog.
- Hands-on experience with AZURE Function and/or REST/FAST API, LogicApps.
- Hands-on Databricks expertise in Performance Tuning & Cost Optimization, Structural Streaming, Delta Live Table, and DBU/compute Performance optimization.
- AI experience with Databricks MosaicAI and/or Databricks MLFlow, PowerBI, Vector Embedding/Search, LLM Integration, Prompt Engineering.
- Proficient with Python & PySpark.
- Strong command of SQL and data modeling concepts.
Salary (Rate): undetermined
City: London Area
Country: United Kingdom
Working Arrangements: undetermined
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Insight Global are in search for a Data Engineer. The Data Engineer is the one who designs and builds data foundations and end to end solutions for the business to maximize value from data. The role helps create a data-driven thinking within the organization, not just within IT teams, but also in the wider business stakeholder community. The Sr. Data engineer is expected to be a subject matter expert, who design & build data solutions and mentor junior engineers. They are also the key drivers to convert Vison and Data Strategy for IT solutions and deliver.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines using Azure Data Factory and Azure Databricks .
- Implement data transformation workflows using PySpark and Delta Live Tables (DLT) .
- Manage and govern data assets using Unity Catalog .
- Write efficient and optimized SQL queries for data extraction, transformation, and analysis.
- Collaborate with cross-functional teams to understand data requirements and deliver high-quality solutions.
- Demonstrate strong ownership and accountability in delivering end-to-end data solutions.
- Communicate effectively with stakeholders to gather requirements, provide updates, and manage expectations.
The Applicant must have the following:
- Proven hands-on experience with Azure Data Factory, Databricks, DLT, and Unity Catalog
- Hands on experience with AZURE Function and/or REST/FAST API, LogicApps
- Hands on Databricks expertise in: Performance Tunning & Cost Optimization
- Databricks Structural Streaming
- Databricks Delta live Table
- Databricks DBU/compute Performance optimization
- AI experience with Databricks MosiacAI and/or Databricks MLFlow, PowerBI, Vector Embedding/Search, LLM Integration, Prompt Engineering
- Proficient with Python & PySpark
- Strong command of SQL and data modelling concepts