Lead Data Engineer
Posted Today by Natobotics
Negotiable
Undetermined
Hybrid
Sheffield, England, United Kingdom
Summary: The Lead Data Engineer – Consultant role in Sheffield, UK, is a contract position requiring a professional with extensive experience in developing data engineering platforms. The ideal candidate will have hands-on expertise in technologies such as Hadoop, Spark, and Splunk, along with strong skills in data handling and pipeline optimization. This position emphasizes collaboration with BI and Analytics teams and promoting a Site Reliability Engineering culture. The role also involves ensuring high code quality through automated testing and adherence to service resilience standards.
Key Responsibilities:
- Lead and develop data engineering platforms.
- Handle raw, structured, semi-structured, and unstructured data using SQL and NoSQL.
- Build and optimize ETL/ELT data pipelines.
- Integrate large, disparate datasets using modern tools and frameworks.
- Implement Continuous Integration, Delivery, and Deployment via CI/CD pipelines.
- Support and collaborate with BI and Analytics teams.
- Drive the creation of technical test plans and maintain records for automated test environments.
- Promote SRE culture and ensure service resilience and sustainability.
Key Skills:
- Extensive experience with Hadoop, Spark, and Splunk.
- Proficiency in Python and scripting.
- Experience with SQL and NoSQL databases.
- Strong background in ETL/ELT processes.
- Familiarity with agile methodologies like Scrum or Kanban.
- Excellent analytical and problem-solving abilities.
- Ability to pair program and collaborate effectively.
Salary (Rate): undetermined
City: Sheffield
Country: United Kingdom
Working Arrangements: hybrid
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Role : Lead Data Engineer – Consultant
Location : Sheffield, UK
Is it Permanent/ Contract: Contract
Is it Onsite/Remote/Hybrid: Hybrid (3 days weekly from office)
We Are Seeking a Lead Data Engineering Consultant With Proven Experience In Leading And Developing Data Engineering Platforms. The Ideal Candidate Will Possess Hands-on Expertise In The Following Areas
- Extensive enterprise experience with Hadoop, Spark, and Splunk.
- Proficiency in object-oriented and functional scripting, particularly in Python.
- Skilled in handling raw, structured, semi-structured, and unstructured data (SQL and NoSQL).
- Experience integrating large, disparate datasets using modern tools and frameworks.
- Strong background in building and optimizing ETL/ELT data pipelines.
- Familiarity with source control and implementing Continuous Integration, Delivery, and Deployment via CI/CD pipelines.
- Experience supporting and collaborating with BI and Analytics teams in fast-paced environments.
- Ability to pair program and work effectively with other engineers.
- Excellent analytical and problem-solving abilities.
- Knowledge of agile methodologies such as Scrum or Kanban is a plus.
- Comfortable representing the team in standups and problem-solving sessions.
- Capable of driving the creation of technical test plans and maintaining records, including unit and integration tests, within automated test environments to ensure high code quality.
- Promote SRE (Site Reliability Engineering) culture by addressing challenges through data engineering.
- Ensure service resilience, sustainability, and adherence to recovery time objectives for all delivered software solutions.