Data Engineer
Posted 2 days ago by Trust In SODA
£500 Per day
Outside
Remote
Greater London, South East, England
Summary: We are currently working with a new generation consultancy based across the UK and EU, focused on engineering excellence and empowering people to make an impact. The role involves collaborating with businesses to analyze and cleanse data, design storage solutions, and communicate effectively with the team. The consultancy values modern tech stacks and agile methodologies, with all consultants holding equity in the company.
Key Responsibilities:
- Work closely with businesses to understand and analyze their current data problems.
- Design data storage solutions.
- Communicate effectively with team members.
- Cleanse and optimize data.
Key Skills:
- Proficiency in Python at a software engineering level, including unit and integration testing experience.
- Knowledge of distributed computing with PySpark or Scala, including debugging in SparkUI and optimization.
- Experience with AWS.
- Understanding of data modeling, change data capture, and ACID compliant table structures.
- Good understanding of data lakes and traditional data platforms.
- Experience with data ingestion via API calls, batch, and/or streaming.
- Proficiency in SQL and NoSQL databases.
- Experience with Git version control and CI/CD.
- Experience building data platforms in at least one cloud environment, preferably AWS.
- Container experience with Docker or Kubernetes.
- Consulting experience working on projects with multiple stakeholders.
Salary (Rate): £500
City: Greater London
Country: England
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: undetermined
Industry: IT