Data Engineer - ETRM
Posted 2 weeks ago by Vallum Associates
Negotiable
Undetermined
Undetermined
London Area, United Kingdom
Summary: The Senior Data Engineer role focuses on designing, developing, and maintaining scalable data pipelines and ETRM systems, with a strong emphasis on real-time data replication and integration within the Energy Trading and Risk Management domain. The position requires collaboration with cross-functional teams to ensure data quality and integrity while optimizing data architecture for performance. Candidates should possess extensive experience in relevant technologies and data integration tools, particularly within the ETRM sector.
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETRM systems.
- Implement real-time data replication from various source systems to Snowflake using Fivetran.
- Work on data integration projects within the Energy Trading and Risk Management (ETRM) domain.
- Collaborate with cross-functional teams to integrate data from ETRM trading systems like Allegro, RightAngle, and Endur.
- Optimize and manage data storage solutions in Data Lake and Snowflake.
- Develop and maintain ETL processes using Azure Data Factory and Databricks.
- Write efficient and maintainable code in Python for data processing and analysis.
- Ensure data quality and integrity across various data sources and platforms.
- Develop and maintain data pipelines using FiveTran, Snowflake, and Azure Data Factory (ADF).
- Ensure data accuracy, integrity, and availability across various trading systems.
- Collaborate with traders, analysts, and IT teams to understand data requirements and deliver robust solutions.
- Optimize and enhance data architecture for performance and scalability.
Key Skills:
- Azure Data Factory (ADF)
- Data Lake
- Snowflake
- Python
- Databricks
- Fivetran
- Experience in ETRM domain
- Integration with trading systems like Allegro, RightAngle, Endur
Salary (Rate): undetermined
City: London Area
Country: United Kingdom
Working Arrangements: undetermined
IR35 Status: undetermined
Seniority Level: Senior
Industry: IT
Job Title: Senior Data Engineer
Experience: 10+ years
Key Responsibilities:
- Design, develop, and maintain scalable data pipelines and ETRM systems.
- Implement real-time data replication from various source systems to Snowflake using Fivetran.
- Work on data integration projects within the Energy Trading and Risk Management (ETRM) domain.
- Collaborate with cross-functional teams to integrate data from ETRM trading systems like Allegro, RightAngle, and Endur.
- Optimize and manage data storage solutions in Data Lake and Snowflake.
- Develop and maintain ETL processes using Azure Data Factory and Databricks.
- Write efficient and maintainable code in Python for data processing and analysis.
- Ensure data quality and integrity across various data sources and platforms.
- Develop and maintain data pipelines using FiveTran, Snowflake, and Azure Data Factory (ADF).
- Ensure data accuracy, integrity, and availability across various trading systems.
- Collaborate with traders, analysts, and IT teams to understand data requirements and deliver robust solutions.
- Optimize and enhance data architecture for performance and scalability.
Mandatory Skills:
- Azure Data Factory (ADF)
- Data Lake
- Snowflake
- Python
- Databricks
- Fivetran
- Experience in ETRM domain
- Integration with trading systems like Allegro, RightAngle, Endur
Preferred Qualifications:
- Strong problem-solving skills and attention to detail.
- Excellent communication and collaboration skills.
- Ability to work in a fast-paced and dynamic environment.
- Experience with other data integration tools and technologies is a plus.
Education:
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- Certification in relevant technologies is a plus.