Data Engineer - ETL, PySpark, Python, SQL - SC Clearance
Posted 1 day ago by Apps IT Ltd
Negotiable
Undetermined
Undetermined
Wales, UK
Summary: The Data Engineer role focuses on developing and constructing data products and services, integrating them into systems and business processes. The position requires SC Clearance and involves working with ETL processes, Python, and SQL to optimize data flows and support the technical development of strategic processing systems. Collaboration with team members and adherence to Agile methodologies are essential components of the role.
Key Responsibilities:
- Develop and construct data products and services and integrate them into systems and business processes.
- Implement data flows to connect operational systems, data for analytics and Business Intelligence (BI) systems.
- Re-engineer manual data flows to enable scaling and repeatable use.
- Write ETL scripts and code to ensure optimal ETL process performance.
- Write PySpark, Python, SQL code to systematize data processing based on design documents.
- Write unit/integration tests and manage code versions with Git.
- Support the build of maintainable, adaptable, and re-runnable pipelines by the operational team.
- Maintain and review documentation for statistical and data processing functions and pipelines.
- Collaborate with colleagues to brainstorm solutions and share knowledge.
- Report on work progress to the line manager and peers.
Key Skills:
- Experience in programming and data engineering, including design, coding, testing, and documentation.
- Ability to review requirements and specifications, define test conditions, and analyze test activities.
- Understanding of core technical concepts related to the role.
- Strong role behaviors in teamwork, communication, influencing, and delivering at pace.
- Experience with Agile methodologies, JIRA, and Confluence.
Salary (Rate): undetermined
City: undetermined
Country: UK
Working Arrangements: undetermined
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Data Engineer, ETL, Python
Location: Remote/onsite
SC Clearance Required - please only apply if you have been a resident of the UK for the past 5 years.
Description:
- Develop and construct data products and services and integrate them into systems and business processes.
- Implement data flows to connect operational systems, data for analytics and Business Intelligence (BI) systems,
- Re-engineer manual data flows to enable scaling and repeatable use.
- Write ETL scripts and code to make sure the ETL process performs optimally,
- Write PySpark, Python, SQL code that will systemise the processing of data, based on initial design documents.
- This includes writing unit/integration tests and managing versions of code with Git.
- Support the build of pipelines by the operational team that can be easily maintained, adapted and re-run.
- Maintain and review documentation for statistical and data processing functions, as well as data processing pipelines.
- Collaboration - work with colleagues in the team to brainstorm possible solutions to problems and share knowledge and support each other.
- Report on the progress of your work to your LM and peers.
Working knowledge/experience of:
- Programming and build (data engineering) - you can design, code, test, correct and document simple programs or scripts under the direction of others
- Testing - You can review requirements and specifications, and define test conditions. You can identify issues and risks associated with work. You can analyse and report test activities and results
- Technical Understanding - you understand the core technical concepts related to the role, and apply them with guidance
- Role Behaviours - Working Together; Communicating and Influencing; Changing and Improving; Delivering at Pace.
Your primary focus will be to support the lead Data Engineer with the ongoing technical development of the strategic processing system. With an initial focus on processing Transformed Labour Force Survey (TLFS) data but expanding to meet requirements for other social surveys in due course.
You will work within a small coding team along with collaborators from other teams, including teams working on shorter term tactical solutions, within the division and across ONS to meet development and operational needs.
You will be required to work in an Agile manner aided via the use of JIRA and Confluence.