Data Engineer

Job Description

What is the Customer Insight Platform (CIP)?

CIP is composed of multiple agile, lean teams who together provide a platform and services to collect, evaluate and analyse customer interaction data.

Context: Teams across CIP

• CIP focuses on capturing customer interaction data across the HMRC estate and making it easily available to a wide audience through a number of different channels.
• CIP processes and stores a LOT of customer interaction data, using cutting edge & emerging technologies to perform bulk operations at scale, whilst operating, maintaining, and improving a critical platform that our consumers depend on.
• It also provides a number of different channels that allow teams within HMRC to access and analyse the data in a digestible format to support their day to day responsibilities.
• CIP is driven by the principal need to detect and prevent fraudulent access to taxpayer data and prevent organised repayment fraud.
• Attribute Reputation is the process through which large amounts of data are collated from tax transactions to glean insights into common attributes.
• In particular, we assert their validity and derive risk scores to transactions based on whether the attributes in question, or other attributes used in combination with that attribute, have been used in historic transactions associated with past suspicious or fraudulent activity.
• Collaboration with other data product and Front End product teams will form the core of creating a frictionless customer experience to exploit the digital audit data that CIP holds.
• Insights from Attribute Reputation are crucial for HMRC services, and the team is pioneering data sharing with other government departments and local authorities.

Role description

• As a Data Engineer in the AttRep Team you will define and shape the way we build and transform our audit data products.
• These data products will be used by our Front End Attribute Reputation services (APIs) to provide attribute insights for fraud detection and prevention across HMRC.
• It is vital that CIP Data Engineers are as equally committed to operating, maintaining and improving our pipelines and channels as they are to creating the forward-looking features.
• We live by the You build it, you run it ethos. We would like to develop good software engineers with an interest in working on a data platform as CIP Data Engineers.
• The sheer scale of information our data products provide CIP Data Engineers with the opportunity to explore cutting edge & emerging technologies.
• Engineers should be comfortable reaching out to technology communities in order to learn, share and create new possibilities for our data products.
• Whether this is focussed on improving the quality of the data we capture or how we summarise and present that data to different audiences, creativity and imagination is a must.
• CIP Data Engineers must understand the importance of testing, automation and repeatable reliability, for all of the things we build and run.

This work involves:

• Munging large data sets from multiple sources, including audit data captured from Digital Services
• Collaborating with multidisciplinary team to provide data business solutions, including publishing data and sharing data
• Creating and productionising Jupyter notebooks
• Data processing, validation and cleansing
• Summarizing and enriching data sets for convenient use by consumers
• Set up, maintain and improve automated data processes

Desired behaviours

• Be proactive
• Responsible and motivated self-starter
• User centric attitude
• Prioritise people over processes
• Drive lean results; understand the value of iterative delivery
• Be evidence driven
• Advocate for doing the right thing'
• Help others understand your work through co-participatory design, pairing, demos and documentation
• Make complex look and sound simple
• Be kind, creative and convincing

How we work:

We favour:

Breaking down a high-level problem as a team upfront and delivering the core value first

Delivering individual business benefits to production, rather than bundling tasks together

Collaboration, communication, regular pairing and thoughtful course-correction

Taking the time to write simple and easy to maintain code

Willingness to make and participate in improvements for the greater good of CIP

Working cross-discipline rather than adhering to tightly defined roles.

Knowledge & experience

• Security clearance (SC-level)
• Data Analysis
• Data Transformation and ETL
• Databases
• Understanding of Data Science principles
• AWS suite experience
• Skilled at returning services to good states in outage situations
• Understands the importance of testing and automation
• Working in a lean, agile environment
• Working in a you build it you run it team
• Experience coding in continuous integration context

Skills & technologies

Essential:

Python

Spark

SQL

Database experience

AWS Suite

Desirable:

Scikit Learn/Spark MLlib

Hudi or other open table format

AWS EMR

Continuous Integration

Airflow

Jenkins

Kibana

Grafana