Summary: The Data Engineer role requires a strong understanding of data concepts, particularly in JSON manipulation and data security principles. The position involves working with data pipelines using Python and PySpark frameworks, and requires active SC clearance. The role is remote and has a duration of 6 months with a day rate of £470, classified as inside IR35.

Key Responsibilities:

• Strong understanding of data concepts - data types, data structures, schemas (both JSON and Spark), schema management etc.
• Strong understanding of complex JSON manipulation.
• Experience working with Data Pipelines using a custom Python/PySpark frameworks.
• Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.
• Strong understanding of Data Security principles - data owners, access controls - row and column level, GDPR etc including experience of handling sensitive datasets.
• Strong problem solving and analytical skills, particularly able to demonstrate these intuitively.

Key Skills:

• Languages/Frameworks - JSON, YAML, Python, SQL, PySpark, Delta Lake, Bash, Git, Markdown, Scala (bonus), Azure SQL Server as a HIVE Metastore (bonus).
• Technologies - Azure Databricks, Apache Spark, Delta Tables, Data processing with Python, PowerBI (Integration/Data Ingestion), JIRA.

Salary (Rate): £470

City: London

Country: UK

Working Arrangements: remote

IR35 Status: inside IR35

Seniority Level: undetermined

Industry: IT