lead pyspark engineer - data, sas, aws
Posted Today by 1770880906
£380 Per day
Undetermined
Remote
london, london
Summary: The Lead Data Engineer role focuses on designing and developing complex data processing solutions using PySpark on AWS within the financial sector. The position involves modernising legacy data workflows and supporting large-scale migrations from SAS to PySpark. Candidates must possess strong engineering skills and a deep understanding of data to deliver production-ready data pipelines. The role is fully remote, offering flexibility in work arrangements.
Key Responsibilities:
- Design, develop, and fix complex data processing solutions using PySpark on AWS.
- Modernise legacy data workflows and support SAS-to-PySpark migrations.
- Write production-ready PySpark code and ensure data pipeline integrity.
- Implement performance tuning and optimisation for Spark execution.
- Document code, data flows, and technical decisions clearly.
- Build and execute data and ETL test cases.
- Utilise core AWS services for data pipeline operations.
Key Skills:
- 5+ years of hands-on experience with PySpark.
- Experience with SAS to PySpark migration.
- Strong understanding of data warehousing concepts.
- Proficiency in Python coding and engineering quality practices.
- Strong foundation in SAS and experience with legacy code modernisation.
- Ability to understand end-to-end data flows and integrations.
- Experience with Git-based workflows and CI/CD pipelines.
- Strong understanding of core AWS services.
Salary (Rate): £380 per day
City: London
Country: United Kingdom
Working Arrangements: remote
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Lead Data Engineer - Pyspark / AWS / Python / SAS - Financial Sector
As a Lead PySpark Engineer, you will design, develop, and fix complex data processing solutions using PySpark on AWS. You will work hands-on with code, modernising legacy data workflows and supporting large-scale SAS-to-PySpark migrations. The role requires strong engineering discipline, deep data understanding, and the ability to deliver production-ready data pipelines in a financial services environment.
Essential Skills
PySpark & Data Engineering
- Minimum 5+ years of hands-on PySpark experience.
- SAS to Pyspark migration experience
- Proven ability to write production-ready PySpark code.
- Strong understanding of data and data warehousing concepts, including: ETL/ELT, Data models, Dimensions and facts, Data marts, SCDs
Spark Performance & Optimisation
- Strong knowledge of Spark execution concepts, including partitioning, optimisation, and performance tuning.
- Experience troubleshooting and improving distributed data processing pipelines.
- Python & Engineering Quality
- Strong Python coding skills with the ability to refactor, optimise, and stabilise existing codebases.
- Experience implementing parameterisation, configuration, logging, exception handling, and modular design.
SAS & Legacy Analytics
- Strong foundation in SAS (Base SAS, SAS Macros, SAS DI Studio).
- Experience understanding, debugging, and modernising legacy SAS code.
Data Engineering & Testing
- Ability to understand end-to-end data flows, integrations, orchestration, and CDC.
- Experience writing and executing data and ETL test cases.
- Ability to build unit tests, comparative testing, and validate data pipelines.
Engineering Practices
- Proficiency in Git-based workflows, branching strategies, pull requests, and code reviews.
- Ability to document code, data flows, and technical decisions clearly.
- Exposure to CI/CD pipelines for data engineering workloads.
AWS & Platform Skills
- Strong understanding of core AWS services, including: S3, EMR / Glue, Workflows, Athena, IAM
- Experience building and operating data pipelines on AWS.
- Big data processing on cloud platforms.
Desirable Skills
- Experience in banking or financial services.
- Experience working on SAS modernisation or cloud migration programmes.
- Familiarity with DevOps practices and tools.
- Experience working in Agile/Scrum delivery environments.
I have three roles available all of which can be worked remotely so dont delay and apply today. I have interview slots ready to be filled
Randstad Technologies is acting as an Employment Business in relation to this vacancy.