Lead PySpark Engineer - Data, SAS, AWS
Posted 1 day ago by 1770824001
£380 Per day
Undetermined
Remote
London
Summary: The Lead Data Engineer role focuses on designing and developing complex data processing solutions using PySpark on AWS within the financial sector. The position involves modernizing legacy data workflows and supporting large-scale migrations from SAS to PySpark. Candidates are expected to have strong engineering skills and a deep understanding of data to deliver production-ready data pipelines. This role is remote and offers competitive daily rates.
Key Responsibilities:
- Design, develop, and fix complex data processing solutions using PySpark on AWS.
- Modernize legacy data workflows and support SAS-to-PySpark migrations.
- Deliver production-ready data pipelines in a financial services environment.
- Write and execute data and ETL test cases.
- Document code, data flows, and technical decisions clearly.
- Build and operate data pipelines on AWS.
Key Skills:
- Minimum 5+ years of hands-on PySpark experience.
- SAS to PySpark migration experience.
- Strong understanding of data warehousing concepts.
- Strong knowledge of Spark execution concepts.
- Strong Python coding skills.
- Experience with AWS core services.
- Proficiency in Git-based workflows.
- Experience in banking or financial services is desirable.
Salary (Rate): £380 per day
City: London
Country: United Kingdom
Working Arrangements: remote
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Lead Data Engineer - Pyspark/AWS/Python/SAS - Financial Sector
As a Lead PySpark Engineer, you will design, develop, and fix complex data processing solutions using PySpark on AWS. You will work hands-on with code, modernising Legacy data workflows and supporting large-scale SAS-to-PySpark migrations. The role requires strong engineering discipline, deep data understanding, and the ability to deliver production-ready data pipelines in a financial services environment.
Essential Skills
PySpark & Data Engineering
- Minimum 5+ years of hands-on PySpark experience.
- SAS to Pyspark migration experience
- Proven ability to write production-ready PySpark code.
- Strong understanding of data and data warehousing concepts, including: ETL/ELT, Data models, Dimensions and facts, Data marts, SCDs
Spark Performance & Optimisation
- Strong knowledge of Spark execution concepts, including partitioning, optimisation, and performance tuning.
- Experience troubleshooting and improving distributed data processing pipelines.
- Python & Engineering Quality
- Strong Python coding skills with the ability to refactor, optimise, and stabilise existing codebases.
- Experience implementing parameterisation, configuration, logging, exception handling, and modular design.
SAS & Legacy Analytics
- Strong foundation in SAS (Base SAS, SAS Macros, SAS DI Studio).
- Experience understanding, debugging, and modernising Legacy SAS code.
Data Engineering & Testing
- Ability to understand end-to-end data flows, integrations, orchestration, and CDC.
- Experience writing and executing data and ETL test cases.
- Ability to build unit tests, comparative testing, and validate data pipelines.
Engineering Practices
- Proficiency in Git-based workflows, branching strategies, pull requests, and code reviews.
- Ability to document code, data flows, and technical decisions clearly.
- Exposure to CI/CD pipelines for data engineering workloads.
AWS & Platform Skills
- Strong understanding of core AWS services, including: S3, EMR/Glue, Workflows, Athena, IAM
- Experience building and operating data pipelines on AWS.
- Big data processing on cloud platforms.
Desirable Skills
- Experience in banking or financial services.
- Experience working on SAS modernisation or cloud migration programmes.
- Familiarity with DevOps practices and tools.
- Experience working in Agile/Scrum delivery environments.
I have three roles available all of which can be worked remotely so don't delay and apply today. I have interview slots ready to be filled
Randstad Technologies is acting as an Employment Business in relation to this vacancy.