Senior Data Engineer - Python, Databrick, AI

Position: Senior Data Engineer - Python, Databrick

Location: London

Job Type: Contract

Job Responsibilities

• Develop workflows and ELT data pipelines using Python, Spark/PySpark, and Databricks
• Onboard enterprise datasets into Palmos, including ingestion, transformation, and validation of data assets
• Build, test, and maintain scalable data pipelines and data architectures that support enterprise controls and analytics use cases
• Build business controls algorithms for communications data to identify anomalies
• Build data completeness and integrity controls for the pipelines
• Apply data engineering best practices for performance optimization, reliability, and maintainability
• Use SQL extensively and work with both relational and NoSQL data stores
• Partner with producers to understand data requirements and translate them into production-ready solutions
• Apply SDLC practices including CI/CD, testing, and operational monitoring to ensure pipeline stability
• Contribute to reusable frameworks and standards to accelerate onboarding and pipeline delivery
• Identify data issues, anomalies, and optimization opportunities to improve data quality and performance
• Leverage enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity across complex deliverables (eg, code generation/refactoring, unit test creation, documentation), while validating outputs through peer review, automated testing, and secure coding standards; contribute learnings and reusable patterns to improve broader team effectiveness

Required Qualifications, Capabilities, and Skills:

• Hands-on experience with Databricks, Spark/PySpark, Python, and SQL
• Experience developing and maintaining data pipelines and data processing systems
• Understanding of the data life cycle, including ingestion, transformation, storage, and consumption
• Knowledge of cloud platforms (AWS) and distributed data processing
• Experience with SDLC practices including CI/CD, testing, and deployment
• Strong problem-solving skills and ability to troubleshoot data and pipeline issues
• Ability to collaborate effectively within agile teams and across stakeholders
• Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (eg, for coding, test creation, troubleshooting, or documentation) with demonstrated ability to critically evaluate, validate, and refine AI-generated outputs for correctness, performance, and security
• Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; ability to guide peers on safe and effective usage within team practices

Preferred Qualifications, Capabilities, and Skills:

• Experience with Databricks lakehouse, Databricks Genie, Delta Lake, and medallion architecture
• Familiarity with enterprise data platforms and data mesh principles
• Exposure to data quality, observability, and metadata management tools
• Experience supporting analytics, reporting, or AI/ML workloads
• Experience working on regulatory control