CDI Data Engineer

3 CDI Data Engineering vacancies, differing mainly in where they sit within the customer data lifecycle.

The first is the most data-centric, the second is the most DevOps/MLOps-oriented, and the third is the most transformation-focused.

2-3 days a week in office based in Canary Wharf.

Client Details: Our client operates within the Oil and Gas sector. British Petroleum.

Description: Role 1: CDI Data Engineer

• Implement and maintain SCI base, fact and curated pipelines.
• Deliver Elavon/PAR/WRE/payment and loyalty matching.
• Support ICE-to-EV data engineering and dashboard datasets.
• Implement DO-site and market-specific logic.
• Investigate volume drops, match-rate changes and data completeness defects.
• Execute controlled reloads, production validation, DQ/freshness controls and BAU support.
• Prepare PRs, tests, technical designs, CAB/release evidence and stakeholder

• Automate end-to-end BNA orchestration
• Reliable retry/recovery and idempotent processing
• Production monitoring and alerting automated activation and acknowledgement handling
• Reducing manual intervention Stable QA ? production deployment process
• Runbooks and operational handover
• Reducing runtime and production failures

• Repointing remaining CIS/SCI dependencies
• Stabilising migrated pipelines
• Reusable error-handling patterns
• Reducing Legacy technical debt
• Removing dual-platform monitoring
• Resolving SFMC/Comarch integration issues
• Reducing production support burden
• Clear handover into BAU

Profile: Role 1: CDI Data Engineer

Must Have worked with:

• Strong SQL, Python/Spark and Databricks pipeline engineering background.
• Large-scale transaction, payment, loyalty, EPOS/WRE and identity-matching work history
• Historical/full/incremental loads, high-water marks and reconciliation.
• Azure DevOps PRs, unit testing, CI/CD and production release support.

Nice to have:

• Reporting/Power BI dataset exposure desirable.
• SFMC/CIS integration and multi-market SCI knowledge desirable.

Role 2: BNA Production-Readiness, MLOps & Integration Engineer: Must Have:

• Strong hands-on background in Databricks, PySpark, Python and SQL Production pipeline engineering and troubleshooting
• Workflow orchestration and dependency management Idempotency, retry/restart and failure-recovery patterns CI/CD, version control and automated deployment
• Worked across monitoring, logging and operational alerting
• Completed data-quality and reconciliation controls API/file/SFTP integration patterns

Nice to have:

• Delta Lake/Delta MERGE Databricks Asset Bundles Azure DevOps MLOps/model deployment work history
• SFMC or Comarch integration
• Customer decisioning/personalisation platforms

Role 3: CIS/SCI Integration & Post-Migration Stabilisation Engineer: Must Have worked with:

• Strong hands-on background with Databricks, SQL, Python/PySpark ETL/ELT Data-pipeline migration
• Legacy-to-cloud modernisation
• Production troubleshooting
• Dependency analysis
• Pipeline repointing
• Error handling and recovery
• Data validation/reconciliation
• Integration testing
• Production support/L2-L3 troubleshooting

Nice to have:

• AWS-to-Databricks migration
• Informatica/SSIS/Legacy ETL
• SFMC Comarch SFTP/file interfaces
• Azure/AWS CI/CD

Job Offer

£200-£400 day rate

2-3 days a week in office

12 months contract

Inside IR35 only.