Sr Data Engineer (Databricks, Spark) – Production Support Lead
Posted 2 weeks ago by Next Level Business Services, Inc.
Negotiable
Undetermined
Remote
Remote
Summary: The Lead Data Engineer role focuses on providing production support for enterprise data platforms, specifically utilizing Databricks. The position requires a combination of hands-on data engineering and incident management to ensure the reliability and performance of data pipelines. The engineer will serve as the primary escalation point for critical issues and collaborate with various teams to maintain operational standards. This is a contract position with potential for extension, based remotely in the USA.
Key Responsibilities:
- Lead end-to-end production support for data pipelines, ETL/ELT workflows, and Databricks environments
- Serve as the L3 escalation point for high-priority incidents and outages
- Monitor, troubleshoot, and resolve issues across batch and real-time data pipelines
- Perform root cause analysis (RCA) and implement preventive fixes
- Collaborate with Data Engineering, DevOps, and business teams for SLA adherence
- Optimize Databricks jobs, clusters, and cost efficiency
- Build and maintain monitoring, alerting, and support runbooks
- Lead on-call rotations and ensure operational readiness
- Communicate effectively with stakeholders during incidents and provide post-incident reports
Key Skills:
- 8+ years of experience in Data Engineering / Production Support
- Strong hands-on experience with Databricks (Spark, Delta Lake, Workflows)
- Proficiency in Python, SQL, and Apache Spark
- Experience with cloud platforms: AWS / Azure / Google Cloud Platform
- Strong understanding of ETL/ELT pipelines and orchestration tools (Airflow preferred)
- Experience with monitoring tools, incident management, and support models (L2/L3)
- Knowledge of data quality, governance, and security best practices
Salary (Rate): £55 hourly
City: undetermined
Country: USA
Working Arrangements: remote
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
Job Title: Lead Data Engineer – Databricks | Production Support
Location: Remote (USA)
Type: 6 Months Contract + Possible extension
Rate: Upto $60 Per hour W2 + benefits
Job Overview
We are looking for a Lead Data Engineer with strong Databricks experience to own and lead production support for enterprise data platforms. This role combines hands-on data engineering with incident management, ensuring high availability, performance, and reliability of data pipelines and analytics systems.
You will act as the primary escalation point for critical issues, drive root cause analysis, and partner with engineering and business teams to maintain SLA-driven operations.
Key Responsibilities
Lead end-to-end production support for data pipelines, ETL/ELT workflows, and Databricks environments
Serve as the L3 escalation point for high-priority incidents and outages
Monitor, troubleshoot, and resolve issues across batch and real-time data pipelines
Perform root cause analysis (RCA) and implement preventive fixes
Collaborate with Data Engineering, DevOps, and business teams for SLA adherence
Optimize Databricks jobs, clusters, and cost efficiency
Build and maintain monitoring, alerting, and support runbooks
Lead on-call rotations and ensure operational readiness
Communicate effectively with stakeholders during incidents and provide post-incident reports
Required Skills & Experience
8+ years of experience in Data Engineering / Production Support
Strong hands-on experience with Databricks (Spark, Delta Lake, Workflows)
Proficiency in Python, SQL, and Apache Spark
Experience with cloud platforms: AWS / Azure / Google Cloud Platform
Strong understanding of ETL/ELT pipelines and orchestration tools (Airflow preferred)
Experience with monitoring tools, incident management, and support models (L2/L3)
Knowledge of data quality, governance, and security best practices
Preferred Qualifications
Experience supporting large-scale enterprise data platforms
Familiarity with CI/CD pipelines and DevOps practices
Exposure to real-time/streaming data processing
Soft Skills
Strong incident leadership and decision-making under pressure
Ability to mentor and guide junior engineers
Excellent communication and stakeholder management skills
Proactive mindset with focus on automation and continuous improvement
Why Join
Opportunity to work on modern data platforms (Databricks, Spark)
High-impact role with ownership of critical data systems
Collaborative and fast-paced environment