We are seeking an experienced AWS Databricks Data Engineer to design, build, and maintain scalable data pipelines and cloudnative data infrastructure. The ideal candidate will have strong expertise in Apache Spark, PySpark, Python, SQL, Delta Lake, and AWS services, with proven experience in production Databricks environments.
Key Responsibilities
- Design and implement batch and streaming data pipelines using PySpark, Delta Lake, and Delta Live Tables (DLT).
- Develop scalable ingestion and transformation solutions within Databricks.
- Build integrations with AWS services such as S3, IAM, and Lambda.
- Configure and manage data governance, security, and access controls via Databricks Unity Catalog.
- Monitor and optimize pipeline performance, reliability, and data quality.
- Automate testing and deployment using CI/CD tools (GitHub/GitLab).
- Collaborate with data scientists and stakeholders to prepare datasets for analytics and machine learning.
- Support healthcare payer datasets including claims, provider, member enrollment, and MMIS data.
Required Skills & Experience
- 3 5 years of handson data engineering experience.
- Strong productionlevel experience with Databricks.
- Advanced programming skills in Python and PySpark.
- Advanced SQL expertise.
- Deep understanding of Apache Spark, Delta Lake, and Data Lakehouse architecture.
- Experience with Delta Live Tables, Unity Catalog, and Databricks Workflows.
- Familiarity with AWS cloud services for data storage and management.
Work Hours
- Must be available to work in the CST time zone.
Rate:
Not specified
Location:
Remote
IR35 Status:
Outside
Remote Status:
Remote
Industry:
Data & Analytics
Seniority Level:
Mid-Level