Data Engineer with AWS, Databricks and Healthcare Experience : Remote

Role Summary

Senior engineering role responsible for designing and developing scalable AWS Databricks data pipelines, transformation frameworks, and reusable components for complex healthcare data workloads.

Roles & Responsibilities

• Design and develop PySpark and Spark SQL pipelines on AWS Databricks.
• Build reusable transformation frameworks, Delta tables, workflows, and metadata/configuration-driven processing components.
• Implement complex source-to-target mappings, state-specific transformation logic, crosswalks, reference-data processing, and business rules.
• Develop batch pipelines for full-refresh, incremental, historical, and reprocessing scenarios.
• Implement logging, error handling, audit controls, balance/control checks, lineage, and operational monitoring.
• Optimize Spark workloads for performance, scalability, and cost.
• Migrate or re-engineer legacy ETL logic from DataStage, SSIS, Informatica, C/Linux, SQL, and other batch platforms.
• Support data validation, reconciliation, defect analysis, and production-readiness activities.
• Participate in code reviews, technical design reviews, testing, and CI/CD.

Required Skills & Experience

• 10+ years of data engineering experience with strong hands-on development skills.
• Strong AWS Databricks, PySpark, Spark SQL, Delta Lake, and Databricks Workflows experience.
• Strong understanding of AWS S3-based data-lake patterns and secure integration with enterprise source systems.
• Strong SQL and experience building high-volume batch ETL / ELT pipelines.
• Experience with reusable and metadata-driven transformation frameworks.
• Experience with schema evolution, partitioning, Delta optimization, performance tuning, and large-volume data processing.
• Experience with Oracle, SQL Server, flat files, APIs, and enterprise data integrations.
• Experience with Git, CI/CD, automated testing, and deployment practices.
• Exposure to DataStage, SSIS, Informatica, C/Linux, or other legacy ETL technologies is preferred.
• Experience with Unity Catalog, access controls, lineage, and data-governance concepts is preferred.

Domain Knowledge

Healthcare payer / Medicaid / MMIS experience is preferred.

T-MSIS or other healthcare regulatory-reporting experience is a strong plus.

Understanding of claims, encounters, eligibility, provider, managed care, pharmacy, TPL, and financial transaction data is preferred.

Familiarity with source-to-target mappings, crosswalks, regulatory validation rules, and reconciliation is beneficial.