Position Summary
The QA Data Engineer is responsible for validating data pipelines, transformations, integrations, and reporting outputs across the Databricks platform. This role ensures data accuracy, completeness, reliability, and compliance while supporting enterprise analytics and AI initiatives.
Key Responsibilities
- Develop and execute data validation and testing strategies for Databricks pipelines.
- Create automated test cases for ETL/ELT processes and data transformations.
- Verify source-to-target mappings and business rule implementations.
- Perform reconciliation testing between source systems and target datasets.
- Develop data quality monitoring, controls, and exception reporting.
- Validate data model integrity and reporting outputs.
- Perform performance testing of large-scale data workloads.
- Collaborate with Data Engineers and Data Modelers to resolve defects.
- Support regression, integration, and user acceptance testing activities.
- Track data quality KPIs and testing metrics.
Required Skills
- SQL
- Databricks
- Data Warehousing concepts
- Data Validation and Reconciliation techniques
- Python
- ETL Testing
- QA methodologies
- Source-to-target mapping validation
Preferred Skills
- Great Expectations
- Databricks Data Quality Frameworks
- Azure DevOps
- Test automation tools
- Healthcare claims, member, provider, or clinical data experience
Experience
- 4+ years in Data Testing or Data QA
- Experience validating large-scale Databricks data platforms
- Strong analytical and troubleshooting skills
Success Metrics
- Defect leakage reduction
- Data quality score improvements
- Test automation coverage
- Reduced production data incidents
Rate:
Not specified
Location:
Remote
IR35 Status:
Outside
Remote Status:
Remote
Industry:
Data & Analytics
Seniority Level:
Mid-Level