Job Description
- Review and translate requirements to end-end technical design with a goal of building data products
- Develop pipelines in Databricks using PySpark and DBT to onboard various sources
- Working knowledge in Kafka to stream the data into databricks platform
- Identify Common approaches and best practices to data integration such as metadata drive ingestion, data validations, complex data type handling
- Drive standards for reusability, while developing pipelines
- Create or update documentation associated to key decisions on requirements, design and development
- Liaise with architects, data owners and users for reviews on day-day basis to ensure the quality of deliverables are high.
Skill Proficiency Requirements
- Experience in providing data engineering solutions
- Working knowledge of below technology toolsets:
- Cloud (Azure/AWS)
- Databricks, Data Mesh, Data as a Product
- Python
- Data streaming (Kafka, EventHub, Kinesis, SNS)
- Cloud-based API management - good to have
- Immuta - good to have
- Multi-cloud and on-prem application integration - good to have
Rate:
Not specified
Location:
United States
IR35 Status:
Outside
Remote Status:
Remote
Industry:
Data & Analytics
Seniority Level:
Not Specified