Job Summary
We are looking for an experienced Data Engineer with 9+ years of experience to design, build, and maintain data pipelines using Databricks, Spark, and Azure cloud services.
The ideal candidate has hands-on experience with modern data engineering tools, cloud automation, and event-driven architectures.
Job Responsibilities
- Design, develop, and optimize scalable data pipelines using Databricks and Spark (PySpark)
- Build and maintain ETL/ELT workflows integrating with Snowflake and other data platforms
- Develop and manage Azure-based data solutions using ADF, ADLS, Azure Functions, and Key Vault
- Implement CI/CD pipelines and infrastructure automation using GitHub Actions, Terraform, and DevOps best practices
- Build and maintain event-driven architectures using Kafka, including CDC and streaming integrations
- Work with AI/GenAI initiatives involving LLM fine-tuning, RAG, and prompt engineering
- Monitor, troubleshoot, and optimize data pipeline performance and reliability
- Collaborate with data architects, analysts, and business stakeholders to translate requirements into technical solutions
- Ensure data quality, security, and governance standards are met across pipelines
- Orchestrate and schedule workflows using Airflow
Required Skills
- 8+ years of experience in Data Engineering
- Data Engineering: Databricks, Spark (PySpark), Python/Scala, SQL, Snowflake
- Azure: Azure Cloud Services, ADF, ADLS, Azure Functions, Key Vault, DevOps
- AI/GenAI: LLM Fine-Tuning, RAG, Prompt Engineering
- Automation & DevOps: Airflow, GitHub, GitHub Actions, CI/CD, Terraform, Infrastructure Automation
- Kafka, Event-Driven Architecture, CDC/Streaming Integrations
Preferred Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, or related field
- Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect) is a plus
- Experience working in large-scale enterprise or regulated industry environments (finance, healthcare, insurance)
- Strong communication skills with ability to work directly with client stakeholders
Rate:
Not specified
Location:
Remote
IR35 Status:
Outside
Remote Status:
Remote
Industry:
Data & Analytics
Seniority Level:
Senior