Key Responsibilities
- Design, develop, and maintain large-scale data pipelines across Google Cloud Platform.
- Build scalable ETL/ELT solutions using BigQuery, Dataproc, PySpark, Python, and Cloud Composer.
- Develop and optimize data processing workflows using Spark and Spark SQL.
- Create and maintain workflows and DAGs using Apache Airflow / Cloud Composer.
- Develop data solutions using Google Cloud Storage (GCS) and other Google Cloud Platform services.
- Implement and support enterprise Data Quality frameworks.
- Perform data validation, reconciliation, profiling, and quality checks.
- Collaborate with business stakeholders, data architects, and governance teams to understand requirements and deliver reliable data solutions.
- Apply data modeling and performance-tuning techniques to improve data processing and query performance.
- Troubleshoot and resolve data pipeline and production issues.
- Participate in requirements gathering, technical design, development, testing, deployment, and production support.
- Follow Agile development practices and participate in sprint planning, stand-ups, reviews, and retrospectives.
- Maintain code using Git/GitHub and support CI/CD processes.
Required Qualifications
- 5+ years of experience in Data Engineering.
- 5+ years of hands-on experience with Google Cloud Platform (Google Cloud Platform).
- Strong experience with:
- Google BigQuery
- Google Dataproc
- Cloud Composer / Apache Airflow
- Google Cloud Storage (GCS)
- Advanced experience with:
- PySpark
- Python
- Spark SQL
- ETL / ELT development
- Strong understanding of data quality, data validation, reconciliation, and data governance.
- Experience with data modeling and performance tuning.
- Strong understanding of enterprise-scale cloud data architecture and solutions.
- Experience working in Agile/Scrum environments.
- Hands-on experience with Git, GitHub, Jira, and CI/CD pipelines.
- Strong troubleshooting, analytical, and problem-solving skills.
Rate:
Not specified
Location:
Remote
IR35 Status:
Outside
Remote Status:
Remote
Industry:
Data & Analytics
Seniority Level:
Senior