Introduction:
As a Data Engineer / ETL Developer, you will play a crucial role in our team by developing and maintaining ETL pipelines, optimizing data workflows, and ensuring the smooth operation of our data infrastructure.
You will work closely with cross-functional teams to support various projects and initiatives that require efficient data processing and analysis.
Responsibilities:
- Develop and optimize ETL pipelines using Apache NiFi, PySpark, and AWS Glue.
- Write and optimize complex SQL queries for data extraction, transformation, and loading.
- Work with MySQL, PostgreSQL, and AWS DocumentDB to manage and manipulate data efficiently.
- Utilize Apache Kafka or similar streaming platforms for real-time data processing.
- Provide development and production support for various projects and initiatives.
- Troubleshoot production issues, perform root cause analysis, and implement necessary fixes.
- Coordinate and execute System Integration Testing, Regression Testing, and data validation processes.
Requirements:: Required Skills:
- At least 3 years of hands-on experience in Apache Kafka or similar streaming platforms.
- Strong proficiency in Apache NiFi, PySpark, and AWS Glue for ETL development.
- Expertise in writing and optimizing complex SQL queries, performance tuning, and database management (MySQL, PostgreSQL, AWS DocumentDB).
- Experience with other data processing technologies such as Apache Flink or Apache Beam.
- Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform.
- Knowledge of data warehousing concepts and tools.
Rate:
Not specified
Location:
Richmond
IR35 Status:
Outside
Remote Status:
Remote
Industry:
Data & Analytics
Seniority Level:
Mid-Level