Data Engineer

Introduction

As a Data Engineer, you will be responsible for designing, building, and maintaining the infrastructure that supports the extraction, transformation, and loading of data from various sources into a data warehouse.

You will work closely with data scientists, analysts, and other stakeholders to ensure that the data is accessible, accurate, and secure.

Responsibilities

• Develop and maintain ETL processes to move data from source systems to the data warehouse
• Design and implement data pipelines for real-time and batch processing
• Optimize data pipelines for performance and scalability
• Collaborate with data scientists and analysts to understand their data requirements
• Ensure data quality and integrity through data validation and testing
• Identify and address performance bottlenecks in the data infrastructure
• Implement data security and privacy measures to protect sensitive information
• Document data infrastructure and processes for knowledge sharing

Required Skills:

• Proficiency in SQL and one or more programming languages (e.g. Python, Java)
• Experience with ETL tools and data processing frameworks (e.g. Apache Spark, Talend)
• Knowledge of data modeling and database design principles
• Strong problem-solving and analytical skills
• Ability to work independently and collaboratively in a team environment

Preferred Skills:

• Experience with cloud data services (e.g. AWS, Google Cloud Platform)
• Familiarity with big data technologies (e.g. Hadoop, Hive)
• Certifications in data engineering or related field
• Knowledge of machine learning and data visualization tools