Data Engineer

Description

Remote

Our client seeks a data engineer to collaborate with cross-functional teams and own systems, processes, and the tech stack across the SDLC. The role focuses on building and optimizing scalable data pipelines across Azure and Google Cloud Platform, implementing feature engineering for machine learning with Python and Vertex AI, and ensuring quality through automated testing and CI/CD. The engineer will modernize and extend data assets across SQL, NoSQL, and streaming platforms while documenting architectures and interfaces.

We can facilitate w2 and corp-to-corp consultants. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.

Responsibilities

• Design, build, and maintain scalable and reliable data pipelines for ingestion, transformation, and integration using tools such as Kafka and Databricks.
• Leverage innovative technologies to modernize and extend core data assets across SQL, NoSQL, cloud, and real-time streaming platforms.
• Develop and manage feature engineering pipelines for machine learning using Vertex AI, BigQuery ML, and Python.
• Design and implement automated unit, integration, and performance testing frameworks to ensure data quality and reliability.
• Optimize data workflows for performance, cost efficiency, and scalability across large datasets and complex environments.
• Draft and review architectural diagrams, interface specifications, and design documents to communicate data solutions and requirements.
• Own systems, processes, and tech stack elements and drive features to completion across internal and external applications.

Experience Requirements

• 4+ years of professional data development experience.
• 4+ years with SQL and NoSQL technologies.
• 3+ years building and maintaining data pipelines and workflows.
• 2+ years developing with Python.
• Experience with feature engineering for machine learning pipelines.
• Experience with CI/CD pipelines and processes.
• Experience with automated unit, integration, and performance testing.
• Experience with version control such as Git.
• Understanding of ETL and data warehousing concepts.
• Understanding of Agile principles, including Scrum.
• Preferred: Knowledge of structured streaming with Spark, Kafka, or EventHub.
• Preferred: Experience with Google Cloud Platform services such as BigQuery, Vertex AI Platform, Cloud Storage, AutoMLOps, and Dataflow, or Databricks equivalents.
• Preferred: Experience with GitHub SaaS and GitHub Actions.
• Preferred: Understanding of Databricks concepts, PySpark, and Spark development.
• Preferred: Experience with service-oriented architecture.