Senior Data Engineer – Middle Office Data-Remote(USA)-W2 Role

Key Responsibilities

• Onboard new datasets required to support Middle Office business processes, analytics, reporting, and downstream applications.
• Work with business and technology stakeholders to understand data requirements, source systems, data structures, business rules, and downstream consumption patterns.
• Design scalable data ingestion, transformation, and processing solutions using AWS and Apache Spark/PySpark.
• Develop and maintain production-grade data pipelines using services such as AWS EMR, AWS Glue, Amazon S3, and Amazon Athena.
• Design and work with large-scale data lake tables using technologies such as Apache Hudi and Apache Iceberg.
• Build workflow orchestration and scheduling solutions using Apache Airflow or equivalent orchestration technologies.
• Perform data modeling and design appropriate schemas, partitioning strategies, data structures, and transformation layers for analytical and operational use cases.
• Build pipelines with appropriate operational capabilities, including monitoring, logging, alerting, data-quality validation, failure handling, retries, and recovery mechanisms.
• Develop clean, reusable, testable, and maintainable code primarily using Python and PySpark.
• Implement and maintain automated deployment processes using CI/CD and infrastructure-as-code practices, including GitHub Actions and Terraform.
• Troubleshoot data-quality, pipeline-performance, and production issues and identify appropriate long-term solutions.
• Produce appropriate technical documentation covering data lineage, pipeline design, business rules, dependencies, and operational procedures.
• Collaborate effectively with data platform, business, Middle Office, and downstream technology teams.

Required Technical Skills

• Strong hands-on experience with AWS data and analytics services, particularly: AWS EMR, AWS Glue and Glue Data Catalog, Amazon Athena, Amazon S3, AWS IAM and general AWS security concepts
• The candidate should understand how these services work together when designing and operating enterprise-scale data pipelines.
• Expert-level or strong hands-on experience with: Apache Spark, PySpark, Distributed data processing, Large-scale dataset transformation and optimization, Spark performance tuning and troubleshooting, Partitioning and efficient data processing strategies.
• Strong experience working with modern data lake/table formats, particularly: Apache Hudi, Apache Iceberg
• The candidate should understand concepts such as schema evolution, partitioning, incremental processing, upserts, snapshots, and management of large analytical tables.
• Strong programming skills in: Python, PySpark
• Candidate should consistently produce clean, modular, maintainable, and production-quality code, with experience in: Object-oriented and modular design where appropriate, Unit and integration testing, Error and exception handling, Logging, Code reviews, Git-based development workflows
• Strong hands-on experience with Apache Airflow or a comparable enterprise orchestration platform.
• Candidate should be comfortable designing: DAGs and workflow dependencies, Scheduling, Retry and failure-handling strategies, Backfill and rerun processes, Monitoring and alerting, Parameterized and reusable workflows
• Experience with Amazon MWAA is an asset.
• Strong understanding of: Data modeling, Data pipeline architecture, Data lake architecture, Schema design, Data partitioning, Incremental data processing, Historical data management, Data quality and reconciliation, Data lineage
• Candidate should be capable of translating business requirements into a scalable technical data model and an operationally supportable data pipeline.
• Working knowledge of modern DevOps and infrastructure-as-code practices, including: Terraform, GitHub Actions, Git, CI/CD pipelines, Automated testing and deployment, Environment configuration and promotion
• The candidate does not need to be a dedicated DevOps engineer but should be comfortable working with infrastructure definitions and deployment pipelines as part of normal data engineering development.

Preferred Qualifications

• Previous experience building data platforms or data products within financial services, asset management, capital markets, or investment management.
• Experience supporting Middle Office, investment operations, risk, portfolio management, or trading-related datasets.
• Experience onboarding datasets from external vendors, APIs, files, databases, or enterprise data platforms.
• Understanding of data governance, lineage, entitlement, security, and audit requirements in an enterprise environment.
• Experience designing pipelines that process both historical and incremental/daily datasets.
• Experience troubleshooting and optimizing large-scale Spark workloads.
• Experience operating production data pipelines with defined SLAs and operational support requirements.

Education:

• Bachelor's or Master’s degree in Computer Science, Computer or Electrical Engineering, Mathematics, or a related field.