Job Description
We are seeking a Senior Data Engineer to design, build, and operate highly scalable data platforms supporting a large, high-traffic digital business. The role involves massive datasets, complex data flows, and business-critical analytics workloads, where performance, reliability, scalability, and automation are essential.
The ideal candidate has hands-on experience building and operating production-grade data pipelines processing billions of rows/records, with strong expertise across distributed data processing, orchestration, data modeling, monitoring, and optimization.
This is not a dashboard-focused or basic ETL role. We are looking for an engineer who can take complex data problems from business requirements architecture pipeline development data modeling production deployment monitoring and optimization, while ensuring high performance, reliability, and scalability.
Responsibilities
- Design and implement scalable, production-grade data pipelines capable of processing billions of rows/records across multiple data sources.
- Build reliable data ingestion, transformation, processing, and storage workflows for large-scale analytical and operational use cases.
- Develop and optimize distributed data processing solutions using technologies such as Apache Spark.
- Build and manage workflow orchestration using Apache Airflow or similar platforms.
- Design data pipelines with strong focus on performance, scalability, reliability, recoverability, and operational efficiency.
- Monitor production pipelines, identify bottlenecks and failures, and proactively improve data reliability, pipeline uptime, and processing efficiency.
- Develop robust data models that translate complex business requirements into clean, scalable, and easily evolvable analytical structures.
- Write highly efficient and complex SQL for large-scale data transformation, analysis, validation, and troubleshooting.
- Use Python, Java, or Scala to develop scalable data processing and automation solutions.
- Identify repetitive manual processes and continuously automate, simplify, and eliminate operational overhead.
- Troubleshoot large-scale data and pipeline issues and drive root-cause resolution rather than relying on manual workarounds.
Experience
- Strong experience in Data Engineering / Big Data Engineering within high-volume production environments.
- Proven experience designing and operating data pipelines processing billions of rows/records.
- Extensive hands-on experience with Apache Spark and distributed data processing.
- Strong experience with Apache Airflow or comparable workflow orchestration frameworks.
- Advanced proficiency in SQL and experience working with very large datasets.
- Strong programming experience in Python, Java, or Scala.
- Proven expertise in data modeling, dimensional modeling, analytical data structures, and scalable data architecture.
- Experience with production data environments where availability, reliability, latency, scalability, and data quality are critical.
- Demonstrated mindset of automation and continuous improvement—you naturally look for ways to eliminate repetitive manual work.
Skillsets
- Experience handling multi-billion-row datasets or similarly high-volume event/data workloads.
- Experience with high-traffic consumer platforms, digital products, marketplaces, advertising, search, transaction, or behavioral analytics data.
- Experience building data platforms that support near-real-time or high-frequency data processing.
- Background in geographic information systems (GIS) or spatial analysis
Education
- Bachelor's degree or equivalent experience in related field.