Hadoop Developer

Hadoop Developer

Posted 1 week ago by GIOS Technology

Apply
Negotiable
Undetermined
Onsite
Northampton, England, United Kingdom
Apply
Apache Flume Apache Hadoop Apache HBase Apache Hive Apache Kafka Apache Pig Apache Spark Big Data Data Ingestion Data Security Debugging Extract Transform Load (ETL) Google Cloud Platform (GCP) Hadoop Distributed File System (HDFS) Java (Programming Language) MapReduce Microsoft Azure Nice (Unix Utility) Python (Programming Language) Scala (Programming Language) Software Development Sqoop Technical Documentation

Summary: The role of Hadoop Developer involves developing and debugging Spark jobs in Scala, with a focus on Hadoop-based applications and data pipelines. The position requires collaboration with data scientists and analysts to meet data needs while ensuring compliance with organizational policies. The developer should also be familiar with various big data tools and technologies, including Hive, Pig, and cloud-based solutions. This is an onsite role based in Northampton, UK, requiring 2-3 days per week attendance.

Key Responsibilities:

  • Design and develop Hadoop-based applications and data pipelines.
  • Build, operate, monitor, and troubleshoot Hadoop clusters.
  • Write scalable ETL processes using tools like Hive, Pig, and Spark.
  • Develop and maintain data ingestion processes using Sqoop, Flume, or Kafka.
  • Optimize MapReduce jobs and manage HDFS storage.
  • Collaborate with data scientists and analysts to support data needs.
  • Ensure data security and compliance with organizational policies.
  • Create and maintain technical documentation and playbooks.
  • Evaluate and integrate cloud-based big data solutions (AWS, GCP, Azure).

Key Skills:

  • Proficiency in Hadoop, Spark, Scala, and Python.
  • Experience with Hadoop core concepts (MapReduce), Hive, Pig, and HBase.
  • Ability to debug and develop Spark jobs.
  • Familiarity with data ingestion tools like Sqoop, Flume, or Kafka.
  • Knowledge of cloud-based big data solutions (AWS, GCP, Azure).
  • Strong problem-solving and troubleshooting skills.
  • Ability to create and maintain technical documentation.
  • Understanding of data security and compliance requirements.

Salary (Rate): undetermined

City: Northampton

Country: United Kingdom

Working Arrangements: on-site

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

We are hiring for Hadoop (Spark/Scala) Developer for our client's project based at Northampton, UK – 2-3 days per week Onsite

Role Description: In general, the resource should be able to comfortably debug and develop Spark jobs in Scala. Alternatively, comfortability in Java Bigdata development and exposure is acceptable. Python is also nice to have. Looking for Hadoop, Spark, Scala / Python developer. Candidate should be conversant with concept of Hadoop core (MapReduce), Hive, Pig HBase

Key Responsibilities:

  • Design and develop Hadoop-based applications and data pipelines.
  • Build, operate, monitor, and troubleshoot Hadoop clusters.
  • Write scalable ETL processes using tools like Hive, Pig, and Spark.
  • Develop and maintain data ingestion processes using Sqoop, Flume, or Kafka.
  • Optimize MapReduce jobs and manage HDFS storage.
  • Collaborate with data scientists and analysts to support data needs.
  • Ensure data security and compliance with organizational policies.
  • Create and maintain technical documentation and playbooks.
  • Evaluate and integrate cloud-based big data solutions (AWS, GCP, Azure).
Apply
Similar Jobs
Loading data...