Big Data Hadoop Engineer
Posted 2 weeks ago by 1752739506
Negotiable
Outside
Remote
USA
Summary: The Big Data Hadoop Engineer role focuses on developing and maintaining data pipelines using Hadoop components and various programming languages, including Python, Java, and Scala. The position requires strong expertise in Big Data technologies, REST API development, and experience in AI/Machine Learning projects. The role is fully remote and classified as outside IR35, indicating a flexible working arrangement for independent contractors. Candidates should possess a robust analytical skill set and the ability to communicate effectively across various levels of an organization.
Key Responsibilities:
- Develop and maintain data pipelines using Hadoop components such as Sqoop, Hive, SOLR, MR, Impala, Spark, and HBase.
- Implement REST API development using Python frameworks (Django, Flask, Fast API) and Java Springboot frameworks.
- Engage in AI/Machine Learning and NLP project development.
- Analyze information and formulate solutions to complex problems.
- Collaborate with team members and communicate effectively with users at all levels.
- Document processes and procedures as needed, ensuring project management practices are followed.
- Work independently on assigned tasks and deliverables with minimal supervision.
Key Skills:
- Strong experience in Big Data and Cloudera Distribution 7.x.
- 4-5 years of programming experience in Python, Java, Scala, and SQL.
- Proficiency in building data pipelines and REST API development.
- Experience with AI/Machine Learning and NLP development.
- Strong analytical skills and problem-solving abilities.
- Familiarity with Gitlab, Jenkins, JIRA, and Unix/Linux environments.
- Knowledge of Machine Learning algorithms and real-time analytics.
Salary (Rate): undetermined
City: undetermined
Country: USA
Working Arrangements: remote
IR35 Status: outside IR35
Seniority Level: undetermined
Industry: IT
Big Data Hadoop Engineer - MT 1780
Remote
Primary Skills: Scala, Java, Spark, Hive, Spark SQL, impala, Flask, Strong experience in Big Data, Cloudera Distribution 7.x,RDBMS development 4-5 years of programming experience in Python, and SQL is must. Strong experience building data pipelines using Hadoop components Sqoop, SOLR, MR, HBase. Strong experience with REST API development using Python frameworks (Django, Fast API etc.) and Java Springboot frameworks Project experience in AI/Machine Learning and NLP development.
Must Haves:
Strong experience in Big Data, Cloudera Distribution 7.x, RDBMS development
4-5 years of programming experience in Python, Java, Scala, and SQL is must.
Strong experience building data pipelines using Hadoop components Sqoop, Hive, SOLR, MR, Impala, Spark, Spark SQL, HBase.
Strong experience with REST API development using Python frameworks (Django, Flask, Fast API etc.) and Java Springboot frameworks
Project experience in AI/Machine Learning and NLP development.
TECHNICAL KNOWLEDGE AND SKILLS: Strong experience in Big Data, Cloudera Distribution 7.x, RDBMS
4-5 years of programming experience in Python, Java, Scala and SQL is must.
Strong experience building data pipelines using Hadoop components Sqoop, Hive, SOLR, MR, Impala, Spark, Spark SQL., HBase.
Strong experience with REST API development using Python frameworks (Django, Flask, Fast API etc.) and Java Springboot frameworks, Micro Services/Web service development experience using Spring framework.
Experience with Dask, Numpy, Pandas, Scikit-Learn
Proficient in Machine Learning Algorithms: Supervised Learning (Regression, Classification, SVM, Decision Trees etc.), Unsupervised Learning (Clustering) and Reinforcement Learning
Strong experience working in Real-Time analytics like Spark/Kafka/Storm
Experience with Gitlab, Jenkins, JIRA
Expertise in Unix/Linux environment in writing scripts and schedule/execute jobs.
Strong Experience with Data Science Notebooks like Jupyter, Zeppelin, RStudio. PyCharm etc.
PROFESSIONAL SKILLS:
Strong analytical skills with the ability to analyze information and identify and formulate solutions to problems.
Provide more in-depth analysis with a high-level view of goals and end deliverables.
Complete work within a reasonable time frame under the supervision of a manager or team lead.
Plan and manage all aspects of the support function.
Extensive knowledge of and proven experience with data processing systems, and methods of developing, testing and moving solutions to implementation.
Strong knowledge in project management practices and ability to document processes and procedures as needed.
Work collaboratively with other support team members and independently on assigned tasks and deliverables with minimum supervision
Communicate effectively with users at all levels, from data entry technicians up to senior management, verbally and in writing.
Self-motivated, working closely and actively communicating with team members to accomplish time critical tasks and deliverables
Ask questions and share information gained with other support team members, recording and documenting this knowledge, Elicit and gather user requirements and/or problem description information, and record this information accurately, Listen carefully and act upon user requirements
Convey and explain complex problems and solutions in an understandable language to both technical and non-technical persons
Present technical solutions to management and decision makers
Follow the lead of others on assigned projects as well as take the lead when deemed appropriate
Think creatively and critically, analyzing complex problems, weighing multiple solutions, and carefully selecting solutions appropriate to the business needs, project scope, and available resources
Take responsibility for the integrity of the solution.