Big Data Developer
Posted 3 days ago by Intuition IT Solutions Ltd on JobServe
Negotiable
Inside
Hybrid
London, UK
Summary: The role of Big Data Developer requires senior experience in Data Engineering, specifically in building automated data pipelines using IBM Datastage, DB2, AWS, and Databricks. The position emphasizes the delivery of complex data pipelines and the management of scalable data infrastructures. The candidate will lead a team, ensuring best practices in data governance and collaboration across various business functions. This hybrid role is based in London.
Key Responsibilities:
- Design, develop, and maintain scalable data infrastructures.
- Manage and scale Databricks environments for ETL, data science, and analytics.
- Utilize AWS services for data processing and storage.
- Lead Data Engineering teams and mentor data engineers.
- Implement data governance, security, and quality best practices.
- Drive automation of data pipelines and workflows.
- Collaborate with business stakeholders for seamless data integration.
- Establish monitoring solutions for data pipeline performance.
- Architect and manage data platforms for real-time and batch processing.
- Integrate and scale cloud data services on AWS.
Key Skills:
- Expertise in Databricks and AWS services (S3, Glue, Lambda, RDS, IAM).
- Proficiency in IBM technologies (DB2, Datastage, Tivoli Workload Scheduler).
- Strong programming skills in Python and SQL.
- Experience with modern ETL frameworks and data warehousing techniques.
- Familiarity with DevOps practices and CI/CD pipelines.
- Knowledge of big data technologies (Apache Spark, Hadoop).
- Test automation skills.
- Leadership and team management capabilities.
- Understanding of data governance and compliance standards.
- Ability to optimize cloud data services and costs.
Salary (Rate): 400
City: London
Country: UK
Working Arrangements: Hybrid
IR35 Status: Inside IR35
Seniority Level: Senior
Industry: IT