Summary: The role of Security Data Engineer involves designing, developing, and implementing data solutions on the Azure cloud platform, specifically focusing on Azure Data Factory, Databricks, and Synapse Analytics. The engineer will collaborate with various teams to ensure data needs are met while optimizing data pipelines and storage solutions. This position requires a strong background in data engineering within a banking or financial services context, with an emphasis on security and data governance. The contract is for six months and is classified as inside IR35.

Key Responsibilities:

• Support the Design and Implementation of Data Architectures: Design and build scalable data pipelines using Azure Data Factory and DataBricks to ingest, transform, and load data from various sources. Develop and optimize data and pipeline architectures within the Azure cloud environment.
• Data Processing and Transformation: Develop and maintain data integration processes to ensure seamless data flow across systems, leveraging Databricks and Synapse Analytics.
• Data Storage Management: Select and manage appropriate data storage solutions like Azure SQL Database, Azure Data Lake, or Azure Blob Storage.
• Data Warehousing: Be able to guide and support the building of scalable data warehousing solutions utilizing Azure Synapse Analytics for handling large datasets.
• Data Modelling and Analysis: Collaborate with data modellers and business analysts to design and implement efficient data models that support business requirements.
• Real Time Data Processing: Configure and optimize streaming pipelines with Azure Stream Analytics for extracting insights from live data streams.
• Data Governance and Security: Ensure data governance practices, implement security measures, and monitor data quality within Azure.
• DevOps and Automation: Automate data pipelines, monitor processes, and manage deployments using tools like Azure DevOps for continuous integration and delivery.
• Performance Optimization: Optimize data processing performance by fine-tuning data pipelines, queries, and storage solutions.
• Data Quality: Implement data quality checks and monitoring processes to ensure data accuracy, consistency, and reliability.
• Collaboration: Work closely with cross-functional teams, including data scientists, software engineers, and product managers, and Senior stakeholders to deliver end-to-end data solutions.
• Documentation: Create and maintain comprehensive documentation of data engineering processes, architectures, and workflows.
• Troubleshooting: Diagnose and resolve data-related issues, ensuring minimal disruption to business operations.
• Continuous Improvement: Stay current with emerging technologies and best practices in data engineering, cloud computing, and the banking industry.

Key Skills:

• Proficiency in Azure Data Factory, Databricks, and Synapse.
• Hands-on experience with Python, SQL, and Spark for data processing.
• Strong analytical skills for working with unstructured datasets.
• Ability to design and implement well-written code.
• Experience in automating tasks and deploying production-standard code.
• Excellent communication skills and the ability to collaborate effectively within teams.
• Familiarity with visualization tools and building insightful dashboards.
• Effective communication and collaboration skills with the ability to work in a team-oriented environment.
• Certification in Azure or related technologies (eg, Microsoft Certified: Azure Data Engineer Associate).
• Excellent organisational, time management skills.
• Know of security tools and risk frameworks.
• Knowledge of relevant technology standards, roadmaps and best practices.
• Knowledge of leading industry standards for retail banking channels and products.
• Knowledge of change management principles.
• Knowledge of Data Management principles.
• Experience with Agile development methodologies.

Salary (Rate): £431

City: Edinburgh

Country: UK

Working Arrangements: Hybrid

IR35 Status: Inside IR35

Seniority Level: Mid-Level

Industry: IT