Senior Data bricks Architect Security Data Platform

Job Overview

We are seeking an experienced Senior Data bricks Architect with a strong background in Pharmaceutical / Life Sciences environments to lead the architecture and technical direction of a large-scale Security Data Platform.

Key Responsibilities

• Lead the architecture and technical strategy for a large-scale Data bricks Security Data Platform.
• Design and optimize Data bricks solutions capable of processing approximately 300 TB of data per day.
• Architect highly scalable, secure, and governed data platforms supporting approximately 13,000 tables and 3,000+ users.
• Develop architecture standards for security data ingestion, processing, storage, analytics, and consumption.
• Work closely with CISO, EVP, SVP, and VP-level Security executives to understand business and security objectives and translate them into technical solutions.
• Partner with engineering teams to implement and operationalize the target architecture.
• Design secure data architectures incorporating RBAC, data governance, access controls, auditing, lineage, encryption, and compliance.
• Establish best practices around Databricks security, governance, performance, scalability, and cost optimization.
• Work with Observability teams to develop solutions for monitoring data pipelines, platform health, security events, performance, and operational issues.
• Provide technical leadership across data engineering, security engineering, cloud, platform, and architecture teams.
• Perform architecture reviews, technical design reviews, troubleshooting, and code reviews.
• Identify performance bottlenecks and optimize large-scale Spark/Databricks workloads.
• Define strategies for data lifecycle management, retention, access management, and secure data sharing.
• Ensure the platform meets enterprise security, regulatory, privacy, and compliance requirements applicable to the pharmaceutical industry.
• Communicate complex technical concepts effectively to both executive stakeholders and engineering teams.

Required Qualifications

• 8+ years of experience in Data Engineering, Data Architecture, or Cloud Data Platforms.
• Strong hands-on experience with Databricks and Apache Spark.
• Proven experience architecting very large-scale Data bricks environments.
• Experience handling 300 TB+ of data per day or comparable large-scale data volumes.
• Experience managing or architecting environments with thousands of users and thousands of tables.
• Strong knowledge of Delta Lake, Lakehouse Architecture, Medallion Architecture, Spark, ETL/ELT, and distributed data processing.
• Strong experience with Data Security and Security Data Platforms.
• Experience implementing enterprise RBAC, access controls, auditing, lineage, governance, and secure data sharing.
• Strong understanding of Databricks Unity Catalog and enterprise data governance.
• Experience working with security, observability, compliance, or cybersecurity data.
• Understanding of security monitoring and observability platforms and their integration with data platforms.
• Experience working with highly sensitive or regulated enterprise data.

Executive & Stakeholder Management

• Experience working directly with CISO, CIO, CTO, EVP, SVP, and VP-level executives.
• Ability to communicate architecture strategy, security risks, platform capabilities, and business value to executive stakeholders.
• Strong experience gathering requirements from senior leadership and translating them into actionable technical roadmaps.
• Ability to work across executive sponsors, security teams, engineering teams, and enterprise architecture organizations.

Observability Experience

• Experience designing or supporting data/platform observability solutions.
• Knowledge of monitoring data pipelines, Spark jobs, Databricks workloads, infrastructure, data quality, and platform health.
• Experience integrating observability and monitoring capabilities into enterprise data platforms.
• Ability to establish operational metrics, alerting, logging, and performance monitoring.

Preferred Qualifications

• Databricks certifications.
• Experience with Unity Catalog, Delta Live Tables / Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming, Workflows, MLflow, and Databricks SQL.
• Experience with Terraform / Infrastructure as Code.
• Experience with cloud security and identity platforms.
• Experience with SIEM, security analytics, or cybersecurity data platforms.
• Knowledge of HIPAA, GxP, FDA, SOX, GDPR, or other relevant regulatory/compliance frameworks.
• Experience supporting global pharmaceutical enterprises.
• Experience leading architecture initiatives across multiple engineering teams.

Must-Have Skills

• Data bricks
• Apache Spark
• Data Architecture
• Security Data
• 300+ TB/Day Data Processing
• Unity Catalog
• Data Governance
• RBAC
• Data Security
• Observability
• Large-Scale Data Platforms
• Pharmaceutical / Life Sciences Experience
• Executive Stakeholder Management

Industry Requirement: Pharmaceutical / Life Sciences experience is mandatory.