Iceberg DBA / Lakehouse Operations Engineer_remote in dallas,TX

Job Summary

We are seeking a highly skilled Iceberg DBA / Lakehouse Operations Engineer to manage the reliability, performance, and operational integrity of the Apache Iceberg data layer supporting enterprise analytics and business-critical applications.

The role involves managing a large-scale, multi-engine Lakehouse environment supporting workloads across Spark, Hive, and Impala. The engineer will play a key role in enterprise data modernization initiatives, particularly migrations from Hive and Teradata to Apache Iceberg.

The ideal candidate will have strong hands-on experience with Iceberg table operations, metadata management, data modeling, and query performance optimization in cloud-based environments.

This role is critical to maintaining data accuracy, availability, and performance, as any degradation can directly impact downstream reporting, analytics, and business-critical decision-making.

Key Responsibilities

• Iceberg Data Layer Ownership & Operations
• Own day-to-day operations and administration of Apache Iceberg tables supporting multiple enterprise applications.
• Ensure data reliability, consistency, and availability across Lakehouse workloads.
• Maintain operational integrity of datasets at multi-terabyte to petabyte scale.
• Monitor table health, metadata growth, and storage utilization.
• Ensure consistent data access and operational stability across environments.
• Advanced Table Management & Optimization
• Perform advanced Iceberg table maintenance and optimization activities, including:
• Minor and major compaction.
• Small file mitigation and optimization.
• Snapshot expiration and metadata compaction.
• Orphan file cleanup and storage management.
• Optimize data layout and query performance through file sizing, distribution, and partitioning strategies.
• Implement partition evolution and partition pruning techniques.
• Apply clustering and data ordering strategies, such as Z-ordering or equivalent approaches.
• Monitor and optimize table maintenance operations to improve query performance and storage efficiency.
• Data Modeling & Lakehouse Architecture
• Support and enforce data modeling best practices, including:
• Third Normal Form (3NF) for source-aligned datasets.
• Medallion architecture (Bronze, Silver, and Gold) for curated data pipelines.
• Ensure Iceberg table designs align with ingestion patterns, downstream consumption, and performance requirements.
• Balance data integrity, normalization, and analytical query performance.
• Collaborate with data engineering teams to ensure consistent implementation of layered data architecture across applications.
• Multi-Engine Query Performance & Consistency
• Ensure consistent and optimized query performance across:
• Apache Spark / Spark SQL (CDE).
• Hive.
• Impala (CDW).
• Troubleshoot query performance bottlenecks, metadata inconsistencies, and inefficient execution plans.
• Analyze query execution plans, scan patterns, and data access behavior.
• Ensure consistent table metadata and query results across multiple processing engines.
• Hive & Teradata Modernization Support
• Support enterprise data modernization initiatives involving Hive-to-Iceberg and Teradata-to-Iceberg migrations.
• Assist with schema alignment, data type mapping, and table structure conversion.
• Perform data validation, reconciliation, and accuracy checks between source and target datasets.
• Troubleshoot migration-related issues and ensure post-migration stability and performance.
• Collaborate with data engineering teams to optimize migrated workloads.
• Metadata & Data Lifecycle Management
• Manage Iceberg metadata to ensure scalability, consistency, and performance.
• Maintain consistent table states and metadata across multiple query engines.
• Implement data retention and archival policies.
• Manage snapshot lifecycle operations, expiration, and cleanup.
• Optimize time-travel capabilities and metadata maintenance.
• Monitor metadata growth and prevent performance degradation.
• Production Support & Incident Management
• Provide L2/L3 operational support for production issues across Iceberg-based Lakehouse workloads.
• Participate in on-call rotations to support critical data platforms.
• Respond to and resolve P1/P2 incidents within defined SLAs.
• Troubleshoot data inconsistencies, reporting discrepancies, query failures, and performance degradation.
• Perform Root Cause Analysis (RCA) and implement permanent corrective and preventive actions.
• Collaborate with platform, infrastructure, and application teams during incident triage and resolution.
• Security & Data Governance
• Support fine-grained data access controls using Apache Ranger policies and Role-Based Access Control (RBAC).
• Ensure data validation, reconciliation, and accuracy between source systems and Iceberg datasets.
• Maintain secure and compliant data access across applications and processing environments.
• Collaborate with security and governance teams to enforce enterprise data access policies.

Required Skills & Qualifications

• Strong hands-on experience with Apache Iceberg and/or Hive-based data lake environments.
• Strong understanding of Iceberg table architecture, metadata management, and table maintenance operations.
• Experience with table-level optimization, compaction, snapshot management, and performance tuning.
• Understanding of data modeling concepts, including normalization and 3NF.
• Knowledge of modern Lakehouse architecture and Medallion architecture (Bronze/Silver/Gold).
• Hands-on experience with Spark SQL, Hive, and Impala.
• Working knowledge of NiFi and Trino.
• Strong understanding of partitioning strategies, partition evolution, and partition pruning.
• Experience with columnar file formats such as Parquet and ORC.
• Understanding of distributed query processing and large-scale data management.
• Experience supporting multi-terabyte to petabyte-scale data platforms.

Preferred Skills

• Experience with Hive-to-Iceberg and/or Teradata-to-Iceberg migrations.
• Experience with Cloudera CDP, particularly CDE and CDW.
• Familiarity with cloud platforms such as AWS and Azure.
• Experience with Apache Ranger, data governance, and access management.
• Scripting and automation experience using Python and Shell.
• Experience supporting enterprise-scale Lakehouse modernization initiatives.