Production Engineer - HPC Reliability & Automation

Key Responsibilities

• Develop deep technical understanding of large-scale HPC and cloud infrastructure environments, including compute clusters, workload schedulers, accelerator platforms, storage systems, networking, and authentication services.
• Proactively identify reliability, scalability, and performance issues by correlating logs, metrics, telemetry, and events across multiple cloud and on-premises environments.
• Troubleshoot complex production issues across Linux systems, containers, distributed applications, schedulers, storage, networking, and compute infrastructure.
• Design and build automation solutions that improve detection, diagnosis, remediation, and operational efficiency while ensuring safety, auditability, and reliability.
• Enhance observability, monitoring, alerting, and operational intelligence capabilities to reduce downtime and improve platform health.
• Collaborate with infrastructure, cloud, security, and engineering teams to implement long-term solutions and improve service resiliency.
• Participate in incident management, root cause analysis, postmortems, and continuous improvement initiatives.
• Develop tooling, scripts, and platform automation to minimize manual operational activities.

Required Skills & Experience

• Strong experience supporting production HPC environments, large-scale distributed compute platforms, GPU-accelerated infrastructure, or reliability-focused engineering environments.
• Deep Linux administration, troubleshooting, and performance engineering experience.
• Strong understanding of networking fundamentals, DNS, authentication, storage I/O, containers, and distributed system operations.
• Hands-on experience with observability and monitoring platforms such as: Prometheus, Grafana, OpenTelemetry, ELK Stack, Datadog, Splunk.
• Strong programming and automation experience using: Python, Go (preferred), Shell/Bash scripting.
• Experience conducting complex production investigations from issue identification through root cause analysis, resolution, validation, and long-term prevention.
• Experience supporting highly available production platforms with operational ownership responsibilities.

Preferred Qualifications

• Experience with workload orchestration and scheduling platforms such as: Kubernetes, Slurm, OpenShift, Similar cluster management solutions.
• Experience supporting AI, machine learning, data science, or GPU-intensive computing environments.
• Understanding of distributed training infrastructure, checkpoint management, job scheduling, workload placement, and compute cluster operations.
• Experience working with cloud platforms such as: AWS, Microsoft Azure, Google Cloud Platform (Google Cloud Platform), Similar large-scale cloud providers.
• Experience with enterprise-scale storage technologies, including: NFS, POSIX-compliant storage, Lustre, Object Storage, High-performance storage systems.
• Experience with Infrastructure as Code and platform automation tools: Terraform, Helm, Kubernetes Operators, Ansible.
• Experience designing and validating resilient infrastructure through testing, fault simulation, workload replay, and controlled recovery procedures.
• Exposure to AI-assisted operations, intelligent automation, or autonomous remediation workflows is beneficial.