Lead Observability & Data Architect/Engineer (SC/DV Cleared)
Posted 1 day ago by GIOS Technology Limited
Negotiable
Undetermined
Hybrid
Hybrid - London/Reading/Birmingham/Manchester/Glasgow/Liverpool/Newcastle
Summary: The role of Lead Observability & Data Architect/Engineer involves designing and scaling national-critical monitoring fabrics and data hubs, focusing on transforming vast amounts of raw event logs into actionable telemetry. Candidates should have a strong background in observability platforms or data engineering, with expertise in building high-performance data pipelines and managing large-scale deployments. The position requires an active UK security clearance (SC/DV) and experience in regulated sectors. The role is hybrid, allowing work from multiple locations including London, Reading, Birmingham, Manchester, and Glasgow.
Key Responsibilities:
- Design and scale national-critical monitoring fabrics and data hubs.
- Build high-performance data pipelines to ingest, parse, and structure massive event telemetry.
- Manage large-scale cluster deployments, optimizing indexing, data life cycle policies, and schema mapping.
- Develop visualization dashboards and manage telemetry collectors.
- Design and scale data pipelines for high-velocity telemetry.
- Architect scalable, containerized solutions using Kubernetes and infrastructure-as-code.
- Utilize search and indexing engines effectively.
- Automate log parsing, testing, and deployment workflows.
- Work within highly regulated sectors with an active UK security clearance.
Key Skills:
- Expertise in Elasticsearch/OpenSearch, Splunk, or Solr.
- Strong proficiency in Python, Rust, or Bash.
- Experience with Apache Kafka, Logstash, or streaming/orchestration frameworks.
- Hands-on experience with Kubernetes, Docker, and infrastructure-as-code tools.
- Knowledge of building visualization dashboards (Kibana, Grafana).
- Experience in regulated sectors (MoD, Home Office, BAE, HMRC).
- Active UK security clearance (SC/DV).
Salary (Rate): undetermined
City: undetermined
Country: undetermined
Working Arrangements: hybrid
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
We are hiring for Lead Observability & Data Architect/Engineer (SC/DV Cleared)
Location: Hybrid - London/Reading/Birmingham/Manchester/Glasgow
- We are seeking an Enterprise Platform Architect or Senior Data Engineer to design and scale national-critical monitoring fabrics and data hubs. Whether your background is rooted in core Observability platforms (Opensearch/Elastic/Splunk/OpenTelemetry) or high-scale Data Engineering (Kafka, Python, Airflow, OpenSearch/Lucene), your core mission is the same: transforming billions of raw event logs into actionable, secure streaming telemetry.
- Should be able to build high-performance data pipelines to ingest, parse, and structure massive event telemetry across hybrid-cloud and air-gapped environments.
- Manage large-scale cluster deployments, designing optimized indexing, data life cycle policies, and schema mapping (such as ECS).
- Expertise in Elasticsearch/OpenSearch, Splunk, or Solr. Strong proficiency in building visualization dashboards (Kibana, Grafana) and managing telemetry collectors (OpenTelemetry/OTel, Beats, Prometheus).
- Experience in designing and scaling data pipelines for high-velocity telemetry. Essential expertise in Apache Kafka, Logstash, or streaming/orchestration frameworks (Airflow, Vector, Debezium).
- Hands-on experience architecting scalable, containerized solutions using Kubernetes (K8s), Docker, and infrastructure-as-code (Terraform/Ansible).
- Search & Indexing Engine Depth: Prior architecture experience with OpenSearch, Apache Solr, or Lucene core or Elasticsearch.
- Strong proficiency in Python, Rust, or Bash for automating log parsing, testing, and deployment workflows.
- A strong track record of engineering within highly regulated sectors (eg, MoD, Home Office, BAE, HMRC) with an active UK security clearance (SC/DV).
We highly encourage applications if you bring any of the following equivalent expertise:
- From Big Data/ETL to Telemetry Pipelines: If you have built high-throughput streaming pipelines using Apache Kafka, Airflow, NiFi, or Spark, you already know how to route and transform complex data feeds at scale.
- From Enterprise Search to System Observability: If you have engineered core search, indexing, or sharding architectures using Apache Solr, Lucene, or OpenSearch, you possess the exact foundational knowledge needed to scale an enterprise Elasticsearch cluster.
- From Cloud Infrastructure to Monitoring Fabrics: If you are a seasoned Cloud or Infrastructure Architect specializing in secure Kubernetes orchestration, Terraform automation, and air-gapped system design, you can easily pivot to defining secure logging architectures.