Principal Engineer, Observability

Firmus Technologies Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific. Founded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability. At Firmus, we are unique in our approach. We design, build, and operate a new class of digital infrastructure – the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally. Firmus AI Cloud Our large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers. It empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale. ROLE SUMMARY Firmus Technologies is seeking a Principal Engineer, Observability to own how Firmus operations teams and customers see the AI factory as one system, from site power and cooling to the GPU and the workload running on it. You will own the telemetry standard every team emits to, the platform that collects, stores, and queries it, and the applications that operations teams and customers use. Every team should be able to build its own dashboards and alerts without waiting on yours. Every new site adds millions of samples per second, and the platform must keep up as Firmus grows to hundreds of thousands of GPUs in gigawatt-scale AI factories, without losing critical data or slowing queries. KEY RESPONSIBILITIES Architecture and decisions • Own the architecture of the observability platform across every data centre and cloud region, from collection and transport to storage, query, alerting, and the applications on top. Set a target architecture two to three years ahead and plan the steps from today's system to it. • Design the platform to grow tenfold without a redesign, through sharding, retention tiers, cardinality budgets, and capacity plans. Prove the headroom with load tests before each new site goes live. • Design around the same repeatable unit as the estate. Each site keeps observing itself when its link to the rest of Firmus is down, one global view spans every site, and new sites come on through the site deployment kit with no local fork. • Own the telemetry standard: what each layer emits and how signals are named, labelled, and versioned, enforced by checks in every team's build pipeline. • Lead design reviews for significant changes across the teams, sign off architecture decision records, and settle technical disputes escalated by engineers. Decide at pace with incomplete information, write down the assumptions, and revisit a decision when the evidence changes. Build and evolve • Write production code for the hardest parts, including collection at fleet scale, the telemetry pipeline, query performance, and the observability API and applications. • Make observability self-service for every team. Teams onboard a service and build their own dashboards, alerts, and SLOs as code from templates, within the telemetry and alerting standards and their own budgets for cardinality and retention, without filing a ticket. • Set delivery guarantees for each class of telemetry. Critical operational and customer-billable signals are never lost, and any loss elsewhere is measured and recoverable. • Build and own the live topology model (digital twin) of every site, hall, rack, tray, GPU, network link, and cooling loop. Keep it in step with the inventory source of truth and map every signal to it. • Cover all layer customers depend on, from GPU and interconnect health and network fabric events to storage performance per job and Kubernetes control-plane metrics. Run passive and active health checks that signal failing GPUs, links, and nodes for draining before customers notice. • Bring facilities telemetry into the same model as compute. Operations teams see power, cooling, and leak events immediately, together with the workloads they affect. • Serve telemetry to internal consumers, including the data platform and the intelligence platform, through stable, versioned interfaces with agreed freshness and completeness, and make it usable by AI agents for diagnosis, with scoped access and an audit trail. • Deliver telemetry to customers and partner platforms on open standards such as OpenTelemetry, within seconds of collection, with ready-made dashboards, query access, alerts to their own channels, and forwarding to their own tools. Each customer sees only their own tenancy, enforced at every layer. Reliability, security, and cost • Set SLOs and error budgets for observability and use the budget to decide when reliability work comes first. • Lead the technical response to major incidents in observability and make sure every post-mortem ends with a fix that stops it happening again. • Set the alerting standard. Every alert is actionable, reaches the right responder, and links to a runbook. Prove detection works by injecting faults. • Make service-level measurement reproducible. Firmus and a customer should get the same availability from the same data. • Design the telemetry stores to upgrade in rolling waves with no gap in data, and keep dependencies and images patched within policy. • Keep Firmus' SOC 2 Type 2 and ISO 27001 controls passing at every new site. • Own cost at the design level, including retention, cardinality, and data volume. Technical leadership • Grow engineers through mentorship, design review, and code review. Hiring and performance review sit with the Engineering Manager. • Set how the team uses AI to build software effectively and securely, with review gates and automated architecture checks that keep AI-generated code within the agreed design. • Own the technical interviews and assessments for engineers and hold every candidate to that bar. Partnership and communication • Shape the observability roadmap with operations, infrastructure, facilities, and product, and have telemetry in place before each new site or product launches. • Agree ownership in writing at each boundary, including which team emits each signal, which team consumes it, and which team acts on it, and settle disagreements directly. • Work with the operations centre and the intelligence platform team to make first-level incident response and root-cause analysis faster and turn the problems they hit into product improvements. • Represent observability in customer briefings on telemetry, health, and how service levels are measured. SKILLS AND EXPERIENCE • Bachelor's degree in computer science or a related technical field, or equivalent practical experience. • 10+ years in software, infrastructure, or platform engineering, including at least 3 years at principal level or equivalent, setting the technical direction that multiple teams delivered against. You are still hands-on and intend to stay that way. • Has designed a metrics, logs, and traces platform handling millions of samples per second and run it in-house on Prometheus-compatible stores such as VictoriaMetrics, Thanos, or Mimir, with high cardinality, long retention, and a global query view across more than one site. Scaled it several times over as load grew, without a redesign. • Has run a streaming ingestion layer for telemetry (such as Kafka or Redpanda, with Vector or the OpenTelemetry Collector) and handled consumer lag, backpressure, and broker failures without losing critical data. • Has defined a telemetry schema on open standards such as OpenTelemetry or OpenMetrics that other teams relied on as a contract and given those teams self-service to build their own dashboards and alerts on it. • Has collected telemetry from hardware or networks, not only applications, through at least one of Redfish, gNMI with OpenConfig models, or DCGM. • Production Kubernetes as the runtime for observability: running and upgrading stateful, high-throughput services such as Kafka or a metrics store with no data loss. • Strong in Go or Python, with production services, APIs, or applications that other teams or customers use. • Has built security into production services: access control, secrets handling, and patching, under SOC 2 Type 2, ISO 27001, or an equivalent control set. • Has set an alerting standard across teams that cut noise and measurably reduced time to detect and time to restore, backed by SLOs and incident reviews that changed the system. • Clear written and verbal communication in English, able to explain problems, decisions, and trade-offs to engineers, executives, and customers. • Willing to join the on-call rotation for the services you own and to travel overseas occasionally. Bonus Points • Background at an AI cloud or high-performance compute provider, ideally with GPU health checks, burn-in testing, or failure prediction across a fleet. • Integrated facilities or operational technology telemetry, such as OPC UA, Modbus, or data from SCADA and building management systems. • Built a digital twin or live topology model of a physical estate from an inventory source of truth. • Delivered telemetry to external customers with tenant isolation and access controls. • Served telemetry to a data platform or to analytics and machine-learning pipelines used for incident response or root-cause analysis. • Wrote or extended Kubernetes operators for stateful services. • Led a team's use of AI-assisted and agentic engineering, measured by delivery outcomes and token cost, and kept it secure with sandboxed agents, least-privilege access, and review gates. • Certifications such as CKA or CKAD, Prometheus or OpenTelemetry Certified Associate, or Confluent Certified Administrator for Apache Kafka. Location & Reporting • Location: Singapore • Hybrid: 4 days in office, 1 day work from home • Report to: Head of Platform Engineering Employment Basis Full-time Diversity At Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions. Join us in our mission to revolutionize the AI industry through sustainable practices and cutting-edge engineering. Apply now to be part of shaping the future of sustainable AI infrastructure.