Infrastructure Engineer

Aion is the enterprise AI platform, a full-stack solution for building, fine-tuning, and deploying AI at scale. A fast-growing, VC-backed startup with teams across the US, UK, and India.

• Build and deploy comprehensive monitoring for GPU infrastructure using DCGM, NVML, and custom exporters
• Deploy and manage production-scale Loki, Grafana, Tempo, and Mimir alongside Prometheus and Thanos/VictoriaMetrics
• Write Prometheus exporters in Go or Python for GPU metrics, platform services, and infrastructure components
• Build custom controllers and operators for GPU workload management, scheduling, and resource allocation
• Design and implement specialized observability for AI training workloads and inference services
• Deploy and manage SLURM clusters for HPC workloads, build observability for batch jobs, create bridges between SLURM and Kubernetes
• Write and deploy systemd services for bare-metal GPU nodes
• Manage infrastructure using ArgoCD and GitOps workflows
• Design intelligent, actionable alerting systems for GPU failures, thermal throttling, performance degradation, and workload anomalies
• Build secure observability isolation ensuring customers access only their metrics and logs
• Implement automated collection of GPU and platform telemetry; integrate with OpenTelemetry
• Build monitoring pipelines that track GPU utilization, idle time, efficiency metrics, and cost allocation per tenant
• Design systems that use observability data to detect hardware failures, trigger workload migration, and handle graceful degradation
• Build intuitive dashboards, APIs, and alerting interfaces
• Use observability systems to rapidly identify root causes during production issues
• Leverage observability data to identify bottlenecks in distributed training, inference latency issues, networking inefficiencies

• 6-10 years of experience in infrastructure engineering with strong focus on observability, monitoring systems, and production service development
• GPU Observability expertise with production experience monitoring NVIDIA GPUs using DCGM, NVML, and nvidia-smi
• Understanding GPU-specific metrics (utilization, thermal management, performance)