Hardware Automation
In-Band Hardware Lifecycle Automation for Real Infrastructure
A practical operating model for taking ownership of cloud, NeoCloud, or hosted bare-metal Linux systems when you have root on the disk but no Redfish, iDRAC, iLO, or BMC access.
Context
Physical AI expands the infrastructure problem beyond standard servers. Linux-type operating environments will run across robots, edge boxes, accelerators, gateways, test rigs, sensors, industrial systems, and hardware variants with different chipsets, firmware paths, drivers, and boot assumptions. All of that hardware still has to be imaged, owned, validated, updated, and orchestrated reliably.
The common problem is not greenfield provisioning. It is receiving a fleet that already exists in an unknown or unwanted state, or being handed pre-imaged cloud-provider or NeoCloud hardware that the team must take ownership of even though access stops at Linux root. There may be no Redfish API, Dell iDRAC, HPE iLO, IPMI, serial console, PXE control, or BMC of any kind available to you.
Hardware platforms are usually managed from two directions. Out-of-band controllers are essential for power, firmware, serial console, and remote recovery, but they are not the whole operating picture. The running host can see kernel state, PCIe behavior, drivers, filesystems, workload symptoms, local logs, performance counters, package state, and the exact configuration that applications inherit.
The agent is not a kubelet replacement. It complements kubelet by filling the missing bare-metal substrate before a tenant node joins a cluster or a VM runtime takes over. The public pattern is simple: establish enough hardware, package, identity, and recovery evidence that higher-level orchestration can trust the host.
In-band hardware lifecycle automation uses that host perspective deliberately. A small stateful agent or operator workflow can collect inventory, preserve important host configuration, validate assumptions, prepare bounded changes, and report what changed. The exact command inventory, recovery mechanics, credential model, and rollout details belong in private implementation material, not in a public overview.
01
Root Access, No BMC
Physical AI increases hardware diversity. Teams will need to operate Linux-like systems across many device classes and chipsets, often outside a clean datacenter model, while still proving that the right image, drivers, firmware, services, and validation profile are present.
This page is for operators who do not fully control the hardware they are given. A cloud provider, hosting company, or NeoCloud may hand you a running Linux install with root credentials, but not the out-of-band management plane that normally makes bare-metal lifecycle work comfortable.
You may not have Redfish, Dell iDRAC, HPE iLO, IPMI, PXE, serial console, firmware controls, or a BMC interface. You still need to discover what is actually present, decide what can safely change from inside the OS, and move the machine toward a known-good operating profile using only in-band access.
Or you are provided pre-imaged hardware that must become yours operationally, but the access model does not allow normal hardware administration. You still need inventory, validation, drift detection, evidence capture, and safe remediation paths before the platform can be trusted.
02
What Root Can Still See
Operators can ask a running machine what it is, what it sees, how it is configured, and whether it matches the expected profile. That includes CPU, memory, PCIe, GPUs, NICs, storage, kernel modules, firmware versions, network state, mount state, services, logs, and workload readiness.
For AI and hardware infrastructure, this matters because the expensive failures are often cross-layer. A GPU may be present but degraded by topology, driver, cooling, firmware, NUMA locality, or scheduler placement. In-band automation can collect the evidence needed to separate those causes before escalation.
Stateful agents also make distributed state management practical. A host can continue collecting facts, preserving task state, and staging approved work while isolated, then reconcile evidence and task status once the network path returns.
Network state is part of that record. The public point is not a command recipe; it is that connectivity-sensitive intent has to be preserved before lifecycle automation changes a host.
A bare-metal agent API enables automation directly against the machine before Kubernetes, virtualization, or workload schedulers own it. Publicly, that means validation, evidence capture, package staging, and controlled handoff. The destructive recovery path stays behind the private gate.
03
Where It Fits Without Out-of-Band Control
This pattern complements out-of-band management when you have it, but it is most useful when you do not. Out-of-band remains the recovery and power-control layer for teams that own it. In-band automation is the inspection, validation, configuration, and evidence layer available when the only reliable control surface is the running OS.
It also complements kubelet rather than replacing it. Kubelet is the Kubernetes node agent once the node is admitted into a cluster. A bare-metal lifecycle agent operates earlier and lower: while the machine is just Linux, while packages and drivers are being prepared, while VM substrates are being created, or while hardware needs to be recovered before it can safely join anything.
The strongest implementations treat hardware lifecycle as a stateful workflow: discover, compare against intent, validate, apply bounded changes, record evidence, reconcile after disconnection, and expose the result to operators or higher-level orchestration.
Deterministic tooling is useful here without requiring a specific host distribution. The agent should be able to bring repeatable diagnostics and support tools to ordinary Linux systems without turning the base OS into a dependency trap.
The same pattern can extend beyond conventional servers. Future support can cover embedded Linux and SoC hardware, and eventually Apple Silicon-style systems where boot, firmware, device trees, accelerators, and platform-specific recovery paths need the same evidence-first lifecycle treatment.
Architecture Patterns
Public Diagrams for the Operating Model
Inventory Intake to Lifecycle Truth
A safe control plane treats spreadsheets, YAML, and existing SSH inventories as intake adapters, not as the long-term source of truth.
Import
Provider handoff data arrives as provisional inventory.
Normalize
Hosts, ownership, platform class, and operating intent are mapped into one model.
Observe
Reachable Linux systems report enough host facts to establish a current-state baseline.
Compare
Observed state is compared with expected profiles and lifecycle policy.
Operate
Tasks, events, logs, and state transitions become durable operating evidence for review.
Control Surface Selection
The same lifecycle intent can use different control surfaces while keeping the operation visible.
Local Agent
Preferred path when the host can run a bounded stateful agent.
Nearby Worker
A worker close to the environment can assist when the host is not ready for a resident agent.
Break Glass
A narrow fallback exists for bootstrap and recovery situations.
Task Record
Each path records state, checkpoints, and failure context for review.
Recovery Readiness Envelope
Publicly, recovery is described as a gated readiness envelope. Concrete execution mechanics stay private.
Prepare
Collect prerequisites and preserve host-specific intent before any risky action.
Verify
Require explicit readiness checks and operator-visible approval gates.
Handoff
Move from observation to a controlled lifecycle action only when the host is inside policy.
Execute
Run bounded work inside an approved window with progress evidence.
Review
Inspect lifecycle state and recovery notes before widening rollout.
Evidence and Observability Split
Lifecycle truth and runtime telemetry are different systems. Good automation preserves both without making dashboards the source of truth.
Control State
Inventory, desired state, lifecycle state, task records, events, and audit logs persist in the product system.
Telemetry
Traces, metrics, and runtime logs flow through OpenTelemetry, collectors, log shippers, and operational dashboards.
Correlation
Host identity, operation phases, and lifecycle status join product state to runtime debugging evidence.
Decision
Operators use durable state for workflow decisions and telemetry for diagnosis, performance, and alerting.
Agent Evidence
Hardware Report as Public Evidence
Hardware Report is the open-source evidence collector that can support this operating model without exposing private control-plane mechanics. It turns reachable host facts into structured reports that can be compared, archived, or reviewed.
That gives the public page a safe read-side example: collect facts, compare observations against intent, and feed acceptance policy with source-backed hardware evidence. Mutation paths, recovery mechanics, credential flow, and rollout design stay private.
The same pattern is server-first today, but the principle generalizes to other Linux-like and hardware-adjacent systems where platform evidence needs to be captured before orchestration can safely proceed.
View the Hardware Report pageHardware Report Hexagonal Architecture
The crate keeps the hardware reporting domain separate from CLI, OS adapters, command execution, file output, and HTTP publishing.
CLI / Library
The hardware_report binary and create_service API call the primary reporting port.
Primary Port
HardwareReportingService exposes generate_report, publish_report, dependency validation, and privilege checks.
Domain Service
HardwareCollectionService coordinates collection, summary generation, and publishing without knowing OS details.
Entities
HardwareReport, SystemSummary, CpuInfo, MemoryInfo, StorageInfo, GpuInfo, NetworkInfo, BIOS, chassis, and motherboard data form the schema.
Secondary Ports
SystemInfoProvider, CommandExecutor, ConfigurationProvider, DataPublisher, and FileRepository isolate external dependencies.
Adapters
Linux/macOS providers, Unix commands, HTTP publisher, and file repository implement the external side.
Collection and Publication Flow
A run turns host-local discovery into portable evidence for files, APIs, CMDBs, acceptance gates, or agent sync.
Validate
Check dependencies and privileges before relying on privileged host discovery paths.
Collect Hardware
CPU, memory, storage, GPU, and network data are collected concurrently.
Collect Metadata
System identity, BIOS, chassis, motherboard, NUMA topology, and filesystems are gathered.
Summarize
CPU topology, memory config, storage totals, GPU count, NIC count, and machine identity become SystemSummary.
Serialize
The HardwareReport aggregate is emitted as TOML or JSON for review and automation.
Publish
Optional HTTP publishing sends JSON with labels, bearer auth, and an optional system identifier.
What to Understand
- The target audience is teams operating cloud-provider, hosted bare-metal, or NeoCloud machines where hardware ownership is incomplete: you may have Linux root access, but no Redfish, iDRAC, iLO, IPMI, PXE, serial console, or BMC control plane.
- The running OS has evidence that BMCs and inventory databases cannot fully see: kernel state, drivers, topology, logs, mounted storage, performance counters, and workload symptoms.
- In-band automation is most valuable when it is bounded: inspect, validate, collect evidence, apply safe changes, and report state rather than hiding risky mutation behind a button.
- The agent is not there to replace kubelet. It is the missing bare-metal layer that prepares and validates Linux machines before kubelet, QEMU, Firecracker, Cloud Hypervisor, or another higher-level runtime takes over.
- Stateful agents are critical for distributed fleets: they can preserve local lifecycle state while offline, continue safe observation, and reconcile queued evidence when connectivity returns.
- Agent identity and transport need bounded trust, but the public page intentionally avoids publishing credential flows or control-plane mechanics.
- Deterministic tooling lets the agent run repeatable diagnostics and support utilities on ordinary Linux systems without relying on the host's global package state.
- A bare-metal agent API makes hardware automation programmable before cluster admission or VM placement, but destructive recovery details belong in private implementation material.
- Network state is first-class evidence. Publicly, the important point is preserving connectivity intent before lifecycle changes, not publishing a command recipe.
- Compared with one-shot configuration tools, the agent provides a stateful reliability layer: local reconciliation, durable checkpoints, offline operation, declarative intent, and sync back to the control plane.
- The model is aimed at servers first, but the same approach can extend to embedded Linux, SoC devices, and Apple Silicon-style systems where platform-specific boot and accelerator state need lifecycle evidence.
- Hardware lifecycle workflows should connect expected profiles to observed state so operators can see drift across firmware, BIOS settings, NICs, GPUs, storage, kernel modules, and services.
- Physical AI makes chipset and device diversity a lifecycle concern: imaging, driver validation, firmware evidence, peripheral discovery, and recovery workflows have to work across more than one golden server shape.
- The pattern works best with deterministic tooling, clear privilege boundaries, signed or reviewed actions, and durable audit trails.
Common Failure Modes
- The agent becomes a privileged mystery box with broad mutation rights and weak audit trails.
- Automation collects inventory but does not compare it to expected profiles, so operators still have to interpret every host by hand.
- Out-of-band and in-band views disagree, but the workflow does not show which source is authoritative for each lifecycle action.
- Recovery commands are mixed with diagnostics without guardrails, approvals, staged execution, or rollback notes.
- The implementation ignores degraded states where the OS is reachable but networking, drivers, storage, or accelerators are partially broken.
What Good Looks Like
- A host can report its hardware, firmware, drivers, topology, services, and workload readiness in a form operators and automation can compare against intent.
- Agents maintain local state, checkpoints, and evidence queues so disconnected sites can keep operating safely and converge back to the control plane after reconnect.
- Agent communication uses authenticated mTLS identity, commonly SPIFFE/SPIRE, so callbacks, task claims, and sync replay can be tied back to a specific workload identity.
- Kubelet, VM runtimes, and schedulers receive a cleaner substrate: hardware has been inspected, packages are reproducible, accelerator dependencies are staged, and lifecycle evidence exists before the node or VM is trusted.
- Nix-backed package isolation allows ML, GPU, storage, firmware, and validation toolchains to coexist on the same Linux host without contaminating the base system or each other.
- The Baremetal Agent API exposes safe operations against physical hardware that were previously manual or vendor-specific, including in-band re-imaging across heterogeneous Linux machines.
- Network state survives lifecycle work because the agent records netplan, interface state, routes, DNS, and related tool-call evidence before making or staging changes.
- Declarative desired state is paired with local state, retries, and sync semantics, so the workflow is more reliable than fire-and-forget host automation.
- Future embedded SoC and Apple Silicon support would use the same pattern: collect platform-specific evidence, preserve state, and hand off to the correct runtime or recovery path.
- Diagnostics are safe by default, mutating actions are explicit, and every lifecycle operation leaves durable evidence for review.
- Operators can move from symptom to likely failure domain quickly: hardware, firmware, kernel, driver, network, storage, scheduler, or workload.
- The workflow complements BMC and vendor tooling while giving product, support, and platform teams a clearer view of what the machine is actually doing.
Further Resources
Private Resources
Private implementation details, command inventories, recovery workflows, credential flows, and source-backed agent designs stay in the protected area.
View Private Resources