The machine that runs the most services should not automatically become the machine I am most afraid to rebuild.
Every homelab eventually develops a workhorse.
Mine became the application layer: Linux, containers, media services, monitoring, management tools, game workloads, and the experiments that are useful enough to keep around. It has enough compute to carry a lot of responsibility, which creates an obvious temptation to let it become responsible for everything.
I went the other direction.
The more important the application host became, the more I wanted its role to stay narrow enough that I could replace it.
That meant separating application compute from authoritative bulk storage, documenting the dependencies between them, and treating persistent application state differently from data that merely passes through the machine.
The result is not high availability. It is something less glamorous and, for my lab, more useful: a failure boundary I can actually reason about.
I designed a role, not a favorite server
The application host has a clear job:
- Run containerized applications
- Provide compute for media and game workloads
- Use shared data provided by the storage layer
- Keep application experiments away from authoritative bulk storage
- Remain manageable without depending on a local desktop session
- Be rebuildable from documented configuration and protected state
That last point changes the design.
If rebuilding the host would require me to remember months of shell commands, rediscover which directories matter, or hope an application database happens to survive, then the machine is not really replaceable. It is just a pet server with a theoretical recovery plan.
I do not need every component to be stateless. I need to know exactly where the state lives and which parts belong to the application host.
That distinction is the center of the architecture.
Persistent state is not one category
A containerized service can touch several very different kinds of data:
- Application configuration — settings required to recreate the service as I operate it.
- Mutable application state — databases, indexes, user state, and metadata the application owns.
- Authoritative user data — files whose value exists independently of the application presenting them.
- Rebuildable data — caches, thumbnails, transcodes, temporary files, and derived indexes.
- Secrets — credentials, keys, tokens, and other values that require their own handling and access controls.
Treating all five as “the volume” makes recovery harder than it needs to be.
Docker volumes persist outside the lifecycle of an individual container, while bind mounts map a host path into a container. Docker’s current documentation also notes that bind mounts depend on the host filesystem layout and are writable by default unless configured read-only. (Volumes, bind mounts)
Those are implementation tools, not a backup strategy.
For each service, I still need to answer:
- Which state must survive a rebuild?
- Which state can be regenerated?
- Which data belongs somewhere other than this host?
- Which items have to be restored together to remain consistent?
- Which permissions and identities must exist before the service starts?
- Which secrets must be restored without embedding them in general documentation?
Once those answers exist, “rebuildable” becomes something I can test instead of something I say optimistically.
Separating storage creates a dependency on purpose
Moving bulk data to another system does not remove complexity. It relocates it.
The application host now depends on the storage layer, the network path between them, the expected filesystem identity, and the permissions that make the data usable. A network filesystem can be perfectly healthy while the application host cannot reach it. The network can be healthy while the mount is missing. The mount can exist while the container sees the wrong path.
That is a real dependency chain.
But it is visible.
I would rather operate a visible dependency than blur compute, application state, and irreplaceable data into one machine simply because that produces fewer boxes on a diagram.
The recent failure I covered in When an Empty Mount Looked Like an Empty Library is exactly the kind of problem this architecture can produce. The path existed, but the filesystem I expected was not mounted there. The service could therefore look alive while consuming the wrong reality.
Docker’s bind-mount documentation describes a related behavior inside containers: mounting a filesystem over an existing directory obscures the pre-existing contents at that path. That is a useful reminder that a pathname alone never proves which storage is actually behind it. Docker documents that behavior directly.
The lesson was not that separate storage was a bad idea.
The lesson was that the storage dependency deserved first-class health checks.
Process state is only the first layer of health
A running container is useful information.
It is not the same thing as a working service.
A service can have a running process while:
- Its database is unavailable
- Its expected storage is missing
- Its reverse proxy cannot reach it
- Authentication is broken
- A required dependency is still initializing
- The application can answer a health endpoint but cannot complete the user’s workflow
Docker Compose makes the startup distinction explicit. Short-form depends_on establishes dependency startup order, but it does not wait for the dependency to become healthy. The long form can use condition: service_healthy when a dependency has a meaningful health check. The current Compose service reference documents both behaviors.
That is useful, but it still does not replace end-to-end validation.
I think about health in layers:
- Is the host itself healthy?
- Are required filesystems and devices present?
- Are the container runtime and networks available?
- Are dependency services ready?
- Is the application itself responding correctly?
- Can a real client complete the workflow the service exists to provide?
The lower layers help localize failure. The final layer tells me whether the service is actually useful.
That same idea appears in The Test Matrix I Use Before I Trust a Change: validation should prove the intended outcome, not merely prove that the change command completed.
Hardware acceleration is another dependency chain
The application host can expose supported accelerator hardware to workloads that benefit from it.
It is easy to reduce that to a checkbox: GPU present, hardware acceleration enabled, done.
In practice, the path is longer.
The host operating system needs a working driver. The container runtime must be able to expose the appropriate device. The application must be configured to use a supported acceleration path. Then the workload itself has to require the operation I am trying to accelerate.
For Docker Compose deployments that use GPU reservations, Docker’s current documentation requires compatible host prerequisites and allows GPU devices to be declared through the Compose device-reservation model. Docker maintains a current Compose GPU-access guide.
My troubleshooting sequence is therefore deliberately boring:
- Confirm the host can see and use the device.
- Confirm the container has the expected device access.
- Confirm the application selected the intended acceleration path.
- Reproduce a workload that should actually use it.
- Verify the result instead of assuming utilization from configuration alone.
That last step matters. A capable server cannot erase every limitation elsewhere in the path. The client, source format, network, or application behavior may still determine what happens.
The point is not to memorize one media-server setting. It is to avoid skipping layers.
Consolidation increases the blast radius
The application host is convenient precisely because it runs several unrelated workloads.
The same convenience makes maintenance more consequential.
A host reboot can interrupt multiple services. A runtime problem can affect several Compose projects. A network failure can make applications appear unrelatedly broken. A bad storage dependency can impact every service that consumes the same shared data.
That does not make consolidation wrong. It means I need to know which services matter most.
I classify workloads loosely by operational consequence:
- Household-facing: an outage is noticeable quickly.
- Administrative: important for managing or observing the environment, but usually recoverable without panic.
- Experimental: allowed to fail without becoming an incident.
That classification influences update timing, monitoring, backup priority, and recovery order.
It also keeps me from calling everything “critical,” which is another way of saying nothing has a priority.
A second machine is not high availability
Separating compute and storage gives me options.
It does not give me automatic failover.
If the application host dies, I still need another compatible system, the required runtime, protected configuration and state, network access, storage access, secrets, and a tested process for recreating the workloads. If the storage system dies, the application host does not magically contain the authoritative data just because it used to mount it.
High availability would require additional architecture: redundant components, automated or well-defined failover behavior, synchronization rules, quorum or consistency decisions where appropriate, health detection, and regular testing of the transition between states.
I have not built that for every service, so I do not describe the lab as highly available.
Accurate language matters because it prevents architecture from getting credit for capabilities it has never demonstrated.
What I have built is recoverability through separation.
That is enough to be valuable.
Rebuildability has to be practiced
A rebuildable host should have an answer to four questions before it fails:
What recreates the host?
The operating-system baseline, required packages, container runtime, drivers, accounts, permissions, and management access should be documented well enough that rebuilding is a process rather than archaeology.
What recreates the applications?
Compose definitions, version choices, network declarations, environment configuration, and service-specific prerequisites should be preserved outside the running containers.
That is one reason Docker became my default rather than my rule. The value is not merely that a container launches quickly; the deployment model helps make application boundaries visible. I wrote more about that in Why Docker Became My Default, Not My Rule.
What must be restored?
Protected configuration, mutable application state, and secrets need explicit recovery paths. Bulk authoritative data should already live in the storage layer or another location appropriate to its role.
What proves the recovery worked?
The test is not docker ps.
The test is that the service can perform its intended job with the expected data, identity, dependencies, and client path.
If I cannot answer all four, the host is only theoretically replaceable.
New services have to earn a place on the host
Consolidated machines tend to accumulate software simply because they are available.
That is how workhorses become junk drawers.
Before adding a service, I now want to know what job it performs, what it depends on, which state it owns, what access it needs, how I will update it, and what would be required to remove it cleanly. That service-admission process is covered in Before I Add a Service, I Write Down the Job.
The question is not whether the server has enough RAM for one more container.
The better question is whether I am willing to own one more dependency.
The boundary is the real feature
The most useful thing about the application host is not its processor, graphics device, memory capacity, or the number of containers it can run.
It is the fact that its job has a boundary.
Applications live there. Some application state lives there. Authoritative bulk data lives elsewhere. Dependencies are documented. Health is tested in layers. Recovery assumes the machine can disappear.
That structure gives me freedom to change the compute layer without treating every maintenance task like open-heart surgery on the entire lab.
I still expect the machine to fail eventually.
The architecture is designed so that when it does, the failure belongs to a layer I understand.
AI transparency
AI assisted with organizing, technical verification, and editorial refinement. The architecture, failure patterns, recovery approach, and conclusions are based on my own homelab experience. Infrastructure identifiers and environment-specific details have been generalized or omitted.