When an Empty Mount Looked Like an Empty Library

A directory can exist, a container can be healthy, and the service can still be pointed at the wrong reality.

The media had not been deleted.

The application still behaved as if the library were empty.

That combination was the clue.

My first instinct could have been to blame the application: rescan the library, inspect metadata, restart the container, or start changing paths in the interface until something moved. Instead, the failure turned out to be one layer lower.

The application host expected a network filesystem at a particular mount point. The directory was there. The remote filesystem was not.

The container therefore received a perfectly valid directory containing none of the data I expected.

Nothing about that looked dramatic. No missing path. No spectacular crash. No obvious red banner saying, “The storage dependency underneath this service is gone.”

Just an empty library.

That incident changed how I think about storage-backed services. A path name is not proof of what is mounted there.

The incident path, reconstructed without pretending I saved a transcript

I did not preserve a verbatim terminal transcript from the original incident. I am not going to manufacture one after the fact.

The command blocks in this section are a sanitized reconstruction of the state I encountered and the checks I now use to distinguish the healthy and failed cases. The hostnames, paths, export names, and sample output are generic. The important part is the state transition.

Healthy baseline

In the healthy state, the mount point should be an actual mount point, and the filesystem mounted there should be the one I intended.

Using a generic path:

mountpoint -q /srv/storage/media && echo "mountpoint: yes"
findmnt --mountpoint /srv/storage/media --output TARGET,SOURCE,FSTYPE,OPTIONS

A healthy result would look conceptually like this:

mountpoint: yes
TARGET              SOURCE                 FSTYPE OPTIONS
/srv/storage/media  storage-host:/media    nfs4   rw,...

The exact source and options depend on the environment. What matters is that the exact directory is a mount point and that its source and filesystem type match the design.

I also want a cheap proof that the expected dataset is visible:

test -r /srv/storage/media/.mount-ready && echo "dataset marker: present"

That marker is optional, but it is useful when the mount itself could be valid while pointing at the wrong export or dataset. The marker belongs on the authoritative storage, not in the local directory underneath the mount.

Failed state

The dangerous state looked healthy at the pathname level.

The directory could still exist:

ls -ld /srv/storage/media

Conceptually:

drwxr-xr-x 2 root root ... /srv/storage/media

But that only proves a directory exists.

The checks that matter are different:

mountpoint -q /srv/storage/media
echo $?

findmnt --mountpoint /srv/storage/media --output TARGET,SOURCE,FSTYPE

In the failed state, mountpoint returns a non-zero status and findmnt --mountpoint returns no matching mounted filesystem.

Meanwhile, Docker can still bind the ordinary local directory into a container:

docker inspect media-service --format '{{range .Mounts}}{{println .Source "->" .Destination}}{{end}}'

The mapping can be syntactically correct while the host-side source represents the wrong filesystem.

That was the core failure:

expected: remote filesystem -> host mount point -> container bind mount -> application library
actual:   local empty directory -> container bind mount -> application library

The application was not lying. Docker was not lying. They were faithfully using the host path I gave them. The broken contract existed underneath both of them.

Why the failure looked healthy

On Linux, a mount point is still a directory when nothing is mounted on it. Mounting another filesystem there changes what is visible through that path; losing that mount does not inherently remove the directory itself.

That creates a quiet failure sequence:

1. Remote storage is unavailable or fails to mount.
2. The expected local directory still exists.
3. The container runtime receives that directory as a bind mount.
4. The application starts because its configured path is technically present.
5. The application sees an empty or incomplete dataset.
6. A process-level health check can still report success.

Every process in the stack can be running while the service outcome is wrong.

Docker bind mounts reinforce the distinction: Docker maps a host file or directory into a container. It does not prove that the host directory represents the remote filesystem I intended to mount there. That contract belongs to the host and to the deployment design.

This is the same reason I have become increasingly skeptical of “the container is running” as a meaningful definition of health. I wrote about that more broadly in Before I Add a Service, I Write Down the Job: the useful question is not whether a component exists. It is whether the job it owns is actually being performed.

The findmnt --target trap

This is the technical detail I now care about most when writing a mount check.

findmnt --target PATH answers a useful question: which mounted filesystem contains this path?

That is not the same question as: is this exact path a mount point?

If the supplied path is not itself a mount point, current findmnt behavior can walk the path components in reverse until it finds the filesystem containing the path.

So this:

findmnt --target /srv/storage/media

can still return a filesystem even when /srv/storage/media is merely an ordinary directory on /.

A conceptual failed result might look like:

TARGET SOURCE    FSTYPE
/      /dev/...  ext4

That output is not wrong. It is answering the broader --target question.

For a readiness gate where /srv/storage/media must itself be a mounted filesystem, I use one of these instead:

mountpoint -q /srv/storage/media

or:

findmnt --mountpoint /srv/storage/media

mountpoint gives the clean yes-or-no result. findmnt --mountpoint is useful when I also want the source, filesystem type, and options for that exact mount point.

That distinction is small enough to miss and important enough to turn a broken dependency into a false pass.

The dangerous version is writable

An empty read-only library is annoying. An unexpected writable directory is worse.

Imagine an application that normally writes into remote storage. If that remote filesystem is missing but the local mount-point directory remains writable, the application may continue writing into the host’s underlying local filesystem.

Later, when the remote filesystem mounts again, those locally written files become hidden behind the mounted filesystem. Nothing necessarily vanished; two different storage locations occupied the same pathname at different times.

That can create two problems at once:

• application data is split between the intended remote filesystem and an accidental local directory;
• the host filesystem quietly fills with data it was never meant to store.

Docker bind mounts are writable by default unless configured read-only. For workloads that only consume media or other static datasets, that is one reason to grant read access instead of write access.

Read-only does not solve the missing-mount problem. It limits the consequences.

A path has an identity, not just a name

I used to think about a storage path mainly as a string configured in several places:

• the host;
• the container definition;
• the application.

If those strings lined up, the configuration looked correct.

Now I treat the path as having an identity.

For a storage-backed service, the identity can include:

• whether the exact path is a mount point;
• the filesystem source;
• the filesystem type;
• read/write state;
• the service identity’s permissions;
• a known marker or representative file that proves the intended dataset is visible.

That is a much stronger contract than test -d /some/path.

A directory-existence check proves the address exists. It does not prove who lives there.

The troubleshooting order changed permanently

Once I understood the failure, the immediate repair was simple: stop depending workloads, restore the storage dependency, verify the mount, and then confirm the application could see its library again.

The useful outcome was changing the order I use whenever a storage-backed application looks empty.

I now move from the bottom upward:

Layer	Question
Authoritative storage	Is the source filesystem healthy and available?
Network	Can the application host reach the storage service?
Exact mount point	Is the required filesystem mounted on the exact directory?
Filesystem identity	Do source and filesystem type match the design?
Dataset identity	Is a known marker or representative object visible?
Host permissions	Can the service identity traverse and read or write as intended?
Container mapping	Is the intended host path mapped into the container?
Application	Can the application interpret and use the files?

If the mount is wrong, rescanning the media library is not troubleshooting. It is asking the top layer to compensate for a broken dependency beneath it.

This follows the same dependency-first approach I use in The Storage Host Became a Contract, Not Just a Server. Storage is not merely a pile of bytes. It is part of the service architecture, with availability, access, failure behavior, and recovery consequences.

The guardrail I use as the reference pattern now

The public article does not reproduce my private hostnames, exports, service units, or deployment files. The guard below is therefore a sanitized reference implementation of the rule I adopted after the incident: do not start the dependent workload until the storage contract is proven.

A simple preflight script can check three things:

1. the exact path is a mount point;
2. the mount source and filesystem type are the expected ones;
3. a marker stored on the authoritative dataset is readable.

For example:

#!/usr/bin/env bash
set -euo pipefail

MOUNT=/srv/storage/media
EXPECTED_SOURCE='storage-host:/media'
EXPECTED_FSTYPE='nfs4'
MARKER="$MOUNT/.mount-ready"

if ! mountpoint -q "$MOUNT"; then
  echo "ERROR: $MOUNT is not a mount point" >&2
  exit 1
fi

actual_source=$(findmnt --mountpoint "$MOUNT" --noheadings --output SOURCE)
actual_fstype=$(findmnt --mountpoint "$MOUNT" --noheadings --output FSTYPE)

if [[ "$actual_source" != "$EXPECTED_SOURCE" ]]; then
  echo "ERROR: unexpected mount source: $actual_source" >&2
  exit 1
fi

if [[ "$actual_fstype" != "$EXPECTED_FSTYPE" ]]; then
  echo "ERROR: unexpected filesystem type: $actual_fstype" >&2
  exit 1
fi

if [[ ! -r "$MARKER" ]]; then
  echo "ERROR: expected dataset marker is missing" >&2
  exit 1
fi

echo "storage preflight: ready"

The exact source string and filesystem type must match the environment. An NFS environment, SMB mount, local block filesystem, or bind-mounted dataset will have a different identity.

The important property is the fail-closed behavior: an ordinary local directory cannot satisfy the preflight.

I can run that before docker compose up -d, call it from an ExecStartPre= line in a systemd-managed Compose service, or use the same checks in whatever orchestration mechanism owns startup.

A startup dependency that only says “network is online” is not enough. The application depends on a specific filesystem, not on the abstract existence of networking.

What the failed and recovered checks should prove

The retest should mirror the failure instead of merely confirming that the container restarted.

During the failure

mountpoint -q /srv/storage/media

Expected result: non-zero exit status.

findmnt --mountpoint /srv/storage/media

Expected result: no exact mount entry.

/path/to/storage-preflight

Expected result: non-zero exit status, so the dependent workload is not started.

After the storage dependency is restored

mountpoint -q /srv/storage/media && echo ok
findmnt --mountpoint /srv/storage/media --output TARGET,SOURCE,FSTYPE
/path/to/storage-preflight

Expected result: the exact mount is present, the expected source/type are visible, and the preflight reports ready.

Only then do I start or restart the dependent workload and verify from inside the container:

docker exec media-service ls -la /media

The final proof is application-level: representative library content returns and the service can perform its actual job.

That gives the incident a complete test loop:

known-good baseline
→ storage dependency disappears
→ exact mount check fails
→ workload is prevented from accepting the local directory
→ storage returns
→ exact source/type/dataset checks pass
→ container sees the expected files
→ application outcome is restored

Health checks should follow the user-visible outcome

This incident is a good example of a wider rule: health checks need to move far enough through the dependency chain to catch the failures users actually experience.

Consider several increasingly useful checks for a media service:

Check	What it proves
Container process exists	The process has not exited
HTTP endpoint responds	The web service is answering
Exact filesystem is mounted	The host storage dependency is present
Dataset marker is readable	The expected dataset appears to be mounted
Library path is readable in the container	The container mapping and permissions work
Known media is visible	The expected content is present
A representative item plays	A large part of the actual service path works

I would not run the most expensive end-to-end test every few seconds.

A cheap dependency check can run frequently. A deeper functional check can run less often. When the deeper test fails, the lower-level checks help identify which boundary broke.

That makes an alert much more useful than “media server down.”

Mount failures deserve their own alert

An empty dataset and an unavailable dependency are not the same condition.

If a library is genuinely empty, the mount can still be healthy. If a required network filesystem is missing, I want the alert to say that directly before applications begin producing secondary symptoms.

The best alert is attached as close as practical to the failed layer.

Storage unavailable should look like storage unavailable.

A dependent application should not be the first component responsible for telling me the storage underneath it disappeared.

This is part of the broader standard I described in The Test Matrix I Use Before I Trust a Change: validation should prove the intended outcome and its important boundaries, not just the easiest successful state to measure.

Recovery includes checking underneath the mount

There is one more recovery step that matters when a missing mount was writable: determine whether anything was written to the underlying local directory while the remote filesystem was absent.

Once the remote filesystem is mounted again, those local files are hidden by the mount.

A safe maintenance sequence is:

1. Stop every workload that can write to the path.
2. Record the currently expected mount source and state.
3. Unmount the remote filesystem during a maintenance window.
4. Inspect the now-visible underlying local directory.
5. Quarantine or reconcile unexpected local files rather than blindly copying them into the authoritative dataset.
6. Restore the mount.
7. Re-run the exact mount/source/dataset preflight.
8. Start the dependent workload.
9. Validate the application outcome.

Do not inspect or reconcile split data while writers are still active. The point is to remove ambiguity, not create a second race condition during recovery.

What I changed after the incident

The lasting changes were more valuable than the remount itself.

For storage-backed services, my deployment standard is now:

• test the exact mount point, not just whether the path exists;
• use findmnt --mountpoint or mountpoint when exact mount identity matters;
• verify source and filesystem type where the distinction matters;
• optionally use a dataset marker for another cheap identity check;
• keep read-only data read-only wherever practical;
• fail startup when the required storage contract is absent;
• alert on the missing mount separately from application symptoms;
• verify representative data after the storage returns;
• inspect for accidental local writes after a writable mount outage;
• document which workloads depend on each remote filesystem.

None of that is glamorous infrastructure work.

It is exactly the kind that makes the lab less mysterious six months later.

The lesson was larger than NFS

My incident involved network storage, but the principle is broader.

A path can exist while the wrong filesystem is mounted there.

A bind mount can be valid while pointing at the wrong host data.

A device node can exist while the device behind it is unavailable or incorrect.

A port can listen while an upstream dependency is dead.

A login page can load while the service behind it cannot perform its actual job.

Infrastructure often fails in states that are syntactically valid and operationally wrong.

Those are the failures that simple “is it running?” checks miss.

The empty library taught me to ask a better question:

What does this healthy-looking layer depend on, and have I actually proven that dependency?

That question has been far more useful than another restart command.

Security note

This article intentionally omits or generalizes infrastructure-specific details. Real hostnames, addresses, export names, account identifiers, service names, mount points, internal directory structures, and topology remain in private operational documentation. Reconstructed command output is explicitly labeled and does not claim to be a verbatim incident transcript.

References

• Docker Docs: Bind mounts
• Linux manual: findmnt(8)
• Linux manual: mountpoint(1)

AI transparency

AI assisted with structure, technical verification, sanitized reconstruction, and editing. The missing-mount incident, troubleshooting sequence, and operational lessons are based on my own homelab experience. The September 13, 2026 revision deliberately distinguishes reconstructed example output from historical evidence that was not preserved.