mirror of
https://github.com/vmware-tanzu/velero.git
synced 2026-08-15 19:56:06 +00:00
Signed-off-by: wolf-06 <Uajjawal.06@gmail.com>
This commit is contained in:
Uajjawal
@@ -170,6 +170,7 @@ kubectl -n velero get datadownloads -l velero.io/restore-name=YOUR_RESTORE_NAME
|
||||
that anyone who has access to your backup storage can decrypt your backup data**. Make sure that you limit access
|
||||
to the backup storage appropriately.
|
||||
- [Velero built-in data mover] Even though the backup data could be incrementally preserved, for a single file data, Velero built-in data mover leverages on deduplication to find the difference to be saved. This means that large files (such as ones storing a database) will take a long time to scan for data deduplication, even if the actual difference is small.
|
||||
- [Velero built-in data mover] On volumes where the underlying filesystem enforces mount-constant identity (Azure Files SMB/CIFS, Azure Blob via blobfuse, GCP Cloud Storage FUSE, and similar), data download's `chown`/`chmod` can report success while changing nothing, silently losing file ownership (and on FUSE mounts, permission bits) with no error surfaced anywhere. See [File Ownership and Permission Preservation](file-system-backup.md#file-ownership-and-permission-preservation) for details and remediation.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@@ -367,7 +367,98 @@ For this reason, FSB can only backup volumes that are mounted by a pod and not d
|
||||
(without running pods), some Velero users overcame this limitation running a staging pod (i.e. a busybox or alpine container
|
||||
with an infinite sleep) to mount these PVC/PV pairs prior taking a Velero backup.
|
||||
- Velero File System Backup expects volumes to be mounted under `<hostPath>/<pod UID>` (`hostPath` is configurable as mentioned in [Configure Node Agent DaemonSet spec](#configure-node-agent-daemonset-spec)). Some Kubernetes systems (i.e., [vCluster][11]) don't mount volumes under the `<pod UID>` sub-dir, Velero File System Backup is not working with them.
|
||||
- File system restores of the same pod won't start until all the volumes of the pod get bound, even though some of the volumes have been bound and ready for restore. An a result, if a pod has multiple volumes, while only part of the volumes are restored by file system restore, these file system restores won't start until the other volumes are restored completely by other restore types (i.e., [CSI Snapshot Restore][12], [CSI Snapshot Data Movement][13]), the file system restores won't happen concurrently with those other types of restores.
|
||||
- File system restores of the same pod won't start until all the volumes of the pod get bound, even though some of the volumes have been bound and ready for restore. An a result, if a pod has multiple volumes, while only part of the volumes are restored by file system restore, these file system restores won't start until the other volumes are restored completely by other restore types (i.e., [CSI Snapshot Restore][12], [CSI Snapshot Data Movement][13]), the file system restores won't happen concurrently with those other types of restores.
|
||||
- On volumes where the underlying filesystem enforces mount-constant identity (Azure Files SMB/CIFS, Azure Blob via blobfuse, GCP Cloud Storage FUSE, and similar), FSB restore's `chown`/`chmod` can report success while changing nothing, silently losing file ownership (and on FUSE mounts, permission bits) with no error surfaced anywhere. See [File Ownership and Permission Preservation](#file-ownership-and-permission-preservation) below.
|
||||
|
||||
## File Ownership and Permission Preservation
|
||||
|
||||
[#file-ownership-and-permission-preservation](#file-ownership-and-permission-preservation)
|
||||
|
||||
Some volume types enforce a **mount-constant identity**: file ownership and/or permission mode are determined
|
||||
entirely by the mount configuration rather than being stored per-file on the underlying storage. On these
|
||||
filesystems, when FSB restore runs `chown`/`chmod` as root, the system call **returns success while changing
|
||||
nothing**: the restored files simply present whatever owner/mode the mount is configured to force. Because no
|
||||
error is ever raised, this is a silent failure: the restore reports `Completed` with zero warnings, and nothing
|
||||
in the node-agent or data mover pod logs indicates a problem.
|
||||
|
||||
This is a distinct failure mode from cases where the storage backend actively rejects the ownership change
|
||||
(for example, NFS server-side `root_squash`, which returns a real `EPERM`). That class of failure can, in
|
||||
principle, be caught by inspecting the error path. The mount-constant-identity case cannot, because there is no
|
||||
error to catch.
|
||||
|
||||
**Affected volume types (verified or by design):**
|
||||
|
||||
| Storage | Ownership storage | `chown` as root | `chmod` as root |
|
||||
| ---------------------------------------------------------------- | ----------------------------------------------- | ------------------------------------------ | ------------------------------------------ |
|
||||
| Azure Files SMB/CIFS, default or forced `uid=`/`gid=` mount options | Mount-constant | Silent no-op | Forced by `file_mode=`/`dir_mode=` |
|
||||
| Azure Files SMB with `idsfromsid,modefromsid` mount options | Stored in NTFS security descriptors | Works, persists, survives remount | Works, persists |
|
||||
| Azure Blob via blobfuse (`blobfuse2`) | Mount-constant | Silent no-op | Silent no-op |
|
||||
| Azure Blob NFSv3 (Premium) | Real POSIX (server-side) | Works | Works |
|
||||
| Azure Files NFS 4.1 (Premium) | Real POSIX (server-side) | Works | Works |
|
||||
| Azure Files NFS 4.1 with `rootSquashType: RootSquash` | Real POSIX, but root is squashed | Real `EPERM` (see NFS ownership caveat below) | Works |
|
||||
| GCP Cloud Storage FUSE (`gcsfuse.csi.storage.gke.io`) | Mount-time uid/gid/mode, not stored | Not supported - silent-loss class | Not supported - silent-loss class |
|
||||
| AWS FSx for Windows (SMB, NTFS ACLs) | NTFS ACLs, don't map to POSIX ownership | Doesn't map | Doesn't map |
|
||||
| AWS EFS Access Points with a `PosixUser` | Access Point overrides uid/gid for all operations | Neutralized server-side | N/A |
|
||||
|
||||
Azure Disk (block storage) and plain Azure Files/EFS without the above configurations use real POSIX semantics
|
||||
and are not affected.
|
||||
|
||||
### Verified remediation for Azure Files SMB
|
||||
|
||||
Add `idsfromsid,modefromsid` to the StorageClass `mountOptions`, and do **not** force `uid=`/`gid=`/`mode=`
|
||||
alongside them. This stores real per-file ownership and mode in the share's NTFS security descriptors and gives
|
||||
full fidelity across backup and restore.
|
||||
|
||||
Caveats:
|
||||
|
||||
- On a fresh share, the volume root receives a translated security descriptor on first mount (typically
|
||||
`uid=0 gid=<fsGroup> mode=1707`). Non-root workloads may need a one-time root init container to `chown`/`chmod`
|
||||
the volume root before the main container starts; this operation itself works correctly on this mount.
|
||||
- A restrictive owner/mode on the volume root can prevent Velero's FSB restore-wait init container from
|
||||
accessing the volume if its identity doesn't match the workload's. If you hit a restore stuck at
|
||||
`Init:0/1`, configure the restore helper's security context (`secCtxRunAsUser`, `secCtxRunAsGroup`, or `secCtx`)
|
||||
to match your workload's UID/GID. See [Customize Restore Helper Container](#customize-restore-helper-container).
|
||||
|
||||
As an alternative, Azure Files NFS 4.1 (Premium tier) or Azure Blob NFSv3 (Premium tier) also preserve ownership
|
||||
and mode with full fidelity, **provided you avoid `rootSquashType: RootSquash`**. Root-squashed NFS mounts
|
||||
reject root's `chown` with a real `EPERM`, which is a different (but related) failure. See the NFS ownership
|
||||
note below.
|
||||
|
||||
### No remediation exists for blobfuse or gcsfuse
|
||||
|
||||
For Azure Blob via blobfuse and GCP Cloud Storage FUSE, there is currently no mount option or configuration
|
||||
that preserves per-file ownership or mode. This is a limitation of the FUSE drivers themselves, not something
|
||||
Velero or its restore path can work around. If your workload depends on stat-level ownership fidelity (for
|
||||
example, databases like PostgreSQL or MySQL that refuse to start if the data directory's ownership doesn't
|
||||
match the running user), avoid these volume types for that data. Use block storage, a real POSIX-backed
|
||||
protocol (e.g. Azure Files NFS 4.1, Azure Blob NFSv3), or Azure Files SMB with `idsfromsid,modefromsid` instead.
|
||||
|
||||
### Related: NFS root_squash ownership loss
|
||||
|
||||
A related but mechanically distinct issue affects NFS mounts with server-side `root_squash` enabled: the
|
||||
`chown` call receives a real `EPERM` from the server, but Velero's kopia integration currently sets
|
||||
`IgnorePermissionErrors: true`, which silently discards that error. The end result looks the same to the user
|
||||
(a `Completed` restore with lost ownership), but the underlying mechanism differs. Here an error genuinely
|
||||
occurs, it is simply swallowed, whereas on mount-constant-identity filesystems no error is ever generated in
|
||||
the first place. If you're troubleshooting ownership loss on NFS-backed volumes with root squashing enabled,
|
||||
this is the more likely cause.
|
||||
|
||||
### Diagnosing which case you're hitting
|
||||
|
||||
Check the mount options inside the affected pod:
|
||||
|
||||
`mount | grep -E 'cifs|fuse|nfs'`
|
||||
|
||||
Look for `uid=`/`gid=` (CIFS) or a FUSE filesystem type. The typical symptom in all these cases is a restore
|
||||
that reports `Completed` with no warnings, followed by an application failing immediately afterward with an
|
||||
ownership-related error, for example:
|
||||
|
||||
FATAL: data directory "/var/lib/postgresql/data/pgdata" has wrong ownership
|
||||
HINT: The server must be started by the user that owns the data directory.
|
||||
|
||||
This signature, a clean restore followed by an immediate ownership-related crash, is the indicator that
|
||||
you're affected by one of the limitations described above rather than a genuine restore failure.
|
||||
|
||||
|
||||
## Customize Restore Helper Container
|
||||
|
||||
|
||||
Reference in New Issue
Block a user