This commit makes backup sync controller delete the volumesnapshot and
volumesnapshotcontent created by the backup which is cleaned up as orphan
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
Fixes#4760
This commit make changes in 2 parts:
1) When a volumesnapshotcontent is persisted during backup, velero will reset its
`Source` field to remove the VolumeHandle, so that the
csi-snapshotter will not try to call `CreateSnapshot` when its synced
to another cluster with a backup.
2) Make sure the referenced volumesnapshotclasses are persisted and
synced with the backup, so that when the volumesnapshotcontent is
deleted the storage snapshot is also removed.
Signed-off-by: Daniel Jiang <jiangd@vmware.com>
* Migrate backup sync controller from code-generator to kubebuilder
1. use kubebuilder's reconcile logic to replace controller's old logic.
2. use ginkgo and gomega to replace testing.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* Fix: modify code according to comments
1. Remove DefaultBackupLocation
2. Remove unneccessary comment line
3. Add syncPeriod default value setting logic
4. Modify ListBackupStorageLocations function's context parameter
5. Add RequeueAfter parameter in Reconcile function return value
Signed-off-by: Xun Jiang <jxun@vmware.com>
* Reconcile function use context passed from parameter
1. Use context passed from parameter, instead of using Reconciler struct's context.
2. Delete Reconciler struct's context member.
3. Modify test case accordingly.
Signed-off-by: Xun Jiang <jxun@vmware.com>
logrusr is a open source convertor, which can convert logrus logger into logr.
By using logrusr, velero can use exsiting formatted logrus logger, other than introducing zap as a new logger.
Signed-off-by: Xun Jiang <jxun@vmware.com>
* Use Credential from BSL for restic commands
This change introduces support for restic to make use of per-BSL
credentials. It makes use of the `credentials.FileStore` introduced in
PR #3442 to write the BSL credentials to disk. To support per-BSL
credentials for restic, the environment for the restic commands needs to
be modified for each provider to ensure that the credentials are
provided via the correct provider specific environment variables.
This change introduces a new function `restic.CmdEnv` to check the BSL
provider and create the correct mapping of environment variables for
each provider.
Previously, AWS and GCP could rely on the environment variables in the
Velero deployments to obtain the credentials file, but now these
environment variables need to be set with the path to the serialized
credentials file if a credential is set on the BSL.
For Azure, the credentials file in the environment was loaded and parsed
to set the environment variables for restic. Now, we check if the BSL
has a credential, and if it does, load and parse that file instead.
This change also introduces a few other small improvements. Now that we
are fetching the BSL to check for the `Credential` field, we can use the
BSL directly to get the `CACert` which means that we can remove the
`GetCACert` function. Also, now that we have a way to serialize secrets
to disk, we can use the `credentials.FileStore` to get a temp file for
the restic repo password and remove the `restic.TempCredentialsFile`
function.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Add documentation for per-BSL credentials
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Address review feedback
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Address review comments
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Load credentials and pass to ObjectStorage plugins
Update NewObjectBackupStore to take a CredentialsGetter which can be
used to get the credentials for a BackupStorageLocation if it has been
configured with a Credential. If the BSL has a credential, use that
SecretKeySelector to fetch the secret, write the contents to a temp file
and then pass that file through to the plugin via the config map using
the key `credentialsFile`. This relies on the plugin being able to use
this new config field.
This does not yet handle VolumeSnapshotLocations or ResticRepositories.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Address code reviews
Add godocs and comments.
Improve formatting and test names.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Address code reviews
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Use kubebuilder client for fetching restic secrets
Instead of using a SecretInformer for fetching secrets for restic, use
the cached client provided by the controller-runtime manager.
In order to use this client, the scheme for Secrets must be added to the
scheme used by the manager so this is added when creating the manager in
both the velero and restic servers.
This change also refactors some of the tests to add a shared utility for
creating a fake controller-runtime client which is now used among all
tests which use that client. This has been added to ensure that all
tests use the same client with the same scheme.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Add builder for SecretKeySelector
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
Restoring CAPI workload clusters without this ordering caused the
capi-controller-manager code to panic, resulting in an unhealthy cluster
state.
This can be worked around
(https://community.pivotal.io/s/article/5000e00001pJyN41611954332537?language=en_US),
but we provide the inclusion of these resources as a default in order to
provide a better out-of-the-box experience.
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
In preparation for modifying the instantiation of `BackupStores` to be
able to load credentials, change the function `NewObjectBackupStore` to
be an interface that is passed in to all controllers.
Previously, the function to get a new backup store was configurable but
for many controllers was fixed to use `NewObjectBackupStore`. This
change introduces an interface for getting the backup store and wraps
the functionality from `NewObjectBackupStore` in a type which implements
this interface. This will allow more flexibility when introducing
credentials for a specific backup store as it will allow us to create a
new `ObjectBackupStoreGetter` type which can be configured to add
credentials config when creating the ObjectBackupStore without needing
to change the API used by the controllers.
Signed-off-by: Bridget McErlean <bmcerlean@vmware.com>
* Add default field to BSL CRD
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Add a new flag `--default` under `velero backup-location create`
add a new flag `--default` under `velero backup-location create`
to specify this new location to be the new default BSL.
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Add a new default field under `velero backup-location get`
add a new default field under `velero backup-location get` to indicate
which BSL is the default one.
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Add a new sub-command and flag under `velero backup-location`
Add a new sub-command called `velero backup-location set` sub-command
and a new flag `velero backup-cation set --default` to configure which
BSL is the default one.
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Add new flag to get the default backup-location
Add a new flag `--default` under `velero backup-location get`
to displays the current default BSL.
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Configures default BSL in BSL controller
When upgrade the BSL CRDs, none of the BSL has been labeled as default.
Sets the BSL default field to true if the BSL name matches to the default BSL setting.
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Configures the default BSL in BSL controller for velero upgrade
When upgrade the BSL CRDs, none of the BSL be marked as the default.
Sets the BSL `.spec.default: true` if the BSL name matches against the
`velero server --default-backup-storage-location`.
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Add unit test to test default BSL behavior
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Update check which one is the default BSL in backup/backup_sync/restore controller
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Add changelog
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Update docs locations.md and upgrade-to-1.6.md
Signed-off-by: JenTing Hsiao <jenting.hsiao@suse.com>
* Exec hooks in restored pods
Signed-off-by: Andrew Reed <andrew@replicated.com>
* WaitExecHookHandler implements ItemHookHandler
This required adding a context.Context argument to the ItemHookHandler
interface which is unused by the DefaultItemHookHandler implementation.
It also means passing nil for the []ResourceHook argument since that
holds BackupResourceHook.
Signed-off-by: Andrew Reed <andrew@replicated.com>
* WaitExecHookHandler unit tests
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Changelog and go fmt
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Fix double import
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Default to first contaienr in pod
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Use constants for hook error modes in tests
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Revert to separate WaitExecHookHandler interface
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Negative tests for invalid timeout annotations
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Rename NamedExecRestoreHook PodExecRestoreHook
Also make field names more descriptive.
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Cleanup test names
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Separate maxHookWait and add unit tests
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Comment on maxWait <= 0
Also info log container is not running for hooks to execute in.
Also add context error to hooks not executed errors.
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Remove log about default for invalid timeout
There is no default wait or exec timeout.
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Linting
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Fix log message and rename controller to podWatcher
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Comment on exactly-once semantics for handler
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Fix logging and comments
Use filed logger for pod in handler.
Add comment about pod changes in unit tests.
Use kube util NamespaceAndName in messages.
Signed-off-by: Andrew Reed <andrew@replicated.com>
* Fix maxHookWait
Signed-off-by: Andrew Reed <andrew@replicated.com>
* k8s 1.18 import wip
backup, cmd, controller, generated, restic, restore, serverstatusrequest, test and util
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* go mod tidy
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* add changelog file
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* go fmt
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* update code-generator and controller-gen in CI
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* checkout proper code-generator version, regen
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* fix remaining calls
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* regenerate CRDs with ./hack/update-generated-crd-code.sh
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* use existing context in restic and server
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* fix test cases by resetting resource version
also use main library go context, not golang.org/x/net/context, in pkg/restore/restore.go
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* clarify changelog message
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* use github.com/kubernetes-csi/external-snapshotter/v2@v2.2.0-rc1
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* run 'go mod tidy' to remove old external-snapshotter version
Signed-off-by: Andrew Lavery <laverya@umich.edu>
* kubebuilder init - minimalist version
Signed-off-by: Carlisia <carlisia@vmware.com>
* Add back main.go, apparently kb needs it
Signed-off-by: Carlisia <carlisia@vmware.com>
* Tweak makefile to accomodate kubebuilder expectations
Signed-off-by: Carlisia <carlisia@vmware.com>
* Port BSL to kubebuilder api client
Signed-off-by: Carlisia <carlisia@vmware.com>
* s/cache/client bc client fetches from cache
And other naming improvements
Signed-off-by: Carlisia <carlisia@vmware.com>
* So, .GetAPIReader is how we bypass the cache
In this case, the cache hasn't started yet
Signed-off-by: Carlisia <carlisia@vmware.com>
* Oh that's what this code was for... adding back
We still need to embed the CRDs as binary data in the Velero binary to
access the generated CRDs at runtime.
Signed-off-by: Carlisia <carlisia@vmware.com>
* Tie in CRD/code generation w/ existing scripts
Signed-off-by: Carlisia <carlisia@vmware.com>
* Mostly result of running update-fmt, updated file formatting
Signed-off-by: Carlisia <carlisia@vmware.com>
* Just a copyright fix
Signed-off-by: Carlisia <carlisia@vmware.com>
* All the test fixes
Signed-off-by: Carlisia <carlisia@vmware.com>
* Add changelog + some cleanup
Signed-off-by: Carlisia <carlisia@vmware.com>
* Update backup manifest
Signed-off-by: Carlisia <carlisia@vmware.com>
* Remove unneeded auto-generated files
Signed-off-by: Carlisia <carlisia@vmware.com>
* Keep everything in the same (existing) package
Signed-off-by: Carlisia <carlisia@vmware.com>
* Fix/clean scripts, generated code, and calls
Deleting the entire `generated` directory and running `make update`
works. Modifying an api and running `make verify` works as expected.
Signed-off-by: Carlisia <carlisia@vmware.com>
* Clean up schema and client calls + code reviews
Signed-off-by: Carlisia <carlisia@vmware.com>
* Move all code gen to inside builder container
Signed-off-by: Carlisia <carlisia@vmware.com>
* Address code review
Signed-off-by: Carlisia <carlisia@vmware.com>
* Fix imports/aliases
Signed-off-by: Carlisia <carlisia@vmware.com>
* More code reviews
Signed-off-by: Carlisia <carlisia@vmware.com>
* Add waitforcachesync
Signed-off-by: Carlisia <carlisia@vmware.com>
* Have manager register ALL controllers
This will allow for proper cache management.
Signed-off-by: Carlisia <carlisia@vmware.com>
* Status subresource is now enabled; cleanup
Signed-off-by: Carlisia <carlisia@vmware.com>
* More code reviews
Signed-off-by: Carlisia <carlisia@vmware.com>
* Clean up
Signed-off-by: Carlisia <carlisia@vmware.com>
* Manager registers ALL controllers for restic too
Signed-off-by: Carlisia <carlisia@vmware.com>
* More code reviews
Signed-off-by: Carlisia <carlisia@vmware.com>
* Add deprecation warning/todo
Signed-off-by: Carlisia <carlisia@vmware.com>
* Add documentation
Signed-off-by: Carlisia <carlisia@vmware.com>
* Add helpful comments
Signed-off-by: Carlisia <carlisia@vmware.com>
* Address code review
Signed-off-by: Carlisia <carlisia@vmware.com>
* More idiomatic Runnable
Signed-off-by: Carlisia <carlisia@vmware.com>
* Clean up imports
Signed-off-by: Carlisia <carlisia@vmware.com>
