Commit Graph

982 Commits

Author SHA1 Message Date
Nolan Brubaker
bbcbde084d Create hook package (#2734)
* Move pkg/backup/item_hook_handler to internal/hoo

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add internal packages to test target

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-07-22 14:26:14 -07:00
Ashish Amarnath
2636730ef2 fix copyright year
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-07-22 12:11:59 -07:00
Martin Odstrčilík
86efd1577e add support for setting SecurityContext (user, group) for restic restore (#2621)
* add support for setting SecurityContext (user, group) for restic restore

Signed-off-by: Martin Odstrcilik <martin.odstrcilik@gmail.com>
2020-07-22 12:10:25 -07:00
Ashish Amarnath
91ccc4adb2 Add metrics for restic back up operation (#2719)
* add metrics for restic back up operation

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* changelog

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-07-22 15:07:52 -04:00
Thejas Babu
d0d143e119 Add StartTimestamp and CompletionTimestamp in Restore Status (#2748)
Signed-off-by: thejas <thejasb99@gmail.com>
2020-07-22 11:40:39 -07:00
Carlisia Campos
e60d9f2821 Remove unneeded auto-generated files (#2732)
Signed-off-by: Carlisia <carlisia@vmware.com>
2020-07-20 15:37:27 -07:00
fvsqr
01e2dcb364 StorageGrid compatibility (#2712)
* remove explicit Accept-Encoding header

For StorageGrid compatibility the Accept-Encoding header should not be set, otherwise StorageGrid compresses the already compressed log files which are only decompressed by the client once

Signed-off-by: fvsqr <48791253+fvsqr@users.noreply.github.com>

* Removed explicit gzip Accept-Encoding header

For StorageGrid compatibility the Accept-Encoding header should not be set, otherwise StorageGrid compresses the already compressed log files which are only decompressed by the client once.
Unclear, how this affects Backup endpoints from Azure or GCP

Signed-off-by: fvsqr <48791253+fvsqr@users.noreply.github.com>

* Create 2712-fvsqr

Signed-off-by: fvsqr <48791253+fvsqr@users.noreply.github.com>
2020-07-20 13:11:26 -04:00
Nolan Brubaker
f42c63af1b Address insensitive language (#2677)
* Change master to main in most uses

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-07-17 14:59:51 -07:00
kathpeony
243ac62e3f Add backupValidationFailureTotal to metrics (#2714)
* Add backValidationFailureTotal to metrics

Signed-off-by: Kathrin Mao <kathrin.mao@sap.com>
2020-07-16 10:13:17 -07:00
Andrew Lavery
a368370bef k8s 1.18 import (#2651)
* k8s 1.18 import wip

backup, cmd, controller, generated, restic, restore, serverstatusrequest, test and util

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* go mod tidy

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* add changelog file

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* go fmt

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* update code-generator and controller-gen in CI

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* checkout proper code-generator version, regen

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* fix remaining calls

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* regenerate CRDs with ./hack/update-generated-crd-code.sh

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* use existing context in restic and server

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* fix test cases by resetting resource version

also use main library go context, not golang.org/x/net/context, in pkg/restore/restore.go

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* clarify changelog message

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* use github.com/kubernetes-csi/external-snapshotter/v2@v2.2.0-rc1

Signed-off-by: Andrew Lavery <laverya@umich.edu>

* run 'go mod tidy' to remove old external-snapshotter version

Signed-off-by: Andrew Lavery <laverya@umich.edu>
2020-07-16 12:21:37 -04:00
Carlisia Campos
13e1eeabfe Return early from a BSL controller (#2709)
Signed-off-by: Carlisia <carlisia@vmware.com>
2020-07-14 15:09:11 -07:00
Carlisia Campos
dbd0aa4915 Add a BSL controller to handle validation + update BSL status phase (#2674)
* Add BSL controller

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add changelog

Signed-off-by: Carlisia <carlisia@vmware.com>

* Make update

Signed-off-by: Carlisia <carlisia@vmware.com>

* Update docs

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add kubebuilder dependency

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add export

Signed-off-by: Carlisia <carlisia@vmware.com>

* add kubebuilder binaries into velero builder image

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* Reset velero dockerfile

Signed-off-by: Carlisia <carlisia@vmware.com>

* Consolidate all logic

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add copyright header

Signed-off-by: Carlisia <carlisia@vmware.com>

* Clean up + add "last validated" column

Signed-off-by: Carlisia <carlisia@vmware.com>

* Better tests

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add more tests

Signed-off-by: Carlisia <carlisia@vmware.com>

* Better logging

Signed-off-by: Carlisia <carlisia@vmware.com>

* Format

Signed-off-by: Carlisia <carlisia@vmware.com>

* Code reviews

Signed-off-by: Carlisia <carlisia@vmware.com>

* Address code review

Signed-off-by: Carlisia <carlisia@vmware.com>

* Remove redundant logic

Signed-off-by: Carlisia <carlisia@vmware.com>

Co-authored-by: Ashish Amarnath <ashisham@vmware.com>
2020-07-14 17:47:00 -04:00
Ashish Amarnath
3d3b9e312a Revert "🐛 fix file perissions on the manifest JSON in backup archive (#2685)" (#2700)
dae5230aae

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-07-10 14:07:05 -07:00
Nolan Brubaker
54aa75a4cd Adjust restic timeout and pod values up (#2696)
* Adjust restic timeout and pod values up

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-07-08 14:59:19 -07:00
Ashish Amarnath
dae5230aae 🐛 fix file perissions on the manifest JSON in backup archive (#2685)
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-07-08 06:37:59 -07:00
Ashish Amarnath
b4465e92b8 🐛 Use CRD version prior to remap_crd_version backup item action (#2683)
* 🐛 preserve crd version before remapping

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-07-07 10:59:41 -07:00
Daniel Thrasher
94a9522f75 updated acceptable values on cron schedule for day of the week from 0-7 to 0-6 (#2676)
* updated acceptable values on cron schedule for day of the week from 0-7 to 0-6

Signed-off-by: Daniel Thrasher <dannythrasher@gmail.com>

* added a changelog file to changelog directory

Signed-off-by: Daniel Thrasher <dannythrasher@gmail.com>

Co-authored-by: Daniel Thrasher <dannythrasher@gmail.com>
2020-07-06 17:14:44 -04:00
Carlisia Campos
4048c020a8 Convert manifests + BSL api client to kubebuilder (#2561)
* kubebuilder init - minimalist version

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add back main.go, apparently kb needs it

Signed-off-by: Carlisia <carlisia@vmware.com>

* Tweak makefile to accomodate kubebuilder expectations

Signed-off-by: Carlisia <carlisia@vmware.com>

* Port BSL to kubebuilder api client

Signed-off-by: Carlisia <carlisia@vmware.com>

* s/cache/client bc client fetches from cache
And other naming improvements

Signed-off-by: Carlisia <carlisia@vmware.com>

* So, .GetAPIReader is how we bypass the cache
In this case, the cache hasn't started yet

Signed-off-by: Carlisia <carlisia@vmware.com>

* Oh that's what this code was for... adding back

We still need to embed the CRDs as binary data in the Velero binary to
access the generated CRDs at runtime.

Signed-off-by: Carlisia <carlisia@vmware.com>

* Tie in CRD/code generation w/ existing scripts

Signed-off-by: Carlisia <carlisia@vmware.com>

* Mostly result of running update-fmt, updated file formatting

Signed-off-by: Carlisia <carlisia@vmware.com>

* Just a copyright fix

Signed-off-by: Carlisia <carlisia@vmware.com>

* All the test fixes

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add changelog + some cleanup

Signed-off-by: Carlisia <carlisia@vmware.com>

* Update backup manifest

Signed-off-by: Carlisia <carlisia@vmware.com>

* Remove unneeded auto-generated files

Signed-off-by: Carlisia <carlisia@vmware.com>

* Keep everything in the same (existing) package

Signed-off-by: Carlisia <carlisia@vmware.com>

* Fix/clean scripts, generated code, and calls

Deleting the entire `generated` directory and running `make update`
works. Modifying an api and running `make verify` works as expected.

Signed-off-by: Carlisia <carlisia@vmware.com>

* Clean up schema and client calls + code reviews

Signed-off-by: Carlisia <carlisia@vmware.com>

* Move all code gen to inside builder container

Signed-off-by: Carlisia <carlisia@vmware.com>

* Address code review

Signed-off-by: Carlisia <carlisia@vmware.com>

* Fix imports/aliases

Signed-off-by: Carlisia <carlisia@vmware.com>

* More code reviews

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add waitforcachesync

Signed-off-by: Carlisia <carlisia@vmware.com>

* Have manager register ALL controllers

This will allow for proper cache management.

Signed-off-by: Carlisia <carlisia@vmware.com>

* Status subresource is now enabled; cleanup

Signed-off-by: Carlisia <carlisia@vmware.com>

* More code reviews

Signed-off-by: Carlisia <carlisia@vmware.com>

* Clean up

Signed-off-by: Carlisia <carlisia@vmware.com>

* Manager registers ALL controllers for restic too

Signed-off-by: Carlisia <carlisia@vmware.com>

* More code reviews

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add deprecation warning/todo

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add documentation

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add helpful comments

Signed-off-by: Carlisia <carlisia@vmware.com>

* Address code review

Signed-off-by: Carlisia <carlisia@vmware.com>

* More idiomatic Runnable

Signed-off-by: Carlisia <carlisia@vmware.com>

* Clean up imports

Signed-off-by: Carlisia <carlisia@vmware.com>
2020-06-24 12:55:18 -04:00
Carlisia Campos
0b8e2cbbe5 Improve velero download doc (#2660)
* Improve velero download doc

Signed-off-by: Carlisia <carlisia@vmware.com>

* Add changelog

Signed-off-by: Carlisia <carlisia@vmware.com>

* Better message

Signed-off-by: Carlisia <carlisia@vmware.com>
2020-06-23 11:12:01 -07:00
Ashish Amarnath
e5e7c025dd fix copyright boilerplate
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-18 14:14:15 -07:00
Ashish Amarnath
7abd2c6db9 doc updates
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-18 13:57:59 -07:00
Ashish Amarnath
6a8dca6b84 add default-volumes-to-restic flag to velero installation
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-16 14:46:36 -07:00
Ashish Amarnath
63f7690f44 more tests
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-16 14:46:36 -07:00
Ashish Amarnath
b0fd3d35c1 rename field
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-15 15:26:44 -07:00
Ashish Amarnath
b71173228a add velero server flag to allow default restic use on all pod volumes
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-15 14:46:06 -07:00
Ashish Amarnath
8a2a852b87 use backup's defaultRestic flag to identify pod volumes using restic
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-15 14:46:06 -07:00
Ashish Amarnath
f34aab251e add default restic flag to backup create cli
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-15 14:46:06 -07:00
Ashish Amarnath
69cceb0d7e add backup level flag to opt-in for default restic use
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-06-15 14:46:06 -07:00
Steve Kriss
a5346c1a87 azure: support aad-pod-identity auth when using restic (#2602)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2020-06-08 11:17:43 -07:00
Nolan Brubaker
d9d9dc60da Log when a hook timeout duration can't be parsed (#2610)
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-06-05 12:45:50 -07:00
Steve Kriss
1c80ba903e don't error during backup when additional items returned by plugin don't exist (#2595)
* log a warning instead of erroring if additional item can't be found

Signed-off-by: Steve Kriss <krisss@vmware.com>

* always show backup warning/error count in get/describe

Signed-off-by: Steve Kriss <krisss@vmware.com>

* changelog

Signed-off-by: Steve Kriss <krisss@vmware.com>
2020-06-04 16:29:55 -07:00
Christoph Blecker
ceae7a2fdc When creating backup from schedule, allow default name (#2569)
* When creating backup from schedule, allow default name

Signed-off-by: Christoph Blecker <admin@toph.ca>
2020-05-28 12:31:25 -07:00
Carlisia Campos
40d8511f64 Add status column to get BSL output (#2493)
* Add status column to get BSL output

Signed-off-by: Carlisia <carlisia@vmware.com>

* Address code review

Signed-off-by: Carlisia <carlisia@vmware.com>
2020-05-28 10:52:46 -07:00
Steve Kriss
5b52fd3efd re-instantiate backup store just before persisting artifacts (#2550)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2020-05-27 16:03:52 -07:00
Jonas Rosland
d10eea3ba3 Add up and down votes to the issue templates and contributing guide (#2583)
Signed-off-by: jonasrosland <jrosland@vmware.com>
2020-05-27 14:09:07 -07:00
Nolan Brubaker
0effb5d74b Merge pull request #2576 from skriss/fix-2562
bug fix: pass ca cert file to restic stats command on restore
2020-05-26 15:16:28 -04:00
Steve Kriss
98c1a34558 bug fix: pass ca cert file to restic stats command on restore
Signed-off-by: Steve Kriss <krisss@vmware.com>
2020-05-26 10:16:03 -06:00
Steve Kriss
92693ad422 update backup struct with progress updates
Signed-off-by: Steve Kriss <krisss@vmware.com>
2020-05-21 15:40:59 -06:00
Steve Kriss
3f6183b522 add missing kubebuilder tags
Signed-off-by: Steve Kriss <krisss@vmware.com>
2020-05-21 15:30:12 -06:00
Ashish Amarnath
532c6c5dc3 increment restic volumesnapshot count after successful pvb create (#2542)
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-05-20 09:05:23 -07:00
Nolan Brubaker
c1494d6804 Tweak VolumeSnapshotContent describe output (#2534)
Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-05-12 13:58:47 -07:00
Nolan Brubaker
e400be9c8f Include CSI volume snapshot information in velero backup describe (#2448)
* Add download methods for CSI objects

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add support for downloading CSI volume objects

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add new methods to mock.

Remove generated information from file since mockery no longer appears
to work. It isn't maintained anymore and doesn't support go module-based
projects.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add describe command for CSI

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add csi package with helpers

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove duplicate import from server

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove CSI API that will not be used with describe

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add VolumeSnapshotContents output to describe command

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Document NewCSIListOptions function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Document csi package

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove stutter in function name

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fix CI

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fix nil pointer error when not using CSI snapshots

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove unused CSI download request kinds

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add back mocks

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Change persistent volumes to velero-native snapshots

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove unused function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Address review feedback

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add changelog

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Remove unnecessary doc.go

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-05-08 12:42:30 -07:00
Nolan Brubaker
e91dfe3d91 Remove comment that has been turned into an issue (#2500)
Logged comment as issue #2499

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-05-08 09:27:01 -07:00
Nolan Brubaker
de8962ea18 Bug fix: Calculate label using backup name for CSI objects (#2510)
* Use a helper function when querying w/ backup label

Setting or querying for a backup label name should always pass the value
through the GetValidName function. This change passes query uses of the
backup label value through the GetValidName function by introducing 2
new helpers, one for making a Selector, one for making a ListOptions.

It also removes functions returning the same data, but under
unecessarily specific names.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Document using the label.GetValidName function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Update copyright year

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Clarify labels.GetValidName and annotations

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Move functions to pkg/label

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fix function comments

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-05-07 11:56:13 -07:00
Nolan Brubaker
abae81ddc8 Merge pull request #2440 from skriss/backup-progress
report backup progress
2020-05-07 14:51:20 -04:00
Ashish Amarnath
c49afd44bd Use label selector to list VSCs to persist (#2502)
Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-05-06 11:04:34 -07:00
Ashish Amarnath
911a607cf8 Allow feature flags to be passed from install CLI (#2503)
* allow feature flags to be passed from install CLI

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-05-06 10:52:44 -07:00
Ashish Amarnath
577e87d1b8 Sync CSI API objects during backup sync (#2496)
* add changelog

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* Sync CSI API volumesnapshotcontents during backup sync

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-05-06 08:00:05 -06:00
Nolan Brubaker
8671a639c9 Backup entire v1beta1 CRD instead of just changing version string (#2478)
* Switch to backing up v1beta1 CRDs from API server

Instead of simply switching out the APIVersion string on a v1
CustomResourceDefinition object, re-download the object from the API
server entirely to get the correct fields.

This should fix validation errors upon restore.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fix existing tests

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add full example CRDs to automated tests

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Move beta CRD lookup into helper function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add case for preserveUnknownFields CRDs

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add PreserveUnknownFields case and refactor execute

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add older prometheus CRD test cases

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add changelog

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2020-05-05 13:21:28 -07:00
Ashish Amarnath
6b5a084f32 Delete dangling volumesnapshotcontents from velero backups (#2480)
* Delete dangling volumesnapshotcontents from velero backups

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* add changelog

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* handle not found errors from VSC delete

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>

* clean up unit tests

Signed-off-by: Ashish Amarnath <ashisham@vmware.com>
2020-05-04 15:41:27 -04:00