Commit Graph

148 Commits

Author SHA1 Message Date
KubeKween
e054a2f0b3 Make "not restored" message a bit better (#1901)
* Make "not restoring" message a bit better

Signed-off-by: Carlisia <carlisia@vmware.com>

* Tweak error msg

Signed-off-by: Carlisia <carlisia@vmware.com>

* Address feedback

Signed-off-by: Carlisia <carlisia@vmware.com>
2019-10-01 16:47:21 -04:00
Steve Kriss
48792ece1f Updates for org move to vmware-tanzu (#1920)
* update import paths to github.com/vmware-tanzu/...

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update other GH org refs to vmware-tanzu

Signed-off-by: Steve Kriss <krisss@vmware.com>

* site and docs: update GH org to vmware-tanzu

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update travis badge links on docs readmes

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-09-30 17:26:56 -04:00
Steve Kriss
2e849dcb99 Fix for #1888: check item's original namespace, not remapped one, for inclusion/exclusion (#1909)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-09-25 18:01:16 -07:00
Steve Kriss
29f29f9d61 Merge pull request #1828 from carlisia/c-1826-test-bug
Sort mounts in test case to avoid intermittent fail
2019-09-03 12:39:59 -06:00
Carlisia
966222ac0f Sort mounts in test case to avoid intermitent fail
Closes #1826.

Signed-off-by: Carlisia <carlisia@vmware.com>
2019-08-29 17:55:13 -07:00
Steve Kriss
e210626a36 create pkg/archive with functionality extracted from pkg/restore (#1801)
* move pkg/restore's backup extractor to pkg/archive

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-08-28 18:03:01 -07:00
Steve Kriss
60f9898ca0 restore: rename PV when remapping a namespace if PV exists in-cluster (#1779)
* rename PV during restore when cloning a namespace

Signed-off-by: Steve Kriss <krisss@vmware.com>

* rename func and vars, switch to if..else

Signed-off-by: Steve Kriss <krisss@vmware.com>

* make pv renamer func configurable for testing purposes

Signed-off-by: Steve Kriss <krisss@vmware.com>

* add unit test cases

Signed-off-by: Steve Kriss <krisss@vmware.com>

* changelog

Signed-off-by: Steve Kriss <krisss@vmware.com>

* address review feedback

Signed-off-by: Steve Kriss <krisss@vmware.com>

* address review feedback

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-08-27 16:42:38 -07:00
KubeKween
6b66a49a21 Unit tests for restic restore (#1747)
* Add unit tests for PVB restore functionality

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Add tests for restore action

Signed-off-by: Carlisia <carlisiac@vmware.com>

* TestRestoreWithRestic wip

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Fix build

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Mockery

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Cleanup mocks

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Remove unused mock

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Use consistent pattern for test building

Signed-off-by: Carlisia <carlisia@vmware.com>

* Test cleanup

Signed-off-by: Carlisia <carlisia@vmware.com>

* Better godoc

Signed-off-by: Carlisia <carlisia@vmware.com>

* Improve test cases

Signed-off-by: Carlisia <carlisia@vmware.com>

* Fix build

Signed-off-by: Carlisia <carlisia@vmware.com>

* Minor test cleanup

Signed-off-by: Carlisia <carlisia@vmware.com>

* New pvb test input names

Signed-off-by: Carlisia <carlisia@vmware.com>
2019-08-27 15:49:23 -07:00
Steve Kriss
ffa3251efc move contents of pkg/util/test into pkg/test
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-08-07 08:26:00 -06:00
Steve Kriss
80692a8a39 Properly restore PVs with a reclaim policy of Retain and restic backups (#1713)
* reorganize persistent volume restore code for clarity

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-08-06 13:40:35 -07:00
KubeKween
4accb8512a Restore from PodVolumeBackups (#1723)
* Restore from PodVolumeBackups

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Partially address code reviews

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Partially address code reviews #2

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Clean up struct

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Fix log messages

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Fix tests

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Clean up

Signed-off-by: Carlisia <carlisiac@vmware.com>

* Add changelog

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-08-06 13:17:36 -07:00
Nolan Brubaker
a4e70456a1 Add resource limits to restic init container (#1677)
* Add resource limits to restic init container

Fixes #1201

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Start restic restore item action tests

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Get initial tests for restore action working

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add new test case

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Move resource parsing into a shared function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fetch request/limits from plugin's ConfigMap

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Use builders

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Use moved ParseResourceRequirements function

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Move init container building inline

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Drop CPU limit down a bit and clarify error message

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Fix godoc

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

* Add resource requirements to doc

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2019-08-05 13:18:11 -06:00
Steve Kriss
4543258970 Update tests to use object builders in pkg/builder (#1707)
* add pkg/builder with BackupBuilder and ObjectMeta functional options

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-07-31 07:46:48 -07:00
KubeKween
3b9af8c654 Store PodVolumeBackups in obj storage & use as source of truth (#1577)
* Store PodVolumeBackups in object storage

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-07-24 15:51:20 -04:00
Steve Kriss
1bb167ef90 add restore item action to change PVC/PV storage class name (#1621)
* add restore item action to change PVC/PV storage class name

Signed-off-by: Steve Kriss <krisss@vmware.com>

* code review

Signed-off-by: Steve Kriss <krisss@vmware.com>

* change existing plugin names to lowercase/hyphenated

Signed-off-by: Steve Kriss <krisss@vmware.com>

* add validation for new storage class name

Signed-off-by: Steve Kriss <krisss@vmware.com>

* add test cases

Signed-off-by: Steve Kriss <krisss@vmware.com>

* changelog

Signed-off-by: Steve Kriss <krisss@vmware.com>

* fix imports

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update plugin names to be more consistent

Signed-off-by: Steve Kriss <krisss@vmware.com>

* update unit tests to use pkg/test object constructors

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-07-15 11:19:38 -07:00
Nolan Brubaker
63964fc6f9 Append "/mount" to directory for CSI volumes (#1615)
CSI volumes are mounted one level deeper than "native" kubernetes
volumes, and this needs to be appended for proper restic support.

Fixes #1313.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2019-07-10 15:16:21 -07:00
Steve Kriss
d1025f7547 rename restore_new_test.go to restore_test.go
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-07-10 09:59:27 -06:00
Steve Kriss
678c02c560 move contents of restore_test.go into other files and delete it
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-07-10 09:59:01 -06:00
Steve Kriss
567802299b move PV/PVC setup helpers into pkg/test
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-07-03 08:50:11 -06:00
Steve Kriss
adb93c33b1 migrate restore PV tests
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-07-03 08:50:09 -06:00
Steve Kriss
db393ec199 properly filter additional items returned from restore item actions
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-28 08:59:23 -06:00
Steve Kriss
022099a62e Refactor pkg/restore tests (part 3) (#1611)
* add modifying restore item action tests

Signed-off-by: Steve Kriss <krisss@vmware.com>

* add restore item action additional items tests

Signed-off-by: Steve Kriss <krisss@vmware.com>

* migrate ShouldRestore test

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-27 15:43:59 -07:00
Steve Kriss
75a96dfa92 fix bugs in finding applicable restore item actions
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-27 13:06:16 -06:00
Steve Kriss
81c2adc059 Refactor pkg/restore tests (part 2) (#1606)
* update TestPrioritizeResources to use real discovery helper

Signed-off-by: Steve Kriss <krisss@vmware.com>

* migrate invalid tarball contents tests

Signed-off-by: Steve Kriss <krisss@vmware.com>

* migrate item restore tests

Signed-off-by: Steve Kriss <krisss@vmware.com>

* migrate restore item action tests

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-27 11:57:47 -07:00
Steve Kriss
0089fa4d93 migrate restore resource priorities test
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-26 10:24:02 -06:00
Steve Kriss
5d8ba1b90d migrate namespace mapping tests
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-24 16:57:30 -06:00
Steve Kriss
1a339f06ac migrate restore resource filtering tests
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-24 15:53:15 -06:00
Nolan Brubaker
67512a3808 Merge pull request #1595 from skriss/restore-test-refactor-1
initial refactoring for pkg/restore tests
2019-06-24 17:08:51 -04:00
Nolan Brubaker
37c7b618ad Merge pull request #1598 from skriss/rm-has-controller-owner
remove unused hasControllerOwner func and tests
2019-06-24 16:51:28 -04:00
Steve Kriss
19052994ed replace TestBackup with pkg/backup.Builder (#1593)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-21 17:08:08 -04:00
Steve Kriss
0d326a3903 remove unused hasControllerOwner func and tests
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-21 11:44:13 -06:00
Steve Kriss
d421fcd85c initial structure for refactored pkg/restore tests
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-06-21 09:15:18 -06:00
Anshul Chandra
4e12b08953 Validate restore name label length
Velero should handle cases when the label length exceeds 63 characters.

- if the length of the backup/restore name is <= 63 characters, use it as the value of the label
- if it's > 63 characters, take the SHA256 hash of the name. the value of
  the label will be the first 57 characters of the backup/restore name
  plus the first six characters of the SHA256 hash.

Fixes heptio#1021

Signed-off-by: Anshul Chandra <anshulc@vmware.com>
2019-05-01 10:25:11 -07:00
Steve Kriss
3155bb159f remove VolumeBackupInfo from API pkg (#1388)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-04-24 13:16:03 -07:00
Steve Kriss
5bc6695109 move RestoreResult from api to pkg/restore, rename to Result
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-04-16 12:57:02 -06:00
Steve Kriss
c59544cb79 remove backup.status.volumeBackups and all related code
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-04-15 10:17:40 -06:00
Steve Kriss
892673816b remove legacy restore label
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-04-15 10:17:03 -06:00
Steve Kriss
77e648eafa remove Ark field from RestoreResult
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-04-15 10:17:02 -06:00
sseago
a519547efc Adds support for allowing a RestoreItemAction to skip item restore (#1336)
* Adds support for allowing a RestoreItemAction to skip item restore

This allows a RestoreItemAction plugin to signal to velero that
the returned item should be skipped rather than restored to the
cluster.

To support this, a boolean SkipRestore attribute is added to
RestoreItemActionExecuteOutput. If restore.restoreResource finds
this set to true, any remaining actions on this item are skipped,
and restore on this item is skipped. Execution continues with
the next item of this resource type.

To signal this for a particular item, the RestoreItemAction's
Execute method should call WithoutRestore() on the
RestoreItemActionExecuteOutput before returning it.

Signed-off-by: Scott Seago <sseago@redhat.com>

* Autogenerated code to support SkipRestore

Signed-off-by: Scott Seago <sseago@redhat.com>

* Added changelog for #1336

Signed-off-by: Scott Seago <sseago@redhat.com>
2019-04-04 13:39:54 -06:00
Steve Kriss
7298a4eda0 allow restic restore helper image to be specified via ConfigMap (#1311)
* allow restic restore helper image to be specified via ConfigMap

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-29 17:11:34 -04:00
Steve Kriss
a696cd09f2 remove Warning from restore item action output (#1318)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-28 13:08:37 -07:00
Steve Kriss
9b635c0e14 add additionalItems to restore item actions (#1304)
* add additionalItems to restore item actions

Signed-off-by: Steve Kriss <krisss@vmware.com>
Co-authored-by: Andy Goldstein <goldsteina@vmware.com>
2019-03-28 12:21:56 -07:00
Steve Kriss
bb9c3f6a1a rename BlockStore to VolumeSnapshotter
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-27 14:55:28 -06:00
Steve Kriss
a111eed2af update license headers to Velero contributors (#1302)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-20 12:32:48 -07:00
KubeKween
73514a003b Move plugin interfaces to same package (#1264)
* Move plugin interfaces to same package

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-03-14 16:35:06 -04:00
Nolan Brubaker
394548afcd Merge pull request #1254 from skriss/remove-wait-for-pv
remove restore code that waits for a PV to become Available
2019-03-11 13:20:59 -04:00
KubeKween
4ee41a13a0 Merge pull request #1261 from asaf-erlich/patch-1
Update ark restore to not open every single file open during extracti…
2019-03-08 14:59:09 -08:00
asaf-erlich
4041044a93 Update ark restore to not open every single file open during extraction of the data
Original error was:

```
ark -n <redacted> restore logs <redacted>
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=backups.ark.heptio.com logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=restores.ark.heptio.com logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Starting restore of backup backup/<redacted>" logSource="pkg/restore/restore.go:342"
time="2019-03-06T18:31:06Z" level=info msg="error unzipping and extracting: open /tmp/604421455/resources/rolebindings.rbac.authorization.k8s.io/namespaces/<redacted>/<redacted>: too many open files" logSource="pkg/restore/restore.go:346"
```

Downloading the directory from s3 and untarring it I found 1036 files. The ulimit -n output says 1024. This is our team's best guess at a root cause. But the code fixed in the PR definitely is holding all the files open until the method closes: https://blog.learngoprogramming.com/gotchas-of-defer-in-go-1-8d070894cb01

Please note my go code abilities are not great and I did not test this. I just edited the file in github. All I did was remove the defer and put fileClose after the copy is done. Theoretically this should only hold one file open at a time now. Let me know if you want me to do any further steps.

Thank you,
-Asaf

Signed-off-by: Asaf Erlich <aerlich@groupon.com>
2019-03-07 11:23:35 -05:00
Michal Wieczorek
1354e2b6ff Add original item to restore plugin interface
Signed-off-by: Michal Wieczorek <wieczorek-michal@wp.pl>
2019-03-05 17:09:42 -08:00
Steve Kriss
e29aa74a23 remove restore code that waits for a PV to become Available
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-05 17:04:52 -07:00