Commit Graph

609 Commits

Author SHA1 Message Date
Rohan Vora
4beb8aab3c Set default backup TTL
Set default backup TTL to 30 days when TTL
is not provided in the backup yaml configuration.

Updates #138

Signed-off-by: Rohan Vora <vorar@vmware.com>
2019-04-09 14:13:29 -07:00
sseago
a519547efc Adds support for allowing a RestoreItemAction to skip item restore (#1336)
* Adds support for allowing a RestoreItemAction to skip item restore

This allows a RestoreItemAction plugin to signal to velero that
the returned item should be skipped rather than restored to the
cluster.

To support this, a boolean SkipRestore attribute is added to
RestoreItemActionExecuteOutput. If restore.restoreResource finds
this set to true, any remaining actions on this item are skipped,
and restore on this item is skipped. Execution continues with
the next item of this resource type.

To signal this for a particular item, the RestoreItemAction's
Execute method should call WithoutRestore() on the
RestoreItemActionExecuteOutput before returning it.

Signed-off-by: Scott Seago <sseago@redhat.com>

* Autogenerated code to support SkipRestore

Signed-off-by: Scott Seago <sseago@redhat.com>

* Added changelog for #1336

Signed-off-by: Scott Seago <sseago@redhat.com>
2019-04-04 13:39:54 -06:00
Fábio Franco Uechi
0167539a14 add new counter metrics for backup deletion (#1280)
* compute backup deletion metrics (attempt, success, fail)

Signed-off-by: fabito <fuechi@ciandt.com>
2019-04-04 14:25:59 -04:00
Steve Kriss
2e3f00f64d Merge pull request #1340 from carlisia/c-copy
Fix copyright
2019-04-01 15:49:06 -06:00
Nolan Brubaker
c3a933d3e3 Merge pull request #1338 from skriss/validate-config-keys
objectstores/volumesnapshotters: check for invalid keys in config
2019-04-01 15:11:15 -04:00
Nolan Brubaker
bbd28a9fb9 Merge pull request #1337 from skriss/logs-cmd-validation
logs commands: validate item exists & is finished processing
2019-04-01 15:10:52 -04:00
Carlisia
23b1098950 Fix copyright
Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-04-01 12:06:14 -07:00
Steve Kriss
1d3d66aa77 logs commands: validate backup/restore exists & is finished processing
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-04-01 12:39:02 -06:00
Fábio Franco Uechi
40c7fbce09 Gracefully handle failed API groups from the discovery API (#1293)
* log details and continue executing if error of type ErrGroupDiscoveryFailed is returned by discovery API

Signed-off-by: fabito <fuechi@ciandt.com>
2019-04-01 14:27:37 -04:00
Steve Kriss
6bf29e17aa objectstores/volumesnapshotters: check for invalid keys in config
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-04-01 08:15:24 -06:00
Steve Kriss
7298a4eda0 allow restic restore helper image to be specified via ConfigMap (#1311)
* allow restic restore helper image to be specified via ConfigMap

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-29 17:11:34 -04:00
Steve Kriss
2a36cdcbf6 set backup start timestamp before patching to inprogress (#1330)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-29 13:33:50 -07:00
Steve Kriss
dcee310745 improve handling of custom S3 URLs (#1331)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-29 13:32:43 -07:00
Steve Kriss
a696cd09f2 remove Warning from restore item action output (#1318)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-28 13:08:37 -07:00
Steve Kriss
be42ea782d turn down log levels in plugin server to DEBUG (#1325)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-28 12:44:26 -07:00
Steve Kriss
9b635c0e14 add additionalItems to restore item actions (#1304)
* add additionalItems to restore item actions

Signed-off-by: Steve Kriss <krisss@vmware.com>
Co-authored-by: Andy Goldstein <goldsteina@vmware.com>
2019-03-28 12:21:56 -07:00
KubeKween
477e42286c Bump plugin client version (#1319)
* Bump plugin client version

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-03-28 11:34:43 -04:00
Nolan Brubaker
21f3169ad3 Merge pull request #1321 from skriss/rename-block-store
rename BlockStore to VolumeSnapshotter
2019-03-28 11:33:06 -04:00
Steve Kriss
86293b68b3 don't wrap io.EOF errors during gRPC streaming
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-27 16:22:28 -06:00
Steve Kriss
bb9c3f6a1a rename BlockStore to VolumeSnapshotter
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-27 14:55:28 -06:00
Nolan Brubaker
3f2c28f6bb Merge pull request #1301 from skriss/plugins-error-location
log error locations from plugin logger and don't overwrite in client
2019-03-27 11:21:52 -04:00
Steve Kriss
7b0d8217de send plugin error stack traces over gRPC and log error locations
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-27 08:34:03 -06:00
Matt Stump
f8baf4f4f0 Fix for #1312, use describe to determine if AWS EBS snapshot is encrypted and explicitly pass that value in EC2 CreateVolume call. (#1316)
Signed-off-by: Matt Stump <mstump@vorstella.com>
2019-03-26 17:30:27 -07:00
Steve Kriss
b1c0e9c49b update plugins to work with updated go-plugin (#1308)
* update plugins to work with updated go-plugin

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-21 12:32:18 -07:00
pei0804
7af9f8d74e compile only once for regexp.MustCompile
Signed-off-by: pei0804 <peeeei0804@gmail.com>
2019-03-21 21:51:48 +09:00
Steve Kriss
ff2db31b32 log error locations from plugin logger and don't overwrite in client
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-20 16:13:37 -06:00
Steve Kriss
a111eed2af update license headers to Velero contributors (#1302)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-20 12:32:48 -07:00
Carlisia
a71e43b2b7 Split velero plugin client into its own package
Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-03-19 16:05:37 -07:00
Steve Kriss
1eac10ca9f Merge pull request #1288 from carlisia/c-plugins-II
Split plugin framework into its own package
2019-03-19 16:44:58 -06:00
Carlisia
7dfe58d37f Split plugin framework into its own package
Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-03-19 15:36:31 -07:00
Steve Kriss
7d66fc31bd pkg/restic: check for & skip hostPath PVC/PVs
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-19 08:57:47 -06:00
Steve Kriss
183bea369d make resticrepositories non-restorable resources (#1296)
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-18 11:38:37 -07:00
KubeKween
73514a003b Move plugin interfaces to same package (#1264)
* Move plugin interfaces to same package

Signed-off-by: Carlisia <carlisiac@vmware.com>
2019-03-14 16:35:06 -04:00
Steve Kriss
7674332313 pass --log-level to plugins (#1278)
Plumb the log level through to plugin processes


Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-14 10:53:36 -07:00
Steve Kriss
409116fce8 add basic plugin panic handlers (#1270)
* add server-side panic handlers to all plugin methods

Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-13 14:07:52 -04:00
Nolan Brubaker
503b112638 Add location resources and tests (#1277)
Add locations and tests to install package

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>
2019-03-13 11:23:00 -06:00
tsturzl
89ca2571f3 AWS zone on volume IDs
Signed-off-by: Travis Sturzl<travis@metismachine.com>
2019-03-12 13:04:33 -06:00
Nolan Brubaker
394548afcd Merge pull request #1254 from skriss/remove-wait-for-pv
remove restore code that waits for a PV to become Available
2019-03-11 13:20:59 -04:00
KubeKween
4ee41a13a0 Merge pull request #1261 from asaf-erlich/patch-1
Update ark restore to not open every single file open during extracti…
2019-03-08 14:59:09 -08:00
asaf-erlich
4041044a93 Update ark restore to not open every single file open during extraction of the data
Original error was:

```
ark -n <redacted> restore logs <redacted>
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=nodes logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=events logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=events.events.k8s.io logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=backups.ark.heptio.com logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Not including resource" groupResource=restores.ark.heptio.com logSource="pkg/restore/restore.go:124"
time="2019-03-06T18:31:06Z" level=info msg="Starting restore of backup backup/<redacted>" logSource="pkg/restore/restore.go:342"
time="2019-03-06T18:31:06Z" level=info msg="error unzipping and extracting: open /tmp/604421455/resources/rolebindings.rbac.authorization.k8s.io/namespaces/<redacted>/<redacted>: too many open files" logSource="pkg/restore/restore.go:346"
```

Downloading the directory from s3 and untarring it I found 1036 files. The ulimit -n output says 1024. This is our team's best guess at a root cause. But the code fixed in the PR definitely is holding all the files open until the method closes: https://blog.learngoprogramming.com/gotchas-of-defer-in-go-1-8d070894cb01

Please note my go code abilities are not great and I did not test this. I just edited the file in github. All I did was remove the defer and put fileClose after the copy is done. Theoretically this should only hold one file open at a time now. Let me know if you want me to do any further steps.

Thank you,
-Asaf

Signed-off-by: Asaf Erlich <aerlich@groupon.com>
2019-03-07 11:23:35 -05:00
Michal Wieczorek
1354e2b6ff Add original item to restore plugin interface
Signed-off-by: Michal Wieczorek <wieczorek-michal@wp.pl>
2019-03-05 17:09:42 -08:00
Steve Kriss
e29aa74a23 remove restore code that waits for a PV to become Available
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-05 17:04:52 -07:00
Nolan Brubaker
ce3f43e876 Merge pull request #1251 from skriss/backup-extractor
move backup extraction logic to its own type
2019-03-05 16:37:48 -05:00
Nolan Brubaker
5912fe66e5 Merge pull request #1250 from skriss/extract-pv-restorer
move pvRestorer and tests to their own files
2019-03-05 16:37:32 -05:00
DheerajSShetty
1b031f0cc4 Improve describe output
* Move Phase to right under Metadata(name/namespace/label/annotations)
 * Move Validation errors: section right after Phase: section and only
   show it if the item has a phase of FailedValidation
 * For restores move Warnings and Errors under Validation errors. Do not
   show Warnings or Errors if there are none.

Signed-off-by: DheerajSShetty <dheerajs@vmware.com>

Fixes #987
2019-03-04 13:21:18 -08:00
Steve Kriss
88e6a740f2 move pvRestorer and tests to their own files
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-01 15:07:25 -07:00
Steve Kriss
0fec56f488 move backup extraction logic to its own type
Signed-off-by: Steve Kriss <krisss@vmware.com>
2019-03-01 15:05:58 -07:00
KubeKween
e21940bee1 Merge pull request #1231 from skriss/k8s-1.12-deps
update kubernetes and azure dependencies to 1.12
2019-02-28 15:09:06 -08:00
Nolan Brubaker
fcf21813a5 Merge pull request #1246 from skriss/preserve-storageclass
when restoring a PV, don't remove its spec.storageClassName
2019-02-28 18:02:20 -05:00
KubeKween
65f3926caa Merge pull request #1146 from skriss/replace-map-utils-final
replace ark's map_utils.go with structured types and apimachinery's unstructured helpers
2019-02-28 14:37:07 -08:00