Commit Graph

77 Commits

Author SHA1 Message Date
Steve Kriss
cb321db21f azure: refactor to not use helpers/ pkg, validate all env/config inputs
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss
9d7ea7483c azure: support different RGs/storage accounts per backup location
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss
cd4e9f5336 azure: fix for breaking change in blob.GetSASURI
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss
0e94fa37f9 update sync controller for backup locations
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-08-28 13:19:21 -07:00
Steve Kriss
1c26fbde32 remove SnapshotService, replace with direct BlockStore usage
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-31 08:58:11 -07:00
Andy Goldstein
130512187a Refactor plugin management
Refactor plugin management:
- support multiple plugins per executable
- support restarting a plugin process in the event it terminates
- simplify plugin lifecycle management by using separate managers for
  each scope (server vs backup vs restore)

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-07-31 08:34:57 -07:00
Dave Parfitt
06d6665abb check s3URL scheme upon AWS ObjectStore Init()
Signed-off-by: Dave Parfitt <diparfitt@gmail.com>
2018-07-24 14:01:05 -04:00
Steve Kriss
85a61b8e8d return nil error if 404 encountered when deleting snapshots
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-20 11:42:34 -07:00
Mike Arpaia
3efe677007 Update Ark library code to work with Kubernetes 1.11
Signed-off-by: Mike Arpaia <mike@arpaia.co>
2018-07-16 20:46:20 -06:00
Steve Kriss
5ad21854f7 restic: if S3, get bucket's region up-front
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-21 17:20:53 -07:00
Steve Kriss
170034787d Merge pull request #513 from ncdc/510-follow-ups
Fix copyright date, add happy path test case
2018-05-23 08:54:22 -07:00
Andy Goldstein
18e2401e79 Merge pull request #455 from skriss/remove-aws-region
AWS: remove region requirement from config for backupStorageProvider
2018-05-23 11:02:57 -04:00
Andy Goldstein
849297e623 Fix copyright date, add happy path test case
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-05-23 09:59:32 -04:00
Nolan Brubaker
09bbe072cd Check errors when closing GCP objects
Writing to GCP's object store is any async operation, so errors need to
be checked both on write and close calls, since errors like permission
violations aren't reported until a close.

Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-22 17:38:49 -04:00
Steve Kriss
b029860b46 AWS: remove region requirement from config for backupStorageProvider
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-22 11:59:53 -07:00
Steve Kriss
e81de2491f remove duplicate github.com/satori/go.uuid dependency
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 10:02:59 -07:00
Steve Kriss
72d7e5e01d azure: pass full snapshot name when restoring disk
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-23 12:51:05 -07:00
Steve Kriss
a466208975 add backwards-compatibility for azure snapshot ID
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-23 11:34:37 -07:00
Steve Kriss
0d2b49acce get backup name from restore spec when handling restore DownloadRequests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-09 10:50:10 -07:00
Andy Goldstein
3172e907a1 Merge pull request #341 from skriss/snapshot-tags
Add tags to snapshots, restored volumes
2018-03-14 12:50:40 -04:00
Andy Goldstein
07fcc927b7 Merge pull request #359 from skriss/reduce-azure-polling-interval
Azure: reduce client polling interval from default 60s to 5s
2018-03-14 12:17:54 -04:00
Steve Kriss
45cee7d222 Azure: reduce client polling interval from default 60s to 5s
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-14 09:07:20 -07:00
Steve Kriss
b2b16b3c85 Azure: fix bug preventing snapshot deletion
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 17:03:14 -07:00
Steve Kriss
0388845bca AWS: tag snapshots during create
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss
a05ae1a7cf add useful Ark tags to snapshots
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss
df985bca74 Azure: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 12:17:50 -07:00
Steve Kriss
cab904570f GCP: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 11:52:09 -07:00
Steve Kriss
9673e9d158 AWS: copy tags from volume to snapshot, and snapshot to volume
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 11:52:09 -07:00
Steve Kriss
409f17361d Azure: store snapshot URI to support cross-resource group restores
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-13 09:18:20 -07:00
Andy Goldstein
331e0c28cc Remove Azure location requirement
Instead of requiring the Ark admin to specify a "location" in the azure
persistentVolumeProvider config (meaning only a single location is
supported), get info about the disk (for its location) when creating a
snapshot, and get info about the snapshot (for its location) when
creating a disk from a snapshot.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-03-05 15:20:58 -05:00
Steve Kriss
cbff5ef53b extract GCP project ID from creds file, remove from config
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-03-02 09:15:12 -08:00
Steve Kriss
af81f589f6 update GCP block store to use v1 library, and tag snapshots on create
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-08 09:23:32 -08:00
Steve Kriss
b53ee44176 update GCP object store to use new/recommended lib
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-05 14:21:13 -08:00
Steve Kriss
656428d0b9 Merge pull request #250 from ncdc/backup-controller-do-as-much-as-possible
BackupController: do as much as possible
2018-01-03 12:10:51 -08:00
Steve Kriss
1503796419 make language and casing consistent across copyrights
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-01-02 10:51:49 -08:00
Andy Goldstein
1e581f1ead BackupController: do as much as possible
When running a backup, try to do as much as possible, collecting errors
along the way, and return an aggregate at the end. This way, if a backup
fails for most reasons, we'll be able to upload the backup log file to
object storage, which wasn't happening before.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-12-20 14:49:27 -05:00
Andy Goldstein
1210cb36e1 Merge pull request #253 from skriss/update-license
Update all license headers
2017-12-19 19:26:54 -05:00
Steve Kriss
3641c2c043 update all license headers
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-15 13:38:12 -08:00
Steve Kriss
56c9d68137 switch to logrus.FieldLogger and arktest.NewLogger() everywhere
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-12-13 09:38:00 -08:00
Steve Kriss
cbcd15d603 add/update documentation for plugins
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-29 14:24:23 -08:00
Steve Kriss
038fa39451 remove ListSnapshots and GetAllSnapshots
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-29 13:59:29 -08:00
Andy Goldstein
c700455272 Support custom volume snapshots & restores
The main Ark code was hard-coding specific support for AWS, GCE, and
Azure volume snapshots and restores, and anything else was considered
unsupported.

Add GetVolumeID and SetVolumeID to the BlockStore interface, to allow
block store plugins to handle volume snapshots and restores.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-29 13:19:40 -05:00
Steve Kriss
24ce316788 switch built-in cloud providers to run as plugins
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-14 09:47:36 -08:00
Steve Kriss
21e2019540 rename Block/ObjectStoreAdapter -> Block/ObjectStore
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-08 16:58:47 -08:00
Andy Goldstein
b2d80471ac Move restore warnings/errors to object storage
If you have a large number of warnings and/or errors, the restore
object's size can exceed the maximum allowed by etcd. Move them to
object storage, and add a new describe command to fetch and display them
on the fly.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-11-02 15:53:59 -04:00
Steve Kriss
e7703d88ec switch UploadBackup, UploadRestoreLog, PutObject to take io.Reader
Signed-off-by: Steve Kriss <steve@heptio.com>
2017-11-01 15:48:13 -07:00
Andy Goldstein
c1bc52eb65 Update gofmt
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:31:03 -04:00
Andy Goldstein
15fe87aea3 Fix clientset imports
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2017-10-30 13:31:03 -04:00
ashish-amarnath
6dc81b42f5 Merge branch 'master' into support-multi-az-pvs
Signed-off-by: Ashish Amarnath <ashish.amarnath@gmail.com>
2017-10-06 23:24:41 -07:00
Ashish Amarnath
9fc9dbb413 Preserve PV's AZ info when snapshotting and restoring PVs.
- Read PV's AZ info from fault-domain label of the PV object for snapshotting.
- Store PV's AZ info in the VolumeInfo.
- Add tests for reading the label from the PV object.
- Remove availability zone validation in AWS and GCP BlockStorageAdaptor.
- Add volumeAZ as a parameter to methods in the BlockStorageAdapter interface.
- Get AZ from VolumeInfo when restoring PV snapshot.
- Remove references to PV availability zone in docs.

Signed-off-by: Ashish Amarnath <ashish.amarnath@gmail.com>
2017-10-06 23:24:41 -07:00