Commit Graph

94 Commits

Author SHA1 Message Date
Nolan Brubaker
7cebfe2df0 Add restore attempt and success/failure counters
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-07-25 11:22:43 -04:00
Nolan Brubaker
b0af81e780 Merge pull request #689 from lemaral/init-schedule-metrics
Init schedule Prometheus metrics
2018-07-23 16:16:26 -04:00
Alex Lemaresquier
450fa72fbb Initialize schedule Prometheus metrics to have them created beforehand (see https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics)
Signed-off-by: Alex Lemaresquier <alex+github@lemaresquier.org>
2018-07-20 23:30:24 +02:00
Steve Kriss
78cbdf95f3 delete old deletion requests for backup when processing a new one
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-20 11:42:42 -07:00
Nolan Brubaker
1e2b141e5d Merge pull request #443 from skriss/restore-from-schedule
Add --from-schedule flag to `ark restore create`
2018-07-11 13:50:08 -04:00
Steve Kriss
683f7afc0d switch to using .status.startTimestamp for sorting backups
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-11 10:35:55 -07:00
Nolan Brubaker
b71a37dbfc Record backup completion time before uploading
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-07-11 13:21:29 -04:00
Nolan Brubaker
fae00a7622 Merge pull request #659 from carlisia/cc-backupsync
Skip backup sync if it already exists on Kubernetes
2018-07-10 12:50:20 -04:00
Carlisia Pinto
5b89f7b6db Skip backup sync if it already exists in k8s
Signed-off-by: Carlisia Pinto <carlisia@grokkingtech.io>
2018-07-09 21:55:24 -07:00
Steve Kriss
c6050845a0 restore controller: switch to 'c' for receiver name
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-09 15:07:38 -07:00
Steve Kriss
706ae07d0d enable a schedule to be provided as the source for a restore
- ScheduleName is added as an API field to the Restore object
- Restore controller validates that exactly one of BackupName
  or ScheduleName has been provided
- If ScheduleName is provided, Restore controller populates
  BackupName with the name of the most recent successful backup
  created from the schedule
- --from-schedule flag is added to `ark restore create` CLI cmd

Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-09 15:07:38 -07:00
Steve Kriss
c18decc89b pkg/restic unit tests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-07-05 11:44:17 -07:00
Andy Goldstein
fe286ff564 Don't restore backups or restores
Add backups and restores the list of non restorable resources. Backups,
if applicable, are synced from object storage by the backup sync
controller. Restores are specific to a cluster and don't have value
moving across clusters.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-06-28 16:58:09 -04:00
Andy Goldstein
539de6d361 Merge pull request #564 from nrb/backup-timing
Record backup start and completion times, add timing metrics
2018-06-28 11:37:39 -04:00
Nolan Brubaker
96b72acb2d Record backup start and completion times
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-06-28 11:18:14 -04:00
Steve Kriss
22e8f23e2c replace ark restic repo init cmd with automatic initialization of repos
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-27 13:50:37 -07:00
Steve Kriss
a697ad164e refine what gets enqueued in PVB/PVR controllers, and log better
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-27 13:31:11 -07:00
Steve Kriss
f2072e5868 use repo identifier from ResticRepo CR when backing up/restoring
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 10:42:53 -07:00
Steve Kriss
3481618324 move runCommand to pkg/util/exec and use in restic repo mgr
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 09:21:43 -07:00
Steve Kriss
65ed8da4b7 add ResticRepository CRD and move repo-level actions to a controller
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-20 09:21:43 -07:00
Steve Kriss
6fb11b8087 get rid of restore staging dir by backing up/restoring within volume dir
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-14 13:49:27 -07:00
Steve Kriss
dc273e3bed remove existing .ark dir and contents during restic restores
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-13 16:52:06 -07:00
Andy Goldstein
9245e9d5dc Merge pull request #547 from skriss/restic-rsync
use rsync to copy restored data from staging dir to volume dir
2018-06-13 16:56:42 -04:00
Steve Kriss
f7a42f378f use rsync to copy restored data from staging dir to volume dir
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-13 13:41:55 -07:00
Andy Goldstein
d1e3688468 Merge pull request #531 from ashish-amarnath/add-backup-metrics
Add metrics package to publish metrics
2018-06-13 16:33:30 -04:00
Ashish Amarnath
83658e891e Add a metrics package to add and expose metrics
* add a metrics package to handle metric registration and publishing
* add a metricsAddress field to the server struct
* make metrics a part of the server
* start a metrics endpoint as part of starting the controllers
* instrument backup_controller to report metrics
* update cli-reference docs
* update example deployments with prometheus annotations
* update 'pkg/install' tooling with prometheus annotations

Signed-off-by: Ashish Amarnath <ashish.amarnath@gmail.com>
2018-06-13 13:17:08 -07:00
Steve Kriss
2c6fc5bd90 don't fail the restore if cleanup of the staging directory post-restore fails
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 09:48:26 -07:00
Steve Kriss
6897c2f901 move dotfiles from staging to volume dir on restore
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 09:31:03 -07:00
Steve Kriss
e354b1c130 replace complete-restore.sh with go code
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-11 09:31:03 -07:00
Steve Kriss
50d4084fac add restic integration for doing pod volume backups/restores
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-06-06 09:48:10 -07:00
Andy Goldstein
2fde1f5fc1 Merge pull request #500 from nrb/fix-497
Exit when failing to update a backup's phase
2018-05-17 14:23:41 -04:00
Andy Goldstein
f288902e3e Merge pull request #490 from skriss/jsonpatch
Use JSON merge patches everywhere
2018-05-17 13:02:47 -04:00
Nolan Brubaker
7eac6675e8 Exit when failing to update a backup's phase
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-05-17 12:47:38 -04:00
Steve Kriss
6d6f734bc9 use json merge patches
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-14 14:34:24 -07:00
Steve Kriss
20f56e9868 cleanup plugin subprocesses on server exit
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-11 09:48:46 -07:00
Steve Kriss
51928e9177 use typed structs for decoding patch JSON in unit tests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 09:22:20 -07:00
Steve Kriss
6754955bcd when syncing backups, set their namespace to current cluster's Ark ns
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-05-07 09:12:51 -07:00
Nolan Brubaker
300a010959 Remove finalizers from backups before syncing
Signed-off-by: Nolan Brubaker <nolan@heptio.com>
2018-04-18 17:32:39 -04:00
Andy Goldstein
258f3e011e Merge pull request #409 from skriss/fix-398
get backup name from restore spec when handling restore DownloadRequests
2018-04-10 13:29:07 -04:00
Steve Kriss
0d2b49acce get backup name from restore spec when handling restore DownloadRequests
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-09 10:50:10 -07:00
Steve Kriss
5bb47d2b7a extract a helper for comparing slices of Actions reliably
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-04-09 09:39:33 -07:00
Andy Goldstein
3f2e222ae4 Don't allow deletion of in-progress backups
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-06 13:08:39 -04:00
Andy Goldstein
644a75e3c1 Set DeleteBackupRequest labels if missing
When the BackupDeletionController processes a request, set the request's
backup-name and backup-uid labels if they aren't currently set.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 15:38:44 -04:00
Andy Goldstein
ef57a44827 Require DeleteBackupRequest.Spec.BackupName
Make sure a DeleteBackupRequest has its Spec.BackupName filled in. If
not, record an error in the status and mark the request as processed.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 14:53:10 -04:00
Andy Goldstein
4328b67f93 backup delete/describe enhancements
Always request DeleteBackupRequests for a given backup so we can show
failed deletion attempts if you try to delete a backup that has PV
snapshots when Ark doesn't have a persistentVolumeProvider configured.

When creating a DeleteBackupRequest, include a label for the UID so we
can match based on name and UID when associated DeleteBackupRequests
with a given backup.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 14:21:45 -04:00
Andy Goldstein
1ae492da6c Expire processed deleted backup requests > 24hr
Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 11:19:11 -04:00
Andy Goldstein
74f60b1ee1 Switch backup finalizer to DeleteBackupRequest
We ran into a lot of problems using a finalizer on the backup to allow
the Ark server to clean up all associated backup data when deleting a
backup.

Users also found it less than desirable that deleting the heptio-ark
namespace resulted in all the backup data being deleted.

This removes the finalizer and replaces it with an explicit
DeleteBackupRequest that is created as a means of requesting the
deletion of a backup and all its associated data. This is what `ark
backup delete` does.

If you use kubectl to delete a backup or to delete the heptio-ark
namespace, this no longer deletes associated backups. Additionally, as
long as the heptio-ark namespace still exists, the Ark server's
BackupSyncController will continually sync backups into the heptio-ark
namespace from object storage.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-04-05 11:16:15 -04:00
Sanket Patel
7257a75f82 Add events to nonRestorableResources and cohabitatingResources
Fixes #367
Fixes #368

Signed-off-by: Sanket Patel <sanketpatel.301090@gmail.com>
2018-03-12 19:55:32 -05:00
Steve Kriss
509431909a add an AddFunc to handle finalizers in GC for initial lists/resyncs
Signed-off-by: Steve Kriss <steve@heptio.com>
2018-02-28 16:56:23 -08:00
Andy Goldstein
d24fb232cc Allow forced backup deletion
Add --force and --confirm to `ark backup delete` to support forced
backup deletion. This forcibly removes the Ark GC finalizer (if it's
present) from a backup and will orphan any resources associated with the
backup, such as backup tarballs in object storage, persistent volume
snapshots, and restores for the backup.

If a backup has a deletion timestamp, display `Deleting` in `ark backup
describe` and `ark backup get`.

Signed-off-by: Andy Goldstein <andy.goldstein@gmail.com>
2018-02-26 16:25:22 -05:00