Wait for CRDs to be ready before restoring CRs (#1937)

* Wait for CRDs to be available and ready

When restoring CRDs, we should wait for the definition to be ready and
available before moving on to restoring specific CRs.

While the CRDs are often ready by the time we get to restoring a CR,
there is a race condition where the CRD isn't ready.

This change waits on each CRD at restore time.

Signed-off-by: Nolan Brubaker <brubakern@vmware.com>

pkg/cmd/server/server.go

@@ -440,7 +440,9 @@ func (s *server) validateBackupStorageLocations() error {
 	return nil
 }
 
-// - Namespaces go first because all namespaced resources depend on them.
+// - Custom Resource Definitions come before Custom Resource so that they can be
+//   restored with their corresponding CRD.
+// - Namespaces go second because all namespaced resources depend on them.
 // - Storage Classes are needed to create PVs and PVCs correctly.
 // - PVs go before PVCs because PVCs depend on them.
 // - PVCs go before pods or controllers so they can be mounted as volumes.
@@ -452,9 +454,8 @@ func (s *server) validateBackupStorageLocations() error {
 //	 have restic restores run before controllers adopt the pods.
 // - Replica sets go before deployments/other controllers so they can be explicitly
 //	 restored and be adopted by controllers.
-// - Custom Resource Definitions come before Custom Resource so that they can be
-//   restored with their corresponding CRD.
 var defaultRestorePriorities = []string{
+	"customresourcedefinitions",
 	"namespaces",
 	"storageclasses",
 	"persistentvolumes",
@@ -469,7 +470,6 @@ var defaultRestorePriorities = []string{
 	// to ensure that we prioritize restoring from "apps" too, since this is how they're stored
 	// in the backup.
 	"replicasets.apps",
-	"customresourcedefinitions",
 }
 
 func (s *server) initRestic() error {