From dfbd9db9e3b2ce93e78309bd4d092fc7ec33f143 Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Tue, 11 Mar 2025 18:51:00 -0700 Subject: [PATCH 01/31] Add design for VolumeGroupSnapshot support Signed-off-by: Shubham Pampattiwar add changelog file Signed-off-by: Shubham Pampattiwar fix codespell checks Signed-off-by: Shubham Pampattiwar address PR feedback: add itemblock:VGS digrams and extra notes for clarification Signed-off-by: Shubham Pampattiwar update backup workflow Signed-off-by: Shubham Pampattiwar --- .../unreleased/8778-shubham-pampattiwar | 1 + design/volume-group-snapshot.md | 605 ++++++++++++++++++ 2 files changed, 606 insertions(+) create mode 100644 changelogs/unreleased/8778-shubham-pampattiwar create mode 100644 design/volume-group-snapshot.md diff --git a/changelogs/unreleased/8778-shubham-pampattiwar b/changelogs/unreleased/8778-shubham-pampattiwar new file mode 100644 index 000000000..569f43ecf --- /dev/null +++ b/changelogs/unreleased/8778-shubham-pampattiwar @@ -0,0 +1 @@ +Add design for VolumeGroupSnapshot support \ No newline at end of file diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md new file mode 100644 index 000000000..b7480e855 --- /dev/null +++ b/design/volume-group-snapshot.md @@ -0,0 +1,605 @@ +# Add Support for VolumeGroupSnapshots + +This proposal outlines the design and implementation plan for incorporating VolumeGroupSnapshot support into Velero. The enhancement will allow Velero to perform consistent, atomic snapshots of groups of Volumes using the new Kubernetes [VolumeGroupSnapshot API](https://kubernetes.io/blog/2024/12/18/kubernetes-1-32-volume-group-snapshot-beta/). This capability is especially critical for stateful applications that rely on multiple volumes to ensure data consistency, such as databases and analytics workloads. + +## Background + +Velero currently enables snapshot-based backups on an individual Volume basis through CSI drivers. However, modern stateful applications often require multiple volumes for data, logs, and backups. This distributed data architecture increases the risk of inconsistencies when volumes are captured individually. Kubernetes has introduced the VolumeGroupSnapshot(VGS) API [(KEP-3476)](https://github.com/kubernetes/enhancements/pull/1551), which allows for the atomic snapshotting of multiple volumes in a coordinated manner. By integrating this feature, Velero can offer enhanced disaster recovery for multi-volume applications, ensuring consistency across all related data. + +## Goals +- Ensure that multiple related volumes are snapshotted simultaneously, preserving consistency for stateful applications via VolumeGroupSnapshots(VGS) API. +- Integrate VolumeGroupSnapshot functionality into Velero’s existing backup and restore workflows. +- Allow users to opt in to volume group snapshots via specifying the group label. + +## Non-Goals +- The proposal does not require a complete overhaul of Velero’s CSI integration, it will extend the current mechanism to support group snapshots. +- No any changes pertaining to execution of Restore Hooks + +## High-Level Design + +### Backup workflow: +#### Accept the label to be used for VGS from the user: + - Accept the label from the user, we will do this in 3 ways: + - Firstly, we will have a hard-coded default label key like `velero.io/volume-group-snapshot` that the users can directly use on their PVCs. + - Secondly, we will let the users override this default VGS label via a velero server arg, `--volume-group-nsaphot-label-key`, if needed. + - And Finally we will have the option to override the default label via Backup API spec, `backup.spec.volumeGroupSnapshotLabelKey` + - In all the instances, the VGS label key will be present on the backup spec, this makes the label key accessible to plugins during the execution of backup operation. + - This label will enable velero to filter the PVC to be included in the VGS spec. + - Users will have to label the PVCs before invoking the backup operation. + - This label would act as a group identifier for the PVCs to be grouped under a specific VGS. + - It will be used to collect the PVCs to be used for a particular instance of VGS object. +**Note:** Modifying or adding VGS label on PVCs during an active backup operation may lead to unexpected or undesirable backup results. To avoid inconsistencies, ensure PVC labels remain unchanged throughout the backup execution. + +#### Changes to the Existing PVC ItemBlockAction plugin: + - Currently the PVC IBA plugin is applied to PVCs and adds the RelatedItems for the particular PVC into the ItemBlock. + - At first it checks whether the PVC is bound and VolumeName is non-empty. + - Then it adds the related PV under the list of relatedItems. + - Following on, the plugin adds the pods mounting the PVC as relatedItems. + - Now we need to extend this PVC IBA plugin to add the PVCs to be grouped for a particular VGS object, so that they are processed together under an ItemBlock by Velero. + - First we will check if the PVC that is being processed by the plugin has the user specified VGS label. + - If it is present then we will execute a List call in the namespace with the label as a matching criteria and see if this results in any PVCs (other than the current one). + - If there are PVCs matching the criteria then we add the PVCs to the relatedItems list. + - This helps in building the ItemBlock we need for VGS processing, i.e. we have the relevant pods and PVCs in the ItemBlock. + +**Note:** The ItemBlock to VGS relationship will not always be 1:1. There might be scenarios when the ItemBlock might have multiple VGS instances associated with it. +Lets go ove some ItemBlock/VGS scenarios that we might encounter and visualize them for clarity: +1. Pod Mounts: Pod1 mounts both PVC1 and PVC2. + Grouping: PVC1 and PVC2 share the same group label (group: A) + ItemBlock: The item block includes Pod1, PVC1, and PVC2. + VolumeGroupSnapshot (VGS): Because PVC1 and PVC2 are grouped together by their label, they trigger the creation of a single VGS (labeled with group: A). + +```mermaid +flowchart TD + subgraph ItemBlock + P1[Pod1] + PVC1[PVC1 group: A] + PVC2[PVC2 group: A] + end + + P1 -->|mounts| PVC1 + P1 -->|mounts| PVC2 + + PVC1 --- PVC2 + + PVC1 -- "group: A" --> VGS[VGS group: A] + PVC2 -- "group: A" --> VGS + +``` +2. Pod Mounts: Pod1 mounts each of the four PVCs. + Grouping: + Group A: PVC1 and PVC2 share the same grouping label (group: A). + Group B: PVC3 and PVC4 share the grouping label (group: B) + ItemBlock: All objects (Pod1, PVC1, PVC2, PVC3, and PVC4) are collected into a single item block. + VolumeGroupSnapshots: + PVC1 and PVC2 (group A) point to the same VGS (VGS (group: A)). + PVC3 and PVC4 (group B) point to a different VGS (VGS (group: B)). +```mermaid +flowchart TD + subgraph ItemBlock + P1[Pod1] + PVC1[PVC1 group: A] + PVC2[PVC2 group: A] + PVC3[PVC3 group: B] + PVC4[PVC4 group: B] + end + + %% Pod mounts all PVCs + P1 -->|mounts| PVC1 + P1 -->|mounts| PVC2 + P1 -->|mounts| PVC3 + P1 -->|mounts| PVC4 + + %% Group A relationships: PVC1 and PVC2 + PVC1 --- PVC2 + PVC1 -- "group: A" --> VGS_A[VGS-A group: A] + PVC2 -- "group: A" --> VGS_A + + %% Group B relationships: PVC3 and PVC4 + PVC3 --- PVC4 + PVC3 -- "group: B" --> VGS_B[VGS-B group: B] + PVC4 -- "group: B" --> VGS_B +``` + +3. Pod Mounts: Pod1 mounts both PVC1 and PVC2, Pod2 mounts PVC1 and PVC3. + Grouping: + Group A: PVC1 and PVC2 + Group B: PVC1 and PVC3 + ItemBlock: All objects-Pod1, Pod2, PVC1, PVC2, and PVC3, are collected into a single item block. + VolumeGroupSnapshots: + PVC1 and PVC2 (group A) point to the same VGS (VGS (group: A)). + PVC3 (group B) point to a different VGS (VGS (group: B)). +```mermaid +flowchart TD + subgraph ItemBlock + P1[Pod1] + P2[Pod2] + PVC1[PVC1 group: A, group: B] + PVC2[PVC2 group: A] + PVC3[PVC3 group: B] + end + + %% Pod mount relationships + P1 -->|mounts| PVC1 + P1 -->|mounts| PVC2 + P2 -->|mounts| PVC1 + P2 -->|mounts| PVC3 + + %% Grouping for Group A: PVC1 and PVC2 are grouped into VGS_A + PVC1 --- PVC2 + PVC1 -- "Group A" --> VGS_A[VGS Group A] + PVC2 -- "Group A" --> VGS_A + + %% Grouping for Group B: PVC3 grouped into VGS_B + PVC3 -- "Group B" --> VGS_B[VGS Group B] + +``` + +#### Updates to CSI PVC plugin: + - This is the plugin which creates VolumeSnapshots for the CSI PVCs if required. + - We might encounter the following cases: + - PVC has VGS label and VGS already exists: + - We can check if VGS exists by listing the VGS in the namespace using the backup UUID + along with the label used for the VGS (we will be creating VGS with backup UUID and user specified VGS group label) and see if the PVC is part of any of the existing VGS for the particular backup operation we are currently in. + - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label) and also add the PVC names (that are included in the VGS) as + annotations. (This will be useful for our VGS restore operation where we would need PVC resource identifiers). + - For non-datamover case: + - Once VGS is created, add the VGS as an additional item to be included in the backup. + - The VGS BIA plugin will add the related VS as additional items later. + - For datamover case: + - We need to bypass creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. + - Fetch the correct VS instances that got created via VGS and is related to the current PVC. + - Finally create DataUpload object for this VS/PVC pair. + - Add the DataUpload as an additional item. + - PVC has VGS label and VGS exists: + - In this case we do not want to re-create VGS object. + - The required VS/DU for the PVC already exists as VGS instance exists for the PVC and backup UUID. + - PVC does not possess VGS label: + - Create VS and follow the existing legacy workflow. + + +#### Add a new VolumeGroupSnapshot(VGS) BIA plugin + - We need this plugin only for non-datamover case. + - For a particular instance of VGS, this plugin will wait for all the related VS instances to be in ready state. + - And once they are ready, add all the VS instances as additional items for the backup. + - This plugin will also add VolumeGroupSnapshotClass(VGSClass) and VolumeGroupSnapshotContent(VGSC) as additional items to be included in the backup. + +### Add a new VolumeGroupSnapshotClass(VGSClass) BIA plugin + - We need this plugin only for non-datamover case. + - This plugin will be similar to the VSClass plugin. + - This will check if the VGSClass has any lister secret annotations and then add it as an additional item. + +### Add a new VolumeGroupSnapshotContent(VGSC) BIA Plugin + - We need this plugin only for non-datamover case. + - This will also be similar to the VSC plugin + - This will check if the VGSC has any delete secret annotations and then add it as an additional item. + +```mermaid +flowchart TD +%% User Input Section + subgraph User_Input [User Input] + A[User sets VGS label
via default, server arg, or Backup spec] + end + +%% PVC ItemBlockAction Plugin Section + subgraph PVC_IBA [PVC ItemBlockAction Plugin] + B[PVC has VGS label?] + C[If Yes: List all PVCs in namespace
with same label] + D[Add grouped PVCs to relatedItems] + E[Pod IBA adds pods mounting the PVC] + end + +%% CSI PVC Plugin Section + subgraph CSI_PVC [CSI PVC Plugin] + F[Does PVC have VGS label?] + G[If Yes: Check if VGS exists
using backup UID & label] + H[If VGS NOT found,
create new VGS object
backup UID, label, PVC names in annotation] + I[If VGS exists, skip VGS as well as VS creation] + J[If No label: Create individual VS legacy flow] + K[For non-datamover:
Add VGS as additional item] + L[For datamover:
Create DataUpload for VS and add as additional item] + end + +%% VGS BackupItemAction Plugin Section + subgraph VGS_BIA [VGS BackupItemAction Plugin] + M[Wait for all related VS to be ready] + N[Add all ready VS, VGSClass and VGSC as additional items] + end + +%% Existing VS and VSC BIA Plugins + subgraph VS_VSC_BIA [VS and VSC BIA Plugin] + P[Legacy VS and VSC BIA actions] + end + +%% VGSClass and VGSC BIA Plugins + subgraph VSClass_VGSC_BIA [VGSClass and VGSC BIA Plugin] + Q[Similar actions to VSClass and VSC BIA Actions] + end + +%% Flow Connections + A --> B + B -- Yes --> C + C --> D + D --> E + B -- No --> E + + E --> F + F -- Yes --> G + G -- VGS not found --> H + H --> K + G -- VGS exists --> I + F -- No --> J + + K --> VGS_BIA + J -- snapshotMoveData is true --> L + + VGS_BIA --> M + M --> N + N --> P + N --> Q + +``` + + +Restore workflow (WIP): + +- Update the Default Resource Restore priority for Velero restore + - Modify the default priority so that VGS, VGSClass are restored earlier than Pods and PVCs. + +- Add a new VGS RIA plugin + - This plugin will skip restore of the VGS resource. + - It will add the PVCs of the VGS as additional Items. + - The PVC identifiers will be obtained from the VGS annotation that we added during the backup workflow. + + +## Detailed Design + +Backup workflow: +- Accept the label to be used for VGS from the user as a server argument: + - Set a default VGS label key to be used: + ```go + // default VolumeGroupSnapshot Label + defaultVGSLabelKey = "velero.io/volume-group-snapshot" + + ``` + - Add this as a server flag and pass it to backup reconciler, so that we can use it during the backup request execution. + ```go + flags.StringVar(&c.DefaultVGSLabelKey, "volume-group-snapshot-label-key", c.DefaultVGSLabelKey, "Label key for grouping PVCs into VolumeGroupSnapshot") + ``` + + - Update the Backup CRD to accept the VGS Label Key as a spec value: + ```go + // VolumeGroupSnapshotLabelKey specifies the label key to be used for grouping the PVCs under + // an instance of VolumeGroupSnapshot, if left unspecified velero.io/volume-group-snapshot is used + // +optional + VolumeGroupSnapshotLabelKey string `json:"volumeGroupSnapshotLabelKey,omitempty"` + ``` + - Modify the [`prepareBackupRequest` function](https://github.com/openshift/velero/blob/8c8a6cccd78b78bd797e40189b0b9bee46a97f9e/pkg/controller/backup_controller.go#L327) to set the default label key as a backup spec if the user does not specify any value: + ```go + if len(request.Spec.VolumeGroupSnapshotLabelKey) == 0 { + // set the default key value + request.Spec.VolumeGroupSnapshotLabelKey = b.defaultVGSLabelKey + } + ``` + +- Changes to the Existing [PVC ItemBlockAction plugin](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/itemblock/actions/pvc_action.go#L64) (Update the GetRelatedItems function): +```go +// Retrieve the VGS label key from the Backup spec. + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey + if vgsLabelKey != "" { + // Check if the PVC has the specified VGS label. + if groupID, ok := pvc.Labels[vgsLabelKey]; ok { + // List all PVCs in the namespace with the same label key and value (i.e. same group). + pvcList := new(corev1api.PersistentVolumeClaimList) + if err := a.crClient.List(context.Background(), pvcList, crclient.InNamespace(pvc.Namespace), crclient.MatchingLabels{vgsLabelKey: groupID}); err != nil { + return nil, errors.Wrap(err, "failed to list PVCs for VGS grouping") + } + // Add each matching PVC (except the current one) to the relatedItems. + for _, groupPVC := range pvcList.Items { + if groupPVC.Name == pvc.Name { + continue + } + a.log.Infof("Adding grouped PVC %s to relatedItems for PVC %s", groupPVC.Name, pvc.Name) + relatedItems = append(relatedItems, velero.ResourceIdentifier{ + GroupResource: kuberesource.PersistentVolumeClaims, + Namespace: groupPVC.Namespace, + Name: groupPVC.Name, + }) + } + } + } else { + a.log.Info("No VolumeGroupSnapshotLabelKey provided in backup spec; skipping PVC grouping") + } +``` + +- Updates to [CSI PVC plugin](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/backup/actions/csi/pvc_action.go#L200) (Update the Execute method): +```go +// Retrieve the VGS label key from the backup spec. + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey + // Check if the PVC has the specified VGS label key. + if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { + // PVC is marked for group snapshot. + existingVGS := p.findExistingVGS(backup.UID, vgsLabelKey, group, pvc.Namespace) + if existingVGS == nil { + // No existing VGS found; list all PVCs in the same group. + groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + // Extract the names of all grouped PVCs. + pvcNames := extractPVCNames(groupedPVCs) + // Create a new VGS object with the backup UID and group. + newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "snapshot.storage.k8s.io", + Resource: "volumegroupsnapshots", + }, + Namespace: newVGS.Namespace, + Name: newVGS.Name, + }) + } else { + // Existing VGS found; skip creating an individual VS. + p.log.Infof("PVC %s is part of existing VGS %s, skipping individual VolumeSnapshot creation", pvc.Name, existingVGS.Name) + } + } else { + // PVC is not part of a VGS group; create an individual VolumeSnapshot. + + //. + //. + //. + } + +// helper functions used + +// findExistingVGS checks for an existing VolumeGroupSnapshot (VGS) for the given backup UID and group in the namespace. +func (p *pvcBackupItemAction) findExistingVGS(backupUID types.UID, vgsLabelKey, group, namespace string) *VolumeGroupSnapshot { + var vgsList VolumeGroupSnapshotList + err := p.crClient.List( + context.TODO(), + &vgsList, + crclient.InNamespace(namespace), + crclient.MatchingLabels{ + "backup-uid": string(backupUID), + vgsLabelKey: group, + }, + ) + if err != nil { + p.log.Errorf("Error listing VGS: %v", err) + return nil + } + if len(vgsList.Items) > 0 { + return &vgsList.Items[0] + } + return nil +} + +// listGroupedPVCs returns all PVCs in the given namespace that have the specified VGS label key and group value. +func (p *pvcBackupItemAction) listGroupedPVCs(backup *velerov1api.Backup, namespace, vgsLabelKey, group string) ([]corev1api.PersistentVolumeClaim, error) { + var pvcList corev1api.PersistentVolumeClaimList + err := p.crClient.List(context.TODO(), &pvcList, crclient.InNamespace(namespace), crclient.MatchingLabels{vgsLabelKey: group}) + if err != nil { + return nil, err + } + return pvcList.Items, nil +} + +func (p *pvcBackupItemAction) createVolumeGroupSnapshot( + backup *velerov1api.Backup, + pvc corev1api.PersistentVolumeClaim, + pvcNames []string, + vgsLabelKey, group string, +) (*VolumeGroupSnapshot, error) { + // Generate a name for the new VGS object. + name := fmt.Sprintf("vgs-%s-%d", string(backup.UID), time.Now().Unix()) + + // Construct the VGS object + vgs := &VolumeGroupSnapshot{ + TypeMeta: metav1.TypeMeta{ + Kind: "VolumeGroupSnapshot", + APIVersion: "groupsnapshot.storage.k8s.io/v1beta1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: pvc.Namespace, + Annotations: map[string]string{ + "pvcList": strings.Join(pvcNames, ","), + }, + Labels: map[string]string{ + velerov1api.BackupUIDLabel: string(backup.UID), + vgsLabelKey: group, + }, + }, + Spec: VolumeGroupSnapshotSpec{ + // we will use default VGSClass for now + Source: VolumeGroupSnapshotSource{ + Selector: metav1.LabelSelector{ + MatchLabels: map[string]string{ + vgsLabelKey: group, + }, + }, + }, + }, + } + + // Create the VGS object + if err := p.crClient.Create(context.TODO(), vgs); err != nil { + return nil, err + } + return vgs, nil +} + +// extractPVCNames returns a slice of PVC names from the provided list. +func extractPVCNames(pvcs []corev1api.PersistentVolumeClaim) []string { + names := []string{} + for _, pvc := range pvcs { + names = append(names, pvc.Name) + } + return names +} + +``` + +- Add a new VolumeGroupSnapshot(VGS) BIA plugin +```go +// AppliesTo specifies that this plugin applies to VolumeGroupSnapshot objects. +func (p *vgsBackupItemAction) AppliesTo() (velero.ResourceSelector, error) { + return velero.ResourceSelector{ + IncludedResources: []string{"volumegroupsnapshots"}, + }, nil +} + +// Execute waits for all related VolumeSnapshot (VS) instances to be ready, then adds them +// along with the VolumeGroupSnapshotClass as additional backup items. +func (p *vgsBackupItemAction) Execute( + item runtime.Unstructured, + backup *velerov1api.Backup, +) ( + runtime.Unstructured, + []velero.ResourceIdentifier, + string, + []velero.ResourceIdentifier, + error, +) { + p.log.Infof("Starting VGS backup plugin Execute for item: %s", item.GetName()) + + // Convert the unstructured object to a VolumeGroupSnapshot. + var vgs VolumeGroupSnapshot + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(item.UnstructuredContent(), &vgs); err != nil { + return nil, nil, "", nil, errors.Wrap(err, "unable to convert item to VolumeGroupSnapshot") + } + + // Wait until all related VolumeSnapshots are ready. + p.log.Infof("Waiting for all related VolumeSnapshots for VGS %s to be ready", vgs.Name) + ready, err := waitForGroupedVolumeSnapshots(&vgs, vgs.Namespace, p.log) + if err != nil { + return nil, nil, "", nil, errors.Wrap(err, "error waiting for grouped VolumeSnapshots") + } + if !ready { + p.log.Infof("Not all related VolumeSnapshots are ready for VGS %s", vgs.Name) + return item, nil, "", nil, nil + } + + // Retrieve the related VolumeSnapshot instances. + groupedVS := getGroupedVolumeSnapshots(&vgs, vgs.Namespace) + var additionalItems []velero.ResourceIdentifier + for _, vs := range groupedVS { + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "snapshot.storage.k8s.io", + Resource: "volumesnapshots", + }, + Namespace: vs.Namespace, + Name: vs.Name, + }) + } + + // Add the VolumeGroupSnapshotClass as an additional item. + // Here we assume that the VGS spec defines the class name. + // We will be using the default VGSClass for now + vgsClassName := vgs.Spec.VolumeGroupSnapshotClassName + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "snapshot.storage.k8s.io", + Resource: "volumegroupsnapshotclasses", + }, + Namespace: "", + Name: vgsClassName, + }) + + p.log.Infof("VGS %s ready; adding %d additional items to backup", vgs.Name, len(additionalItems)) + return item, additionalItems, "", nil, nil +} +``` + +Restore workflow: + +- Update the [Default Resource Restore priority](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/cmd/server/config/config.go#L116) for Velero restore +```go +defaultRestorePriorities = types.Priorities{ + HighPriorities: []string{ + "customresourcedefinitions", + "namespaces", + "storageclasses", + // updated list from here + "volumegroupsnaoshotclass.snapshot.k8s.io", + "volumegroupsnapshots.snapshot.storage.k8s.io", + "volumesnapshotclass.snapshot.storage.k8s.io", + "volumesnapshotcontents.snapshot.storage.k8s.io", + "volumesnapshots.snapshot.storage.k8s.io", + "datauploads.velero.io", + "persistentvolumes", + "persistentvolumeclaims", +``` +- Add a VGS RIA plugin +```go + +// AppliesTo returns a selector that matches VolumeGroupSnapshot resources. +func (p *vgsRestoreItemAction) AppliesTo() (velero.ResourceSelector, error) { + return velero.ResourceSelector{ + IncludedResources: []string{"volumegroupsnapshots"}, + }, nil +} + + +// Execute skips restoring the VGS resource and extracts the PVC identifiers from the "pvcList" annotation, +// adding each PVC as an additional restore item. +func (p *vgsRestoreItemAction) Execute( + input *velero.RestoreItemActionExecuteInput, +) (*velero.RestoreItemActionExecuteOutput, error) { + p.log.Info("Starting VGS RestoreItemAction") + + // Convert the backup VGS object to an unstructured type. + // We use input.ItemFromBackup as the backed-up version of the VGS. + var vgs unstructured.Unstructured + vgs.SetUnstructuredContent(input.ItemFromBackup.UnstructuredContent()) + + // Retrieve the pvcList annotation. + annotations := vgs.GetAnnotations() + pvcListStr, ok := annotations["pvcList"] + if !ok || pvcListStr == "" { + p.log.Infof("No pvcList annotation found on VGS %s", vgs.GetName()) + return &velero.RestoreItemActionExecuteOutput{SkipRestore: true}, nil + } + + // Parse the comma-separated list of PVC names. + pvcNames := strings.Split(pvcListStr, ",") + var additionalItems []velero.ResourceIdentifier + namespace := vgs.GetNamespace() + for _, pvcName := range pvcNames { + pvcName = strings.TrimSpace(pvcName) + if pvcName == "" { + continue + } + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "", + Resource: "persistentvolumeclaims", + }, + Namespace: namespace, + Name: pvcName, + }) + } + + p.log.Infof("Skipping restore of VGS %s and adding %d PVC(s) as additional restore items", vgs.GetName(), len(additionalItems)) + + // Return output indicating that the VGS resource itself should be skipped, + // and providing the list of PVCs to be restored as additional items + return &velero.RestoreItemActionExecuteOutput{ + SkipRestore: true, + AdditionalItems: additionalItems, + }, nil +} +``` +## Implementation + +This design proposal is targeted for velero 1.16. + +The implementation of this proposed design is targeted for velero 1.17. + +## Open Questions + +How to handle VGSC ? Do we need VGSC for non-datamover operations ? From 48d6aff78606338d2c274a9e4e7d992da0292332 Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Mon, 24 Mar 2025 12:45:36 -0700 Subject: [PATCH 02/31] update itemblock case 3 Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index b7480e855..1eb23f131 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -103,7 +103,7 @@ flowchart TD 3. Pod Mounts: Pod1 mounts both PVC1 and PVC2, Pod2 mounts PVC1 and PVC3. Grouping: Group A: PVC1 and PVC2 - Group B: PVC1 and PVC3 + Group B: PVC3 ItemBlock: All objects-Pod1, Pod2, PVC1, PVC2, and PVC3, are collected into a single item block. VolumeGroupSnapshots: PVC1 and PVC2 (group A) point to the same VGS (VGS (group: A)). @@ -113,7 +113,7 @@ flowchart TD subgraph ItemBlock P1[Pod1] P2[Pod2] - PVC1[PVC1 group: A, group: B] + PVC1[PVC1 group: A] PVC2[PVC2 group: A] PVC3[PVC3 group: B] end From 0c87e2f64d4618a99aa15feab8c6e50a0303b04c Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Tue, 25 Mar 2025 15:03:34 -0700 Subject: [PATCH 03/31] Update the VGS B/R workflows Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 229 +++++++++----------------------- 1 file changed, 64 insertions(+), 165 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 1eb23f131..ca5f1ca83 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -42,7 +42,7 @@ Velero currently enables snapshot-based backups on an individual Volume basis th - This helps in building the ItemBlock we need for VGS processing, i.e. we have the relevant pods and PVCs in the ItemBlock. **Note:** The ItemBlock to VGS relationship will not always be 1:1. There might be scenarios when the ItemBlock might have multiple VGS instances associated with it. -Lets go ove some ItemBlock/VGS scenarios that we might encounter and visualize them for clarity: +Lets go over some ItemBlock/VGS scenarios that we might encounter and visualize them for clarity: 1. Pod Mounts: Pod1 mounts both PVC1 and PVC2. Grouping: PVC1 and PVC2 share the same group label (group: A) ItemBlock: The item block includes Pod1, PVC1, and PVC2. @@ -137,119 +137,97 @@ flowchart TD #### Updates to CSI PVC plugin: - This is the plugin which creates VolumeSnapshots for the CSI PVCs if required. - We might encounter the following cases: - - PVC has VGS label and VGS already exists: + - PVC has VGS label and VGS does not exist: - We can check if VGS exists by listing the VGS in the namespace using the backup UUID along with the label used for the VGS (we will be creating VGS with backup UUID and user specified VGS group label) and see if the PVC is part of any of the existing VGS for the particular backup operation we are currently in. - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label) and also add the PVC names (that are included in the VGS) as - annotations. (This will be useful for our VGS restore operation where we would need PVC resource identifiers). + annotations. + - Once VGS is created, wait for VGS object to be `readyToUse`, it implies that all the related VS/VSC objects are ready to use too. + - Add the VGS object as an additional item. - For non-datamover case: - - Once VGS is created, add the VGS as an additional item to be included in the backup. - - The VGS BIA plugin will add the related VS as additional items later. + - Here, we need to skip the creation of VS as the VS object get created via creation of VGS object. + - We just need to add the VS objects that got created via VGS object as additional items and the rest of the workflow remains the same for non-datamover cases. - For datamover case: - - We need to bypass creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. + - We need to skip creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. - Fetch the correct VS instances that got created via VGS and is related to the current PVC. - Finally create DataUpload object for this VS/PVC pair. - Add the DataUpload as an additional item. - - PVC has VGS label and VGS exists: + - PVC has VGS label and VGS already exists: - In this case we do not want to re-create VGS object. - The required VS/DU for the PVC already exists as VGS instance exists for the PVC and backup UUID. - PVC does not possess VGS label: - Create VS and follow the existing legacy workflow. - #### Add a new VolumeGroupSnapshot(VGS) BIA plugin - - We need this plugin only for non-datamover case. - - For a particular instance of VGS, this plugin will wait for all the related VS instances to be in ready state. - - And once they are ready, add all the VS instances as additional items for the backup. - - This plugin will also add VolumeGroupSnapshotClass(VGSClass) and VolumeGroupSnapshotContent(VGSC) as additional items to be included in the backup. - -### Add a new VolumeGroupSnapshotClass(VGSClass) BIA plugin - - We need this plugin only for non-datamover case. - - This plugin will be similar to the VSClass plugin. - - This will check if the VGSClass has any lister secret annotations and then add it as an additional item. - -### Add a new VolumeGroupSnapshotContent(VGSC) BIA Plugin - - We need this plugin only for non-datamover case. - - This will also be similar to the VSC plugin - - This will check if the VGSC has any delete secret annotations and then add it as an additional item. + - Cleanup the VGS object, here we need to be careful, we want to delete VGS and VGSC objects but keep the VS and VSC objects for further processing of our backup. + - We will first check the `deletionPolicy` of VGSC, + - if its `Retain` then we can go ahead with the deletion of VGS and VGSC objects. + - But if its `Delete` then we need to first patch the VGSC `deletionPolicy` to `Retain` and then delete the VGS and VGSC objects. ```mermaid flowchart TD -%% User Input Section - subgraph User_Input [User Input] - A[User sets VGS label
via default, server arg, or Backup spec] +%% User Input + subgraph "User Input" + A[User sets VGS label key
default, server arg, or Backup spec] + B[User labels PVCs before backup] + A --> B end -%% PVC ItemBlockAction Plugin Section - subgraph PVC_IBA [PVC ItemBlockAction Plugin] - B[PVC has VGS label?] - C[If Yes: List all PVCs in namespace
with same label] - D[Add grouped PVCs to relatedItems] - E[Pod IBA adds pods mounting the PVC] +%% PVC ItemBlockAction Plugin + subgraph "PVC IBA Plugin" + C[Check: PVC is bound & VolumeName non-empty] + D[Add related PV and pods] + E[Check if PVC has VGS label] + F[List all PVCs with same label in namespace] + G[Add PVCs to relatedItems] + C --> D + D --> E + E -- Yes --> F + F --> G end -%% CSI PVC Plugin Section - subgraph CSI_PVC [CSI PVC Plugin] - F[Does PVC have VGS label?] - G[If Yes: Check if VGS exists
using backup UID & label] - H[If VGS NOT found,
create new VGS object
backup UID, label, PVC names in annotation] - I[If VGS exists, skip VGS as well as VS creation] - J[If No label: Create individual VS legacy flow] - K[For non-datamover:
Add VGS as additional item] - L[For datamover:
Create DataUpload for VS and add as additional item] - end +%% CSI PVC Plugin +subgraph "CSI PVC Plugin" +H[For each PVC, check for VGS label] +I[If VGS label exists:] +J[Lookup VGS using backup UID and label] +K[Create VGS object] +L[Wait until VGS is readyToUse, add VGS as an additional item] +M[Non-datamover: Skip individual VS creation; add VS objects from VGS as additional items] +N[Datamover: Create DataUpload for VS PVC pair; add DU as an additional item] +O[If VGS already exists, skip creation] +P[If no VGS label, follow legacy workflow: create VS] +H --> I +I -- Yes --> J +J -- Not found --> K +K --> L +L --> M +L --> N +I -- Yes and found --> O +I -- No --> P +end -%% VGS BackupItemAction Plugin Section - subgraph VGS_BIA [VGS BackupItemAction Plugin] - M[Wait for all related VS to be ready] - N[Add all ready VS, VGSClass and VGSC as additional items] - end +%% VGS BackupItemAction Plugin +subgraph "VGS BIA Plugin" +Q[Check VGSC deletionPolicy] +R[If Retain, delete VGS and VGSC objects] +S[If Delete, patch VGSC to Retain then delete VGS and VGSC] +Q --> R +Q --> S +end -%% Existing VS and VSC BIA Plugins - subgraph VS_VSC_BIA [VS and VSC BIA Plugin] - P[Legacy VS and VSC BIA actions] - end - -%% VGSClass and VGSC BIA Plugins - subgraph VSClass_VGSC_BIA [VGSClass and VGSC BIA Plugin] - Q[Similar actions to VSClass and VSC BIA Actions] - end - -%% Flow Connections - A --> B - B -- Yes --> C - C --> D - D --> E - B -- No --> E - - E --> F - F -- Yes --> G - G -- VGS not found --> H - H --> K - G -- VGS exists --> I - F -- No --> J - - K --> VGS_BIA - J -- snapshotMoveData is true --> L - - VGS_BIA --> M - M --> N - N --> P - N --> Q +%% Connect the flows +B --> C +G --> H +M --> Q +N --> Q ``` -Restore workflow (WIP): - -- Update the Default Resource Restore priority for Velero restore - - Modify the default priority so that VGS, VGSClass are restored earlier than Pods and PVCs. - -- Add a new VGS RIA plugin - - This plugin will skip restore of the VGS resource. - - It will add the PVCs of the VGS as additional Items. - - The PVC identifiers will be obtained from the VGS annotation that we added during the backup workflow. +Restore workflow: +No changes required for the restore workflow. ## Detailed Design @@ -515,85 +493,6 @@ func (p *vgsBackupItemAction) Execute( } ``` -Restore workflow: - -- Update the [Default Resource Restore priority](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/cmd/server/config/config.go#L116) for Velero restore -```go -defaultRestorePriorities = types.Priorities{ - HighPriorities: []string{ - "customresourcedefinitions", - "namespaces", - "storageclasses", - // updated list from here - "volumegroupsnaoshotclass.snapshot.k8s.io", - "volumegroupsnapshots.snapshot.storage.k8s.io", - "volumesnapshotclass.snapshot.storage.k8s.io", - "volumesnapshotcontents.snapshot.storage.k8s.io", - "volumesnapshots.snapshot.storage.k8s.io", - "datauploads.velero.io", - "persistentvolumes", - "persistentvolumeclaims", -``` -- Add a VGS RIA plugin -```go - -// AppliesTo returns a selector that matches VolumeGroupSnapshot resources. -func (p *vgsRestoreItemAction) AppliesTo() (velero.ResourceSelector, error) { - return velero.ResourceSelector{ - IncludedResources: []string{"volumegroupsnapshots"}, - }, nil -} - - -// Execute skips restoring the VGS resource and extracts the PVC identifiers from the "pvcList" annotation, -// adding each PVC as an additional restore item. -func (p *vgsRestoreItemAction) Execute( - input *velero.RestoreItemActionExecuteInput, -) (*velero.RestoreItemActionExecuteOutput, error) { - p.log.Info("Starting VGS RestoreItemAction") - - // Convert the backup VGS object to an unstructured type. - // We use input.ItemFromBackup as the backed-up version of the VGS. - var vgs unstructured.Unstructured - vgs.SetUnstructuredContent(input.ItemFromBackup.UnstructuredContent()) - - // Retrieve the pvcList annotation. - annotations := vgs.GetAnnotations() - pvcListStr, ok := annotations["pvcList"] - if !ok || pvcListStr == "" { - p.log.Infof("No pvcList annotation found on VGS %s", vgs.GetName()) - return &velero.RestoreItemActionExecuteOutput{SkipRestore: true}, nil - } - - // Parse the comma-separated list of PVC names. - pvcNames := strings.Split(pvcListStr, ",") - var additionalItems []velero.ResourceIdentifier - namespace := vgs.GetNamespace() - for _, pvcName := range pvcNames { - pvcName = strings.TrimSpace(pvcName) - if pvcName == "" { - continue - } - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "", - Resource: "persistentvolumeclaims", - }, - Namespace: namespace, - Name: pvcName, - }) - } - - p.log.Infof("Skipping restore of VGS %s and adding %d PVC(s) as additional restore items", vgs.GetName(), len(additionalItems)) - - // Return output indicating that the VGS resource itself should be skipped, - // and providing the list of PVCs to be restored as additional items - return &velero.RestoreItemActionExecuteOutput{ - SkipRestore: true, - AdditionalItems: additionalItems, - }, nil -} -``` ## Implementation This design proposal is targeted for velero 1.16. From 5ce4b5ad643a855a4fd84765fb040f93f0a99dfb Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Thu, 27 Mar 2025 11:28:36 -0700 Subject: [PATCH 04/31] remove vgsc open question Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index ca5f1ca83..47f06acfc 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -499,6 +499,3 @@ This design proposal is targeted for velero 1.16. The implementation of this proposed design is targeted for velero 1.17. -## Open Questions - -How to handle VGSC ? Do we need VGSC for non-datamover operations ? From d4296aa78c963e366d3b983b95572c3cd11b9340 Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Thu, 27 Mar 2025 18:18:29 -0700 Subject: [PATCH 05/31] delegate cleanup to VGS BIA Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 116 ++++++++++++++++++-------------- 1 file changed, 64 insertions(+), 52 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 47f06acfc..6b9bd0dd5 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -140,17 +140,18 @@ flowchart TD - PVC has VGS label and VGS does not exist: - We can check if VGS exists by listing the VGS in the namespace using the backup UUID along with the label used for the VGS (we will be creating VGS with backup UUID and user specified VGS group label) and see if the PVC is part of any of the existing VGS for the particular backup operation we are currently in. - - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label) and also add the PVC names (that are included in the VGS) as - annotations. - - Once VGS is created, wait for VGS object to be `readyToUse`, it implies that all the related VS/VSC objects are ready to use too. - - Add the VGS object as an additional item. + - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label). + - Once VGS is created, we just wait for the related VS/VSC objects to exist and not wait for them to be `readyToUse`. + - Add the VGS object to the list of itemToUpdate, so that its BIA plugin runs in finalize backup phase. (we will run all the cleanup tasks in this plugin) - For non-datamover case: - Here, we need to skip the creation of VS as the VS object get created via creation of VGS object. + - We will skip the cleanup of VS here in VS BIA plugin as VGS related VS/VSC will get cleaned up in VGS BIA plugin. - We just need to add the VS objects that got created via VGS object as additional items and the rest of the workflow remains the same for non-datamover cases. - For datamover case: - We need to skip creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. - Fetch the correct VS instances that got created via VGS and is related to the current PVC. - Finally create DataUpload object for this VS/PVC pair. + - We will skip the cleanup of VS here as VGS related VS/VSC will get cleaned up in VGS BIA plugin. - Add the DataUpload as an additional item. - PVC has VGS label and VGS already exists: - In this case we do not want to re-create VGS object. @@ -158,76 +159,87 @@ flowchart TD - PVC does not possess VGS label: - Create VS and follow the existing legacy workflow. +#### Modify the CleanupVolumeSnapshot function to skip deletion of VGS related VS +- We use a common cleanupVolumeSnapshot function to cleanup VS objects in both datamover and non-datamover cases +- We are delegating the VGS related cleanup tasks to VGS BIA plugin which will run during the backup finalize phase when all the async actions are complete +- We need to modify this function skip deletion of VS objects that were created via VGS objects. + #### Add a new VolumeGroupSnapshot(VGS) BIA plugin - - Cleanup the VGS object, here we need to be careful, we want to delete VGS and VGSC objects but keep the VS and VSC objects for further processing of our backup. - - We will first check the `deletionPolicy` of VGSC, - - if its `Retain` then we can go ahead with the deletion of VGS and VGSC objects. - - But if its `Delete` then we need to first patch the VGSC `deletionPolicy` to `Retain` and then delete the VGS and VGSC objects. + - This plugin will run in backup finalize phase. + - We will process cleanup of all the VGS related artifacts like VGS, VGSC, VS and VSC that got created durring the backup process. ```mermaid flowchart TD -%% User Input +%% User Input Section subgraph "User Input" - A[User sets VGS label key
default, server arg, or Backup spec] - B[User labels PVCs before backup] - A --> B + U1[User sets VGS label key using default, server arg or Backup API spec] + U2[User labels PVCs before backup] + U1 --> U2 end -%% PVC ItemBlockAction Plugin +%% PVC ItemBlockAction Plugin Section subgraph "PVC IBA Plugin" - C[Check: PVC is bound & VolumeName non-empty] - D[Add related PV and pods] - E[Check if PVC has VGS label] - F[List all PVCs with same label in namespace] - G[Add PVCs to relatedItems] - C --> D - D --> E - E -- Yes --> F - F --> G + I1[Check PVC is bound and has VolumeName] + I2[Add related PV to relatedItems] + I3[Add pods mounting PVC to relatedItems] + I4[Check if PVC has user-specified VGS label] + I5[List PVCs in namespace matching label criteria] + I6[Add matching PVCs to relatedItems] + I1 --> I2 + I2 --> I3 + I3 --> I4 + I4 -- Yes --> I5 + I5 --> I6 end -%% CSI PVC Plugin +%% CSI PVC Plugin Section subgraph "CSI PVC Plugin" -H[For each PVC, check for VGS label] -I[If VGS label exists:] -J[Lookup VGS using backup UID and label] -K[Create VGS object] -L[Wait until VGS is readyToUse, add VGS as an additional item] -M[Non-datamover: Skip individual VS creation; add VS objects from VGS as additional items] -N[Datamover: Create DataUpload for VS PVC pair; add DU as an additional item] -O[If VGS already exists, skip creation] -P[If no VGS label, follow legacy workflow: create VS] -H --> I -I -- Yes --> J -J -- Not found --> K -K --> L -L --> M -L --> N -I -- Yes and found --> O -I -- No --> P +C1[For each PVC, check for VGS label] +C2[If VGS label exists then lookup VGS using backup UID and label] +C3[If VGS not found then create new VGS with backup UID and group label] +C4[Wait for related VS and VGSC objects to exist] +C5[Add VGS object to itemToUpdate] +C6[Non-datamover case: Skip individual VS creation; add VS from VGS as additional item] +C7[Datamover case: Create DataUpload for VS-PVC pair; add DU as additional item] +C8[If VGS already exists then do not re-create VGS] +C9[If no VGS label then follow legacy workflow to create VS] +C1 --> C2 +C2 -- Not Found --> C3 +C3 --> C4 +C4 --> C5 +C5 --> C6 +C2 -- Found --> C8 +C1 -- No VGS label --> C9 end -%% VGS BackupItemAction Plugin +%% Cleanup Modification Section +subgraph "Cleanup Modification" +CL1[Modify cleanupVolumeSnapshot to skip deletion of VS created via VGS] +CL2[Delegate VGS related cleanup to VGS BIA Plugin in finalize phase] +CL1 --> CL2 +end + +%% VGS BackupItemAction Plugin Section subgraph "VGS BIA Plugin" -Q[Check VGSC deletionPolicy] -R[If Retain, delete VGS and VGSC objects] -S[If Delete, patch VGSC to Retain then delete VGS and VGSC] -Q --> R -Q --> S +V1[Run in backup finalize phase] +V2[Process cleanup of VGS, VGSC, VS and VSC artifacts] +V1 --> V2 end -%% Connect the flows -B --> C -G --> H -M --> Q -N --> Q +%% Connect Major Sections +U2 --> I1 +I6 --> C1 +C6 --> CL1 +C7 --> CL1 +CL2 --> V1 ``` Restore workflow: -No changes required for the restore workflow. +#### Add a new VolumeGroupSnapshot(VGS) RIA plugin +- This plugin will just skip the restore VGS object ## Detailed Design From e9f23a32eef2f7576d332cbcf9a4384733b1e083 Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Thu, 27 Mar 2025 19:00:34 -0700 Subject: [PATCH 06/31] fix typo Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 6b9bd0dd5..f4d0dc53a 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -166,7 +166,7 @@ flowchart TD #### Add a new VolumeGroupSnapshot(VGS) BIA plugin - This plugin will run in backup finalize phase. - - We will process cleanup of all the VGS related artifacts like VGS, VGSC, VS and VSC that got created durring the backup process. + - We will process cleanup of all the VGS related artifacts like VGS, VGSC, VS and VSC that got created during the backup process. ```mermaid flowchart TD From 0ab2253f46752741742db09daea9543607993ada Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Thu, 3 Apr 2025 13:35:49 -0700 Subject: [PATCH 07/31] update csi plugin changes, diagram and snippets Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 548 ++++++++++++++++---------------- 1 file changed, 278 insertions(+), 270 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index f4d0dc53a..71e47d2ec 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -135,111 +135,85 @@ flowchart TD ``` #### Updates to CSI PVC plugin: - - This is the plugin which creates VolumeSnapshots for the CSI PVCs if required. - - We might encounter the following cases: - - PVC has VGS label and VGS does not exist: - - We can check if VGS exists by listing the VGS in the namespace using the backup UUID - along with the label used for the VGS (we will be creating VGS with backup UUID and user specified VGS group label) and see if the PVC is part of any of the existing VGS for the particular backup operation we are currently in. - - If not then create the VGS object with the required labels (in this case backup UUID and the user specified VGS group label). - - Once VGS is created, we just wait for the related VS/VSC objects to exist and not wait for them to be `readyToUse`. - - Add the VGS object to the list of itemToUpdate, so that its BIA plugin runs in finalize backup phase. (we will run all the cleanup tasks in this plugin) - - For non-datamover case: - - Here, we need to skip the creation of VS as the VS object get created via creation of VGS object. - - We will skip the cleanup of VS here in VS BIA plugin as VGS related VS/VSC will get cleaned up in VGS BIA plugin. - - We just need to add the VS objects that got created via VGS object as additional items and the rest of the workflow remains the same for non-datamover cases. - - For datamover case: - - We need to skip creation of VS but need to create DataUploads for the VS instance that got created due to VGS creation. - - Fetch the correct VS instances that got created via VGS and is related to the current PVC. - - Finally create DataUpload object for this VS/PVC pair. - - We will skip the cleanup of VS here as VGS related VS/VSC will get cleaned up in VGS BIA plugin. - - Add the DataUpload as an additional item. - - PVC has VGS label and VGS already exists: - - In this case we do not want to re-create VGS object. - - The required VS/DU for the PVC already exists as VGS instance exists for the PVC and backup UUID. - - PVC does not possess VGS label: - - Create VS and follow the existing legacy workflow. +- When a PVC has a VGS label and no VS (created via VGS) exists: + - Create VGS: + - This triggers creation of the corresponding VGSC, VS, and VSC objects. + - Wait for VS Status: + - Wait until each VS (one per PVC in the group) has its volumeGroupSnapshotName set. This confirms that the snapshot controller has done its work. + - Update VS Objects: + - Remove owner references and VGS-related finalizers from the VS objects (decoupling them to prevent cascading deletion). + - Add backup metadata (BackupName, BackupUUID, PVC name) as labels. This metadata is later used to skip re-creating a VGS when another PVC of the same group is processed. + - Patch and Cleanup: + - Patch the VGSC deletionPolicy to Retain so that when you delete the VGSC, the underlying VSC (and the storage snapshots) remain. + - Delete the temporary VGS and VGSC objects. + - Branching: + - For non‑datamover cases, skip the creation of an individual VS (since it was created via VGS) and add the VS objects as additional items. + - For datamover cases, create DataUploads for the VS–PVC pair (using the VS created by the VGS workflow) and add those as additional items. + +- When a PVC has a VGS label and a VS created via an earlier VGS workflow already exists: + - List VS objects in the PVC’s namespace using labels (BackupUUID, BackupName, PVCName). + - Verify that a VS exists and that its status shows a non‑empty volumeGroupSnapshotName. + - If so, skip VGS (and VS) creation and continue with the legacy workflow. + - If a VS is found but it wasn’t created by the VGS workflow (i.e. it lacks the volumeGroupSnapshotName), then the backup for that PVC is failed, resulting in a partially failed backup. -#### Modify the CleanupVolumeSnapshot function to skip deletion of VGS related VS -- We use a common cleanupVolumeSnapshot function to cleanup VS objects in both datamover and non-datamover cases -- We are delegating the VGS related cleanup tasks to VGS BIA plugin which will run during the backup finalize phase when all the async actions are complete -- We need to modify this function skip deletion of VS objects that were created via VGS objects. +- When a PVC does not have a VGS label: + - The legacy workflow is followed, creating an individual VolumeSnapshot as before. -#### Add a new VolumeGroupSnapshot(VGS) BIA plugin - - This plugin will run in backup finalize phase. - - We will process cleanup of all the VGS related artifacts like VGS, VGSC, VS and VSC that got created during the backup process. ```mermaid flowchart TD -%% User Input Section - subgraph "User Input" - U1[User sets VGS label key using default, server arg or Backup API spec] - U2[User labels PVCs before backup] - U1 --> U2 +%% Section 1: Accept VGS Label from User + subgraph Accept_Label + A1[User sets VGS label] + A2[User labels PVCs before backup] + A1 --> A2 end -%% PVC ItemBlockAction Plugin Section - subgraph "PVC IBA Plugin" - I1[Check PVC is bound and has VolumeName] - I2[Add related PV to relatedItems] - I3[Add pods mounting PVC to relatedItems] - I4[Check if PVC has user-specified VGS label] - I5[List PVCs in namespace matching label criteria] - I6[Add matching PVCs to relatedItems] - I1 --> I2 - I2 --> I3 - I3 --> I4 - I4 -- Yes --> I5 - I5 --> I6 +%% Section 2: PVC ItemBlockAction Plugin + subgraph PVC_ItemBlockAction + B1[Check PVC is bound and has VolumeName] + B2[Add related PV to relatedItems] + B3[Add pods mounting PVC to relatedItems] + B4[Check if PVC has user-specified VGS label] + B5[List PVCs in namespace matching label criteria] + B6[Add matching PVCs to relatedItems] + B1 --> B2 --> B3 --> B4 + B4 -- Yes --> B5 + B5 --> B6 end -%% CSI PVC Plugin Section -subgraph "CSI PVC Plugin" +%% Section 3: CSI PVC Plugin Updates +subgraph CSI_PVC_Plugin C1[For each PVC, check for VGS label] -C2[If VGS label exists then lookup VGS using backup UID and label] -C3[If VGS not found then create new VGS with backup UID and group label] -C4[Wait for related VS and VGSC objects to exist] -C5[Add VGS object to itemToUpdate] -C6[Non-datamover case: Skip individual VS creation; add VS from VGS as additional item] -C7[Datamover case: Create DataUpload for VS-PVC pair; add DU as additional item] -C8[If VGS already exists then do not re-create VGS] -C9[If no VGS label then follow legacy workflow to create VS] -C1 --> C2 -C2 -- Not Found --> C3 -C3 --> C4 -C4 --> C5 -C5 --> C6 -C2 -- Found --> C8 -C1 -- No VGS label --> C9 +C1 -- Yes --> C2[Case 1: VGS for volume group not yet created] +C2 -- True --> C3[Create new VGS triggering VGSC, VS and VSC creation] +C3 --> C4[Wait for VS objects to show volumeGroupSnapshotName using CSISnapshotTimeout] +C4 --> C5[Update VS objects: remove VGS owner refs and VGS finalizers; add BackupName, BackupUUID, PVC name as labels] +C5 --> C6[Patch VGSC deletionPolicy to Retain] +C6 --> C7[Delete VGS object] +C7 --> C8[Delete VGSC] +C8 --> C9[For non-datamover: Skip individual VS creation; add VS from VGS as additional item] +C8 --> C10[For datamover: Create DataUpload for VS-PVC pair; add DU as additional item] + +C1 -- Yes --> C11[Case 2: VGS was created then deleted] +C11 --> C12[List VS using labels BackupUID, BackupName, PVCName] +C12 --> C13[Check if VS has non-empty volumeGroupSnapshotName] +C13 -- Yes --> C14[Skip VGS creation; use existing VS/DU; legacy workflow continues] +C13 -- No --> C15[Fail backup for PVC] + +C1 -- No --> C16[Case 3: PVC lacks VGS label; follow legacy workflow to create VS] end -%% Cleanup Modification Section -subgraph "Cleanup Modification" -CL1[Modify cleanupVolumeSnapshot to skip deletion of VS created via VGS] -CL2[Delegate VGS related cleanup to VGS BIA Plugin in finalize phase] -CL1 --> CL2 -end - -%% VGS BackupItemAction Plugin Section -subgraph "VGS BIA Plugin" -V1[Run in backup finalize phase] -V2[Process cleanup of VGS, VGSC, VS and VSC artifacts] -V1 --> V2 -end - -%% Connect Major Sections -U2 --> I1 -I6 --> C1 -C6 --> CL1 -C7 --> CL1 -CL2 --> V1 +%% Connect Main Sections +A2 --> B1 +B6 --> C1 ``` Restore workflow: -#### Add a new VolumeGroupSnapshot(VGS) RIA plugin -- This plugin will just skip the restore VGS object +- No changes required for the restore workflow. ## Detailed Design @@ -303,148 +277,7 @@ Backup workflow: - Updates to [CSI PVC plugin](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/backup/actions/csi/pvc_action.go#L200) (Update the Execute method): ```go -// Retrieve the VGS label key from the backup spec. - vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey - // Check if the PVC has the specified VGS label key. - if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { - // PVC is marked for group snapshot. - existingVGS := p.findExistingVGS(backup.UID, vgsLabelKey, group, pvc.Namespace) - if existingVGS == nil { - // No existing VGS found; list all PVCs in the same group. - groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - // Extract the names of all grouped PVCs. - pvcNames := extractPVCNames(groupedPVCs) - // Create a new VGS object with the backup UID and group. - newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "snapshot.storage.k8s.io", - Resource: "volumegroupsnapshots", - }, - Namespace: newVGS.Namespace, - Name: newVGS.Name, - }) - } else { - // Existing VGS found; skip creating an individual VS. - p.log.Infof("PVC %s is part of existing VGS %s, skipping individual VolumeSnapshot creation", pvc.Name, existingVGS.Name) - } - } else { - // PVC is not part of a VGS group; create an individual VolumeSnapshot. - - //. - //. - //. - } - -// helper functions used - -// findExistingVGS checks for an existing VolumeGroupSnapshot (VGS) for the given backup UID and group in the namespace. -func (p *pvcBackupItemAction) findExistingVGS(backupUID types.UID, vgsLabelKey, group, namespace string) *VolumeGroupSnapshot { - var vgsList VolumeGroupSnapshotList - err := p.crClient.List( - context.TODO(), - &vgsList, - crclient.InNamespace(namespace), - crclient.MatchingLabels{ - "backup-uid": string(backupUID), - vgsLabelKey: group, - }, - ) - if err != nil { - p.log.Errorf("Error listing VGS: %v", err) - return nil - } - if len(vgsList.Items) > 0 { - return &vgsList.Items[0] - } - return nil -} - -// listGroupedPVCs returns all PVCs in the given namespace that have the specified VGS label key and group value. -func (p *pvcBackupItemAction) listGroupedPVCs(backup *velerov1api.Backup, namespace, vgsLabelKey, group string) ([]corev1api.PersistentVolumeClaim, error) { - var pvcList corev1api.PersistentVolumeClaimList - err := p.crClient.List(context.TODO(), &pvcList, crclient.InNamespace(namespace), crclient.MatchingLabels{vgsLabelKey: group}) - if err != nil { - return nil, err - } - return pvcList.Items, nil -} - -func (p *pvcBackupItemAction) createVolumeGroupSnapshot( - backup *velerov1api.Backup, - pvc corev1api.PersistentVolumeClaim, - pvcNames []string, - vgsLabelKey, group string, -) (*VolumeGroupSnapshot, error) { - // Generate a name for the new VGS object. - name := fmt.Sprintf("vgs-%s-%d", string(backup.UID), time.Now().Unix()) - - // Construct the VGS object - vgs := &VolumeGroupSnapshot{ - TypeMeta: metav1.TypeMeta{ - Kind: "VolumeGroupSnapshot", - APIVersion: "groupsnapshot.storage.k8s.io/v1beta1", - }, - ObjectMeta: metav1.ObjectMeta{ - Name: name, - Namespace: pvc.Namespace, - Annotations: map[string]string{ - "pvcList": strings.Join(pvcNames, ","), - }, - Labels: map[string]string{ - velerov1api.BackupUIDLabel: string(backup.UID), - vgsLabelKey: group, - }, - }, - Spec: VolumeGroupSnapshotSpec{ - // we will use default VGSClass for now - Source: VolumeGroupSnapshotSource{ - Selector: metav1.LabelSelector{ - MatchLabels: map[string]string{ - vgsLabelKey: group, - }, - }, - }, - }, - } - - // Create the VGS object - if err := p.crClient.Create(context.TODO(), vgs); err != nil { - return nil, err - } - return vgs, nil -} - -// extractPVCNames returns a slice of PVC names from the provided list. -func extractPVCNames(pvcs []corev1api.PersistentVolumeClaim) []string { - names := []string{} - for _, pvc := range pvcs { - names = append(names, pvc.Name) - } - return names -} - -``` - -- Add a new VolumeGroupSnapshot(VGS) BIA plugin -```go -// AppliesTo specifies that this plugin applies to VolumeGroupSnapshot objects. -func (p *vgsBackupItemAction) AppliesTo() (velero.ResourceSelector, error) { - return velero.ResourceSelector{ - IncludedResources: []string{"volumegroupsnapshots"}, - }, nil -} - -// Execute waits for all related VolumeSnapshot (VS) instances to be ready, then adds them -// along with the VolumeGroupSnapshotClass as additional backup items. -func (p *vgsBackupItemAction) Execute( +func (p *pvcBackupItemAction) Execute( item runtime.Unstructured, backup *velerov1api.Backup, ) ( @@ -454,55 +287,230 @@ func (p *vgsBackupItemAction) Execute( []velero.ResourceIdentifier, error, ) { - p.log.Infof("Starting VGS backup plugin Execute for item: %s", item.GetName()) + p.log.Info("Starting PVCBackupItemAction") - // Convert the unstructured object to a VolumeGroupSnapshot. - var vgs VolumeGroupSnapshot - if err := runtime.DefaultUnstructuredConverter.FromUnstructured(item.UnstructuredContent(), &vgs); err != nil { - return nil, nil, "", nil, errors.Wrap(err, "unable to convert item to VolumeGroupSnapshot") - } - - // Wait until all related VolumeSnapshots are ready. - p.log.Infof("Waiting for all related VolumeSnapshots for VGS %s to be ready", vgs.Name) - ready, err := waitForGroupedVolumeSnapshots(&vgs, vgs.Namespace, p.log) - if err != nil { - return nil, nil, "", nil, errors.Wrap(err, "error waiting for grouped VolumeSnapshots") - } - if !ready { - p.log.Infof("Not all related VolumeSnapshots are ready for VGS %s", vgs.Name) + if valid := p.validateBackup(*backup); !valid { return item, nil, "", nil, nil } - // Retrieve the related VolumeSnapshot instances. - groupedVS := getGroupedVolumeSnapshots(&vgs, vgs.Namespace) - var additionalItems []velero.ResourceIdentifier - for _, vs := range groupedVS { - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "snapshot.storage.k8s.io", - Resource: "volumesnapshots", - }, - Namespace: vs.Namespace, - Name: vs.Name, - }) + var pvc corev1api.PersistentVolumeClaim + if err := runtime.DefaultUnstructuredConverter.FromUnstructured( + item.UnstructuredContent(), &pvc, + ); err != nil { + return nil, nil, "", nil, errors.WithStack(err) } - // Add the VolumeGroupSnapshotClass as an additional item. - // Here we assume that the VGS spec defines the class name. - // We will be using the default VGSClass for now - vgsClassName := vgs.Spec.VolumeGroupSnapshotClassName - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "snapshot.storage.k8s.io", - Resource: "volumegroupsnapshotclasses", - }, - Namespace: "", - Name: vgsClassName, - }) + if valid, item, err := p.validatePVCandPV(pvc, item); !valid { + if err != nil { + return nil, nil, "", nil, err + } + return item, nil, "", nil, nil + } - p.log.Infof("VGS %s ready; adding %d additional items to backup", vgs.Name, len(additionalItems)) - return item, additionalItems, "", nil, nil + shouldSnapshot, err := volumehelper.ShouldPerformSnapshotWithBackup( + item, + kuberesource.PersistentVolumeClaims, + *backup, + p.crClient, + p.log, + ) + if err != nil { + return nil, nil, "", nil, err + } + if !shouldSnapshot { + p.log.Debugf("CSI plugin skip snapshot for PVC %s according to VolumeHelper setting", pvc.Namespace+"/"+pvc.Name) + return nil, nil, "", nil, nil + } + + var additionalItems []velero.ResourceIdentifier + operationID := "" + var itemToUpdate []velero.ResourceIdentifier + + // vsRef will be used to apply common labels/annotations + var vsRef *corev1api.ObjectReference + + // VGS branch: check if PVC has the VGS label key set on it. + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey + if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { + p.log.Infof("PVC %s has VGS label with group %s", pvc.Name, group) + // First, check if a VS created via a VGS workflow exists for this PVC. + existingVS, err := p.findExistingVSForBackup(backup.UID, backup.Name, pvc.Name, pvc.Namespace) + if err != nil { + return nil, nil, "", nil, err + } + if existingVS != nil && existingVS.Status.VolumeGroupSnapshotName != "" { + p.log.Infof("Existing VS %s found for PVC %s in group %s; skipping VGS creation", existingVS.Name, pvc.Name, group) + vsRef = &corev1api.ObjectReference{ + Namespace: existingVS.Namespace, + Name: existingVS.Name, + } + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "snapshot.storage.k8s.io", + Resource: "volumesnapshots", + }, + Namespace: existingVS.Namespace, + Name: existingVS.Name, + }) + } else { + // No existing VS found for the group; execute VGS creation workflow. + groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + pvcNames := extractPVCNames(groupedPVCs) + newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) + + // Wait for the VS objects created via VGS to have VolumeGroupSnapshotName in status. + if err := p.waitForVGSAssociatedVS(newVGS, pvc.Namespace, backup.Spec.CSISnapshotTimeout.Duration); err != nil { + return nil, nil, "", nil, err + } + // Update VS objects: remove VGS owner references and finalizers; add BackupName, BackupUUID and PVC name as labels. + if err := p.updateVGSCreatedVS(newVGS, backup); err != nil { + return nil, nil, "", nil, err + } + // Patch the VGSC deletionPolicy to Retain. + if err := p.patchVGSCDeletionPolicy(newVGS, pvc.Namespace); err != nil { + return nil, nil, "", nil, err + } + // Delete the VGS and VGSC objects to prevent cascading deletion. + if err := p.deleteVGSAndVGSC(newVGS, pvc.Namespace); err != nil { + return nil, nil, "", nil, err + } + + // Branch based on datamover flag. + if !boolptr.IsSetToTrue(backup.Spec.SnapshotMoveData) { + // Non-datamover: list VS objects created via VGS and use them. + vsList, err := p.listVSForVGSGroup(backup, pvc.Namespace, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + if len(vsList) == 0 { + return nil, nil, "", nil, errors.New("no VS objects found for VGS group " + group) + } + vsRef = &corev1api.ObjectReference{ + Namespace: vsList[0].Namespace, + Name: vsList[0].Name, + } + additionalItems = append(additionalItems, convertVSToResourceIdentifiers(vsList)...) + } else { + // Datamover: retrieve the VS for the PVC and create a DataUpload. + vs, err := p.getVSForPVC(backup, pvc, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + operationID = label.GetValidName(string(velerov1api.AsyncOperationIDPrefixDataUpload) + string(backup.UID) + "." + string(pvc.UID)) + dataUploadLog := p.log.WithFields(logrus.Fields{ + "Source PVC": fmt.Sprintf("%s/%s", pvc.Namespace, pvc.Name), + "VolumeSnapshot": fmt.Sprintf("%s/%s", vs.Namespace, vs.Name), + "Operation ID": operationID, + "Backup": backup.Name, + }) + // Wait until VS associated VSC snapshot handle is created. + _, err = csi.WaitUntilVSCHandleIsReady( + vs, + p.crClient, + p.log, + true, + backup.Spec.CSISnapshotTimeout.Duration, + ) + if err != nil { + dataUploadLog.Errorf("Fail to wait VolumeSnapshot turned to ReadyToUse: %s", err.Error()) + csi.CleanupVolumeSnapshot(vs, p.crClient, p.log) + return nil, nil, "", nil, errors.WithStack(err) + } + dataUploadLog.Info("Starting data upload of backup") + dataUpload, err := createDataUpload( + context.Background(), + backup, + p.crClient, + vs, + &pvc, + operationID, + ) + if err != nil { + dataUploadLog.WithError(err).Error("failed to submit DataUpload") + if deleteErr := p.crClient.Delete(context.TODO(), vs); deleteErr != nil { + if !apierrors.IsNotFound(deleteErr) { + dataUploadLog.WithError(deleteErr).Error("fail to delete VolumeSnapshot") + } + } + return item, nil, "", nil, nil + } else { + itemToUpdate = []velero.ResourceIdentifier{ + { + GroupResource: schema.GroupResource{ + Group: "velero.io", + Resource: "datauploads", + }, + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + }, + } + annotations[velerov1api.DataUploadNameAnnotation] = dataUpload.Namespace + "/" + dataUpload.Name + dataUploadLog.Info("DataUpload is submitted successfully.") + } + vsRef = &corev1api.ObjectReference{ + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + } + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "velero.io", + Resource: "datauploads", + }, + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + }) + } + } + } else { + // Legacy workflow: PVC does not have a VGS label; create an individual VolumeSnapshot. + vs, err := p.createVolumeSnapshot(pvc, backup) + if err != nil { + return nil, nil, "", nil, err + } + vsRef = vs + additionalItems = []velero.ResourceIdentifier{ + { + GroupResource: kuberesource.VolumeSnapshots, + Namespace: vs.Namespace, + Name: vs.Name, + }, + } + } + + labels := map[string]string{ + velerov1api.VolumeSnapshotLabel: vsRef.Name, + velerov1api.BackupNameLabel: backup.Name, + } + + annotations := map[string]string{ + velerov1api.VolumeSnapshotLabel: vsRef.Name, + velerov1api.MustIncludeAdditionalItemAnnotation: "true", + } + + kubeutil.AddAnnotations(&pvc.ObjectMeta, annotations) + kubeutil.AddLabels(&pvc.ObjectMeta, labels) + + p.log.Infof("Returning from PVCBackupItemAction with %d additionalItems to backup", len(additionalItems)) + for _, ai := range additionalItems { + p.log.Debugf("%s: %s", ai.GroupResource.String(), ai.Name) + } + + pvcMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&pvc) + if err != nil { + return nil, nil, "", nil, errors.WithStack(err) + } + + return &unstructured.Unstructured{Object: pvcMap}, + additionalItems, operationID, itemToUpdate, nil } + ``` ## Implementation From 2372c4ecf3a61e27799dbc7344252033c93e3b1b Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Tue, 8 Apr 2025 14:52:41 -0700 Subject: [PATCH 08/31] Update CSI plugin common branch flow and add mechanism to determine VGSClass Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 575 +++++++++++++++++--------------- 1 file changed, 301 insertions(+), 274 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 71e47d2ec..716a938b0 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -135,78 +135,101 @@ flowchart TD ``` #### Updates to CSI PVC plugin: -- When a PVC has a VGS label and no VS (created via VGS) exists: - - Create VGS: - - This triggers creation of the corresponding VGSC, VS, and VSC objects. - - Wait for VS Status: - - Wait until each VS (one per PVC in the group) has its volumeGroupSnapshotName set. This confirms that the snapshot controller has done its work. - - Update VS Objects: - - Remove owner references and VGS-related finalizers from the VS objects (decoupling them to prevent cascading deletion). - - Add backup metadata (BackupName, BackupUUID, PVC name) as labels. This metadata is later used to skip re-creating a VGS when another PVC of the same group is processed. - - Patch and Cleanup: - - Patch the VGSC deletionPolicy to Retain so that when you delete the VGSC, the underlying VSC (and the storage snapshots) remain. - - Delete the temporary VGS and VGSC objects. - - Branching: - - For non‑datamover cases, skip the creation of an individual VS (since it was created via VGS) and add the VS objects as additional items. - - For datamover cases, create DataUploads for the VS–PVC pair (using the VS created by the VGS workflow) and add those as additional items. - -- When a PVC has a VGS label and a VS created via an earlier VGS workflow already exists: - - List VS objects in the PVC’s namespace using labels (BackupUUID, BackupName, PVCName). - - Verify that a VS exists and that its status shows a non‑empty volumeGroupSnapshotName. - - If so, skip VGS (and VS) creation and continue with the legacy workflow. - - If a VS is found but it wasn’t created by the VGS workflow (i.e. it lacks the volumeGroupSnapshotName), then the backup for that PVC is failed, resulting in a partially failed backup. +The CSI PVC plugin now supports obtaining a VolumeSnapshot (VS) reference for a PVC in three ways, and then applies common branching for datamover and non‑datamover workflows: -- When a PVC does not have a VGS label: - - The legacy workflow is followed, creating an individual VolumeSnapshot as before. +- Scenario 1: PVC has a VGS label and no VS (created via the VGS workflow) exists for its volume group: + - Determine VGSClass: The plugin checks for CSI driver of all the grouped PVCs and then uses the corresponding VGSClass in VGS spec. If the grouped PVC has more than one CSI driver, VGS creation is skipped and backup fails. + - Create VGS: The plugin creates a new VolumeGroupSnapshot (VGS) for the PVC’s volume group. This action automatically triggers creation of the corresponding VGSC, VS, and VSC objects. + - Wait for VS Status: The plugin waits until each VS (one per PVC in the group) has its `volumeGroupSnapshotName` populated. This confirms that the snapshot controller has completed its work. `CSISnapshotTimeout` will be used here. + - Update VS Objects: Once the VS objects are provisioned, the plugin updates them by removing VGS owner references and VGS-related finalizers, and by adding backup metadata labels (including BackupName, BackupUUID, and PVC name). These labels are later used to detect an existing VS when processing another PVC of the same group. + - Patch and Cleanup: The plugin patches the deletionPolicy of the VGSC to "Retain" (ensuring that deletion of the VGSC does not remove the underlying VSC objects or storage snapshots) and then deletes the temporary VGS and VGSC objects. + +- Scenario 2: PVC has a VGS label and a VS created via an earlier VGS workflow already exists: + - The plugin lists VS objects in the PVC’s namespace using backup metadata labels (BackupUID, BackupName, and PVCName). + - It verifies that at least one VS has a non‑empty `volumeGroupSnapshotName` in its status. + - If such a VS exists, the plugin skips creating a new VGS (or VS) and proceeds with the legacy workflow using the existing VS. + - If a VS is found but its status does not indicate it was created by the VGS workflow (i.e. its `volumeGroupSnapshotName` is empty), the backup for that PVC is failed, resulting in a partially failed backup. +- Scenario 3: PVC does not have a VGS label: + - The legacy workflow is followed, and an individual VolumeSnapshot (VS) is created for the PVC. +- Common Branching for Datamover and Non‑datamover Workflows: + - Once a VS reference (`vsRef`) is determined—whether through the VGS workflow (Scenario 1 or 2) or the legacy workflow (Scenario 3)—the plugin then applies the common branching: + - Non‑datamover Case: The VS reference is directly added as an additional backup item. + + - Datamover Case: The plugin waits until the VS’s associated VSC snapshot handle is ready (using the configured CSISnapshotTimeout), then creates a DataUpload for the VS–PVC pair. The resulting DataUpload is then added as an additional backup item. ```mermaid flowchart TD -%% Section 1: Accept VGS Label from User + %% Section 1: Accept VGS Label from User subgraph Accept_Label - A1[User sets VGS label] - A2[User labels PVCs before backup] - A1 --> A2 + A1[User sets VGS label key using default velero.io/volume-group-snapshot or via server arg or Backup API spec] + A2[User labels PVCs before backup] + A1 --> A2 end -%% Section 2: PVC ItemBlockAction Plugin + %% Section 2: PVC ItemBlockAction Plugin Extension subgraph PVC_ItemBlockAction - B1[Check PVC is bound and has VolumeName] - B2[Add related PV to relatedItems] - B3[Add pods mounting PVC to relatedItems] - B4[Check if PVC has user-specified VGS label] - B5[List PVCs in namespace matching label criteria] - B6[Add matching PVCs to relatedItems] - B1 --> B2 --> B3 --> B4 - B4 -- Yes --> B5 - B5 --> B6 + B1[Check PVC is bound and has VolumeName] + B2[Add related PV to relatedItems] + B3[Add pods mounting PVC to relatedItems] + B4[Check if PVC has user-specified VGS label] + B5[List PVCs in namespace matching label criteria] + B6[Add matching PVCs to relatedItems] + B1 --> B2 --> B3 --> B4 + B4 -- Yes --> B5 + B5 --> B6 end -%% Section 3: CSI PVC Plugin Updates -subgraph CSI_PVC_Plugin -C1[For each PVC, check for VGS label] -C1 -- Yes --> C2[Case 1: VGS for volume group not yet created] -C2 -- True --> C3[Create new VGS triggering VGSC, VS and VSC creation] -C3 --> C4[Wait for VS objects to show volumeGroupSnapshotName using CSISnapshotTimeout] -C4 --> C5[Update VS objects: remove VGS owner refs and VGS finalizers; add BackupName, BackupUUID, PVC name as labels] -C5 --> C6[Patch VGSC deletionPolicy to Retain] -C6 --> C7[Delete VGS object] -C7 --> C8[Delete VGSC] -C8 --> C9[For non-datamover: Skip individual VS creation; add VS from VGS as additional item] -C8 --> C10[For datamover: Create DataUpload for VS-PVC pair; add DU as additional item] + %% Section 3: CSI PVC Plugin Updates + subgraph CSI_PVC_Plugin + C1[For each PVC, check for VGS label] + C1 -- Has VGS label --> C2[Determine scenario] + C1 -- No VGS label --> C16[Scenario 3: Legacy workflow - create individual VS] -C1 -- Yes --> C11[Case 2: VGS was created then deleted] -C11 --> C12[List VS using labels BackupUID, BackupName, PVCName] -C12 --> C13[Check if VS has non-empty volumeGroupSnapshotName] -C13 -- Yes --> C14[Skip VGS creation; use existing VS/DU; legacy workflow continues] -C13 -- No --> C15[Fail backup for PVC] + %% Scenario 1: No existing VS via VGS exists + subgraph Scenario1[Scenario 1: No existing VS via VGS] + S1[List grouped PVCs using VGS label] + S2[Determine CSI driver for grouped PVCs] + S3[If single CSI driver then select matching VGSClass; else fail backup] + S4[Create new VGS triggering VGSC, VS, and VSC creation] + S5[Wait for VS objects to have nonempty volumeGroupSnapshotName] + S6[Update VS objects; remove VGS owner refs and finalizers; add backup metadata labels] + S7[Patch VGSC deletionPolicy to Retain] + S8[Delete transient VGS and VGSC] + S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7 --> S8 + + end -C1 -- No --> C16[Case 3: PVC lacks VGS label; follow legacy workflow to create VS] -end + %% Scenario 2: Existing VS via VGS exists + subgraph Scenario2[Scenario 2: Existing VS via VGS exists] + S9[List VS objects using backup metadata - BackupUID, BackupName, PVCName] + S10[Check if any VS has nonempty volumeGroupSnapshotName] + S9 --> S10 + S10 -- Yes --> S11[Use existing VS] + S10 -- No --> S12[Fail backup for PVC] + end -%% Connect Main Sections -A2 --> B1 -B6 --> C1 + C2 -- Scenario1 applies --> S1 + C2 -- Scenario2 applies --> S9 + + %% Common Branch: After obtaining a VS reference + subgraph Common_Branch[Common Branch] + CB1[Obtain VS reference as vsRef] + CB2[If non-datamover, add vsRef as additional backup item] + CB3[If datamover, wait for VSC handle and create DataUpload; add DataUpload as additional backup item] + CB1 --> CB2 + CB1 --> CB3 + end + + %% Connect Scenario outcomes and legacy branch to the common branch + S8 --> CB1 + S11 --> CB1 + C16 --> CB1 + end + + %% Overall Flow Connections + A2 --> B1 + B6 --> C1 ``` @@ -278,239 +301,243 @@ Backup workflow: - Updates to [CSI PVC plugin](https://github.com/vmware-tanzu/velero/blob/512199723ff95d5016b32e91e3bf06b65f57d608/pkg/backup/actions/csi/pvc_action.go#L200) (Update the Execute method): ```go func (p *pvcBackupItemAction) Execute( - item runtime.Unstructured, - backup *velerov1api.Backup, + item runtime.Unstructured, + backup *velerov1api.Backup, ) ( - runtime.Unstructured, - []velero.ResourceIdentifier, - string, - []velero.ResourceIdentifier, - error, + runtime.Unstructured, + []velero.ResourceIdentifier, + string, + []velero.ResourceIdentifier, + error, ) { - p.log.Info("Starting PVCBackupItemAction") + p.log.Info("Starting PVCBackupItemAction") - if valid := p.validateBackup(*backup); !valid { - return item, nil, "", nil, nil - } + // Validate backup policy and PVC/PV + if valid := p.validateBackup(*backup); !valid { + return item, nil, "", nil, nil + } - var pvc corev1api.PersistentVolumeClaim - if err := runtime.DefaultUnstructuredConverter.FromUnstructured( - item.UnstructuredContent(), &pvc, - ); err != nil { - return nil, nil, "", nil, errors.WithStack(err) - } + var pvc corev1api.PersistentVolumeClaim + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(item.UnstructuredContent(), &pvc); err != nil { + return nil, nil, "", nil, errors.WithStack(err) + } + if valid, item, err := p.validatePVCandPV(pvc, item); !valid { + if err != nil { + return nil, nil, "", nil, err + } + return item, nil, "", nil, nil + } - if valid, item, err := p.validatePVCandPV(pvc, item); !valid { - if err != nil { - return nil, nil, "", nil, err - } - return item, nil, "", nil, nil - } + shouldSnapshot, err := volumehelper.ShouldPerformSnapshotWithBackup( + item, + kuberesource.PersistentVolumeClaims, + *backup, + p.crClient, + p.log, + ) + if err != nil { + return nil, nil, "", nil, err + } + if !shouldSnapshot { + p.log.Debugf("CSI plugin skip snapshot for PVC %s according to VolumeHelper setting", pvc.Namespace+"/"+pvc.Name) + return nil, nil, "", nil, nil + } - shouldSnapshot, err := volumehelper.ShouldPerformSnapshotWithBackup( - item, - kuberesource.PersistentVolumeClaims, - *backup, - p.crClient, - p.log, - ) - if err != nil { - return nil, nil, "", nil, err - } - if !shouldSnapshot { - p.log.Debugf("CSI plugin skip snapshot for PVC %s according to VolumeHelper setting", pvc.Namespace+"/"+pvc.Name) - return nil, nil, "", nil, nil - } + var additionalItems []velero.ResourceIdentifier + var operationID string + var itemToUpdate []velero.ResourceIdentifier - var additionalItems []velero.ResourceIdentifier - operationID := "" - var itemToUpdate []velero.ResourceIdentifier + // vsRef will be our common reference to the VolumeSnapshot (VS) + var vsRef *corev1api.ObjectReference - // vsRef will be used to apply common labels/annotations - var vsRef *corev1api.ObjectReference + // Retrieve the VGS label key from the backup spec. + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey - // VGS branch: check if PVC has the VGS label key set on it. - vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey - if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { - p.log.Infof("PVC %s has VGS label with group %s", pvc.Name, group) - // First, check if a VS created via a VGS workflow exists for this PVC. - existingVS, err := p.findExistingVSForBackup(backup.UID, backup.Name, pvc.Name, pvc.Namespace) - if err != nil { - return nil, nil, "", nil, err - } - if existingVS != nil && existingVS.Status.VolumeGroupSnapshotName != "" { - p.log.Infof("Existing VS %s found for PVC %s in group %s; skipping VGS creation", existingVS.Name, pvc.Name, group) - vsRef = &corev1api.ObjectReference{ - Namespace: existingVS.Namespace, - Name: existingVS.Name, - } - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "snapshot.storage.k8s.io", - Resource: "volumesnapshots", - }, - Namespace: existingVS.Namespace, - Name: existingVS.Name, - }) - } else { - // No existing VS found for the group; execute VGS creation workflow. - groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - pvcNames := extractPVCNames(groupedPVCs) - newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) - - // Wait for the VS objects created via VGS to have VolumeGroupSnapshotName in status. - if err := p.waitForVGSAssociatedVS(newVGS, pvc.Namespace, backup.Spec.CSISnapshotTimeout.Duration); err != nil { - return nil, nil, "", nil, err - } - // Update VS objects: remove VGS owner references and finalizers; add BackupName, BackupUUID and PVC name as labels. - if err := p.updateVGSCreatedVS(newVGS, backup); err != nil { - return nil, nil, "", nil, err - } - // Patch the VGSC deletionPolicy to Retain. - if err := p.patchVGSCDeletionPolicy(newVGS, pvc.Namespace); err != nil { - return nil, nil, "", nil, err - } - // Delete the VGS and VGSC objects to prevent cascading deletion. - if err := p.deleteVGSAndVGSC(newVGS, pvc.Namespace); err != nil { - return nil, nil, "", nil, err - } - - // Branch based on datamover flag. - if !boolptr.IsSetToTrue(backup.Spec.SnapshotMoveData) { - // Non-datamover: list VS objects created via VGS and use them. - vsList, err := p.listVSForVGSGroup(backup, pvc.Namespace, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - if len(vsList) == 0 { - return nil, nil, "", nil, errors.New("no VS objects found for VGS group " + group) - } - vsRef = &corev1api.ObjectReference{ - Namespace: vsList[0].Namespace, - Name: vsList[0].Name, - } - additionalItems = append(additionalItems, convertVSToResourceIdentifiers(vsList)...) - } else { - // Datamover: retrieve the VS for the PVC and create a DataUpload. - vs, err := p.getVSForPVC(backup, pvc, vgsLabelKey, group) - if err != nil { - return nil, nil, "", nil, err - } - operationID = label.GetValidName(string(velerov1api.AsyncOperationIDPrefixDataUpload) + string(backup.UID) + "." + string(pvc.UID)) - dataUploadLog := p.log.WithFields(logrus.Fields{ - "Source PVC": fmt.Sprintf("%s/%s", pvc.Namespace, pvc.Name), - "VolumeSnapshot": fmt.Sprintf("%s/%s", vs.Namespace, vs.Name), - "Operation ID": operationID, - "Backup": backup.Name, - }) - // Wait until VS associated VSC snapshot handle is created. - _, err = csi.WaitUntilVSCHandleIsReady( - vs, - p.crClient, - p.log, - true, - backup.Spec.CSISnapshotTimeout.Duration, - ) - if err != nil { - dataUploadLog.Errorf("Fail to wait VolumeSnapshot turned to ReadyToUse: %s", err.Error()) - csi.CleanupVolumeSnapshot(vs, p.crClient, p.log) - return nil, nil, "", nil, errors.WithStack(err) - } - dataUploadLog.Info("Starting data upload of backup") - dataUpload, err := createDataUpload( - context.Background(), - backup, - p.crClient, - vs, - &pvc, - operationID, - ) - if err != nil { - dataUploadLog.WithError(err).Error("failed to submit DataUpload") - if deleteErr := p.crClient.Delete(context.TODO(), vs); deleteErr != nil { - if !apierrors.IsNotFound(deleteErr) { - dataUploadLog.WithError(deleteErr).Error("fail to delete VolumeSnapshot") - } - } - return item, nil, "", nil, nil - } else { - itemToUpdate = []velero.ResourceIdentifier{ - { - GroupResource: schema.GroupResource{ - Group: "velero.io", - Resource: "datauploads", - }, - Namespace: dataUpload.Namespace, - Name: dataUpload.Name, - }, - } - annotations[velerov1api.DataUploadNameAnnotation] = dataUpload.Namespace + "/" + dataUpload.Name - dataUploadLog.Info("DataUpload is submitted successfully.") - } - vsRef = &corev1api.ObjectReference{ - Namespace: dataUpload.Namespace, - Name: dataUpload.Name, - } - additionalItems = append(additionalItems, velero.ResourceIdentifier{ - GroupResource: schema.GroupResource{ - Group: "velero.io", - Resource: "datauploads", - }, - Namespace: dataUpload.Namespace, - Name: dataUpload.Name, - }) - } - } - } else { - // Legacy workflow: PVC does not have a VGS label; create an individual VolumeSnapshot. - vs, err := p.createVolumeSnapshot(pvc, backup) - if err != nil { - return nil, nil, "", nil, err - } - vsRef = vs - additionalItems = []velero.ResourceIdentifier{ - { - GroupResource: kuberesource.VolumeSnapshots, - Namespace: vs.Namespace, - Name: vs.Name, - }, - } - } + // Check if the PVC has the user-specified VGS label. + if group, ok := pvc.Labels[vgsLabelKey]; ok && group != "" { + p.log.Infof("PVC %s has VGS label with group %s", pvc.Name, group) + // --- VGS branch --- + // 1. Check if a VS created via a VGS workflow exists for this PVC. + existingVS, err := p.findExistingVSForBackup(backup.UID, backup.Name, pvc.Name, pvc.Namespace) + if err != nil { + return nil, nil, "", nil, err + } + if existingVS != nil && existingVS.Status.VolumeGroupSnapshotName != "" { + p.log.Infof("Existing VS %s found for PVC %s in group %s; skipping VGS creation", existingVS.Name, pvc.Name, group) + vsRef = &corev1api.ObjectReference{ + Namespace: existingVS.Namespace, + Name: existingVS.Name, + } + } else { + // 2. No existing VS via VGS; execute VGS creation workflow. + groupedPVCs, err := p.listGroupedPVCs(backup, pvc.Namespace, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + pvcNames := extractPVCNames(groupedPVCs) + // Determine the CSI driver used by the grouped PVCs. + driver, err := p.determineCSIDriver(groupedPVCs) + if err != nil { + return nil, nil, "", nil, errors.Wrap(err, "failed to determine CSI driver for grouped PVCs") + } + if driver == "" { + return nil, nil, "", nil, errors.New("multiple CSI drivers found for grouped PVCs; failing backup") + } + // Retrieve the appropriate VGSClass for the CSI driver. + vgsClass := p.getVGSClassForDriver(driver) + p.log.Infof("Determined CSI driver %s with VGSClass %s for PVC group %s", driver, vgsClass, group) - labels := map[string]string{ - velerov1api.VolumeSnapshotLabel: vsRef.Name, - velerov1api.BackupNameLabel: backup.Name, - } + newVGS, err := p.createVolumeGroupSnapshot(backup, pvc, pvcNames, vgsLabelKey, group, vgsClass) + if err != nil { + return nil, nil, "", nil, err + } + p.log.Infof("Created new VGS %s for PVC group %s", newVGS.Name, group) + + // Wait for the VS objects created via VGS to have volumeGroupSnapshotName in status. + if err := p.waitForVGSAssociatedVS(newVGS, pvc.Namespace, backup.Spec.CSISnapshotTimeout.Duration); err != nil { + return nil, nil, "", nil, err + } + // Update the VS objects: remove VGS owner references and finalizers; add backup metadata labels. + if err := p.updateVGSCreatedVS(newVGS, backup); err != nil { + return nil, nil, "", nil, err + } + // Patch the VGSC deletionPolicy to Retain. + if err := p.patchVGSCDeletionPolicy(newVGS, pvc.Namespace); err != nil { + return nil, nil, "", nil, err + } + // Delete the VGS and VGSC + if err := p.deleteVGSAndVGSC(newVGS, pvc.Namespace); err != nil { + return nil, nil, "", nil, err + } + // Fetch the VS that was created for this PVC via VGS. + vs, err := p.getVSForPVC(backup, pvc, vgsLabelKey, group) + if err != nil { + return nil, nil, "", nil, err + } + vsRef = &corev1api.ObjectReference{ + Namespace: vs.Namespace, + Name: vs.Name, + } + } + } else { + // Legacy workflow: PVC does not have a VGS label; create an individual VS. + vs, err := p.createVolumeSnapshot(pvc, backup) + if err != nil { + return nil, nil, "", nil, err + } + vsRef = &corev1api.ObjectReference{ + Namespace: vs.Namespace, + Name: vs.Name, + } + } - annotations := map[string]string{ - velerov1api.VolumeSnapshotLabel: vsRef.Name, - velerov1api.MustIncludeAdditionalItemAnnotation: "true", - } + // --- Common Branch --- + // Now we have vsRef populated from one of the above cases. + // Branch further based on backup.Spec.SnapshotMoveData. + if boolptr.IsSetToTrue(backup.Spec.SnapshotMoveData) { + // Datamover case: + operationID = label.GetValidName( + string(velerov1api.AsyncOperationIDPrefixDataUpload) + string(backup.UID) + "." + string(pvc.UID), + ) + dataUploadLog := p.log.WithFields(logrus.Fields{ + "Source PVC": fmt.Sprintf("%s/%s", pvc.Namespace, pvc.Name), + "VolumeSnapshot": fmt.Sprintf("%s/%s", vsRef.Namespace, vsRef.Name), + "Operation ID": operationID, + "Backup": backup.Name, + }) + // Retrieve the current VS using vsRef + vs := &snapshotv1api.VolumeSnapshot{} + if err := p.crClient.Get(context.TODO(), crclient.ObjectKey{Namespace: vsRef.Namespace, Name: vsRef.Name}, vs); err != nil { + return nil, nil, "", nil, errors.Wrapf(err, "failed to get VolumeSnapshot %s", vsRef.Name) + } + // Wait until the VS-associated VSC snapshot handle is ready. + _, err := csi.WaitUntilVSCHandleIsReady( + vs, + p.crClient, + p.log, + true, + backup.Spec.CSISnapshotTimeout.Duration, + ) + if err != nil { + dataUploadLog.Errorf("Failed to wait for VolumeSnapshot to become ReadyToUse: %s", err.Error()) + csi.CleanupVolumeSnapshot(vs, p.crClient, p.log) + return nil, nil, "", nil, errors.WithStack(err) + } + dataUploadLog.Info("Starting data upload of backup") + dataUpload, err := createDataUpload( + context.Background(), + backup, + p.crClient, + vs, + &pvc, + operationID, + ) + if err != nil { + dataUploadLog.WithError(err).Error("Failed to submit DataUpload") + if deleteErr := p.crClient.Delete(context.TODO(), vs); deleteErr != nil && !apierrors.IsNotFound(deleteErr) { + dataUploadLog.WithError(deleteErr).Error("Failed to delete VolumeSnapshot") + } + return item, nil, "", nil, nil + } + dataUploadLog.Info("DataUpload submitted successfully") + itemToUpdate = []velero.ResourceIdentifier{ + { + GroupResource: schema.GroupResource{ + Group: "velero.io", + Resource: "datauploads", + }, + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + }, + } + annotations[velerov1api.DataUploadNameAnnotation] = dataUpload.Namespace + "/" + dataUpload.Name + // For the datamover case, add the dataUpload as an additional item directly. + vsRef = &corev1api.ObjectReference{ + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + } + additionalItems = append(additionalItems, velero.ResourceIdentifier{ + GroupResource: schema.GroupResource{ + Group: "velero.io", + Resource: "datauploads", + }, + Namespace: dataUpload.Namespace, + Name: dataUpload.Name, + }) + } else { + // Non-datamover case: + // Use vsRef for snapshot purposes. + additionalItems = append(additionalItems, convertVSToResourceIdentifiersFromRef(vsRef)...) + p.log.Infof("VolumeSnapshot additional item added for VS %s", vsRef.Name) + } - kubeutil.AddAnnotations(&pvc.ObjectMeta, annotations) - kubeutil.AddLabels(&pvc.ObjectMeta, labels) + // Update PVC metadata with common labels and annotations. + labels := map[string]string{ + velerov1api.VolumeSnapshotLabel: vsRef.Name, + velerov1api.BackupNameLabel: backup.Name, + } + annotations := map[string]string{ + velerov1api.VolumeSnapshotLabel: vsRef.Name, + velerov1api.MustIncludeAdditionalItemAnnotation: "true", + } + kubeutil.AddAnnotations(&pvc.ObjectMeta, annotations) + kubeutil.AddLabels(&pvc.ObjectMeta, labels) - p.log.Infof("Returning from PVCBackupItemAction with %d additionalItems to backup", len(additionalItems)) - for _, ai := range additionalItems { - p.log.Debugf("%s: %s", ai.GroupResource.String(), ai.Name) - } + p.log.Infof("Returning from PVCBackupItemAction with %d additionalItems to backup", len(additionalItems)) + for _, ai := range additionalItems { + p.log.Debugf("%s: %s", ai.GroupResource.String(), ai.Name) + } - pvcMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&pvc) - if err != nil { - return nil, nil, "", nil, errors.WithStack(err) - } + pvcMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&pvc) + if err != nil { + return nil, nil, "", nil, errors.WithStack(err) + } - return &unstructured.Unstructured{Object: pvcMap}, - additionalItems, operationID, itemToUpdate, nil + return &unstructured.Unstructured{Object: pvcMap}, + additionalItems, operationID, itemToUpdate, nil } + ``` ## Implementation From 71b889aa6efc9a569f0ca888794bca4f04e003fc Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Tue, 22 Apr 2025 13:16:58 -0700 Subject: [PATCH 09/31] Update VGSClass determination mechanism Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index 716a938b0..acc13455d 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -138,7 +138,30 @@ flowchart TD The CSI PVC plugin now supports obtaining a VolumeSnapshot (VS) reference for a PVC in three ways, and then applies common branching for datamover and non‑datamover workflows: - Scenario 1: PVC has a VGS label and no VS (created via the VGS workflow) exists for its volume group: - - Determine VGSClass: The plugin checks for CSI driver of all the grouped PVCs and then uses the corresponding VGSClass in VGS spec. If the grouped PVC has more than one CSI driver, VGS creation is skipped and backup fails. + - Determine VGSClass: The plugin will pick `VolumeGroupSnapshotClass` by following the same tier based precedence as it does for individual `VolumeSnapshotClasses`: + - Default by Label: Use the one VGSClass labeled + ```yaml + metadata: + labels: + velero.io/csi-volumegroupsnapshot-class: "true" + + ``` + whose `spec.driver` matches the CSI driver used by the PVCs. + - Backup‑level Override: If the Backup CR has an annotation + ```yaml + metadata: + annotations: + velero.io/csi-volumegroupsnapshot-class_: + ``` + (with equal to the PVCs’ CSI driver), use that class. + - PVC‑level Override: Finally, if the PVC itself carries an annotation + ```yaml + metadata: + annotations: + velero.io/csi-volume-group-snapshot-class: + ``` + and that class exists, use it. + At each step, if the plugin finds zero or multiple matching classes, VGS creation is skipped and backup fails. - Create VGS: The plugin creates a new VolumeGroupSnapshot (VGS) for the PVC’s volume group. This action automatically triggers creation of the corresponding VGSC, VS, and VSC objects. - Wait for VS Status: The plugin waits until each VS (one per PVC in the group) has its `volumeGroupSnapshotName` populated. This confirms that the snapshot controller has completed its work. `CSISnapshotTimeout` will be used here. - Update VS Objects: Once the VS objects are provisioned, the plugin updates them by removing VGS owner references and VGS-related finalizers, and by adding backup metadata labels (including BackupName, BackupUUID, and PVC name). These labels are later used to detect an existing VS when processing another PVC of the same group. From b30e43998aae479481d0b19b0b81ace94713748e Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Wed, 23 Apr 2025 19:52:22 -0700 Subject: [PATCH 10/31] Add notes regarding compat, perf, reqs and testing Signed-off-by: Shubham Pampattiwar --- design/volume-group-snapshot.md | 50 +++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 5 deletions(-) diff --git a/design/volume-group-snapshot.md b/design/volume-group-snapshot.md index acc13455d..83bc5a574 100644 --- a/design/volume-group-snapshot.md +++ b/design/volume-group-snapshot.md @@ -2,6 +2,16 @@ This proposal outlines the design and implementation plan for incorporating VolumeGroupSnapshot support into Velero. The enhancement will allow Velero to perform consistent, atomic snapshots of groups of Volumes using the new Kubernetes [VolumeGroupSnapshot API](https://kubernetes.io/blog/2024/12/18/kubernetes-1-32-volume-group-snapshot-beta/). This capability is especially critical for stateful applications that rely on multiple volumes to ensure data consistency, such as databases and analytics workloads. +## Glossary & Abbreviation + +Terminology used in this document: +- VGS: VolumeGroupSnapshot +- VS: VolumeSnapshot +- VGSC: VolumeGroupSnapshotContent +- VSC: VolumeSnapshotContent +- VGSClass: VolumeGroupSnapshotClass +- VSClass: VolumeSnapshotClass + ## Background Velero currently enables snapshot-based backups on an individual Volume basis through CSI drivers. However, modern stateful applications often require multiple volumes for data, logs, and backups. This distributed data architecture increases the risk of inconsistencies when volumes are captured individually. Kubernetes has introduced the VolumeGroupSnapshot(VGS) API [(KEP-3476)](https://github.com/kubernetes/enhancements/pull/1551), which allows for the atomic snapshotting of multiple volumes in a coordinated manner. By integrating this feature, Velero can offer enhanced disaster recovery for multi-volume applications, ensuring consistency across all related data. @@ -20,16 +30,23 @@ Velero currently enables snapshot-based backups on an individual Volume basis th ### Backup workflow: #### Accept the label to be used for VGS from the user: - Accept the label from the user, we will do this in 3 ways: - - Firstly, we will have a hard-coded default label key like `velero.io/volume-group-snapshot` that the users can directly use on their PVCs. - - Secondly, we will let the users override this default VGS label via a velero server arg, `--volume-group-nsaphot-label-key`, if needed. - - And Finally we will have the option to override the default label via Backup API spec, `backup.spec.volumeGroupSnapshotLabelKey` - - In all the instances, the VGS label key will be present on the backup spec, this makes the label key accessible to plugins during the execution of backup operation. + - Firstly, we will have a hard-coded default label key like `velero.io/volume-group-snapshot` that the users can directly use on their PVCs. + - Secondly, we will let the users override this default VGS label via a velero server arg, `--volume-group-nsaphot-label-key`, if needed. + - And Finally we will have the option to override the default label via Backup API spec, `backup.spec.volumeGroupSnapshotLabelKey` + - In all the instances, the VGS label key will be present on the backup spec, this makes the label key accessible to plugins during the execution of backup operation. - This label will enable velero to filter the PVC to be included in the VGS spec. - Users will have to label the PVCs before invoking the backup operation. - This label would act as a group identifier for the PVCs to be grouped under a specific VGS. - It will be used to collect the PVCs to be used for a particular instance of VGS object. -**Note:** Modifying or adding VGS label on PVCs during an active backup operation may lead to unexpected or undesirable backup results. To avoid inconsistencies, ensure PVC labels remain unchanged throughout the backup execution. +**Note:** + - Modifying or adding VGS label on PVCs during an active backup operation may lead to unexpected or undesirable backup results. To avoid inconsistencies, ensure PVC labels remain unchanged throughout the backup execution. + - Label Key Precedence: When determining which label key to use for grouping PVCs into a VolumeGroupSnapshot, Velero applies overrides in the following order (highest to lowest): + - Backup API spec (`backup.spec.volumeGroupSnapshotLabelKey`) + - Server flag (`--volume-group-snapshot-label-key`) + - Built-in default (`velero.io/volume-group-snapshot`) + + Whichever key wins this precedence is then injected into the Backup spec so that all Velero plugins can uniformly discover and use it during the backup execution. #### Changes to the Existing PVC ItemBlockAction plugin: - Currently the PVC IBA plugin is applied to PVCs and adds the RelatedItems for the particular PVC into the ItemBlock. - At first it checks whether the PVC is bound and VolumeName is non-empty. @@ -569,3 +586,26 @@ This design proposal is targeted for velero 1.16. The implementation of this proposed design is targeted for velero 1.17. +**Note:** +- VGS support isn't a requirement on restore. The design does not have any VGS related elements/considerations in the restore workflow. + +## Requirements and Assumptions +- Kubernetes Version: + - Minimum: v1.32.0 or later, since the VolumeGroupSnapshot API goes beta in 1.32. + - Assumption: CRDs for `VolumeGroupSnapshot`, `VolumeGroupSnapshotClass`, and `VolumeGroupSnapshotContent` are already installed. + +- VolumeGroupSnapshot API Availability: + - If the VGS API group (`groupsnapshot.storage.k8s.io/v1beta1`) is not present, Velero backup will fail. + +- CSI Driver Compatibility + - Only CSI drivers that implement the VolumeGroupSnapshot admission and controller support this feature. + - Upon VGS creation, we assume the driver will atomically snapshot all matching PVCs; if it does not, the plugin may time out. + +## Performance Considerations +- Use VGS if you have many similar volumes that must be snapped together and you want to minimize API/server load. +- Use individual VS if you have only a few volumes, or want one‐volume failures to be isolated. + +## Testing Strategy + +- Unit tests: We will add targeted unit tests to cover all new code paths—including existing-VS detection, VGS creation, legacy VS fallback, and error scenarios. +- E2E tests: For E2E we would need, a Kind cluster with a CSI driver that supports group snapshots, deploy an application with multiple PVCs, execute a Velero backup and restore, and verify that VGS is created, all underlying VS objects reach ReadyToUse, and every PVC is restored successfully. \ No newline at end of file From b7d997130d2f173076d352e64040373591d7b600 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Thu, 29 May 2025 17:06:20 +0800 Subject: [PATCH 11/31] issue 8960: implement PodVolume exposer for PVB/PVR Signed-off-by: Lyndon-Li --- changelogs/unreleased/8985-Lyndon-Li | 1 + pkg/cmd/cli/nodeagent/server.go | 4 +- .../pod_volume_backup_controller.go | 7 +- .../pod_volume_restore_controller.go | 7 +- pkg/exposer/host_path.go | 31 +- pkg/exposer/host_path_test.go | 72 ++- pkg/exposer/pod_volume.go | 404 ++++++++++++ pkg/exposer/pod_volume_test.go | 590 ++++++++++++++++++ pkg/exposer/types.go | 12 +- pkg/nodeagent/node_agent.go | 40 ++ pkg/nodeagent/node_agent_test.go | 161 +++++ pkg/util/kube/utils.go | 28 +- pkg/util/kube/utils_test.go | 25 +- 13 files changed, 1333 insertions(+), 49 deletions(-) create mode 100644 changelogs/unreleased/8985-Lyndon-Li create mode 100644 pkg/exposer/pod_volume.go create mode 100644 pkg/exposer/pod_volume_test.go diff --git a/changelogs/unreleased/8985-Lyndon-Li b/changelogs/unreleased/8985-Lyndon-Li new file mode 100644 index 000000000..7bb89251c --- /dev/null +++ b/changelogs/unreleased/8985-Lyndon-Li @@ -0,0 +1 @@ +Fix issue #8960, implement PodVolume exposer for PVB/PVR \ No newline at end of file diff --git a/pkg/cmd/cli/nodeagent/server.go b/pkg/cmd/cli/nodeagent/server.go index c7a2576c9..0dca03665 100644 --- a/pkg/cmd/cli/nodeagent/server.go +++ b/pkg/cmd/cli/nodeagent/server.go @@ -305,14 +305,14 @@ func (s *nodeAgentServer) run() { credentialGetter := &credentials.CredentialGetter{FromFile: credentialFileStore, FromSecret: credSecretStore} repoEnsurer := repository.NewEnsurer(s.mgr.GetClient(), s.logger, s.config.resourceTimeout) - pvbReconciler := controller.NewPodVolumeBackupReconciler(s.mgr.GetClient(), s.dataPathMgr, repoEnsurer, + pvbReconciler := controller.NewPodVolumeBackupReconciler(s.mgr.GetClient(), s.kubeClient, s.dataPathMgr, repoEnsurer, credentialGetter, s.nodeName, s.mgr.GetScheme(), s.metrics, s.logger) if err := pvbReconciler.SetupWithManager(s.mgr); err != nil { s.logger.Fatal(err, "unable to create controller", "controller", constant.ControllerPodVolumeBackup) } - if err = controller.NewPodVolumeRestoreReconciler(s.mgr.GetClient(), s.dataPathMgr, repoEnsurer, credentialGetter, s.logger).SetupWithManager(s.mgr); err != nil { + if err = controller.NewPodVolumeRestoreReconciler(s.mgr.GetClient(), s.kubeClient, s.dataPathMgr, repoEnsurer, credentialGetter, s.logger).SetupWithManager(s.mgr); err != nil { s.logger.WithError(err).Fatal("Unable to create the pod volume restore controller") } diff --git a/pkg/controller/pod_volume_backup_controller.go b/pkg/controller/pod_volume_backup_controller.go index 03cb3fe8e..254a39c91 100644 --- a/pkg/controller/pod_volume_backup_controller.go +++ b/pkg/controller/pod_volume_backup_controller.go @@ -29,6 +29,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/kubernetes" clocks "k8s.io/utils/clock" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" @@ -48,10 +49,11 @@ import ( const pVBRRequestor string = "pod-volume-backup-restore" // NewPodVolumeBackupReconciler creates the PodVolumeBackupReconciler instance -func NewPodVolumeBackupReconciler(client client.Client, dataPathMgr *datapath.Manager, ensurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter, +func NewPodVolumeBackupReconciler(client client.Client, kubeClient kubernetes.Interface, dataPathMgr *datapath.Manager, ensurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter, nodeName string, scheme *runtime.Scheme, metrics *metrics.ServerMetrics, logger logrus.FieldLogger) *PodVolumeBackupReconciler { return &PodVolumeBackupReconciler{ Client: client, + kubeClient: kubeClient, logger: logger.WithField("controller", "PodVolumeBackup"), repositoryEnsurer: ensurer, credentialGetter: credentialGetter, @@ -67,6 +69,7 @@ func NewPodVolumeBackupReconciler(client client.Client, dataPathMgr *datapath.Ma // PodVolumeBackupReconciler reconciles a PodVolumeBackup object type PodVolumeBackupReconciler struct { client.Client + kubeClient kubernetes.Interface scheme *runtime.Scheme clock clocks.WithTickerAndDelayedExecution metrics *metrics.ServerMetrics @@ -155,7 +158,7 @@ func (r *PodVolumeBackupReconciler) Reconcile(ctx context.Context, req ctrl.Requ return r.errorOut(ctx, &pvb, err, fmt.Sprintf("getting pod %s/%s", pvb.Spec.Pod.Namespace, pvb.Spec.Pod.Name), log) } - path, err := exposer.GetPodVolumeHostPath(ctx, &pod, pvb.Spec.Volume, r.Client, r.fileSystem, log) + path, err := exposer.GetPodVolumeHostPath(ctx, &pod, pvb.Spec.Volume, r.kubeClient, r.fileSystem, log) if err != nil { r.closeDataPath(ctx, pvb.Name) return r.errorOut(ctx, &pvb, err, "error exposing host path for pod volume", log) diff --git a/pkg/controller/pod_volume_restore_controller.go b/pkg/controller/pod_volume_restore_controller.go index f4645657f..e65d1b606 100644 --- a/pkg/controller/pod_volume_restore_controller.go +++ b/pkg/controller/pod_volume_restore_controller.go @@ -30,6 +30,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/kubernetes" clocks "k8s.io/utils/clock" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" @@ -49,10 +50,11 @@ import ( "github.com/vmware-tanzu/velero/pkg/util/filesystem" ) -func NewPodVolumeRestoreReconciler(client client.Client, dataPathMgr *datapath.Manager, ensurer *repository.Ensurer, +func NewPodVolumeRestoreReconciler(client client.Client, kubeClient kubernetes.Interface, dataPathMgr *datapath.Manager, ensurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter, logger logrus.FieldLogger) *PodVolumeRestoreReconciler { return &PodVolumeRestoreReconciler{ Client: client, + kubeClient: kubeClient, logger: logger.WithField("controller", "PodVolumeRestore"), repositoryEnsurer: ensurer, credentialGetter: credentialGetter, @@ -64,6 +66,7 @@ func NewPodVolumeRestoreReconciler(client client.Client, dataPathMgr *datapath.M type PodVolumeRestoreReconciler struct { client.Client + kubeClient kubernetes.Interface logger logrus.FieldLogger repositoryEnsurer *repository.Ensurer credentialGetter *credentials.CredentialGetter @@ -135,7 +138,7 @@ func (c *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Req return c.errorOut(ctx, pvr, err, "error to update status to in progress", log) } - volumePath, err := exposer.GetPodVolumeHostPath(ctx, pod, pvr.Spec.Volume, c.Client, c.fileSystem, log) + volumePath, err := exposer.GetPodVolumeHostPath(ctx, pod, pvr.Spec.Volume, c.kubeClient, c.fileSystem, log) if err != nil { c.closeDataPath(ctx, pvr.Name) return c.errorOut(ctx, pvr, err, "error exposing host path for pod volume", log) diff --git a/pkg/exposer/host_path.go b/pkg/exposer/host_path.go index d249dda39..a5668f8c0 100644 --- a/pkg/exposer/host_path.go +++ b/pkg/exposer/host_path.go @@ -19,13 +19,15 @@ package exposer import ( "context" "fmt" + "strings" "github.com/pkg/errors" "github.com/sirupsen/logrus" corev1api "k8s.io/api/core/v1" - ctrlclient "sigs.k8s.io/controller-runtime/pkg/client" + "k8s.io/client-go/kubernetes" "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/nodeagent" "github.com/vmware-tanzu/velero/pkg/uploader" "github.com/vmware-tanzu/velero/pkg/util/filesystem" "github.com/vmware-tanzu/velero/pkg/util/kube" @@ -37,17 +39,17 @@ var singlePathMatch = kube.SinglePathMatch // GetPodVolumeHostPath returns a path that can be accessed from the host for a given volume of a pod func GetPodVolumeHostPath(ctx context.Context, pod *corev1api.Pod, volumeName string, - cli ctrlclient.Client, fs filesystem.Interface, log logrus.FieldLogger) (datapath.AccessPoint, error) { + kubeClient kubernetes.Interface, fs filesystem.Interface, log logrus.FieldLogger) (datapath.AccessPoint, error) { logger := log.WithField("pod name", pod.Name).WithField("pod UID", pod.GetUID()).WithField("volume", volumeName) - volDir, err := getVolumeDirectory(ctx, logger, pod, volumeName, cli) + volDir, err := getVolumeDirectory(ctx, logger, pod, volumeName, kubeClient) if err != nil { return datapath.AccessPoint{}, errors.Wrapf(err, "error getting volume directory name for volume %s in pod %s", volumeName, pod.Name) } logger.WithField("volDir", volDir).Info("Got volume dir") - volMode, err := getVolumeMode(ctx, logger, pod, volumeName, cli) + volMode, err := getVolumeMode(ctx, logger, pod, volumeName, kubeClient) if err != nil { return datapath.AccessPoint{}, errors.Wrapf(err, "error getting volume mode for volume %s in pod %s", volumeName, pod.Name) } @@ -57,7 +59,7 @@ func GetPodVolumeHostPath(ctx context.Context, pod *corev1api.Pod, volumeName st volSubDir = "volumeDevices" } - pathGlob := fmt.Sprintf("/host_pods/%s/%s/*/%s", string(pod.GetUID()), volSubDir, volDir) + pathGlob := fmt.Sprintf("/%s/%s/%s/*/%s", nodeagent.HostPodVolumeMountPoint, string(pod.GetUID()), volSubDir, volDir) logger.WithField("pathGlob", pathGlob).Debug("Looking for path matching glob") path, err := singlePathMatch(pathGlob, fs, logger) @@ -72,3 +74,22 @@ func GetPodVolumeHostPath(ctx context.Context, pod *corev1api.Pod, volumeName st VolMode: volMode, }, nil } + +var getHostPodPath = nodeagent.GetHostPodPath + +func ExtractPodVolumeHostPath(ctx context.Context, path string, kubeClient kubernetes.Interface, veleroNamespace string, osType string) (string, error) { + podPath, err := getHostPodPath(ctx, kubeClient, veleroNamespace, osType) + if err != nil { + return "", errors.Wrap(err, "error getting host pod path from node-agent") + } + + if osType == kube.NodeOSWindows { + podPath = strings.Replace(podPath, "/", "\\", -1) + } + + if osType == kube.NodeOSWindows { + return strings.Replace(path, "\\"+nodeagent.HostPodVolumeMountPoint, podPath, 1), nil + } else { + return strings.Replace(path, "/"+nodeagent.HostPodVolumeMountPoint, podPath, 1), nil + } +} diff --git a/pkg/exposer/host_path_test.go b/pkg/exposer/host_path_test.go index 9faff5f14..411833f64 100644 --- a/pkg/exposer/host_path_test.go +++ b/pkg/exposer/host_path_test.go @@ -18,26 +18,29 @@ package exposer import ( "context" + "fmt" "testing" "github.com/pkg/errors" "github.com/sirupsen/logrus" "github.com/stretchr/testify/assert" corev1api "k8s.io/api/core/v1" - ctrlclient "sigs.k8s.io/controller-runtime/pkg/client" + "k8s.io/client-go/kubernetes" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/nodeagent" velerotest "github.com/vmware-tanzu/velero/pkg/test" "github.com/vmware-tanzu/velero/pkg/uploader" "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/kube" ) func TestGetPodVolumeHostPath(t *testing.T) { tests := []struct { name string - getVolumeDirFunc func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, ctrlclient.Client) (string, error) - getVolumeModeFunc func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, ctrlclient.Client) (uploader.PersistentVolumeMode, error) + getVolumeDirFunc func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, kubernetes.Interface) (string, error) + getVolumeModeFunc func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, kubernetes.Interface) (uploader.PersistentVolumeMode, error) pathMatchFunc func(string, filesystem.Interface, logrus.FieldLogger) (string, error) pod *corev1api.Pod pvc string @@ -45,7 +48,7 @@ func TestGetPodVolumeHostPath(t *testing.T) { }{ { name: "get volume dir fail", - getVolumeDirFunc: func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, ctrlclient.Client) (string, error) { + getVolumeDirFunc: func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, kubernetes.Interface) (string, error) { return "", errors.New("fake-error-1") }, pod: builder.ForPod(velerov1api.DefaultNamespace, "fake-pod-1").Result(), @@ -54,10 +57,10 @@ func TestGetPodVolumeHostPath(t *testing.T) { }, { name: "single path match fail", - getVolumeDirFunc: func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, ctrlclient.Client) (string, error) { + getVolumeDirFunc: func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, kubernetes.Interface) (string, error) { return "", nil }, - getVolumeModeFunc: func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, ctrlclient.Client) (uploader.PersistentVolumeMode, error) { + getVolumeModeFunc: func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, kubernetes.Interface) (uploader.PersistentVolumeMode, error) { return uploader.PersistentVolumeFilesystem, nil }, pathMatchFunc: func(string, filesystem.Interface, logrus.FieldLogger) (string, error) { @@ -69,7 +72,7 @@ func TestGetPodVolumeHostPath(t *testing.T) { }, { name: "get block volume dir success", - getVolumeDirFunc: func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, ctrlclient.Client) ( + getVolumeDirFunc: func(context.Context, logrus.FieldLogger, *corev1api.Pod, string, kubernetes.Interface) ( string, error) { return "fake-pvc-1", nil }, @@ -102,3 +105,58 @@ func TestGetPodVolumeHostPath(t *testing.T) { }) } } + +func TestExtractPodVolumeHostPath(t *testing.T) { + tests := []struct { + name string + getHostPodPathFunc func(context.Context, kubernetes.Interface, string, string) (string, error) + path string + osType string + expectedErr string + expected string + }{ + { + name: "get host pod path error", + getHostPodPathFunc: func(context.Context, kubernetes.Interface, string, string) (string, error) { + return "", errors.New("fake-error-1") + }, + + expectedErr: "error getting host pod path from node-agent: fake-error-1", + }, + { + name: "Windows os", + getHostPodPathFunc: func(context.Context, kubernetes.Interface, string, string) (string, error) { + return "/var/lib/kubelet/pods", nil + }, + path: fmt.Sprintf("\\%s\\pod-id-xxx\\volumes\\kubernetes.io~csi\\pvc-id-xxx\\mount", nodeagent.HostPodVolumeMountPoint), + osType: kube.NodeOSWindows, + expected: "\\var\\lib\\kubelet\\pods\\pod-id-xxx\\volumes\\kubernetes.io~csi\\pvc-id-xxx\\mount", + }, + { + name: "linux OS", + getHostPodPathFunc: func(context.Context, kubernetes.Interface, string, string) (string, error) { + return "/var/lib/kubelet/pods", nil + }, + path: fmt.Sprintf("/%s/pod-id-xxx/volumes/kubernetes.io~csi/pvc-id-xxx/mount", nodeagent.HostPodVolumeMountPoint), + osType: kube.NodeOSLinux, + expected: "/var/lib/kubelet/pods/pod-id-xxx/volumes/kubernetes.io~csi/pvc-id-xxx/mount", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + if test.getHostPodPathFunc != nil { + getHostPodPath = test.getHostPodPathFunc + } + + path, err := ExtractPodVolumeHostPath(context.Background(), test.path, nil, "", test.osType) + + if test.expectedErr != "" { + assert.EqualError(t, err, test.expectedErr) + } else { + assert.NoError(t, err) + assert.Equal(t, test.expected, path) + } + }) + } +} diff --git a/pkg/exposer/pod_volume.go b/pkg/exposer/pod_volume.go new file mode 100644 index 000000000..6e3910d12 --- /dev/null +++ b/pkg/exposer/pod_volume.go @@ -0,0 +1,404 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package exposer + +import ( + "context" + "fmt" + "time" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/wait" + "k8s.io/client-go/kubernetes" + "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/vmware-tanzu/velero/pkg/nodeagent" + "github.com/vmware-tanzu/velero/pkg/util/boolptr" + "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/kube" +) + +const ( + PodVolumeExposeTypeBackup = "pod-volume-backup" + PodVolumeExposeTypeRestore = "pod-volume-restore" +) + +// PodVolumeExposeParam define the input param for pod volume Expose +type PodVolumeExposeParam struct { + // ClientPodName is the name of pod to be backed up or restored + ClientPodName string + + // ClientNamespace is the namespace to be backed up or restored + ClientNamespace string + + // ClientNamespace is the pod volume for the client PVC + ClientPodVolume string + + // HostingPodLabels is the labels that are going to apply to the hosting pod + HostingPodLabels map[string]string + + // HostingPodAnnotations is the annotations that are going to apply to the hosting pod + HostingPodAnnotations map[string]string + + // Resources defines the resource requirements of the hosting pod + Resources corev1.ResourceRequirements + + // OperationTimeout specifies the time wait for resources operations in Expose + OperationTimeout time.Duration + + // Type specifies the type of the expose, either backup or erstore + Type string +} + +// PodVolumeExposer is the interfaces for a pod volume exposer +type PodVolumeExposer interface { + // Expose starts the process to a pod volume expose, the expose process may take long time + Expose(context.Context, corev1.ObjectReference, PodVolumeExposeParam) error + + // GetExposed polls the status of the expose. + // If the expose is accessible by the current caller, it waits the expose ready and returns the expose result. + // Otherwise, it returns nil as the expose result without an error. + GetExposed(context.Context, corev1.ObjectReference, client.Client, string, time.Duration) (*ExposeResult, error) + + // PeekExposed tests the status of the expose. + // If the expose is incomplete but not recoverable, it returns an error. + // Otherwise, it returns nil immediately. + PeekExposed(context.Context, corev1.ObjectReference) error + + // DiagnoseExpose generate the diagnostic info when the expose is not finished for a long time. + // If it finds any problem, it returns an string about the problem. + DiagnoseExpose(context.Context, corev1.ObjectReference) string + + // CleanUp cleans up any objects generated during the restore expose + CleanUp(context.Context, corev1.ObjectReference) +} + +// NewPodVolumeExposer creates a new instance of pod volume exposer +func NewPodVolumeExposer(kubeClient kubernetes.Interface, log logrus.FieldLogger) PodVolumeExposer { + return &podVolumeExposer{ + kubeClient: kubeClient, + fs: filesystem.NewFileSystem(), + log: log, + } +} + +type podVolumeExposer struct { + kubeClient kubernetes.Interface + fs filesystem.Interface + log logrus.FieldLogger +} + +var getPodVolumeHostPath = GetPodVolumeHostPath +var extractPodVolumeHostPath = ExtractPodVolumeHostPath + +func (e *podVolumeExposer) Expose(ctx context.Context, ownerObject corev1.ObjectReference, param PodVolumeExposeParam) error { + curLog := e.log.WithFields(logrus.Fields{ + "owner": ownerObject.Name, + "client pod": param.ClientPodName, + "client pod volume": param.ClientPodVolume, + "client namespace": param.ClientNamespace, + "type": param.Type, + }) + + pod, err := e.kubeClient.CoreV1().Pods(param.ClientNamespace).Get(ctx, param.ClientPodName, metav1.GetOptions{}) + if err != nil { + return errors.Wrapf(err, "error getting client pod %s", param.ClientPodName) + } + + if pod.Spec.NodeName == "" { + return errors.Errorf("client pod %s doesn't have a node name", pod.Name) + } + + nodeOS, err := kube.GetNodeOS(ctx, pod.Spec.NodeName, e.kubeClient.CoreV1()) + if err != nil { + return errors.Wrapf(err, "error getting OS for node %s", pod.Spec.NodeName) + } + + curLog.Infof("Client pod is running in node %s, os %s", pod.Spec.NodeName, nodeOS) + + path, err := getPodVolumeHostPath(ctx, pod, param.ClientPodVolume, e.kubeClient, e.fs, e.log) + if err != nil { + return errors.Wrapf(err, "error to get pod volume path") + } + + path.ByPath, err = extractPodVolumeHostPath(ctx, path.ByPath, e.kubeClient, ownerObject.Namespace, nodeOS) + if err != nil { + return errors.Wrapf(err, "error to extract pod volume path") + } + + curLog.WithField("path", path).Infof("Host path is retrieved for pod %s, volume %s", param.ClientPodName, param.ClientPodVolume) + + hostingPod, err := e.createHostingPod(ctx, ownerObject, param.Type, path.ByPath, param.OperationTimeout, param.HostingPodLabels, param.HostingPodAnnotations, pod.Spec.NodeName, param.Resources, nodeOS) + if err != nil { + return errors.Wrapf(err, "error to create hosting pod") + } + + defer func() { + if err != nil { + kube.DeletePodIfAny(ctx, e.kubeClient.CoreV1(), hostingPod.Name, hostingPod.Namespace, curLog) + } + }() + + curLog.WithField("pod name", hostingPod.Name).Info("Hosting pod is created") + + return nil +} + +func (e *podVolumeExposer) GetExposed(ctx context.Context, ownerObject corev1.ObjectReference, nodeClient client.Client, nodeName string, timeout time.Duration) (*ExposeResult, error) { + hostingPodName := ownerObject.Name + + containerName := string(ownerObject.UID) + volumeName := string(ownerObject.UID) + + curLog := e.log.WithFields(logrus.Fields{ + "owner": ownerObject.Name, + "node": nodeName, + }) + + var updated *corev1.Pod + err := wait.PollUntilContextTimeout(ctx, 2*time.Second, timeout, true, func(ctx context.Context) (bool, error) { + pod := &corev1.Pod{} + err := nodeClient.Get(ctx, types.NamespacedName{ + Namespace: ownerObject.Namespace, + Name: hostingPodName, + }, pod) + + if err != nil { + return false, errors.Wrapf(err, "error to get pod %s/%s", ownerObject.Namespace, hostingPodName) + } + + if pod.Status.Phase != corev1.PodRunning { + return false, nil + } + + updated = pod + + return true, nil + }) + + if err != nil { + if apierrors.IsNotFound(err) { + curLog.WithField("hosting pod", hostingPodName).Debug("Hosting pod is not running in the current node") + return nil, nil + } else { + return nil, errors.Wrapf(err, "error to wait for rediness of pod %s", hostingPodName) + } + } + + curLog.WithField("pod", updated.Name).Infof("Hosting pod is in running state in node %s", updated.Spec.NodeName) + + return &ExposeResult{ByPod: ExposeByPod{ + HostingPod: updated, + HostingContainer: containerName, + VolumeName: volumeName, + }}, nil +} + +func (e *podVolumeExposer) PeekExposed(ctx context.Context, ownerObject corev1.ObjectReference) error { + hostingPodName := ownerObject.Name + + curLog := e.log.WithFields(logrus.Fields{ + "owner": ownerObject.Name, + }) + + pod, err := e.kubeClient.CoreV1().Pods(ownerObject.Namespace).Get(ctx, hostingPodName, metav1.GetOptions{}) + if apierrors.IsNotFound(err) { + return nil + } + + if err != nil { + curLog.WithError(err).Warnf("error to peek hosting pod %s", hostingPodName) + return nil + } + + if podFailed, message := kube.IsPodUnrecoverable(pod, curLog); podFailed { + return errors.New(message) + } + + return nil +} + +func (e *podVolumeExposer) DiagnoseExpose(ctx context.Context, ownerObject corev1.ObjectReference) string { + hostingPodName := ownerObject.Name + + diag := "begin diagnose pod volume exposer\n" + + pod, err := e.kubeClient.CoreV1().Pods(ownerObject.Namespace).Get(ctx, hostingPodName, metav1.GetOptions{}) + if err != nil { + pod = nil + diag += fmt.Sprintf("error getting hosting pod %s, err: %v\n", hostingPodName, err) + } + + if pod != nil { + diag += kube.DiagnosePod(pod) + + if pod.Spec.NodeName != "" { + if err := nodeagent.KbClientIsRunningInNode(ctx, ownerObject.Namespace, pod.Spec.NodeName, e.kubeClient); err != nil { + diag += fmt.Sprintf("node-agent is not running in node %s, err: %v\n", pod.Spec.NodeName, err) + } + } + } + + diag += "end diagnose pod volume exposer" + + return diag +} + +func (e *podVolumeExposer) CleanUp(ctx context.Context, ownerObject corev1.ObjectReference) { + restorePodName := ownerObject.Name + kube.DeletePodIfAny(ctx, e.kubeClient.CoreV1(), restorePodName, ownerObject.Namespace, e.log) +} + +func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject corev1.ObjectReference, exposeType string, hostPath string, + operationTimeout time.Duration, label map[string]string, annotation map[string]string, selectedNode string, resources corev1.ResourceRequirements, nodeOS string) (*corev1.Pod, error) { + hostingPodName := ownerObject.Name + + containerName := string(ownerObject.UID) + clientVolumeName := string(ownerObject.UID) + clientVolumePath := "/" + clientVolumeName + + podInfo, err := getInheritedPodInfo(ctx, e.kubeClient, ownerObject.Namespace, nodeOS) + if err != nil { + return nil, errors.Wrap(err, "error to get inherited pod info from node-agent") + } + + var gracePeriod int64 + mountPropagation := corev1.MountPropagationHostToContainer + volumeMounts := []corev1.VolumeMount{{ + Name: clientVolumeName, + MountPath: clientVolumePath, + MountPropagation: &mountPropagation, + }} + volumeMounts = append(volumeMounts, podInfo.volumeMounts...) + + volumes := []corev1.Volume{{ + Name: clientVolumeName, + VolumeSource: corev1.VolumeSource{ + HostPath: &corev1.HostPathVolumeSource{ + Path: hostPath, + }, + }, + }} + volumes = append(volumes, podInfo.volumes...) + + if label == nil { + label = make(map[string]string) + } + + args := []string{ + fmt.Sprintf("--volume-path=%s", clientVolumePath), + fmt.Sprintf("--resource-timeout=%s", operationTimeout.String()), + } + + command := []string{ + "/velero", + "pod-volume", + } + + if exposeType == PodVolumeExposeTypeBackup { + args = append(args, fmt.Sprintf("--pod-volume-backup=%s", ownerObject.Name)) + command = append(command, "backup") + label[podGroupLabel] = podGroupPodVolumeBackup + } else { + args = append(args, fmt.Sprintf("--pod-volume-restore=%s", ownerObject.Name)) + command = append(command, "restore") + label[podGroupLabel] = podGroupPodVolumeRestore + } + + args = append(args, podInfo.logFormatArgs...) + args = append(args, podInfo.logLevelArgs...) + + var securityCtx *corev1.PodSecurityContext + nodeSelector := map[string]string{} + podOS := corev1.PodOS{} + toleration := []corev1.Toleration{} + if nodeOS == kube.NodeOSWindows { + userID := "ContainerAdministrator" + securityCtx = &corev1.PodSecurityContext{ + WindowsOptions: &corev1.WindowsSecurityContextOptions{ + RunAsUserName: &userID, + }, + } + + nodeSelector[kube.NodeOSLabel] = kube.NodeOSWindows + podOS.Name = kube.NodeOSWindows + + toleration = append(toleration, corev1.Toleration{ + Key: "os", + Operator: "Equal", + Effect: "NoSchedule", + Value: "windows", + }) + } else { + userID := int64(0) + securityCtx = &corev1.PodSecurityContext{ + RunAsUser: &userID, + } + + nodeSelector[kube.NodeOSLabel] = kube.NodeOSLinux + podOS.Name = kube.NodeOSLinux + } + + pod := &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Name: hostingPodName, + Namespace: ownerObject.Namespace, + OwnerReferences: []metav1.OwnerReference{ + { + APIVersion: ownerObject.APIVersion, + Kind: ownerObject.Kind, + Name: ownerObject.Name, + UID: ownerObject.UID, + Controller: boolptr.True(), + }, + }, + Labels: label, + Annotations: annotation, + }, + Spec: corev1.PodSpec{ + NodeSelector: nodeSelector, + OS: &podOS, + Containers: []corev1.Container{ + { + Name: containerName, + Image: podInfo.image, + ImagePullPolicy: corev1.PullNever, + Command: command, + Args: args, + VolumeMounts: volumeMounts, + Env: podInfo.env, + EnvFrom: podInfo.envFrom, + Resources: resources, + }, + }, + ServiceAccountName: podInfo.serviceAccount, + TerminationGracePeriodSeconds: &gracePeriod, + Volumes: volumes, + NodeName: selectedNode, + RestartPolicy: corev1.RestartPolicyNever, + SecurityContext: securityCtx, + Tolerations: toleration, + }, + } + + return e.kubeClient.CoreV1().Pods(ownerObject.Namespace).Create(ctx, pod, metav1.CreateOptions{}) +} diff --git a/pkg/exposer/pod_volume_test.go b/pkg/exposer/pod_volume_test.go new file mode 100644 index 000000000..8f4abab67 --- /dev/null +++ b/pkg/exposer/pod_volume_test.go @@ -0,0 +1,590 @@ +package exposer + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/assert" + velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/datapath" + velerotest "github.com/vmware-tanzu/velero/pkg/test" + "github.com/vmware-tanzu/velero/pkg/util/filesystem" + appsv1api "k8s.io/api/apps/v1" + corev1api "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/kubernetes/fake" + clientFake "sigs.k8s.io/controller-runtime/pkg/client/fake" +) + +func TestPodVolumeExpose(t *testing.T) { + backup := &velerov1.Backup{ + TypeMeta: metav1.TypeMeta{ + APIVersion: velerov1.SchemeGroupVersion.String(), + Kind: "Backup", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1.DefaultNamespace, + Name: "fake-backup", + UID: "fake-uid", + }, + } + + podWithNoNode := builder.ForPod("fake-ns", "fake-client-pod").Result() + podWithNode := builder.ForPod("fake-ns", "fake-client-pod").NodeName("fake-node").Result() + + node := builder.ForNode("fake-node").Result() + + daemonSet := &appsv1api.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "velero", + Name: "node-agent", + }, + TypeMeta: metav1.TypeMeta{ + Kind: "DaemonSet", + APIVersion: appsv1api.SchemeGroupVersion.String(), + }, + Spec: appsv1api.DaemonSetSpec{ + Template: corev1api.PodTemplateSpec{ + Spec: corev1api.PodSpec{ + Containers: []corev1api.Container{ + { + Name: "node-agent", + }, + }, + }, + }, + }, + } + + tests := []struct { + name string + snapshotClientObj []runtime.Object + kubeClientObj []runtime.Object + ownerBackup *velerov1.Backup + exposeParam PodVolumeExposeParam + funcGetPodVolumeHostPath func(context.Context, *corev1api.Pod, string, kubernetes.Interface, filesystem.Interface, logrus.FieldLogger) (datapath.AccessPoint, error) + funcExtractPodVolumeHostPath func(context.Context, string, kubernetes.Interface, string, string) (string, error) + err string + }{ + { + name: "get client pod fail", + ownerBackup: backup, + exposeParam: PodVolumeExposeParam{ + ClientNamespace: "fake-ns", + ClientPodName: "fake-client-pod", + }, + err: "error getting client pod fake-client-pod: pods \"fake-client-pod\" not found", + }, + { + name: "client pod with no node name", + ownerBackup: backup, + exposeParam: PodVolumeExposeParam{ + ClientNamespace: "fake-ns", + ClientPodName: "fake-client-pod", + }, + kubeClientObj: []runtime.Object{ + podWithNoNode, + }, + err: "client pod fake-client-pod doesn't have a node name", + }, + { + name: "get node os fail", + ownerBackup: backup, + exposeParam: PodVolumeExposeParam{ + ClientNamespace: "fake-ns", + ClientPodName: "fake-client-pod", + }, + kubeClientObj: []runtime.Object{ + podWithNode, + }, + err: "error getting OS for node fake-node: error getting node fake-node: nodes \"fake-node\" not found", + }, + { + name: "get pod volume path fail", + ownerBackup: backup, + exposeParam: PodVolumeExposeParam{ + ClientNamespace: "fake-ns", + ClientPodName: "fake-client-pod", + ClientPodVolume: "fake-client-volume", + }, + kubeClientObj: []runtime.Object{ + podWithNode, + node, + }, + funcGetPodVolumeHostPath: func(context.Context, *corev1api.Pod, string, kubernetes.Interface, filesystem.Interface, logrus.FieldLogger) (datapath.AccessPoint, error) { + return datapath.AccessPoint{}, errors.New("fake-get-pod-volume-path-error") + }, + err: "error to get pod volume path: fake-get-pod-volume-path-error", + }, + { + name: "extract pod volume path fail", + ownerBackup: backup, + exposeParam: PodVolumeExposeParam{ + ClientNamespace: "fake-ns", + ClientPodName: "fake-client-pod", + ClientPodVolume: "fake-client-volume", + }, + kubeClientObj: []runtime.Object{ + podWithNode, + node, + }, + funcGetPodVolumeHostPath: func(context.Context, *corev1api.Pod, string, kubernetes.Interface, filesystem.Interface, logrus.FieldLogger) (datapath.AccessPoint, error) { + return datapath.AccessPoint{ + ByPath: "/var/lib/kubelet/pods/pod-id-xxx/volumes/kubernetes.io~csi/pvc-id-xxx/mount", + }, nil + }, + funcExtractPodVolumeHostPath: func(context.Context, string, kubernetes.Interface, string, string) (string, error) { + return "", errors.New("fake-extract-error") + }, + err: "error to extract pod volume path: fake-extract-error", + }, + { + name: "create hosting pod fail", + ownerBackup: backup, + exposeParam: PodVolumeExposeParam{ + ClientNamespace: "fake-ns", + ClientPodName: "fake-client-pod", + ClientPodVolume: "fake-client-volume", + }, + kubeClientObj: []runtime.Object{ + podWithNode, + node, + }, + funcGetPodVolumeHostPath: func(context.Context, *corev1api.Pod, string, kubernetes.Interface, filesystem.Interface, logrus.FieldLogger) (datapath.AccessPoint, error) { + return datapath.AccessPoint{ + ByPath: "/host_pods/pod-id-xxx/volumes/kubernetes.io~csi/pvc-id-xxx/mount", + }, nil + }, + funcExtractPodVolumeHostPath: func(context.Context, string, kubernetes.Interface, string, string) (string, error) { + return "/var/lib/kubelet/pods/pod-id-xxx/volumes/kubernetes.io~csi/pvc-id-xxx/mount", nil + }, + err: "error to create hosting pod: error to get inherited pod info from node-agent: error to get node-agent pod template: error to get node-agent daemonset: daemonsets.apps \"node-agent\" not found", + }, + { + name: "succeed", + ownerBackup: backup, + exposeParam: PodVolumeExposeParam{ + ClientNamespace: "fake-ns", + ClientPodName: "fake-client-pod", + ClientPodVolume: "fake-client-volume", + }, + kubeClientObj: []runtime.Object{ + podWithNode, + node, + daemonSet, + }, + funcGetPodVolumeHostPath: func(context.Context, *corev1api.Pod, string, kubernetes.Interface, filesystem.Interface, logrus.FieldLogger) (datapath.AccessPoint, error) { + return datapath.AccessPoint{ + ByPath: "/host_pods/pod-id-xxx/volumes/kubernetes.io~csi/pvc-id-xxx/mount", + }, nil + }, + funcExtractPodVolumeHostPath: func(context.Context, string, kubernetes.Interface, string, string) (string, error) { + return "/var/lib/kubelet/pods/pod-id-xxx/volumes/kubernetes.io~csi/pvc-id-xxx/mount", nil + }, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fakeKubeClient := fake.NewSimpleClientset(test.kubeClientObj...) + + exposer := podVolumeExposer{ + kubeClient: fakeKubeClient, + log: velerotest.NewLogger(), + } + + var ownerObject corev1api.ObjectReference + if test.ownerBackup != nil { + ownerObject = corev1api.ObjectReference{ + Kind: test.ownerBackup.Kind, + Namespace: test.ownerBackup.Namespace, + Name: test.ownerBackup.Name, + UID: test.ownerBackup.UID, + APIVersion: test.ownerBackup.APIVersion, + } + } + + if test.funcGetPodVolumeHostPath != nil { + getPodVolumeHostPath = test.funcGetPodVolumeHostPath + } + + if test.funcExtractPodVolumeHostPath != nil { + extractPodVolumeHostPath = test.funcExtractPodVolumeHostPath + } + + err := exposer.Expose(context.Background(), ownerObject, test.exposeParam) + if err == nil { + assert.NoError(t, err) + + _, err = exposer.kubeClient.CoreV1().Pods(ownerObject.Namespace).Get(context.Background(), ownerObject.Name, metav1.GetOptions{}) + assert.NoError(t, err) + } else { + assert.EqualError(t, err, test.err) + } + }) + } +} + +func TestGetPodVolumeExpose(t *testing.T) { + backup := &velerov1.Backup{ + TypeMeta: metav1.TypeMeta{ + APIVersion: velerov1.SchemeGroupVersion.String(), + Kind: "Backup", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1.DefaultNamespace, + Name: "fake-backup", + UID: "fake-uid", + }, + } + + backupPodNotRunning := builder.ForPod(backup.Namespace, backup.Name).Result() + backupPodRunning := builder.ForPod(backup.Namespace, backup.Name).Phase(corev1api.PodRunning).Result() + + scheme := runtime.NewScheme() + corev1api.AddToScheme(scheme) + + tests := []struct { + name string + kubeClientObj []runtime.Object + ownerBackup *velerov1.Backup + nodeName string + Timeout time.Duration + err string + expectedResult *ExposeResult + }{ + { + name: "backup pod is not found", + ownerBackup: backup, + nodeName: "fake-node", + }, + { + name: "wait backup pod running fail", + ownerBackup: backup, + nodeName: "fake-node", + kubeClientObj: []runtime.Object{ + backupPodNotRunning, + }, + Timeout: time.Second, + err: "error to wait for rediness of pod fake-backup: context deadline exceeded", + }, + { + name: "succeed", + ownerBackup: backup, + nodeName: "fake-node", + kubeClientObj: []runtime.Object{ + backupPodRunning, + }, + Timeout: time.Second, + expectedResult: &ExposeResult{ + ByPod: ExposeByPod{ + HostingPod: backupPodRunning, + VolumeName: string(backup.UID), + }, + }, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fakeKubeClient := fake.NewSimpleClientset(test.kubeClientObj...) + + fakeClientBuilder := clientFake.NewClientBuilder() + fakeClientBuilder = fakeClientBuilder.WithScheme(scheme) + + fakeClient := fakeClientBuilder.WithRuntimeObjects(test.kubeClientObj...).Build() + + exposer := podVolumeExposer{ + kubeClient: fakeKubeClient, + log: velerotest.NewLogger(), + } + + var ownerObject corev1api.ObjectReference + if test.ownerBackup != nil { + ownerObject = corev1api.ObjectReference{ + Kind: test.ownerBackup.Kind, + Namespace: test.ownerBackup.Namespace, + Name: test.ownerBackup.Name, + UID: test.ownerBackup.UID, + APIVersion: test.ownerBackup.APIVersion, + } + } + + result, err := exposer.GetExposed(context.Background(), ownerObject, fakeClient, test.nodeName, test.Timeout) + if test.err == "" { + assert.NoError(t, err) + + if test.expectedResult == nil { + assert.Nil(t, result) + } else { + assert.NoError(t, err) + assert.Equal(t, test.expectedResult.ByPod.VolumeName, result.ByPod.VolumeName) + assert.Equal(t, test.expectedResult.ByPod.HostingPod.Name, result.ByPod.HostingPod.Name) + } + } else { + assert.EqualError(t, err, test.err) + } + }) + } +} + +func TestPodVolumePeekExpose(t *testing.T) { + backup := &velerov1.Backup{ + TypeMeta: metav1.TypeMeta{ + APIVersion: velerov1.SchemeGroupVersion.String(), + Kind: "Backup", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1.DefaultNamespace, + Name: "fake-backup", + UID: "fake-uid", + }, + } + + backupPodUrecoverable := &corev1api.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: backup.Namespace, + Name: backup.Name, + }, + Status: corev1api.PodStatus{ + Phase: corev1api.PodFailed, + }, + } + + backupPod := &corev1api.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: backup.Namespace, + Name: backup.Name, + }, + } + + scheme := runtime.NewScheme() + corev1api.AddToScheme(scheme) + + tests := []struct { + name string + kubeClientObj []runtime.Object + ownerBackup *velerov1.Backup + err string + }{ + { + name: "backup pod is not found", + ownerBackup: backup, + }, + { + name: "pod is unrecoverable", + ownerBackup: backup, + kubeClientObj: []runtime.Object{ + backupPodUrecoverable, + }, + err: "Pod is in abnormal state [Failed], message []", + }, + { + name: "succeed", + ownerBackup: backup, + kubeClientObj: []runtime.Object{ + backupPod, + }, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fakeKubeClient := fake.NewSimpleClientset(test.kubeClientObj...) + + exposer := podVolumeExposer{ + kubeClient: fakeKubeClient, + log: velerotest.NewLogger(), + } + + var ownerObject corev1api.ObjectReference + if test.ownerBackup != nil { + ownerObject = corev1api.ObjectReference{ + Kind: test.ownerBackup.Kind, + Namespace: test.ownerBackup.Namespace, + Name: test.ownerBackup.Name, + UID: test.ownerBackup.UID, + APIVersion: test.ownerBackup.APIVersion, + } + } + + err := exposer.PeekExposed(context.Background(), ownerObject) + if test.err == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, test.err) + } + }) + } +} + +func TestPodVolumeDiagnoseExpose(t *testing.T) { + backup := &velerov1.Backup{ + TypeMeta: metav1.TypeMeta{ + APIVersion: velerov1.SchemeGroupVersion.String(), + Kind: "Backup", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1.DefaultNamespace, + Name: "fake-backup", + UID: "fake-uid", + }, + } + + backupPodWithoutNodeName := corev1api.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1.DefaultNamespace, + Name: "fake-backup", + OwnerReferences: []metav1.OwnerReference{ + { + APIVersion: backup.APIVersion, + Kind: backup.Kind, + Name: backup.Name, + UID: backup.UID, + }, + }, + }, + Status: corev1api.PodStatus{ + Phase: corev1api.PodPending, + Conditions: []corev1api.PodCondition{ + { + Type: corev1api.PodInitialized, + Status: corev1api.ConditionTrue, + Message: "fake-pod-message", + }, + }, + }, + } + + backupPodWithNodeName := corev1api.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1.DefaultNamespace, + Name: "fake-backup", + OwnerReferences: []metav1.OwnerReference{ + { + APIVersion: backup.APIVersion, + Kind: backup.Kind, + Name: backup.Name, + UID: backup.UID, + }, + }, + }, + Spec: corev1api.PodSpec{ + NodeName: "fake-node", + }, + Status: corev1api.PodStatus{ + Phase: corev1api.PodPending, + Conditions: []corev1api.PodCondition{ + { + Type: corev1api.PodInitialized, + Status: corev1api.ConditionTrue, + Message: "fake-pod-message", + }, + }, + }, + } + + nodeAgentPod := corev1api.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1.DefaultNamespace, + Name: "node-agent-pod-1", + Labels: map[string]string{"role": "node-agent"}, + }, + Spec: corev1api.PodSpec{ + NodeName: "fake-node", + }, + Status: corev1api.PodStatus{ + Phase: corev1api.PodRunning, + }, + } + + tests := []struct { + name string + ownerBackup *velerov1.Backup + kubeClientObj []runtime.Object + snapshotClientObj []runtime.Object + expected string + }{ + { + name: "no pod", + ownerBackup: backup, + expected: `begin diagnose pod volume exposer +error getting hosting pod fake-backup, err: pods "fake-backup" not found +end diagnose pod volume exposer`, + }, + { + name: "pod without node name, pvc without volume name, vs without status", + ownerBackup: backup, + kubeClientObj: []runtime.Object{ + &backupPodWithoutNodeName, + }, + expected: `begin diagnose pod volume exposer +Pod velero/fake-backup, phase Pending, node name +Pod condition Initialized, status True, reason , message fake-pod-message +end diagnose pod volume exposer`, + }, + { + name: "pod without node name", + ownerBackup: backup, + kubeClientObj: []runtime.Object{ + &backupPodWithoutNodeName, + }, + expected: `begin diagnose pod volume exposer +Pod velero/fake-backup, phase Pending, node name +Pod condition Initialized, status True, reason , message fake-pod-message +end diagnose pod volume exposer`, + }, + { + name: "pod with node name, no node agent", + ownerBackup: backup, + kubeClientObj: []runtime.Object{ + &backupPodWithNodeName, + }, + expected: `begin diagnose pod volume exposer +Pod velero/fake-backup, phase Pending, node name fake-node +Pod condition Initialized, status True, reason , message fake-pod-message +node-agent is not running in node fake-node, err: daemonset pod not found in running state in node fake-node +end diagnose pod volume exposer`, + }, + { + name: "pod with node name, node agent is running", + ownerBackup: backup, + kubeClientObj: []runtime.Object{ + &backupPodWithNodeName, + &nodeAgentPod, + }, + expected: `begin diagnose pod volume exposer +Pod velero/fake-backup, phase Pending, node name fake-node +Pod condition Initialized, status True, reason , message fake-pod-message +end diagnose pod volume exposer`, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + fakeKubeClient := fake.NewSimpleClientset(tt.kubeClientObj...) + e := &podVolumeExposer{ + kubeClient: fakeKubeClient, + log: velerotest.NewLogger(), + } + var ownerObject corev1api.ObjectReference + if tt.ownerBackup != nil { + ownerObject = corev1api.ObjectReference{ + Kind: tt.ownerBackup.Kind, + Namespace: tt.ownerBackup.Namespace, + Name: tt.ownerBackup.Name, + UID: tt.ownerBackup.UID, + APIVersion: tt.ownerBackup.APIVersion, + } + } + + diag := e.DiagnoseExpose(context.Background(), ownerObject) + assert.Equal(t, tt.expected, diag) + }) + } +} diff --git a/pkg/exposer/types.go b/pkg/exposer/types.go index 5cdb9d497..b8f875a88 100644 --- a/pkg/exposer/types.go +++ b/pkg/exposer/types.go @@ -21,11 +21,13 @@ import ( ) const ( - AccessModeFileSystem = "by-file-system" - AccessModeBlock = "by-block-device" - podGroupLabel = "velero.io/exposer-pod-group" - podGroupSnapshot = "snapshot-exposer" - podGroupGenericRestore = "generic-restore-exposer" + AccessModeFileSystem = "by-file-system" + AccessModeBlock = "by-block-device" + podGroupLabel = "velero.io/exposer-pod-group" + podGroupSnapshot = "snapshot-exposer" + podGroupGenericRestore = "generic-restore-exposer" + podGroupPodVolumeBackup = "pod-volume-backup" + podGroupPodVolumeRestore = "pod-volume-restore" ) // ExposeResult defines the result of expose. diff --git a/pkg/nodeagent/node_agent.go b/pkg/nodeagent/node_agent.go index 34b782cbc..1c0378901 100644 --- a/pkg/nodeagent/node_agent.go +++ b/pkg/nodeagent/node_agent.go @@ -41,6 +41,12 @@ const ( // nodeAgentRole marks pods with node-agent role on all nodes. nodeAgentRole = "node-agent" + + // hostPodVolume is the name of the volume in node-agent for host-pod mount + hostPodVolume = "host-pods" + + // HostPodVolumeMountPoint is the mount point of the volume in node-agent for host-pod mount + HostPodVolumeMountPoint = "host_pods" ) var ( @@ -249,3 +255,37 @@ func GetAnnotationValue(ctx context.Context, kubeClient kubernetes.Interface, na return val, nil } + +func GetHostPodPath(ctx context.Context, kubeClient kubernetes.Interface, namespace string, osType string) (string, error) { + dsName := daemonSet + if osType == kube.NodeOSWindows { + dsName = daemonsetWindows + } + + ds, err := kubeClient.AppsV1().DaemonSets(namespace).Get(ctx, dsName, metav1.GetOptions{}) + if err != nil { + return "", errors.Wrapf(err, "error getting daemonset %s", dsName) + } + + var volume *corev1api.Volume + for _, v := range ds.Spec.Template.Spec.Volumes { + if v.Name == hostPodVolume { + volume = &v + break + } + } + + if volume == nil { + return "", errors.New("host pod volume is not found") + } + + if volume.HostPath == nil { + return "", errors.New("host pod volume is not a host path volume") + } + + if volume.HostPath.Path == "" { + return "", errors.New("host pod volume path is empty") + } + + return volume.HostPath.Path, nil +} diff --git a/pkg/nodeagent/node_agent_test.go b/pkg/nodeagent/node_agent_test.go index 9a78ca033..118f36ecf 100644 --- a/pkg/nodeagent/node_agent_test.go +++ b/pkg/nodeagent/node_agent_test.go @@ -590,3 +590,164 @@ func TestGetAnnotationValue(t *testing.T) { }) } } + +func TestGetHostPodPath(t *testing.T) { + daemonSet := &appsv1api.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "fake-ns", + Name: "node-agent", + }, + TypeMeta: metav1.TypeMeta{ + Kind: "DaemonSet", + }, + } + + daemonSetWithHostPodVolume := &appsv1api.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "fake-ns", + Name: "node-agent", + }, + TypeMeta: metav1.TypeMeta{ + Kind: "DaemonSet", + }, + Spec: appsv1api.DaemonSetSpec{ + Template: corev1api.PodTemplateSpec{ + Spec: corev1api.PodSpec{ + Volumes: []corev1api.Volume{ + { + Name: hostPodVolume, + }, + }, + }, + }, + }, + } + + daemonSetWithHostPodVolumeAndEmptyPath := &appsv1api.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "fake-ns", + Name: "node-agent", + }, + TypeMeta: metav1.TypeMeta{ + Kind: "DaemonSet", + }, + Spec: appsv1api.DaemonSetSpec{ + Template: corev1api.PodTemplateSpec{ + Spec: corev1api.PodSpec{ + Volumes: []corev1api.Volume{ + { + Name: hostPodVolume, + VolumeSource: corev1api.VolumeSource{ + HostPath: &corev1api.HostPathVolumeSource{}, + }, + }, + }, + }, + }, + }, + } + + daemonSetWithHostPodVolumeAndValidPath := &appsv1api.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "fake-ns", + Name: "node-agent", + }, + TypeMeta: metav1.TypeMeta{ + Kind: "DaemonSet", + }, + Spec: appsv1api.DaemonSetSpec{ + Template: corev1api.PodTemplateSpec{ + Spec: corev1api.PodSpec{ + Volumes: []corev1api.Volume{ + { + Name: hostPodVolume, + VolumeSource: corev1api.VolumeSource{ + HostPath: &corev1api.HostPathVolumeSource{ + Path: "/var/lib/kubelet/pods", + }, + }, + }, + }, + }, + }, + }, + } + + tests := []struct { + name string + kubeClientObj []runtime.Object + namespace string + osType string + expectedValue string + expectErr string + }{ + { + name: "ds get error", + namespace: "fake-ns", + osType: kube.NodeOSWindows, + kubeClientObj: []runtime.Object{ + daemonSet, + }, + expectErr: "error getting daemonset node-agent-windows: daemonsets.apps \"node-agent-windows\" not found", + }, + { + name: "no host pod volume", + namespace: "fake-ns", + osType: kube.NodeOSLinux, + kubeClientObj: []runtime.Object{ + daemonSet, + }, + expectErr: "host pod volume is not found", + }, + { + name: "no host pod volume path", + namespace: "fake-ns", + osType: kube.NodeOSLinux, + kubeClientObj: []runtime.Object{ + daemonSetWithHostPodVolume, + }, + expectErr: "host pod volume is not a host path volume", + }, + { + name: "empty host pod volume path", + namespace: "fake-ns", + osType: kube.NodeOSLinux, + kubeClientObj: []runtime.Object{ + daemonSetWithHostPodVolumeAndEmptyPath, + }, + expectErr: "host pod volume path is empty", + }, + { + name: "succeed", + namespace: "fake-ns", + osType: kube.NodeOSLinux, + kubeClientObj: []runtime.Object{ + daemonSetWithHostPodVolumeAndValidPath, + }, + expectedValue: "/var/lib/kubelet/pods", + }, + { + name: "succeed on empty os type", + namespace: "fake-ns", + kubeClientObj: []runtime.Object{ + daemonSetWithHostPodVolumeAndValidPath, + }, + expectedValue: "/var/lib/kubelet/pods", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fakeKubeClient := fake.NewSimpleClientset(test.kubeClientObj...) + + path, err := GetHostPodPath(context.TODO(), fakeKubeClient, test.namespace, test.osType) + + if test.expectErr == "" { + assert.NoError(t, err) + assert.Equal(t, test.expectedValue, path) + } else { + assert.EqualError(t, err, test.expectErr) + } + }) + } +} diff --git a/pkg/util/kube/utils.go b/pkg/util/kube/utils.go index 5d64f117b..eab3193fc 100644 --- a/pkg/util/kube/utils.go +++ b/pkg/util/kube/utils.go @@ -25,7 +25,6 @@ import ( "github.com/pkg/errors" "github.com/sirupsen/logrus" corev1api "k8s.io/api/core/v1" - storagev1api "k8s.io/api/storage/v1" apiextv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" apiextv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1" apierrors "k8s.io/apimachinery/pkg/api/errors" @@ -33,8 +32,8 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/wait" + "k8s.io/client-go/kubernetes" corev1client "k8s.io/client-go/kubernetes/typed/core/v1" - "sigs.k8s.io/controller-runtime/pkg/client" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" "github.com/vmware-tanzu/velero/pkg/label" @@ -148,8 +147,8 @@ func EnsureNamespaceExistsAndIsReady(namespace *corev1api.Namespace, client core // GetVolumeDirectory gets the name of the directory on the host, under /var/lib/kubelet/pods//volumes/, // where the specified volume lives. // For volumes with a CSIVolumeSource, append "/mount" to the directory name. -func GetVolumeDirectory(ctx context.Context, log logrus.FieldLogger, pod *corev1api.Pod, volumeName string, cli client.Client) (string, error) { - pvc, pv, volume, err := GetPodPVCVolume(ctx, log, pod, volumeName, cli) +func GetVolumeDirectory(ctx context.Context, log logrus.FieldLogger, pod *corev1api.Pod, volumeName string, kubeClient kubernetes.Interface) (string, error) { + pvc, pv, volume, err := GetPodPVCVolume(ctx, log, pod, volumeName, kubeClient) if err != nil { // This case implies the administrator created the PV and attached it directly, without PVC. // Note that only one VolumeSource can be populated per Volume on a pod @@ -164,7 +163,7 @@ func GetVolumeDirectory(ctx context.Context, log logrus.FieldLogger, pod *corev1 // Most common case is that we have a PVC VolumeSource, and we need to check the PV it points to for a CSI source. // PV's been created with a CSI source. - isProvisionedByCSI, err := isProvisionedByCSI(log, pv, cli) + isProvisionedByCSI, err := isProvisionedByCSI(log, pv, kubeClient) if err != nil { return "", errors.WithStack(err) } @@ -179,9 +178,9 @@ func GetVolumeDirectory(ctx context.Context, log logrus.FieldLogger, pod *corev1 } // GetVolumeMode gets the uploader.PersistentVolumeMode of the volume. -func GetVolumeMode(ctx context.Context, log logrus.FieldLogger, pod *corev1api.Pod, volumeName string, cli client.Client) ( +func GetVolumeMode(ctx context.Context, log logrus.FieldLogger, pod *corev1api.Pod, volumeName string, kubeClient kubernetes.Interface) ( uploader.PersistentVolumeMode, error) { - _, pv, _, err := GetPodPVCVolume(ctx, log, pod, volumeName, cli) + _, pv, _, err := GetPodPVCVolume(ctx, log, pod, volumeName, kubeClient) if err != nil { if err == ErrorPodVolumeIsNotPVC { @@ -198,7 +197,7 @@ func GetVolumeMode(ctx context.Context, log logrus.FieldLogger, pod *corev1api.P // GetPodPVCVolume gets the PVC, PV and volume for a pod volume name. // Returns pod volume in case of ErrorPodVolumeIsNotPVC error -func GetPodPVCVolume(ctx context.Context, log logrus.FieldLogger, pod *corev1api.Pod, volumeName string, cli client.Client) ( +func GetPodPVCVolume(ctx context.Context, log logrus.FieldLogger, pod *corev1api.Pod, volumeName string, kubeClient kubernetes.Interface) ( *corev1api.PersistentVolumeClaim, *corev1api.PersistentVolume, *corev1api.Volume, error) { var volume *corev1api.Volume @@ -217,14 +216,12 @@ func GetPodPVCVolume(ctx context.Context, log logrus.FieldLogger, pod *corev1api return nil, nil, volume, ErrorPodVolumeIsNotPVC // There is a pod volume but it is not a PVC } - pvc := &corev1api.PersistentVolumeClaim{} - err := cli.Get(ctx, client.ObjectKey{Namespace: pod.Namespace, Name: volume.VolumeSource.PersistentVolumeClaim.ClaimName}, pvc) + pvc, err := kubeClient.CoreV1().PersistentVolumeClaims(pod.Namespace).Get(ctx, volume.VolumeSource.PersistentVolumeClaim.ClaimName, metav1.GetOptions{}) if err != nil { return nil, nil, nil, errors.WithStack(err) } - pv := &corev1api.PersistentVolume{} - err = cli.Get(ctx, client.ObjectKey{Name: pvc.Spec.VolumeName}, pv) + pv, err := kubeClient.CoreV1().PersistentVolumes().Get(ctx, pvc.Spec.VolumeName, metav1.GetOptions{}) if err != nil { return nil, nil, nil, errors.WithStack(err) } @@ -235,7 +232,7 @@ func GetPodPVCVolume(ctx context.Context, log logrus.FieldLogger, pod *corev1api // isProvisionedByCSI function checks whether this is a CSI PV by annotation. // Either "pv.kubernetes.io/provisioned-by" or "pv.kubernetes.io/migrated-to" indicates // PV is provisioned by CSI. -func isProvisionedByCSI(log logrus.FieldLogger, pv *corev1api.PersistentVolume, kbClient client.Client) (bool, error) { +func isProvisionedByCSI(log logrus.FieldLogger, pv *corev1api.PersistentVolume, kubeClient kubernetes.Interface) (bool, error) { if pv.Spec.CSI != nil { return true, nil } @@ -245,10 +242,11 @@ func isProvisionedByCSI(log logrus.FieldLogger, pv *corev1api.PersistentVolume, driverName := pv.Annotations[KubeAnnDynamicallyProvisioned] migratedDriver := pv.Annotations[KubeAnnMigratedTo] if len(driverName) > 0 || len(migratedDriver) > 0 { - list := &storagev1api.CSIDriverList{} - if err := kbClient.List(context.TODO(), list); err != nil { + list, err := kubeClient.StorageV1().CSIDrivers().List(context.TODO(), metav1.ListOptions{}) + if err != nil { return false, err } + for _, driver := range list.Items { if driverName == driver.Name || migratedDriver == driver.Name { log.Debugf("the annotation %s or %s equals to %s indicates the volume is provisioned by a CSI driver", KubeAnnDynamicallyProvisioned, KubeAnnMigratedTo, driver.Name) diff --git a/pkg/util/kube/utils_test.go b/pkg/util/kube/utils_test.go index b36ab8ef2..37f990778 100644 --- a/pkg/util/kube/utils_test.go +++ b/pkg/util/kube/utils_test.go @@ -34,11 +34,12 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" - "sigs.k8s.io/controller-runtime/pkg/client/fake" "github.com/vmware-tanzu/velero/pkg/builder" velerotest "github.com/vmware-tanzu/velero/pkg/test" "github.com/vmware-tanzu/velero/pkg/uploader" + + "k8s.io/client-go/kubernetes/fake" ) func TestNamespaceAndName(t *testing.T) { @@ -216,17 +217,18 @@ func TestGetVolumeDirectorySuccess(t *testing.T) { ObjectMeta: metav1.ObjectMeta{Name: "csi.test.com"}, } for _, tc := range tests { - clientBuilder := fake.NewClientBuilder().WithLists(&storagev1api.CSIDriverList{Items: []storagev1api.CSIDriver{csiDriver}}) - + objs := []runtime.Object{&csiDriver} if tc.pvc != nil { - clientBuilder = clientBuilder.WithObjects(tc.pvc) + objs = append(objs, tc.pvc) } if tc.pv != nil { - clientBuilder = clientBuilder.WithObjects(tc.pv) + objs = append(objs, tc.pv) } + fakeKubeClient := fake.NewSimpleClientset(objs...) + // Function under test - dir, err := GetVolumeDirectory(context.Background(), logrus.StandardLogger(), tc.pod, tc.pod.Spec.Volumes[0].Name, clientBuilder.Build()) + dir, err := GetVolumeDirectory(context.Background(), logrus.StandardLogger(), tc.pod, tc.pod.Spec.Volumes[0].Name, fakeKubeClient) require.NoError(t, err) assert.Equal(t, tc.want, dir) @@ -264,17 +266,18 @@ func TestGetVolumeModeSuccess(t *testing.T) { } for _, tc := range tests { - clientBuilder := fake.NewClientBuilder() - + objs := []runtime.Object{} if tc.pvc != nil { - clientBuilder = clientBuilder.WithObjects(tc.pvc) + objs = append(objs, tc.pvc) } if tc.pv != nil { - clientBuilder = clientBuilder.WithObjects(tc.pv) + objs = append(objs, tc.pv) } + fakeKubeClient := fake.NewSimpleClientset(objs...) + // Function under test - mode, err := GetVolumeMode(context.Background(), logrus.StandardLogger(), tc.pod, tc.pod.Spec.Volumes[0].Name, clientBuilder.Build()) + mode, err := GetVolumeMode(context.Background(), logrus.StandardLogger(), tc.pod, tc.pod.Spec.Volumes[0].Name, fakeKubeClient) require.NoError(t, err) assert.Equal(t, tc.want, mode) From f604a5da48c7dd13608c0e5ce12febe1199d367d Mon Sep 17 00:00:00 2001 From: Xun Jiang Date: Mon, 26 May 2025 17:38:16 +0800 Subject: [PATCH 12/31] Add BSL status check for backup/restore operations. Signed-off-by: Xun Jiang --- changelogs/unreleased/8976-blackpiglet | 1 + pkg/controller/backup_controller.go | 8 +++++ pkg/controller/backup_controller_test.go | 29 ++++++++++++--- pkg/controller/backup_deletion_controller.go | 12 ++++--- .../backup_deletion_controller_test.go | 35 ++++++++++++++++++- pkg/controller/backup_sync_controller.go | 5 +++ pkg/controller/backup_sync_controller_test.go | 21 +++++++++++ pkg/controller/gc_controller.go | 9 +++++ pkg/controller/gc_controller_test.go | 12 +++++-- pkg/controller/restore_controller.go | 10 ++++++ pkg/controller/restore_controller_test.go | 26 +++++++++++--- pkg/util/velero/velero.go | 6 ++++ pkg/util/velero/velero_test.go | 11 ++++++ 13 files changed, 168 insertions(+), 17 deletions(-) create mode 100644 changelogs/unreleased/8976-blackpiglet diff --git a/changelogs/unreleased/8976-blackpiglet b/changelogs/unreleased/8976-blackpiglet new file mode 100644 index 000000000..c5aab6a51 --- /dev/null +++ b/changelogs/unreleased/8976-blackpiglet @@ -0,0 +1 @@ +Add BSL status check for backup/restore operations. diff --git a/pkg/controller/backup_controller.go b/pkg/controller/backup_controller.go index 9ae3b3590..11ced989c 100644 --- a/pkg/controller/backup_controller.go +++ b/pkg/controller/backup_controller.go @@ -56,6 +56,7 @@ import ( kubeutil "github.com/vmware-tanzu/velero/pkg/util/kube" "github.com/vmware-tanzu/velero/pkg/util/logging" "github.com/vmware-tanzu/velero/pkg/util/results" + veleroutil "github.com/vmware-tanzu/velero/pkg/util/velero" ) const ( @@ -417,6 +418,13 @@ func (b *backupReconciler) prepareBackupRequest(backup *velerov1api.Backup, logg request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("backup can't be created because backup storage location %s is currently in read-only mode", request.StorageLocation.Name)) } + + if !veleroutil.BSLIsAvailable(*request.StorageLocation) { + request.Status.ValidationErrors = append( + request.Status.ValidationErrors, + fmt.Sprintf("backup can't be created because BackupStorageLocation %s is in Unavailable status.", request.StorageLocation.Name), + ) + } } // add the storage location as a label for easy filtering later. diff --git a/pkg/controller/backup_controller_test.go b/pkg/controller/backup_controller_test.go index 548cb9b88..633e0c8a3 100644 --- a/pkg/controller/backup_controller_test.go +++ b/pkg/controller/backup_controller_test.go @@ -155,7 +155,7 @@ func TestProcessBackupNonProcessedItems(t *testing.T) { } func TestProcessBackupValidationFailures(t *testing.T) { - defaultBackupLocation := builder.ForBackupStorageLocation("velero", "loc-1").Result() + defaultBackupLocation := builder.ForBackupStorageLocation("velero", "loc-1").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result() tests := []struct { name string @@ -183,7 +183,7 @@ func TestProcessBackupValidationFailures(t *testing.T) { { name: "backup for read-only backup location fails validation", backup: defaultBackup().StorageLocation("read-only").Result(), - backupLocation: builder.ForBackupStorageLocation("velero", "read-only").AccessMode(velerov1api.BackupStorageLocationAccessModeReadOnly).Result(), + backupLocation: builder.ForBackupStorageLocation("velero", "read-only").AccessMode(velerov1api.BackupStorageLocationAccessModeReadOnly).Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result(), expectedErrs: []string{"backup can't be created because backup storage location read-only is currently in read-only mode"}, }, { @@ -203,6 +203,12 @@ func TestProcessBackupValidationFailures(t *testing.T) { backupLocation: defaultBackupLocation, expectedErrs: []string{"include-resources, exclude-resources and include-cluster-resources are old filter parameters.\ninclude-cluster-scoped-resources, exclude-cluster-scoped-resources, include-namespace-scoped-resources and exclude-namespace-scoped-resources are new filter parameters.\nThey cannot be used together"}, }, + { + name: "BSL in unavailable state", + backup: defaultBackup().StorageLocation("unavailable").Result(), + backupLocation: builder.ForBackupStorageLocation("velero", "unavailable").Phase(velerov1api.BackupStorageLocationPhaseUnavailable).Result(), + expectedErrs: []string{"backup can't be created because BackupStorageLocation unavailable is in Unavailable status."}, + }, } for _, test := range tests { @@ -592,7 +598,7 @@ func TestDefaultVolumesToResticDeprecation(t *testing.T) { } func TestProcessBackupCompletions(t *testing.T) { - defaultBackupLocation := builder.ForBackupStorageLocation("velero", "loc-1").Default(true).Bucket("store-1").Result() + defaultBackupLocation := builder.ForBackupStorageLocation("velero", "loc-1").Default(true).Bucket("store-1").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result() now, err := time.Parse(time.RFC1123Z, time.RFC1123Z) require.NoError(t, err) @@ -652,7 +658,7 @@ func TestProcessBackupCompletions(t *testing.T) { { name: "backup with a specific backup location keeps it", backup: defaultBackup().StorageLocation("alt-loc").Result(), - backupLocation: builder.ForBackupStorageLocation("velero", "alt-loc").Bucket("store-1").Result(), + backupLocation: builder.ForBackupStorageLocation("velero", "alt-loc").Bucket("store-1").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result(), defaultVolumesToFsBackup: false, expectedResult: &velerov1api.Backup{ TypeMeta: metav1.TypeMeta{ @@ -692,6 +698,7 @@ func TestProcessBackupCompletions(t *testing.T) { backupLocation: builder.ForBackupStorageLocation("velero", "read-write"). Bucket("store-1"). AccessMode(velerov1api.BackupStorageLocationAccessModeReadWrite). + Phase(velerov1api.BackupStorageLocationPhaseAvailable). Result(), defaultVolumesToFsBackup: true, expectedResult: &velerov1api.Backup{ @@ -1414,11 +1421,13 @@ func TestProcessBackupCompletions(t *testing.T) { } func TestValidateAndGetSnapshotLocations(t *testing.T) { + defaultBSL := builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "bsl").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result() tests := []struct { name string backup *velerov1api.Backup locations []*velerov1api.VolumeSnapshotLocation defaultLocations map[string]string + bsl velerov1api.BackupStorageLocation expectedVolumeSnapshotLocationNames []string // adding these in the expected order will allow to test with better msgs in case of a test failure expectedErrors string expectedSuccess bool @@ -1432,6 +1441,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { builder.ForVolumeSnapshotLocation(velerov1api.DefaultNamespace, "some-name").Provider("fake-provider").Result(), }, expectedErrors: "a VolumeSnapshotLocation CRD for the location random-name with the name specified in the backup spec needs to be created before this snapshot can be executed. Error: volumesnapshotlocations.velero.io \"random-name\" not found", expectedSuccess: false, + bsl: *defaultBSL, }, { name: "duplicate locationName per provider: should filter out dups", @@ -1442,6 +1452,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedVolumeSnapshotLocationNames: []string{"aws-us-west-1"}, expectedSuccess: true, + bsl: *defaultBSL, }, { name: "multiple non-dupe location names per provider should error", @@ -1453,6 +1464,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedErrors: "more than one VolumeSnapshotLocation name specified for provider aws: aws-us-west-1; unexpected name was aws-us-east-1", expectedSuccess: false, + bsl: *defaultBSL, }, { name: "no location name for the provider exists, only one VSL for the provider: use it", @@ -1462,6 +1474,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedVolumeSnapshotLocationNames: []string{"aws-us-east-1"}, expectedSuccess: true, + bsl: *defaultBSL, }, { name: "no location name for the provider exists, no default, more than one VSL for the provider: error", @@ -1471,6 +1484,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { builder.ForVolumeSnapshotLocation(velerov1api.DefaultNamespace, "aws-us-west-1").Provider("aws").Result(), }, expectedErrors: "provider aws has more than one possible volume snapshot location, and none were specified explicitly or as a default", + bsl: *defaultBSL, }, { name: "no location name for the provider exists, more than one VSL for the provider: the provider's default should be added", @@ -1482,11 +1496,13 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedVolumeSnapshotLocationNames: []string{"aws-us-east-1"}, expectedSuccess: true, + bsl: *defaultBSL, }, { name: "no existing location name and no default location name given", backup: defaultBackup().Phase(velerov1api.BackupPhaseNew).Result(), expectedSuccess: true, + bsl: *defaultBSL, }, { name: "multiple location names for a provider, default location name for another provider", @@ -1498,6 +1514,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedVolumeSnapshotLocationNames: []string{"aws-us-west-1", "some-name"}, expectedSuccess: true, + bsl: *defaultBSL, }, { name: "location name does not correspond to any existing location and snapshotvolume disabled; should return error", @@ -1509,6 +1526,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedVolumeSnapshotLocationNames: nil, expectedErrors: "a VolumeSnapshotLocation CRD for the location random-name with the name specified in the backup spec needs to be created before this snapshot can be executed. Error: volumesnapshotlocations.velero.io \"random-name\" not found", expectedSuccess: false, + bsl: *defaultBSL, }, { name: "duplicate locationName per provider and snapshotvolume disabled; should return only one BSL", @@ -1519,6 +1537,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedVolumeSnapshotLocationNames: []string{"aws-us-west-1"}, expectedSuccess: true, + bsl: *defaultBSL, }, { name: "no location name for the provider exists, only one VSL created and snapshotvolume disabled; should return the VSL", @@ -1528,6 +1547,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedVolumeSnapshotLocationNames: []string{"aws-us-east-1"}, expectedSuccess: true, + bsl: *defaultBSL, }, { name: "multiple location names for a provider, no default location and backup has no location defined, but snapshotvolume disabled, should return error", @@ -1538,6 +1558,7 @@ func TestValidateAndGetSnapshotLocations(t *testing.T) { }, expectedVolumeSnapshotLocationNames: nil, expectedErrors: "provider aws has more than one possible volume snapshot location, and none were specified explicitly or as a default", + bsl: *defaultBSL, }, } diff --git a/pkg/controller/backup_deletion_controller.go b/pkg/controller/backup_deletion_controller.go index 3952c790d..edd176f3a 100644 --- a/pkg/controller/backup_deletion_controller.go +++ b/pkg/controller/backup_deletion_controller.go @@ -22,9 +22,8 @@ import ( "fmt" "time" - "github.com/vmware-tanzu/velero/pkg/util/csi" - jsonpatch "github.com/evanphx/json-patch/v5" + snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" "github.com/pkg/errors" "github.com/sirupsen/logrus" corev1api "k8s.io/api/core/v1" @@ -37,8 +36,6 @@ import ( ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" - snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" - "github.com/vmware-tanzu/velero/internal/credentials" "github.com/vmware-tanzu/velero/internal/delete" "github.com/vmware-tanzu/velero/internal/volume" @@ -56,8 +53,10 @@ import ( repomanager "github.com/vmware-tanzu/velero/pkg/repository/manager" repotypes "github.com/vmware-tanzu/velero/pkg/repository/types" "github.com/vmware-tanzu/velero/pkg/util/boolptr" + "github.com/vmware-tanzu/velero/pkg/util/csi" "github.com/vmware-tanzu/velero/pkg/util/filesystem" "github.com/vmware-tanzu/velero/pkg/util/kube" + veleroutil "github.com/vmware-tanzu/velero/pkg/util/velero" ) const ( @@ -202,6 +201,11 @@ func (r *backupDeletionReconciler) Reconcile(ctx context.Context, req ctrl.Reque return ctrl.Result{}, err } + if !veleroutil.BSLIsAvailable(*location) { + err := r.patchDeleteBackupRequestWithError(ctx, dbr, fmt.Errorf("cannot delete backup because backup storage location %s is currently in Unavailable state", location.Name)) + return ctrl.Result{}, err + } + // if the request object has no labels defined, initialize an empty map since // we will be updating labels if dbr.Labels == nil { diff --git a/pkg/controller/backup_deletion_controller_test.go b/pkg/controller/backup_deletion_controller_test.go index d99a5c3c5..142c4af43 100644 --- a/pkg/controller/backup_deletion_controller_test.go +++ b/pkg/controller/backup_deletion_controller_test.go @@ -126,6 +126,9 @@ func TestBackupDeletionControllerReconcile(t *testing.T) { }, }, }, + Status: velerov1api.BackupStorageLocationStatus{ + Phase: velerov1api.BackupStorageLocationPhaseAvailable, + }, } dbr := defaultTestDbr() td := setupBackupDeletionControllerTest(t, dbr, location, backup) @@ -254,7 +257,7 @@ func TestBackupDeletionControllerReconcile(t *testing.T) { t.Run("backup storage location is in read-only mode", func(t *testing.T) { backup := builder.ForBackup(velerov1api.DefaultNamespace, "foo").StorageLocation("default").Result() - location := builder.ForBackupStorageLocation("velero", "default").AccessMode(velerov1api.BackupStorageLocationAccessModeReadOnly).Result() + location := builder.ForBackupStorageLocation("velero", "default").Phase(velerov1api.BackupStorageLocationPhaseAvailable).AccessMode(velerov1api.BackupStorageLocationAccessModeReadOnly).Result() td := setupBackupDeletionControllerTest(t, defaultTestDbr(), location, backup) @@ -268,6 +271,24 @@ func TestBackupDeletionControllerReconcile(t *testing.T) { assert.Len(t, res.Status.Errors, 1) assert.Equal(t, "cannot delete backup because backup storage location default is currently in read-only mode", res.Status.Errors[0]) }) + + t.Run("backup storage location is in unavailable state", func(t *testing.T) { + backup := builder.ForBackup(velerov1api.DefaultNamespace, "foo").StorageLocation("default").Result() + location := builder.ForBackupStorageLocation("velero", "default").Phase(velerov1api.BackupStorageLocationPhaseUnavailable).Result() + + td := setupBackupDeletionControllerTest(t, defaultTestDbr(), location, backup) + + _, err := td.controller.Reconcile(context.TODO(), td.req) + require.NoError(t, err) + + res := &velerov1api.DeleteBackupRequest{} + err = td.fakeClient.Get(ctx, td.req.NamespacedName, res) + require.NoError(t, err) + assert.Equal(t, "Processed", string(res.Status.Phase)) + assert.Len(t, res.Status.Errors, 1) + assert.Equal(t, "cannot delete backup because backup storage location default is currently in Unavailable state", res.Status.Errors[0]) + }) + t.Run("full delete, no errors", func(t *testing.T) { input := defaultTestDbr() @@ -297,6 +318,9 @@ func TestBackupDeletionControllerReconcile(t *testing.T) { }, }, }, + Status: velerov1api.BackupStorageLocationStatus{ + Phase: velerov1api.BackupStorageLocationPhaseAvailable, + }, } snapshotLocation := &velerov1api.VolumeSnapshotLocation{ @@ -416,6 +440,9 @@ func TestBackupDeletionControllerReconcile(t *testing.T) { }, }, }, + Status: velerov1api.BackupStorageLocationStatus{ + Phase: velerov1api.BackupStorageLocationPhaseAvailable, + }, } snapshotLocation := &velerov1api.VolumeSnapshotLocation{ @@ -518,6 +545,9 @@ func TestBackupDeletionControllerReconcile(t *testing.T) { }, }, }, + Status: velerov1api.BackupStorageLocationStatus{ + Phase: velerov1api.BackupStorageLocationPhaseAvailable, + }, } snapshotLocation := &velerov1api.VolumeSnapshotLocation{ @@ -600,6 +630,9 @@ func TestBackupDeletionControllerReconcile(t *testing.T) { }, }, }, + Status: velerov1api.BackupStorageLocationStatus{ + Phase: velerov1api.BackupStorageLocationPhaseAvailable, + }, } snapshotLocation := &velerov1api.VolumeSnapshotLocation{ diff --git a/pkg/controller/backup_sync_controller.go b/pkg/controller/backup_sync_controller.go index b80ae82e9..00eb9dafd 100644 --- a/pkg/controller/backup_sync_controller.go +++ b/pkg/controller/backup_sync_controller.go @@ -41,6 +41,7 @@ import ( "github.com/vmware-tanzu/velero/pkg/persistence" "github.com/vmware-tanzu/velero/pkg/plugin/clientmgmt" "github.com/vmware-tanzu/velero/pkg/util/kube" + veleroutil "github.com/vmware-tanzu/velero/pkg/util/velero" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" @@ -92,6 +93,10 @@ func (b *backupSyncReconciler) Reconcile(ctx context.Context, req ctrl.Request) } return ctrl.Result{}, errors.Wrapf(err, "error getting BackupStorageLocation %s", req.String()) } + if !veleroutil.BSLIsAvailable(*location) { + log.Errorf("BackupStorageLocation is in unavailable state, skip syncing backup from it.") + return ctrl.Result{}, nil + } pluginManager := b.newPluginManager(log) defer pluginManager.CleanupClients() diff --git a/pkg/controller/backup_sync_controller_test.go b/pkg/controller/backup_sync_controller_test.go index 398165515..a4db93b16 100644 --- a/pkg/controller/backup_sync_controller_test.go +++ b/pkg/controller/backup_sync_controller_test.go @@ -62,6 +62,9 @@ func defaultLocation(namespace string) *velerov1api.BackupStorageLocation { }, Default: true, }, + Status: velerov1api.BackupStorageLocationStatus{ + Phase: velerov1api.BackupStorageLocationPhaseAvailable, + }, } } @@ -141,6 +144,9 @@ func defaultLocationWithLongerLocationName(namespace string) *velerov1api.Backup }, }, }, + Status: velerov1api.BackupStorageLocationStatus{ + Phase: velerov1api.BackupStorageLocationPhaseAvailable, + }, } } @@ -177,6 +183,21 @@ var _ = Describe("Backup Sync Reconciler", func() { namespace: "ns-1", location: defaultLocation("ns-1"), }, + { + name: "unavailable BSL", + namespace: "ns-1", + location: builder.ForBackupStorageLocation("ns-1", "default").Phase(velerov1api.BackupStorageLocationPhaseUnavailable).Result(), + cloudBackups: []*cloudBackupData{ + { + backup: builder.ForBackup("ns-1", "backup-1").Result(), + backupShouldSkipSync: true, + }, + { + backup: builder.ForBackup("ns-1", "backup-2").Result(), + backupShouldSkipSync: true, + }, + }, + }, { name: "normal case", namespace: "ns-1", diff --git a/pkg/controller/gc_controller.go b/pkg/controller/gc_controller.go index c162a1c28..bb3c60ae5 100644 --- a/pkg/controller/gc_controller.go +++ b/pkg/controller/gc_controller.go @@ -18,6 +18,7 @@ package controller import ( "context" + "fmt" "time" "github.com/pkg/errors" @@ -36,6 +37,7 @@ import ( "github.com/vmware-tanzu/velero/pkg/constant" "github.com/vmware-tanzu/velero/pkg/label" "github.com/vmware-tanzu/velero/pkg/util/kube" + veleroutil "github.com/vmware-tanzu/velero/pkg/util/velero" ) const ( @@ -44,6 +46,7 @@ const ( gcFailureBSLNotFound = "BSLNotFound" gcFailureBSLCannotGet = "BSLCannotGet" gcFailureBSLReadOnly = "BSLReadOnly" + gcFailureBSLUnavailable = "BSLUnavailable" ) // gcReconciler creates DeleteBackupRequests for expired backups. @@ -144,12 +147,18 @@ func (c *gcReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Re } else { backup.Labels[garbageCollectionFailure] = gcFailureBSLCannotGet } + if err := c.Update(ctx, backup); err != nil { log.WithError(err).Error("error updating backup labels") } return ctrl.Result{}, errors.Wrap(err, "error getting backup storage location") } + if !veleroutil.BSLIsAvailable(*loc) { + log.Infof("BSL %s is unavailable, cannot gc backup", loc.Name) + return ctrl.Result{}, fmt.Errorf("bsl %s is unavailable, cannot gc backup", loc.Name) + } + if loc.Spec.AccessMode == velerov1api.BackupStorageLocationAccessModeReadOnly { log.Infof("Backup cannot be garbage-collected because backup storage location %s is currently in read-only mode", loc.Name) backup.Labels[garbageCollectionFailure] = gcFailureBSLReadOnly diff --git a/pkg/controller/gc_controller_test.go b/pkg/controller/gc_controller_test.go index 0241790c8..197deb050 100644 --- a/pkg/controller/gc_controller_test.go +++ b/pkg/controller/gc_controller_test.go @@ -46,7 +46,7 @@ func mockGCReconciler(fakeClient kbclient.Client, fakeClock *testclocks.FakeCloc func TestGCReconcile(t *testing.T) { fakeClock := testclocks.NewFakeClock(time.Now()) - defaultBackupLocation := builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "default").Result() + defaultBackupLocation := builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "default").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result() tests := []struct { name string @@ -66,12 +66,12 @@ func TestGCReconcile(t *testing.T) { { name: "expired backup in read-only storage location is not deleted", backup: defaultBackup().Expiration(fakeClock.Now().Add(-time.Minute)).StorageLocation("read-only").Result(), - backupLocation: builder.ForBackupStorageLocation("velero", "read-only").AccessMode(velerov1api.BackupStorageLocationAccessModeReadOnly).Result(), + backupLocation: builder.ForBackupStorageLocation("velero", "read-only").AccessMode(velerov1api.BackupStorageLocationAccessModeReadOnly).Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result(), }, { name: "expired backup in read-write storage location is deleted", backup: defaultBackup().Expiration(fakeClock.Now().Add(-time.Minute)).StorageLocation("read-write").Result(), - backupLocation: builder.ForBackupStorageLocation("velero", "read-write").AccessMode(velerov1api.BackupStorageLocationAccessModeReadWrite).Result(), + backupLocation: builder.ForBackupStorageLocation("velero", "read-write").AccessMode(velerov1api.BackupStorageLocationAccessModeReadWrite).Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result(), }, { name: "expired backup with no pending deletion requests is deleted", @@ -118,6 +118,12 @@ func TestGCReconcile(t *testing.T) { }, }, }, + { + name: "BSL is unavailable", + backup: defaultBackup().Expiration(fakeClock.Now().Add(-time.Second)).StorageLocation("default").Result(), + backupLocation: builder.ForBackupStorageLocation(velerov1api.DefaultNamespace, "default").Phase(velerov1api.BackupStorageLocationPhaseUnavailable).Result(), + expectError: true, + }, } for _, test := range tests { diff --git a/pkg/controller/restore_controller.go b/pkg/controller/restore_controller.go index 248ccbc49..208a3ddca 100644 --- a/pkg/controller/restore_controller.go +++ b/pkg/controller/restore_controller.go @@ -58,6 +58,7 @@ import ( kubeutil "github.com/vmware-tanzu/velero/pkg/util/kube" "github.com/vmware-tanzu/velero/pkg/util/logging" "github.com/vmware-tanzu/velero/pkg/util/results" + veleroutil "github.com/vmware-tanzu/velero/pkg/util/velero" pkgrestoreUtil "github.com/vmware-tanzu/velero/pkg/util/velero/restore" ) @@ -393,6 +394,11 @@ func (r *restoreReconciler) validateAndComplete(restore *api.Restore) (backupInf return backupInfo{}, nil } + if !veleroutil.BSLIsAvailable(*info.location) { + restore.Status.ValidationErrors = append(restore.Status.ValidationErrors, fmt.Sprintf("The BSL %s is unavailable, cannot retrieve the backup", info.location.Name)) + return backupInfo{}, nil + } + // Fill in the ScheduleName so it's easier to consume for metrics. if restore.Spec.ScheduleName == "" { restore.Spec.ScheduleName = info.backup.GetLabels()[api.ScheduleNameLabel] @@ -728,6 +734,10 @@ func (r *restoreReconciler) deleteExternalResources(restore *api.Restore) error return errors.Wrap(err, fmt.Sprintf("can't get backup info, backup: %s", restore.Spec.BackupName)) } + if !veleroutil.BSLIsAvailable(*backupInfo.location) { + return fmt.Errorf("bsl %s is unavailable, cannot get the backup info", backupInfo.location.Name) + } + // delete restore files in object storage pluginManager := r.newPluginManager(r.logger) defer pluginManager.CleanupClients() diff --git a/pkg/controller/restore_controller_test.go b/pkg/controller/restore_controller_test.go index 4fce9d312..2333943e1 100644 --- a/pkg/controller/restore_controller_test.go +++ b/pkg/controller/restore_controller_test.go @@ -66,7 +66,7 @@ func TestFetchBackupInfo(t *testing.T) { { name: "lister has backup", backupName: "backup-1", - informerLocations: []*velerov1api.BackupStorageLocation{builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Result()}, + informerLocations: []*velerov1api.BackupStorageLocation{builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result()}, informerBackups: []*velerov1api.Backup{defaultBackup().StorageLocation("default").Result()}, expectedRes: defaultBackup().StorageLocation("default").Result(), }, @@ -74,7 +74,7 @@ func TestFetchBackupInfo(t *testing.T) { name: "lister does not have a backup, but backupSvc does", backupName: "backup-1", backupStoreBackup: defaultBackup().StorageLocation("default").Result(), - informerLocations: []*velerov1api.BackupStorageLocation{builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Result()}, + informerLocations: []*velerov1api.BackupStorageLocation{builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result()}, informerBackups: []*velerov1api.Backup{defaultBackup().StorageLocation("default").Result()}, expectedRes: defaultBackup().StorageLocation("default").Result(), }, @@ -211,7 +211,7 @@ func TestProcessQueueItemSkips(t *testing.T) { } func TestRestoreReconcile(t *testing.T) { - defaultStorageLocation := builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Result() + defaultStorageLocation := builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result() now, err := time.Parse(time.RFC1123Z, time.RFC1123Z) require.NoError(t, err) @@ -464,6 +464,22 @@ func TestRestoreReconcile(t *testing.T) { expectedCompletedTime: ×tamp, expectedRestorerCall: NewRestore("foo", "bar", "backup-1", "ns-1", "", velerov1api.RestorePhaseInProgress).Result(), }, + { + name: "Restore creation is rejected when BSL is unavailable", + location: builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Phase(velerov1api.BackupStorageLocationPhaseUnavailable).Result(), + restore: NewRestore("foo", "bar", "backup-1", "ns-1", "", velerov1api.RestorePhaseNew).Result(), + backup: defaultBackup().StorageLocation("default").Result(), + expectedErr: false, + expectedPhase: string(velerov1api.RestorePhaseNew), + expectedValidationErrors: []string{"The BSL default is unavailable, cannot retrieve the backup"}, + }, + { + name: "Restore deletion is rejected when BSL is unavailable.", + location: builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Phase(velerov1api.BackupStorageLocationPhaseUnavailable).Result(), + restore: NewRestore("foo", "bar", "backup-1", "ns-1", "", velerov1api.RestorePhaseCompleted).ObjectMeta(builder.WithFinalizers(ExternalResourcesFinalizer), builder.WithDeletionTimestamp(timestamp.Time)).Result(), + backup: defaultBackup().StorageLocation("default").Result(), + expectedErr: true, + }, } formatFlag := logging.FormatText @@ -738,7 +754,7 @@ func TestValidateAndCompleteWhenScheduleNameSpecified(t *testing.T) { Result(), )) - location := builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Result() + location := builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result() require.NoError(t, r.kbClient.Create(context.Background(), location)) restore = &velerov1api.Restore{ @@ -797,7 +813,7 @@ func TestValidateAndCompleteWithResourceModifierSpecified(t *testing.T) { }, } - location := builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Result() + location := builder.ForBackupStorageLocation("velero", "default").Provider("myCloud").Bucket("bucket").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result() require.NoError(t, r.kbClient.Create(context.Background(), location)) require.NoError(t, r.kbClient.Create( diff --git a/pkg/util/velero/velero.go b/pkg/util/velero/velero.go index d52ecc59c..c0ab9771d 100644 --- a/pkg/util/velero/velero.go +++ b/pkg/util/velero/velero.go @@ -19,6 +19,8 @@ package velero import ( appsv1api "k8s.io/api/apps/v1" corev1api "k8s.io/api/core/v1" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" ) // GetNodeSelectorFromVeleroServer get the node selector from the Velero server deployment @@ -105,3 +107,7 @@ func GetVeleroServerAnnotationValue(deployment *appsv1api.Deployment, key string return deployment.Spec.Template.Annotations[key] } + +func BSLIsAvailable(bsl velerov1api.BackupStorageLocation) bool { + return bsl.Status.Phase == velerov1api.BackupStorageLocationPhaseAvailable +} diff --git a/pkg/util/velero/velero_test.go b/pkg/util/velero/velero_test.go index 5e552c80c..06e9ed070 100644 --- a/pkg/util/velero/velero_test.go +++ b/pkg/util/velero/velero_test.go @@ -24,6 +24,9 @@ import ( appsv1api "k8s.io/api/apps/v1" corev1api "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/builder" ) func TestGetNodeSelectorFromVeleroServer(t *testing.T) { @@ -759,3 +762,11 @@ func TestGetVeleroServerLabelValue(t *testing.T) { }) } } + +func TestBSLIsAvailable(t *testing.T) { + availableBSL := builder.ForBackupStorageLocation("velero", "available").Phase(velerov1api.BackupStorageLocationPhaseAvailable).Result() + unavailableBSL := builder.ForBackupStorageLocation("velero", "unavailable").Phase(velerov1api.BackupStorageLocationPhaseUnavailable).Result() + + assert.True(t, BSLIsAvailable(*availableBSL)) + assert.False(t, BSLIsAvailable(*unavailableBSL)) +} From d903e9eda75b46b12a1a3d7ac95815167045c8d1 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Thu, 29 May 2025 17:15:29 +0800 Subject: [PATCH 13/31] issue 8960: implement PodVolume exposer for PVB/PVR Signed-off-by: Lyndon-Li --- pkg/cmd/cli/nodeagent/server.go | 8 ++- pkg/cmd/cli/nodeagent/server_test.go | 2 +- pkg/exposer/host_path.go | 6 +-- pkg/exposer/pod_volume.go | 74 +++++++++++++--------------- pkg/exposer/pod_volume_test.go | 11 +++-- pkg/exposer/types.go | 12 ++--- pkg/install/daemonset.go | 5 +- pkg/nodeagent/node_agent.go | 14 ++++-- pkg/nodeagent/node_agent_test.go | 6 +-- 9 files changed, 69 insertions(+), 69 deletions(-) diff --git a/pkg/cmd/cli/nodeagent/server.go b/pkg/cmd/cli/nodeagent/server.go index 0dca03665..bf43d18e1 100644 --- a/pkg/cmd/cli/nodeagent/server.go +++ b/pkg/cmd/cli/nodeagent/server.go @@ -80,8 +80,6 @@ const ( // files will be written to defaultCredentialsDirectory = "/tmp/credentials" - defaultHostPodsPath = "/host_pods" - defaultResourceTimeout = 10 * time.Minute defaultDataMoverPrepareTimeout = 30 * time.Minute defaultDataPathConcurrentNum = 1 @@ -416,10 +414,10 @@ func (s *nodeAgentServer) waitCacheForResume() error { // validatePodVolumesHostPath validates that the pod volumes path contains a // directory for each Pod running on this node func (s *nodeAgentServer) validatePodVolumesHostPath(client kubernetes.Interface) error { - files, err := s.fileSystem.ReadDir(defaultHostPodsPath) + files, err := s.fileSystem.ReadDir(nodeagent.HostPodVolumeMountPath()) if err != nil { if errors.Is(err, os.ErrNotExist) { - s.logger.Warnf("Pod volumes host path [%s] doesn't exist, fs-backup is disabled", defaultHostPodsPath) + s.logger.Warnf("Pod volumes host path [%s] doesn't exist, fs-backup is disabled", nodeagent.HostPodVolumeMountPath()) return nil } return errors.Wrap(err, "could not read pod volumes host path") @@ -452,7 +450,7 @@ func (s *nodeAgentServer) validatePodVolumesHostPath(client kubernetes.Interface valid = false s.logger.WithFields(logrus.Fields{ "pod": fmt.Sprintf("%s/%s", pod.GetNamespace(), pod.GetName()), - "path": defaultHostPodsPath + "/" + dirName, + "path": nodeagent.HostPodVolumeMountPath() + "/" + dirName, }).Debug("could not find volumes for pod in host path") } } diff --git a/pkg/cmd/cli/nodeagent/server_test.go b/pkg/cmd/cli/nodeagent/server_test.go index 3d0290a70..2551bfac3 100644 --- a/pkg/cmd/cli/nodeagent/server_test.go +++ b/pkg/cmd/cli/nodeagent/server_test.go @@ -99,7 +99,7 @@ func Test_validatePodVolumesHostPath(t *testing.T) { for _, dir := range tt.dirs { if tt.createDir { - err := fs.MkdirAll(filepath.Join(defaultHostPodsPath, dir), os.ModePerm) + err := fs.MkdirAll(filepath.Join(nodeagent.HostPodVolumeMountPath(), dir), os.ModePerm) if err != nil { t.Error(err) } diff --git a/pkg/exposer/host_path.go b/pkg/exposer/host_path.go index a5668f8c0..e51178711 100644 --- a/pkg/exposer/host_path.go +++ b/pkg/exposer/host_path.go @@ -59,7 +59,7 @@ func GetPodVolumeHostPath(ctx context.Context, pod *corev1api.Pod, volumeName st volSubDir = "volumeDevices" } - pathGlob := fmt.Sprintf("/%s/%s/%s/*/%s", nodeagent.HostPodVolumeMountPoint, string(pod.GetUID()), volSubDir, volDir) + pathGlob := fmt.Sprintf("%s/%s/%s/*/%s", nodeagent.HostPodVolumeMountPath(), string(pod.GetUID()), volSubDir, volDir) logger.WithField("pathGlob", pathGlob).Debug("Looking for path matching glob") path, err := singlePathMatch(pathGlob, fs, logger) @@ -88,8 +88,8 @@ func ExtractPodVolumeHostPath(ctx context.Context, path string, kubeClient kuber } if osType == kube.NodeOSWindows { - return strings.Replace(path, "\\"+nodeagent.HostPodVolumeMountPoint, podPath, 1), nil + return strings.Replace(path, nodeagent.HostPodVolumeMountPathWin(), podPath, 1), nil } else { - return strings.Replace(path, "/"+nodeagent.HostPodVolumeMountPoint, podPath, 1), nil + return strings.Replace(path, nodeagent.HostPodVolumeMountPath(), podPath, 1), nil } } diff --git a/pkg/exposer/pod_volume.go b/pkg/exposer/pod_volume.go index 6e3910d12..6a62705d6 100644 --- a/pkg/exposer/pod_volume.go +++ b/pkg/exposer/pod_volume.go @@ -23,7 +23,7 @@ import ( "github.com/pkg/errors" "github.com/sirupsen/logrus" - corev1 "k8s.io/api/core/v1" + corev1api "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" @@ -60,7 +60,7 @@ type PodVolumeExposeParam struct { HostingPodAnnotations map[string]string // Resources defines the resource requirements of the hosting pod - Resources corev1.ResourceRequirements + Resources corev1api.ResourceRequirements // OperationTimeout specifies the time wait for resources operations in Expose OperationTimeout time.Duration @@ -72,24 +72,24 @@ type PodVolumeExposeParam struct { // PodVolumeExposer is the interfaces for a pod volume exposer type PodVolumeExposer interface { // Expose starts the process to a pod volume expose, the expose process may take long time - Expose(context.Context, corev1.ObjectReference, PodVolumeExposeParam) error + Expose(context.Context, corev1api.ObjectReference, PodVolumeExposeParam) error // GetExposed polls the status of the expose. // If the expose is accessible by the current caller, it waits the expose ready and returns the expose result. // Otherwise, it returns nil as the expose result without an error. - GetExposed(context.Context, corev1.ObjectReference, client.Client, string, time.Duration) (*ExposeResult, error) + GetExposed(context.Context, corev1api.ObjectReference, client.Client, string, time.Duration) (*ExposeResult, error) // PeekExposed tests the status of the expose. // If the expose is incomplete but not recoverable, it returns an error. // Otherwise, it returns nil immediately. - PeekExposed(context.Context, corev1.ObjectReference) error + PeekExposed(context.Context, corev1api.ObjectReference) error // DiagnoseExpose generate the diagnostic info when the expose is not finished for a long time. // If it finds any problem, it returns an string about the problem. - DiagnoseExpose(context.Context, corev1.ObjectReference) string + DiagnoseExpose(context.Context, corev1api.ObjectReference) string // CleanUp cleans up any objects generated during the restore expose - CleanUp(context.Context, corev1.ObjectReference) + CleanUp(context.Context, corev1api.ObjectReference) } // NewPodVolumeExposer creates a new instance of pod volume exposer @@ -110,7 +110,7 @@ type podVolumeExposer struct { var getPodVolumeHostPath = GetPodVolumeHostPath var extractPodVolumeHostPath = ExtractPodVolumeHostPath -func (e *podVolumeExposer) Expose(ctx context.Context, ownerObject corev1.ObjectReference, param PodVolumeExposeParam) error { +func (e *podVolumeExposer) Expose(ctx context.Context, ownerObject corev1api.ObjectReference, param PodVolumeExposeParam) error { curLog := e.log.WithFields(logrus.Fields{ "owner": ownerObject.Name, "client pod": param.ClientPodName, @@ -163,7 +163,7 @@ func (e *podVolumeExposer) Expose(ctx context.Context, ownerObject corev1.Object return nil } -func (e *podVolumeExposer) GetExposed(ctx context.Context, ownerObject corev1.ObjectReference, nodeClient client.Client, nodeName string, timeout time.Duration) (*ExposeResult, error) { +func (e *podVolumeExposer) GetExposed(ctx context.Context, ownerObject corev1api.ObjectReference, nodeClient client.Client, nodeName string, timeout time.Duration) (*ExposeResult, error) { hostingPodName := ownerObject.Name containerName := string(ownerObject.UID) @@ -174,9 +174,9 @@ func (e *podVolumeExposer) GetExposed(ctx context.Context, ownerObject corev1.Ob "node": nodeName, }) - var updated *corev1.Pod + var updated *corev1api.Pod err := wait.PollUntilContextTimeout(ctx, 2*time.Second, timeout, true, func(ctx context.Context) (bool, error) { - pod := &corev1.Pod{} + pod := &corev1api.Pod{} err := nodeClient.Get(ctx, types.NamespacedName{ Namespace: ownerObject.Namespace, Name: hostingPodName, @@ -186,7 +186,7 @@ func (e *podVolumeExposer) GetExposed(ctx context.Context, ownerObject corev1.Ob return false, errors.Wrapf(err, "error to get pod %s/%s", ownerObject.Namespace, hostingPodName) } - if pod.Status.Phase != corev1.PodRunning { + if pod.Status.Phase != corev1api.PodRunning { return false, nil } @@ -213,7 +213,7 @@ func (e *podVolumeExposer) GetExposed(ctx context.Context, ownerObject corev1.Ob }}, nil } -func (e *podVolumeExposer) PeekExposed(ctx context.Context, ownerObject corev1.ObjectReference) error { +func (e *podVolumeExposer) PeekExposed(ctx context.Context, ownerObject corev1api.ObjectReference) error { hostingPodName := ownerObject.Name curLog := e.log.WithFields(logrus.Fields{ @@ -237,7 +237,7 @@ func (e *podVolumeExposer) PeekExposed(ctx context.Context, ownerObject corev1.O return nil } -func (e *podVolumeExposer) DiagnoseExpose(ctx context.Context, ownerObject corev1.ObjectReference) string { +func (e *podVolumeExposer) DiagnoseExpose(ctx context.Context, ownerObject corev1api.ObjectReference) string { hostingPodName := ownerObject.Name diag := "begin diagnose pod volume exposer\n" @@ -263,13 +263,13 @@ func (e *podVolumeExposer) DiagnoseExpose(ctx context.Context, ownerObject corev return diag } -func (e *podVolumeExposer) CleanUp(ctx context.Context, ownerObject corev1.ObjectReference) { +func (e *podVolumeExposer) CleanUp(ctx context.Context, ownerObject corev1api.ObjectReference) { restorePodName := ownerObject.Name kube.DeletePodIfAny(ctx, e.kubeClient.CoreV1(), restorePodName, ownerObject.Namespace, e.log) } -func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject corev1.ObjectReference, exposeType string, hostPath string, - operationTimeout time.Duration, label map[string]string, annotation map[string]string, selectedNode string, resources corev1.ResourceRequirements, nodeOS string) (*corev1.Pod, error) { +func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject corev1api.ObjectReference, exposeType string, hostPath string, + operationTimeout time.Duration, label map[string]string, annotation map[string]string, selectedNode string, resources corev1api.ResourceRequirements, nodeOS string) (*corev1api.Pod, error) { hostingPodName := ownerObject.Name containerName := string(ownerObject.UID) @@ -282,28 +282,24 @@ func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject cor } var gracePeriod int64 - mountPropagation := corev1.MountPropagationHostToContainer - volumeMounts := []corev1.VolumeMount{{ + mountPropagation := corev1api.MountPropagationHostToContainer + volumeMounts := []corev1api.VolumeMount{{ Name: clientVolumeName, MountPath: clientVolumePath, MountPropagation: &mountPropagation, }} volumeMounts = append(volumeMounts, podInfo.volumeMounts...) - volumes := []corev1.Volume{{ + volumes := []corev1api.Volume{{ Name: clientVolumeName, - VolumeSource: corev1.VolumeSource{ - HostPath: &corev1.HostPathVolumeSource{ + VolumeSource: corev1api.VolumeSource{ + HostPath: &corev1api.HostPathVolumeSource{ Path: hostPath, }, }, }} volumes = append(volumes, podInfo.volumes...) - if label == nil { - label = make(map[string]string) - } - args := []string{ fmt.Sprintf("--volume-path=%s", clientVolumePath), fmt.Sprintf("--resource-timeout=%s", operationTimeout.String()), @@ -317,24 +313,22 @@ func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject cor if exposeType == PodVolumeExposeTypeBackup { args = append(args, fmt.Sprintf("--pod-volume-backup=%s", ownerObject.Name)) command = append(command, "backup") - label[podGroupLabel] = podGroupPodVolumeBackup } else { args = append(args, fmt.Sprintf("--pod-volume-restore=%s", ownerObject.Name)) command = append(command, "restore") - label[podGroupLabel] = podGroupPodVolumeRestore } args = append(args, podInfo.logFormatArgs...) args = append(args, podInfo.logLevelArgs...) - var securityCtx *corev1.PodSecurityContext + var securityCtx *corev1api.PodSecurityContext nodeSelector := map[string]string{} - podOS := corev1.PodOS{} - toleration := []corev1.Toleration{} + podOS := corev1api.PodOS{} + toleration := []corev1api.Toleration{} if nodeOS == kube.NodeOSWindows { userID := "ContainerAdministrator" - securityCtx = &corev1.PodSecurityContext{ - WindowsOptions: &corev1.WindowsSecurityContextOptions{ + securityCtx = &corev1api.PodSecurityContext{ + WindowsOptions: &corev1api.WindowsSecurityContextOptions{ RunAsUserName: &userID, }, } @@ -342,7 +336,7 @@ func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject cor nodeSelector[kube.NodeOSLabel] = kube.NodeOSWindows podOS.Name = kube.NodeOSWindows - toleration = append(toleration, corev1.Toleration{ + toleration = append(toleration, corev1api.Toleration{ Key: "os", Operator: "Equal", Effect: "NoSchedule", @@ -350,7 +344,7 @@ func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject cor }) } else { userID := int64(0) - securityCtx = &corev1.PodSecurityContext{ + securityCtx = &corev1api.PodSecurityContext{ RunAsUser: &userID, } @@ -358,7 +352,7 @@ func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject cor podOS.Name = kube.NodeOSLinux } - pod := &corev1.Pod{ + pod := &corev1api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: hostingPodName, Namespace: ownerObject.Namespace, @@ -374,14 +368,14 @@ func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject cor Labels: label, Annotations: annotation, }, - Spec: corev1.PodSpec{ + Spec: corev1api.PodSpec{ NodeSelector: nodeSelector, OS: &podOS, - Containers: []corev1.Container{ + Containers: []corev1api.Container{ { Name: containerName, Image: podInfo.image, - ImagePullPolicy: corev1.PullNever, + ImagePullPolicy: corev1api.PullNever, Command: command, Args: args, VolumeMounts: volumeMounts, @@ -394,7 +388,7 @@ func (e *podVolumeExposer) createHostingPod(ctx context.Context, ownerObject cor TerminationGracePeriodSeconds: &gracePeriod, Volumes: volumes, NodeName: selectedNode, - RestartPolicy: corev1.RestartPolicyNever, + RestartPolicy: corev1api.RestartPolicyNever, SecurityContext: securityCtx, Tolerations: toleration, }, diff --git a/pkg/exposer/pod_volume_test.go b/pkg/exposer/pod_volume_test.go index 8f4abab67..aac24990f 100644 --- a/pkg/exposer/pod_volume_test.go +++ b/pkg/exposer/pod_volume_test.go @@ -8,11 +8,6 @@ import ( "github.com/sirupsen/logrus" "github.com/stretchr/testify/assert" - velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" - "github.com/vmware-tanzu/velero/pkg/builder" - "github.com/vmware-tanzu/velero/pkg/datapath" - velerotest "github.com/vmware-tanzu/velero/pkg/test" - "github.com/vmware-tanzu/velero/pkg/util/filesystem" appsv1api "k8s.io/api/apps/v1" corev1api "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -20,6 +15,12 @@ import ( "k8s.io/client-go/kubernetes" "k8s.io/client-go/kubernetes/fake" clientFake "sigs.k8s.io/controller-runtime/pkg/client/fake" + + velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/datapath" + velerotest "github.com/vmware-tanzu/velero/pkg/test" + "github.com/vmware-tanzu/velero/pkg/util/filesystem" ) func TestPodVolumeExpose(t *testing.T) { diff --git a/pkg/exposer/types.go b/pkg/exposer/types.go index b8f875a88..5cdb9d497 100644 --- a/pkg/exposer/types.go +++ b/pkg/exposer/types.go @@ -21,13 +21,11 @@ import ( ) const ( - AccessModeFileSystem = "by-file-system" - AccessModeBlock = "by-block-device" - podGroupLabel = "velero.io/exposer-pod-group" - podGroupSnapshot = "snapshot-exposer" - podGroupGenericRestore = "generic-restore-exposer" - podGroupPodVolumeBackup = "pod-volume-backup" - podGroupPodVolumeRestore = "pod-volume-restore" + AccessModeFileSystem = "by-file-system" + AccessModeBlock = "by-block-device" + podGroupLabel = "velero.io/exposer-pod-group" + podGroupSnapshot = "snapshot-exposer" + podGroupGenericRestore = "generic-restore-exposer" ) // ExposeResult defines the result of expose. diff --git a/pkg/install/daemonset.go b/pkg/install/daemonset.go index 1dee41a33..7eaa2a094 100644 --- a/pkg/install/daemonset.go +++ b/pkg/install/daemonset.go @@ -25,6 +25,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "github.com/vmware-tanzu/velero/internal/velero" + "github.com/vmware-tanzu/velero/pkg/nodeagent" ) func DaemonSet(namespace string, opts ...podTemplateOption) *appsv1api.DaemonSet { @@ -126,8 +127,8 @@ func DaemonSet(namespace string, opts ...podTemplateOption) *appsv1api.DaemonSet }, VolumeMounts: []corev1api.VolumeMount{ { - Name: "host-pods", - MountPath: "/host_pods", + Name: nodeagent.HostPodVolumeMount, + MountPath: nodeagent.HostPodVolumeMountPath(), MountPropagation: &mountPropagationMode, }, { diff --git a/pkg/nodeagent/node_agent.go b/pkg/nodeagent/node_agent.go index 1c0378901..3d1159085 100644 --- a/pkg/nodeagent/node_agent.go +++ b/pkg/nodeagent/node_agent.go @@ -42,8 +42,8 @@ const ( // nodeAgentRole marks pods with node-agent role on all nodes. nodeAgentRole = "node-agent" - // hostPodVolume is the name of the volume in node-agent for host-pod mount - hostPodVolume = "host-pods" + // HostPodVolumeMount is the name of the volume in node-agent for host-pod mount + HostPodVolumeMount = "host-pods" // HostPodVolumeMountPoint is the mount point of the volume in node-agent for host-pod mount HostPodVolumeMountPoint = "host_pods" @@ -269,7 +269,7 @@ func GetHostPodPath(ctx context.Context, kubeClient kubernetes.Interface, namesp var volume *corev1api.Volume for _, v := range ds.Spec.Template.Spec.Volumes { - if v.Name == hostPodVolume { + if v.Name == HostPodVolumeMount { volume = &v break } @@ -289,3 +289,11 @@ func GetHostPodPath(ctx context.Context, kubeClient kubernetes.Interface, namesp return volume.HostPath.Path, nil } + +func HostPodVolumeMountPath() string { + return "/" + HostPodVolumeMountPoint +} + +func HostPodVolumeMountPathWin() string { + return "\\" + HostPodVolumeMountPoint +} diff --git a/pkg/nodeagent/node_agent_test.go b/pkg/nodeagent/node_agent_test.go index 118f36ecf..8f94b1ff6 100644 --- a/pkg/nodeagent/node_agent_test.go +++ b/pkg/nodeagent/node_agent_test.go @@ -615,7 +615,7 @@ func TestGetHostPodPath(t *testing.T) { Spec: corev1api.PodSpec{ Volumes: []corev1api.Volume{ { - Name: hostPodVolume, + Name: HostPodVolumeMount, }, }, }, @@ -636,7 +636,7 @@ func TestGetHostPodPath(t *testing.T) { Spec: corev1api.PodSpec{ Volumes: []corev1api.Volume{ { - Name: hostPodVolume, + Name: HostPodVolumeMount, VolumeSource: corev1api.VolumeSource{ HostPath: &corev1api.HostPathVolumeSource{}, }, @@ -660,7 +660,7 @@ func TestGetHostPodPath(t *testing.T) { Spec: corev1api.PodSpec{ Volumes: []corev1api.Volume{ { - Name: hostPodVolume, + Name: HostPodVolumeMount, VolumeSource: corev1api.VolumeSource{ HostPath: &corev1api.HostPathVolumeSource{ Path: "/var/lib/kubelet/pods", From 92c72b1a63458132e68ae2b33e53dbed567f07ec Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Fri, 30 May 2025 12:59:14 +0800 Subject: [PATCH 14/31] data path for vgdp ms pvb Signed-off-by: Lyndon-Li --- changelogs/unreleased/8998-Lyndon-Li | 1 + .../v1/bases/velero.io_podvolumebackups.yaml | 5 + config/crd/v1/crds/crds.go | 2 +- internal/credentials/local.go | 7 + pkg/apis/velero/v1/pod_volume_backup_types.go | 4 + pkg/cmd/cli/datamover/backup.go | 13 +- pkg/cmd/cli/datamover/data_mover.go | 30 -- pkg/cmd/cli/datamover/data_mover_test.go | 131 ----- pkg/cmd/cli/datamover/restore.go | 5 +- pkg/cmd/cli/nodeagent/server.go | 6 +- pkg/cmd/cli/podvolume/backup.go | 291 ++++++++++++ pkg/cmd/cli/podvolume/backup_test.go | 216 +++++++++ pkg/cmd/cli/podvolume/podvolume.go | 43 ++ pkg/cmd/cli/repomantenance/maintenance.go | 2 +- pkg/cmd/server/config/config.go | 7 +- pkg/cmd/velero/velero.go | 2 + pkg/podvolume/backup_micro_service.go | 313 ++++++++++++ pkg/podvolume/backup_micro_service_test.go | 447 ++++++++++++++++++ pkg/podvolume/util.go | 15 + pkg/util/kube/pod.go | 28 ++ pkg/util/kube/pod_test.go | 107 +++++ 21 files changed, 1491 insertions(+), 184 deletions(-) create mode 100644 changelogs/unreleased/8998-Lyndon-Li create mode 100644 internal/credentials/local.go delete mode 100644 pkg/cmd/cli/datamover/data_mover_test.go create mode 100644 pkg/cmd/cli/podvolume/backup.go create mode 100644 pkg/cmd/cli/podvolume/backup_test.go create mode 100644 pkg/cmd/cli/podvolume/podvolume.go create mode 100644 pkg/podvolume/backup_micro_service.go create mode 100644 pkg/podvolume/backup_micro_service_test.go diff --git a/changelogs/unreleased/8998-Lyndon-Li b/changelogs/unreleased/8998-Lyndon-Li new file mode 100644 index 000000000..a2385b5ad --- /dev/null +++ b/changelogs/unreleased/8998-Lyndon-Li @@ -0,0 +1 @@ +Fix issue #8988, add data path for VGDP ms pvb \ No newline at end of file diff --git a/config/crd/v1/bases/velero.io_podvolumebackups.yaml b/config/crd/v1/bases/velero.io_podvolumebackups.yaml index 9ccff4124..0eadd8e59 100644 --- a/config/crd/v1/bases/velero.io_podvolumebackups.yaml +++ b/config/crd/v1/bases/velero.io_podvolumebackups.yaml @@ -76,6 +76,11 @@ spec: BackupStorageLocation is the name of the backup storage location where the backup repository is stored. type: string + cancel: + description: |- + Cancel indicates request to cancel the ongoing PodVolumeBackup. It can be set + when the PodVolumeBackup is in InProgress phase + type: boolean node: description: Node is the name of the node that the Pod is running on. diff --git a/config/crd/v1/crds/crds.go b/config/crd/v1/crds/crds.go index 60c18e9eb..16288b1b4 100644 --- a/config/crd/v1/crds/crds.go +++ b/config/crd/v1/crds/crds.go @@ -34,7 +34,7 @@ var rawCRDs = [][]byte{ []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xccYK\x8f\x1b\xb9\x11\xbe\xebW\x14v\x0f{ٖ\xec\x04\t\x02\xdd\xc6r\x02\x18\x19\xc7\x03k2\xb9.EVK\\\xb1\xc9\x0e\x1f\x92\x95\xc7\x7f\x0f\x8a\x0f\xa9\xd5\x0fK\xe3\x04\x9b\xe5eF|\x14\xeb\xf9U\x15\xbb\xaa\xaa\x19k\xe5\vZ'\x8d^\x02k%~\xf1\xa8闛\xef\xff\xe0\xe6\xd2,\x0eog{\xa9\xc5\x12V\xc1y\xd3|Fg\x82\xe5\xf8\x1ek\xa9\xa5\x97F\xcf\x1a\xf4L0ϖ3\x00\xa6\xb5\xf1\x8c\xa6\x1d\xfd\x04\xe0F{k\x94B[mQ\xcf\xf7a\x83\x9b \x95@\x1b\x89\x97\xab\x0fo\xe6o\x7f?\xff\xdd\f@\xb3\x06\x97\xb0a|\x1fZ\xe7\x8de[T\x86'\x92\xf3\x03*\xb4f.\xcd̵\xc8醭5\xa1]\xc2e!Qȷ'\xce\xdfEb\xebD\xec1\x13\x8b\xebJ:\xff\xe7\xe9=\x8f\xd2\xf9\xb8\xafU\xc125\xc5V\xdc\xe2v\xc6\xfa\xbf\\\xae\xae`\xe3TZ\x91z\x1b\x14\xb3\x13\xc7g\x00\x8e\x9b\x16\x97\x10O\xb7\x8c\xa3\x98\x01d\xd5Dj\x150!\xa2\xb2\x99z\xb2R{\xb4+\xa3B\xa3\xcfw\tt\xdc\xca\xd6Ge&Y \v\x03E\x1ap\x9e\xf9\xe0\xc0\x05\xbe\x03\xe6\xe0\xe1\xc0\xa4b\x1b\x85\x8b\xbfjV\xfe\x8f\xf4\x00~vF?1\xbf[\xc2<\x9d\x9a\xb7;\xe6\xcaj\xb2\xd1SgƟH\x00\xe7\xad\xd4\xdb1\x96\x1e\x99\xf3/LI\x119y\x96\r\x82t\xe0w\b\x8a9\x0f\x9e&\xe8W\xd2\x10\x90\x8a\x10\x8a\x86\xe0\xc8\\\xbe\a\xe0\x90\xa8D\x1d\x8ds\xaa\x06w]\xb1M\xac\xc0K\x8fJ\xe2\x9ff2\xf7\x1d\xb2ſ\xe7\xdc♤\xf3\xaci\xaf\xe8>lq\x8aؕ*\xdec͂\xf2]Q\xc9J\xaa\xeb\x97\xd7b\xb5\xc8\xe7\"\x9d\xba\xba\xf1\xfd\xd5\\\xbauc\x8cB\x96\xa8\xa4]\x87\xb7\xc9\v\xf9\x0e\x1b\xb6̛M\x8b\xfa\xe1\xe9\xc3\xcbo\xd7W\xd30\xe6H\xbd\xa0 ñ\x8emvh\x11^b\xfc%\xbb\xb9,ڙ&\x80\xd9\xfc\x8c\xdc_\x8c\xd8ZӢ\xf5\xb2\x04K\x1a\x1d,\xea\xcc\xf6x\xfaWu\xb5\x06@b\xa4S \b\x940\xf9U\x8e\x1f\x14Yr05\xf8\x9dt`\xb1\xb5\xe8P'\x98\xa2i\xa63\x83\xf3\x1e\xe95Z\"C\xb1\x1d\x94 ,;\xa0\xf5`\x91\x9b\xad\x96\xff8\xd3v\xe0Mvf\x8f\xceC\x8cP\xcd\x149k\xc0\x1f\x81iѣܰ\x13X\xa4;!\xe8\x0e\xbdx\xc0\xf5\xf9\xf8H\xd1 um\x96\xb0\xf3\xbeu\xcb\xc5b+}Ahn\x9a&h\xe9O\x8b\b\xb6r\x13\xbc\xb1n!\xf0\x80j\xe1\xe4\xb6b\x96\xef\xa4G\xee\x83\xc5\x05ke\x15\x05\xd1\tR\x1b\xf1\xbd͘\uebae\x1d\x84t\x1a\x11R_a\x1e\x82\xd7\xe42\x89T\x12\xf1b\x05\x9a\"\xd5}\xfe\xe3\xfa\x19\n'\xc9R\xc9(\x97\xad\x03\xbd\x14\xfb\x906\xa5\xaeѦs\xb55M\xa4\x89Z\xb4Fj\x1f\x7fp%Q{pa\xd3HOn\xf0\xf7\x80Γ\xe9\xfadW1\x8b\xc1\x06!\xb4\x11$\xfa\x1b>hX\xb1\x06Պ9\xfc\x85mEVq\x15\x19\xe1.kuss\x7fsRog\xa1\xe4\xd4\tӎ\xa2\xc1\xbaE~\x15w\x02\x9d\xb4\x14\x19\x9ey\x8c\xd1\xd5SP\x86\x8a\xe9\xa4\\\xc68H\xd0`\x9c\xa3s\x1f\x8d\xc0\xfeJ\x8f\xe5\x87\xf3\xc6+\x1e[\xb4\x8dt1\xbdBml?\xf3\xb03\x92wGA\xbc\xbe\xc1\x01P\x87f\xc8H\x05\x9f\x91\x89OZ\x9d&\x96\xfef\xa5\x1f^4aH\x1a\x89\xc5\xf5I\xf3'\xb4҈\x1b¿\xebm?\xab`g\x8ePG\xff\xd7^\x9d\b\xbb\xdcI\xf3!j\x97\xf1\xf0\xf4\xa1 x\x8a\xad\x1c\x98YWsx\xc8Amjx\x03B:*$\\$:T\x96\x0e*\x16\x1aK\xf06\xbcJ|nt-\xb7C\xa1\xbb\xb5є\xc7\xdc \xdd\xd3\xdc*\xdeD\xa8E\xde\xd1Zs\x90\x02mE\xf1!k\xc93'\xc1\xa6\fRKTb\x80M\x93Q\x16E\xb1((\xa8\x99\xbaa\xc3\xd5yc\xac\xa4\x99\xd4Ƀ/\x04\"\xd6\xd8&\xa7f\xedQ\v\xecg\x9bȍ\x89\x80\xe6P\xc0Q\xfa]BJ5\x16w\xf0\xd5أ\xb1\xc7\xd3\xd8t\x8f\xf7\xe7\x1d\xd2Δx\x11\x1cr\x8b>z\x1b*r\x1fr\xa59\xc0\xc7\xe0\"\xd6\xf6q\xa2\x8cX\xf0\x95\xd3{<\r\x15\r\xb7\x8c\x9bK\xa1\t\x96c\x11\xb5\x84ﾻ-\xd2 \xbb\x95A\xa5{\x11\xd4b\x8d\x16\xf5\xa0\x9a(\xe39\xe6(r\x1a\xf20\xack\xe4^\x1eP\x9dbN\"\xf0\xfc\x116\xc1\x83\b\x18\xad\xc6\xf8\xfeȬp\xc0M\xd32/7RI\x7f\x02\xe9&\xe83\xa5\xcc\x11E\xb686\xad?\xcd\xe1\x83v\x9ei\x8e\xee\\\a\x91ƒ+0\x9dv\xe5(\x8e\x05\x1d\xb3c\x18\x98\xc87\xc6y\xe0h\xc9\x1d\xd5\t\x8e\xd6\xe8픰#\xe9\x90z@\xab\xd1c̈\xc2pGɐc\xeb\xdd\xc2\x1c\xd0\x1e$\x1e\x17Gc\xf7Ro+b\xb0\xcaೈ\x9d\xdd\xe2\xfb\xf8\xe7[\xbc\xc0\xb4\t'\xeep\xdeu\x8c\xf5\x13\x95\xb7~\x87)E\xac\x93\x0f\x1a\vT@\x90k7\xd9w\x13\xb2\x8e\x85\xddX]\xde\x1d\xc5\xe4c\xf9c\x8f\xc3\xd4\xf1\x15P\x01\xf8R]t[5\xac\xad\xd2n\xe6M#\xf9\xac/m\xf2\xfb\xaf\xe3OiV\xa4\x16\x92Sq{\x8d\x1b\xa5\x89\x13W=͈\x1a\xfa]\xce\x14Z\x8e\xab)\x89\x9bk\x85\x1b\x1c\x7f\xea\uef74\xbe\t\xbas\xfew\xe8\xa9\xeet\xa0\x91\xea\x03f\x87z\x8e\x80ɍքT\xde\x00;\xa7\x81\x1f\\?\xff\xbd\x12=7\x81\xefqD\xf1\x03Q\xdeōE\xc7\xe9\x18\xf1\x12\x1c\xc6\xc4t\x8b\r\xb8\x1d\x11\x9c\xad\xd0\xde\xc3\xcb\xea\x816\x9eK\b\x06\xab\a\xd8\x04-\x14\x16\x8e\x8e;\xd4\xd4u\xc9\xfa4~\x17\x8d\xe7\xc7u\xd1j\xac\xber\xdfTt;.C\xcaoK\u061cF\xea\xa5;\x84l-\xd6\xf2\xcb\x1dB>ōE\xe1-\xf3;\x90\xdaI\x81\xc0Fԟ\n\xd9\tAϵѧ\x8c9\xdf`\x9e\xafaCb\xe75\xf0Pt|#~\x9e\xf2\xb6\xb3\x16\xca\xef\x9cݮ\xeb\xe4\xa98\x1e\x95\xe8p~\x94\xf9S\xaa>\xf9H\x19q\xc5\xcc\xcb\xf0\xc4W\xaa\xd8\xf244\x16\xccT3\x19kѵF\v\xea9\xef\xaba/,\xff\xef*\xd9q\xb3V\xd7(\xd7[+V\xb8\xab\x8d\x8b\xcf`\xafn\xe4\xd2\xe3`\xb7M2\x1bG\r\xf6\xa5\x97\xeb\xc9\xf8\x8b\xb4p\xa3%W\xa7\xaf\x93\x8eꗠce\x1b\xab\xaa\xf9l\xe4\xc4{l-R\x06\x13K\x92\xcdƃ\xda\x1c\xe9p\x87Z*ˌN\xf9\x9ez[\xa6E~U\xa0\xa5\x11\xcaG\xa9\x14\xd5\x00\x16\x1bCʢ\xb2\xdcR5\xc7b\xadu\xf8\xcd\xfc\xcd\xff\xafeT\xccy\xea\x00Q|ƃ\x1c>\xadݧ\xee\xc7\x01\x95\x82\x0e瘡\x1f?\x95׆\x85\xcd\xdb~\x82Z*\xaa\xff:\xd0qGu0\xf20\xfcn\xfd\xf8\x83\x8b=\x10j\xef\xe0H\x16t\x91%jzL~\xe1\t\xceS\x12\xb9i\xffn\x01\xae\r(\xa3\xb7h\xcbk\x0f\x15xɛ\x8c\x05\x81\x9er\x95\xde\x02\xdf1\xbd\xa5\xc8\x18\x83\xfc\xc8p\xe6\xbe\xcb'yϤ\x83H=\xe1\x1dw\x19\xf4Y\x8e\xb54\xaf1\xe6\xf43\xfc\x99\xffl\xd9\xcbkoO\xefSP[,\xd1_,\xa9\x9c\x14]\xf9\xcb\xd3\xfce|\xfb\xfb\xc0\xf0\xdd\xff[\xd5\xf3_}\xa9\x18|\xa1\xf8U(\xa7\xa1:\xf7f\xf1\xfc1\xedJ\xef\xb5\xf9\b\xb0\x8d\t~$\xf7w\x1c~4\xa6\xe3ǘ\xd7\xf0\x18?1\xdd*OhO\xb1\b\x0f\xd6\xc67\xdd\xf2\xd6\x18\x91b,+ݏ\xc0\x0f\xbd/aݵ\xe1w\xb2;\xe4\x1a\xcd҃ɔi;v\xcdJ\xee΄\xcd\xf9\xa5~\t\xff\xfc\xf7\xec?\x01\x00\x00\xff\xff\x03f\x86Y\xc0\x1d\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcVMo\x1b7\x10\xbd\xebW\f\xd0kwU\xa3hQ\xec\xadqr0\xda\x06\x82\x1d\xe4N\x91#-c.\xc9\xce\f\xe5\xba\x1f\xff\xbd \xb9+K\xab\x95\x93\\\xb27\x91Ù\xc7\xf7f\x1e\xd54\xcdJE\xfb\x11\x89m\xf0\x1d\xa8h\xf1/A\x9f\x7fq\xfb\xf8\v\xb76\xac\x0f7\xabG\xebM\a\xb7\x89%\f\xf7\xc8!\x91Ʒ\xb8\xb3ފ\r~5\xa0(\xa3Du+\x00\xe5}\x10\x95\x979\xff\x04\xd0\xc1\v\x05琚=\xfa\xf61mq\x9b\xac3H%\xf9T\xfa\xf0C{\xf3s\xfb\xd3\n\xc0\xab\x01;0\xe8Pp\xab\xf4c\x8a\x84\x7f&d\xe1\xf6\x80\x0e)\xb46\xac8\xa2\xce\xf9\xf7\x14R\xec\xe0e\xa3\x9e\x1fkW\xdcoK\xaa7%\xd5}MUv\x9de\xf9\xedZ\xc4\xefv\x8c\x8a.\x91rˀJ\x00[\xbfON\xd1b\xc8\n\x80u\x88\xd8\xc1\xfb\f+*\x8df\x050^\xbb\xc0l@\x19S\x88TnC\xd6\v\xd2mpi\x98\bl\xc0 k\xb2Q\nQ\x1fz,W\x84\xb0\x03\xe9\x11j9\x90\x00[\x1c\x11\x98r\x0e\xe0\x13\a\xbfQ\xd2w\xd0f\xbe\xda\x1a\x9a\x81\x8c\x01\x95\xea7\xf3ey\u0380Y\xc8\xfa\xfd5\b,J\x12O J]\x1b<\xd0\t\xbf\xe7\x00J|\x1b{\xc5\xe7\xd5\x1f\xcaƵ\xca5\xe6pS\x99\xd6=\x0e\xaa\x1bcCD\xff\xeb\xe6\xee\xe3\x8f\x0fg\xcbp\x8euAZ\xb0\fjB\x9a\x89\xab\xacA\xf0\b\x81`\b4\xb1\xca\xed1i\xa4\x10\x91\xc4N\xadU\xbf\x93\xe19Y\x9dA\xf8\xb79\xdb\x03Ȩ\xeb)0y\x8a\x90\v\x89cS\xa0\x19/Zɵ\f\x84\x91\x90\xd1\u05f9\xca\xcb\xcaC\xd8~B-\xed,\xf5\x03RN\x03܇\xe4L\x1e\xbe\x03\x92\x00\xa1\x0e{o\xff>\xe6\xe6|\xef\\\xd4))\x94\xe4\xb6\xf3\xca\xc1A\xb9\x84߃\xf2f\x96yP\xcf@\x98kB\xf2'\xf9\xca\x01\x9e\xe3\xf8#\x93h\xfd.tЋD\xee\xd6뽕\xc9Rt\x18\x86\xe4\xad<\xaf\x8b;\xd8m\x92@\xbc6x@\xb7f\xbbo\x14\xe9\xde\njI\x84k\x15mS.⋭\xb4\x83\xf9\x8eF\x13Ⳳ\x17\xddS\xbf\xe2\x02_!O\xf6\x84\xda#5U\xbd\xe2\x8b\ny)Sw\xff\xee\xe1\x03LH\xaaRU\x94\x97\xd0\v^&}2\x9b\xd6\xef\x90\xea\xb9\x1d\x85\xa1\xe4Dob\xb0^\xca\x0f\xed,z\x01N\xdb\xc1\nO\x1d\x9b\xa5\x9b\xa7\xbd-\xb6\x9b\x1d E\xa3\x04\xcd<\xe0\xceí\x1a\xd0\xdd*\xc6o\xacUV\x85\x9b,\xc2\x17\xa9u\xfa\x98̃+\xbd'\x1b\xd33pEڅ\xe1\x7f\x88\xa8\xb3\xb8\x99\xdf|\xda\ueb2ec\xb5\v\x04O\xbd\xd5\xfd4\xfc3\x9a\x8eFq\xce߲1\xe4\xef\xc5n\xe7;W/\x0fEdK8k\xd8\x06.\xbc\xfbu^\x8a\xa9~%3\xd5\xd1Gnt\"*\xcdw\xf4y\xb5t\xe8K\xb9@\xa2@\x17\xab3P\xefJP\xf9Ǡ\xacgP\xfey<\b\xd2+\x81'\xa4\r\x97\x95\x1ax\x8fO\v\xabw~CaO\xc8\xf3\x96ϛ\x9b\xca\x1e\xce߃WXZlʋE\xceVhNXd\t\xa4\xf6\xa7\xbcr\xda\x1e\x9d\xbe\x83\x7f\xfe[\xfd\x1f\x00\x00\xff\xff\xbeM\x1a\xea\xb1\n\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcWMo\xe36\x10\xbd\xfbW\f\xd0K\v\xac\xe4\x06E\x8b·\xd6\xd9C\xb0\xe96\x88\xb7\xb9S\xd4HbC\x91,9t6E\x7f|1\xa4\xe4\x0fYv\x9c\xcb\xea\xe6\xe1p\xf8\xe6\xcd\xcc#]\x14\xc5B8\xf5\x84>(kV \x9c¯\x84\x86\x7f\x85\xf2\xf9\xd7P*\xbb\xdc\xde,\x9e\x95\xa9W\xb0\x8e\x81l\xff\x88\xc1F/\xf1\x16\x1be\x14)k\x16=\x92\xa8\x05\x89\xd5\x02@\x18cI\xb09\xf0O\x00i\ry\xab5\xfa\xa2ES>\xc7\n\xab\xa8t\x8d>\x05\x1f\x8f\xde\xfeX\xde\xfcR\xfe\xbc\x000\xa2\xc7\x15\xd4\xf6\xc5h+j\x8f\xffD\f\x14\xca-j\xf4\xb6Tv\x11\x1cJ\x8e\xddz\x1b\xdd\n\xf6\vy\xefpn\xc6|;\x84y\xccaҊV\x81>ͭޫ\xc1\xc3\xe9\xe8\x85>\x05\x91\x16\x832m\xd4\u009f,/\x00\x82\xb4\x0eW\xf0\x99a8!\xb1^\x00\f)&XŐ\xdd\xf6&\x87\x92\x1d\xf6\"\xe3\x05\xb0\x0e\xcdo\x0fwO?m\x8e\xcc\x005\x06镣D\xd4\x7f\xc5\xce\x0e\xd3\x04@\x05\x100\xc0\x01\xb2;\x84 \f\bO\xaa\x11\x92\xa0\xf1\xb6\x87J\xc8\xe7\xe8\xc0V\x7f\xa3$\bd\xbdh\xf1\x03\x84(;\x10\x1c%;\x1c\x9c\xa5m\v\x8d\xd2X\xeel\xce[\x87\x9e\xd4Hy\xfe\x0e\x1a\xea\xc0z)\v\xfe8\xf1\xbc\vj\xee,\f@\x1d\x8e\xe4a=p\x05\xb6\x01\xeaT\x00\x8f\xcec@\x93{\x8d\xcd\xc2\fٔ\x93\xd0\x1b\xf4\x1c\x06Bg\xa3\xae\xb9!\xb7\xe8\t\x1aE\xaf\xcb41\xaa\x8ad}XָE\xbd\f\xaa-\x84\x97\x9d\"\x94\x14=.\x85SEJĤQ+\xfb\xfa;?\ff8:\x96^\xb9!\x03yeڃ\x854\x1d\xef(\x0f\xcfK\xee\xae\x1c*\xa7\xb8\xaf\x02\x9b\x98\xbaǏ\x9b/0\"ɕ\x1aZl\xe7z\xc2\xcbX\x1ffS\x99\x06}ޗڔc\xa2\xa9\x9dU\x86\xd2\x0f\xa9\x15\x1a\x82\x10\xab^Q\x18{\x9dK7\r\xbbNR\x04\x15Bt\xb5 \xac\xa7\x0ew\x06֢G\xbd\x16\x01\xbfq\xad\xb8*\xa1\xe0\"\\U\xadC\x81\x9d:gz\x0f\x16Fy&j^\x01\x128\xe1[\xa4\xa9u\x82\xe5Kr\xe2\xe3_:q,X\xdfcٖ\xac9a\x00\x92\xf5\xe8\x87i\xa1.a\x80\xd9F\x9fE2\xf67\xd3\xc0\xbc\xb2\xa0\xb0\xd8\x1db:=\x9a?4\xb1\x9f?\xa0\x80\xdf\x13\xe6{\xdb^\\_[C<\x17\x17\x9d\x9e\xac\x8e=n\x8cp\xa1\xb3o\xf8\xde\x11\xf6\x7f:\xf4\xf9\x1a\xbe\xe8:\xde滫\xef\x82c\xd4g\xcf}D\xbeA\xf0|\xa6\x83\xc3UQ\xae\xc04x^\x95\xe8zs\xf7\x1e\nϸ\xbf\xa3Hw\xa6\xb1o\xa4\xb8w\x9c\xf5;#\x03\xe3\x97\xde\x10o\xf74\xbfBƞ\xe6-\xf9\xeeD\xf8\x14+\xf4\x06\t\xc3^\xa9_\x14u\xb3\x11\x01^:%\xbb\xb41\r\x04_\x02!X\xa9\xe6$\xf5\n\xf8\xac#\xca\xe3\xccP\x16iXg\xcc\f\xfe\xc4|F\xfd\xce\x1dP\f\x8at\x95\x82\x92\xa0\x18ޡ\xa1\xc9\x7f\xa4ZF\xef\xd3\x15\x95\xad\xfc2\x99n\xb8VDG\xe5\xf9\xeb\xf1\xfe\r%\xbd\xdd{\xa6\x17\xb7P&\xa3q\x1e\x8b\xa0Z~A\xf1\x1akiҸS2\xf2w\xfc\xc2;&j\xb6\xa2\xf8թ<\x80o@\xfc\xb8ŝ\x8f&\xdf\xf3\xd37l\n\x88\x81\x9f[ \x85\x99\xc1X!Ԩ\x91\xb0\x86\xea5\xdf\\\xaf\x81\xb0?\xc5\xddX\xdf\vZ\x01\xdf\xff\x05\xa9\x9962QkQi\\\x01\xf9x\xae\xcbf\x13w\x9d\b3cx\x94\xf3\x03\xfb\xcc5\xc6n\x18/v\x06\x9c\xbd_\n\xf8\x8c/3\xd6\ao%\x86\x80\xa7ct6\x93\xd9!81\x06~\xa4\xd5\a,\r\x7f\x19\x06\xcb\xff\x01\x00\x00\xff\xffx\xae@\xbaJ\x0e\x00\x00"), - []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4ZK\x93\x1b\xb7\x11\xbe\xef\xaf\xe8Z\x1flWi\xc8HI\\)ޤU\x9c\xda\xc4\xdel\x89+]\\>4\aM\x0e\xbc3\x00\f`\xb8b\x1c\xff\xf7T\xe3A\xce\f\x87\xe4\x92r\xa4\xb9Hģ\xfbC\xbf\xd1آ(\xae\xd0\xc8\x0fd\x9d\xd4j\x06h$}\xf4\xa4\xf8\x97\x9b<\xfe\xcdM\xa4\x9e\xae_^=J%fp\xd3:\xaf\x9bw\xe4tkKzKK\xa9\xa4\x97Z]5\xe4Q\xa0\xc7\xd9\x15\x00*\xa5=\xf2\xb0\xe3\x9f\x00\xa5V\xde\xea\xba&[\xacHM\x1e\xdb\x05-ZY\v\xb2\x81xf\xbd\xfe\xd3\xe4\xe5w\x93\xbf^\x01(lh\x06F\x8b\xb5\xaeۆ\x16X>\xb6\xc6M\xd6T\x93\xd5\x13\xa9\xaf\x9c\xa1\x92i\xaf\xacn\xcd\fv\x13qo\xe2\x1b1\xdfk\xf1!\x90y\x13Ȅ\x99Z:\xff\xaf\xb1\xd9\x1f\xa4\xf3a\x85\xa9[\x8b\xf5>\x880\xe9\xa4Z\xb55ڽ\xe9+\x00WjC3\xb8c\x18\x06K\x12W\x00\xe9\x88\x01V\x01(D\x10\x1a\xd6\xf7V*O\xf6\x86)da\x15 ȕV\x1a\x1f\x84r\xaf\x05D\x80\x10\x11\x82\xf3\xe8[\a\xae-+@\aw\xf44\xbdU\xf7V\xaf,\xb9\b\x0f\xe0\x17\xa7\xd5=\xfaj\x06\x93\xb8|b*t\x94f\xa3x\xe7a\"\r\xf9\r\x83v\xdeJ\xb5\x1a\x83\xf1 \x1b\x82\xa7\x8a\x14\xf8J:\x88\xa7\x85't\f\xc7\xfap\xcaq\xc6a\x9e\xb7;\x8f\x8d\xe9!\xb8\xb1\x84\xbb\xad\x11\x82@Oc\x00\xb6\xf2\x04\xbd\x04_\x11K>\x18\x16J%\xd5*\fEM\x80װ\xa0\x00\x91\x04\xb4f\x04\x99\xa1rb\xb4\x98\xa8L\xb4\a\xebn0zJ6\xbc\xfe\x8fF\xd5\x03t\xaf\xc5\x05P\xce\xe2\x1b\x17\xf7\xb8~\xe8\x0e\x9d\xb4\x8f\x8a\u009a̼5\xb5FA\x96\xd9W\xa8DM\xacY\x04oQ\xb9%\xd9\x030\U000b61cd\xe9\x83y\x9f\xe9uf\xce\x11F\xf2\x9d\xb9\xd7\x16W\x04?\xe82\x04(6iK=\x9bv\x95nk\x01\x8b\xcc\x05\xc0ymG\r\x9c\x11\xc7]\x89n&;\xf0\xb3>\xcf\xc3\xe8;\xb4s<\x9d\x94\xec#R\xabq\x0fz\xbd\xa2q\xef\x89\xd3\xeb\x971\\\x95\x1558K+\xb5!\xf5\xfa\xfe\xf6ß\xe7\xbda\x00c\xb5!\xebe\x0e\x9f\xf1\xeb$\x87\xce(\xf4E\xfdߢ7\a\xc0\f\xe2.\x10\x9c%\xc8E\x9b\x8cc$\x12\xa6\xa8\x1e\xe9\xc0\x92\xb1\xe4Hż\xc1è@/~\xa1\xd2O\x06\xa4\xe7d\x99LVT\xa9՚\xac\aK\xa5^)\xf9\x9f-mǶ\xc7Lk\xf4\xe4<\x84P\xab\xb0\x865\xd6-\xbd\x00Tb@\xb9\xc1\rXb\x9eЪ\x0e\xbd\xb0\xc1\rq\xfc\xa8-\x81TK=\x83\xca{\xe3f\xd3\xe9J\xfa\x9c2K\xdd4\xad\x92~3\r\xd9O.Z\xaf\xad\x9b\nZS=urU\xa0-+\xe9\xa9\xf4\xad\xa5)\x1aY\x84\x83\xa8\x906'\x8d\xf8ʦ$\xebzl\xf7\xac&~!ӝ\xa1\x1e\xce} \x1d`\"\x15\x8f\xb8\xd3B\x8e]\xef\xfe>\x7f\x80\x8c$j**e\xb7tO.Y?,M\xa9\x96\x1c\x03x\xdf\xd2\xea&\xd0$%\x8c\x96ʇ\x1fe-Iyp\xed\xa2\x91\x9e\xcd\xe0ז\x9cg\xd5\r\xc9ބ\xb2\x82cYk\xd8\xcc\xc5p\xc1\xad\x82\x1bl\xa8\xbeAG\x9fYW\xac\x15W\xb0\x12\x9e\xa5\xadn\xb14\\\x1c\xc5ۙȥ\xce\x01\xd5\x0eꗹ\xa1\x92\x15˲\xe5\x9dr)S\xa4[j\v8\\ޗ\xd3x\x00\xe0o4\xca\r\x17\x9d2:\xfeތ\x11ʀU'`\xe7h\x9c\x82g\xdd\x0f\x9e\xdd/\x87\xf0\xed\x1eKF;\xe9\xb5\xdd0\xe1\x18\xbd\x87\x06qP7\xfc)-\xe8\xc4\xe1\ued201ؼ\x15|\x85Ѻ\xb9x\xe3\xe0\xd6*\xb5υ?\xad\xce\x02f\xb48\x81+qD\xb0\xb4$K\xaa\xa4\x1c\x05\x8fU&#Ⱥ5\xc3>\xc6Ö\x02GR\xc6(\xe2\xd7\xf7\xb79-d!&\xec{\x91\xff\xa4|\xf8[J\xaaEȢ\xa7y\x8f\x9a(\x7f\xb7\xcb\b\"\xc4F\xaf\x01\xc1H\x8a\xb5\xe76/\x81T\xce\x13\x8a4\xc8\xe1\xc0R\x9a{\x11c\xdeA\x90\xfc\xed\xf2\x17\xeb\x04\x90c\xb0\x14\xf0\xcf\xf9\xbf\xef\xa6\xff\xd0\xf1\x1c\x80eI.\x14ٞ\x1aR\xfeŶ\xee\x17\xe4\xa4%\xc1Ubcjz\x012\xca|\x1bֳ\xd9H\x17\x0f\xbe\xa5\bO\xd2W\x01\xa8\xd1\"\x1d\xf0)\x1c\xc1\xe3#\x81NGh\tj\xf98\xe2?\xf1\xbb\x0eU\xd3\x0e\xe6o\xec=\xbf_\xc37э\xaf\xf9\xe7u\x84\xb1M\xe0]\a\xdb\xc1\x89^f\xe5jE\xbb\xf2l\xcfX8\xe1p\xa8\xfe\x16\xb4\xe5\xb3*\xdd!\x11\b\xb3\x9eb\xa4$\xb1\a\xef\xa7W?_\xc37}\x19\x1c`%\x95\xa0\x8f\xf0\nd\xba#\x19-\xbe\x9d\xc0C\xb0\x83\x8d\xf2\xf8\x919\x95\x95v\xa4@\xabz\x13K\xe35\x81\xd3|\xb7\xa2\xba.b\xa9$\xe0\t7\xa0\x97\a\xf8d\x15\xb1i\"\x18\xb4\xfeh\xb9\x94\xe4p\xdci\xf6\xeb\x87\xfc=\xcf_B=\xf1,\xef\xfdb\xb9\xf8\x99\x92\b\x85\xf3'H\xa2{\xe9\xb8@\x12\x8f킬\"OA\x18B\x97\x8e\xe5P\x92\xf1n\xaa\xd7dג\x9e\xa6O\xda>J\xb5*\xd8\x18\x8b\xa8u7\rw\xd9\xe9W\xe1\x9fK\x0f\x1en\xbd\x9fz\xfa\xde-\xfd\U000cb039\xbb\xe9%\x12\xc8u\xee\xf3s\xd7A9\xccS\xe95\xa4\xc9>\xffTɲʷ\x9eN\xb4mP\xc4p\x8cj\xf3\x85|\x87\xe5\xdcZF\xb4)RӮ@%\xf8\xffN:\xcf\xe3\x97\b\xb6\x95\x9f\x14\\\xde߾\xfd\x92\x1e\xd5\xcaK\"Ɂj>~\x1f\x8b\x1d\xaa\xa2AS\xc4\xd5\xe8u#\xcb\xc1j\xaefo\x05+i)ɞ(\xff\xde\xf5\x16\xe7\x02u\xa4.ޮ9\xab\xfe\xf4\xb8\x1a)\xf8\xba\xfd\xccce\xe1Qy\x9d6\x85\a\\9@K\x80Рa\x8bx\xa4M\x11+\x0e\x83\x92\xcb\x05\xae\b\xb6\xfd\x1b@cj\xce鱊\x18\xa1\x98\xea\xdf$\x1et\xe1|\x87\x042\xaa\xcaܯ\x9a\x93\xe7;\xf3\x97\x13\xce\xfb\x01\x90?VP\xdbn^\xa9\xd5R\xaeZ\x1b\xeeb\xfb\x92Rm]㢦\x19x\xdb\xee\x13z\x86 \x1fx\xc9\xf1\xf3\xbf\xef,\xcd\x16~\xa2\xf58~\xaa^Cr\xff0\xa4\xdaf\x1fJ\x01\x8f\xdaH\x1c\x19\xb7\xe4\xfc\x9e\xf7\xf2\xc4\xf5\xf59>\x16\x8d\xf2\x92\xbbuz&\x18\xb9\x95&CO\x05|\xbe\x99v;ãJ?#6X\xfa\xb5\xe5\xebH\x1fw1\xde8\x18\xac\xe1;\xf3`\xc8h1\x18\xe9\x87\xc1\xc1d\xaf{\xddE\xba\xdfM\t\x8f\x12g\xf4S\xe2cK\x92iL\x8e>?\xc1p\xd9}iG\xa5\xd4|\xfd\xeauv/\xd1\xf9\xcd>\x99\xd0\t\xb5\"9\x86l8\x0et\xdek\x12㱖H\x97\\\xdc\x19j\x14\xa6F\"\\\xa3\xf8\x96\xb7DY\x93\x80\xfc(w&\x95\x05-9GG'͍\x88\x04\xef\xf0\x05\xe6\xa1\"p\xa1\xaf\xf8\xb5\xdb\xd2l\x1d\x89\xd0\xd6\x1a\x11\xc2~\xc6^j۠\x8f-\xf2\x82I\\\x16\xbdF}\xb6!\xe7pu\xcai\x7f\x8c\xabb\x7f&m\x01\\\xe8\xd6o\x1b4\xbd\x8c\xf4\xb5K\x86v^\x8fh\xb4\xf5ѷq\xf4U6\xe9e[\xd7aO7:\xec\x1el\x03\xaa\x05\x8d\xd7uG\x1aD\xc7\x00V\xe8N\x89\xea\x9e\u05ccy\xdd6\xa4\x1du;8\x12\xbe\xef\xe8idt\xef\x05\xb5;y\x93]fd\xee\xfb\xe0\rg\x9d?1\xba\xc4\xdd3H\xa8t\x9d=\\{\xacA\xb5͂,\vg\xb1\xf1\xe4\x06\x81\x1f\x95\xe8Jr\xec\xf6\xb7۟\x95\x1a)\xa5\x0eF\x89*\xb4\xde],\x13\x84t\xa6\xc6\xcd\xf6,\xa1\xe6f\xff\x1ao\xd1\xee\x8c<{\xba\xa1C5\xc4\xf1\xd6b\xc0\xf4V\xab\x03\xb7\xd4\xec\xe4R\xf9\xef\xfer\xa4h\x97\xca\xd3j\x90F\xd2<\x8b\xf3\rs\xf9\xffp8R\x039\x85\xc6U\xda߾=a\x1a\xf3\xed\xc2\xec\"\xbbz>\x04\xc4\xd0\xfdO\x8b\x92)\x8c@\xdd\x05\x9c\xb3\xfc\xb7\xff\xa0\x7f\x89\x15\xcf{\x14N\xe4\xab\xf4\xf7\x05cYaN\x06-DŽ\xf0\xb6t3|)}\x01N\x86\x068W\xbb\xb1\xfc-+T\xab\xd1\xfe\x88V\xa1\x80\xd3v\xff\xa1\x0fN&\xa0\xfe\x81>g\xee\x195\xa7\xbd\xc1\x80\\th\xa7g\x95\xeeH\xbbؾ8\xce\xe0\xb7߯\xfe\x17\x00\x00\xff\xff\x8fTl=\x19$\x00\x00"), + []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Z_s\x1b\xb7\x11\x7fק\xd8Q\x1e\x92\xcc\xf8\xc8\xdam3\x1d\xbe\xd9r\xd3Q\x9b\xa8\x1aS\xf6K&\x0f\xcbÒ\x87\xe8\x0e@\x01\x1ce6\xcdw\xef,p \xef\x8e )҉}/6\xf1g\xf1\xc3\xfe߅\x8a\xa2\xb8B#?\x90uR\xab\x19\xa0\x91\xf4ѓ\xe2_n\xf2\xf877\x91z\xba~y\xf5(\x95\x98\xc1M\xeb\xbcnޑӭ-\xe9--\xa5\x92^juՐG\x81\x1egW\x00\xa8\x94\xf6\xc8Î\x7f\x02\x94Zy\xab\xeb\x9al\xb1\"5yl\x17\xb4he-\xc8\x06\xe2\xe9\xe8\xf5\x9f&/\xbf\x9b\xfc\xf5\n@aC30Z\xacu\xdd6\xb4\xc0\xf2\xb15n\xb2\xa6\x9a\xac\x9eH}\xe5\f\x95L{eukf\xb0\x9b\x88{\xbbs#\xe6{->\x042o\x02\x990SK\xe7\xff\x95\x9b\xfdA:\x1fV\x98\xba\xb5X\xef\x83\b\x93N\xaaU[\xa3ݛ\xbe\x02p\xa564\x83;\x86a\xb0$q\x05\xd0]1\xc0*\x00\x85\bL\xc3\xfa\xdeJ\xe5\xc9\xde0\x85Ĭ\x02\x04\xb9\xd2J\xe3\x03S\ued40\b\x10\"Bp\x1e}\xeb\xc0\xb5e\x05\xe8\xe0\x8e\x9e\xa6\xb7\xea\xde\xea\x95%\x17\xe1\x01\xfcⴺG_\xcd`\x12\x97OL\x85\x8e\xba\xd9\xc8\xdey\x98\xe8\x86\xfc\x86A;o\xa5Z\xe5`<Ȇ\xe0\xa9\"\x05\xbe\x92\x0e\xe2m\xe1\t\x1dñ>\xdc2\x7fp\x98\xe7\xed\xcecc\x06\bn,\xe1nk\x84 \xd0S\x0e\xc0\x96\x9f\xa0\x97\xe0+b\xce\a\xc5B\xa9\xa4Z\x85\xa1(\t\xf0\x1a\x16\x14 \x92\x80\xd6d\x90\x19*'F\x8b\x89JD\a\xb0\xeeF\xa3\xa7x\xc3\xeb\x7foT\x03@\xf7Z\\\x00\xe5\xacs\xe3\xe2\xc1\xa9\x1f\xfaC'\xf5\xa3\xa2\xb0&\x1dޚZ\xa3 \xcb\xc7W\xa8DM,Y\x04oQ\xb9%\xd9\x030Ҷ\x87\x8d\x19\x82y\x9f\xe8\xf5f\xceaFg;s\xaf-\xae\b~\xd0epP\xacҖ\x06:\xed*\xdd\xd6\x02\x16\xe9\x14\x00\xe7\xb5\xcd*8#\x8e\xbb:\xba\x89\xec\xc8Άg\x1eFߣ\x9d\xfc\xe9\xa4d\x1b\x91Z\xe5-\xe8\xf5\x8a\xf2\xd6\x13\xa7\xd7/\xa3\xbb*+jp֭Ԇ\xd4\xeb\xfb\xdb\x0f\x7f\x9e\x0f\x86\x01\x8cՆ\xac\x97\xc9}Ư\x17\x1cz\xa30d\xf5\xff\x8a\xc1\x1c\x00\x1f\x10w\x81\xe0(A.\xead\x1c#\xd1a\x8a\xe2\x91\x0e,\x19K\x8eT\x8c\x1b<\x8c\n\xf4\xe2\x17*\xfddDzN\x96\xc9$A\x95Z\xad\xc9z\xb0Tꕒ\xff\xdd\xd2v\xac{|h\x8d\x9e\x9c\x87\xe0j\x15ְƺ\xa5\x17\x80J\x8c(7\xb8\x01K|&\xb4\xaaG/lpc\x1c?jK \xd5RϠ\xf2\u07b8\xd9t\xba\x92>\x85\xccR7M\xab\xa4\xdfLC\xf4\x93\x8b\xd6k릂\xd6TO\x9d\\\x15h\xcbJz*}ki\x8aF\x16\xe1\"*\x84\xcdI#\xbe\xb2]\x90u\x83c\xf7\xb4&~!ҝ!\x1e\x8e} \x1d`G*^q'\x85\xe4\xbb\xde\xfd}\xfe\x00\tI\x94T\x14\xcan\xe9\x1e_\x92|\x98\x9bR-\xd9\a\xf0\xbe\xa5\xd5M\xa0IJ\x18-\x95\x0f?\xcaZ\x92\xf2\xe0\xdaE#=\xab\xc1\x7fZr\x9eE7&{\x13\xd2\n\xf6e\xada5\x17\xe3\x05\xb7\nn\xb0\xa1\xfa\x06\x1d}fY\xb1T\\\xc1Bx\x96\xb4\xfa\xc9\xd2xqdoo\"\xa5:\aD;\xca_\xe6\x86J\x16,\xf3\x96wʥ\xec<\xddR[\xc0\xf1\xf2!\x9f\xf2\x0e\x80\xbf\xac\x97\x1b/:\xa5t\xfc\xbd\xc9\x11J\x80U\xcfa'o\xdc9\xcfz\xe8<\xfb_r\xe1\xdb=\x96\x8cv\xd2k\xbba\xc2\xd1{\x8f\x15\xe2\xa0l\xf8+Q\x95T_r\xbd\x9b\xb0\x13\xa4\x12\xccv\xda*4\xbb\xa2H5\x00\xd5j\xa5\xd9\xc4\xc6Ҁ[\xcf\xcbX\xc9\x1d\xf9\xfc]U\xa00\xda\xc9\x17\x95\nvy \xf4\xf3\xbd\xf1\xa5\x17Zׄc^*-\xe8ĝﴠ\x9c\xb0x+\xf8\n}\xc2Ƌl\xab\xd4>o\xf9\xd3\xea,q\x18-N\xe0\xeaND\xb0\xb4$K\xaa\xa4\xe4\xfb\x8f\xe5c\x19d\xfdLi\x1f\xe3a\xfb\x80#\x812\x8b\xf8\xf5\xfdm\n\x86\x89\x89\x1d\xf6\xbdxw\x92?\xfc-%\xd5\"\xe4\x0e\xa7\xcf\xcej.\x7f\xb7\xcb\b\"D\x04\xaf\x01\xc1H\x8a\x19\xf76\x1a\x83T\xce\x13\x8an\x90\x9d\xa0\xa5n\xeeE\xf4\xf4\aA\xf2\xb7\x8b\xda,\x13@\x8e\xb2%\xc7+\xb4\x04\xb5|\xcc\xd8O\xfc\xaeC\xae\xb8\x83\xf9+[\xcfo\xd7\xf0Mt^\xd7\xfc\xf3:\xc2ئ-}\x03\xdb\xc1\x89Vf\xe5jE\xbb\xa4tOY8\xccr\x80\xfa\x16\xb4\xe5\xbb*\xdd#\x11\b\xb3\x9cb| \xb1\a\xef\xa7W?_\xc37C\x1e\x1c8J*A\x1f\xe1\x15{\x9f\xc0\x1b\xa3ŷ\x13x\bz\xb0Q\x1e?\xf2Ie\xa5\x1d)Ъ\xdeĂ`M\xe04W\x94T\xd7EL\x10\x05<\xe1\x06\xf4\xf2\xc09ID\xac\x9a\b\x06\xad?\x9a$v|8n4\xfbYS\xfa\x9eg/!\x8bz\x96\xf5~\xb1\f䙜\b\xe5\xc2'p\xa2_j]\xc0\x89\xc7vAV\x91\xa7\xc0\f\xa1K\xc7|(\xc9x7\xd5k\xb2kIO\xd3'm\x1f\xa5Z\x15\xac\x8cE\x94\xba\x9b\x86\n~\xfaU\xf8\xe7ҋ\x87Z\xffSo?\xe8M|~\x16\xf0\xe9nz\t\aRv\xff\xfc\xd8u\x90\x0f\xf3.\xe1\x1c\xd3d\x9b\x7f\xaadY\xa5Z\xaf\xe7m\x1b\x14\xd1\x1d\xa3\xda|!\xdba>\xb7\x96\x11m\x8a\xaeUY\xa0\x12\xfc\x7f'\x9d\xe7\xf1K\x18\xdb\xcaOr.\xefo\xdf~I\x8bj\xe5%\x9e\xe4@\r\x13\xbf\x8f\xc5\x0eUѠ)\xe2j\xf4\xba\x91\xe5h5\xe7\U00037085\xb4\x94dO\xa4\x7f\xef\x06\x8bS\x82\x9a\xa9\x06\xb6k\xce\xca?=\xae2\t_\xbf\x8b{,-<ʯӪ\xf0\x80+\ah\t\x10\x1a4\xac\x11\x8f\xb4)b\xc6aPr\xba\xc0\x19\xc1\xb6k\x05hL\xcd1=f\x11\x19\x8a]\xfe۱\a]\xb8\xdf!\x86dE\x99\xbats\xf2^\xaa/Ȝ\xf7# \xbf/\xa3\xb6=\xccR\xab\xa5\\\xb56T\xa0\xfb\x9cRm]㢦\x19x\xdb\x1e\xaa\xb9\x8e2\xf2\x81\x97\x1c\xbf\xff\xfb\xdeҤ\xe1'\x1a\xae\xf9[\rڰ\xfb\x97!\xd56\xfbP\nx\xd4Fbfܒ\xf3{\xd6\xcb\x13\xd7\xd7\xe7\xd8XT\xcaKJ\xee\xeeq$S\x95v\x8a\xde%\xf0\xa92\xed\xf7óB?\xc37pu\xcf\xe5\xc8\x10w\x91o\x97\x8c\xd6p\xcd<\x1a2Z\x8cF\x86np49\xe8\xd9\xf7\x91\xee\xf7\x90\xc2S\xcc\x19]\xa4\xf8\xc4\xd4\xf14\x06G\x9f\x1e\x9e8\xed\xbe\xb4\x8fTj.\xbf\x06\xfd\xec\x8b\xda,\xfbdB\xff\u05ca\xce0d\xc3~\xa0\xf7J\xd5\x1d\x9ck\x04\xf5\xc9ŝ!Gaj$B\x19\xc5U\xde\x12eM\x02\xd2S\xe4\x99T\x16\xb4\xe4\x18\x1d\x8d45\":x\x87\v\x98\x87\x8a\xc0\x85n\xea\xd7nK\xb3u$B3/Ä\xfd\x88\xbdԶA\x1f\x1f\x06\n&q\x99\xf7\xca\xdalC\xce\xe1\xea\x94\xd1\xfe\x18W\xc5\xfeL\xb7\x05p\xa1[\xbfm\xd0\f\"\xd2\u05eeS\xb4\xf3zD\xd9\xd6\xc7P\xc7\xd1WI\xa5\x97m]\x87=}\xef\xb0{\xa6\x0e\xa8\x16\x94\xcf\xeb\x8e4\x88\x8e\x01\xacНb\xd5=\xaf\xc9Y\xdd֥\x1d5;8\xe2\xbe\xef\xe8)3\xba\xf7nܟ\xbcI&\x93\x99\xfb>X\xc3Y\xf7\xef\x0e\xba\xc4ܷM\xcdJ\xd7\xc9µ\xc7\x1aT\xdb,\xc82s\x16\x1bOn\xe4\xf8Q\x89>'s\xd5\xdfn\x7f\x12j\xa4\xd4u0\xba^l09\xafAHgj\xdcl\xef\x12rn\xb6\xaf|cz\xa7\xe4\xc9\xd2\r\x1d\xca!\x8e\xb7\x16\x03\xa6\xb7Z\x1d\xa8R\x93\x91K\xe5\xbf\xfbˑ\xa4]*O\xabQ\x18\xe9晝o\xf8\x94?\xe6\x84#9\x90Sh\\\xa5\xfd\xed\xdb\x13\xaa1\xdf.L&\xb2\xcb\xe7\x83C\fo\x1eݢN\x152Pw\x0e\xe7,\xfb\x1d\xfe\x19\xc3%Z<\x1fP8\x11\xaf\xba\xbf\xaa\xc8E\x859\x19\xb4\xec\x13\u008b\xda\xcd\xf8}\xf8\x058\x19\x1a\xe0\x9c\xed\xc6\xf4\xb7\xacP\xad\xb2\xfd\x11\xadB\x02\xa7\xed\xfe\xf3&\x9c\f@\xc3\v}\xceؓU\xa7\xbd\xc1\x80\\\xf4hw\x8fI\xfd\x91v\xb1}g\x9d\xc1\xaf\xbf]\xfd?\x00\x00\xff\xff\f\x19\xe2z\x0f%\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Y_\x93۶\x11\x7fקع<$\x991\xa5\xdam3\x1d\xbd\xd9\xe7\xa6smr\xbd\xb1\xce~\xc9\xe4aE\xacHD$\x80\x00\xa0d5\xcdw\xef,@R\xfc'\xe9tnl\xbe\xdc\t\x00\x17?\xfc\xf6/\x96I\x92\xcc\xd0\xc8\x0fd\x9d\xd4j\th$}\xf4\xa4\xf8\x97\x9bo\xff\xe6\xe6R/v/g[\xa9\xc4\x12n+\xe7u\xf9\x8e\x9c\xaelJoi#\x95\xf4R\xabYI\x1e\x05z\\\xce\x00P)푇\x1d\xff\x04H\xb5\xf2V\x17\x05\xd9$#5\xdfVkZW\xb2\x10d\x83\xf0f\xebݟ\xe6/\xbf\x9b\xffu\x06\xa0\xb0\xa4%\x18-v\xba\xa8J\xb2伶\xe4\xe6;*\xc8\xea\xb9\xd43g(e\xe1\x99ՕY\xc2q\"\xbe\\o\x1cA?h\xf1!\xc8y\x17儩B:\xff\xaf\xc9\xe9\x1f\xa4\xf3a\x89)*\x8b\xc5\x04\x8e0\xeb\xa4ʪ\x02\xedx~\x06\xe0Rmh\t\xf7\f\xc5`Jb\x06P\x9f3@K\x00\x85\b\xcca\xf1`\xa5\xf2doYD\xc3X\x02\x82\\j\xa5\xf1\x81\x99V\x0e\xe8\r\xf8\x9cx\xcb\xc0*J%U\x16\x86\"\x04\xf0\x1a\xd6\x045\x12\x11\x84\x01\xfc\xe2\xb4z@\x9f/a\xce\xc4͍\x16s\xd5Ȭ\xd7D\xce\xef\a\xa3\xfe\xc0\xe7p\xdeJ\x95\x9dB\xf6\x7f\x06\xd5\xc3\xf3\xa0\xc5\x13\x91<\xe6\x14\xd64h*Sh\x14dy\xf3\x1c\x95(\b\xd8@\xc1[TnC\xf6\x04\x8a\xe6\xb5ǃ\xe9#y\xdf\xc8\xeb\xcc\\\xc3\xce5Tĵ\xbd\xed?t\x87.\xed\xfb\xa0E\xfd\x02\xd4F\rΣ\xaf\x1c\xb8*\xcd\x01\x1d\xdc\xd3~q\xa7\x1e\xac\xce,97\x01#,\x9f\x9b\x1c]\x1f\xc7*L\xfc\xb186ږ\xe8\x97 \x95\xff\xee/\xa7\xb1\xd5/ͽ\xf6X\xbc9xr=\xa4\x8f\xc3ሖ\x9d-\xab\xd5\xffE\xe0\xae\x19\xd2[\xad\xfa\xbc\xbe\x19\x8cN\x81\xed\bm\xe2\xed<\xb5\x14B\xed\xa3,\xc9y,MO\xea\xeb\xac/O\xa0\x8f\x03qz\xf72\x86\xb24\xa7\x12\x97\xf5JmH\xbd~\xb8\xfb\xf0\xe7Uo\x18\xc0Xm\xc8z\xd9D\xd7\xf8t\x92Gg\x14\xfa\xcc\xfe7\xe9\xcd\x01\xf0\x06\xf1-\x10\x9cE\xc8E'\x89c$jL\xd1y\xa4\x03Kƒ#\x15\xf3\n\x0f\xa3\x02\xbd\xfe\x85R?\x1f\x88^\x91e1\xe0r]\x15!\"\xed\xc8z\xb0\x94\xeaL\xc9\xff\xb4\xb2\x1d\xfb\"oZ\xa0'\xe7\x03\xd7Va\x01;,*z\x01\xa8\xc4@r\x89\a\xb0\xc4{B\xa5:\xf2\xc2\vn\x88\xe3G\xb6\x1f\xa96z\t\xb9\xf7\xc6-\x17\x8bL\xfa&\xa5\xa6\xba,+%\xfda\x11\xb2\xa3\\W^[\xb7\x10\xb4\xa3b\xe1d\x96\xa0Ms\xe9)\xf5\x95\xa5\x05\x1a\x99\x84\x83\xa8\x90V\xe7\xa5\xf8\xca\xd6I\xd8\xf5\xb6\x1dyd|B\"\xbcB=\x9c\x19A:\xc0ZT<\xe2Q\vM|\x7f\xf7\xf7\xd5#4H\xa2\xa6\xa2R\x8eKG\xbc4\xfaa6\xa5\xdap\x84\xe6\xf76V\x97A&)a\xb4T>\xfcH\vIʃ\xab֥\xf4l\x06\xbfV\xe4<\xabn(\xf66\x94\x1d\x1c\\+\xc3f.\x86\v\xee\x14\xdcbI\xc5-:\xfa̺b\xad\xb8\x84\x95\xf0$mu\x8b\xa9\xe1\xe2Hog\xa2\xa9\x84N\xa8vXݬ\f\xa5\xacY&\x97_\x95\x1b\x99F\x9f\xdah\v8Z\xdfgj:\x04\xf0\xb3\xc6t[\x99\x95\xd7\x163\xfaAG\x99\xc3E\x97̎\x9f7S\x82\x1aĪ\x93P\xe3\x8e\xe0\xe2J(\xea\xa5\x13\"\xf79Y\xea\xbec\xc9h'\xbd\xb6\a\x16\x1cS\xf1\xd0$Nj'\xf0\xa0Ņ\xb3q.\t\x0ediC\x96TJM\xb89W&M\x80\xefT\vc\x88\xa7\xf5\x01gB\xf3$\xe0\xd7\x0fwM\xf8m\x18\xae\xa1\x8f\"\xecEz\xf8\xd9H*D\xc8V\x97\xf7\x9e4\x04~\xee6\x11D\x88A^\x03\x82\x91\x14\xcb\xe06\xfe\x83T\xce\x13\x8az\x90\xdd\xceR=\xf7\"Ɩ\x93 \xf99\xe6\tV\t \xc7:)\xe0\x9f\xab\x7f\xdf/\xfe\xa1\xe39\x00Ӕ\x9c\v\xe5\x00\x95\xa4\xfc\x8b\xb6$\x10\xe4\xa4%\xc1u\x11\xcdKTrC\xce\xcfkid\xddO\xaf~\x9e\xe6\x0f\xe0{m\x81>bi\nz\x012rކ\xcf\xc6j\xa4\x8b\ao%\xc2^\xfa<\x005Z\xd4\a܇#x\xdc\x12\xe8\xfa\b\x15A!\xb74\xcd>\xc0M(4\x8f0\x7fc\xd7\xfa\xfd\x06\xbe\x89\xcer\xc3?o\"\x8c6Qv\xbd\xef\b\xc7\xe7\xe8\xc1[\x99et\xachG\xc6\u0081\x9dCⷠ-\x9fU鎈 \x98\xf5\x14\x03\x12\x89\x11\xbc\x9f^\xfd|\x03\xdf\xf498\xb1\x95T\x82>\xc2+\x90*rc\xb4\xf8v\x0e\x8f\xc1\x0e\x0e\xca\xe3G\xde)͵#\x05Z\x15\x87xA\xd8\x118]\x12\xec\xa9(\x92X\x92\b\xd8\xe3\x01\xf4\xe6\xc4>\x8d\x8a\xd84\x11\fZ\x7f\xb6,\xa9y8\xef4\xe3<\xdd?\x05\xbc\xbb[<\x87\x81\xa6\x9e|z\xee:\xc9ê\xaep\x862\xd9\xe7\xf7\xb9L\xf3\xe6vщ\xb6%\x8a\x18\x8eQ\x1d\xbe\x90\xef0ϕeD\x87\xa4n\x9e%\xa8\x04\xff\xef\xa4\xf3<\xfe\x1cb+\xf9I\xc1\xe5\xfd\xdd\xdb/\xe9Q\x95|N$9Q5\xc7\xe7crD\x95\x94h\x92\xb8\x1a\xbd.e:X\xcd5\xe3\x9d`%m$\xd9\v\xd5\u07fb\xde\xe2\xa6z\x9d\xa8>\xdb5W\x95\x9fN\xa1q\xb9\xf6wo/\xe0X\xb5\v\x1b\fG\x1d\xd6Eg#kЙ\xba\x0eO\xf0\xad\xfbӑ\xab\x0f\xaa\xbf\xbaA\xa6\xad\xcc$߿\xdb\xf0\x11\xae$\nK\xecv$\xbbO\x89\xc6H\x95]\x85\xb5i\xf0\xad\xc8\xf35v\xa2p\xee\xb6fϕ\xd7g\xed\xee\xb2K\xbd\x1f\x00\x01\xb4\x04\xc8gb\rm\xe9\x90\xc4*Π\xe4\x12\x8c\xab\xac\xbaT]\x13\xa01\x05\xd7I\xb12\x9b\xf2\xf5\xa6]\x99j\xb5\x91Ye\xc3\xe5h̔\xaa\x8a\x02\xd7\x05-\xc1\xdbj,\xe8\x8c\xfbt;\xa5\x174\xfe\xbe\xb3\xb4Q\xf7\x85^\xed\xf4\xa9z\x1d\xdc\xf1aHU\xe5\x18J\x02[m$N\x8c\xb3\xb1\x8f\x1c\x9d'nn\xae1\xa9\xe8I\x178\xa8\x1b\x8b\x13\x17\xd9\xda\x11벞G\xf8\xf2\x18\xdcq:;^렖~\xad\xf8\x8e\xd2G\x98L\xdf\xd9\ak\x8c\x16\xb3!i\xdd\xd86\x980\xf5\x9f\xf1ע\xc1|\xfb-\xec\x8f\xd9\xe1L\x99\xe0\x83\x00\x17\x00\xa5h\xdb\xfe\xf7\x0e\x0e.$%\x90\x84d˛-^2\xa6\x80\x03\xe0\xdc\xcf\xc1\x01\xb2X,\xdeц}\x03\xa5\x99\x14W\x846\f\xbe\x1b\x10\xf6/}\xf9\xfco\xfa\x92\xc9\x0f\x9b\x8f\uf799(\xaf\xc8M\xab\x8d\xac\x1fA\xcbV\x15\xf03\xac\x98`\x86I\xf1\xae\x06CKj\xe8\xd5;B\xa8\x10\xd2P\xfbY\xdb?\t)\xa40Jr\x0ej\xb1\x06q\xf9\xdc.a\xd92^\x82B\xe0a\xea\xcd?^~\xfc\xd7\xcb\x7fyG\x88\xa05\\\x11\x05\xdaH\x05\xfar\x03\x1c\x94\xbcd\xf2\x9dn\xa0\xb00\xd7J\xb6\xcd\x15\xe9~pc\xfc|n\xad\x8fn8~\xe1L\x9b\xbf\xf4\xbf\xfe\x95i\x83\xbf4\xbcU\x94w\x93\xe1G\xcdĺ\xe5T\xc5\xcf\xef\bхl\xe0\x8a\xdc\xdbi\x1aZ@\xf9\x8e\x10\xbft\x9cv\xe1W\xbd\xf9\xe8@\x14\x15\xd4ԭ\x87\x10ـ\xb8~\xb8\xfb\xf6OO\x83τ\x94\xa0\v\xc5\x1a\x83\b\xf8\x9fE\xfcN\xc2B\tӄ\x92o\xb8Q\xbb\x1aD<1\x155DA\xa3@\x830\x9a\x98\n\bm\x1a\xce\n\xc4;\x91\xab\x1e\xa40J\x93\x95\x92u\amI\x8b\xe7\xb6!F\x12J\fUk0\xe4/\xed\x12\x94\x00\x03\x9a\x14\xbc\xd5\x06\xd4e\x04\xd4(ـ2,`ٵ\x1e\xef\xf4\xbeNm\xcc6\x8b\v7\x8a\x94\x96\x89\xc0m\xc1\xe3\x13J\x8f>\"W\xc4TLw[\r\xdb#T\x10\xb9\xfc\x1b\x14\xe6r\x0f\xf4\x13(\v\x86\xe8J\xb6\xbc\xb4\xbc\xb7\x01e\x91Uȵ`\xbfG\xd8\xdan\xdcNʩ\x01m\b\x13\x06\x94\xa0\x9cl(o\xe1\x82PQ\xeeA\xae\xe9\x8e(\xb0s\x92V\xf4\xe0\xe1\x00\xbd\xbf\x8e_\x90xb%\xafHeL\xa3\xaf>|X3\x13$\xaa\x90u\xdd\nfv\x1fP8ز5R\xe9\x0f%l\x80\x7f\xd0l\xbd\xa0\xaa\xa8\x98\x81´\n>І-p#\x02\xa5\xea\xb2.\xff.\x12u0\xad\xd9Y\x1e\xd5F1\xb1\xee\xfd\x80\x02q\x04y\xac\xa88\xc6s\xa0\xdc\x16;*\xd8O\x16u\x8f\xb7O_\xfbLɴ'J\x8f7\xc7\xe8c\xb1\xc9\xc4\n\x94\x1b\x87\xacia\x82(\x1bɄ\xc1?\n\xce@\x18\xa2\xdbe͌e\x83\xdfZЖ\xdf\xe5>\xd8\x1b\xd4:d\t\xa4mJj\xa0\xdc\xefp'\xc8\r\xad\x81\xdfP\roL+K\x15\xbd\xb0DȢV_\x97\xeewv\xe8\xed\xfd\x104\xe2\bi\xbd\x16yj\xa0\x18H\x9a\x1d\xc6VA]\xac\xa4\x1a(\x19;d\x88\xa3\xb4\xf0\xdb洈U\x8b\xfb\xbf\xccq\x99m\xff\x1eG[~\xb3+k\x05\xfb\xad\x05T\xa6N\xfc\xe1P_\xa9\x9ej\x1f6\xcbF\xfb\xd4\x1dE\xb4m\xf0\xbd\xe0m\te\xd4\xeb\a\x1b\xcc\xd9\xc6\xed\x01\x144z\x94\t+D\xd6\xfaؽ\x88\xeeWT\xe0T\x01\x11\xd2$\xe01\xe1\xe0\x11&\x10\x03I\x9a`G\x03ubœ[&D\xb4\x9c\xd3%\x87+bT{\x88F7\x96*Ew#\xd8\n\x1e\xc0\x8b\x90\x15\x81xU\xc3Y\x81$\x8f\n\x05\xf1\xf5\xe7E\x15\xd3VQ\x86]>HΊ\xdd\f\xben\x93\x83\x82\xb4z\xd9\xf5;$K\xa8\xe8\x86I\x95\x12\x03\xa9\xb0kϞwjZZ-\xe9\x81\xec۸\xcc\r'\x91UI\xf9<\xc7\x10\x9fm\x9f\xce:\x90\x02\x1dʸ\x15Omo\xbb\x97@\xe0;\x14\xadI,\x93\x90\xb2E\xd3$\x15i\xa46\xe3t\x1fW]\xa4\xef\x1c\xa5~\x9c`\x9a\x83\x9d%Y\xdd5\xaf\x84\x03Q-\x0e\x06\nY\n\xb0ۨ-Q\xbb\xbeJ\xb6\xae\xef(RȒj(\x89\x14\xa33#\xbb\xb4\x1c\xb4\x9f\xabD\xce\xe8\xf4\xd0E\xb7\x7f\xf4x\b\xa7K\xe0D\x03\x87\xc2Hu\x88\xcc\x1c\x94\xba\x96\xa3XGP\x99ЦC\t\xe860\x01\x92XN\xdfV\xac\xa8\x9c\x87a\xd9\x13\xe1\x90R\x82\xb6\xda\x04]\xe6\xdd\xd8&\xc9\x1c\xf9\xfd$Sڣk3b\xb5\x0f/\xa5Q\xba\x96\xa1\x86\xbb\x96Dm\xa7{\x0ft\x8b\xffn\xe4\xe4\xb6\xff\x7f\"6\x18\x93\x13\x98vB\xfe\t\xba\x9f\xd9<=ʷ\x18ၾ$w+\x02ucv\x17\x84\x99\xf0uN\x12(\xe7\xbd9\xfeĴ9\x9e\xe93I\x93#\x13g\"L\x9c\xe2OH\x174\x19O\xdebd\xd3\xe4\xaf\xfdQ\x17\x84\xad\"\xd2\xcb\v\xb2b܀\xda\xc3\xfeI\xaa>P\xe65\x90\x91c\xf5\b\xe6\tLQ\xdd~\xb7.\x8e\xee\x92`\x99x\xd9\x1f\xec|\xe3\x10A\f\xcd\xf3\f\\\x82\xf12SPc\x1cN\xbe\"6\xbb/\xe8T_\xdf\xff|\x18+\xef\xb7\f\xce;\xd8Ȍйv\xbd\xb7\xa3\xfe\xfa|T\x10~A\x1f(\x06U.\xe7rA(y\x86\x9ds]\xa8 \x96>4tΘ^\x01&\x7f\x90Ϟa\x87`\xd2ٜÖ\xcb\r\xae=C\xc2\xf5O\xb5\x01\x0e\xed\x9a|X\xec\xf0d? \"0\x86\xcfe\x03\u05fc($r'閩KB\v\xb8?a\x9bY\xacҟ\xa3\x9f\xfaD\x0e\xf8I;ZZ\x89\xa9\x98\xcfij@\x99\xc9%\xa8k\xdf(ge\x9c\xc8\xc9ȝ\xb8 \xf7\xd2\xd8\x7f0@\xd3\xc8(?K\xd0\xf7\xd2\xe0\x97\xb3`\xd4-\xfc\x9c\xf8t3\xa0\xa0\t\xa7\xe5-\xc2\xfa9?g\xd3,\xb7E\xdc3M\ue10dW\x1cJ2\xa7\xc2\xf4\xae\x9b\xceMT\xb7\x1a\xd3uB\x8a\x05\xda\xcc\xe4L\x1e\xdfR\r\xd0\xfd\xe2I\xfd\x84_\xad\xb1p\xbf\xb8$3\xa7\x05\x94!\xb2\xc4\xec'5\xb0fE\xe6|5\xa85\x90ƪ\xf0<\x8e\xc8T\xac~7DZO\x9e\xf5\xee\xb7\xef\x8b\xe7\x98/XX\x93\xb3\xf0\x10\x8c\xac3p\xe0uw9\xbf\x9f\x85\x95ٌ^\x81\x13f\xbb\x8e$Gǻ\xe6 \xe5\x05\xe8@+\x8e.\xce,uiY\xe2\x11\x1a\xe5\x0fGX\x94#x\xe1X\xd5\xd0[\xbb3\xc15m\xacZ\xf8okiQ\x9a\xfe\x974\x94)}I\xae\xf1\xa4\x8c\xc3\xe07\x9f\x87\xeb\x81ɘ\xb2\xb1SY\xfe\xd9Pnm\xbfU\xe0\x82\x00w\x9e\x80\\\x1d\xf8E\x17d[I\xed\xcc\xf6\x8a\x01\xc7\xf3\x8a\xf7ϰ{\x7fa\xa7\x9f\x9d\xb2\xafd\xde߉\xf7·8P\x18\xd1ᐂ\xef\xc8{\xfc\xed\xfdK\\\xa9LN\xcd\xec6`њ6y\x1c*\x92\xc9\xfa\xae\r8\xa6\x9f\x9b\xef\x92\xf2\xdeɞ\xdam\x16\x8b6R\x9b\xcf\xe9\xbc\xe1\xc8z\x1e\u0088\xa1g\x9cȱ\xcdF\f>\x8f\x16\xf5\xbdu\"W\x06\x94\xcf%:\x1b\x10\xe2\x8f\x17Ff\xa9S\x99\xfebc2\x90\xc6\xfc\xaeE\xf0\f7\xb9\x83\x9b\x9c%\x1e\xe3\xb0Z\xbc\x1c\xe9\xed\xdf~\xef\xe53\xad\xe4ڿ\xfb\x1bym\x87\xba\x90uM\xf7O5\xb3\x96z\xe3F\x06\x9e\xf6\x80\x1c\xf5պEyε\xc8\x1d\x0f\xe1\xf9喙\x8a\tB\x83\xda\x00\xe5\x19\x8a\x92F\xa6rةVQM\x96\x00\"\xa6\xe8\x7f\x04W\xa2f\xe2\x0e' \x1f\xcf\xe0zDt\x9d\xd3ٽ\x894\x89\x94\x8f\x1f\x9c\xc9jdI\xb6\x15(\x180\xc6a\xde\x1d=U!M/eq\x84C\xda\xc8\xf2'MVLi\xd3_\x82&\xadΥ\xf5\x91\xe4\xb3\xeb\xfe\xcaj\x90\xad9'\x82o\xbbi\x06g\xcd5\xfd\xce\xea\xb6&\xb4\x96\xad3\xe6\x86\xd5\xf1TףwK\x99\x89\xc7V\x98\xbf1Ғ\xa0\xe1`\x80,a\x95>\xefM\xb5B\n\xcdJP\xa1J\xc1\x91\x8dI+\x98+\xcax\x9b:%J\xb5c#`q\xab\xd4I\x01\xf0\x177\xb2\x97w\xac\xe4v\x88\xa0̽\xe3A\x1a\x10\xb6\"\xcc\x10\x10\x85\xc58(\xa7\x92q\n\x8f\fD\r\xcb\xd5sy\n\xdc6\x10m\x9d\x87\x80\x05\n$\x13\x93)\xb7~\xf7O\x94\xf1s\x90\xcdr\xde'\xa9\x1e\x81\x96\xa7\xe4h~\xed\r' t\xab\xf0\xf0\xdf\xe9\x8e-\xe3yk\xb6\x94#\x9c\xb6\xa2\xa8\x00\x95\x90\x18\xea\x06\a\x9e\tm\x80\xe6\xf2\x82\xf5\x8aZ!\x98X\xe7\xd1.;\x11\xda5\x87\ua954\x1c\xe8\xf8)d\xd7,\xae\xdf@\x13\xfd\xdaM\xf3BM\xd4\x11\xc1\x1d\x9b#\x1d\xb2)j\x95\x16\xa1\xc6@\xdd8\x91\x93D\xb5\xa2o]Π\x88\x8e\t\xc3\xfd*^3\xbef\x82e\xd0v@\xd7;\xc1L\xdfy\xb4 \xce\xea<\xda\t\xa2;pJ\x86\xedn\x00\xc0\nh\x88Cp\xed\x91k\x8ep$\x97@hYB\xe9r\x97\xd6\x15\xf1a\x89+|\x1b)nH\xee\xeexO0\x8b\xb2\xa1\r\x82N\xccê\r,Z\xf1,\xe4V,0\x18\xd7G\xeb\x90\x13\xb3T/\x9dޜ\xac\x8c\xe6\xf5K\xbe\x9a\x9e\xd3BC~\xcd\xe7\xa9\xe0?\x9dA\xcbd\xf3\xcdQ\t\x8f).\x98\xd3k\xae\x00{\xe4\xc7\xd9UL\xcd?1\xd8\x1fJ߸b\xe9\x17\x95\xc5ݥA\xf5\x9c\xc2m\x05\xa6\x02\x15J\xb3\x17X\x92^N\x9e\x90v\xc1K\xac\x93\xb3L\x15\\dW\xfe\xb9W9\x87\xd1M\xcb\xf9\x85\xe5m\xda\xf2d8l$\x8a\xd8!geՏ\xa5=\x86\x9c\xea\x8bl<\xf6+-\x86\xf5\x85\xb1\n\"\x14\x18\xca0\xb3\xa7qj\xbfXX\xda;\xdf\x1f\x96S`\xfe/,\xff\x0f/=̨\x94\xc8Gcn\x95fDb\x02V\x82\xc1zh\xec\xea+|?_\xe8\xfbc\xe1\xd4@\xfd\xa5\xf1\x123\xea\xc2f\xa05\x01g\xaf\xde\x04\xadA\xab\x9d+\x10\xed\x80\xcf\x19\xda\xf1ׅ\xbb\x05\x11\xc0\xa4\xf8\xf5k\x05A|}\xf5>\xd3\xe4\x9fI%\xdbDU\xdf\x04\xcaf\xaa;\xe67<(\xf4\xf0\a\n`\xe8\xe6\xe3\xe5\xf0\x17#}\xd9\af\xd1\x12\x800(\xea2\xb3L\x94l\xc3ʖ\xf2 \xb5\xdd\x1d\x02\xc7@\x1d\x9f%\xa0IE\x04\xe3\x8e\x01\xc3\xf8\x01Ñ/\x8d;\x969Z\xc5M\xfb\xa2y\xd5!'ׄ\fk>F\xac\xe1\xb1\xc7\x17\xafR\x05\xfb\x87\xd4z\x1c_\xe1\x91\x13I\xccTs\x9cPÑY,\xf6\xe2\xf3\x96\x9c*\x8dcb\xee\xb3Ud\xbc~\x1dF\x16~\xe6k.\x8e\xc1\xce\xd9\xeb+ް\xaa\xe2mj)2+(^\xaf\x142/\xfa<\xa9\x14`>`\x19\xaf\x82\x98\xad}xQ@sҖfk\x1a\x8e\xa9d\x98\xa5N\x9e\x98\xbdY\xad\u009bU(\xbcm]\xc2$\x17M\xfexL\xe5A\x8c\x93~\xa1M\xc3\xc4\xfa\x90)rYg\x92m\xe6Y\xe6~o!\x03\x9e\xe9\x873]t8\x12\xfa\xba\xeb҉H2\xa4-\x990\xf2\x92\\\x8b\x9d\x87\x9b\x80\xd3\v\x1f\x854\a\x17\xd9첶\x8c\xf3\xfem-\x04;\r\xcaߙԴv\xab\x1a\xf3\xf6\x93t\x95j\xe0\x94\x9f\x148~ك\xd1ώ\xbe\xa5\xe7_\xb7ܰ\x86\x83\xf5\xe86\xacL\xde!3\x15\xec\"\x92\xff&\xf1\x86\xd4r\x87\x90\xbe]\xf4\x9eQ\xec\x9eq\x926\xbf\xc9\x13\xb6\x97Q\xcc~\\\x11{\x06\xcdrE\xf1\r\x8b\xd5߰H\xfd\xad\x8b\xd3g8k\xe6\xe7\xe3\x8a\xd0O>\x81\tG\xfd\xf7\xb2\x84\a\xa9\xcc\\p\xf2\xb0\xdf?q\x92\xda\v\xd8$/\x89\b]\x13\xbb\xc4\x10Ç\x17\xa7m*}\xe8\x19\xdc\xe9_di\xd76w\xc6\xf2\xb8\xd7\xfd\xe0\xae\xf2\n\x14\b\xf7\xcc\xc7\x7f>}\xb9\x8f\xf0S>\xaf\xf7\x8c\xf7\x9e\x97p\x1eL\xe9\x91\xe3\x8f\xe6|1\x93\xc3\x16\xfa\x00\xaf|.B\x1b\xf6\x1f\xf8\xaa\xdb\v\xd2A\xd7\x0fw\b#\xf8i\xf8L\\\xac\xa2\x88'\x96K\xb0\x16+\xa2jT,\xeeV\x03\x88Ê\xdf\xfe3JP\xba'\xb3\x82\xc5d\xa1\xc6\xcb\n\xdeÝ[\xc7\xd8,\x9f\xac\xd3(vD:\x8e\xac\x98*\x17\rUf\x87l\xa3/\x06k\bff*\x9d3\xaaX\x0f\x9f\x01K\xa27\xbc\xfe\x85g\x91\xbbfxڻ\x8f\xbbS\xd61~\xffd\xf6\xe6\xc9+\xaec\xdcb/\x10S\x89\xcf\xc9\x02\x93WK\x93yM\xf4\xf0\xed\xa4\xb4\xcbc\x1c=\xad\xe7l\x14\x1dRM\t0v<\xaa:-h\xa3\xabēK/\xd3u\xf8\x1a\x99\xa1\xa6}\xc9&\x1d\x80\xc1>YQ\xf5\xb4\xd5\x16\x82>\v\xdbFi\xc5a)\xddnm\xb3\xab{a\xfc\xa2\x97)x\x9b#\xe1\xcc\xe7\\N~\xc8šgD\xfd`\xf6˪\xb6CL\x9dp\x18<\xeb\xdae\x14\x19O;\xb1\x99π\xe4\x19\x8c\x13\x9e\xfe@|\xe5\xe2\x8a$_\x04\xc9|\xf5\xe3\x0fE\xf4\x84V\xd3E\x05e\xcb\xe1\xd47\xff\x9ez\xe3\xe7_\xfd\v\xb3e\xbc\xfbg\x91\xdd3\xd0\xd6g\x1e\xbe/\xe8)\xe1!\xf7)9\xe6\xf0ap\xe0\x9e\x17+\xdcK\x94E\x01Z\xafZ\x1e\xaa\x94\n\x05\xd4@\x19\xba3\x1dW|T\x9dM\xdbpIKP7R\xacX\xe2\x84d\x80\xd6\xff\x1at\xde\xe3\xd9\x02?\xb6\xaa{\xdaq\xf2Y\xbc\x17i\xae\x86*\xca9\xf0O\x8c\x83\xfeYn\x85]W\x86@>\xa4\xc6\xf5\xeee\x15\xad\xb2f}GD[/\xad\x93\vƌ\a\x8b+\xa9\xa6+\xa4\x1dޙ0\xb0\x86T|\xbdU\xcc\xc0SC\x95\x06\\Q\xc6\x0e~\xdd\x1b\xe2\xa2\xcf\x15\xa7kW\nW\xb2\x82\x1a\x88\x06\x18g\x18[>\x8e\xd7\b\x8b\xef\xb02I\x8e$\xbd\xb2\x85z\xecJƨX\x8f=/\x9a0\xd5\xc9\aF\x9dE.hc\xf0\x02\f\xd2\x11\x89h<\f|\xb4w\xef\x8d\xd1\x01\xd8qN\xf3e̾`N\x1bZ'\xa2\x84y\xbdss\b\x06\x9f\x05Ve\xaf\xee\xae\xff\xc0b,\xb0#[\xaac1u\xd2\xf7\xee`;0\xe8\xaa[\xd0P\x12\u0600 V\x14)\xe3PNq\xeaWL$\xab\r\xa8\x9ft\x84\x83\x95\x80\x96ş\fU&.\xfdЏYIUSsEJj`aG\x9f溥\x9fIU\xea\xc4\xe3@\xbc\xd9\xe6ţ\b\xd7n\xac\xf5s\xf7\xd1jК\xaeC\x10\xba\x05\x05d\r\xc2\xe2=\xe6\x16\x93\x1eS\xb8\xd2\xe7\x8dE,-\xb5(\xa4\x85i\xa9\x9f\xc0\xb9p\xf1\xf44\xbcO\x8cQ\xeczTE\xa7U\x85\xbf<\xf8\bT\xef?w}\x80\x8bO\xfd\xbe>I\xecv\xec\xceF\xa8+\xf0\xc4\a\x8f\r\x8b\x91uJ\xa6\x8dę\x8f2'\x95\x94\xcfYn\xf6\xe7رK'1\xe1X\t\xafL.ekz~\x8eGxb\x99\xf8\xfc\xe7+\xdb\x17\x84y\xed.P\x8d\xe5V\xf3<\xbd\xcf\x03H1\xbc\x95\x86\xf2`d,_\xc6\x0e\xd5\xc4\x03\x02O\xe1\xf1d\xcew\x17\xfb\x90\xf7^e\xef`W\xddS\x9e^\x13t\xd7\xc7\xc7Ҫ>\xeb\x97\x04\x12_\x01\xed|\x92\xb17\x17\xe7\xec\x1fB\xfd\x84\x8b\xca\xc0\xf1\xe7\xae\xf7\x18\x1e\xdd2\x9d\xc3\f\"\x1di\x12\f>L\x15%ㄥOx\xa9ME\xf5\x9c{\xfa`\xfbD\xb7\xa3g\xae\xa2\x13\xfa8\"\x95\xe9{\xae\vr\x0f\xdb\xc4W\x87,<\xfdB\xa9Jt\xb9\x13\x0fJ\xae\x15\xe8C\xa6[\xe0}F&֟\xa4z\xe0횉/\xe3\x95\xdfS\x9d\x1f\xa82\xcc2\xad[Ob\xecM\xb0q\x89\xdf\xe6G\x8f\xff\xc0\x04\xe5\xec\xf7\x94.\xef\xff87Ä\xbek<\xf2N\xb1P\x01\xf1s\n\xd0k\xe8\x9ft\xcf\xfc\x84y/ɽL\x8a\xb1? fC\xa0L\x93%h\xb3\x80\xd5J*\xe3\xf2\xf7\x8b\x05a\xab\xe0 Y\r\x81q\xa2{͞\xb0T\xe2=\x1e\xbd\x05\x87e\xe5S\x89\n\xad\x0e\x86\x9c5ݹ\x8c$-\n\x1b\x13\xc0\amh*6y\x91\x9e\xc6P\xd5\xcbJ\x8e\n\xb9\xeb\xf7\x8f9\xbe\xa8>\x10\x9cC\x1d^gw\x06\x9d\x8f\x9di\r^\xcb \xdab\xef\x14eB\x9c\x1a\xbb\x1b\x0f\xbb\xf3L\xcd\xd7\beL=\xfa\xfd\r\x1e\xe2\xf6\a\xac\xbe\x93%[QQ\xb1\x1e\xbd\xd0V)ٮ\xab\xc0\x9bc\x0e\x11)[\x8c\x9c\x1bT\x05:\xfc\xc7!\xa6U\xa2wh\xe7k,ƴt\\\uee0f\xf2\x02E\xad\xba\x8b-\x9d\xaa\x9a\xb0\xf9\xd9Y\xc2\x11\x88\xb3\xb6?\x01\x91\xea\x9d(&\xaf\xe0\xf8@\x9bM\xdc՝\xc2P\x12\tQ\x1b\xbf\x1a\x12\"\xc41$\xf4}\x89.\xe2\xf9a02棜\x88\x8ei'\x06\xb78\rj~\xd3}'h\xe8\xee\x1c\x87\x0e=\b\xfeNJ\xbb\r \x1c\x13\xf9\xe2\xdc\xe9\xb8\xf7ǍX7\xd1ۺ=9v\xfd\xb6\ac\xef\n\xa4\x8db\xbbiB\xbc\xf9\xf7l\x95\x92\x17\xf7\xbf3-9\xfc\xc3\xc1\xafo|\x95qK\x95`b}\x12F~\xf5c\x13\xf1\xbc\a{Έ>\xac\xfc\xd5b\xfa\xa4Y:\xf8\x88\f^\xf6\xf0\xecg\xf2_\xfe/\x00\x00\xff\xffP\a\xb5\x16Cm\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xec=]s\x1c)\x92\xef\xfa\x15\x84\xeea?B\xdd^\xc7}ą\xde|\xb2\xe7\xae\xe3\xfe\xa7Y\n\xf5f\xf7\xf6\xe2Q\xc8\xfc\x9a\xdd\xd4ƪ\xf23\x18U\xeb\f\xde\xc3FHa\x85\x92\x17%X\x9es˯/\x18\xe3R*\xcb\xf1\xb3\xc1?\x19˔\xb4Z\x15\x05\xe8\xc5\x03\xc8\xe5c\xbd\x86u-\x8a\x1c4\x01\x0f]\xef\xfe\xb4|\xfb\x1f\xcb\x7f\xbf`L\xf2\x12\xae\x99ɶ\x90\xd7\x05\x98\xe5\x0e\n\xd0j)ԅ\xa9 C\xa0\x0fZ\xd5\xd55k\x7fp\x8d|\x87\x0e\xd9;ߞ>\x15\xc2\xd8\xff\xed}\xfe(\x8c\xa5\x9f\xaa\xa2ּ\xe8\xf4G_\x8d\x90\x0fu\xc1u\xfb\xfd\x821\x93\xa9\n\xae\xd9O\xd8U\xc53\xc8/\x18\xf3\xf8S\xd7\v\xc6\xf3\x9c(\u008b[-\xa4\x05}\xa3\x8a\xba\f\x94X\xb0\x1cL\xa6Eei\xc4w\x96\xdb\xda0\xb5av\v\xdd~\xb0\xfcb\x94\xbc\xe5v{͖\x86\xea-\xab-7\xe1WG\"\a\xc0\x7f\xb2{\xc4\xcdX-\xe4\xc3Xo\xef؍V\x92\xc1\xd7J\x83A\x94YN\f\x94\x0f\xeci\v\x92Y\xc5t-\t\x95\xff\xe2\xd9c]\x8d RA\xb6\x1c\xe0\xe91\xe9\x7f\x9c\xc2\xe5~\v\xac\xe0\xc62+J`\xdcwȞ\xb8!\x1c6J3\xbb\x15f\x9a&\b\xa4\x87\xadC\xe7\xe3\xf0\xb3C(\xe7\x16<:\x1dPAx\x97\x99\x06\x92\xdb{Q\x82\xb1\xbc\xec\xc3|\xf7\x00\t\xc0\x88D\x15\xaf\r\tG\xdb\xfa\xb6\xfb\xc9\x01X+U\x00\x97\x17m\xa5\xdd['{\xd9\x16J~\xed+\xab\n\xe4\xbb\xdb\u0557\x7f\xbd\xeb}f}\x8a\xfe\xff\xa2\xf9\xce\x1an0a\x18g_h\x960\xed\xa7-\xb3[n\x99\x06\x14\x03\x90\x16kT\x1a\x16\x81\xd49S\xba\x03\xaa\x02-T.\xb2\xc0\"jl\xb6\xaa.r\xb6\x06\xe4ֲ\xa9]iU\x81\xb6\"\xccCW:\xea\xa5\xf3\xf5\x18\xfaXpĮ\x95\x13S0$\x99~\xb6A\xee\x89\xe4&\x8f0\xedx\x88\x83\xf8\x99K\xa6ֿ@f\x97\x03\xd0w\xa0\x11L\x18E\xa6\xe4\x0e4R$S\x0fR\xfc_\x03\xdb\xe0\x94\xb0$\xa9\x16\x8ce4\x9f%/؎\x175\\1.\xf3\x01\xe4\x92\xef\x99\x06\xec\x93ղ\x03\x8f\x1a\x98!\x1e?*\rLȍ\xbaf[k+s\xfd\xe6̓\xb0A\xe9f\xaa,k)\xec\xfe\r\xe9O\xb1\xae\xad\xd2\xe6M\x0e;(\xde\x18\xf1\xb0\xe0:\xdb\n\v\x99\xad5\xbc\xe1\x95X\xd0@$)\xdee\x99\xffK\xe0\xb7\xe9u{03]!\x959\x83=\xa8K\x9dt9Pn\x88-\x17\xf0\x13\x92\xee\xf3\x87\xbb\xfb\xae\xe4\t\xe3\x99\xd2\x11\xc0\x18\x7f\x90\x9aBn\xc0낍V%\xc1\x04\x99WJHK\x7fd\x85\x00i\x99\xa9ץ\xb0(\x06\xbf\xd6`,\xb2n\b\xf6\x86\f\x13\nm]\xe1\xdc͇\x15V\x92\xdd\xf0\x12\x8a\x1bn\xe0\x95y\x85\\1\vdB\x12\xb7\xba\xe6vXّ\xb7\xf3C\xb0\x99\x11\xd6\x06]qWA֛j\xd8NlD\xe6&\x14\xaa\xe4F\x95\fԲ+㳟~!\xdd7\xfc:\xc0\xc3)\xc8\xd0+\x184JvK\xccom#\x8a\x9c\x83ƔfR\xd9\x03\x98\x87\xaa\xb5C\t\x0fe\x02\x93\x03ag\a*5Œ\x8e\x00im\xebP\xbe\xa2\xac&\xbc\x1fE\xb5*K\xc8\x05\xb7P\xecOB\xbf\x0fb\x8c̊\xfaak\xa7\xe7ŦG\xf4\xbc\x06&:\xedi2\xfe9\xd48\xb4\xc6\x7f&\xcbNF\xd4Ѥ\v\xac\x96-\x0f\a\xfdHx:$\rc\xab\r\xb3\x1au\xae\xc7\xeeI\x14\x05\xcedĸ\x82\xbc\x87Z\xbc;\xb1a\u0086Ѭ9a \xd9\xd2yQ\xcb\xd6gh\xec?\"8\xc0\x8eԾ\xeb\x1f=\x15n\x99\x84\xaf\xb6\xad\x85Î\x8c`\xc3\v3\x18\x82WH\xb3\x86q\xc5ֵ=\r\x03(+\xbb\xbfrm7\xaa(\xd4\x133\xa4l\xd1\bn\xc4C\xad\xddd\xff}\x0e\x1b^\x17\xf6\xda\xe1\xfc\x87\x98\xb4\x8eO3\ve\x85&\xf3\x149\xbd\xf7mq\xc08[\xf2&\xc6\bnr\xf0C\x94w?F\x80(\xe7\xc5VZ\xedD\xee\xcd\xf9\x81\xbabGU\x16\x96̈;\xc9+\xb3U\x16%B\xd5v\xacVʨ\xb0\xdcܭ\x06\xd0:\x93\x10\xd1%W\x98\xe4\xd4*\xf6ą%\x9d{s\xb7b_0\x86\x80К\xb9\xc9\xc6l\xad%ڹH\x7f\x9f\x81\xe7\xfb{\xf5\xb3\x01\x96\xd7d\xa2\x83{{\xc5ְA{\xa6\x01a\xe0O\xa05\xeawCH\xa8\xfa\xc0ej\xd8\xe3X\x82\xb2\xe1-\xbe0\xec\xed\x9fX)dmG\xa5\xee\xa8b#\xeaq\xcbK\xb5\x03\xfd\x1c\xe2\xbe\xe7\x96\xff\x88@\x064E\xe0\x8c\xa0{\x81!\xfa\xae\xf7\xf4\xe3:\xa2\x89]Ym:P\x85a\x97\x97\xa8\r.]\xc8yy\xe5 Ԣ\xb0\v!\xbb\xfd\x04Մ=\x9dF\x10G_\xc7ts\xaf~0N\xe4\x9fE\x9f\b\xcc\x11;P\xa9\x9c\xed\xa8\x1eۈ\x02\x98\xd9\x1b\ve\xd0Z\xad\xe7\xdf\tg\x86\x85|\x85\xa2\xf0`\f\xd2\xdb\x0fj\x9c \xb2.\n\xbe.\xe0\x9a\x94\xfc\x11\x9a\x8d\xeb\x9b1\xa2}\x06cEvN\x929\x88#\x04\xd3\xfe\x87\x1ee(t\xe0\x8f\xc0x\x04\xbc\xa7'\xc6)E\xd1!z\x9fZQ\xdc*\r\x19\xfa\xb0\xd7\xde7\x16P\x90?.\x15+\x94|\x00\xed\xb0hl\x15\xeaJ@\x01\xcd\x19\xba\x9d\x1a-\x8c\x90lS\xa3G\xbad\xa8%\xa22\"\xa4\xb1\xc0#\xc2|\x06\xde\xc1\u05ec\xa8s\xc8o\x8a\xdaX\xd0w\x99\xaa \x0f\x8bL\xa3\x9a9\x95\x87\x1f\x8eB\xf6\xf1K!2@>d\xae҂\x16yb\xa2݆2\xfb\nܚ\x13\xb2\xda\x0f\xa1\x8dQ&u\x8b\x01\x8b\r/\xffxyE\x12\xd0\xef\xbdߏa\\CC\xa6Y\xba\x99,\xfex\va\xa1\x8cPwRG\xcd\xe0;ך\xef\x8fp\xbdYL{\x01\xbe\xc7`\x0f8/C\xb5o\xc4\xfba\xff\xff\x8c\xdc?/\xbf\r-:s!\x91υ0\xb6\xc7f\xe3V\xb1\x90\xacc!\xa4'\x90t0QMNq\xf5\uf118g\x9d;\xb1\xc9\xd2Ȧ\x9f\x00\xffP\x94\xdc*\xf5\x98B\xbd\xff\xc1z\xed\x12\x16\xcbhc\x84\xada\xcbwBi3\\&\x85\xaf\x90\xd56\xaaY\xb8e\xb9\xd8l@#,Z\xe6ov\x05\x8e\x11\xebx\xf8\xc2:*+Za0\xae\x96\xe9\xc8R\xa2Fl(\x14\xa0F\xa1:\a\aC\vr r\xb1\x13y\xcd\v\xf2%\xb8\xcc\xdc\xf8x\x83_L\xabM\b\xc4\x01\xfeQ\xa9v\xc594a\x90\xc8\xc4ު\x97\x92\x80>~\x89\xb1\xd1a\xd58%\xc2R\xc2Ѿ\x91\x99\xba.\xc0\xf8\xeerr\x93[\x9dt\xd52˭1\x14|\r\x053P@f\x95\x8eS(E\x0e\\IU\xba\x11\xe2\x8eh\xd9~\xb4\xd5\x0ef\x02,\xa3\x10w+\xb2\xads_Q\xd0\b\x16\xcb\x15\x18Z\x15\xe1UUDLW[&\x85\xc3w6\xa57ڒ\xa0A\x86pc\xba\xa4-\x89\xfa\xb9-\xa3do\xe7f\x9f\xea\xe3\xeb\xfc\xa3\xf8\xfe3\x11=X\x9d\x13\x85}B\x930\xda/H\x9e\x0fQ\xd2#\xc5\x05\x98eguN\xd8\xf05\x85\xa1=\xff\xf1`+\xe5\x80(\xbf-ޝ6af\xb0nrN\xbd,\xe3\x9an\xfeA\xf8F&\xeb\xce[\xacY<\xfb\xd8myE\xbb\x02\x9e!\xf9\x15ۈ\xc2\x029US\x88\xb2\x19\x9c;'\x81R-0\xa3Mb\x9bm?4{G\t-\x06\xb4\x1a\x02p\x0ez\x88r\x88\a\t Y\xe3ZЦ\xa9\xd0P\xd2f,E\x92\xdd/\xe4\n\xbe\xfb\xe9}<\xf6\xec\x96DI=\x18T¤u\xe5\xdd\xc01\xea\xe2\xeaC\x95\xf0\v\xf9kM \xe86\xe1\xaf\x18g\x8f\xb0w.\x16\x97\f\xf9\xc6C\xe5D\x144PF\x00i\x8aG\xd8\x13\xa8\xf1-\xfe\xf12GZ\\y\x84\x91]\xbfX\xe9\xd1\x15\xf1\xf3{)\x8en\xf8\x81\b\x932\x9b\xda\xd2\x10\xd5O\x9f\x91\r\xf6x\x99\xa1\x97B\t|9q\xd8\xc9\xe2\xd4\xed\xab\x9f\x14\xf3\b\xfb\xdf\x19\xc7k\x9ce[A\x9bN\x9cVo\xd4f\x16\xc3]\xf9\xc2\v\x917\x9d\xb9y\xb5\x92W\xec'e\xf1\x9f\x0f_\x85\xc1\x8ee\xce\xde+0?)K_^\x94\xcan\x10\xafAc\xd7\x13MP\xe9,\t\x12\xb1\x9b<\xe2l)\nj\xc3\x0fa\xd8JbH\xe6H4\xa3;\xca\x15r]\xba\xce\xca\xda\xd0V\xabTr\xe1\x96\xc5\xc6z\xf3\xa7N\xd1Ə\x86\x1es\x0f\xf7DȻ\x96\xcav\x96qf:ѕ\xca\x7fg\xd8Fhc\xbbh\x98#\x89U\xa3\xa0N\b=\xe5\a\xadO\x8ems\x06\x1e\x99\x92F\xe4И~/\x02J2\xce6\\\x14\xb5\x9e\xa1Ug\x93|n\x10\xe6\xb5\xc9\xf9#\xabtD\x16D\xa2\xc4u\xf6\x19^\xf0\xb4Ư\xf4c\n\x86\vZ\xe3Unu\xf0d\xbfiz\r+=\x11wn\xfam\x93-9\xed\xbe?#\xe9\xf6\xacG\xa3\xbeYZ\xediɴ\xa9+\x94\t\x89\xb3\xe9\xe9\xb2)lu%=I69BNM\x88\x9d\xbb\x02\xf1\xa2ɯ/\x93\xf2\x9aL\xb3\xb4\xf4ֹ\x14{\x95T\xd6WN`}\xbd\xb4\xd5\x19ɪ\xe7?\xf5\x92\xbe\x96~rveڲ\xcc\xf1\x84Ӥ4Ӥ\xa5\x9b\x94\x01\x9f4Ԥ\xf4ѹI\xa3I\x9cL\x9f\xae\xaf\x9a\x16\xfa\xaaɠ\xaf\x9f\x02:)m\x93\x15\xe6&y\x8e_r\x18ʴ\x03P|\v\xe1|.\x99\x94\xee\xb9\xe6ϊ;?\r`\xa1\xb0\x047\xf5\x15〲.\xac\xa8\x8a\xf6>\xb6X\xc0\xb9\x85}sY\xd1/\x8a\x8e\xc8\xfb\x9b\xba>}n$~9\x88j\xb8aOP\x14\x8c\xc7\xe6\xe6\x01\x152w\x0fh\xa6\x16\x80\xb6\x11g\xb9\xbf\x8c\xc9_\x1ez\xe5\xa6\v\xdd\x06@\x16\xb6\x8c-\xf5qy\xfc\xa6\xaf\xa3\x06,U\x8f\x1dx\xe6.ޠo\xbf֠\xf7\x8c\xee\x1dk|\xb3\xf6P\xa9\x9f\xe8\x06\x03Ӡ~\xbc:<\xb6gr\x10\xe0\xb4ꁽ\x93\xce#\x18\xe2DmP\xef\xb4\x01\x1d*U\x8cӢ\xfdD@H\xd5@\x884Mq\xfe眲|\x89\xf0\xee\x1c\x01^\x92\a4\xcf{\xfd\x86\xa7'O=5\x99\x9e\x8c\x92tJ\xf2%½9\x01\xdf,\x7f5\xfd\x14\xe4\xfc\x8d\xe7\x17>\xf5\xf8R\xa7\x1dgP/\xf5t\xe3|ڽ\xd2i\xc6W?\xc5\xf8\x9a\xa7\x17g\x9dZLNϚ\x95q0'\xb5\xea\x19\xc7\xed\xd2r\t\xa6O!&\x9e>L\xcc4H\x1b\xfc\x89\xc3N<]8\xffTa\"\x7f\xe7L\xe9W>=\xf8ʧ\x06\xbf\xc5i\xc1\x04\tL\xa82\xffT\u0cf7\xa4\x94\xceAOn\xfb͑\xdaIyM\x8d\xe5\xfa\x88\r\xf6\xb5\xc2m\xb2X\xab\x17\x03\x90Y\xf2\x17\xf9ӣ\rǶ\xc1Q2;\x1eQo_\xb2u\xd7\xfa\x0e\xb1\x7f\xcd\xc1m]\x1a\xa88\x1a\x00\n\xdc(5+\xea*|\xe0\xd9v\xd0Ö\x1b\xb6Q\xba\xe4\x96]6\x9b\xc5o\\\a\xf8\xf7咱\x1fT\x93\xabӽ/͈\xb2*\xf6\x18\x89\xb1\xcbn\x83\xe7IIT:CϷ\xaa\x10Y\xc4\xe7\x1c\xbdW\xcf58\xb8l\x88n\xfe\xcb:\xd9\"\xb1\xc0\a\x9b\x8bp\xebb\xffJfw\x9f\xfb\x89k%\xbc\x12\xffMO*\x9da\xd5\xed\xdd\xed\x8a`\x051\xa2\xb7\x9a\x9a\x04ņ\xe5k@\x97\xa1\x1d\xfb1}\xb2\xda\xf4\xa0\xf6s\x84\xbb\x8fU@\xee^&\tn\x8bW͙B\xadu\xbbr\xb8\x1c\xeb\t\xe5\x8b\xcb=S\xfe\xe9\t\xa1\xf3Eŵݻd\xa2\xab\x1e\x1e\xc1\xaeO\xad\x9a\x1d\xb5V\x87/\xaftK\x8f\xec\xe1\xd1\x15\xda\xc9\xdeW\xfd\xe4\x81!=\x9f\x83\xd3\xf1SՓ\xe7\xa9_\x00\xa7\xe3.Ԃ\xa8\x18\xf9)\x9a\x01y\xf6\x15K\xe3o\xe8\xffQ\xed\xe0}t\xe5\xb2\xff\xfaʠ\xc9Hjb\x80J\x97\xccG(\xd8\xe6#\xd2\x1d\xdf\xcfS{\xf1\\À\x8a\xbf#\xfc9\x8b\x93w}P\xe3\x0f\x92\xd0\r\xea\xa1ӘWEO=\xed\xd9\xed\x17\x8a[\x1bUꧾ\x8f[\xc3\xf2dH0\x88\xc0\x12\xf2\xe8\x1b-\xe7\"\xa3U\x9a?\xc0G\xe5\xde\xd6I\x11\x93~\x8b\xde\xcbK\xdes\v\xf9\xda~\x12\xc6\x14\xbd\x1f\xdb\x10`{>\xe3\xe0\xa2\x7f\xc4\xf6ħ\f\xac-\x9e##\xf7\xf7\x1f\xddH\xe9I\x93\xf7\xfeu\x12\xd4\xc7\x06\x90\x05\x81\x02\x0e\xda\x1a\xff\xbbUOt\x01~|\x8d9< \xd2y\xc3\f\xe8\xa0\b\xa5\xf0\x9e4̺*\x14\xcfA\xdf\xd0#*\t#\xfe\xb9\xd7`\xe0\x0e\xf4\x9fb\xf1v32\x9e\xd0\xf3\vfɠGW\x14P\xfc \n0\x0e\xf1D\xd3p{ز\xb1\x14u\xb9v\x9e\xea\x06\x7fl:9b\x99\xddPi\x83\xa1\x02\x8d~\xa2ۊ\xa8M\x90\xfc\xe3\xc4`\r\x1f\x85\xb4\xf0\x00\xe31\xf4\x84M\xd8\xf5^b\t\xb3'E\x11~\x19o\xd9q\xa6;\xf3\x98\xbc߸\xba\x8b\xc1\xe2ƨL\x90\xff\xfd$\xac\xbf\xf8\xf0\xe5n\xdb>\x16J\x1d\xa1cm\xe0ӓ\x04\xfd9\xe8j\xb3\x92\xb1\x17N\xa6\xf5\xc4\xcf\aТ/\x9bX\x85}\x8f\xc0\x18\x00`*\xec\b\x19\xf7fN؈\x12\xa6y\x06쐞\x13\x93-n\x13\xc6]\x9b\xc5\xf8\xabE\x8b\xe6u\xa5\x8b\x04r\xbb\x97\x82\xfa\x80\xc7\x1f\x7fsO\ne\xbc\xb2\xb5\x0ez\xa8\xd6t\x1f9\x02\x01w]\xf7iϿ\xb5\xaf\x82\x9d\xc2\xe0\xf6Y\xaev\xa5~\xf2\xe1\xd0\x118\xcd\x03n\xd1נ\\\xec\xe9\x1e\xf6\\ \xfc\xd3x<:c\x10\xe7;\xf7\xca\xd7\x04\x11>\xb65\xc7\x06\xdc\f\x03\x87\xec\xdf\r{Ց\xd0\xf5\xf4\x13c\xb8\xc5:\xcdyP/G\xd40\\k\x7f\x17c\xc2\xf8\xa1\xc1\x05\xfb\t\x0ec\xdb\x05\xfb q\x10\x87\x04p'\x03!\xa7M\bҎs\x86\xb8kZѱ\xcc\x11\r9-\xb6_\x060\x069\xdf\xf4=\xc0\xfe\xf2\u05cb\xbf\x05\x00\x00\xff\xffq\xe5\x82\xc1hz\x00\x00"), diff --git a/internal/credentials/local.go b/internal/credentials/local.go new file mode 100644 index 000000000..00ccc3174 --- /dev/null +++ b/internal/credentials/local.go @@ -0,0 +1,7 @@ +package credentials + +import "os" + +func DefaultStoreDirectory() string { + return os.TempDir() + "/credentials" +} diff --git a/pkg/apis/velero/v1/pod_volume_backup_types.go b/pkg/apis/velero/v1/pod_volume_backup_types.go index b3070e3dd..ced436a82 100644 --- a/pkg/apis/velero/v1/pod_volume_backup_types.go +++ b/pkg/apis/velero/v1/pod_volume_backup_types.go @@ -57,6 +57,10 @@ type PodVolumeBackupSpec struct { // +optional // +nullable UploaderSettings map[string]string `json:"uploaderSettings,omitempty"` + + // Cancel indicates request to cancel the ongoing PodVolumeBackup. It can be set + // when the PodVolumeBackup is in InProgress phase + Cancel bool `json:"cancel,omitempty"` } // PodVolumeBackupPhase represents the lifecycle phase of a PodVolumeBackup. diff --git a/pkg/cmd/cli/datamover/backup.go b/pkg/cmd/cli/datamover/backup.go index 94fe613d1..2b647bb24 100644 --- a/pkg/cmd/cli/datamover/backup.go +++ b/pkg/cmd/cli/datamover/backup.go @@ -39,6 +39,7 @@ import ( "github.com/vmware-tanzu/velero/pkg/repository" "github.com/vmware-tanzu/velero/pkg/uploader" "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/kube" "github.com/vmware-tanzu/velero/pkg/util/logging" ctrl "sigs.k8s.io/controller-runtime" @@ -78,7 +79,7 @@ func NewBackupCommand(f client.Factory) *cobra.Command { f.SetBasename(fmt.Sprintf("%s-%s", c.Parent().Name(), c.Name())) s, err := newdataMoverBackup(logger, f, config) if err != nil { - exitWithMessage(logger, false, "Failed to create data mover backup, %v", err) + kube.ExitPodWithMessage(logger, false, "Failed to create data mover backup, %v", err) } s.run() @@ -100,12 +101,6 @@ func NewBackupCommand(f client.Factory) *cobra.Command { return command } -const ( - // defaultCredentialsDirectory is the path on disk where credential - // files will be written to - defaultCredentialsDirectory = "/tmp/credentials" -) - type dataMoverBackup struct { logger logrus.FieldLogger ctx context.Context @@ -215,7 +210,7 @@ func newdataMoverBackup(logger logrus.FieldLogger, factory client.Factory, confi return s, nil } -var funcExitWithMessage = exitWithMessage +var funcExitWithMessage = kube.ExitPodWithMessage var funcCreateDataPathService = (*dataMoverBackup).createDataPathService func (s *dataMoverBackup) run() { @@ -277,7 +272,7 @@ func (s *dataMoverBackup) createDataPathService() (dataPathService, error) { credentialFileStore, err := funcNewCredentialFileStore( s.client, s.namespace, - defaultCredentialsDirectory, + credentials.DefaultStoreDirectory(), filesystem.NewFileSystem(), ) if err != nil { diff --git a/pkg/cmd/cli/datamover/data_mover.go b/pkg/cmd/cli/datamover/data_mover.go index 6786f4e7c..265d57bd8 100644 --- a/pkg/cmd/cli/datamover/data_mover.go +++ b/pkg/cmd/cli/datamover/data_mover.go @@ -15,10 +15,7 @@ package datamover import ( "context" - "fmt" - "os" - "github.com/sirupsen/logrus" "github.com/spf13/cobra" "github.com/vmware-tanzu/velero/pkg/client" @@ -45,30 +42,3 @@ type dataPathService interface { RunCancelableDataPath(context.Context) (string, error) Shutdown() } - -var funcExit = os.Exit -var funcCreateFile = os.Create - -func exitWithMessage(logger logrus.FieldLogger, succeed bool, message string, a ...any) { - exitCode := 0 - if !succeed { - exitCode = 1 - } - - toWrite := fmt.Sprintf(message, a...) - - podFile, err := funcCreateFile("/dev/termination-log") - if err != nil { - logger.WithError(err).Error("Failed to create termination log file") - exitCode = 1 - } else { - if _, err := podFile.WriteString(toWrite); err != nil { - logger.WithError(err).Error("Failed to write error to termination log file") - exitCode = 1 - } - - podFile.Close() - } - - funcExit(exitCode) -} diff --git a/pkg/cmd/cli/datamover/data_mover_test.go b/pkg/cmd/cli/datamover/data_mover_test.go deleted file mode 100644 index 206ba36ee..000000000 --- a/pkg/cmd/cli/datamover/data_mover_test.go +++ /dev/null @@ -1,131 +0,0 @@ -/* -Copyright The Velero Contributors. -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - http://www.apache.org/licenses/LICENSE-2.0 -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package datamover - -import ( - "errors" - "fmt" - "io" - "os" - "path/filepath" - "testing" - - "github.com/google/uuid" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - - velerotest "github.com/vmware-tanzu/velero/pkg/test" -) - -type exitWithMessageMock struct { - createErr error - writeFail bool - filePath string - exitCode int -} - -func (em *exitWithMessageMock) Exit(code int) { - em.exitCode = code -} - -func (em *exitWithMessageMock) CreateFile(name string) (*os.File, error) { - if em.createErr != nil { - return nil, em.createErr - } - - if em.writeFail { - return os.OpenFile(em.filePath, os.O_CREATE|os.O_RDONLY, 0500) - } else { - return os.Create(em.filePath) - } -} - -func TestExitWithMessage(t *testing.T) { - tests := []struct { - name string - message string - succeed bool - args []any - createErr error - writeFail bool - expectedExitCode int - expectedMessage string - }{ - { - name: "create pod file failed", - createErr: errors.New("fake-create-file-error"), - succeed: true, - expectedExitCode: 1, - }, - { - name: "write pod file failed", - writeFail: true, - succeed: true, - expectedExitCode: 1, - }, - { - name: "not succeed", - message: "fake-message-1, arg-1 %s, arg-2 %v, arg-3 %v", - args: []any{ - "arg-1-1", - 10, - false, - }, - expectedExitCode: 1, - expectedMessage: fmt.Sprintf("fake-message-1, arg-1 %s, arg-2 %v, arg-3 %v", "arg-1-1", 10, false), - }, - { - name: "not succeed", - message: "fake-message-2, arg-1 %s, arg-2 %v, arg-3 %v", - args: []any{ - "arg-1-2", - 20, - true, - }, - succeed: true, - expectedMessage: fmt.Sprintf("fake-message-2, arg-1 %s, arg-2 %v, arg-3 %v", "arg-1-2", 20, true), - }, - } - - for _, test := range tests { - t.Run(test.name, func(t *testing.T) { - podFile := filepath.Join(os.TempDir(), uuid.NewString()) - - em := exitWithMessageMock{ - createErr: test.createErr, - writeFail: test.writeFail, - filePath: podFile, - } - - funcExit = em.Exit - funcCreateFile = em.CreateFile - - exitWithMessage(velerotest.NewLogger(), test.succeed, test.message, test.args...) - - assert.Equal(t, test.expectedExitCode, em.exitCode) - - if test.createErr == nil && !test.writeFail { - reader, err := os.Open(podFile) - require.NoError(t, err) - - message, err := io.ReadAll(reader) - require.NoError(t, err) - - reader.Close() - - assert.Equal(t, test.expectedMessage, string(message)) - } - }) - } -} diff --git a/pkg/cmd/cli/datamover/restore.go b/pkg/cmd/cli/datamover/restore.go index 65e4e0f5e..7e11df822 100644 --- a/pkg/cmd/cli/datamover/restore.go +++ b/pkg/cmd/cli/datamover/restore.go @@ -42,6 +42,7 @@ import ( "github.com/vmware-tanzu/velero/pkg/repository" "github.com/vmware-tanzu/velero/pkg/uploader" "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/kube" "github.com/vmware-tanzu/velero/pkg/util/logging" ctlcache "sigs.k8s.io/controller-runtime/pkg/cache" @@ -76,7 +77,7 @@ func NewRestoreCommand(f client.Factory) *cobra.Command { f.SetBasename(fmt.Sprintf("%s-%s", c.Parent().Name(), c.Name())) s, err := newdataMoverRestore(logger, f, config) if err != nil { - exitWithMessage(logger, false, "Failed to create data mover restore, %v", err) + kube.ExitPodWithMessage(logger, false, "Failed to create data mover restore, %v", err) } s.run() @@ -263,7 +264,7 @@ func (s *dataMoverRestore) createDataPathService() (dataPathService, error) { credentialFileStore, err := funcNewCredentialFileStore( s.client, s.namespace, - defaultCredentialsDirectory, + credentials.DefaultStoreDirectory(), filesystem.NewFileSystem(), ) if err != nil { diff --git a/pkg/cmd/cli/nodeagent/server.go b/pkg/cmd/cli/nodeagent/server.go index c7a2576c9..5bfec7c87 100644 --- a/pkg/cmd/cli/nodeagent/server.go +++ b/pkg/cmd/cli/nodeagent/server.go @@ -76,10 +76,6 @@ const ( // the port where prometheus metrics are exposed defaultMetricsAddress = ":8085" - // defaultCredentialsDirectory is the path on disk where credential - // files will be written to - defaultCredentialsDirectory = "/tmp/credentials" - defaultHostPodsPath = "/host_pods" defaultResourceTimeout = 10 * time.Minute @@ -291,7 +287,7 @@ func (s *nodeAgentServer) run() { credentialFileStore, err := credentials.NewNamespacedFileStore( s.mgr.GetClient(), s.namespace, - defaultCredentialsDirectory, + credentials.DefaultStoreDirectory(), filesystem.NewFileSystem(), ) if err != nil { diff --git a/pkg/cmd/cli/podvolume/backup.go b/pkg/cmd/cli/podvolume/backup.go new file mode 100644 index 000000000..537caa3cb --- /dev/null +++ b/pkg/cmd/cli/podvolume/backup.go @@ -0,0 +1,291 @@ +/* +Copyright The Velero Contributors. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "context" + "fmt" + "os" + "strings" + "time" + + "github.com/bombsimon/logrusr/v3" + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + "github.com/spf13/cobra" + v1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/fields" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes" + "k8s.io/klog/v2" + + "github.com/vmware-tanzu/velero/internal/credentials" + "github.com/vmware-tanzu/velero/pkg/buildinfo" + "github.com/vmware-tanzu/velero/pkg/client" + "github.com/vmware-tanzu/velero/pkg/cmd/util/signals" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/podvolume" + "github.com/vmware-tanzu/velero/pkg/repository" + "github.com/vmware-tanzu/velero/pkg/uploader" + "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/kube" + "github.com/vmware-tanzu/velero/pkg/util/logging" + + ctrl "sigs.k8s.io/controller-runtime" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + + ctlcache "sigs.k8s.io/controller-runtime/pkg/cache" + ctlclient "sigs.k8s.io/controller-runtime/pkg/client" +) + +type podVolumeBackupConfig struct { + volumePath string + pvbName string + resourceTimeout time.Duration +} + +func NewBackupCommand(f client.Factory) *cobra.Command { + config := podVolumeBackupConfig{} + + logLevelFlag := logging.LogLevelFlag(logrus.InfoLevel) + formatFlag := logging.NewFormatFlag() + + command := &cobra.Command{ + Use: "backup", + Short: "Run the velero pod volume backup", + Long: "Run the velero pod volume backup", + Hidden: true, + Run: func(c *cobra.Command, args []string) { + logLevel := logLevelFlag.Parse() + logrus.Infof("Setting log-level to %s", strings.ToUpper(logLevel.String())) + + logger := logging.DefaultLogger(logLevel, formatFlag.Parse()) + logger.Infof("Starting Velero pod volume backup %s (%s)", buildinfo.Version, buildinfo.FormattedGitSHA()) + + f.SetBasename(fmt.Sprintf("%s-%s", c.Parent().Name(), c.Name())) + s, err := newPodVolumeBackup(logger, f, config) + if err != nil { + kube.ExitPodWithMessage(logger, false, "Failed to create pod volume backup, %v", err) + } + + s.run() + }, + } + + command.Flags().Var(logLevelFlag, "log-level", fmt.Sprintf("The level at which to log. Valid values are %s.", strings.Join(logLevelFlag.AllowedValues(), ", "))) + command.Flags().Var(formatFlag, "log-format", fmt.Sprintf("The format for log output. Valid values are %s.", strings.Join(formatFlag.AllowedValues(), ", "))) + command.Flags().StringVar(&config.volumePath, "volume-path", config.volumePath, "The full path of the volume to be backed up") + command.Flags().StringVar(&config.pvbName, "pod-volume-backup", config.pvbName, "The PVB name") + command.Flags().DurationVar(&config.resourceTimeout, "resource-timeout", config.resourceTimeout, "How long to wait for resource processes which are not covered by other specific timeout parameters.") + + _ = command.MarkFlagRequired("volume-path") + _ = command.MarkFlagRequired("pod-volume-backup") + _ = command.MarkFlagRequired("resource-timeout") + + return command +} + +type podVolumeBackup struct { + logger logrus.FieldLogger + ctx context.Context + cancelFunc context.CancelFunc + client ctlclient.Client + cache ctlcache.Cache + namespace string + nodeName string + config podVolumeBackupConfig + kubeClient kubernetes.Interface + dataPathMgr *datapath.Manager +} + +func newPodVolumeBackup(logger logrus.FieldLogger, factory client.Factory, config podVolumeBackupConfig) (*podVolumeBackup, error) { + ctx, cancelFunc := context.WithCancel(context.Background()) + + clientConfig, err := factory.ClientConfig() + if err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to create client config") + } + + ctrl.SetLogger(logrusr.New(logger)) + klog.SetLogger(logrusr.New(logger)) // klog.Logger is used by k8s.io/client-go + + scheme := runtime.NewScheme() + if err := velerov1api.AddToScheme(scheme); err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to add velero v1 scheme") + } + + if err := v1.AddToScheme(scheme); err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to add core v1 scheme") + } + + nodeName := os.Getenv("NODE_NAME") + + // use a field selector to filter to only pods scheduled on this node. + cacheOption := ctlcache.Options{ + Scheme: scheme, + ByObject: map[ctlclient.Object]ctlcache.ByObject{ + &v1.Pod{}: { + Field: fields.Set{"spec.nodeName": nodeName}.AsSelector(), + }, + &velerov1api.PodVolumeBackup{}: { + Field: fields.Set{"metadata.namespace": factory.Namespace()}.AsSelector(), + }, + }, + } + + cli, err := ctlclient.New(clientConfig, ctlclient.Options{ + Scheme: scheme, + }) + if err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to create client") + } + + var cache ctlcache.Cache + retry := 10 + for { + cache, err = ctlcache.New(clientConfig, cacheOption) + if err == nil { + break + } + + retry-- + if retry == 0 { + break + } + + logger.WithError(err).Warn("Failed to create client cache, need retry") + + time.Sleep(time.Second) + } + + if err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to create client cache") + } + + s := &podVolumeBackup{ + logger: logger, + ctx: ctx, + cancelFunc: cancelFunc, + client: cli, + cache: cache, + config: config, + namespace: factory.Namespace(), + nodeName: nodeName, + } + + s.kubeClient, err = factory.KubeClient() + if err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to create kube client") + } + + s.dataPathMgr = datapath.NewManager(1) + + return s, nil +} + +var funcExitWithMessage = kube.ExitPodWithMessage +var funcCreateDataPathService = (*podVolumeBackup).createDataPathService + +func (s *podVolumeBackup) run() { + signals.CancelOnShutdown(s.cancelFunc, s.logger) + go func() { + if err := s.cache.Start(s.ctx); err != nil { + s.logger.WithError(err).Warn("error starting cache") + } + }() + + s.runDataPath() +} + +func (s *podVolumeBackup) runDataPath() { + s.logger.Infof("Starting micro service in node %s for PVB %s", s.nodeName, s.config.pvbName) + + dpService, err := funcCreateDataPathService(s) + if err != nil { + s.cancelFunc() + funcExitWithMessage(s.logger, false, "Failed to create data path service for PVB %s: %v", s.config.pvbName, err) + return + } + + s.logger.Infof("Starting data path service %s", s.config.pvbName) + + err = dpService.Init() + if err != nil { + dpService.Shutdown() + s.cancelFunc() + funcExitWithMessage(s.logger, false, "Failed to init data path service for PVB %s: %v", s.config.pvbName, err) + return + } + + s.logger.Infof("Running data path service %s", s.config.pvbName) + + result, err := dpService.RunCancelableDataPath(s.ctx) + if err != nil { + dpService.Shutdown() + s.cancelFunc() + funcExitWithMessage(s.logger, false, "Failed to run data path service for PVB %s: %v", s.config.pvbName, err) + return + } + + s.logger.WithField("PVB", s.config.pvbName).Info("Data path service completed") + + dpService.Shutdown() + + s.logger.WithField("PVB", s.config.pvbName).Info("Data path service is shut down") + + s.cancelFunc() + + funcExitWithMessage(s.logger, true, result) +} + +var funcNewCredentialFileStore = credentials.NewNamespacedFileStore +var funcNewCredentialSecretStore = credentials.NewNamespacedSecretStore + +func (s *podVolumeBackup) createDataPathService() (dataPathService, error) { + credentialFileStore, err := funcNewCredentialFileStore( + s.client, + s.namespace, + credentials.DefaultStoreDirectory(), + filesystem.NewFileSystem(), + ) + if err != nil { + return nil, errors.Wrapf(err, "error to create credential file store") + } + + credSecretStore, err := funcNewCredentialSecretStore(s.client, s.namespace) + if err != nil { + return nil, errors.Wrapf(err, "error to create credential secret store") + } + + credGetter := &credentials.CredentialGetter{FromFile: credentialFileStore, FromSecret: credSecretStore} + + pvbInformer, err := s.cache.GetInformer(s.ctx, &velerov1api.PodVolumeBackup{}) + if err != nil { + return nil, errors.Wrap(err, "error to get controller-runtime informer from manager") + } + + repoEnsurer := repository.NewEnsurer(s.client, s.logger, s.config.resourceTimeout) + + return podvolume.NewBackupMicroService(s.ctx, s.client, s.kubeClient, s.config.pvbName, s.namespace, s.nodeName, datapath.AccessPoint{ + ByPath: s.config.volumePath, + VolMode: uploader.PersistentVolumeFilesystem, + }, s.dataPathMgr, repoEnsurer, credGetter, pvbInformer, s.logger), nil +} diff --git a/pkg/cmd/cli/podvolume/backup_test.go b/pkg/cmd/cli/podvolume/backup_test.go new file mode 100644 index 000000000..1c59af751 --- /dev/null +++ b/pkg/cmd/cli/podvolume/backup_test.go @@ -0,0 +1,216 @@ +/* +Copyright The Velero Contributors. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "context" + "errors" + "fmt" + "testing" + + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + ctlclient "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/vmware-tanzu/velero/internal/credentials" + cacheMock "github.com/vmware-tanzu/velero/pkg/cmd/cli/datamover/mocks" + velerotest "github.com/vmware-tanzu/velero/pkg/test" + "github.com/vmware-tanzu/velero/pkg/util/filesystem" +) + +func fakeCreateDataPathServiceWithErr(_ *podVolumeBackup) (dataPathService, error) { + return nil, errors.New("fake-create-data-path-error") +} + +var frHelper *fakeRunHelper + +func fakeCreateDataPathService(_ *podVolumeBackup) (dataPathService, error) { + return frHelper, nil +} + +type fakeRunHelper struct { + initErr error + runCancelableDataPathErr error + runCancelableDataPathResult string + exitMessage string + succeed bool +} + +func (fr *fakeRunHelper) Init() error { + return fr.initErr +} + +func (fr *fakeRunHelper) RunCancelableDataPath(_ context.Context) (string, error) { + if fr.runCancelableDataPathErr != nil { + return "", fr.runCancelableDataPathErr + } else { + return fr.runCancelableDataPathResult, nil + } +} + +func (fr *fakeRunHelper) Shutdown() { + +} + +func (fr *fakeRunHelper) ExitWithMessage(logger logrus.FieldLogger, succeed bool, message string, a ...any) { + fr.succeed = succeed + fr.exitMessage = fmt.Sprintf(message, a...) +} + +func TestRunDataPath(t *testing.T) { + tests := []struct { + name string + pvbName string + createDataPathFail bool + initDataPathErr error + runCancelableDataPathErr error + runCancelableDataPathResult string + expectedMessage string + expectedSucceed bool + }{ + { + name: "create data path failed", + pvbName: "fake-name", + createDataPathFail: true, + expectedMessage: "Failed to create data path service for PVB fake-name: fake-create-data-path-error", + }, + { + name: "init data path failed", + pvbName: "fake-name", + initDataPathErr: errors.New("fake-init-data-path-error"), + expectedMessage: "Failed to init data path service for PVB fake-name: fake-init-data-path-error", + }, + { + name: "run data path failed", + pvbName: "fake-name", + runCancelableDataPathErr: errors.New("fake-run-data-path-error"), + expectedMessage: "Failed to run data path service for PVB fake-name: fake-run-data-path-error", + }, + { + name: "succeed", + pvbName: "fake-name", + runCancelableDataPathResult: "fake-run-data-path-result", + expectedMessage: "fake-run-data-path-result", + expectedSucceed: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + frHelper = &fakeRunHelper{ + initErr: test.initDataPathErr, + runCancelableDataPathErr: test.runCancelableDataPathErr, + runCancelableDataPathResult: test.runCancelableDataPathResult, + } + + if test.createDataPathFail { + funcCreateDataPathService = fakeCreateDataPathServiceWithErr + } else { + funcCreateDataPathService = fakeCreateDataPathService + } + + funcExitWithMessage = frHelper.ExitWithMessage + + s := &podVolumeBackup{ + logger: velerotest.NewLogger(), + cancelFunc: func() {}, + config: podVolumeBackupConfig{ + pvbName: test.pvbName, + }, + } + + s.runDataPath() + + assert.Equal(t, test.expectedMessage, frHelper.exitMessage) + assert.Equal(t, test.expectedSucceed, frHelper.succeed) + }) + } +} + +type fakeCreateDataPathServiceHelper struct { + fileStoreErr error + secretStoreErr error +} + +func (fc *fakeCreateDataPathServiceHelper) NewNamespacedFileStore(_ ctlclient.Client, _ string, _ string, _ filesystem.Interface) (credentials.FileStore, error) { + return nil, fc.fileStoreErr +} + +func (fc *fakeCreateDataPathServiceHelper) NewNamespacedSecretStore(_ ctlclient.Client, _ string) (credentials.SecretStore, error) { + return nil, fc.secretStoreErr +} + +func TestCreateDataPathService(t *testing.T) { + tests := []struct { + name string + fileStoreErr error + secretStoreErr error + mockGetInformer bool + getInformerErr error + expectedError string + }{ + { + name: "create credential file store error", + fileStoreErr: errors.New("fake-file-store-error"), + expectedError: "error to create credential file store: fake-file-store-error", + }, + { + name: "create credential secret store", + secretStoreErr: errors.New("fake-secret-store-error"), + expectedError: "error to create credential secret store: fake-secret-store-error", + }, + { + name: "get informer error", + mockGetInformer: true, + getInformerErr: errors.New("fake-get-informer-error"), + expectedError: "error to get controller-runtime informer from manager: fake-get-informer-error", + }, + { + name: "succeed", + mockGetInformer: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fcHelper := &fakeCreateDataPathServiceHelper{ + fileStoreErr: test.fileStoreErr, + secretStoreErr: test.secretStoreErr, + } + + funcNewCredentialFileStore = fcHelper.NewNamespacedFileStore + funcNewCredentialSecretStore = fcHelper.NewNamespacedSecretStore + + cache := cacheMock.NewCache(t) + if test.mockGetInformer { + cache.On("GetInformer", mock.Anything, mock.Anything).Return(nil, test.getInformerErr) + } + + funcExitWithMessage = frHelper.ExitWithMessage + + s := &podVolumeBackup{ + cache: cache, + } + + _, err := s.createDataPathService() + + if test.expectedError != "" { + assert.EqualError(t, err, test.expectedError) + } else { + assert.NoError(t, err) + } + }) + } +} diff --git a/pkg/cmd/cli/podvolume/podvolume.go b/pkg/cmd/cli/podvolume/podvolume.go new file mode 100644 index 000000000..e7f347533 --- /dev/null +++ b/pkg/cmd/cli/podvolume/podvolume.go @@ -0,0 +1,43 @@ +/* +Copyright The Velero Contributors. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "context" + + "github.com/spf13/cobra" + + "github.com/vmware-tanzu/velero/pkg/client" +) + +func NewCommand(f client.Factory) *cobra.Command { + command := &cobra.Command{ + Use: "pod-volume", + Short: "Run the velero pod volume backup/restore", + Long: "Run the velero pod volume backup/restore", + Hidden: true, + } + + command.AddCommand( + NewBackupCommand(f), + ) + + return command +} + +type dataPathService interface { + Init() error + RunCancelableDataPath(context.Context) (string, error) + Shutdown() +} diff --git a/pkg/cmd/cli/repomantenance/maintenance.go b/pkg/cmd/cli/repomantenance/maintenance.go index 93ebebff3..46c54f7d2 100644 --- a/pkg/cmd/cli/repomantenance/maintenance.go +++ b/pkg/cmd/cli/repomantenance/maintenance.go @@ -121,7 +121,7 @@ func initRepoManager(namespace string, cli client.Client, kubeClient kubernetes. credentialFileStore, err := credentials.NewNamespacedFileStore( cli, namespace, - "/tmp/credentials", + credentials.DefaultStoreDirectory(), filesystem.NewFileSystem(), ) if err != nil { diff --git a/pkg/cmd/server/config/config.go b/pkg/cmd/server/config/config.go index 33a40a3c4..29c53b821 100644 --- a/pkg/cmd/server/config/config.go +++ b/pkg/cmd/server/config/config.go @@ -8,6 +8,7 @@ import ( "github.com/sirupsen/logrus" "github.com/spf13/pflag" + "github.com/vmware-tanzu/velero/internal/credentials" "github.com/vmware-tanzu/velero/pkg/cmd/util/flag" "github.com/vmware-tanzu/velero/pkg/constant" podvolumeconfigs "github.com/vmware-tanzu/velero/pkg/podvolume/configs" @@ -44,10 +45,6 @@ const ( defaultMaxConcurrentK8SConnections = 30 defaultDisableInformerCache = false - // defaultCredentialsDirectory is the path on disk where credential - // files will be written to - defaultCredentialsDirectory = "/tmp/credentials" - DefaultKeepLatestMaintenanceJobs = 3 DefaultMaintenanceJobCPURequest = "0" DefaultMaintenanceJobCPULimit = "0" @@ -211,7 +208,7 @@ func GetDefaultConfig() *Config { DefaultSnapshotMoveData: false, DisableInformerCache: defaultDisableInformerCache, ScheduleSkipImmediately: false, - CredentialsDirectory: defaultCredentialsDirectory, + CredentialsDirectory: credentials.DefaultStoreDirectory(), PodResources: kube.PodResources{ CPURequest: DefaultMaintenanceJobCPULimit, CPULimit: DefaultMaintenanceJobCPURequest, diff --git a/pkg/cmd/velero/velero.go b/pkg/cmd/velero/velero.go index eab96d62f..a74c68fbc 100644 --- a/pkg/cmd/velero/velero.go +++ b/pkg/cmd/velero/velero.go @@ -26,6 +26,7 @@ import ( "k8s.io/klog/v2" "github.com/vmware-tanzu/velero/pkg/cmd/cli/debug" + "github.com/vmware-tanzu/velero/pkg/cmd/cli/podvolume" "github.com/vmware-tanzu/velero/pkg/cmd/cli/repomantenance" "github.com/vmware-tanzu/velero/pkg/client" @@ -126,6 +127,7 @@ operations can also be performed as 'velero backup get' and 'velero schedule cre debug.NewCommand(f), repomantenance.NewCommand(f), datamover.NewCommand(f), + podvolume.NewCommand(f), ) // init and add the klog flags diff --git a/pkg/podvolume/backup_micro_service.go b/pkg/podvolume/backup_micro_service.go new file mode 100644 index 000000000..275b57fb9 --- /dev/null +++ b/pkg/podvolume/backup_micro_service.go @@ -0,0 +1,313 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "context" + "encoding/json" + "time" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/wait" + "k8s.io/client-go/kubernetes" + "sigs.k8s.io/controller-runtime/pkg/client" + + cachetool "k8s.io/client-go/tools/cache" + "sigs.k8s.io/controller-runtime/pkg/cache" + + "github.com/vmware-tanzu/velero/internal/credentials" + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/repository" + "github.com/vmware-tanzu/velero/pkg/uploader" + "github.com/vmware-tanzu/velero/pkg/util/kube" + + apierrors "k8s.io/apimachinery/pkg/api/errors" +) + +const ( + podVolumeRequestor = "snapshot-pod-volume" +) + +// BackupMicroService process data mover backups inside the backup pod +type BackupMicroService struct { + ctx context.Context + client client.Client + kubeClient kubernetes.Interface + repoEnsurer *repository.Ensurer + credentialGetter *credentials.CredentialGetter + logger logrus.FieldLogger + dataPathMgr *datapath.Manager + eventRecorder kube.EventRecorder + + namespace string + pvbName string + pvb *velerov1api.PodVolumeBackup + sourceTargetPath datapath.AccessPoint + + resultSignal chan dataPathResult + + pvbInformer cache.Informer + pvbHandler cachetool.ResourceEventHandlerRegistration + nodeName string +} + +type dataPathResult struct { + err error + result string +} + +func NewBackupMicroService(ctx context.Context, client client.Client, kubeClient kubernetes.Interface, pvbName string, namespace string, nodeName string, + sourceTargetPath datapath.AccessPoint, dataPathMgr *datapath.Manager, repoEnsurer *repository.Ensurer, cred *credentials.CredentialGetter, + pvbInformer cache.Informer, log logrus.FieldLogger) *BackupMicroService { + return &BackupMicroService{ + ctx: ctx, + client: client, + kubeClient: kubeClient, + credentialGetter: cred, + logger: log, + repoEnsurer: repoEnsurer, + dataPathMgr: dataPathMgr, + namespace: namespace, + pvbName: pvbName, + sourceTargetPath: sourceTargetPath, + nodeName: nodeName, + resultSignal: make(chan dataPathResult), + pvbInformer: pvbInformer, + } +} + +func (r *BackupMicroService) Init() error { + r.eventRecorder = kube.NewEventRecorder(r.kubeClient, r.client.Scheme(), r.pvbName, r.nodeName, r.logger) + + handler, err := r.pvbInformer.AddEventHandler( + cachetool.ResourceEventHandlerFuncs{ + UpdateFunc: func(oldObj any, newObj any) { + oldPvb := oldObj.(*velerov1api.PodVolumeBackup) + newPvb := newObj.(*velerov1api.PodVolumeBackup) + + if newPvb.Name != r.pvbName { + return + } + + if newPvb.Status.Phase != velerov1api.PodVolumeBackupPhaseInProgress { + return + } + + if newPvb.Spec.Cancel && !oldPvb.Spec.Cancel { + r.cancelPodVolumeBackup(newPvb) + } + }, + }, + ) + + if err != nil { + return errors.Wrap(err, "error adding PVB handler") + } + + r.pvbHandler = handler + + return err +} + +func (r *BackupMicroService) RunCancelableDataPath(ctx context.Context) (string, error) { + log := r.logger.WithFields(logrus.Fields{ + "PVB": r.pvbName, + }) + + pvb := &velerov1api.PodVolumeBackup{} + err := wait.PollUntilContextCancel(ctx, 500*time.Millisecond, true, func(ctx context.Context) (bool, error) { + err := r.client.Get(ctx, types.NamespacedName{ + Namespace: r.namespace, + Name: r.pvbName, + }, pvb) + + if apierrors.IsNotFound(err) { + return false, nil + } + + if err != nil { + return true, errors.Wrapf(err, "error to get PVB %s", r.pvbName) + } + + if pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseInProgress { + return true, nil + } else { + return false, nil + } + }) + + if err != nil { + log.WithError(err).Error("Failed to wait PVB") + return "", errors.Wrap(err, "error waiting for PVB") + } + + r.pvb = pvb + + log.Info("Run cancelable PVB") + + callbacks := datapath.Callbacks{ + OnCompleted: r.OnDataPathCompleted, + OnFailed: r.OnDataPathFailed, + OnCancelled: r.OnDataPathCancelled, + OnProgress: r.OnDataPathProgress, + } + + fsBackup, err := r.dataPathMgr.CreateFileSystemBR(pvb.Name, podVolumeRequestor, ctx, r.client, pvb.Namespace, callbacks, log) + if err != nil { + return "", errors.Wrap(err, "error to create data path") + } + + log.Debug("Async fs br created") + + if err := fsBackup.Init(ctx, &datapath.FSBRInitParam{ + BSLName: pvb.Spec.BackupStorageLocation, + SourceNamespace: pvb.Spec.Pod.Namespace, + UploaderType: pvb.Spec.UploaderType, + RepositoryType: velerov1api.BackupRepositoryTypeKopia, + RepoIdentifier: "", + RepositoryEnsurer: r.repoEnsurer, + CredentialGetter: r.credentialGetter, + }); err != nil { + return "", errors.Wrap(err, "error to initialize data path") + } + + log.Info("Async fs br init") + + tags := map[string]string{} + + if err := fsBackup.StartBackup(r.sourceTargetPath, pvb.Spec.UploaderSettings, &datapath.FSBRStartParam{ + RealSource: GetRealSource(pvb), + ParentSnapshot: "", + ForceFull: false, + Tags: tags, + }); err != nil { + return "", errors.Wrap(err, "error starting data path backup") + } + + log.Info("Async fs backup data path started") + r.eventRecorder.Event(pvb, false, datapath.EventReasonStarted, "Data path for %s started", pvb.Name) + + result := "" + select { + case <-ctx.Done(): + err = errors.New("timed out waiting for fs backup to complete") + break + case res := <-r.resultSignal: + err = res.err + result = res.result + break + } + + if err != nil { + log.WithError(err).Error("Async fs backup was not completed") + } + + r.eventRecorder.EndingEvent(pvb, false, datapath.EventReasonStopped, "Data path for %s stopped", pvb.Name) + + return result, err +} + +func (r *BackupMicroService) Shutdown() { + r.eventRecorder.Shutdown() + r.closeDataPath(r.ctx, r.pvbName) + + if r.pvbHandler != nil { + if err := r.pvbInformer.RemoveEventHandler(r.pvbHandler); err != nil { + r.logger.WithError(err).Warn("Failed to remove pod handler") + } + } +} + +var funcMarshal = json.Marshal + +func (r *BackupMicroService) OnDataPathCompleted(ctx context.Context, namespace string, pvbName string, result datapath.Result) { + log := r.logger.WithField("PVB", pvbName) + + backupBytes, err := funcMarshal(result.Backup) + if err != nil { + log.WithError(err).Errorf("Failed to marshal backup result %v", result.Backup) + r.resultSignal <- dataPathResult{ + err: errors.Wrapf(err, "Failed to marshal backup result %v", result.Backup), + } + } else { + r.eventRecorder.Event(r.pvb, false, datapath.EventReasonCompleted, string(backupBytes)) + r.resultSignal <- dataPathResult{ + result: string(backupBytes), + } + } + + log.Info("Async fs backup completed") +} + +func (r *BackupMicroService) OnDataPathFailed(ctx context.Context, namespace string, pvbName string, err error) { + log := r.logger.WithField("PVB", pvbName) + log.WithError(err).Error("Async fs backup data path failed") + + r.eventRecorder.Event(r.pvb, false, datapath.EventReasonFailed, "Data path for PVB %s failed, error %v", r.pvbName, err) + r.resultSignal <- dataPathResult{ + err: errors.Wrapf(err, "Data path for PVB %s failed", r.pvbName), + } +} + +func (r *BackupMicroService) OnDataPathCancelled(ctx context.Context, namespace string, pvbName string) { + log := r.logger.WithField("PVB", pvbName) + log.Warn("Async fs backup data path canceled") + + r.eventRecorder.Event(r.pvb, false, datapath.EventReasonCancelled, "Data path for PVB %s canceled", pvbName) + r.resultSignal <- dataPathResult{ + err: errors.New(datapath.ErrCancelled), + } +} + +func (r *BackupMicroService) OnDataPathProgress(ctx context.Context, namespace string, pvbName string, progress *uploader.Progress) { + log := r.logger.WithFields(logrus.Fields{ + "PVB": pvbName, + }) + + progressBytes, err := funcMarshal(progress) + if err != nil { + log.WithError(err).Errorf("Failed to marshal progress %v", progress) + return + } + + r.eventRecorder.Event(r.pvb, false, datapath.EventReasonProgress, string(progressBytes)) +} + +func (r *BackupMicroService) closeDataPath(ctx context.Context, duName string) { + fsBackup := r.dataPathMgr.GetAsyncBR(duName) + if fsBackup != nil { + fsBackup.Close(ctx) + } + + r.dataPathMgr.RemoveAsyncBR(duName) +} + +func (r *BackupMicroService) cancelPodVolumeBackup(pvb *velerov1api.PodVolumeBackup) { + r.logger.WithField("pvb", pvb.Name).Info("PVB is being canceled") + + r.eventRecorder.Event(pvb, false, datapath.EventReasonCancelling, "Canceling for PVB %s", pvb.Name) + + fsBackup := r.dataPathMgr.GetAsyncBR(pvb.Name) + if fsBackup == nil { + r.OnDataPathCancelled(r.ctx, pvb.GetNamespace(), pvb.GetName()) + } else { + fsBackup.Cancel() + } +} diff --git a/pkg/podvolume/backup_micro_service_test.go b/pkg/podvolume/backup_micro_service_test.go new file mode 100644 index 000000000..eb3a484d1 --- /dev/null +++ b/pkg/podvolume/backup_micro_service_test.go @@ -0,0 +1,447 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "context" + "fmt" + "sync" + "testing" + "time" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + "k8s.io/apimachinery/pkg/runtime" + + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/uploader" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + + clientFake "sigs.k8s.io/controller-runtime/pkg/client/fake" + + velerotest "github.com/vmware-tanzu/velero/pkg/test" + + kbclient "sigs.k8s.io/controller-runtime/pkg/client" + + datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks" +) + +type backupMsTestHelper struct { + eventReason string + eventMsg string + marshalErr error + marshalBytes []byte + withEvent bool + eventLock sync.Mutex +} + +func (bt *backupMsTestHelper) Event(_ runtime.Object, _ bool, reason string, message string, a ...any) { + bt.eventLock.Lock() + defer bt.eventLock.Unlock() + + bt.withEvent = true + bt.eventReason = reason + bt.eventMsg = fmt.Sprintf(message, a...) +} + +func (bt *backupMsTestHelper) EndingEvent(_ runtime.Object, _ bool, reason string, message string, a ...any) { + bt.eventLock.Lock() + defer bt.eventLock.Unlock() + + bt.withEvent = true + bt.eventReason = reason + bt.eventMsg = fmt.Sprintf(message, a...) +} +func (bt *backupMsTestHelper) Shutdown() {} + +func (bt *backupMsTestHelper) Marshal(v any) ([]byte, error) { + if bt.marshalErr != nil { + return nil, bt.marshalErr + } + + return bt.marshalBytes, nil +} + +func (bt *backupMsTestHelper) EventReason() string { + bt.eventLock.Lock() + defer bt.eventLock.Unlock() + + return bt.eventReason +} + +func (bt *backupMsTestHelper) EventMessage() string { + bt.eventLock.Lock() + defer bt.eventLock.Unlock() + + return bt.eventMsg +} + +func TestOnDataPathFailed(t *testing.T) { + pvbName := "fake-pvb" + bt := &backupMsTestHelper{} + + bs := &BackupMicroService{ + pvbName: pvbName, + dataPathMgr: datapath.NewManager(1), + eventRecorder: bt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + expectedErr := "Data path for PVB fake-pvb failed: fake-error" + expectedEventReason := datapath.EventReasonFailed + expectedEventMsg := "Data path for PVB fake-pvb failed, error fake-error" + + go bs.OnDataPathFailed(context.TODO(), velerov1api.DefaultNamespace, pvbName, errors.New("fake-error")) + + result := <-bs.resultSignal + assert.EqualError(t, result.err, expectedErr) + assert.Equal(t, expectedEventReason, bt.EventReason()) + assert.Equal(t, expectedEventMsg, bt.EventMessage()) +} + +func TestOnDataPathCancelled(t *testing.T) { + pvbName := "fake-pvb" + bt := &backupMsTestHelper{} + + bs := &BackupMicroService{ + pvbName: pvbName, + dataPathMgr: datapath.NewManager(1), + eventRecorder: bt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + expectedErr := datapath.ErrCancelled + expectedEventReason := datapath.EventReasonCancelled + expectedEventMsg := "Data path for PVB fake-pvb canceled" + + go bs.OnDataPathCancelled(context.TODO(), velerov1api.DefaultNamespace, pvbName) + + result := <-bs.resultSignal + assert.EqualError(t, result.err, expectedErr) + assert.Equal(t, expectedEventReason, bt.EventReason()) + assert.Equal(t, expectedEventMsg, bt.EventMessage()) +} + +func TestOnDataPathCompleted(t *testing.T) { + tests := []struct { + name string + expectedErr string + expectedEventReason string + expectedEventMsg string + marshalErr error + marshallStr string + }{ + { + name: "marshal fail", + marshalErr: errors.New("fake-marshal-error"), + expectedErr: "Failed to marshal backup result { false { } 0}: fake-marshal-error", + }, + { + name: "succeed", + marshallStr: "fake-complete-string", + expectedEventReason: datapath.EventReasonCompleted, + expectedEventMsg: "fake-complete-string", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + pvbName := "fake-pvb" + + bt := &backupMsTestHelper{ + marshalErr: test.marshalErr, + marshalBytes: []byte(test.marshallStr), + } + + bs := &BackupMicroService{ + dataPathMgr: datapath.NewManager(1), + eventRecorder: bt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + funcMarshal = bt.Marshal + + go bs.OnDataPathCompleted(context.TODO(), velerov1api.DefaultNamespace, pvbName, datapath.Result{}) + + result := <-bs.resultSignal + if test.marshalErr != nil { + assert.EqualError(t, result.err, test.expectedErr) + } else { + assert.NoError(t, result.err) + assert.Equal(t, test.expectedEventReason, bt.EventReason()) + assert.Equal(t, test.expectedEventMsg, bt.EventMessage()) + } + }) + } +} + +func TestOnDataPathProgress(t *testing.T) { + tests := []struct { + name string + expectedErr string + expectedEventReason string + expectedEventMsg string + marshalErr error + marshallStr string + }{ + { + name: "marshal fail", + marshalErr: errors.New("fake-marshal-error"), + expectedErr: "Failed to marshal backup result", + }, + { + name: "succeed", + marshallStr: "fake-progress-string", + expectedEventReason: datapath.EventReasonProgress, + expectedEventMsg: "fake-progress-string", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + pvbName := "fake-pvb" + + bt := &backupMsTestHelper{ + marshalErr: test.marshalErr, + marshalBytes: []byte(test.marshallStr), + } + + bs := &BackupMicroService{ + dataPathMgr: datapath.NewManager(1), + eventRecorder: bt, + logger: velerotest.NewLogger(), + } + + funcMarshal = bt.Marshal + + bs.OnDataPathProgress(context.TODO(), velerov1api.DefaultNamespace, pvbName, &uploader.Progress{}) + + if test.marshalErr != nil { + assert.False(t, bt.withEvent) + } else { + assert.True(t, bt.withEvent) + assert.Equal(t, test.expectedEventReason, bt.EventReason()) + assert.Equal(t, test.expectedEventMsg, bt.EventMessage()) + } + }) + } +} + +func TestCancelPodVolumeBackup(t *testing.T) { + tests := []struct { + name string + expectedEventReason string + expectedEventMsg string + expectedErr string + }{ + { + name: "no fs backup", + expectedEventReason: datapath.EventReasonCancelled, + expectedEventMsg: "Data path for PVB fake-pvb canceled", + expectedErr: datapath.ErrCancelled, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + pvbName := "fake-pvb" + pvb := builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, pvbName).Result() + + bt := &backupMsTestHelper{} + + bs := &BackupMicroService{ + dataPathMgr: datapath.NewManager(1), + eventRecorder: bt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + go bs.cancelPodVolumeBackup(pvb) + + result := <-bs.resultSignal + + assert.EqualError(t, result.err, test.expectedErr) + assert.True(t, bt.withEvent) + assert.Equal(t, test.expectedEventReason, bt.EventReason()) + assert.Equal(t, test.expectedEventMsg, bt.EventMessage()) + }) + } +} + +func TestRunCancelableDataPath(t *testing.T) { + pvbName := "fake-pvb" + pvb := builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, pvbName).Phase(velerov1api.PodVolumeBackupPhaseNew).Result() + pvbInProgress := builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, pvbName).Phase(velerov1api.PodVolumeBackupPhaseInProgress).Result() + ctxTimeout, cancel := context.WithTimeout(context.Background(), time.Second) + + tests := []struct { + name string + ctx context.Context + result *dataPathResult + dataPathMgr *datapath.Manager + kubeClientObj []runtime.Object + initErr error + startErr error + dataPathStarted bool + expectedEventMsg string + expectedErr string + }{ + { + name: "no pvb", + ctx: ctxTimeout, + expectedErr: "error waiting for PVB: context deadline exceeded", + }, + { + name: "pvb not in in-progress", + ctx: ctxTimeout, + kubeClientObj: []runtime.Object{pvb}, + expectedErr: "error waiting for PVB: context deadline exceeded", + }, + { + name: "create data path fail", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvbInProgress}, + dataPathMgr: datapath.NewManager(0), + expectedErr: "error to create data path: Concurrent number exceeds", + }, + { + name: "init data path fail", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvbInProgress}, + initErr: errors.New("fake-init-error"), + expectedErr: "error to initialize data path: fake-init-error", + }, + { + name: "start data path fail", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvbInProgress}, + startErr: errors.New("fake-start-error"), + expectedErr: "error starting data path backup: fake-start-error", + }, + { + name: "data path timeout", + ctx: ctxTimeout, + kubeClientObj: []runtime.Object{pvbInProgress}, + dataPathStarted: true, + expectedEventMsg: fmt.Sprintf("Data path for %s stopped", pvbName), + expectedErr: "timed out waiting for fs backup to complete", + }, + { + name: "data path returns error", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvbInProgress}, + dataPathStarted: true, + result: &dataPathResult{ + err: errors.New("fake-data-path-error"), + }, + expectedEventMsg: fmt.Sprintf("Data path for %s stopped", pvbName), + expectedErr: "fake-data-path-error", + }, + { + name: "succeed", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvbInProgress}, + dataPathStarted: true, + result: &dataPathResult{ + result: "fake-succeed-result", + }, + expectedEventMsg: fmt.Sprintf("Data path for %s stopped", pvbName), + }, + } + + scheme := runtime.NewScheme() + velerov1api.AddToScheme(scheme) + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fakeClientBuilder := clientFake.NewClientBuilder() + fakeClientBuilder = fakeClientBuilder.WithScheme(scheme) + + fakeClient := fakeClientBuilder.WithRuntimeObjects(test.kubeClientObj...).Build() + + bt := &backupMsTestHelper{} + + bs := &BackupMicroService{ + namespace: velerov1api.DefaultNamespace, + pvbName: pvbName, + ctx: context.Background(), + client: fakeClient, + dataPathMgr: datapath.NewManager(1), + eventRecorder: bt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + if test.ctx != nil { + bs.ctx = test.ctx + } + + if test.dataPathMgr != nil { + bs.dataPathMgr = test.dataPathMgr + } + + datapath.FSBRCreator = func(string, string, kbclient.Client, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { + fsBR := datapathmockes.NewAsyncBR(t) + if test.initErr != nil { + fsBR.On("Init", mock.Anything, mock.Anything).Return(test.initErr) + } + + if test.startErr != nil { + fsBR.On("Init", mock.Anything, mock.Anything).Return(nil) + fsBR.On("StartBackup", mock.Anything, mock.Anything, mock.Anything).Return(test.startErr) + } + + if test.dataPathStarted { + fsBR.On("Init", mock.Anything, mock.Anything).Return(nil) + fsBR.On("StartBackup", mock.Anything, mock.Anything, mock.Anything).Return(nil) + } + + return fsBR + } + + if test.result != nil { + go func() { + time.Sleep(time.Millisecond * 500) + bs.resultSignal <- *test.result + }() + } + + result, err := bs.RunCancelableDataPath(test.ctx) + + if test.expectedErr != "" { + assert.EqualError(t, err, test.expectedErr) + } else { + assert.NoError(t, err) + assert.Equal(t, test.result.result, result) + } + + if test.expectedEventMsg != "" { + assert.True(t, bt.withEvent) + assert.Equal(t, test.expectedEventMsg, bt.EventMessage()) + } + }) + } + + cancel() +} diff --git a/pkg/podvolume/util.go b/pkg/podvolume/util.go index e9bb3075c..dab291768 100644 --- a/pkg/podvolume/util.go +++ b/pkg/podvolume/util.go @@ -17,12 +17,14 @@ limitations under the License. package podvolume import ( + "fmt" "strings" corev1api "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/podvolume/configs" repotypes "github.com/vmware-tanzu/velero/pkg/repository/types" "github.com/vmware-tanzu/velero/pkg/uploader" ) @@ -143,6 +145,19 @@ func GetSnapshotIdentifier(podVolumeBackups *velerov1api.PodVolumeBackupList) ma return res } +func GetRealSource(pvb *velerov1api.PodVolumeBackup) string { + pvcName := "" + if pvb.Annotations != nil { + pvcName = pvb.Annotations[configs.PVCNameAnnotation] + } + + if pvcName != "" { + return fmt.Sprintf("%s/%s/%s", pvb.Spec.Pod.Namespace, pvb.Spec.Pod.Name, pvcName) + } else { + return fmt.Sprintf("%s/%s/%s", pvb.Spec.Pod.Namespace, pvb.Spec.Pod.Name, pvb.Spec.Volume) + } +} + func getUploaderTypeOrDefault(uploaderType string) string { if uploaderType != "" { return uploaderType diff --git a/pkg/util/kube/pod.go b/pkg/util/kube/pod.go index 04457f0d4..ba506dae1 100644 --- a/pkg/util/kube/pod.go +++ b/pkg/util/kube/pod.go @@ -19,6 +19,7 @@ import ( "context" "fmt" "io" + "os" "time" "github.com/pkg/errors" @@ -273,3 +274,30 @@ func DiagnosePod(pod *corev1api.Pod) string { return diag } + +var funcExit = os.Exit +var funcCreateFile = os.Create + +func ExitPodWithMessage(logger logrus.FieldLogger, succeed bool, message string, a ...any) { + exitCode := 0 + if !succeed { + exitCode = 1 + } + + toWrite := fmt.Sprintf(message, a...) + + podFile, err := funcCreateFile("/dev/termination-log") + if err != nil { + logger.WithError(err).Error("Failed to create termination log file") + exitCode = 1 + } else { + if _, err := podFile.WriteString(toWrite); err != nil { + logger.WithError(err).Error("Failed to write error to termination log file") + exitCode = 1 + } + + podFile.Close() + } + + funcExit(exitCode) +} diff --git a/pkg/util/kube/pod_test.go b/pkg/util/kube/pod_test.go index 2c19091f3..c1f8ba912 100644 --- a/pkg/util/kube/pod_test.go +++ b/pkg/util/kube/pod_test.go @@ -18,14 +18,19 @@ package kube import ( "context" + "fmt" "io" + "os" + "path/filepath" "reflect" "strings" "testing" "time" + "github.com/google/uuid" "github.com/pkg/errors" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" corev1api "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -932,3 +937,105 @@ func TestDiagnosePod(t *testing.T) { }) } } + +type exitWithMessageMock struct { + createErr error + writeFail bool + filePath string + exitCode int +} + +func (em *exitWithMessageMock) Exit(code int) { + em.exitCode = code +} + +func (em *exitWithMessageMock) CreateFile(name string) (*os.File, error) { + if em.createErr != nil { + return nil, em.createErr + } + + if em.writeFail { + return os.OpenFile(em.filePath, os.O_CREATE|os.O_RDONLY, 0500) + } else { + return os.Create(em.filePath) + } +} + +func TestExitPodWithMessage(t *testing.T) { + tests := []struct { + name string + message string + succeed bool + args []any + createErr error + writeFail bool + expectedExitCode int + expectedMessage string + }{ + { + name: "create pod file failed", + createErr: errors.New("fake-create-file-error"), + succeed: true, + expectedExitCode: 1, + }, + { + name: "write pod file failed", + writeFail: true, + succeed: true, + expectedExitCode: 1, + }, + { + name: "not succeed", + message: "fake-message-1, arg-1 %s, arg-2 %v, arg-3 %v", + args: []any{ + "arg-1-1", + 10, + false, + }, + expectedExitCode: 1, + expectedMessage: fmt.Sprintf("fake-message-1, arg-1 %s, arg-2 %v, arg-3 %v", "arg-1-1", 10, false), + }, + { + name: "not succeed", + message: "fake-message-2, arg-1 %s, arg-2 %v, arg-3 %v", + args: []any{ + "arg-1-2", + 20, + true, + }, + succeed: true, + expectedMessage: fmt.Sprintf("fake-message-2, arg-1 %s, arg-2 %v, arg-3 %v", "arg-1-2", 20, true), + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + podFile := filepath.Join(os.TempDir(), uuid.NewString()) + + em := exitWithMessageMock{ + createErr: test.createErr, + writeFail: test.writeFail, + filePath: podFile, + } + + funcExit = em.Exit + funcCreateFile = em.CreateFile + + ExitPodWithMessage(velerotest.NewLogger(), test.succeed, test.message, test.args...) + + assert.Equal(t, test.expectedExitCode, em.exitCode) + + if test.createErr == nil && !test.writeFail { + reader, err := os.Open(podFile) + require.NoError(t, err) + + message, err := io.ReadAll(reader) + require.NoError(t, err) + + reader.Close() + + assert.Equal(t, test.expectedMessage, string(message)) + } + }) + } +} From b54404fc56c75d1aa906b2627a139af8abd26b6e Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Tue, 3 Jun 2025 14:35:27 +0800 Subject: [PATCH 15/31] vgdp ms pvr data path Signed-off-by: Lyndon-Li --- pkg/apis/velero/v1/pod_volume_restore_type.go | 4 + pkg/cmd/cli/podvolume/restore.go | 284 +++++++++++++++ pkg/podvolume/pvr_micro_service.go | 334 ++++++++++++++++++ 3 files changed, 622 insertions(+) create mode 100644 pkg/cmd/cli/podvolume/restore.go create mode 100644 pkg/podvolume/pvr_micro_service.go diff --git a/pkg/apis/velero/v1/pod_volume_restore_type.go b/pkg/apis/velero/v1/pod_volume_restore_type.go index 34bc7e530..d8871b708 100644 --- a/pkg/apis/velero/v1/pod_volume_restore_type.go +++ b/pkg/apis/velero/v1/pod_volume_restore_type.go @@ -54,6 +54,10 @@ type PodVolumeRestoreSpec struct { // +optional // +nullable UploaderSettings map[string]string `json:"uploaderSettings,omitempty"` + + // Cancel indicates request to cancel the ongoing PodVolumeRestore. It can be set + // when the PodVolumeRestore is in InProgress phase + Cancel bool `json:"cancel,omitempty"` } // PodVolumeRestorePhase represents the lifecycle phase of a PodVolumeRestore. diff --git a/pkg/cmd/cli/podvolume/restore.go b/pkg/cmd/cli/podvolume/restore.go new file mode 100644 index 000000000..2894cf8c6 --- /dev/null +++ b/pkg/cmd/cli/podvolume/restore.go @@ -0,0 +1,284 @@ +/* +Copyright The Velero Contributors. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "context" + "fmt" + "os" + "strings" + "time" + + "github.com/bombsimon/logrusr/v3" + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + "github.com/spf13/cobra" + corev1api "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/fields" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + + "github.com/vmware-tanzu/velero/internal/credentials" + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" + "github.com/vmware-tanzu/velero/pkg/buildinfo" + "github.com/vmware-tanzu/velero/pkg/client" + "github.com/vmware-tanzu/velero/pkg/cmd/util/signals" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/podvolume" + "github.com/vmware-tanzu/velero/pkg/repository" + "github.com/vmware-tanzu/velero/pkg/uploader" + "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/logging" + + ctlcache "sigs.k8s.io/controller-runtime/pkg/cache" + ctlclient "sigs.k8s.io/controller-runtime/pkg/client" +) + +type pvrConfig struct { + volumePath string + pvrName string + resourceTimeout time.Duration + winHPC bool +} + +func NewRestoreCommand(f client.Factory) *cobra.Command { + logLevelFlag := logging.LogLevelFlag(logrus.InfoLevel) + formatFlag := logging.NewFormatFlag() + + config := pvrConfig{} + + command := &cobra.Command{ + Use: "restore", + Short: "Run the velero pod volume restore", + Long: "Run the velero pod volume restore", + Hidden: true, + Run: func(c *cobra.Command, args []string) { + logLevel := logLevelFlag.Parse() + logrus.Infof("Setting log-level to %s", strings.ToUpper(logLevel.String())) + + logger := logging.DefaultLogger(logLevel, formatFlag.Parse()) + logger.Infof("Starting Velero pod volume restore %s (%s)", buildinfo.Version, buildinfo.FormattedGitSHA()) + + f.SetBasename(fmt.Sprintf("%s-%s", c.Parent().Name(), c.Name())) + s, err := newPodVolumeRestore(logger, f, config) + if err != nil { + exitWithMessage(logger, false, "Failed to create pod volume restore, %v", err) + } + + s.run() + }, + } + + command.Flags().Var(logLevelFlag, "log-level", fmt.Sprintf("The level at which to log. Valid values are %s.", strings.Join(logLevelFlag.AllowedValues(), ", "))) + command.Flags().Var(formatFlag, "log-format", fmt.Sprintf("The format for log output. Valid values are %s.", strings.Join(formatFlag.AllowedValues(), ", "))) + command.Flags().StringVar(&config.volumePath, "volume-path", config.volumePath, "The full path of the volume to be restored") + command.Flags().StringVar(&config.pvrName, "pod-volume-restore", config.pvrName, "The PVR name") + command.Flags().DurationVar(&config.resourceTimeout, "resource-timeout", config.resourceTimeout, "How long to wait for resource processes which are not covered by other specific timeout parameters.") + + _ = command.MarkFlagRequired("volume-path") + _ = command.MarkFlagRequired("pod-volume-restore") + _ = command.MarkFlagRequired("resource-timeout") + + return command +} + +type podVolumeRestore struct { + logger logrus.FieldLogger + ctx context.Context + cancelFunc context.CancelFunc + client ctlclient.Client + cache ctlcache.Cache + namespace string + nodeName string + config pvrConfig + kubeClient kubernetes.Interface + dataPathMgr *datapath.Manager +} + +func newPodVolumeRestore(logger logrus.FieldLogger, factory client.Factory, config pvrConfig) (*podVolumeRestore, error) { + ctx, cancelFunc := context.WithCancel(context.Background()) + + clientConfig, err := factory.ClientConfig() + if err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to create client config") + } + + ctrl.SetLogger(logrusr.New(logger)) + klog.SetLogger(logrusr.New(logger)) // klog.Logger is used by k8s.io/client-go + + scheme := runtime.NewScheme() + if err := velerov1api.AddToScheme(scheme); err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to add velero v1 scheme") + } + + if err := corev1api.AddToScheme(scheme); err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to add core v1 scheme") + } + + nodeName := os.Getenv("NODE_NAME") + + // use a field selector to filter to only pods scheduled on this node. + cacheOption := ctlcache.Options{ + Scheme: scheme, + ByObject: map[ctlclient.Object]ctlcache.ByObject{ + &corev1api.Pod{}: { + Field: fields.Set{"spec.nodeName": nodeName}.AsSelector(), + }, + &velerov1api.PodVolumeRestore{}: { + Field: fields.Set{"metadata.namespace": factory.Namespace()}.AsSelector(), + }, + }, + } + + cli, err := ctlclient.New(clientConfig, ctlclient.Options{ + Scheme: scheme, + }) + if err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to create client") + } + + var cache ctlcache.Cache + retry := 10 + for { + cache, err = ctlcache.New(clientConfig, cacheOption) + if err == nil { + break + } + + retry-- + if retry == 0 { + break + } + + logger.WithError(err).Warn("Failed to create client cache, need retry") + + time.Sleep(time.Second) + } + + if err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to create client cache") + } + + s := &podVolumeRestore{ + logger: logger, + ctx: ctx, + cancelFunc: cancelFunc, + client: cli, + cache: cache, + config: config, + namespace: factory.Namespace(), + nodeName: nodeName, + } + + s.kubeClient, err = factory.KubeClient() + if err != nil { + cancelFunc() + return nil, errors.Wrap(err, "error to create kube client") + } + + s.dataPathMgr = datapath.NewManager(1) + + return s, nil +} + +var funcCreateDataPathRestore = (*podVolumeRestore).createDataPathService + +func (s *podVolumeRestore) run() { + signals.CancelOnShutdown(s.cancelFunc, s.logger) + go func() { + if err := s.cache.Start(s.ctx); err != nil { + s.logger.WithError(err).Warn("error starting cache") + } + }() + + s.runDataPath() +} + +func (s *podVolumeRestore) runDataPath() { + s.logger.Infof("Starting micro service in node %s for PVR %s", s.nodeName, s.config.pvrName) + + dpService, err := funcCreateDataPathRestore(s) + if err != nil { + s.cancelFunc() + funcExitWithMessage(s.logger, false, "Failed to create data path service for PVR %s: %v", s.config.pvrName, err) + return + } + + s.logger.Infof("Starting data path service %s", s.config.pvrName) + + err = dpService.Init() + if err != nil { + dpService.Shutdown() + s.cancelFunc() + funcExitWithMessage(s.logger, false, "Failed to init data path service for PVR %s: %v", s.config.pvrName, err) + return + } + + result, err := dpService.RunCancelableDataPath(s.ctx) + if err != nil { + dpService.Shutdown() + s.cancelFunc() + funcExitWithMessage(s.logger, false, "Failed to run data path service for PVR %s: %v", s.config.pvrName, err) + return + } + + s.logger.WithField("PVR", s.config.pvrName).Info("Data path service completed") + + dpService.Shutdown() + + s.logger.WithField("PVR", s.config.pvrName).Info("Data path service is shut down") + + s.cancelFunc() + + funcExitWithMessage(s.logger, true, result) +} + +func (s *podVolumeRestore) createDataPathService() (dataPathService, error) { + credentialFileStore, err := funcNewCredentialFileStore( + s.client, + s.namespace, + credentials.DefaultStoreDirectory(), + filesystem.NewFileSystem(), + ) + if err != nil { + return nil, errors.Wrapf(err, "error to create credential file store") + } + + credSecretStore, err := funcNewCredentialSecretStore(s.client, s.namespace) + if err != nil { + return nil, errors.Wrapf(err, "error to create credential secret store") + } + + credGetter := &credentials.CredentialGetter{FromFile: credentialFileStore, FromSecret: credSecretStore} + + pvrInformer, err := s.cache.GetInformer(s.ctx, &velerov2alpha1api.DataDownload{}) + if err != nil { + return nil, errors.Wrap(err, "error to get controller-runtime informer from manager") + } + + repoEnsurer := repository.NewEnsurer(s.client, s.logger, s.config.resourceTimeout) + + return podvolume.NewRestoreMicroService(s.ctx, s.client, s.kubeClient, s.config.pvrName, s.namespace, s.nodeName, datapath.AccessPoint{ + ByPath: s.config.volumePath, + VolMode: uploader.PersistentVolumeFilesystem, + }, s.dataPathMgr, repoEnsurer, credGetter, pvrInformer, s.logger), nil +} diff --git a/pkg/podvolume/pvr_micro_service.go b/pkg/podvolume/pvr_micro_service.go new file mode 100644 index 000000000..146c73992 --- /dev/null +++ b/pkg/podvolume/pvr_micro_service.go @@ -0,0 +1,334 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "context" + "fmt" + "os" + "path/filepath" + "time" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/wait" + "k8s.io/client-go/kubernetes" + "sigs.k8s.io/controller-runtime/pkg/cache" + "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/vmware-tanzu/velero/internal/credentials" + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/repository" + "github.com/vmware-tanzu/velero/pkg/uploader" + "github.com/vmware-tanzu/velero/pkg/util/boolptr" + "github.com/vmware-tanzu/velero/pkg/util/kube" + + cachetool "k8s.io/client-go/tools/cache" +) + +// RestoreMicroService process data mover restores inside the restore pod +type RestoreMicroService struct { + ctx context.Context + client client.Client + kubeClient kubernetes.Interface + repoEnsurer *repository.Ensurer + credentialGetter *credentials.CredentialGetter + logger logrus.FieldLogger + dataPathMgr *datapath.Manager + eventRecorder kube.EventRecorder + + namespace string + pvrName string + pvr *velerov1api.PodVolumeRestore + sourceTargetPath datapath.AccessPoint + + resultSignal chan dataPathResult + + ddInformer cache.Informer + pvrHandler cachetool.ResourceEventHandlerRegistration + nodeName string +} + +func NewRestoreMicroService(ctx context.Context, client client.Client, kubeClient kubernetes.Interface, pvrName string, namespace string, nodeName string, + sourceTargetPath datapath.AccessPoint, dataPathMgr *datapath.Manager, repoEnsurer *repository.Ensurer, cred *credentials.CredentialGetter, + ddInformer cache.Informer, log logrus.FieldLogger) *RestoreMicroService { + return &RestoreMicroService{ + ctx: ctx, + client: client, + kubeClient: kubeClient, + credentialGetter: cred, + logger: log, + repoEnsurer: repoEnsurer, + dataPathMgr: dataPathMgr, + namespace: namespace, + pvrName: pvrName, + sourceTargetPath: sourceTargetPath, + nodeName: nodeName, + resultSignal: make(chan dataPathResult), + ddInformer: ddInformer, + } +} + +func (r *RestoreMicroService) Init() error { + r.eventRecorder = kube.NewEventRecorder(r.kubeClient, r.client.Scheme(), r.pvrName, r.nodeName, r.logger) + + handler, err := r.ddInformer.AddEventHandler( + cachetool.ResourceEventHandlerFuncs{ + UpdateFunc: func(oldObj any, newObj any) { + oldPvr := oldObj.(*velerov1api.PodVolumeRestore) + newPvr := newObj.(*velerov1api.PodVolumeRestore) + + if newPvr.Name != r.pvrName { + return + } + + if newPvr.Status.Phase != velerov1api.PodVolumeRestorePhaseInProgress { + return + } + + if newPvr.Spec.Cancel && !oldPvr.Spec.Cancel { + r.cancelPodVolumeRestore(newPvr) + } + }, + }, + ) + + if err != nil { + return errors.Wrap(err, "error adding PVR handler") + } + + r.pvrHandler = handler + + return err +} + +func (r *RestoreMicroService) RunCancelableDataPath(ctx context.Context) (string, error) { + log := r.logger.WithFields(logrus.Fields{ + "PVR": r.pvrName, + }) + + pvr := &velerov1api.PodVolumeRestore{} + err := wait.PollUntilContextCancel(ctx, 500*time.Millisecond, true, func(ctx context.Context) (bool, error) { + err := r.client.Get(ctx, types.NamespacedName{ + Namespace: r.namespace, + Name: r.pvrName, + }, pvr) + if apierrors.IsNotFound(err) { + return false, nil + } + + if err != nil { + return true, errors.Wrapf(err, "error to get PVR %s", r.pvrName) + } + + if pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseInProgress { + return true, nil + } else { + return false, nil + } + }) + if err != nil { + log.WithError(err).Error("Failed to wait PVR") + return "", errors.Wrap(err, "error waiting for PVR") + } + + r.pvr = pvr + + log.Info("Run cancelable PVR") + + callbacks := datapath.Callbacks{ + OnCompleted: r.OnPvrCompleted, + OnFailed: r.OnPvrFailed, + OnCancelled: r.OnPvrCancelled, + OnProgress: r.OnPvrProgress, + } + + fsRestore, err := r.dataPathMgr.CreateFileSystemBR(pvr.Name, podVolumeRequestor, ctx, r.client, pvr.Namespace, callbacks, log) + if err != nil { + return "", errors.Wrap(err, "error to create data path") + } + + log.Debug("Found volume path") + if err := fsRestore.Init(ctx, + &datapath.FSBRInitParam{ + BSLName: pvr.Spec.BackupStorageLocation, + SourceNamespace: pvr.Spec.SourceNamespace, + UploaderType: pvr.Spec.UploaderType, + RepositoryType: velerov1api.BackupRepositoryTypeKopia, + RepoIdentifier: "", + RepositoryEnsurer: r.repoEnsurer, + CredentialGetter: r.credentialGetter, + }); err != nil { + return "", errors.Wrap(err, "error to initialize data path") + } + log.Info("fs init") + + if err := fsRestore.StartRestore(pvr.Spec.SnapshotID, r.sourceTargetPath, pvr.Spec.UploaderSettings); err != nil { + return "", errors.Wrap(err, "error starting data path restore") + } + + log.Info("Async fs restore data path started") + r.eventRecorder.Event(pvr, false, datapath.EventReasonStarted, "Data path for %s started", pvr.Name) + + result := "" + select { + case <-ctx.Done(): + err = errors.New("timed out waiting for fs restore to complete") + break + case res := <-r.resultSignal: + err = res.err + result = res.result + break + } + + if err != nil { + log.WithError(err).Error("Async fs restore was not completed") + } + + r.eventRecorder.EndingEvent(pvr, false, datapath.EventReasonStopped, "Data path for %s stopped", pvr.Name) + + return result, err +} + +func (r *RestoreMicroService) Shutdown() { + r.eventRecorder.Shutdown() + r.closeDataPath(r.ctx, r.pvrName) + + if r.pvrHandler != nil { + if err := r.ddInformer.RemoveEventHandler(r.pvrHandler); err != nil { + r.logger.WithError(err).Warn("Failed to remove pod handler") + } + } +} + +func (r *RestoreMicroService) OnPvrCompleted(ctx context.Context, namespace string, pvrName string, result datapath.Result) { + log := r.logger.WithField("PVR", pvrName) + + volumePath := result.Restore.Target.ByPath + if volumePath == "" { + r.recordPvrFailed(pvrName, "invalid restore target", errors.New("path is empty")) + return + } + + // Remove the .velero directory from the restored volume (it may contain done files from previous restores + // of this volume, which we don't want to carry over). If this fails for any reason, log and continue, since + // this is non-essential cleanup (the done files are named based on restore UID and the init container looks + // for the one specific to the restore being executed). + if err := os.RemoveAll(filepath.Join(volumePath, ".velero")); err != nil { + log.WithError(err).Warnf("error removing .velero directory from directory %s", volumePath) + } + + var restoreUID types.UID + for _, owner := range r.pvr.OwnerReferences { + if boolptr.IsSetToTrue(owner.Controller) { + restoreUID = owner.UID + break + } + } + + // Create the .velero directory within the volume dir so we can write a done file + // for this restore. + if err := os.MkdirAll(filepath.Join(volumePath, ".velero"), 0755); err != nil { + r.recordPvrFailed(pvrName, "error creating .velero directory for done file", err) + return + } + + // Write a done file with name= into the just-created .velero dir + // within the volume. The velero init container on the pod is waiting + // for this file to exist in each restored volume before completing. + if err := os.WriteFile(filepath.Join(volumePath, ".velero", string(restoreUID)), nil, 0644); err != nil { //nolint:gosec // Internal usage. No need to check. + r.recordPvrFailed(pvrName, "error writing done file", err) + return + } + + restoreBytes, err := funcMarshal(result.Restore) + if err != nil { + log.WithError(err).Errorf("Failed to marshal restore result %v", result.Restore) + r.recordPvrFailed(pvrName, fmt.Sprintf("error marshaling restore result %v", result.Restore), err) + } else { + r.eventRecorder.Event(r.pvr, false, datapath.EventReasonCompleted, string(restoreBytes)) + r.resultSignal <- dataPathResult{ + result: string(restoreBytes), + } + } + + log.Info("Async fs restore data path completed") +} + +func (r *RestoreMicroService) recordPvrFailed(pvrName string, msg string, err error) { + evtMsg := fmt.Sprintf("%s, error %v", msg, err) + r.eventRecorder.Event(r.pvr, false, datapath.EventReasonFailed, evtMsg) + r.resultSignal <- dataPathResult{ + err: errors.Wrapf(err, msg, pvrName), + } +} + +func (r *RestoreMicroService) OnPvrFailed(ctx context.Context, namespace string, pvrName string, err error) { + log := r.logger.WithField("PVR", pvrName) + log.WithError(err).Error("Async fs restore data path failed") + + r.recordPvrFailed(pvrName, fmt.Sprintf("Data path for PVR %s failed", pvrName), err) +} + +func (r *RestoreMicroService) OnPvrCancelled(ctx context.Context, namespace string, pvrName string) { + log := r.logger.WithField("PVR", pvrName) + log.Warn("Async fs restore data path canceled") + + r.eventRecorder.Event(r.pvr, false, datapath.EventReasonCancelled, "Data path for PVR %s canceled", pvrName) + r.resultSignal <- dataPathResult{ + err: errors.New(datapath.ErrCancelled), + } +} + +func (r *RestoreMicroService) OnPvrProgress(ctx context.Context, namespace string, pvrName string, progress *uploader.Progress) { + log := r.logger.WithFields(logrus.Fields{ + "PVR": pvrName, + }) + + progressBytes, err := funcMarshal(progress) + if err != nil { + log.WithError(err).Errorf("Failed to marshal progress %v", progress) + return + } + + r.eventRecorder.Event(r.pvr, false, datapath.EventReasonProgress, string(progressBytes)) +} + +func (r *RestoreMicroService) closeDataPath(ctx context.Context, ddName string) { + fsRestore := r.dataPathMgr.GetAsyncBR(ddName) + if fsRestore != nil { + fsRestore.Close(ctx) + } + + r.dataPathMgr.RemoveAsyncBR(ddName) +} + +func (r *RestoreMicroService) cancelPodVolumeRestore(pvr *velerov1api.PodVolumeRestore) { + r.logger.WithField("PVR", pvr.Name).Info("PVR is being canceled") + + r.eventRecorder.Event(pvr, false, datapath.EventReasonCancelling, "Canceling for PVR %s", pvr.Name) + + fsBackup := r.dataPathMgr.GetAsyncBR(pvr.Name) + if fsBackup == nil { + r.OnPvrCancelled(r.ctx, pvr.GetNamespace(), pvr.GetName()) + } else { + fsBackup.Cancel() + } +} From a65e11e04dab7e5a81c53f6fec8c47070a8cacc0 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Tue, 3 Jun 2025 13:45:40 +0800 Subject: [PATCH 16/31] data path for vgdp ms pvb Signed-off-by: Lyndon-Li --- pkg/builder/pod_volume_backup_builder.go | 6 +++++ pkg/cmd/cli/podvolume/backup.go | 6 ++--- pkg/podvolume/backup_micro_service.go | 2 +- pkg/podvolume/util_test.go | 31 ++++++++++++++++++++++++ 4 files changed, 41 insertions(+), 4 deletions(-) diff --git a/pkg/builder/pod_volume_backup_builder.go b/pkg/builder/pod_volume_backup_builder.go index 80cb55042..ac89f21de 100644 --- a/pkg/builder/pod_volume_backup_builder.go +++ b/pkg/builder/pod_volume_backup_builder.go @@ -113,3 +113,9 @@ func (b *PodVolumeBackupBuilder) UploaderType(uploaderType string) *PodVolumeBac b.object.Spec.UploaderType = uploaderType return b } + +// Annotations sets the PodVolumeBackup's Annotations. +func (b *PodVolumeBackupBuilder) Annotations(annotations map[string]string) *PodVolumeBackupBuilder { + b.object.Annotations = annotations + return b +} diff --git a/pkg/cmd/cli/podvolume/backup.go b/pkg/cmd/cli/podvolume/backup.go index 537caa3cb..ed2d6c09a 100644 --- a/pkg/cmd/cli/podvolume/backup.go +++ b/pkg/cmd/cli/podvolume/backup.go @@ -24,7 +24,7 @@ import ( "github.com/pkg/errors" "github.com/sirupsen/logrus" "github.com/spf13/cobra" - v1 "k8s.io/api/core/v1" + corev1api "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/runtime" "k8s.io/client-go/kubernetes" @@ -128,7 +128,7 @@ func newPodVolumeBackup(logger logrus.FieldLogger, factory client.Factory, confi return nil, errors.Wrap(err, "error to add velero v1 scheme") } - if err := v1.AddToScheme(scheme); err != nil { + if err := corev1api.AddToScheme(scheme); err != nil { cancelFunc() return nil, errors.Wrap(err, "error to add core v1 scheme") } @@ -139,7 +139,7 @@ func newPodVolumeBackup(logger logrus.FieldLogger, factory client.Factory, confi cacheOption := ctlcache.Options{ Scheme: scheme, ByObject: map[ctlclient.Object]ctlcache.ByObject{ - &v1.Pod{}: { + &corev1api.Pod{}: { Field: fields.Set{"spec.nodeName": nodeName}.AsSelector(), }, &velerov1api.PodVolumeBackup{}: { diff --git a/pkg/podvolume/backup_micro_service.go b/pkg/podvolume/backup_micro_service.go index 275b57fb9..3c9cbab61 100644 --- a/pkg/podvolume/backup_micro_service.go +++ b/pkg/podvolume/backup_micro_service.go @@ -300,7 +300,7 @@ func (r *BackupMicroService) closeDataPath(ctx context.Context, duName string) { } func (r *BackupMicroService) cancelPodVolumeBackup(pvb *velerov1api.PodVolumeBackup) { - r.logger.WithField("pvb", pvb.Name).Info("PVB is being canceled") + r.logger.WithField("PVB", pvb.Name).Info("PVB is being canceled") r.eventRecorder.Event(pvb, false, datapath.EventReasonCancelling, "Canceling for PVB %s", pvb.Name) diff --git a/pkg/podvolume/util_test.go b/pkg/podvolume/util_test.go index 395b45a3f..bfd33e55e 100644 --- a/pkg/podvolume/util_test.go +++ b/pkg/podvolume/util_test.go @@ -300,3 +300,34 @@ func TestVolumeHasNonRestorableSource(t *testing.T) { }) } } + +func TestGetRealSource(t *testing.T) { + testCases := []struct { + name string + pvb *velerov1api.PodVolumeBackup + expected string + }{ + { + name: "pvb with empty annotation", + pvb: builder.ForPodVolumeBackup("fake-ns", "fake-name").PodNamespace("fake-pod-ns").PodName("fake-pod-name").Volume("fake-volume").Result(), + expected: "fake-pod-ns/fake-pod-name/fake-volume", + }, + { + name: "pvb without pvc name annotation", + pvb: builder.ForPodVolumeBackup("fake-ns", "fake-name").PodNamespace("fake-pod-ns").PodName("fake-pod-name").Volume("fake-volume").Annotations(map[string]string{}).Result(), + expected: "fake-pod-ns/fake-pod-name/fake-volume", + }, + { + name: "pvb with pvc name annotation", + pvb: builder.ForPodVolumeBackup("fake-ns", "fake-name").PodNamespace("fake-pod-ns").PodName("fake-pod-name").Volume("fake-volume").Annotations(map[string]string{"velero.io/pvc-name": "fake-pvc-name"}).Result(), + expected: "fake-pod-ns/fake-pod-name/fake-pvc-name", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + actual := GetRealSource(tc.pvb) + assert.Equal(t, tc.expected, actual) + }) + } +} From 829e75e9b7bc18d79c68544d9696707ac536f394 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Wed, 4 Jun 2025 13:44:10 +0800 Subject: [PATCH 17/31] issue 8960: implement PodVolume exposer for PVB/PVR Signed-off-by: Lyndon-Li --- pkg/exposer/pod_volume.go | 6 ------ 1 file changed, 6 deletions(-) diff --git a/pkg/exposer/pod_volume.go b/pkg/exposer/pod_volume.go index 6a62705d6..b402ffecb 100644 --- a/pkg/exposer/pod_volume.go +++ b/pkg/exposer/pod_volume.go @@ -152,12 +152,6 @@ func (e *podVolumeExposer) Expose(ctx context.Context, ownerObject corev1api.Obj return errors.Wrapf(err, "error to create hosting pod") } - defer func() { - if err != nil { - kube.DeletePodIfAny(ctx, e.kubeClient.CoreV1(), hostingPod.Name, hostingPod.Namespace, curLog) - } - }() - curLog.WithField("pod name", hostingPod.Name).Info("Hosting pod is created") return nil From d795f41a472c4648495b4c0fcb6d8f8a63959e47 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Fri, 6 Jun 2025 12:42:10 +0800 Subject: [PATCH 18/31] vgdp ms pvr data path Signed-off-by: Lyndon-Li --- changelogs/unreleased/9005-Lyndon-Li | 1 + .../v1/bases/velero.io_podvolumerestores.yaml | 5 + config/crd/v1/crds/crds.go | 2 +- pkg/cmd/cli/podvolume/podvolume.go | 1 + pkg/cmd/cli/podvolume/restore.go | 17 +- ...ro_service.go => restore_micro_service.go} | 111 +++-- pkg/podvolume/restore_micro_service_test.go | 470 ++++++++++++++++++ 7 files changed, 546 insertions(+), 61 deletions(-) create mode 100644 changelogs/unreleased/9005-Lyndon-Li rename pkg/podvolume/{pvr_micro_service.go => restore_micro_service.go} (85%) create mode 100644 pkg/podvolume/restore_micro_service_test.go diff --git a/changelogs/unreleased/9005-Lyndon-Li b/changelogs/unreleased/9005-Lyndon-Li new file mode 100644 index 000000000..8a40d62cb --- /dev/null +++ b/changelogs/unreleased/9005-Lyndon-Li @@ -0,0 +1 @@ +Fix issue #8988, add data path for VGDP ms PVR \ No newline at end of file diff --git a/config/crd/v1/bases/velero.io_podvolumerestores.yaml b/config/crd/v1/bases/velero.io_podvolumerestores.yaml index 888ac1669..4c4b1b91d 100644 --- a/config/crd/v1/bases/velero.io_podvolumerestores.yaml +++ b/config/crd/v1/bases/velero.io_podvolumerestores.yaml @@ -77,6 +77,11 @@ spec: BackupStorageLocation is the name of the backup storage location where the backup repository is stored. type: string + cancel: + description: |- + Cancel indicates request to cancel the ongoing PodVolumeRestore. It can be set + when the PodVolumeRestore is in InProgress phase + type: boolean pod: description: Pod is a reference to the pod containing the volume to be restored. diff --git a/config/crd/v1/crds/crds.go b/config/crd/v1/crds/crds.go index a412073f6..1157d1933 100644 --- a/config/crd/v1/crds/crds.go +++ b/config/crd/v1/crds/crds.go @@ -35,7 +35,7 @@ var rawCRDs = [][]byte{ []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcVMo\x1b7\x10\xbd\xebW\f\xd0kwU\xa3hQ\xec\xadqr0\xda\x06\x82\x1d\xe4N\x91#-c.\xc9\xce\f\xe5\xba\x1f\xff\xbd \xb9+K\xab\x95\x93\\\xb27\x91Ù\xc7\xf7f\x1e\xd54\xcdJE\xfb\x11\x89m\xf0\x1d\xa8h\xf1/A\x9f\x7fq\xfb\xf8\v\xb76\xac\x0f7\xabG\xebM\a\xb7\x89%\f\xf7\xc8!\x91Ʒ\xb8\xb3ފ\r~5\xa0(\xa3Du+\x00\xe5}\x10\x95\x979\xff\x04\xd0\xc1\v\x05琚=\xfa\xf61mq\x9b\xac3H%\xf9T\xfa\xf0C{\xf3s\xfb\xd3\n\xc0\xab\x01;0\xe8Pp\xab\xf4c\x8a\x84\x7f&d\xe1\xf6\x80\x0e)\xb46\xac8\xa2\xce\xf9\xf7\x14R\xec\xe0e\xa3\x9e\x1fkW\xdcoK\xaa7%\xd5}MUv\x9de\xf9\xedZ\xc4\xefv\x8c\x8a.\x91rˀJ\x00[\xbfON\xd1b\xc8\n\x80u\x88\xd8\xc1\xfb\f+*\x8df\x050^\xbb\xc0l@\x19S\x88TnC\xd6\v\xd2mpi\x98\bl\xc0 k\xb2Q\nQ\x1fz,W\x84\xb0\x03\xe9\x11j9\x90\x00[\x1c\x11\x98r\x0e\xe0\x13\a\xbfQ\xd2w\xd0f\xbe\xda\x1a\x9a\x81\x8c\x01\x95\xea7\xf3ey\u0380Y\xc8\xfa\xfd5\b,J\x12O J]\x1b<\xd0\t\xbf\xe7\x00J|\x1b{\xc5\xe7\xd5\x1f\xcaƵ\xca5\xe6pS\x99\xd6=\x0e\xaa\x1bcCD\xff\xeb\xe6\xee\xe3\x8f\x0fg\xcbp\x8euAZ\xb0\fjB\x9a\x89\xab\xacA\xf0\b\x81`\b4\xb1\xca\xed1i\xa4\x10\x91\xc4N\xadU\xbf\x93\xe19Y\x9dA\xf8\xb79\xdb\x03Ȩ\xeb)0y\x8a\x90\v\x89cS\xa0\x19/Zɵ\f\x84\x91\x90\xd1\u05f9\xca\xcb\xcaC\xd8~B-\xed,\xf5\x03RN\x03܇\xe4L\x1e\xbe\x03\x92\x00\xa1\x0e{o\xff>\xe6\xe6|\xef\\\xd4))\x94\xe4\xb6\xf3\xca\xc1A\xb9\x84߃\xf2f\x96yP\xcf@\x98kB\xf2'\xf9\xca\x01\x9e\xe3\xf8#\x93h\xfd.tЋD\xee\xd6뽕\xc9Rt\x18\x86\xe4\xad<\xaf\x8b;\xd8m\x92@\xbc6x@\xb7f\xbbo\x14\xe9\xde\njI\x84k\x15mS.⋭\xb4\x83\xf9\x8eF\x13Ⳳ\x17\xddS\xbf\xe2\x02_!O\xf6\x84\xda#5U\xbd\xe2\x8b\ny)Sw\xff\xee\xe1\x03LH\xaaRU\x94\x97\xd0\v^&}2\x9b\xd6\xef\x90\xea\xb9\x1d\x85\xa1\xe4Dob\xb0^\xca\x0f\xed,z\x01N\xdb\xc1\nO\x1d\x9b\xa5\x9b\xa7\xbd-\xb6\x9b\x1d E\xa3\x04\xcd<\xe0\xceí\x1a\xd0\xdd*\xc6o\xacUV\x85\x9b,\xc2\x17\xa9u\xfa\x98̃+\xbd'\x1b\xd33pEڅ\xe1\x7f\x88\xa8\xb3\xb8\x99\xdf|\xda\ueb2ec\xb5\v\x04O\xbd\xd5\xfd4\xfc3\x9a\x8eFq\xce߲1\xe4\xef\xc5n\xe7;W/\x0fEdK8k\xd8\x06.\xbc\xfbu^\x8a\xa9~%3\xd5\xd1Gnt\"*\xcdw\xf4y\xb5t\xe8K\xb9@\xa2@\x17\xab3P\xefJP\xf9Ǡ\xacgP\xfey<\b\xd2+\x81'\xa4\r\x97\x95\x1ax\x8fO\v\xabw~CaO\xc8\xf3\x96ϛ\x9b\xca\x1e\xce߃WXZlʋE\xceVhNXd\t\xa4\xf6\xa7\xbcr\xda\x1e\x9d\xbe\x83\x7f\xfe[\xfd\x1f\x00\x00\xff\xff\xbeM\x1a\xea\xb1\n\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcWMo\xe36\x10\xbd\xfbW\f\xd0K\v\xac\xe4\x06E\x8b·\xd6\xd9C\xb0\xe96\x88\xb7\xb9S\xd4HbC\x91,9t6E\x7f|1\xa4\xe4\x0fYv\x9c\xcb\xea\xe6\xe1p\xf8\xe6\xcd\xcc#]\x14\xc5B8\xf5\x84>(kV \x9c¯\x84\x86\x7f\x85\xf2\xf9\xd7P*\xbb\xdc\xde,\x9e\x95\xa9W\xb0\x8e\x81l\xff\x88\xc1F/\xf1\x16\x1be\x14)k\x16=\x92\xa8\x05\x89\xd5\x02@\x18cI\xb09\xf0O\x00i\ry\xab5\xfa\xa2ES>\xc7\n\xab\xa8t\x8d>\x05\x1f\x8f\xde\xfeX\xde\xfcR\xfe\xbc\x000\xa2\xc7\x15\xd4\xf6\xc5h+j\x8f\xffD\f\x14\xca-j\xf4\xb6Tv\x11\x1cJ\x8e\xddz\x1b\xdd\n\xf6\vy\xefpn\xc6|;\x84y\xccaҊV\x81>ͭޫ\xc1\xc3\xe9\xe8\x85>\x05\x91\x16\x832m\xd4\u009f,/\x00\x82\xb4\x0eW\xf0\x99a8!\xb1^\x00\f)&XŐ\xdd\xf6&\x87\x92\x1d\xf6\"\xe3\x05\xb0\x0e\xcdo\x0fwO?m\x8e\xcc\x005\x06镣D\xd4\x7f\xc5\xce\x0e\xd3\x04@\x05\x100\xc0\x01\xb2;\x84 \f\bO\xaa\x11\x92\xa0\xf1\xb6\x87J\xc8\xe7\xe8\xc0V\x7f\xa3$\bd\xbdh\xf1\x03\x84(;\x10\x1c%;\x1c\x9c\xa5m\v\x8d\xd2X\xeel\xce[\x87\x9e\xd4Hy\xfe\x0e\x1a\xea\xc0z)\v\xfe8\xf1\xbc\vj\xee,\f@\x1d\x8e\xe4a=p\x05\xb6\x01\xeaT\x00\x8f\xcec@\x93{\x8d\xcd\xc2\fٔ\x93\xd0\x1b\xf4\x1c\x06Bg\xa3\xae\xb9!\xb7\xe8\t\x1aE\xaf\xcb41\xaa\x8ad}XָE\xbd\f\xaa-\x84\x97\x9d\"\x94\x14=.\x85SEJĤQ+\xfb\xfa;?\ff8:\x96^\xb9!\x03yeڃ\x854\x1d\xef(\x0f\xcfK\xee\xae\x1c*\xa7\xb8\xaf\x02\x9b\x98\xbaǏ\x9b/0\"ɕ\x1aZl\xe7z\xc2\xcbX\x1ffS\x99\x06}ޗڔc\xa2\xa9\x9dU\x86\xd2\x0f\xa9\x15\x1a\x82\x10\xab^Q\x18{\x9dK7\r\xbbNR\x04\x15Bt\xb5 \xac\xa7\x0ew\x06֢G\xbd\x16\x01\xbfq\xad\xb8*\xa1\xe0\"\\U\xadC\x81\x9d:gz\x0f\x16Fy&j^\x01\x128\xe1[\xa4\xa9u\x82\xe5Kr\xe2\xe3_:q,X\xdfcٖ\xac9a\x00\x92\xf5\xe8\x87i\xa1.a\x80\xd9F\x9fE2\xf67\xd3\xc0\xbc\xb2\xa0\xb0\xd8\x1db:=\x9a?4\xb1\x9f?\xa0\x80\xdf\x13\xe6{\xdb^\\_[C<\x17\x17\x9d\x9e\xac\x8e=n\x8cp\xa1\xb3o\xf8\xde\x11\xf6\x7f:\xf4\xf9\x1a\xbe\xe8:\xde滫\xef\x82c\xd4g\xcf}D\xbeA\xf0|\xa6\x83\xc3UQ\xae\xc04x^\x95\xe8zs\xf7\x1e\nϸ\xbf\xa3Hw\xa6\xb1o\xa4\xb8w\x9c\xf5;#\x03\xe3\x97\xde\x10o\xf74\xbfBƞ\xe6-\xf9\xeeD\xf8\x14+\xf4\x06\t\xc3^\xa9_\x14u\xb3\x11\x01^:%\xbb\xb41\r\x04_\x02!X\xa9\xe6$\xf5\n\xf8\xac#\xca\xe3\xccP\x16iXg\xcc\f\xfe\xc4|F\xfd\xce\x1dP\f\x8at\x95\x82\x92\xa0\x18ޡ\xa1\xc9\x7f\xa4ZF\xef\xd3\x15\x95\xad\xfc2\x99n\xb8VDG\xe5\xf9\xeb\xf1\xfe\r%\xbd\xdd{\xa6\x17\xb7P&\xa3q\x1e\x8b\xa0Z~A\xf1\x1akiҸS2\xf2w\xfc\xc2;&j\xb6\xa2\xf8թ<\x80o@\xfc\xb8ŝ\x8f&\xdf\xf3\xd37l\n\x88\x81\x9f[ \x85\x99\xc1X!Ԩ\x91\xb0\x86\xea5\xdf\\\xaf\x81\xb0?\xc5\xddX\xdf\vZ\x01\xdf\xff\x05\xa9\x9962QkQi\\\x01\xf9x\xae\xcbf\x13w\x9d\b3cx\x94\xf3\x03\xfb\xcc5\xc6n\x18/v\x06\x9c\xbd_\n\xf8\x8c/3\xd6\ao%\x86\x80\xa7ct6\x93\xd9!81\x06~\xa4\xd5\a,\r\x7f\x19\x06\xcb\xff\x01\x00\x00\xff\xffx\xae@\xbaJ\x0e\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Z_s\x1b\xb7\x11\x7fק\xd8Q\x1e\x92\xcc\xf8\xc8\xdam3\x1d\xbe\xd9r\xd3Q\x9b\xa8\x1aS\xf6K&\x0f\xcbÒ\x87\xe8\x0e@\x01\x1ce6\xcdw\xef,p \xef\x8e )҉}/6\xf1g\xf1\xc3\xfe߅\x8a\xa2\xb8B#?\x90uR\xab\x19\xa0\x91\xf4ѓ\xe2_n\xf2\xf877\x91z\xba~y\xf5(\x95\x98\xc1M\xeb\xbcnޑӭ-\xe9--\xa5\x92^juՐG\x81\x1egW\x00\xa8\x94\xf6\xc8Î\x7f\x02\x94Zy\xab\xeb\x9al\xb1\"5yl\x17\xb4he-\xc8\x06\xe2\xe9\xe8\xf5\x9f&/\xbf\x9b\xfc\xf5\n@aC30Z\xacu\xdd6\xb4\xc0\xf2\xb15n\xb2\xa6\x9a\xac\x9eH}\xe5\f\x95L{eukf\xb0\x9b\x88{\xbbs#\xe6{->\x042o\x02\x990SK\xe7\xff\x95\x9b\xfdA:\x1fV\x98\xba\xb5X\xef\x83\b\x93N\xaaU[\xa3ݛ\xbe\x02p\xa564\x83;\x86a\xb0$q\x05\xd0]1\xc0*\x00\x85\bL\xc3\xfa\xdeJ\xe5\xc9\xde0\x85Ĭ\x02\x04\xb9\xd2J\xe3\x03S\ued40\b\x10\"Bp\x1e}\xeb\xc0\xb5e\x05\xe8\xe0\x8e\x9e\xa6\xb7\xea\xde\xea\x95%\x17\xe1\x01\xfcⴺG_\xcd`\x12\x97OL\x85\x8e\xba\xd9\xc8\xdey\x98\xe8\x86\xfc\x86A;o\xa5Z\xe5`<Ȇ\xe0\xa9\"\x05\xbe\x92\x0e\xe2m\xe1\t\x1dñ>\xdc2\x7fp\x98\xe7\xed\xcecc\x06\bn,\xe1nk\x84 \xd0S\x0e\xc0\x96\x9f\xa0\x97\xe0+b\xce\a\xc5B\xa9\xa4Z\x85\xa1(\t\xf0\x1a\x16\x14 \x92\x80\xd6d\x90\x19*'F\x8b\x89JD\a\xb0\xeeF\xa3\xa7x\xc3\xeb\x7foT\x03@\xf7Z\\\x00\xe5\xacs\xe3\xe2\xc1\xa9\x1f\xfaC'\xf5\xa3\xa2\xb0&\x1dޚZ\xa3 \xcb\xc7W\xa8DM,Y\x04oQ\xb9%\xd9\x030Ҷ\x87\x8d\x19\x82y\x9f\xe8\xf5f\xceaFg;s\xaf-\xae\b~\xd0epP\xacҖ\x06:\xed*\xdd\xd6\x02\x16\xe9\x14\x00\xe7\xb5\xcd*8#\x8e\xbb:\xba\x89\xec\xc8Άg\x1eFߣ\x9d\xfc\xe9\xa4d\x1b\x91Z\xe5-\xe8\xf5\x8a\xf2\xd6\x13\xa7\xd7/\xa3\xbb*+jp֭Ԇ\xd4\xeb\xfb\xdb\x0f\x7f\x9e\x0f\x86\x01\x8cՆ\xac\x97\xc9}Ư\x17\x1cz\xa30d\xf5\xff\x8a\xc1\x1c\x00\x1f\x10w\x81\xe0(A.\xead\x1c#\xd1a\x8a\xe2\x91\x0e,\x19K\x8eT\x8c\x1b<\x8c\n\xf4\xe2\x17*\xfddDzN\x96\xc9$A\x95Z\xad\xc9z\xb0Tꕒ\xff\xdd\xd2v\xac{|h\x8d\x9e\x9c\x87\xe0j\x15ְƺ\xa5\x17\x80J\x8c(7\xb8\x01K|&\xb4\xaaG/lpc\x1c?jK \xd5RϠ\xf2\u07b8\xd9t\xba\x92>\x85\xccR7M\xab\xa4\xdfLC\xf4\x93\x8b\xd6k릂\xd6TO\x9d\\\x15h\xcbJz*}ki\x8aF\x16\xe1\"*\x84\xcdI#\xbe\xb2]\x90u\x83c\xf7\xb4&~!ҝ!\x1e\x8e} \x1d`G*^q'\x85\xe4\xbb\xde\xfd}\xfe\x00\tI\x94T\x14\xcan\xe9\x1e_\x92|\x98\x9bR-\xd9\a\xf0\xbe\xa5\xd5M\xa0IJ\x18-\x95\x0f?\xcaZ\x92\xf2\xe0\xdaE#=\xab\xc1\x7fZr\x9eE7&{\x13\xd2\n\xf6e\xada5\x17\xe3\x05\xb7\nn\xb0\xa1\xfa\x06\x1d}fY\xb1T\\\xc1Bx\x96\xb4\xfa\xc9\xd2xqdoo\"\xa5:\aD;\xca_\xe6\x86J\x16,\xf3\x96wʥ\xec<\xddR[\xc0\xf1\xf2!\x9f\xf2\x0e\x80\xbf\xac\x97\x1b/:\xa5t\xfc\xbd\xc9\x11J\x80U\xcfa'o\xdc9\xcfz\xe8<\xfb_r\xe1\xdb=\x96\x8cv\xd2k\xbba\xc2\xd1{\x8f\x15\xe2\xa0l\xf8+Q\x95T_r\xbd\x9b\xb0\x13\xa4\x12\xccv\xda*4\xbb\xa2H5\x00\xd5j\xa5\xd9\xc4\xc6Ҁ[\xcf\xcbX\xc9\x1d\xf9\xfc]U\xa00\xda\xc9\x17\x95\nvy \xf4\xf3\xbd\xf1\xa5\x17Zׄc^*-\xe8ĝﴠ\x9c\xb0x+\xf8\n}\xc2Ƌl\xab\xd4>o\xf9\xd3\xea,q\x18-N\xe0\xeaND\xb0\xb4$K\xaa\xa4\xe4\xfb\x8f\xe5c\x19d\xfdLi\x1f\xe3a\xfb\x80#\x812\x8b\xf8\xf5\xfdm\n\x86\x89\x89\x1d\xf6\xbdxw\x92?\xfc-%\xd5\"\xe4\x0e\xa7\xcf\xcej.\x7f\xb7\xcb\b\"D\x04\xaf\x01\xc1H\x8a\x19\xf76\x1a\x83T\xce\x13\x8an\x90\x9d\xa0\xa5n\xeeE\xf4\xf4\aA\xf2\xb7\x8b\xda,\x13@\x8e\xb2%\xc7+\xb4\x04\xb5|\xcc\xd8O\xfc\xaeC\xae\xb8\x83\xf9+[\xcfo\xd7\xf0Mt^\xd7\xfc\xf3:\xc2ئ-}\x03\xdb\xc1\x89Vf\xe5jE\xbb\xa4tOY8\xccr\x80\xfa\x16\xb4\xe5\xbb*\xdd#\x11\b\xb3\x9cb| \xb1\a\xef\xa7W?_\xc37C\x1e\x1c8J*A\x1f\xe1\x15{\x9f\xc0\x1b\xa3ŷ\x13x\bz\xb0Q\x1e?\xf2Ie\xa5\x1d)Ъ\xdeĂ`M\xe04W\x94T\xd7EL\x10\x05<\xe1\x06\xf4\xf2\xc09ID\xac\x9a\b\x06\xad?\x9a$v|8n4\xfbYS\xfa\x9eg/!\x8bz\x96\xf5~\xb1\f䙜\b\xe5\xc2'p\xa2_j]\xc0\x89\xc7vAV\x91\xa7\xc0\f\xa1K\xc7|(\xc9x7\xd5k\xb2kIO\xd3'm\x1f\xa5Z\x15\xac\x8cE\x94\xba\x9b\x86\n~\xfaU\xf8\xe7ҋ\x87Z\xffSo?\xe8M|~\x16\xf0\xe9nz\t\aRv\xff\xfc\xd8u\x90\x0f\xf3.\xe1\x1c\xd3d\x9b\x7f\xaadY\xa5Z\xaf\xe7m\x1b\x14\xd1\x1d\xa3\xda|!\xdba>\xb7\x96\x11m\x8a\xaeUY\xa0\x12\xfc\x7f'\x9d\xe7\xf1K\x18\xdb\xcaOr.\xefo\xdf~I\x8bj\xe5%\x9e\xe4@\r\x13\xbf\x8f\xc5\x0eUѠ)\xe2j\xf4\xba\x91\xe5h5\xe7\U00037085\xb4\x94dO\xa4\x7f\xef\x06\x8bS\x82\x9a\xa9\x06\xb6k\xce\xca?=\xae2\t_\xbf\x8b{,-<ʯӪ\xf0\x80+\ah\t\x10\x1a4\xac\x11\x8f\xb4)b\xc6aPr\xba\xc0\x19\xc1\xb6k\x05hL\xcd1=f\x11\x19\x8a]\xfe۱\a]\xb8\xdf!\x86dE\x99\xbats\xf2^\xaa/Ȝ\xf7# \xbf/\xa3\xb6=\xccR\xab\xa5\\\xb56T\xa0\xfb\x9cRm]㢦\x19x\xdb\x1e\xaa\xb9\x8e2\xf2\x81\x97\x1c\xbf\xff\xfb\xdeҤ\xe1'\x1a\xae\xf9[\rڰ\xfb\x97!\xd56\xfbP\nx\xd4Fbfܒ\xf3{\xd6\xcb\x13\xd7\xd7\xe7\xd8XT\xcaKJ\xee\xeeq$S\x95v\x8a\xde%\xf0\xa92\xed\xf7óB?\xc37pu\xcf\xe5\xc8\x10w\x91o\x97\x8c\xd6p\xcd<\x1a2Z\x8cF\x86np49\xe8\xd9\xf7\x91\xee\xf7\x90\xc2S\xcc\x19]\xa4\xf8\xc4\xd4\xf14\x06G\x9f\x1e\x9e8\xed\xbe\xb4\x8fTj.\xbf\x06\xfd\xec\x8b\xda,\xfbdB\xff\u05ca\xce0d\xc3~\xa0\xf7J\xd5\x1d\x9ck\x04\xf5\xc9ŝ!Gaj$B\x19\xc5U\xde\x12eM\x02\xd2S\xe4\x99T\x16\xb4\xe4\x18\x1d\x8d45\":x\x87\v\x98\x87\x8a\xc0\x85n\xea\xd7nK\xb3u$B3/Ä\xfd\x88\xbdԶA\x1f\x1f\x06\n&q\x99\xf7\xca\xdalC\xce\xe1\xea\x94\xd1\xfe\x18W\xc5\xfeL\xb7\x05p\xa1[\xbfm\xd0\f\"\xd2\u05eeS\xb4\xf3zD\xd9\xd6\xc7P\xc7\xd1WI\xa5\x97m]\x87=}\xef\xb0{\xa6\x0e\xa8\x16\x94\xcf\xeb\x8e4\x88\x8e\x01\xacНb\xd5=\xaf\xc9Y\xdd֥\x1d5;8\xe2\xbe\xef\xe8)3\xba\xf7nܟ\xbcI&\x93\x99\xfb>X\xc3Y\xf7\xef\x0e\xba\xc4ܷM\xcdJ\xd7\xc9µ\xc7\x1aT\xdb,\xc82s\x16\x1bOn\xe4\xf8Q\x89>'s\xd5\xdfn\x7f\x12j\xa4\xd4u0\xba^l09\xafAHgj\xdcl\xef\x12rn\xb6\xaf|cz\xa7\xe4\xc9\xd2\r\x1d\xca!\x8e\xb7\x16\x03\xa6\xb7Z\x1d\xa8R\x93\x91K\xe5\xbf\xfbˑ\xa4]*O\xabQ\x18\xe9晝o\xf8\x94?\xe6\x84#9\x90Sh\\\xa5\xfd\xed\xdb\x13\xaa1\xdf.L&\xb2\xcb\xe7\x83C\fo\x1eݢN\x152Pw\x0e\xe7,\xfb\x1d\xfe\x19\xc3%Z<\x1fP8\x11\xaf\xba\xbf\xaa\xc8E\x859\x19\xb4\xec\x13\u008b\xda\xcd\xf8}\xf8\x058\x19\x1a\xe0\x9c\xed\xc6\xf4\xb7\xacP\xad\xb2\xfd\x11\xadB\x02\xa7\xed\xfe\xf3&\x9c\f@\xc3\v}\xceؓU\xa7\xbd\xc1\x80\\\xf4hw\x8fI\xfd\x91v\xb1}g\x9d\xc1\xaf\xbf]\xfd?\x00\x00\xff\xff\f\x19\xe2z\x0f%\x00\x00"), - []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Y_\x93۶\x11\x7fקع<$\x991\xa5\xdam3\x1d\xbd\xd9\xe7\xa6smr\xbd\xb1\xce~\xc9\xe4aE\xacHD$\x80\x00\xa0d5\xcdw\xef,@R\xfc'\xe9tnl\xbe\xdc\t\x00\x17?\xfc\xf6/\x96I\x92\xcc\xd0\xc8\x0fd\x9d\xd4j\th$}\xf4\xa4\xf8\x97\x9bo\xff\xe6\xe6R/v/g[\xa9\xc4\x12n+\xe7u\xf9\x8e\x9c\xaelJoi#\x95\xf4R\xabYI\x1e\x05z\\\xce\x00P)푇\x1d\xff\x04H\xb5\xf2V\x17\x05\xd9$#5\xdfVkZW\xb2\x10d\x83\xf0f\xebݟ\xe6/\xbf\x9b\xffu\x06\xa0\xb0\xa4%\x18-v\xba\xa8J\xb2伶\xe4\xe6;*\xc8\xea\xb9\xd43g(e\xe1\x99ՕY\xc2q\"\xbe\\o\x1cA?h\xf1!\xc8y\x17儩B:\xff\xaf\xc9\xe9\x1f\xa4\xf3a\x89)*\x8b\xc5\x04\x8e0\xeb\xa4ʪ\x02\xedx~\x06\xe0Rmh\t\xf7\f\xc5`Jb\x06P\x9f3@K\x00\x85\b\xcca\xf1`\xa5\xf2doYD\xc3X\x02\x82\\j\xa5\xf1\x81\x99V\x0e\xe8\r\xf8\x9cx\xcb\xc0*J%U\x16\x86\"\x04\xf0\x1a\xd6\x045\x12\x11\x84\x01\xfc\xe2\xb4z@\x9f/a\xce\xc4͍\x16s\xd5Ȭ\xd7D\xce\xef\a\xa3\xfe\xc0\xe7p\xdeJ\x95\x9dB\xf6\x7f\x06\xd5\xc3\xf3\xa0\xc5\x13\x91<\xe6\x14\xd64h*Sh\x14dy\xf3\x1c\x95(\b\xd8@\xc1[TnC\xf6\x04\x8a\xe6\xb5ǃ\xe9#y\xdf\xc8\xeb\xcc\\\xc3\xce5Tĵ\xbd\xed?t\x87.\xed\xfb\xa0E\xfd\x02\xd4F\rΣ\xaf\x1c\xb8*\xcd\x01\x1d\xdc\xd3~q\xa7\x1e\xac\xce,97\x01#,\x9f\x9b\x1c]\x1f\xc7*L\xfc\xb186ږ\xe8\x97 \x95\xff\xee/\xa7\xb1\xd5/ͽ\xf6X\xbc9xr=\xa4\x8f\xc3ሖ\x9d-\xab\xd5\xffE\xe0\xae\x19\xd2[\xad\xfa\xbc\xbe\x19\x8cN\x81\xed\bm\xe2\xed<\xb5\x14B\xed\xa3,\xc9y,MO\xea\xeb\xac/O\xa0\x8f\x03qz\xf72\x86\xb24\xa7\x12\x97\xf5JmH\xbd~\xb8\xfb\xf0\xe7Uo\x18\xc0Xm\xc8z\xd9D\xd7\xf8t\x92Gg\x14\xfa\xcc\xfe7\xe9\xcd\x01\xf0\x06\xf1-\x10\x9cE\xc8E'\x89c$jL\xd1y\xa4\x03Kƒ#\x15\xf3\n\x0f\xa3\x02\xbd\xfe\x85R?\x1f\x88^\x91e1\xe0r]\x15!\"\xed\xc8z\xb0\x94\xeaL\xc9\xff\xb4\xb2\x1d\xfb\"oZ\xa0'\xe7\x03\xd7Va\x01;,*z\x01\xa8\xc4@r\x89\a\xb0\xc4{B\xa5:\xf2\xc2\vn\x88\xe3G\xb6\x1f\xa96z\t\xb9\xf7\xc6-\x17\x8bL\xfa&\xa5\xa6\xba,+%\xfda\x11\xb2\xa3\\W^[\xb7\x10\xb4\xa3b\xe1d\x96\xa0Ms\xe9)\xf5\x95\xa5\x05\x1a\x99\x84\x83\xa8\x90V\xe7\xa5\xf8\xca\xd6I\xd8\xf5\xb6\x1dyd|B\"\xbcB=\x9c\x19A:\xc0ZT<\xe2Q\vM|\x7f\xf7\xf7\xd5#4H\xa2\xa6\xa2R\x8eKG\xbc4\xfaa6\xa5\xdap\x84\xe6\xf76V\x97A&)a\xb4T>\xfcH\vIʃ\xab֥\xf4l\x06\xbfV\xe4<\xabn(\xf66\x94\x1d\x1c\\+\xc3f.\x86\v\xee\x14\xdcbI\xc5-:\xfa̺b\xad\xb8\x84\x95\xf0$mu\x8b\xa9\xe1\xe2Hog\xa2\xa9\x84N\xa8vXݬ\f\xa5\xacY&\x97_\x95\x1b\x99F\x9f\xdah\v8Z\xdfgj:\x04\xf0\xb3\xc6t[\x99\x95\xd7\x163\xfaAG\x99\xc3E\x97̎\x9f7S\x82\x1aĪ\x93P\xe3\x8e\xe0\xe2J(\xea\xa5\x13\"\xf79Y\xea\xbec\xc9h'\xbd\xb6\a\x16\x1cS\xf1\xd0$Nj'\xf0\xa0Ņ\xb3q.\t\x0ediC\x96TJM\xb89W&M\x80\xefT\vc\x88\xa7\xf5\x01gB\xf3$\xe0\xd7\x0fwM\xf8m\x18\xae\xa1\x8f\"\xecEz\xf8\xd9H*D\xc8V\x97\xf7\x9e4\x04~\xee6\x11D\x88A^\x03\x82\x91\x14\xcb\xe06\xfe\x83T\xce\x13\x8az\x90\xdd\xceR=\xf7\"Ɩ\x93 \xf99\xe6\tV\t \xc7:)\xe0\x9f\xab\x7f\xdf/\xfe\xa1\xe39\x00Ӕ\x9c\v\xe5\x00\x95\xa4\xfc\x8b\xb6$\x10\xe4\xa4%\xc1u\x11\xcdKTrC\xce\xcfkid\xddO\xaf~\x9e\xe6\x0f\xe0{m\x81>bi\nz\x012rކ\xcf\xc6j\xa4\x8b\ao%\xc2^\xfa<\x005Z\xd4\a܇#x\xdc\x12\xe8\xfa\b\x15A!\xb74\xcd>\xc0M(4\x8f0\x7fc\xd7\xfa\xfd\x06\xbe\x89\xcer\xc3?o\"\x8c6Qv\xbd\xef\b\xc7\xe7\xe8\xc1[\x99et\xachG\xc6\u0081\x9dCⷠ-\x9fU鎈 \x98\xf5\x14\x03\x12\x89\x11\xbc\x9f^\xfd|\x03\xdf\xf498\xb1\x95T\x82>\xc2+\x90*rc\xb4\xf8v\x0e\x8f\xc1\x0e\x0e\xca\xe3G\xde)͵#\x05Z\x15\x87xA\xd8\x118]\x12\xec\xa9(\x92X\x92\b\xd8\xe3\x01\xf4\xe6\xc4>\x8d\x8a\xd84\x11\fZ\x7f\xb6,\xa9y8\xef4\xe3<\xdd?\x05\xbc\xbb[<\x87\x81\xa6\x9e|z\xee:\xc9ê\xaep\x862\xd9\xe7\xf7\xb9L\xf3\xe6vщ\xb6%\x8a\x18\x8eQ\x1d\xbe\x90\xef0ϕeD\x87\xa4n\x9e%\xa8\x04\xff\xef\xa4\xf3<\xfe\x1cb+\xf9I\xc1\xe5\xfd\xdd\xdb/\xe9Q\x95|N$9Q5\xc7\xe7crD\x95\x94h\x92\xb8\x1a\xbd.e:X\xcd5\xe3\x9d`%m$\xd9\v\xd5\u07fb\xde\xe2\xa6z\x9d\xa8>\xdb5W\x95\x9fN\xa1q\xb9\xf6wo/\xe0X\xb5\v\x1b\fG\x1d\xd6Eg#kЙ\xba\x0eO\xf0\xad\xfbӑ\xab\x0f\xaa\xbf\xbaA\xa6\xad\xcc$߿\xdb\xf0\x11\xae$\nK\xecv$\xbbO\x89\xc6H\x95]\x85\xb5i\xf0\xad\xc8\xf35v\xa2p\xee\xb6fϕ\xd7g\xed\xee\xb2K\xbd\x1f\x00\x01\xb4\x04\xc8gb\rm\xe9\x90\xc4*Π\xe4\x12\x8c\xab\xac\xbaT]\x13\xa01\x05\xd7I\xb12\x9b\xf2\xf5\xa6]\x99j\xb5\x91Ye\xc3\xe5h̔\xaa\x8a\x02\xd7\x05-\xc1\xdbj,\xe8\x8c\xfbt;\xa5\x174\xfe\xbe\xb3\xb4Q\xf7\x85^\xed\xf4\xa9z\x1d\xdc\xf1aHU\xe5\x18J\x02[m$N\x8c\xb3\xb1\x8f\x1c\x9d'nn\xae1\xa9\xe8I\x178\xa8\x1b\x8b\x13\x17\xd9\xda\x11벞G\xf8\xf2\x18\xdcq:;^렖~\xad\xf8\x8e\xd2G\x98L\xdf\xd9\ak\x8c\x16\xb3!i\xdd\xd86\x980\xf5\x9f\xf1ע\xc1|\xfb-\xec\x8f\xd9\xe1L\x99\xe039\xba>\x8eeX\xf8cq\xac\xb5-\xd1/@*\xff\xdd_\x8ec\xab7ͼ\xf6X\xbc\xd9{r=\xa4O\xc3鈖\x9d-\xab\xd5\xffE\xe0\xae\x18\xd2[\xad\xfar}3\x98\x9d\x02\xdba\xda\xc4\xdbYj)\x84\xda'Y\x92\xf3X\x9a\x1e\xd7\xd7Y\x9f\x9f@\x1f'\xe2\xf2\xf6e\feiN%.jJmH\xbd~\xbc\xff\xf0\xe7eo\x1a\xc0Xm\xc8z\xd9D\xd78:ɣ3\v}\xc9\xfe/\xe9\xad\x01\xf0\x01q\x17\b\xce\"䢓\xc49\x125\xa6\xe8<ҁ%cɑ\x8ay\x85\xa7Q\x81^\xfdB\xa9\x9f\rX/\xc92\x1bp\xb9\xae\x8a\x10\x91\xb6d=XJu\xa6\xe4\x7f[ގ}\x91\x0f-Г\xf3A\xd6Va\x01[,*z\x01\xa8Ās\x89{\xb0\xc4gB\xa5:\xfc\xc2\x067\xc4\xf1#ۏTk\xbd\x80\xdc{\xe3\x16\xf3y&}\x93RS]\x96\x95\x92~?\x0f\xd9Q\xae*\xaf\xad\x9b\v\xdaR1w2KЦ\xb9\xf4\x94\xfa\xca\xd2\x1c\x8dL\xc2ETH\xab\xb3R|e\xeb$\xeczǎ<2\x8e\x90\b/P\x0fgF\x90\x0e\xb0f\x15\xafx\xd0B\x13\xdf\xdf\xfd}\xf9\x04\r\x92\xa8\xa9\xa8\x94\x03\xe9H.\x8d~X\x9aR\xad9B\xf3\xbe\xb5\xd5e\xe0IJ\x18-\x95\x0f?\xd2B\x92\xf2\xe0\xaaU)=\x9b\xc1\x7f*r\x9eU7d{\x17\xca\x0e\x0e\xae\x95a3\x17C\x82{\x05wXRq\x87\x8e>\xb3\xaeX+.a%\xca,]\xf9L\x9a\x05\x8f\xb5\xa4B\x84,}\xfe\xecI\v\xe1q\xbf\x8e B\xec\xf5\x1a\x10\x8c\xa4X\xfe\xb7y\x0f\xa4r\x9ePԓ\x1cn,\xd5k/bL=\n\x92\xc7!?\xb2J\x009\xc6K\x01\xff\\\xfe\xfba\xfe\x0f\x1d\xef\x01\x98\xa6l)\\\xc3PIʿhK!ANZ\x12\\\x0fҬD%\xd7\xe4\xfc\xac\xe6F\xd6\xfd\xf4\xea\xe7i\xf9\x01|\xaf-\xd0G,MA/@F\x99\xb7i\xa3\xb1\x1a\xe9\xe2\xc5[\x8e\xb0\x93>\x0f@\x8d\x16\xf5\x05w\xe1\n\x1e7\xec1\xf1\n\x15A!74-}\x80\xdbP`\x1f`\xfe\xca!\xe5\xb7[\xf8&\x06\x89[\xfey\x1ba\xb4\x05B7\xea\x1c\xe0\xf8\x1c=x+\xb3\x8c\x0e\x95\xfc\xc8X8\xa1q*\xf8\x16\xb4\xe5\xbb*\xdda\x11\x18\xb3\x9eb &1\x82\xf7ӫ\x9fo\u16fe\f\x8e\x1c%\x95\xa0\x8f\xf0\x8a}<\xc8\xc6h\xf1\xed\f\x9e\x82\x1d\xec\x95Ǐ|R\x9akG\n\xb4*\xf6\xf1a\xb4%p\xba$\xd8QQ$\xb1\x14\x13\xb0\xc3=\xe8\xf5\x91s\x1a\x15\xb1i\"\x18\xb4\xfed9V\xcb\xe1\xb4ӌ\xeb\x93f<\xcf_B\xbd\xf2,\xef\xfdb\xb9\xfe\x99\x92\b\x85\xf9'H\xa2\xfb\xe4\xbcB\x12\x9bjEV\x91\xa7 \f\xa1S\xc7rH\xc9x7\xd7[\xb2[I\xbb\xf9NۍTY\xc2ƘD\xad\xbby\xe8'̿\n\x7f\xae\xbdx\xe8<|\xea\xed{\x8d\x92\xcf/\x02>\xddͯ\x91@SG??w\x1d\x95ò\xae\xec\x86<\xd9\xe7w\xb9L\xf3\xe6UՉ\xb6%\x8a\x18\x8eQ\xed\xbf\x90ﰜ+ˈ\xf6I\xdd4LP\t\xfe\xdfI\xe7y\xfe\x1a\xc1V\xf2\x93\x82\xcb\xfb\xfb\xb7_ң*yM$9\xf2Z\x88\xe3cr@\x95\x94h\x92H\x8d^\x972\x1dPs\xad|/XIkI\xf6L\xf5\xf7\xaeG\xdcT\xed\x13UwKsQ\xd9\xed\x14\x1a\x97k\x7f\xff\xf6\f\x8eeK\xd8`8\xe8\xb0.:\x1b^\x83\x8e\xdcex\x82o=\x1c\x8f\\}P}\xea\x06\x99\xb62\x93\n\x8bC\x04\fO1\x85%v;\xb1\xddQ\xa21Re\x17am\x1a\x9bK\xf2\xfc|\x9f(\x9c\xbb-\xe9S\xe5\xf5I\xbb;\xefR\xef\a@\x00-\x01\xf2\x9dXC\x1b\xda'\xb1\x8a3(\xb9\x04\xe3*\xab.UW\x04hL\xc1uR\xac̦|\xbdiӦZ\xadeV\xd9\xf0(\x1cKJUE\x81\xab\x82\x16\xe0mu\xec\x114\xe9>\xdd\x0e\xf1\x19\x8d\xbf\xef\x906\xea>ӣ\x9e\xbeU\xafs=\xbe\f\xa9\xaa\x1cCI`\xa3\x8dĉy6\xf6\x91\xa3\xf3\xc2\xed\xed%&\x15=\xe9\x8c\f\xea\x86\xea\xc4\x03\xbevĺ\xac\xaf\x9f\xac\xd1\x1d\xa7\xb3\xe3\xa5\x0e\xcaOk~\xa3\xf4\x11&ӽ\x8a\x01\x8d\xd1\xe2f(\xb4nl\x1b,\x1e\"\xd3p\xa1\xef\xf4\x83\xd5^\xa3\xbf{\x9bq\x9b't\x91/i\xf4\xc4\xceu-\xf7\x98V}\xd3\xcf\xe6\x82\xfd\xeaVO\xaa\xf9\xe5\xd6k:_\xd5\t\x19\xb3\tMZ+jG\x91%\x85>B\xec^\xec\xd05'O\x19A\x97_\xdc\x1a\xea\x1bfG\"<\xc1\xf8\x85\xb8FY\x90\x80\xf6\x83\xe2\x04\x9b\xa7\x9c\xc0\x85\x16\xe5\u05eeeT9\x12!*O\x80\x1e'\xe7惀@O\t\xb3\xb8.\xfaL\xfa\\I\xceav\xce\xe9~\x8cT\xb1\x13So\x01\\\xe9ʷ\xad\x98\xda\xfbjQ|\xedjӸ(\xa7\x84\x06ҹ\x86\x10\xd3L\x99a\x1b\aN\xdb!\x9c\x88o\x0f\xb4\x9b\x98\x1d}\xa0\xe9.\xde5&4\xb1\xf6}\xb0\x8e\x8b\x04P\x1ft\x8d\xfd\xb7m\xb8\\\x17\x8d\xc9k\xcfu@U\xaeȲt§\xa2FLm\xc1\x82Jt\x859\xf5\x96j94Q3\xb2\xaa\xdb\x01u\xff0\x18\xb5\xd7 \xa43\x05\xee\xdb˄\x02\x96-x\xba\x9bz0\xa3\xc659P\x1cI\xb3\xa7\x1bu\xed\xa7\xb0\xe9\xf2|\xea\xc3Z\x7f\x8c\xbf\x92\r\xd6\xdbo\x80\x7f\xcc\t'\xca\x04\xe7\xd1\xfaO\n\x90\xcb\x1e\x87s\xb11\x9c7\x1d\x19O\x87\xb4\xfe1\x9f3\x9aMJo4\x19\x90\x8b\x0e\xef\xba\xe3ߝ\xa9V\xed\xe7\xb0\x05\xfc\xfa\xdb\xcd\xff\x03\x00\x00\xff\xff\xfa\x1dp/\xd6\"\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xdc=[s\xdb8w\xef\xf9\x15\x98\xf4a\xdb\x19\xcbi\xa6\x97\xe9\xf8\xcd\xf5:\x8d\xfb}\xebx\xec4\xfb\f\x91G\">\x83\x00\x17\x00\xa5h\xdb\xfe\xf7\x0e\x0e.$%\x90\x84d˛-^2\xa6\x80\x03\xe0\xdc\xcf\xc1\x01\xb2X,\xdeц}\x03\xa5\x99\x14W\x846\f\xbe\x1b\x10\xf6/}\xf9\xfco\xfa\x92\xc9\x0f\x9b\x8f\uf799(\xaf\xc8M\xab\x8d\xac\x1fA\xcbV\x15\xf03\xac\x98`\x86I\xf1\xae\x06CKj\xe8\xd5;B\xa8\x10\xd2P\xfbY\xdb?\t)\xa40Jr\x0ej\xb1\x06q\xf9\xdc.a\xd92^\x82B\xe0a\xea\xcd?^~\xfc\xd7\xcb\x7fyG\x88\xa05\\\x11\x05\xdaH\x05\xfar\x03\x1c\x94\xbcd\xf2\x9dn\xa0\xb00\xd7J\xb6\xcd\x15\xe9~pc\xfc|n\xad\x8fn8~\xe1L\x9b\xbf\xf4\xbf\xfe\x95i\x83\xbf4\xbcU\x94w\x93\xe1G\xcdĺ\xe5T\xc5\xcf\xef\bхl\xe0\x8a\xdc\xdbi\x1aZ@\xf9\x8e\x10\xbft\x9cv\xe1W\xbd\xf9\xe8@\x14\x15\xd4ԭ\x87\x10ـ\xb8~\xb8\xfb\xf6OO\x83τ\x94\xa0\v\xc5\x1a\x83\b\xf8\x9fE\xfcN\xc2B\tӄ\x92o\xb8Q\xbb\x1aD<1\x155DA\xa3@\x830\x9a\x98\n\bm\x1a\xce\n\xc4;\x91\xab\x1e\xa40J\x93\x95\x92u\amI\x8b\xe7\xb6!F\x12J\fUk0\xe4/\xed\x12\x94\x00\x03\x9a\x14\xbc\xd5\x06\xd4e\x04\xd4(ـ2,`ٵ\x1e\xef\xf4\xbeNm\xcc6\x8b\v7\x8a\x94\x96\x89\xc0m\xc1\xe3\x13J\x8f>\"W\xc4TLw[\r\xdb#T\x10\xb9\xfc\x1b\x14\xe6r\x0f\xf4\x13(\v\x86\xe8J\xb6\xbc\xb4\xbc\xb7\x01e\x91Uȵ`\xbfG\xd8\xdan\xdcNʩ\x01m\b\x13\x06\x94\xa0\x9cl(o\xe1\x82PQ\xeeA\xae\xe9\x8e(\xb0s\x92V\xf4\xe0\xe1\x00\xbd\xbf\x8e_\x90xb%\xafHeL\xa3\xaf>|X3\x13$\xaa\x90u\xdd\nfv\x1fP8ز5R\xe9\x0f%l\x80\x7f\xd0l\xbd\xa0\xaa\xa8\x98\x81´\n>І-p#\x02\xa5\xea\xb2.\xff.\x12u0\xad\xd9Y\x1e\xd5F1\xb1\xee\xfd\x80\x02q\x04y\xac\xa88\xc6s\xa0\xdc\x16;*\xd8O\x16u\x8f\xb7O_\xfbLɴ'J\x8f7\xc7\xe8c\xb1\xc9\xc4\n\x94\x1b\x87\xacia\x82(\x1bɄ\xc1?\n\xce@\x18\xa2\xdbe͌e\x83\xdfZЖ\xdf\xe5>\xd8\x1b\xd4:d\t\xa4mJj\xa0\xdc\xefp'\xc8\r\xad\x81\xdfP\roL+K\x15\xbd\xb0DȢV_\x97\xeewv\xe8\xed\xfd\x104\xe2\bi\xbd\x16yj\xa0\x18H\x9a\x1d\xc6VA]\xac\xa4\x1a(\x19;d\x88\xa3\xb4\xf0\xdb洈U\x8b\xfb\xbf\xccq\x99m\xff\x1eG[~\xb3+k\x05\xfb\xad\x05T\xa6N\xfc\xe1P_\xa9\x9ej\x1f6\xcbF\xfb\xd4\x1dE\xb4m\xf0\xbd\xe0m\te\xd4\xeb\a\x1b\xcc\xd9\xc6\xed\x01\x144z\x94\t+D\xd6\xfaؽ\x88\xeeWT\xe0T\x01\x11\xd2$\xe01\xe1\xe0\x11&\x10\x03I\x9a`G\x03ubœ[&D\xb4\x9c\xd3%\x87+bT{\x88F7\x96*Ew#\xd8\n\x1e\xc0\x8b\x90\x15\x81xU\xc3Y\x81$\x8f\n\x05\xf1\xf5\xe7E\x15\xd3VQ\x86]>HΊ\xdd\f\xben\x93\x83\x82\xb4z\xd9\xf5;$K\xa8\xe8\x86I\x95\x12\x03\xa9\xb0kϞwjZZ-\xe9\x81\xec۸\xcc\r'\x91UI\xf9<\xc7\x10\x9fm\x9f\xce:\x90\x02\x1dʸ\x15Omo\xbb\x97@\xe0;\x14\xadI,\x93\x90\xb2E\xd3$\x15i\xa46\xe3t\x1fW]\xa4\xef\x1c\xa5~\x9c`\x9a\x83\x9d%Y\xdd5\xaf\x84\x03Q-\x0e\x06\nY\n\xb0ۨ-Q\xbb\xbeJ\xb6\xae\xef(RȒj(\x89\x14\xa33#\xbb\xb4\x1c\xb4\x9f\xabD\xce\xe8\xf4\xd0E\xb7\x7f\xf4x\b\xa7K\xe0D\x03\x87\xc2Hu\x88\xcc\x1c\x94\xba\x96\xa3XGP\x99ЦC\t\xe860\x01\x92XN\xdfV\xac\xa8\x9c\x87a\xd9\x13\xe1\x90R\x82\xb6\xda\x04]\xe6\xdd\xd8&\xc9\x1c\xf9\xfd$Sڣk3b\xb5\x0f/\xa5Q\xba\x96\xa1\x86\xbb\x96Dm\xa7{\x0ft\x8b\xffn\xe4\xe4\xb6\xff\x7f\"6\x18\x93\x13\x98vB\xfe\t\xba\x9f\xd9<=ʷ\x18ၾ$w+\x02ucv\x17\x84\x99\xf0uN\x12(\xe7\xbd9\xfeĴ9\x9e\xe93I\x93#\x13g\"L\x9c\xe2OH\x174\x19O\xdebd\xd3\xe4\xaf\xfdQ\x17\x84\xad\"\xd2\xcb\v\xb2b܀\xda\xc3\xfeI\xaa>P\xe65\x90\x91c\xf5\b\xe6\tLQ\xdd~\xb7.\x8e\xee\x92`\x99x\xd9\x1f\xec|\xe3\x10A\f\xcd\xf3\f\\\x82\xf12SPc\x1cN\xbe\"6\xbb/\xe8T_\xdf\xff|\x18+\xef\xb7\f\xce;\xd8Ȍйv\xbd\xb7\xa3\xfe\xfa|T\x10~A\x1f(\x06U.\xe7rA(y\x86\x9ds]\xa8 \x96>4tΘ^\x01&\x7f\x90Ϟa\x87`\xd2ٜÖ\xcb\r\xae=C\xc2\xf5O\xb5\x01\x0e\xed\x9a|X\xec\xf0d? \"0\x86\xcfe\x03\u05fc($r'閩KB\v\xb8?a\x9bY\xacҟ\xa3\x9f\xfaD\x0e\xf8I;ZZ\x89\xa9\x98\xcfij@\x99\xc9%\xa8k\xdf(ge\x9c\xc8\xc9ȝ\xb8 \xf7\xd2\xd8\x7f0@\xd3\xc8(?K\xd0\xf7\xd2\xe0\x97\xb3`\xd4-\xfc\x9c\xf8t3\xa0\xa0\t\xa7\xe5-\xc2\xfa9?g\xd3,\xb7E\xdc3M\ue10dW\x1cJ2\xa7\xc2\xf4\xae\x9b\xceMT\xb7\x1a\xd3uB\x8a\x05\xda\xcc\xe4L\x1e\xdfR\r\xd0\xfd\xe2I\xfd\x84_\xad\xb1p\xbf\xb8$3\xa7\x05\x94!\xb2\xc4\xec'5\xb0fE\xe6|5\xa85\x90ƪ\xf0<\x8e\xc8T\xac~7DZO\x9e\xf5\xee\xb7\xef\x8b\xe7\x98/XX\x93\xb3\xf0\x10\x8c\xac3p\xe0uw9\xbf\x9f\x85\x95ٌ^\x81\x13f\xbb\x8e$Gǻ\xe6 \xe5\x05\xe8@+\x8e.\xce,uiY\xe2\x11\x1a\xe5\x0fGX\x94#x\xe1X\xd5\xd0[\xbb3\xc15m\xacZ\xf8okiQ\x9a\xfe\x974\x94)}I\xae\xf1\xa4\x8c\xc3\xe07\x9f\x87\xeb\x81ɘ\xb2\xb1SY\xfe\xd9Pnm\xbfU\xe0\x82\x00w\x9e\x80\\\x1d\xf8E\x17d[I\xed\xcc\xf6\x8a\x01\xc7\xf3\x8a\xf7ϰ{\x7fa\xa7\x9f\x9d\xb2\xafd\xde߉\xf7·8P\x18\xd1ᐂ\xef\xc8{\xfc\xed\xfdK\\\xa9LN\xcd\xec6`њ6y\x1c*\x92\xc9\xfa\xae\r8\xa6\x9f\x9b\xef\x92\xf2\xdeɞ\xdam\x16\x8b6R\x9b\xcf\xe9\xbc\xe1\xc8z\x1e\u0088\xa1g\x9cȱ\xcdF\f>\x8f\x16\xf5\xbdu\"W\x06\x94\xcf%:\x1b\x10\xe2\x8f\x17Ff\xa9S\x99\xfebc2\x90\xc6\xfc\xaeE\xf0\f7\xb9\x83\x9b\x9c%\x1e\xe3\xb0Z\xbc\x1c\xe9\xed\xdf~\xef\xe53\xad\xe4ڿ\xfb\x1bym\x87\xba\x90uM\xf7O5\xb3\x96z\xe3F\x06\x9e\xf6\x80\x1c\xf5պEyε\xc8\x1d\x0f\xe1\xf9喙\x8a\tB\x83\xda\x00\xe5\x19\x8a\x92F\xa6rةVQM\x96\x00\"\xa6\xe8\x7f\x04W\xa2f\xe2\x0e' \x1f\xcf\xe0zDt\x9d\xd3ٽ\x894\x89\x94\x8f\x1f\x9c\xc9jdI\xb6\x15(\x180\xc6a\xde\x1d=U!M/eq\x84C\xda\xc8\xf2'MVLi\xd3_\x82&\xadΥ\xf5\x91\xe4\xb3\xeb\xfe\xcaj\x90\xad9'\x82o\xbbi\x06g\xcd5\xfd\xce\xea\xb6&\xb4\x96\xad3\xe6\x86\xd5\xf1TףwK\x99\x89\xc7V\x98\xbf1Ғ\xa0\xe1`\x80,a\x95>\xefM\xb5B\n\xcdJP\xa1J\xc1\x91\x8dI+\x98+\xcax\x9b:%J\xb5c#`q\xab\xd4I\x01\xf0\x177\xb2\x97w\xac\xe4v\x88\xa0̽\xe3A\x1a\x10\xb6\"\xcc\x10\x10\x85\xc58(\xa7\x92q\n\x8f\fD\r\xcb\xd5sy\n\xdc6\x10m\x9d\x87\x80\x05\n$\x13\x93)\xb7~\xf7O\x94\xf1s\x90\xcdr\xde'\xa9\x1e\x81\x96\xa7\xe4h~\xed\r' t\xab\xf0\xf0\xdf\xe9\x8e-\xe3yk\xb6\x94#\x9c\xb6\xa2\xa8\x00\x95\x90\x18\xea\x06\a\x9e\tm\x80\xe6\xf2\x82\xf5\x8aZ!\x98X\xe7\xd1.;\x11\xda5\x87\ua954\x1c\xe8\xf8)d\xd7,\xae\xdf@\x13\xfd\xdaM\xf3BM\xd4\x11\xc1\x1d\x9b#\x1d\xb2)j\x95\x16\xa1\xc6@\xdd8\x91\x93D\xb5\xa2o]Π\x88\x8e\t\xc3\xfd*^3\xbef\x82e\xd0v@\xd7;\xc1L\xdfy\xb4 \xce\xea<\xda\t\xa2;pJ\x86\xedn\x00\xc0\nh\x88Cp\xed\x91k\x8ep$\x97@hYB\xe9r\x97\xd6\x15\xf1a\x89+|\x1b)nH\xee\xeexO0\x8b\xb2\xa1\r\x82N\xccê\r,Z\xf1,\xe4V,0\x18\xd7G\xeb\x90\x13\xb3T/\x9dޜ\xac\x8c\xe6\xf5K\xbe\x9a\x9e\xd3BC~\xcd\xe7\xa9\xe0?\x9dA\xcbd\xf3\xcdQ\t\x8f).\x98\xd3k\xae\x00{\xe4\xc7\xd9UL\xcd?1\xd8\x1fJ߸b\xe9\x17\x95\xc5ݥA\xf5\x9c\xc2m\x05\xa6\x02\x15J\xb3\x17X\x92^N\x9e\x90v\xc1K\xac\x93\xb3L\x15\\dW\xfe\xb9W9\x87\xd1M\xcb\xf9\x85\xe5m\xda\xf2d8l$\x8a\xd8!geՏ\xa5=\x86\x9c\xea\x8bl<\xf6+-\x86\xf5\x85\xb1\n\"\x14\x18\xca0\xb3\xa7qj\xbfXX\xda;\xdf\x1f\x96S`\xfe/,\xff\x0f/=̨\x94\xc8Gcn\x95fDb\x02V\x82\xc1zh\xec\xea+|?_\xe8\xfbc\xe1\xd4@\xfd\xa5\xf1\x123\xea\xc2f\xa05\x01g\xaf\xde\x04\xadA\xab\x9d+\x10\xed\x80\xcf\x19\xda\xf1ׅ\xbb\x05\x11\xc0\xa4\xf8\xf5k\x05A|}\xf5>\xd3\xe4\x9fI%\xdbDU\xdf\x04\xcaf\xaa;\xe67<(\xf4\xf0\a\n`\xe8\xe6\xe3\xe5\xf0\x17#}\xd9\af\xd1\x12\x800(\xea2\xb3L\x94l\xc3ʖ\xf2 \xb5\xdd\x1d\x02\xc7@\x1d\x9f%\xa0IE\x04\xe3\x8e\x01\xc3\xf8\x01Ñ/\x8d;\x969Z\xc5M\xfb\xa2y\xd5!'ׄ\fk>F\xac\xe1\xb1\xc7\x17\xafR\x05\xfb\x87\xd4z\x1c_\xe1\x91\x13I\xccTs\x9cPÑY,\xf6\xe2\xf3\x96\x9c*\x8dcb\xee\xb3Ud\xbc~\x1dF\x16~\xe6k.\x8e\xc1\xce\xd9\xeb+ް\xaa\xe2mj)2+(^\xaf\x142/\xfa<\xa9\x14`>`\x19\xaf\x82\x98\xad}xQ@sҖfk\x1a\x8e\xa9d\x98\xa5N\x9e\x98\xbdY\xad\u009bU(\xbcm]\xc2$\x17M\xfexL\xe5A\x8c\x93~\xa1M\xc3\xc4\xfa\x90)rYg\x92m\xe6Y\xe6~o!\x03\x9e\xe9\x873]t8\x12\xfa\xba\xeb҉H2\xa4-\x990\xf2\x92\\\x8b\x9d\x87\x9b\x80\xd3\v\x1f\x854\a\x17\xd9첶\x8c\xf3\xfem-\x04;\r\xcaߙԴv\xab\x1a\xf3\xf6\x93t\x95j\xe0\x94\x9f\x148~ك\xd1ώ\xbe\xa5\xe7_\xb7ܰ\x86\x83\xf5\xe86\xacL\xde!3\x15\xec\"\x92\xff&\xf1\x86\xd4r\x87\x90\xbe]\xf4\x9eQ\xec\x9eq\x926\xbf\xc9\x13\xb6\x97Q\xcc~\\\x11{\x06\xcdrE\xf1\r\x8b\xd5߰H\xfd\xad\x8b\xd3g8k\xe6\xe7\xe3\x8a\xd0O>\x81\tG\xfd\xf7\xb2\x84\a\xa9\xcc\\p\xf2\xb0\xdf?q\x92\xda\v\xd8$/\x89\b]\x13\xbb\xc4\x10Ç\x17\xa7m*}\xe8\x19\xdc\xe9_di\xd76w\xc6\xf2\xb8\xd7\xfd\xe0\xae\xf2\n\x14\b\xf7\xcc\xc7\x7f>}\xb9\x8f\xf0S>\xaf\xf7\x8c\xf7\x9e\x97p\x1eL\xe9\x91\xe3\x8f\xe6|1\x93\xc3\x16\xfa\x00\xaf|.B\x1b\xf6\x1f\xf8\xaa\xdb\v\xd2A\xd7\x0fw\b#\xf8i\xf8L\\\xac\xa2\x88'\x96K\xb0\x16+\xa2jT,\xeeV\x03\x88Ê\xdf\xfe3JP\xba'\xb3\x82\xc5d\xa1\xc6\xcb\n\xdeÝ[\xc7\xd8,\x9f\xac\xd3(vD:\x8e\xac\x98*\x17\rUf\x87l\xa3/\x06k\bff*\x9d3\xaaX\x0f\x9f\x01K\xa27\xbc\xfe\x85g\x91\xbbfxڻ\x8f\xbbS\xd61~\xffd\xf6\xe6\xc9+\xaec\xdcb/\x10S\x89\xcf\xc9\x02\x93WK\x93yM\xf4\xf0\xed\xa4\xb4\xcbc\x1c=\xad\xe7l\x14\x1dRM\t0v<\xaa:-h\xa3\xabēK/\xd3u\xf8\x1a\x99\xa1\xa6}\xc9&\x1d\x80\xc1>YQ\xf5\xb4\xd5\x16\x82>\v\xdbFi\xc5a)\xddnm\xb3\xab{a\xfc\xa2\x97)x\x9b#\xe1\xcc\xe7\\N~\xc8šgD\xfd`\xf6˪\xb6CL\x9dp\x18<\xeb\xdae\x14\x19O;\xb1\x99π\xe4\x19\x8c\x13\x9e\xfe@|\xe5\xe2\x8a$_\x04\xc9|\xf5\xe3\x0fE\xf4\x84V\xd3E\x05e\xcb\xe1\xd47\xff\x9ez\xe3\xe7_\xfd\v\xb3e\xbc\xfbg\x91\xdd3\xd0\xd6g\x1e\xbe/\xe8)\xe1!\xf7)9\xe6\xf0ap\xe0\x9e\x17+\xdcK\x94E\x01Z\xafZ\x1e\xaa\x94\n\x05\xd4@\x19\xba3\x1dW|T\x9dM\xdbpIKP7R\xacX\xe2\x84d\x80\xd6\xff\x1at\xde\xe3\xd9\x02?\xb6\xaa{\xdaq\xf2Y\xbc\x17i\xae\x86*\xca9\xf0O\x8c\x83\xfeYn\x85]W\x86@>\xa4\xc6\xf5\xeee\x15\xad\xb2f}GD[/\xad\x93\vƌ\a\x8b+\xa9\xa6+\xa4\x1dޙ0\xb0\x86T|\xbdU\xcc\xc0SC\x95\x06\\Q\xc6\x0e~\xdd\x1b\xe2\xa2\xcf\x15\xa7kW\nW\xb2\x82\x1a\x88\x06\x18g\x18[>\x8e\xd7\b\x8b\xef\xb02I\x8e$\xbd\xb2\x85z\xecJƨX\x8f=/\x9a0\xd5\xc9\aF\x9dE.hc\xf0\x02\f\xd2\x11\x89h<\f|\xb4w\xef\x8d\xd1\x01\xd8qN\xf3e̾`N\x1bZ'\xa2\x84y\xbdss\b\x06\x9f\x05Ve\xaf\xee\xae\xff\xc0b,\xb0#[\xaac1u\xd2\xf7\xee`;0\xe8\xaa[\xd0P\x12\u0600 V\x14)\xe3PNq\xeaWL$\xab\r\xa8\x9ft\x84\x83\x95\x80\x96ş\fU&.\xfdЏYIUSsEJj`aG\x9f溥\x9fIU\xea\xc4\xe3@\xbc\xd9\xe6ţ\b\xd7n\xac\xf5s\xf7\xd1jК\xaeC\x10\xba\x05\x05d\r\xc2\xe2=\xe6\x16\x93\x1eS\xb8\xd2\xe7\x8dE,-\xb5(\xa4\x85i\xa9\x9f\xc0\xb9p\xf1\xf44\xbcO\x8cQ\xeczTE\xa7U\x85\xbf<\xf8\bT\xef?w}\x80\x8bO\xfd\xbe>I\xecv\xec\xceF\xa8+\xf0\xc4\a\x8f\r\x8b\x91uJ\xa6\x8dę\x8f2'\x95\x94\xcfYn\xf6\xe7رK'1\xe1X\t\xafL.ekz~\x8eGxb\x99\xf8\xfc\xe7+\xdb\x17\x84y\xed.P\x8d\xe5V\xf3<\xbd\xcf\x03H1\xbc\x95\x86\xf2`d,_\xc6\x0e\xd5\xc4\x03\x02O\xe1\xf1d\xcew\x17\xfb\x90\xf7^e\xef`W\xddS\x9e^\x13t\xd7\xc7\xc7Ҫ>\xeb\x97\x04\x12_\x01\xed|\x92\xb17\x17\xe7\xec\x1fB\xfd\x84\x8b\xca\xc0\xf1\xe7\xae\xf7\x18\x1e\xdd2\x9d\xc3\f\"\x1di\x12\f>L\x15%ㄥOx\xa9ME\xf5\x9c{\xfa`\xfbD\xb7\xa3g\xae\xa2\x13\xfa8\"\x95\xe9{\xae\vr\x0f\xdb\xc4W\x87,<\xfdB\xa9Jt\xb9\x13\x0fJ\xae\x15\xe8C\xa6[\xe0}F&֟\xa4z\xe0횉/\xe3\x95\xdfS\x9d\x1f\xa82\xcc2\xad[Ob\xecM\xb0q\x89\xdf\xe6G\x8f\xff\xc0\x04\xe5\xec\xf7\x94.\xef\xff87Ä\xbek<\xf2N\xb1P\x01\xf1s\n\xd0k\xe8\x9ft\xcf\xfc\x84y/ɽL\x8a\xb1? fC\xa0L\x93%h\xb3\x80\xd5J*\xe3\xf2\xf7\x8b\x05a\xab\xe0 Y\r\x81q\xa2{͞\xb0T\xe2=\x1e\xbd\x05\x87e\xe5S\x89\n\xad\x0e\x86\x9c5ݹ\x8c$-\n\x1b\x13\xc0\amh*6y\x91\x9e\xc6P\xd5\xcbJ\x8e\n\xb9\xeb\xf7\x8f9\xbe\xa8>\x10\x9cC\x1d^gw\x06\x9d\x8f\x9di\r^\xcb \xdab\xef\x14eB\x9c\x1a\xbb\x1b\x0f\xbb\xf3L\xcd\xd7\beL=\xfa\xfd\r\x1e\xe2\xf6\a\xac\xbe\x93%[QQ\xb1\x1e\xbd\xd0V)ٮ\xab\xc0\x9bc\x0e\x11)[\x8c\x9c\x1bT\x05:\xfc\xc7!\xa6U\xa2wh\xe7k,ƴt\\\uee0f\xf2\x02E\xad\xba\x8b-\x9d\xaa\x9a\xb0\xf9\xd9Y\xc2\x11\x88\xb3\xb6?\x01\x91\xea\x9d(&\xaf\xe0\xf8@\x9bM\xdc՝\xc2P\x12\tQ\x1b\xbf\x1a\x12\"\xc41$\xf4}\x89.\xe2\xf9a02棜\x88\x8ei'\x06\xb78\rj~\xd3}'h\xe8\xee\x1c\x87\x0e=\b\xfeNJ\xbb\r \x1c\x13\xf9\xe2\xdc\xe9\xb8\xf7ǍX7\xd1ۺ=9v\xfd\xb6\ac\xef\n\xa4\x8db\xbbiB\xbc\xf9\xf7l\x95\x92\x17\xf7\xbf3-9\xfc\xc3\xc1\xafo|\x95qK\x95`b}\x12F~\xf5c\x13\xf1\xbc\a{Έ>\xac\xfc\xd5b\xfa\xa4Y:\xf8\x88\f^\xf6\xf0\xecg\xf2_\xfe/\x00\x00\xff\xffP\a\xb5\x16Cm\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xec=]s\x1c)\x92\xef\xfa\x15\x84\xeea?B\xdd^\xc7}ą\xde|\xb2gO\xb1\x1e[ai\xf4\xbctU\xb6\x9aQ\x15\xd4\x00\xd5r\xdf\xde\xfe\xf7\x8dL\xa0\xbe\xba\xe8\xa2Z-ygǼت\x86$\xc9L\xf2\x03\x12X,\x16g\xbc\x12\xf7\xa0\x8dP\xf2\x92\xf1J\xc0W\v\x12\xff2\xcb\xc7\xff6K\xa1\xdelߞ=\n\x99_\xb2\xab\xdaXU~\x01\xa3j\x9d\xc1{X\v)\xacP\xf2\xac\x04\xcbsn\xf9\xe5\x19c\\Je9~6\xf8'c\x99\x92V\xab\xa2\x00\xbdx\x00\xb9|\xacW\xb0\xaaE\x91\x83&\xe0\xa1\xebퟖo\xffk\xf9\x9fg\x8cI^\xc2%3\xd9\x06\xf2\xba\x00\xb3\xdcB\x01Z-\x85:3\x15d\b\xf4A\xab\xba\xbad\xed\x0f\xae\x91\xef\xd0!{\xeb\xdbӧB\x18\xfb\x97\xde\xe7\x8f\xc2X\xfa\xa9*j͋N\x7f\xf4\xd5\b\xf9P\x17\\\xb7\xdf\xcf\x183\x99\xaa\xe0\x92}®*\x9eA~Ƙǟ\xba^0\x9e\xe7D\x11^\xdch!-\xe8+U\xd4e\xa0Ă\xe5`2-*K#\xbe\xb5\xdcֆ\xa95\xb3\x1b\xe8\xf6\x83\xe5g\xa3\xe4\r\xb7\x9bK\xb64ToYm\xb8\t\xbf:\x129\x00\xfe\x93\xdd!n\xc6j!\x1f\xc6z{Ǯ\xb4\x92\f\xbeV\x1a\f\xa2\xccrb\xa0|`O\x1b\x90\xcc*\xa6kI\xa8\xfc\x0f\xcf\x1e\xebj\x04\x91\n\xb2\xe5\x00O\x8fI\xff\xe3\x14.w\x1b`\x057\x96YQ\x02\xe3\xbeC\xf6\xc4\r\xe1\xb0V\x9aٍ0\xd34A =l\x1d:\x1f\x87\x9f\x1dB9\xb7\xe0\xd1\xe9\x80\n»\xcc4\x90\xdcމ\x12\x8c\xe5e\x1f\xe6\xbb\aH\x00F$\xaaxmH8\xda\xd67\xddO\x0e\xc0J\xa9\x02\xb8\x80vX4\xb6\nu%\xa0\x80\xe6\f\xddN\x8d\x16FH\xb6\xae\xd1#]2\xd4\x12Q\x19\x11\xd2X\xe0\x11a>\x01\xef\xe0kV\xd49\xe4WEm,\xe8\xdbLU\x90\x87E\xa6Q͜\xca\xc3\x0f\a!\xfb\xf8\xa5\x10\x19 \x1f2WiA\x8b<1\xd1nC\x99]\x05n\xcd\tY\xed\x87\xd0\xc6(\x93\xbaŀņ\xe7\x7f<\xbf \t\xe8\xf7\xde\xef\xc70\xae\xa1!\xd3,\xddL\x16\x7f\xbc\x85\xb0PF\xa8;\xa9\xa3f\xf0\x9dk\xcdw\a\xb8\xde,\xa6\xbd\x00\xdfc\xb0\a\x9c\x97\xa1\xda7\xe2\xfd\xb0\xff\xdf\"\xf7O\xcboC\x8b\xce\\H\xe4s!\x8c\xed\xb1ٸU,$\xebX\b\xe9\t$\x1dLT\x93S\\\xfd'!\xe6I\xe7Nl\xb24\xb2\xe9'\xc0\xbf\x14%7J=\xa6P\xef\x7f\xb1^\xbb\x84\xc52\xda\x18a+\xd8\xf0\xadP\xda\f\x97I\xe1+d\xb5\x8dj\x16nY.\xd6k\xd0\b\x8b\x96\xf9\x9b]\x81C\xc4:\x1c\xbe\xb0\x8eʊV\x18\x8c\xabe:\xb2\x94\xa8\x11\x1b\n\x05\xa8Q\xa8\xce\xc1\xc1Ђ\x1c\x88\\lE^\xf3\x82|\t.37>\xde\xe0\x17\xd3j\x13\x02\xb1\x87\x7fT\xaa]q\x0eM\x18$2\xb1\xb7\xea\xa5$\xa0\x8f_bl\xb4_5N\x89\xb0\x94p\xb0od\xa6\xae\v0\xbe\xbb\x9c\xdc\xe4V']\xb4\xccrk\f\x05_A\xc1\f\x14\x90Y\xa5\xe3\x14J\x91\x03WR\x95n\x84\xb8#Z\xb6\x1fm\xb5\x83\x99\x00\xcb(\xc4݈l\xe3\xdcW\x144\x82\xc5r\x05\x86VExU\x15\x11\xd3ՖI\xe1\xf0\x9dM鍶$h\x90!ܘ.iK\xa2~n\xcb(\xd9۹٧\xfa\xf8:\xff(\xbe\xbf%\xa2\a\xabs\xa4\xb0Oh\x12F\xfb\x05\xc9\xf3!Jz\xa4\xb8\x00\xb3\xec\xac\xce\t\x1b\xbe\xa60\xb4\xe7?\xeem\xa5\xec\x11\xe5\xd7Ż\xe3&\xcc\f\xd6MΩ\x97e\\\xd3Ϳ\b\xdf\xc8d\xddz\x8b5\x8bg\x1f\xbb-/hW\xc03$\xbf`kQX \xa7j\nQ6\x83s\xa7$P\xaa\x05f\xb4Il\xb3͇f\xef(\xa1ŀVC\x00\xceA\x0fQ\x0e\xf1 \x01$k\\\v\xda4\x15\x1aJڌ\xa5H\xb2\xfb\x85\\\xc1w\x9f\xde\xc7c\xcfnI\x94ԽA%LZW\xde\r\x1c\xa3.\xae>T\t\xbf\x90\xbf\xd6\x04\x82n\x13\xfe\x82q\xf6\b;\xe7bqɐo)\x8b\xff|\xf8*\fv,s\xf6^\x81\xf9\xa4,}yQ*\xbbA\xbc\x06\x8d]O4A\xa5\xb3$H\xc4n\U00088ce5(\xa8\r?\x84a\xd7\x12C2G\xa2\x19\xddQ\xae\x90\xeb\xd2uVֆ\xb6Z\xa5\x92\v\xb7,6֛\xe7\x81\xd2=\x16\x9c\xa4c\xdf\xe9\x1d\x1a#\xf7\x8b\xcbZ*x\x06yآ\xa3t\x1an\xe1Ad3\xfa,A?\x00\xab\xd0,\xa4K\xcb\fE\xedG6_\xbc\xd2=\x87n\xf9\xbax\xacW\xa0%X0\v4k\v\x0fŪ2\x91.\xde&\x8c䜌\x95\x05\xce\xf5ĚAZ\x92\xaaG2r\x0eWO%\xd63\xc9D^\x04\xb9]IR\xd0Ml\x9dg\xbdf\xca\xcd1*\xa63\x16\xe7\x02\x94\x9c\xb6\xd6\xfe\x86\x96\x9ef\xe3\xdfYŅ6K\xf6\x8e2{\v\xe8\xfd\xe6\x17&;`\x12\xbb\xadh\x95\xfd\x97Zly\x81\xfe\a\x1a\bɠpވZ\xef\xf9j\x17\xeci\xa3\x8cs\x1b\x9aM\xbb\xf3Gع\x1d\xe5\xa4n\xbb\n\xeb\xfcZ\x9e;_fO\xf14\x8e\x8f\x92Ŏ\x9d\xd3o\xe7\xcfu\xeffH\xf4\x8c\xaa=Q.y\x95.ɔ7;'\xd0\xc0`=8DظI \xc5\x00a\x8a\x02ɢ\\)\x13I\x16\x89\xa0\x95 \xe87\xcaX\xb7\x0e\xd9\xf3\xf7G\x17*UX\x9cd|mA3c\x95\x0e)\x99\xa8\xf8S\x96\xe2\xbb\xe5n\x03\x06\xfc>\x94_\xf4t\x801\x8a=ou\x83\xb3*\xe7n/\x8c:\xe2\x19yOԶ\xd2*\x03\x13͋hK\xa2m\xeaQp\x9f\x0eͺ.w\xd1\xdf:Ik\xa7,J\x872ϑG\xd2\x1d\x11\x19}\xf8\xdaY\xa2F\xed\x82\x7f\xa7H\xeb182:\xafQ\x96|\x98\x0e\x9c\x8c\xee\x95k\x1d\xe6\x98\a\xe6\xc2-\xfdP\x93Ι\xe3u4\xa2\xfc\xcf\xe6ڔB^SG\xec\xed\v\xbaC^\x8b\xc7ң\xc6\xca\xf1N\xfaU\xe8\xac\xe5^\xf3\xc1\xe7\xd4)\xda\xf8\xd1\xd0c\xee\xfe\x9e\by\xd7R\xd9\xce2\xceL'\xbaR\xf9\xef\f[\vml\x17\rs \xb1j\x14\xd4\x11\xa1\xa7\xfc\xa0\xf5ё\xe7g\u05fa\xb3\xa0\xb8QO>qzN\xbc\x1dH\xba\xe1[\xf0\x99\xab 3UKZ\nC=\x80\xdd̀\xe8X\xe3\xac@\xa2\xbd\xeb4\x96u\x99N\x90\x05I\x92\x90\x93\xebf\xdd&?p\x91\xb6nŎc\xab=\x94\xc39V\x8e\x9fG!\xc1\xb3\x9bN_\U000af8acK\xc6K\xe4!\xb9\x1d\xa2\x84&\xa3ޱ\xbbI\xfb\xc4\x16d\xb4\xac\xc2YV\x15`\xc1\xa7m\xce\xc0#S҈\x1c\x1a\xd3\xefE@I\xc6ٚ\x8b\xa2\xd63\xb4\xeal\x92\xcf\r¼69}d\x95\x8eȂH\x94\xb8\xce>\xc3\v\x9e\xd6\xf8\x95\x9e\xe7Ǧ8\x8c\x1a\xe6\xfb\x8b\x95\x16\xca\x1d\x068\xbd\xcb\xe8ӎ\xb9\xdc}\xf7\x19\xbf\xfb\x8c\xdf}\xc69\x1d}\xf7\x19'\xcaw\x9f\xf1\xbb\xcfx\xb8|\xf7\x19S\xcaw\x9fq&\"\xdf\xcagL\xc1pAk\x9c\a*$a\x95\x98\n1\x85\xf6D_>\xe9ǟ\xd58I.\xf3\xf58ȑC<\x91\xe3\x171\xaf\xa35^Mr3\xce\xc00w\xdc)\xca\x04\x87\xf9\x04\xa7g\x02\x02\xa7?=s}\x10\xf2\tO\xcf\xf8!\xa4E\x18G\x9d\x9d\tD\x9a\x7fz\xe2\xc2'\x11\x95\xc0\xc3V\x8aK\xff\x88\x8d1&I\tx|\xe3\xe4\xf7\xbd\x8c\xc9\x17\x90\xa5W9\x913K\x9eFY\x7f\xfe\xc7\xf3_\a\x8bN˔(\x1b\xf6i\xeb\xd4xL?b,\xdfM\x8d\xecg\xa9\xfez\xa6\xc2Ie?\xf5DMC\xe4\b\xbc\xbeX\x0f\xa8\xfck\xd27\x16\xcaϕ\xb7\x96'8a\x7f=\x02/\xe9\x8c=7;\x99m\xb4\x92\xaa6~M\ba\xbd\xcbܽ\x03\x01dL\xd8G5\xc8\x7f\xb0\x8d\xaa#\xa76&H\x9b\x90E\x9bF\x90^R\xadO\x8c\x00˷o\x97\xfd_\xac\xf2)\xb6\xecI\xd8M\x04\x18\xddG\xc1\xf3\x1c\xe3\x82\u0381\x1e\xaf\a\xc2UIC\xa1\x8c\x00S\x9aIQ8\x89\r\x10z\xf2\xca>Wnu\xf0h\xbfiz\r+=\x11wn\xfam\x93-9\xed\xbe?#\xe9\xf6\xa4G\xa3\xbeYZ\xedqɴ\xa9+\x94\t\x89\xb3\xe9\xe9\xb2)lu%=I69BNM\x88\x9d\xbb\x02\xf1\xa2ɯ/\x93\xf2\x9aL\xb3\xb4\xf4ֹ\x14{\x95T\xd6WN`}\xbd\xb4\xd5\x19ɪ\xa7?\xf5\x92\xbe\x96~tveڲ\xcc\xe1\x84Ӥ4Ӥ\xa5\x9b\x94\x01\x1f5Ԥ\xf4ѹI\xa3I\x9cL\x9f\xae\xaf\x9a\x16\xfa\xaaɠ\xaf\x9f\x02:)m\x93\x15\xe6&y\x8e_r\x18ʴ\x03P|\v\xe1|.\x99\x94\xee\xb9\xe6ϊ;?\x0f`\xa1\xb0\x047\xf5\x15〲.\xac\xa8\x8a\xf6>\xb6X\xc0\xb9\x81]sY\xd1ϊ\x8e\xc8\xfb\x9b\xba>\x7fi$~9\x88j\xb8aOP\x14\x8c\xc7\xe6\xe6\x1e\x152w\x0fh\xa6\x16\x80\xb6\x11g\xb9\xbf\x8c\xc9_\x1ez\xe1\xa6\v\xdd\x06@\x16\xb6\x8c-\xf5qy\xf8\xa6\xaf\x83\x06,U\x8f\xedy\xe6.ޠo\xbfԠw\x8c\xee\x1dk|\xb3\xf6P\xa9\x9f\xe8\x06\x03Ӡ~\xbc:<\xb4g\xb2\x17\xe0\xb4ꁽ\x93\xce#\x18\xe2DmP\xef\xb4\x01\x1d*U\x8cӢ\xfdD@H\xd5@\x884Mq\xfe眲|\x89\xf0\xee\x14\x01^\x92\a4\xcf{\xfd\x86\xa7'\x8f=5\x99\x9e\x8c\x92tJ\xf2%½9\x01\xdf,\x7f5\xfd\x14\xe4\xfc\x8d\xe7\x17>\xf5\xf8R\xa7\x1dgP/\xf5t\xe3|ڽ\xd2i\xc6W?\xc5\xf8\x9a\xa7\x17g\x9dZLNϚ\x95q0'\xb5\xea\x19\xc7\xed\xd2r\t\xa6O!&\x9e>L\xcc4H\x1b\xfc\x91\xc3N<]8\xffTa\"\x7f\xe7L\xe9W>=\xf8ʧ\x06\xbf\xc5i\xc1\x04\tL\xa82\xffT\u0cf7\xa4\x94\xceAOn\xfb͑\xdaIyM\x8d\xe5\xfa\x88\r\xf6\xb5\xc2m\xb2X\xab\x17\x03\x90Y\xf2\x17\xf9ӣ\r\x87\xb6\xc1Q2;\x1eQo_\xb2u\xd7\xfa\x0e\xb1\x7f\xcd\xc1m]\x1a\xa88\x1a\x00\n\xdc(5+\xea*|\xe0\xd9f\xd0Æ\x1b\xb6V\xba䖝7\x9b\xc5o\\\a\xf8\xf7\xf9\x92\xb1\x1fT\x93\xabӽ/͈\xb2*v\x18\x89\xb1\xf3n\x83\xe7IIT:C\xcf7\xaa\x10Y\xc4\xe7\x1c\xbdW\xcf5ػl\x88n\xfe\xcb:\xd9\"\xb1\xc0\a\x9b\x8bp\xebb\xffJfw\x9f\xfb\x91k%\xbc\x12\x7f\xa6'\x95N\xb0\xea\xf6\xee\xe6\x9a`\x051\xa2\xb7\x9a\x9a\x04ņ\xe5+@\x97\xa1\x1d\xfb!}r\xbd\xeeA\xed\xe7\bw\x1f\xab\x80ܽL\x12\xdc\x16\xaf\x9a3\x85Z\xeb\xe6\xda\xe1r\xa8'\x94/.wL\xf9\xa7'\x84\xce\x17\x15\xd7v璉.zx\x04\xbb>\xb5jv\xd0Z\xed\xbf\xbc\xd2-=\xb2\x87GWh'{W\xf5\x93\a\x86\xf4|\x0eN\x87OUO\x9e\xa7~\x01\x9c\x0e\xbbP\v\xa2b\xe4\xa7h\x06\xe4\xc9W,\x8d\xbf\xa1\xffG\xb5\x85\xf7ѕ\xcb\xfe\xeb+\x83&#\xa9\x89\x01*]2\x1f\xa1`\x9b\x8fHw|?O\xed\xc5s\r\x03*\xfe\x8e\xf0\xe7,N\xde\xf6A\x8d?HB7\xa8\x87Nc^\x15=\xf5\xb4c7\xf7\x14\xb76\xaa\xd4O}\x1f\xb7\x86\xe5ɐ`\x10\x81%\xe4\xc17ZNEF\xab4\x7f\x80\x8fʽ\xad\x93\"&\xfd\x16\xbd\x97\x97\xbc\xe7\x16\xf2\xb5\xfd$\x8c)z?\xb6!\xc0\xf6|\xc6\xdeE\xff\x88\xed\x91O\x19X[\xba\x91ғ&\xef\xfd\xeb$\xa8\x8f\r \v\x02\x05\x1c\xb4\x15\xfew\xa3\x9e\xe8\x02\xfc\xf8\x1asx@\xa4\xf3\x86\x19\xd0A\x11J\xe1=j\x98uU(\x9e\x83\xbe\xa2GT\x12F\xfcS\xaf\xc1\xc0\x1d\xe8?\xc5\xe2\xedfd<\xa1\xe7\x17̒A\x8f\xae(\xa0\xf8A\x14`\x1c≦\xe1f\xbfec)\xear\xe5<\xd55\xfe\xd8tr\xc02\xbb\xa1\xd2\x06C\x05\x1a\xfdD\xb7\x15Q\x9b \xf9\x87\x89\xc1\x1a>\ni\xe1\x01\xc6c\xe8\t\x9b\xe0\xdeh \a (0\x8a\xf8\xfe\x12[y\xec\x11\xe4>\xdez \x03\xcdbdL\x8e\x95w\xabn\xee\xaf\f\xabeN\x1b\x00\xf7\x7f\xbe=J~\xb7\xbd\xf7e\x82NHQ\xef\xf7\xe3-;!BG;\x91O\x1fW\xe21X\xdc\x18\x95\t\x8a*\x9e\x84\xf5\xd79\xbe\xdc\x1d\xe2\x87\x02\xc4\x03\xd2Q\x1b\xf8\xfc$A\x7f\t\x16\xc8\\\xcbػ-\xd3\xda\xef\xa7=h\xd1\xf7Z\xac¾G`\f\x000\x15\xf6\xb9\x8c{\t(l\xaf\t\xd3H\x1c\xc4>\x01\xdcyG\xc8ik\x85\xb4\xe3\x9c!n\x9bVt\xd8tDCN\x8b\xed\xfd\x00\xc6 \x93\x9d\x1e}j\xaa\xb8Ӧ\x86\xfd^\x8cy\xa3\xb4c\x96\xe1@\xff\xb0\xf7kT\x83\x1f\xd4\xde1\xcd=\xaaF\xf6>\xd2CxyGr\xbc\x97\xde\xfdR\xaf\xda\a\x15\xd8\xdf\xfe~\xf6\x8f\x00\x00\x00\xff\xff)\x00\x87w>{\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcV\xcfo+5\x10\xbe\xe7\xaf\x18\x89+\xbb\xa1B \x94\x1b*\x1c*\xe0\xa9j\x9ezw\xbc\x93d\xa8\xd7^f\xc6)A\xfc\xf1\xc8\xf6n\x9b\xee:\xb4\x8f\x03\xbe\xad\xed\xf9\xe6\x9bo~x\x9b\xa6Y\x99\x81\x1e\x91\x85\x82߀\x19\b\xffT\xf4\xe9Kڧ\x1f\xa4\xa5\xb0>ݬ\x9e\xc8w\x1b\xb8\x8d\xa2\xa1\x7f@\t\x91-\xfe\x84{\xf2\xa4\x14\xfc\xaaG5\x9dQ\xb3Y\x01\x18\uf0da\xb4-\xe9\x13\xc0\x06\xaf\x1c\x9cCn\x0e\xe8ۧ\xb8\xc3]$\xd7!g\xf0\xc9\xf5\xe9\x9b\xf6\xe6\xfb\xf6\xbb\x15\x807=n@\x90ә\x1a\x8d\xc2\xf8GDQiO\xe8\x90CKa%\x03ڄ\x7f\xe0\x10\x87\r\xbc\x1e\x14\xfb\xd1w\xe1\xbd\xcdP\xdb\f\xf5P\xa0\xf2\xa9#\xd1_\xae\xdd\xf8\x95\xc6[\x83\x8bl\\\x9dP\xbe \xc7\xc0\xfa\xe9\xd5i\x03\"\\N\xc8\x1f\xa23\\5^\x01\x88\r\x03n \xdb\x0e\xc6b\xb7\x02\x18\x05\xc9Xͨ\xc5\xe9\xa6\xc0\xd9#\xf6\xa68\x01\b\x03\xfa\x1f\xef\xef\x1e\xbfݾ\xd9\x06\xe8P,ӠYֿ\x9b\x97}\xa8\x85\t$``\xa4\x04\x1a\xc0X\x8b\"`#3z\x85B\x19\xc8\xef\x03\xf79\xad`v!\xea\x05\xaa\x1e\x11\x1e\xb3\xfec\x98\xed\xcb\xe1\xc0a@V\x9a\xa4)\xeb\xa2\xe2.v\xff\x8dxZ)\xd6b\x05]*=\x94\xecy\xd4\v\xbbQ\x1e\b{\xd0#\t0\x0e\x8c\x82\xbe\x14c\xda6\x1e\xc2\xeew\xb4\xdaΠ\x8b.\x922\x19]\x97*\xf6\x84\xac\xc0h\xc3\xc1\xd3_/ؒ\x04JN\x9dѬ\x9dWdo\x1c\x9c\x8c\x8b\xf85\x18\xdf͐{s\x06\xc6\xe4\x13\xa2\xbf\xc0\xcb\x062\xe7\xf1[`\xccRo\xe0\xa8:\xc8f\xbd>\x90N}hC\xdfGOz^疢]\xd4\xc0\xb2\xee\xf0\x84n-th\f\xdb#)Z\x8d\x8ck3P\x93\x03\xf1\xb9\x17۾\xfb\x8a\xc7Ε7n\xf5\x9cjP\x94\xc9\x1f.\x0er\xeb|AzR#\x95b*P%\xc4\xd7,\xa4\xad$\xdd\xc3\xcf\xdb\xcf01)\x99*Iy\xbd\xba\xd0e\xcaOR\x93\xfc\x1e\xb9\xd8\xed9\xf4\x19\x13}7\x04\xf2\x9a?\xac\xa3\\\xb8qד\xcaT\xda)us\xd8\xdb<\xab`\x87\x10\x87\xce(v\xf3\vw\x1enM\x8f\xee\xd6\b\xfeϹJY\x91&%\xe1Cٺ\x9c\xc0\xf3\xcbEދ\x83iv^ImeJl\a\xb4)\xb9I\xdfdM{\xb2\xa5\xad\xf6\x81\xc1\xd4L\xda\x0f1\xc9\x16_\xc8e\x9cH\x85\xcdlN\xa5.\x7f\x9fM},哣\x11\x9co\xce8ݧ;s\xff\x8e\xf6h\xcf\xd6a\x81(S\bߧ\x92\x16\xfa\xd8/}6\xf0\t\x9f+\xbb\xf7\x1c҄\xc6\xf9\xa8\xb9Z\x1bP\x1e\xb1\x03\xf9E\xb8\xf3\xc8ʭ\xfc0.G~\x0eh\x04\x02\x8eާ\x96\x0e~\x01Yy\x11\x16wH\xb1\xaf\xb0\xa9\xf2\xb9\xf3\xfb\x90\xff\"Lrl\xb4\xb4\x13\x8e\xc9\x1e\xfd\x14^\x15\xc0\xeb\xb9.k9\xe7>$hY\xf9y\xfeo\xc6i.\x11c\xd5w\x93YU\x0f\x92ǚ\xe2\xf5\xfe\x1aYF\xe7\xcc\xce\xe1\x06\x94\xe3Һ\xd8\x1afs\x9eW\xcdTj\x9f\xa9GQ\xd3\x0f\xef\x14\xd0\xe2UH\xeb~\x81\x92\x9a\xe7\xf9\x88\xfeZ\x8b\xc0\xb3\x91W\xe7\x15\xc8\xdd\xf9\x9a\xe9\xed\xcb\xdf\xe6\xb2\xcfJ=o \xcd\xfaF\xa9\"䇔\xaa\xa6\xb4\xd4y\xf5\xb7f\xa1\xd2\xf6\xf2\xee4H\xde\xf4\xcb\xf4W\xb3\x8c\xe1*\x85j\x05,63|w\x11\x9eh`s\x98\x02\xfe'\x00\x00\xff\xff\xef\xf8\xa6>\x10\f\x00\x00"), diff --git a/pkg/cmd/cli/podvolume/podvolume.go b/pkg/cmd/cli/podvolume/podvolume.go index e7f347533..998e4683a 100644 --- a/pkg/cmd/cli/podvolume/podvolume.go +++ b/pkg/cmd/cli/podvolume/podvolume.go @@ -31,6 +31,7 @@ func NewCommand(f client.Factory) *cobra.Command { command.AddCommand( NewBackupCommand(f), + NewRestoreCommand(f), ) return command diff --git a/pkg/cmd/cli/podvolume/restore.go b/pkg/cmd/cli/podvolume/restore.go index 2894cf8c6..ba7a838d7 100644 --- a/pkg/cmd/cli/podvolume/restore.go +++ b/pkg/cmd/cli/podvolume/restore.go @@ -33,7 +33,6 @@ import ( "github.com/vmware-tanzu/velero/internal/credentials" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" - velerov2alpha1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v2alpha1" "github.com/vmware-tanzu/velero/pkg/buildinfo" "github.com/vmware-tanzu/velero/pkg/client" "github.com/vmware-tanzu/velero/pkg/cmd/util/signals" @@ -42,24 +41,24 @@ import ( "github.com/vmware-tanzu/velero/pkg/repository" "github.com/vmware-tanzu/velero/pkg/uploader" "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util/kube" "github.com/vmware-tanzu/velero/pkg/util/logging" ctlcache "sigs.k8s.io/controller-runtime/pkg/cache" ctlclient "sigs.k8s.io/controller-runtime/pkg/client" ) -type pvrConfig struct { +type podVolumeRestoreConfig struct { volumePath string pvrName string resourceTimeout time.Duration - winHPC bool } func NewRestoreCommand(f client.Factory) *cobra.Command { logLevelFlag := logging.LogLevelFlag(logrus.InfoLevel) formatFlag := logging.NewFormatFlag() - config := pvrConfig{} + config := podVolumeRestoreConfig{} command := &cobra.Command{ Use: "restore", @@ -76,7 +75,7 @@ func NewRestoreCommand(f client.Factory) *cobra.Command { f.SetBasename(fmt.Sprintf("%s-%s", c.Parent().Name(), c.Name())) s, err := newPodVolumeRestore(logger, f, config) if err != nil { - exitWithMessage(logger, false, "Failed to create pod volume restore, %v", err) + kube.ExitPodWithMessage(logger, false, "Failed to create pod volume restore, %v", err) } s.run() @@ -104,12 +103,12 @@ type podVolumeRestore struct { cache ctlcache.Cache namespace string nodeName string - config pvrConfig + config podVolumeRestoreConfig kubeClient kubernetes.Interface dataPathMgr *datapath.Manager } -func newPodVolumeRestore(logger logrus.FieldLogger, factory client.Factory, config pvrConfig) (*podVolumeRestore, error) { +func newPodVolumeRestore(logger logrus.FieldLogger, factory client.Factory, config podVolumeRestoreConfig) (*podVolumeRestore, error) { ctx, cancelFunc := context.WithCancel(context.Background()) clientConfig, err := factory.ClientConfig() @@ -233,6 +232,8 @@ func (s *podVolumeRestore) runDataPath() { return } + s.logger.Infof("Running data path service %s", s.config.pvrName) + result, err := dpService.RunCancelableDataPath(s.ctx) if err != nil { dpService.Shutdown() @@ -270,7 +271,7 @@ func (s *podVolumeRestore) createDataPathService() (dataPathService, error) { credGetter := &credentials.CredentialGetter{FromFile: credentialFileStore, FromSecret: credSecretStore} - pvrInformer, err := s.cache.GetInformer(s.ctx, &velerov2alpha1api.DataDownload{}) + pvrInformer, err := s.cache.GetInformer(s.ctx, &velerov1api.PodVolumeRestore{}) if err != nil { return nil, errors.Wrap(err, "error to get controller-runtime informer from manager") } diff --git a/pkg/podvolume/pvr_micro_service.go b/pkg/podvolume/restore_micro_service.go similarity index 85% rename from pkg/podvolume/pvr_micro_service.go rename to pkg/podvolume/restore_micro_service.go index 146c73992..f7614bf7c 100644 --- a/pkg/podvolume/pvr_micro_service.go +++ b/pkg/podvolume/restore_micro_service.go @@ -37,7 +37,6 @@ import ( "github.com/vmware-tanzu/velero/pkg/datapath" "github.com/vmware-tanzu/velero/pkg/repository" "github.com/vmware-tanzu/velero/pkg/uploader" - "github.com/vmware-tanzu/velero/pkg/util/boolptr" "github.com/vmware-tanzu/velero/pkg/util/kube" cachetool "k8s.io/client-go/tools/cache" @@ -61,14 +60,14 @@ type RestoreMicroService struct { resultSignal chan dataPathResult - ddInformer cache.Informer - pvrHandler cachetool.ResourceEventHandlerRegistration - nodeName string + pvrInformer cache.Informer + pvrHandler cachetool.ResourceEventHandlerRegistration + nodeName string } func NewRestoreMicroService(ctx context.Context, client client.Client, kubeClient kubernetes.Interface, pvrName string, namespace string, nodeName string, sourceTargetPath datapath.AccessPoint, dataPathMgr *datapath.Manager, repoEnsurer *repository.Ensurer, cred *credentials.CredentialGetter, - ddInformer cache.Informer, log logrus.FieldLogger) *RestoreMicroService { + pvrInformer cache.Informer, log logrus.FieldLogger) *RestoreMicroService { return &RestoreMicroService{ ctx: ctx, client: client, @@ -82,14 +81,14 @@ func NewRestoreMicroService(ctx context.Context, client client.Client, kubeClien sourceTargetPath: sourceTargetPath, nodeName: nodeName, resultSignal: make(chan dataPathResult), - ddInformer: ddInformer, + pvrInformer: pvrInformer, } } func (r *RestoreMicroService) Init() error { r.eventRecorder = kube.NewEventRecorder(r.kubeClient, r.client.Scheme(), r.pvrName, r.nodeName, r.logger) - handler, err := r.ddInformer.AddEventHandler( + handler, err := r.pvrInformer.AddEventHandler( cachetool.ResourceEventHandlerFuncs{ UpdateFunc: func(oldObj any, newObj any) { oldPvr := oldObj.(*velerov1api.PodVolumeRestore) @@ -165,7 +164,8 @@ func (r *RestoreMicroService) RunCancelableDataPath(ctx context.Context) (string return "", errors.Wrap(err, "error to create data path") } - log.Debug("Found volume path") + log.Debug("Async fs br created") + if err := fsRestore.Init(ctx, &datapath.FSBRInitParam{ BSLName: pvr.Spec.BackupStorageLocation, @@ -178,7 +178,8 @@ func (r *RestoreMicroService) RunCancelableDataPath(ctx context.Context) (string }); err != nil { return "", errors.Wrap(err, "error to initialize data path") } - log.Info("fs init") + + log.Info("Async fs br init") if err := fsRestore.StartRestore(pvr.Spec.SnapshotID, r.sourceTargetPath, pvr.Spec.UploaderSettings); err != nil { return "", errors.Wrap(err, "error starting data path restore") @@ -212,56 +213,26 @@ func (r *RestoreMicroService) Shutdown() { r.closeDataPath(r.ctx, r.pvrName) if r.pvrHandler != nil { - if err := r.ddInformer.RemoveEventHandler(r.pvrHandler); err != nil { + if err := r.pvrInformer.RemoveEventHandler(r.pvrHandler); err != nil { r.logger.WithError(err).Warn("Failed to remove pod handler") } } } +var funcWriteCompletionMark = writeCompletionMark + func (r *RestoreMicroService) OnPvrCompleted(ctx context.Context, namespace string, pvrName string, result datapath.Result) { log := r.logger.WithField("PVR", pvrName) - volumePath := result.Restore.Target.ByPath - if volumePath == "" { - r.recordPvrFailed(pvrName, "invalid restore target", errors.New("path is empty")) - return - } - - // Remove the .velero directory from the restored volume (it may contain done files from previous restores - // of this volume, which we don't want to carry over). If this fails for any reason, log and continue, since - // this is non-essential cleanup (the done files are named based on restore UID and the init container looks - // for the one specific to the restore being executed). - if err := os.RemoveAll(filepath.Join(volumePath, ".velero")); err != nil { - log.WithError(err).Warnf("error removing .velero directory from directory %s", volumePath) - } - - var restoreUID types.UID - for _, owner := range r.pvr.OwnerReferences { - if boolptr.IsSetToTrue(owner.Controller) { - restoreUID = owner.UID - break - } - } - - // Create the .velero directory within the volume dir so we can write a done file - // for this restore. - if err := os.MkdirAll(filepath.Join(volumePath, ".velero"), 0755); err != nil { - r.recordPvrFailed(pvrName, "error creating .velero directory for done file", err) - return - } - - // Write a done file with name= into the just-created .velero dir - // within the volume. The velero init container on the pod is waiting - // for this file to exist in each restored volume before completing. - if err := os.WriteFile(filepath.Join(volumePath, ".velero", string(restoreUID)), nil, 0644); err != nil { //nolint:gosec // Internal usage. No need to check. - r.recordPvrFailed(pvrName, "error writing done file", err) - return + err := funcWriteCompletionMark(r.pvr, result.Restore, log) + if err != nil { + log.WithError(err).Warnf("Failed to write completion mark, restored pod may failed to start") } restoreBytes, err := funcMarshal(result.Restore) if err != nil { log.WithError(err).Errorf("Failed to marshal restore result %v", result.Restore) - r.recordPvrFailed(pvrName, fmt.Sprintf("error marshaling restore result %v", result.Restore), err) + r.recordPvrFailed(fmt.Sprintf("error marshaling restore result %v", result.Restore), err) } else { r.eventRecorder.Event(r.pvr, false, datapath.EventReasonCompleted, string(restoreBytes)) r.resultSignal <- dataPathResult{ @@ -272,11 +243,11 @@ func (r *RestoreMicroService) OnPvrCompleted(ctx context.Context, namespace stri log.Info("Async fs restore data path completed") } -func (r *RestoreMicroService) recordPvrFailed(pvrName string, msg string, err error) { +func (r *RestoreMicroService) recordPvrFailed(msg string, err error) { evtMsg := fmt.Sprintf("%s, error %v", msg, err) r.eventRecorder.Event(r.pvr, false, datapath.EventReasonFailed, evtMsg) r.resultSignal <- dataPathResult{ - err: errors.Wrapf(err, msg, pvrName), + err: errors.Wrapf(err, msg), } } @@ -284,7 +255,7 @@ func (r *RestoreMicroService) OnPvrFailed(ctx context.Context, namespace string, log := r.logger.WithField("PVR", pvrName) log.WithError(err).Error("Async fs restore data path failed") - r.recordPvrFailed(pvrName, fmt.Sprintf("Data path for PVR %s failed", pvrName), err) + r.recordPvrFailed(fmt.Sprintf("Data path for PVR %s failed", pvrName), err) } func (r *RestoreMicroService) OnPvrCancelled(ctx context.Context, namespace string, pvrName string) { @@ -311,13 +282,13 @@ func (r *RestoreMicroService) OnPvrProgress(ctx context.Context, namespace strin r.eventRecorder.Event(r.pvr, false, datapath.EventReasonProgress, string(progressBytes)) } -func (r *RestoreMicroService) closeDataPath(ctx context.Context, ddName string) { - fsRestore := r.dataPathMgr.GetAsyncBR(ddName) +func (r *RestoreMicroService) closeDataPath(ctx context.Context, pvrName string) { + fsRestore := r.dataPathMgr.GetAsyncBR(pvrName) if fsRestore != nil { fsRestore.Close(ctx) } - r.dataPathMgr.RemoveAsyncBR(ddName) + r.dataPathMgr.RemoveAsyncBR(pvrName) } func (r *RestoreMicroService) cancelPodVolumeRestore(pvr *velerov1api.PodVolumeRestore) { @@ -332,3 +303,39 @@ func (r *RestoreMicroService) cancelPodVolumeRestore(pvr *velerov1api.PodVolumeR fsBackup.Cancel() } } + +func writeCompletionMark(pvr *velerov1api.PodVolumeRestore, result datapath.RestoreResult, log logrus.FieldLogger) error { + volumePath := result.Target.ByPath + if volumePath == "" { + return errors.New("target volume is empty in restore result") + } + + // Remove the .velero directory from the restored volume (it may contain done files from previous restores + // of this volume, which we don't want to carry over). If this fails for any reason, log and continue, since + // this is non-essential cleanup (the done files are named based on restore UID and the init container looks + // for the one specific to the restore being executed). + if err := os.RemoveAll(filepath.Join(volumePath, ".velero")); err != nil { + log.WithError(err).Warnf("Failed to remove .velero directory from directory %s", volumePath) + } + + if len(pvr.OwnerReferences) == 0 { + return errors.New("error finding restore UID") + } + + restoreUID := pvr.OwnerReferences[0].UID + + // Create the .velero directory within the volume dir so we can write a done file + // for this restore. + if err := os.MkdirAll(filepath.Join(volumePath, ".velero"), 0755); err != nil { + return errors.Wrapf(err, "error creating .velero directory for done file") + } + + // Write a done file with name= into the just-created .velero dir + // within the volume. The velero init container on the pod is waiting + // for this file to exist in each restored volume before completing. + if err := os.WriteFile(filepath.Join(volumePath, ".velero", string(restoreUID)), nil, 0644); err != nil { //nolint:gosec // Internal usage. No need to check. + return errors.Wrapf(err, "error writing done file") + } + + return nil +} diff --git a/pkg/podvolume/restore_micro_service_test.go b/pkg/podvolume/restore_micro_service_test.go new file mode 100644 index 000000000..0968bfeec --- /dev/null +++ b/pkg/podvolume/restore_micro_service_test.go @@ -0,0 +1,470 @@ +/* +Copyright The Velero Contributors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "context" + "fmt" + "sync" + "testing" + "time" + + "github.com/pkg/errors" + "github.com/sirupsen/logrus" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + "k8s.io/apimachinery/pkg/runtime" + + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/uploader" + + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + + clientFake "sigs.k8s.io/controller-runtime/pkg/client/fake" + + velerotest "github.com/vmware-tanzu/velero/pkg/test" + + kbclient "sigs.k8s.io/controller-runtime/pkg/client" + + datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks" +) + +type restoreMsTestHelper struct { + eventReason string + eventMsg string + marshalErr error + marshalBytes []byte + withEvent bool + eventLock sync.Mutex + writeCompletionErr error +} + +func (rt *restoreMsTestHelper) Event(_ runtime.Object, _ bool, reason string, message string, a ...any) { + rt.eventLock.Lock() + defer rt.eventLock.Unlock() + + rt.withEvent = true + rt.eventReason = reason + rt.eventMsg = fmt.Sprintf(message, a...) +} + +func (rt *restoreMsTestHelper) EndingEvent(_ runtime.Object, _ bool, reason string, message string, a ...any) { + rt.eventLock.Lock() + defer rt.eventLock.Unlock() + + rt.withEvent = true + rt.eventReason = reason + rt.eventMsg = fmt.Sprintf(message, a...) +} +func (rt *restoreMsTestHelper) Shutdown() {} + +func (rt *restoreMsTestHelper) Marshal(v any) ([]byte, error) { + if rt.marshalErr != nil { + return nil, rt.marshalErr + } + + return rt.marshalBytes, nil +} + +func (rt *restoreMsTestHelper) EventReason() string { + rt.eventLock.Lock() + defer rt.eventLock.Unlock() + + return rt.eventReason +} + +func (rt *restoreMsTestHelper) EventMessage() string { + rt.eventLock.Lock() + defer rt.eventLock.Unlock() + + return rt.eventMsg +} + +func (rt *restoreMsTestHelper) WriteCompletionMark(*velerov1api.PodVolumeRestore, datapath.RestoreResult, logrus.FieldLogger) error { + return rt.writeCompletionErr +} + +func TestOnPvrFailed(t *testing.T) { + pvrName := "fake-pvr" + rt := &restoreMsTestHelper{} + + rs := &RestoreMicroService{ + pvrName: pvrName, + dataPathMgr: datapath.NewManager(1), + eventRecorder: rt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + expectedErr := "Data path for PVR fake-pvr failed: fake-error" + expectedEventReason := datapath.EventReasonFailed + expectedEventMsg := "Data path for PVR fake-pvr failed, error fake-error" + + go rs.OnPvrFailed(context.TODO(), velerov1api.DefaultNamespace, pvrName, errors.New("fake-error")) + + result := <-rs.resultSignal + assert.EqualError(t, result.err, expectedErr) + assert.Equal(t, expectedEventReason, rt.EventReason()) + assert.Equal(t, expectedEventMsg, rt.EventMessage()) +} + +func TestPvrCancelled(t *testing.T) { + pvrName := "fake-pvr" + rt := &restoreMsTestHelper{} + + rs := RestoreMicroService{ + pvrName: pvrName, + dataPathMgr: datapath.NewManager(1), + eventRecorder: rt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + expectedErr := datapath.ErrCancelled + expectedEventReason := datapath.EventReasonCancelled + expectedEventMsg := "Data path for PVR fake-pvr canceled" + + go rs.OnPvrCancelled(context.TODO(), velerov1api.DefaultNamespace, pvrName) + + result := <-rs.resultSignal + assert.EqualError(t, result.err, expectedErr) + assert.Equal(t, expectedEventReason, rt.EventReason()) + assert.Equal(t, expectedEventMsg, rt.EventMessage()) +} + +func TestOnPvrCompleted(t *testing.T) { + tests := []struct { + name string + expectedErr string + expectedEventReason string + expectedEventMsg string + marshalErr error + marshallStr string + writeCompletionErr error + expectedLog string + }{ + { + name: "marshal fail", + marshalErr: errors.New("fake-marshal-error"), + expectedErr: "error marshaling restore result {{ } 0}: fake-marshal-error", + }, + { + name: "succeed", + marshallStr: "fake-complete-string", + expectedEventReason: datapath.EventReasonCompleted, + expectedEventMsg: "fake-complete-string", + }, + { + name: "succeed but write completion mark fail", + marshallStr: "fake-complete-string", + writeCompletionErr: errors.New("fake-write-completion-error"), + expectedEventReason: datapath.EventReasonCompleted, + expectedEventMsg: "fake-complete-string", + expectedLog: "Failed to write completion mark, restored pod may failed to start", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + pvrName := "fake-pvr" + + rt := &restoreMsTestHelper{ + marshalErr: test.marshalErr, + marshalBytes: []byte(test.marshallStr), + writeCompletionErr: test.writeCompletionErr, + } + + logBuffer := []string{} + + rs := &RestoreMicroService{ + dataPathMgr: datapath.NewManager(1), + eventRecorder: rt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewMultipleLogger(&logBuffer), + } + + funcMarshal = rt.Marshal + funcWriteCompletionMark = rt.WriteCompletionMark + + go rs.OnPvrCompleted(context.TODO(), velerov1api.DefaultNamespace, pvrName, datapath.Result{}) + + result := <-rs.resultSignal + if test.marshalErr != nil { + assert.EqualError(t, result.err, test.expectedErr) + } else { + assert.NoError(t, result.err) + assert.Equal(t, test.expectedEventReason, rt.EventReason()) + assert.Equal(t, test.expectedEventMsg, rt.EventMessage()) + + if test.expectedLog != "" { + assert.Contains(t, logBuffer[0], test.expectedLog) + } + } + }) + } +} + +func TestOnPvrProgress(t *testing.T) { + tests := []struct { + name string + expectedErr string + expectedEventReason string + expectedEventMsg string + marshalErr error + marshallStr string + }{ + { + name: "marshal fail", + marshalErr: errors.New("fake-marshal-error"), + expectedErr: "Failed to marshal restore result", + }, + { + name: "succeed", + marshallStr: "fake-progress-string", + expectedEventReason: datapath.EventReasonProgress, + expectedEventMsg: "fake-progress-string", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + pvrName := "fake-pvr" + + rt := &restoreMsTestHelper{ + marshalErr: test.marshalErr, + marshalBytes: []byte(test.marshallStr), + } + + rs := &RestoreMicroService{ + dataPathMgr: datapath.NewManager(1), + eventRecorder: rt, + logger: velerotest.NewLogger(), + } + + funcMarshal = rt.Marshal + + rs.OnPvrProgress(context.TODO(), velerov1api.DefaultNamespace, pvrName, &uploader.Progress{}) + + if test.marshalErr != nil { + assert.False(t, rt.withEvent) + } else { + assert.True(t, rt.withEvent) + assert.Equal(t, test.expectedEventReason, rt.EventReason()) + assert.Equal(t, test.expectedEventMsg, rt.EventMessage()) + } + }) + } +} + +func TestCancelPodVolumeRestore(t *testing.T) { + tests := []struct { + name string + expectedEventReason string + expectedEventMsg string + expectedErr string + }{ + { + name: "no fs restore", + expectedEventReason: datapath.EventReasonCancelled, + expectedEventMsg: "Data path for PVR fake-pvr canceled", + expectedErr: datapath.ErrCancelled, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + pvrName := "fake-pvr" + pvr := builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Result() + + rt := &restoreMsTestHelper{} + + rs := &RestoreMicroService{ + dataPathMgr: datapath.NewManager(1), + eventRecorder: rt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + go rs.cancelPodVolumeRestore(pvr) + + result := <-rs.resultSignal + + assert.EqualError(t, result.err, test.expectedErr) + assert.True(t, rt.withEvent) + assert.Equal(t, test.expectedEventReason, rt.EventReason()) + assert.Equal(t, test.expectedEventMsg, rt.EventMessage()) + }) + } +} + +func TestRunCancelableDataPathRestore(t *testing.T) { + pvrName := "fake-pvr" + pvr := builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseNew).Result() + pvrInProgress := builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Result() + ctxTimeout, cancel := context.WithTimeout(context.Background(), time.Second) + + tests := []struct { + name string + ctx context.Context + result *dataPathResult + dataPathMgr *datapath.Manager + kubeClientObj []runtime.Object + initErr error + startErr error + dataPathStarted bool + expectedEventMsg string + expectedErr string + }{ + { + name: "no pvr", + ctx: ctxTimeout, + expectedErr: "error waiting for PVR: context deadline exceeded", + }, + { + name: "pvr not in in-progress", + ctx: ctxTimeout, + kubeClientObj: []runtime.Object{pvr}, + expectedErr: "error waiting for PVR: context deadline exceeded", + }, + { + name: "create data path fail", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvrInProgress}, + dataPathMgr: datapath.NewManager(0), + expectedErr: "error to create data path: Concurrent number exceeds", + }, + { + name: "init data path fail", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvrInProgress}, + initErr: errors.New("fake-init-error"), + expectedErr: "error to initialize data path: fake-init-error", + }, + { + name: "start data path fail", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvrInProgress}, + startErr: errors.New("fake-start-error"), + expectedErr: "error starting data path restore: fake-start-error", + }, + { + name: "data path timeout", + ctx: ctxTimeout, + kubeClientObj: []runtime.Object{pvrInProgress}, + dataPathStarted: true, + expectedEventMsg: fmt.Sprintf("Data path for %s stopped", pvrName), + expectedErr: "timed out waiting for fs restore to complete", + }, + { + name: "data path returns error", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvrInProgress}, + dataPathStarted: true, + result: &dataPathResult{ + err: errors.New("fake-data-path-error"), + }, + expectedEventMsg: fmt.Sprintf("Data path for %s stopped", pvrName), + expectedErr: "fake-data-path-error", + }, + { + name: "succeed", + ctx: context.Background(), + kubeClientObj: []runtime.Object{pvrInProgress}, + dataPathStarted: true, + result: &dataPathResult{ + result: "fake-succeed-result", + }, + expectedEventMsg: fmt.Sprintf("Data path for %s stopped", pvrName), + }, + } + + scheme := runtime.NewScheme() + velerov1api.AddToScheme(scheme) + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fakeClientBuilder := clientFake.NewClientBuilder() + fakeClientBuilder = fakeClientBuilder.WithScheme(scheme) + + fakeClient := fakeClientBuilder.WithRuntimeObjects(test.kubeClientObj...).Build() + + rt := &restoreMsTestHelper{} + + rs := &RestoreMicroService{ + namespace: velerov1api.DefaultNamespace, + pvrName: pvrName, + ctx: context.Background(), + client: fakeClient, + dataPathMgr: datapath.NewManager(1), + eventRecorder: rt, + resultSignal: make(chan dataPathResult), + logger: velerotest.NewLogger(), + } + + if test.ctx != nil { + rs.ctx = test.ctx + } + + if test.dataPathMgr != nil { + rs.dataPathMgr = test.dataPathMgr + } + + datapath.FSBRCreator = func(string, string, kbclient.Client, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { + fsBR := datapathmockes.NewAsyncBR(t) + if test.initErr != nil { + fsBR.On("Init", mock.Anything, mock.Anything).Return(test.initErr) + } + + if test.startErr != nil { + fsBR.On("Init", mock.Anything, mock.Anything).Return(nil) + fsBR.On("StartRestore", mock.Anything, mock.Anything, mock.Anything).Return(test.startErr) + } + + if test.dataPathStarted { + fsBR.On("Init", mock.Anything, mock.Anything).Return(nil) + fsBR.On("StartRestore", mock.Anything, mock.Anything, mock.Anything).Return(nil) + } + + return fsBR + } + + if test.result != nil { + go func() { + time.Sleep(time.Millisecond * 500) + rs.resultSignal <- *test.result + }() + } + + result, err := rs.RunCancelableDataPath(test.ctx) + + if test.expectedErr != "" { + assert.EqualError(t, err, test.expectedErr) + } else { + assert.NoError(t, err) + assert.Equal(t, test.result.result, result) + } + + if test.expectedEventMsg != "" { + assert.True(t, rt.withEvent) + assert.Equal(t, test.expectedEventMsg, rt.EventMessage()) + } + }) + } + + cancel() +} From 9a9574325b20eba045139244a7ef7bb5d1127d1f Mon Sep 17 00:00:00 2001 From: Michael Steven Fruchtman <40075929+msfrucht@users.noreply.github.com> Date: Fri, 6 Jun 2025 07:25:42 -0700 Subject: [PATCH 19/31] Documentation for ephemeral-storage (#8244) Signed-off-by: MICHAEL S FRUCHTMAN --- site/content/docs/main/customize-installation.md | 6 ++++++ site/content/docs/v1.14/customize-installation.md | 6 ++++++ 2 files changed, 12 insertions(+) diff --git a/site/content/docs/main/customize-installation.md b/site/content/docs/main/customize-installation.md index 8a08fca7f..322263939 100644 --- a/site/content/docs/main/customize-installation.md +++ b/site/content/docs/main/customize-installation.md @@ -178,6 +178,12 @@ Additionally, you may want to update the the default File System Backup operatio - --fs-backup-timeout=240m ``` +### Ephemeral-storage Requests and Limits + +Velero does not set ephemeral-storage limits during installation. Limits and requests can be edited after install for clusters that monitor and restrict ephemeral-storage usage. + +Plugins will use ephemeral-storage. There needs to be a sufficient requests and limit set to account for plugins and the additional ephemeral-storage used to maintain credentials and cache space for datamovers. Object storage plugins will fit comfortably into an allocation of 100MB of ephemeral-storage. + ## Configure more than one storage location for backups or volume snapshots Velero supports any number of backup storage locations and volume snapshot locations. For more details, see [about locations](locations.md). diff --git a/site/content/docs/v1.14/customize-installation.md b/site/content/docs/v1.14/customize-installation.md index b8721ead0..358ea9393 100644 --- a/site/content/docs/v1.14/customize-installation.md +++ b/site/content/docs/v1.14/customize-installation.md @@ -178,6 +178,12 @@ Additionally, you may want to update the the default File System Backup operatio - --fs-backup-timeout=240m ``` +### Ephemeral-storage Requests and Limits + +Velero does not set ephemeral-storage limits during installation. Limits and requests can be edited after install for clusters that monitor and restrict ephemeral-storage usage. + +Plugins will use ephemeral-storage. There needs to be a sufficient requests and limit set to account for plugins and the additional ephemeral-storage used to maintain credentials and cache space for datamovers. Object storage plugins will fit comfortably into an allocation of 100MB of ephemeral-storage. + ## Configure more than one storage location for backups or volume snapshots Velero supports any number of backup storage locations and volume snapshot locations. For more details, see [about locations](locations.md). From 90d13bb609d23aab9bf74245d771e6910d672d62 Mon Sep 17 00:00:00 2001 From: Tiger Kaovilai Date: Fri, 6 Jun 2025 10:48:40 -0400 Subject: [PATCH 20/31] Add .config/ to gitignore Signed-off-by: Tiger Kaovilai Signed-off-by: Tiger Kaovilai --- .gitignore | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.gitignore b/.gitignore index c6fb4076f..ce62679f1 100644 --- a/.gitignore +++ b/.gitignore @@ -58,3 +58,8 @@ debug.test* # make lint cache .cache/ + +# Go telemetry directory created when container sets HOME to working directory +# This happens because Makefile uses 'docker run -w /github.com/vmware-tanzu/velero' +# and Go's os.UserConfigDir() falls back to $HOME/.config when XDG_CONFIG_HOME is unset +.config/ From 1b7175394af1592c2a0d04ca109ec238ec4b6729 Mon Sep 17 00:00:00 2001 From: Xun Jiang Date: Fri, 30 May 2025 22:21:19 +0800 Subject: [PATCH 21/31] Skip VS and VSC not created by backup. Signed-off-by: Xun Jiang --- changelogs/unreleased/8990-blackpiglet | 1 + pkg/backup/actions/csi/pvc_action.go | 1 - .../actions/csi/volumesnapshot_action.go | 111 +++++++---------- .../actions/csi/volumesnapshot_action_test.go | 17 --- pkg/controller/backup_controller.go | 94 ++++++++++++-- pkg/controller/backup_controller_test.go | 115 ++++++++++++++++++ pkg/util/csi/volume_snapshot.go | 24 ---- pkg/util/csi/volume_snapshot_test.go | 41 +------ 8 files changed, 244 insertions(+), 160 deletions(-) create mode 100644 changelogs/unreleased/8990-blackpiglet diff --git a/changelogs/unreleased/8990-blackpiglet b/changelogs/unreleased/8990-blackpiglet new file mode 100644 index 000000000..ec805dc58 --- /dev/null +++ b/changelogs/unreleased/8990-blackpiglet @@ -0,0 +1 @@ +Skip VS and VSC not created by backup. diff --git a/pkg/backup/actions/csi/pvc_action.go b/pkg/backup/actions/csi/pvc_action.go index ffe31d90a..d1d8056a7 100644 --- a/pkg/backup/actions/csi/pvc_action.go +++ b/pkg/backup/actions/csi/pvc_action.go @@ -285,7 +285,6 @@ func (p *pvcBackupItemAction) Execute( vs, p.crClient, p.log, - true, backup.Spec.CSISnapshotTimeout.Duration, ) if err != nil { diff --git a/pkg/backup/actions/csi/volumesnapshot_action.go b/pkg/backup/actions/csi/volumesnapshot_action.go index 7a3bc79d7..595606a39 100644 --- a/pkg/backup/actions/csi/volumesnapshot_action.go +++ b/pkg/backup/actions/csi/volumesnapshot_action.go @@ -96,22 +96,6 @@ func (p *volumeSnapshotBackupItemAction) Execute( }, } - // determine if we are backing up a VolumeSnapshot that was created by velero while - // performing backup of a CSI backed PVC. - // For VolumeSnapshots that were created during the backup of a CSI backed PVC, - // we will wait for the VolumeSnapshotContents to be available. - // For VolumeSnapshots created outside of velero, we expect the VolumeSnapshotContent - // to be available prior to backing up the VolumeSnapshot. In case of a failure, - // backup should be re-attempted after the CSI driver has reconciled the VolumeSnapshot. - // existence of the velerov1api.BackupNameLabel indicates that the VolumeSnapshot was - // created while backing up a CSI backed PVC. - - // We want to await reconciliation of only those VolumeSnapshots created during the - // ongoing backup. For this we will wait only if the backup label exists on the - // VolumeSnapshot object and the backup name is the same as that of the value of the - // backup label. - backupOngoing := vs.Labels[velerov1api.BackupNameLabel] == label.GetValidName(backup.Name) - p.log.Infof("Getting VolumesnapshotContent for Volumesnapshot %s/%s", vs.Namespace, vs.Name) @@ -119,7 +103,6 @@ func (p *volumeSnapshotBackupItemAction) Execute( vs, p.crClient, p.log, - backupOngoing, backup.Spec.CSISnapshotTimeout.Duration, ) if err != nil { @@ -171,42 +154,40 @@ func (p *volumeSnapshotBackupItemAction) Execute( } } - if backupOngoing { - p.log.Infof("Patching VolumeSnapshotContent %s with velero BackupNameLabel", - vsc.Name) - // If we created the VolumeSnapshotContent object during this ongoing backup, - // we would have created it with a DeletionPolicy of Retain. - // But, we want to retain these VolumeSnapshotContent ONLY for the lifetime - // of the backup. To that effect, during velero backup - // deletion, we will update the DeletionPolicy of the VolumeSnapshotContent - // and then delete the VolumeSnapshot object which will cascade delete the - // VolumeSnapshotContent and the associated snapshot in the storage - // provider (handled by the CSI driver and the CSI common controller). - // However, in the event that the VolumeSnapshot object is deleted outside - // of the backup deletion process, it is possible that the dynamically created - // VolumeSnapshotContent object will be left as an orphaned and non-discoverable - // resource in the cluster as well as in the storage provider. To avoid piling - // up of such orphaned resources, we will want to discover and delete the - // dynamically created VolumeSnapshotContent. We do that by adding - // the "velero.io/backup-name" label on the VolumeSnapshotContent. - // Further, we want to add this label only on VolumeSnapshotContents that - // were created during an ongoing velero backup. - originVSC := vsc.DeepCopy() - kubeutil.AddLabels( - &vsc.ObjectMeta, - map[string]string{ - velerov1api.BackupNameLabel: label.GetValidName(backup.Name), - }, - ) + p.log.Infof("Patching VolumeSnapshotContent %s with velero BackupNameLabel", + vsc.Name) + // If we created the VolumeSnapshotContent object during this ongoing backup, + // we would have created it with a DeletionPolicy of Retain. + // But, we want to retain these VolumeSnapshotContent ONLY for the lifetime + // of the backup. To that effect, during velero backup + // deletion, we will update the DeletionPolicy of the VolumeSnapshotContent + // and then delete the VolumeSnapshot object which will cascade delete the + // VolumeSnapshotContent and the associated snapshot in the storage + // provider (handled by the CSI driver and the CSI common controller). + // However, in the event that the VolumeSnapshot object is deleted outside + // of the backup deletion process, it is possible that the dynamically created + // VolumeSnapshotContent object will be left as an orphaned and non-discoverable + // resource in the cluster as well as in the storage provider. To avoid piling + // up of such orphaned resources, we will want to discover and delete the + // dynamically created VolumeSnapshotContent. We do that by adding + // the "velero.io/backup-name" label on the VolumeSnapshotContent. + // Further, we want to add this label only on VolumeSnapshotContents that + // were created during an ongoing velero backup. + originVSC := vsc.DeepCopy() + kubeutil.AddLabels( + &vsc.ObjectMeta, + map[string]string{ + velerov1api.BackupNameLabel: label.GetValidName(backup.Name), + }, + ) - if vscPatchError := p.crClient.Patch( - context.TODO(), - vsc, - crclient.MergeFrom(originVSC), - ); vscPatchError != nil { - p.log.Warnf("Failed to patch VolumeSnapshotContent %s: %v", - vsc.Name, vscPatchError) - } + if vscPatchError := p.crClient.Patch( + context.TODO(), + vsc, + crclient.MergeFrom(originVSC), + ); vscPatchError != nil { + p.log.Warnf("Failed to patch VolumeSnapshotContent %s: %v", + vsc.Name, vscPatchError) } } @@ -244,20 +225,18 @@ func (p *volumeSnapshotBackupItemAction) Execute( var itemToUpdate []velero.ResourceIdentifier // Only return Async operation for VSC created for this backup. - if backupOngoing { - // The operationID is of the form // - operationID = vs.Namespace + "/" + vs.Name + "/" + time.Now().Format(time.RFC3339) - itemToUpdate = []velero.ResourceIdentifier{ - { - GroupResource: kuberesource.VolumeSnapshots, - Namespace: vs.Namespace, - Name: vs.Name, - }, - { - GroupResource: kuberesource.VolumeSnapshotContents, - Name: vsc.Name, - }, - } + // The operationID is of the form // + operationID = vs.Namespace + "/" + vs.Name + "/" + time.Now().Format(time.RFC3339) + itemToUpdate = []velero.ResourceIdentifier{ + { + GroupResource: kuberesource.VolumeSnapshots, + Namespace: vs.Namespace, + Name: vs.Name, + }, + { + GroupResource: kuberesource.VolumeSnapshotContents, + Name: vsc.Name, + }, } return &unstructured.Unstructured{Object: vsMap}, diff --git a/pkg/backup/actions/csi/volumesnapshot_action_test.go b/pkg/backup/actions/csi/volumesnapshot_action_test.go index 5abf7ac9e..a9c666123 100644 --- a/pkg/backup/actions/csi/volumesnapshot_action_test.go +++ b/pkg/backup/actions/csi/volumesnapshot_action_test.go @@ -49,23 +49,6 @@ func TestVSExecute(t *testing.T) { expectedAdditionalItems []velero.ResourceIdentifier expectedItemToUpdate []velero.ResourceIdentifier }{ - { - name: "VS not created by backup, has no status. Backup is finalizing", - backup: builder.ForBackup("velero", "backup"). - Phase(velerov1api.BackupPhaseFinalizing).Result(), - vs: builder.ForVolumeSnapshot("velero", "vs"). - VolumeSnapshotClass("class").Result(), - expectedErr: "", - }, - { - name: "VS is not created by the backup, associated VSC not exists", - backup: builder.ForBackup("velero", "backup"). - Phase(velerov1api.BackupPhaseInProgress).Result(), - vs: builder.ForVolumeSnapshot("velero", "vs"). - VolumeSnapshotClass("class").Status(). - BoundVolumeSnapshotContentName("vsc").Result(), - expectedErr: `error getting volume snapshot content from API: volumesnapshotcontents.snapshot.storage.k8s.io "vsc" not found`, - }, { name: "Normal case", backup: builder.ForBackup("velero", "backup"). diff --git a/pkg/controller/backup_controller.go b/pkg/controller/backup_controller.go index e2b624b0a..bc296f790 100644 --- a/pkg/controller/backup_controller.go +++ b/pkg/controller/backup_controller.go @@ -21,6 +21,7 @@ import ( "context" "fmt" "os" + "slices" "time" snapshotv1api "github.com/kubernetes-csi/external-snapshotter/client/v7/apis/volumesnapshot/v1" @@ -31,6 +32,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" kerrors "k8s.io/apimachinery/pkg/util/errors" + "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/utils/clock" ctrl "sigs.k8s.io/controller-runtime" @@ -63,6 +65,20 @@ const ( backupResyncPeriod = time.Minute ) +var autoExcludeNamespaceScopedResources = []string{ + // CSI VolumeSnapshot and VolumeSnapshotContent are intermediate resources. + // Velero only handle the VS and VSC created during backup, + // not during resource collecting. + "volumesnapshots.snapshot.storage.k8s.io", +} + +var autoExcludeClusterScopedResources = []string{ + // CSI VolumeSnapshot and VolumeSnapshotContent are intermediate resources. + // Velero only handle the VS and VSC created during backup, + // not during resource collecting. + "volumesnapshotcontents.snapshot.storage.k8s.io", +} + type backupReconciler struct { ctx context.Context logger logrus.FieldLogger @@ -481,19 +497,51 @@ func (b *backupReconciler) prepareBackupRequest(backup *velerov1api.Backup, logg request.Status.ValidationErrors = append(request.Status.ValidationErrors, validatedError) } - // validate the included/excluded resources - for _, err := range collections.ValidateIncludesExcludes(request.Spec.IncludedResources, request.Spec.ExcludedResources) { - request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("Invalid included/excluded resource lists: %v", err)) - } + if collections.UseOldResourceFilters(request.Spec) { + // validate the included/excluded resources + ieErr := collections.ValidateIncludesExcludes(request.Spec.IncludedResources, request.Spec.ExcludedResources) + if len(ieErr) > 0 { + for _, err := range ieErr { + request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("Invalid included/excluded resource lists: %v", err)) + } + } else { + request.Spec.IncludedResources, request.Spec.ExcludedResources = + modifyResourceIncludeExclude( + request.Spec.IncludedResources, + request.Spec.ExcludedResources, + append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), + ) + } + } else { + // validate the cluster-scoped included/excluded resources + clusterErr := collections.ValidateScopedIncludesExcludes(request.Spec.IncludedClusterScopedResources, request.Spec.ExcludedClusterScopedResources) + if len(clusterErr) > 0 { + for _, err := range clusterErr { + request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("Invalid cluster-scoped included/excluded resource lists: %s", err)) + } + } else { + request.Spec.IncludedClusterScopedResources, request.Spec.ExcludedClusterScopedResources = + modifyResourceIncludeExclude( + request.Spec.IncludedClusterScopedResources, + request.Spec.ExcludedClusterScopedResources, + autoExcludeClusterScopedResources, + ) + } - // validate the cluster-scoped included/excluded resources - for _, err := range collections.ValidateScopedIncludesExcludes(request.Spec.IncludedClusterScopedResources, request.Spec.ExcludedClusterScopedResources) { - request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("Invalid cluster-scoped included/excluded resource lists: %s", err)) - } - - // validate the namespace-scoped included/excluded resources - for _, err := range collections.ValidateScopedIncludesExcludes(request.Spec.IncludedNamespaceScopedResources, request.Spec.ExcludedNamespaceScopedResources) { - request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("Invalid namespace-scoped included/excluded resource lists: %s", err)) + // validate the namespace-scoped included/excluded resources + namespaceErr := collections.ValidateScopedIncludesExcludes(request.Spec.IncludedNamespaceScopedResources, request.Spec.ExcludedNamespaceScopedResources) + if len(namespaceErr) > 0 { + for _, err := range namespaceErr { + request.Status.ValidationErrors = append(request.Status.ValidationErrors, fmt.Sprintf("Invalid namespace-scoped included/excluded resource lists: %s", err)) + } + } else { + request.Spec.IncludedNamespaceScopedResources, request.Spec.ExcludedNamespaceScopedResources = + modifyResourceIncludeExclude( + request.Spec.IncludedNamespaceScopedResources, + request.Spec.ExcludedNamespaceScopedResources, + autoExcludeNamespaceScopedResources, + ) + } } // validate the included/excluded namespaces @@ -932,3 +980,25 @@ func oldAndNewFilterParametersUsedTogether(backupSpec velerov1api.BackupSpec) bo return haveOldResourceFilterParameters && haveNewResourceFilterParameters } + +func modifyResourceIncludeExclude(include, exclude, addedExclude []string) (modifiedInclude, modifiedExclude []string) { + modifiedInclude = include + modifiedExclude = exclude + + excludeStrSet := sets.NewString(exclude...) + for _, ex := range addedExclude { + if !excludeStrSet.Has(ex) { + modifiedExclude = append(modifiedExclude, ex) + } + } + + for _, exElem := range modifiedExclude { + for inIndex, inElem := range modifiedInclude { + if inElem == exElem { + modifiedInclude = slices.Delete(modifiedInclude, inIndex, inIndex+1) + } + } + } + + return modifiedInclude, modifiedExclude +} diff --git a/pkg/controller/backup_controller_test.go b/pkg/controller/backup_controller_test.go index b5ee3dbf1..e8a28282d 100644 --- a/pkg/controller/backup_controller_test.go +++ b/pkg/controller/backup_controller_test.go @@ -708,6 +708,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.True(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -745,6 +746,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: "alt-loc", DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -786,6 +788,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: "read-write", DefaultVolumesToFsBackup: boolptr.True(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -824,6 +827,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -862,6 +866,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.True(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -901,6 +906,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -940,6 +946,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.True(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -979,6 +986,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.True(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -1018,6 +1026,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -1058,6 +1067,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.True(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFailed, @@ -1098,6 +1108,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.True(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFailed, @@ -1138,6 +1149,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.True(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -1179,6 +1191,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -1220,6 +1233,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -1261,6 +1275,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.True(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -1303,6 +1318,7 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.False(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, @@ -1344,6 +1360,105 @@ func TestProcessBackupCompletions(t *testing.T) { StorageLocation: defaultBackupLocation.Name, DefaultVolumesToFsBackup: boolptr.False(), SnapshotMoveData: boolptr.True(), + ExcludedResources: append(autoExcludeNamespaceScopedResources, autoExcludeClusterScopedResources...), + }, + Status: velerov1api.BackupStatus{ + Phase: velerov1api.BackupPhaseFinalizing, + Version: 1, + FormatVersion: "1.1.0", + StartTimestamp: ×tamp, + Expiration: ×tamp, + CSIVolumeSnapshotsAttempted: 0, + CSIVolumeSnapshotsCompleted: 0, + }, + }, + volumeSnapshot: builder.ForVolumeSnapshot("velero", "testVS").VolumeSnapshotClass("testClass").Status().BoundVolumeSnapshotContentName("testVSC").RestoreSize("10G").SourcePVC("testPVC").ObjectMeta(builder.WithLabels(velerov1api.BackupNameLabel, "backup-1")).Result(), + }, + { + name: "backup with namespace-scoped and cluster-scoped resource filters", + backup: builder.ForBackup(velerov1api.DefaultNamespace, "backup-1"). + ExcludedClusterScopedResources("clusterroles"). + IncludedClusterScopedResources("storageclasses"). + ExcludedNamespaceScopedResources("secrets"). + IncludedNamespaceScopedResources("pods").Result(), + backupLocation: defaultBackupLocation, + defaultVolumesToFsBackup: false, + defaultSnapshotMoveData: true, + expectedResult: &velerov1api.Backup{ + TypeMeta: metav1.TypeMeta{ + Kind: "Backup", + APIVersion: "velero.io/v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1api.DefaultNamespace, + Name: "backup-1", + Annotations: map[string]string{ + "velero.io/source-cluster-k8s-major-version": "1", + "velero.io/source-cluster-k8s-minor-version": "16", + "velero.io/source-cluster-k8s-gitversion": "v1.16.4", + "velero.io/resource-timeout": "0s", + }, + Labels: map[string]string{ + "velero.io/storage-location": "loc-1", + }, + }, + Spec: velerov1api.BackupSpec{ + StorageLocation: defaultBackupLocation.Name, + DefaultVolumesToFsBackup: boolptr.False(), + SnapshotMoveData: boolptr.True(), + IncludedClusterScopedResources: []string{"storageclasses"}, + ExcludedClusterScopedResources: append([]string{"clusterroles"}, autoExcludeClusterScopedResources...), + IncludedNamespaceScopedResources: []string{"pods"}, + ExcludedNamespaceScopedResources: append([]string{"secrets"}, autoExcludeNamespaceScopedResources...), + }, + Status: velerov1api.BackupStatus{ + Phase: velerov1api.BackupPhaseFinalizing, + Version: 1, + FormatVersion: "1.1.0", + StartTimestamp: ×tamp, + Expiration: ×tamp, + CSIVolumeSnapshotsAttempted: 0, + CSIVolumeSnapshotsCompleted: 0, + }, + }, + volumeSnapshot: builder.ForVolumeSnapshot("velero", "testVS").VolumeSnapshotClass("testClass").Status().BoundVolumeSnapshotContentName("testVSC").RestoreSize("10G").SourcePVC("testPVC").ObjectMeta(builder.WithLabels(velerov1api.BackupNameLabel, "backup-1")).Result(), + }, + { + name: "backup's include filter overlap with default exclude resources", + backup: builder.ForBackup(velerov1api.DefaultNamespace, "backup-1"). + ExcludedClusterScopedResources("clusterroles"). + IncludedClusterScopedResources("storageclasses", "volumesnapshotcontents.snapshot.storage.k8s.io"). + ExcludedNamespaceScopedResources("secrets"). + IncludedNamespaceScopedResources("pods", "volumesnapshots.snapshot.storage.k8s.io").Result(), + backupLocation: defaultBackupLocation, + defaultVolumesToFsBackup: false, + defaultSnapshotMoveData: true, + expectedResult: &velerov1api.Backup{ + TypeMeta: metav1.TypeMeta{ + Kind: "Backup", + APIVersion: "velero.io/v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: velerov1api.DefaultNamespace, + Name: "backup-1", + Annotations: map[string]string{ + "velero.io/source-cluster-k8s-major-version": "1", + "velero.io/source-cluster-k8s-minor-version": "16", + "velero.io/source-cluster-k8s-gitversion": "v1.16.4", + "velero.io/resource-timeout": "0s", + }, + Labels: map[string]string{ + "velero.io/storage-location": "loc-1", + }, + }, + Spec: velerov1api.BackupSpec{ + StorageLocation: defaultBackupLocation.Name, + DefaultVolumesToFsBackup: boolptr.False(), + SnapshotMoveData: boolptr.True(), + IncludedClusterScopedResources: []string{"storageclasses"}, + ExcludedClusterScopedResources: append([]string{"clusterroles"}, autoExcludeClusterScopedResources...), + IncludedNamespaceScopedResources: []string{"pods"}, + ExcludedNamespaceScopedResources: append([]string{"secrets"}, autoExcludeNamespaceScopedResources...), }, Status: velerov1api.BackupStatus{ Phase: velerov1api.BackupPhaseFinalizing, diff --git a/pkg/util/csi/volume_snapshot.go b/pkg/util/csi/volume_snapshot.go index f3d1f1a8b..3363ce632 100644 --- a/pkg/util/csi/volume_snapshot.go +++ b/pkg/util/csi/volume_snapshot.go @@ -588,32 +588,8 @@ func WaitUntilVSCHandleIsReady( volSnap *snapshotv1api.VolumeSnapshot, crClient crclient.Client, log logrus.FieldLogger, - shouldWait bool, csiSnapshotTimeout time.Duration, ) (*snapshotv1api.VolumeSnapshotContent, error) { - if !shouldWait { - if volSnap.Status == nil || - volSnap.Status.BoundVolumeSnapshotContentName == nil { - // volumesnapshot hasn't been reconciled and we're - // not waiting for it. - return nil, nil - } - vsc := new(snapshotv1api.VolumeSnapshotContent) - err := crClient.Get( - context.TODO(), - crclient.ObjectKey{ - Name: *volSnap.Status.BoundVolumeSnapshotContentName, - }, - vsc, - ) - if err != nil { - return nil, - errors.Wrap(err, - "error getting volume snapshot content from API") - } - return vsc, nil - } - // We'll wait 10m for the VSC to be reconciled polling // every 5s unless backup's csiSnapshotTimeout is set interval := 5 * time.Second diff --git a/pkg/util/csi/volume_snapshot_test.go b/pkg/util/csi/volume_snapshot_test.go index ba24a91e7..8c736d482 100644 --- a/pkg/util/csi/volume_snapshot_test.go +++ b/pkg/util/csi/volume_snapshot_test.go @@ -1646,26 +1646,22 @@ func TestWaitUntilVSCHandleIsReady(t *testing.T) { name string volSnap *snapshotv1api.VolumeSnapshot exepctedVSC *snapshotv1api.VolumeSnapshotContent - wait bool expectError bool }{ { name: "waitEnabled should find volumesnapshotcontent for volumesnapshot", volSnap: validVS, exepctedVSC: vscObj, - wait: true, expectError: false, }, { name: "waitEnabled should not find volumesnapshotcontent for volumesnapshot with non-existing snapshotcontent name in status.BoundVolumeSnapshotContentName", volSnap: vsWithVSCNotFound, exepctedVSC: nil, - wait: true, expectError: true, }, { name: "waitEnabled should not find volumesnapshotcontent for a non-existent volumesnapshot", - wait: true, exepctedVSC: nil, expectError: true, volSnap: &snapshotv1api.VolumeSnapshot{ @@ -1678,46 +1674,11 @@ func TestWaitUntilVSCHandleIsReady(t *testing.T) { }, }, }, - { - name: "waitDisabled should not find volumesnapshotcontent when volumesnapshot status is nil", - wait: false, - expectError: false, - exepctedVSC: nil, - volSnap: vsWithNilStatus, - }, - { - name: "waitDisabled should not find volumesnapshotcontent when volumesnapshot status.BoundVolumeSnapshotContentName is nil", - wait: false, - expectError: false, - exepctedVSC: nil, - volSnap: vsWithNilStatusField, - }, - { - name: "waitDisabled should find volumesnapshotcontent when volumesnapshotcontent status is nil", - wait: false, - expectError: false, - exepctedVSC: vscWithNilStatus, - volSnap: vsForNilStatusVsc, - }, - { - name: "waitDisabled should find volumesnapshotcontent when volumesnapshotcontent status.SnapshotHandle is nil", - wait: false, - expectError: false, - exepctedVSC: vscWithNilStatusField, - volSnap: vsForNilStatusFieldVsc, - }, - { - name: "waitDisabled should not find a non-existent volumesnapshotcontent", - wait: false, - exepctedVSC: nil, - expectError: true, - volSnap: vsWithVSCNotFound, - }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { - actualVSC, actualError := WaitUntilVSCHandleIsReady(tc.volSnap, fakeClient, logrus.New().WithField("fake", "test"), tc.wait, 0) + actualVSC, actualError := WaitUntilVSCHandleIsReady(tc.volSnap, fakeClient, logrus.New().WithField("fake", "test"), 0) if tc.expectError && actualError == nil { assert.Error(t, actualError) assert.Nil(t, actualVSC) From 00ce103a50807b70615605451870782b919210cc Mon Sep 17 00:00:00 2001 From: Matthieu MOREL Date: Sat, 7 Jun 2025 10:36:07 +0200 Subject: [PATCH 22/31] Refactor affinity tests to use require and assert for better readability and error reporting Signed-off-by: Matthieu MOREL --- pkg/util/velero/velero_test.go | 45 +++++++++++----------------------- 1 file changed, 14 insertions(+), 31 deletions(-) diff --git a/pkg/util/velero/velero_test.go b/pkg/util/velero/velero_test.go index 06e9ed070..4090daeaa 100644 --- a/pkg/util/velero/velero_test.go +++ b/pkg/util/velero/velero_test.go @@ -21,6 +21,7 @@ import ( "testing" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" appsv1api "k8s.io/api/apps/v1" corev1api "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -210,39 +211,21 @@ func TestGetAffinityFromVeleroServer(t *testing.T) { for _, test := range tests { t.Run(test.name, func(t *testing.T) { got := GetAffinityFromVeleroServer(test.deploy) - - if got == nil { - if test.want != nil { - t.Errorf("expected affinity to be %v, got nil", test.want) + if test.want != nil { + require.NotNilf(t, got, "expected affinity to be %v, got nil", test.want) + if test.want.NodeAffinity != nil { + require.NotNilf(t, got.NodeAffinity, "expected node affinity to be %v, got nil", test.want.NodeAffinity) + if test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution != nil { + require.NotNilf(t, got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution, "expected required during scheduling ignored during execution to be %v, got nil", test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution) + assert.Truef(t, reflect.DeepEqual(got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution, test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution), "expected required during scheduling ignored during execution to be %v, got %v", test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution, got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution) + } else { + assert.Nilf(t, got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution, "expected required during scheduling ignored during execution to be nil, got %v", got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution) + } + } else { + assert.Nilf(t, got.NodeAffinity, "expected node affinity to be nil, got %v", got.NodeAffinity) } } else { - if test.want == nil { - t.Errorf("expected affinity to be nil, got %v", got) - } else { - if got.NodeAffinity == nil { - if test.want.NodeAffinity != nil { - t.Errorf("expected node affinity to be %v, got nil", test.want.NodeAffinity) - } - } else { - if test.want.NodeAffinity == nil { - t.Errorf("expected node affinity to be nil, got %v", got.NodeAffinity) - } else { - if got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution == nil { - if test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution != nil { - t.Errorf("expected required during scheduling ignored during execution to be %v, got nil", test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution) - } - } else { - if test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution == nil { - t.Errorf("expected required during scheduling ignored during execution to be nil, got %v", got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution) - } else { - if !reflect.DeepEqual(got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution, test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution) { - t.Errorf("expected required during scheduling ignored during execution to be %v, got %v", test.want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution, got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution) - } - } - } - } - } - } + assert.Nilf(t, got, "expected affinity to be nil, got %v", got) } }) } From 1f5436fe91fe77ef9306b1508d6a205c42d7a95e Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Fri, 6 Jun 2025 17:22:08 +0800 Subject: [PATCH 23/31] vgdp ms pvr controller Signed-off-by: Lyndon-Li --- .../v1/bases/velero.io_podvolumerestores.yaml | 75 +- config/crd/v1/crds/crds.go | 2 +- pkg/apis/velero/v1/labels_annotations.go | 3 + pkg/apis/velero/v1/pod_volume_restore_type.go | 34 +- pkg/apis/velero/v1/zz_generated.deepcopy.go | 4 + pkg/builder/pod_builder.go | 5 + pkg/builder/pod_volume_restore_builder.go | 30 + pkg/cmd/cli/nodeagent/server.go | 10 +- pkg/controller/data_download_controller.go | 5 +- pkg/controller/data_upload_controller.go | 5 +- .../pod_volume_backup_controller.go | 5 +- .../pod_volume_restore_controller.go | 895 +++++++++++++---- .../pod_volume_restore_controller_legacy.go | 10 + .../pod_volume_restore_controller_test.go | 932 +++++++++++++++++- pkg/exposer/mocks/PodVolumeExposer.go | 125 +++ pkg/podvolume/restorer.go | 4 +- 16 files changed, 1901 insertions(+), 243 deletions(-) create mode 100644 pkg/controller/pod_volume_restore_controller_legacy.go create mode 100644 pkg/exposer/mocks/PodVolumeExposer.go diff --git a/config/crd/v1/bases/velero.io_podvolumerestores.yaml b/config/crd/v1/bases/velero.io_podvolumerestores.yaml index 888ac1669..09eda5b28 100644 --- a/config/crd/v1/bases/velero.io_podvolumerestores.yaml +++ b/config/crd/v1/bases/velero.io_podvolumerestores.yaml @@ -15,39 +15,40 @@ spec: scope: Namespaced versions: - additionalPrinterColumns: - - description: Namespace of the pod containing the volume to be restored - jsonPath: .spec.pod.namespace - name: Namespace + - description: PodVolumeRestore status such as New/InProgress + jsonPath: .status.phase + name: Status type: string - - description: Name of the pod containing the volume to be restored - jsonPath: .spec.pod.name - name: Pod + - description: Time duration since this PodVolumeRestore was started + jsonPath: .status.startTimestamp + name: Started + type: date + - description: Completed bytes + format: int64 + jsonPath: .status.progress.bytesDone + name: Bytes Done + type: integer + - description: Total bytes + format: int64 + jsonPath: .status.progress.totalBytes + name: Total Bytes + type: integer + - description: Name of the Backup Storage Location where the backup data is stored + jsonPath: .spec.backupStorageLocation + name: Storage Location + type: string + - description: Time duration since this PodVolumeRestore was created + jsonPath: .metadata.creationTimestamp + name: Age + type: date + - description: Name of the node where the PodVolumeRestore is processed + jsonPath: .status.node + name: Node type: string - description: The type of the uploader to handle data transfer jsonPath: .spec.uploaderType name: Uploader Type type: string - - description: Name of the volume to be restored - jsonPath: .spec.volume - name: Volume - type: string - - description: Pod Volume Restore status such as New/InProgress - jsonPath: .status.phase - name: Status - type: string - - description: Pod Volume Restore status such as New/InProgress - format: int64 - jsonPath: .status.progress.totalBytes - name: TotalBytes - type: integer - - description: Pod Volume Restore status such as New/InProgress - format: int64 - jsonPath: .status.progress.bytesDone - name: BytesDone - type: integer - - jsonPath: .metadata.creationTimestamp - name: Age - type: date name: v1 schema: openAPIV3Schema: @@ -77,6 +78,11 @@ spec: BackupStorageLocation is the name of the backup storage location where the backup repository is stored. type: string + cancel: + description: |- + Cancel indicates request to cancel the ongoing PodVolumeRestore. It can be set + when the PodVolumeRestore is in InProgress phase + type: boolean pod: description: Pod is a reference to the pod containing the volume to be restored. @@ -162,6 +168,13 @@ spec: status: description: PodVolumeRestoreStatus is the current status of a PodVolumeRestore. properties: + acceptedTimestamp: + description: |- + AcceptedTimestamp records the time the pod volume restore is to be prepared. + The server's time is used for AcceptedTimestamp + format: date-time + nullable: true + type: string completionTimestamp: description: |- CompletionTimestamp records the time a restore was completed. @@ -173,11 +186,19 @@ spec: message: description: Message is a message about the pod volume restore's status. type: string + node: + description: Node is name of the node where the pod volume restore + is processed. + type: string phase: description: Phase is the current state of the PodVolumeRestore. enum: - New + - Accepted + - Prepared - InProgress + - Canceling + - Canceled - Completed - Failed type: string diff --git a/config/crd/v1/crds/crds.go b/config/crd/v1/crds/crds.go index a412073f6..0288fbcce 100644 --- a/config/crd/v1/crds/crds.go +++ b/config/crd/v1/crds/crds.go @@ -35,7 +35,7 @@ var rawCRDs = [][]byte{ []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcVMo\x1b7\x10\xbd\xebW\f\xd0kwU\xa3hQ\xec\xadqr0\xda\x06\x82\x1d\xe4N\x91#-c.\xc9\xce\f\xe5\xba\x1f\xff\xbd \xb9+K\xab\x95\x93\\\xb27\x91Ù\xc7\xf7f\x1e\xd54\xcdJE\xfb\x11\x89m\xf0\x1d\xa8h\xf1/A\x9f\x7fq\xfb\xf8\v\xb76\xac\x0f7\xabG\xebM\a\xb7\x89%\f\xf7\xc8!\x91Ʒ\xb8\xb3ފ\r~5\xa0(\xa3Du+\x00\xe5}\x10\x95\x979\xff\x04\xd0\xc1\v\x05琚=\xfa\xf61mq\x9b\xac3H%\xf9T\xfa\xf0C{\xf3s\xfb\xd3\n\xc0\xab\x01;0\xe8Pp\xab\xf4c\x8a\x84\x7f&d\xe1\xf6\x80\x0e)\xb46\xac8\xa2\xce\xf9\xf7\x14R\xec\xe0e\xa3\x9e\x1fkW\xdcoK\xaa7%\xd5}MUv\x9de\xf9\xedZ\xc4\xefv\x8c\x8a.\x91rˀJ\x00[\xbfON\xd1b\xc8\n\x80u\x88\xd8\xc1\xfb\f+*\x8df\x050^\xbb\xc0l@\x19S\x88TnC\xd6\v\xd2mpi\x98\bl\xc0 k\xb2Q\nQ\x1fz,W\x84\xb0\x03\xe9\x11j9\x90\x00[\x1c\x11\x98r\x0e\xe0\x13\a\xbfQ\xd2w\xd0f\xbe\xda\x1a\x9a\x81\x8c\x01\x95\xea7\xf3ey\u0380Y\xc8\xfa\xfd5\b,J\x12O J]\x1b<\xd0\t\xbf\xe7\x00J|\x1b{\xc5\xe7\xd5\x1f\xcaƵ\xca5\xe6pS\x99\xd6=\x0e\xaa\x1bcCD\xff\xeb\xe6\xee\xe3\x8f\x0fg\xcbp\x8euAZ\xb0\fjB\x9a\x89\xab\xacA\xf0\b\x81`\b4\xb1\xca\xed1i\xa4\x10\x91\xc4N\xadU\xbf\x93\xe19Y\x9dA\xf8\xb79\xdb\x03Ȩ\xeb)0y\x8a\x90\v\x89cS\xa0\x19/Zɵ\f\x84\x91\x90\xd1\u05f9\xca\xcb\xcaC\xd8~B-\xed,\xf5\x03RN\x03܇\xe4L\x1e\xbe\x03\x92\x00\xa1\x0e{o\xff>\xe6\xe6|\xef\\\xd4))\x94\xe4\xb6\xf3\xca\xc1A\xb9\x84߃\xf2f\x96yP\xcf@\x98kB\xf2'\xf9\xca\x01\x9e\xe3\xf8#\x93h\xfd.tЋD\xee\xd6뽕\xc9Rt\x18\x86\xe4\xad<\xaf\x8b;\xd8m\x92@\xbc6x@\xb7f\xbbo\x14\xe9\xde\njI\x84k\x15mS.⋭\xb4\x83\xf9\x8eF\x13Ⳳ\x17\xddS\xbf\xe2\x02_!O\xf6\x84\xda#5U\xbd\xe2\x8b\ny)Sw\xff\xee\xe1\x03LH\xaaRU\x94\x97\xd0\v^&}2\x9b\xd6\xef\x90\xea\xb9\x1d\x85\xa1\xe4Dob\xb0^\xca\x0f\xed,z\x01N\xdb\xc1\nO\x1d\x9b\xa5\x9b\xa7\xbd-\xb6\x9b\x1d E\xa3\x04\xcd<\xe0\xceí\x1a\xd0\xdd*\xc6o\xacUV\x85\x9b,\xc2\x17\xa9u\xfa\x98̃+\xbd'\x1b\xd33pEڅ\xe1\x7f\x88\xa8\xb3\xb8\x99\xdf|\xda\ueb2ec\xb5\v\x04O\xbd\xd5\xfd4\xfc3\x9a\x8eFq\xce߲1\xe4\xef\xc5n\xe7;W/\x0fEdK8k\xd8\x06.\xbc\xfbu^\x8a\xa9~%3\xd5\xd1Gnt\"*\xcdw\xf4y\xb5t\xe8K\xb9@\xa2@\x17\xab3P\xefJP\xf9Ǡ\xacgP\xfey<\b\xd2+\x81'\xa4\r\x97\x95\x1ax\x8fO\v\xabw~CaO\xc8\xf3\x96ϛ\x9b\xca\x1e\xce߃WXZlʋE\xceVhNXd\t\xa4\xf6\xa7\xbcr\xda\x1e\x9d\xbe\x83\x7f\xfe[\xfd\x1f\x00\x00\xff\xff\xbeM\x1a\xea\xb1\n\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcWMo\xe36\x10\xbd\xfbW\f\xd0K\v\xac\xe4\x06E\x8b·\xd6\xd9C\xb0\xe96\x88\xb7\xb9S\xd4HbC\x91,9t6E\x7f|1\xa4\xe4\x0fYv\x9c\xcb\xea\xe6\xe1p\xf8\xe6\xcd\xcc#]\x14\xc5B8\xf5\x84>(kV \x9c¯\x84\x86\x7f\x85\xf2\xf9\xd7P*\xbb\xdc\xde,\x9e\x95\xa9W\xb0\x8e\x81l\xff\x88\xc1F/\xf1\x16\x1be\x14)k\x16=\x92\xa8\x05\x89\xd5\x02@\x18cI\xb09\xf0O\x00i\ry\xab5\xfa\xa2ES>\xc7\n\xab\xa8t\x8d>\x05\x1f\x8f\xde\xfeX\xde\xfcR\xfe\xbc\x000\xa2\xc7\x15\xd4\xf6\xc5h+j\x8f\xffD\f\x14\xca-j\xf4\xb6Tv\x11\x1cJ\x8e\xddz\x1b\xdd\n\xf6\vy\xefpn\xc6|;\x84y\xccaҊV\x81>ͭޫ\xc1\xc3\xe9\xe8\x85>\x05\x91\x16\x832m\xd4\u009f,/\x00\x82\xb4\x0eW\xf0\x99a8!\xb1^\x00\f)&XŐ\xdd\xf6&\x87\x92\x1d\xf6\"\xe3\x05\xb0\x0e\xcdo\x0fwO?m\x8e\xcc\x005\x06镣D\xd4\x7f\xc5\xce\x0e\xd3\x04@\x05\x100\xc0\x01\xb2;\x84 \f\bO\xaa\x11\x92\xa0\xf1\xb6\x87J\xc8\xe7\xe8\xc0V\x7f\xa3$\bd\xbdh\xf1\x03\x84(;\x10\x1c%;\x1c\x9c\xa5m\v\x8d\xd2X\xeel\xce[\x87\x9e\xd4Hy\xfe\x0e\x1a\xea\xc0z)\v\xfe8\xf1\xbc\vj\xee,\f@\x1d\x8e\xe4a=p\x05\xb6\x01\xeaT\x00\x8f\xcec@\x93{\x8d\xcd\xc2\fٔ\x93\xd0\x1b\xf4\x1c\x06Bg\xa3\xae\xb9!\xb7\xe8\t\x1aE\xaf\xcb41\xaa\x8ad}XָE\xbd\f\xaa-\x84\x97\x9d\"\x94\x14=.\x85SEJĤQ+\xfb\xfa;?\ff8:\x96^\xb9!\x03yeڃ\x854\x1d\xef(\x0f\xcfK\xee\xae\x1c*\xa7\xb8\xaf\x02\x9b\x98\xbaǏ\x9b/0\"ɕ\x1aZl\xe7z\xc2\xcbX\x1ffS\x99\x06}ޗڔc\xa2\xa9\x9dU\x86\xd2\x0f\xa9\x15\x1a\x82\x10\xab^Q\x18{\x9dK7\r\xbbNR\x04\x15Bt\xb5 \xac\xa7\x0ew\x06֢G\xbd\x16\x01\xbfq\xad\xb8*\xa1\xe0\"\\U\xadC\x81\x9d:gz\x0f\x16Fy&j^\x01\x128\xe1[\xa4\xa9u\x82\xe5Kr\xe2\xe3_:q,X\xdfcٖ\xac9a\x00\x92\xf5\xe8\x87i\xa1.a\x80\xd9F\x9fE2\xf67\xd3\xc0\xbc\xb2\xa0\xb0\xd8\x1db:=\x9a?4\xb1\x9f?\xa0\x80\xdf\x13\xe6{\xdb^\\_[C<\x17\x17\x9d\x9e\xac\x8e=n\x8cp\xa1\xb3o\xf8\xde\x11\xf6\x7f:\xf4\xf9\x1a\xbe\xe8:\xde滫\xef\x82c\xd4g\xcf}D\xbeA\xf0|\xa6\x83\xc3UQ\xae\xc04x^\x95\xe8zs\xf7\x1e\nϸ\xbf\xa3Hw\xa6\xb1o\xa4\xb8w\x9c\xf5;#\x03\xe3\x97\xde\x10o\xf74\xbfBƞ\xe6-\xf9\xeeD\xf8\x14+\xf4\x06\t\xc3^\xa9_\x14u\xb3\x11\x01^:%\xbb\xb41\r\x04_\x02!X\xa9\xe6$\xf5\n\xf8\xac#\xca\xe3\xccP\x16iXg\xcc\f\xfe\xc4|F\xfd\xce\x1dP\f\x8at\x95\x82\x92\xa0\x18ޡ\xa1\xc9\x7f\xa4ZF\xef\xd3\x15\x95\xad\xfc2\x99n\xb8VDG\xe5\xf9\xeb\xf1\xfe\r%\xbd\xdd{\xa6\x17\xb7P&\xa3q\x1e\x8b\xa0Z~A\xf1\x1akiҸS2\xf2w\xfc\xc2;&j\xb6\xa2\xf8թ<\x80o@\xfc\xb8ŝ\x8f&\xdf\xf3\xd37l\n\x88\x81\x9f[ \x85\x99\xc1X!Ԩ\x91\xb0\x86\xea5\xdf\\\xaf\x81\xb0?\xc5\xddX\xdf\vZ\x01\xdf\xff\x05\xa9\x9962QkQi\\\x01\xf9x\xae\xcbf\x13w\x9d\b3cx\x94\xf3\x03\xfb\xcc5\xc6n\x18/v\x06\x9c\xbd_\n\xf8\x8c/3\xd6\ao%\x86\x80\xa7ct6\x93\xd9!81\x06~\xa4\xd5\a,\r\x7f\x19\x06\xcb\xff\x01\x00\x00\xff\xffx\xae@\xbaJ\x0e\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Z_s\x1b\xb7\x11\x7fק\xd8Q\x1e\x92\xcc\xf8\xc8\xdam3\x1d\xbe\xd9r\xd3Q\x9b\xa8\x1aS\xf6K&\x0f\xcbÒ\x87\xe8\x0e@\x01\x1ce6\xcdw\xef,p \xef\x8e )҉}/6\xf1g\xf1\xc3\xfe߅\x8a\xa2\xb8B#?\x90uR\xab\x19\xa0\x91\xf4ѓ\xe2_n\xf2\xf877\x91z\xba~y\xf5(\x95\x98\xc1M\xeb\xbcnޑӭ-\xe9--\xa5\x92^juՐG\x81\x1egW\x00\xa8\x94\xf6\xc8Î\x7f\x02\x94Zy\xab\xeb\x9al\xb1\"5yl\x17\xb4he-\xc8\x06\xe2\xe9\xe8\xf5\x9f&/\xbf\x9b\xfc\xf5\n@aC30Z\xacu\xdd6\xb4\xc0\xf2\xb15n\xb2\xa6\x9a\xac\x9eH}\xe5\f\x95L{eukf\xb0\x9b\x88{\xbbs#\xe6{->\x042o\x02\x990SK\xe7\xff\x95\x9b\xfdA:\x1fV\x98\xba\xb5X\xef\x83\b\x93N\xaaU[\xa3ݛ\xbe\x02p\xa564\x83;\x86a\xb0$q\x05\xd0]1\xc0*\x00\x85\bL\xc3\xfa\xdeJ\xe5\xc9\xde0\x85Ĭ\x02\x04\xb9\xd2J\xe3\x03S\ued40\b\x10\"Bp\x1e}\xeb\xc0\xb5e\x05\xe8\xe0\x8e\x9e\xa6\xb7\xea\xde\xea\x95%\x17\xe1\x01\xfcⴺG_\xcd`\x12\x97OL\x85\x8e\xba\xd9\xc8\xdey\x98\xe8\x86\xfc\x86A;o\xa5Z\xe5`<Ȇ\xe0\xa9\"\x05\xbe\x92\x0e\xe2m\xe1\t\x1dñ>\xdc2\x7fp\x98\xe7\xed\xcecc\x06\bn,\xe1nk\x84 \xd0S\x0e\xc0\x96\x9f\xa0\x97\xe0+b\xce\a\xc5B\xa9\xa4Z\x85\xa1(\t\xf0\x1a\x16\x14 \x92\x80\xd6d\x90\x19*'F\x8b\x89JD\a\xb0\xeeF\xa3\xa7x\xc3\xeb\x7foT\x03@\xf7Z\\\x00\xe5\xacs\xe3\xe2\xc1\xa9\x1f\xfaC'\xf5\xa3\xa2\xb0&\x1dޚZ\xa3 \xcb\xc7W\xa8DM,Y\x04oQ\xb9%\xd9\x030Ҷ\x87\x8d\x19\x82y\x9f\xe8\xf5f\xceaFg;s\xaf-\xae\b~\xd0epP\xacҖ\x06:\xed*\xdd\xd6\x02\x16\xe9\x14\x00\xe7\xb5\xcd*8#\x8e\xbb:\xba\x89\xec\xc8Άg\x1eFߣ\x9d\xfc\xe9\xa4d\x1b\x91Z\xe5-\xe8\xf5\x8a\xf2\xd6\x13\xa7\xd7/\xa3\xbb*+jp֭Ԇ\xd4\xeb\xfb\xdb\x0f\x7f\x9e\x0f\x86\x01\x8cՆ\xac\x97\xc9}Ư\x17\x1cz\xa30d\xf5\xff\x8a\xc1\x1c\x00\x1f\x10w\x81\xe0(A.\xead\x1c#\xd1a\x8a\xe2\x91\x0e,\x19K\x8eT\x8c\x1b<\x8c\n\xf4\xe2\x17*\xfddDzN\x96\xc9$A\x95Z\xad\xc9z\xb0Tꕒ\xff\xdd\xd2v\xac{|h\x8d\x9e\x9c\x87\xe0j\x15ְƺ\xa5\x17\x80J\x8c(7\xb8\x01K|&\xb4\xaaG/lpc\x1c?jK \xd5RϠ\xf2\u07b8\xd9t\xba\x92>\x85\xccR7M\xab\xa4\xdfLC\xf4\x93\x8b\xd6k릂\xd6TO\x9d\\\x15h\xcbJz*}ki\x8aF\x16\xe1\"*\x84\xcdI#\xbe\xb2]\x90u\x83c\xf7\xb4&~!ҝ!\x1e\x8e} \x1d`G*^q'\x85\xe4\xbb\xde\xfd}\xfe\x00\tI\x94T\x14\xcan\xe9\x1e_\x92|\x98\x9bR-\xd9\a\xf0\xbe\xa5\xd5M\xa0IJ\x18-\x95\x0f?\xcaZ\x92\xf2\xe0\xdaE#=\xab\xc1\x7fZr\x9eE7&{\x13\xd2\n\xf6e\xada5\x17\xe3\x05\xb7\nn\xb0\xa1\xfa\x06\x1d}fY\xb1T\\\xc1Bx\x96\xb4\xfa\xc9\xd2xqdoo\"\xa5:\aD;\xca_\xe6\x86J\x16,\xf3\x96wʥ\xec<\xddR[\xc0\xf1\xf2!\x9f\xf2\x0e\x80\xbf\xac\x97\x1b/:\xa5t\xfc\xbd\xc9\x11J\x80U\xcfa'o\xdc9\xcfz\xe8<\xfb_r\xe1\xdb=\x96\x8cv\xd2k\xbba\xc2\xd1{\x8f\x15\xe2\xa0l\xf8+Q\x95T_r\xbd\x9b\xb0\x13\xa4\x12\xccv\xda*4\xbb\xa2H5\x00\xd5j\xa5\xd9\xc4\xc6Ҁ[\xcf\xcbX\xc9\x1d\xf9\xfc]U\xa00\xda\xc9\x17\x95\nvy \xf4\xf3\xbd\xf1\xa5\x17Zׄc^*-\xe8ĝﴠ\x9c\xb0x+\xf8\n}\xc2Ƌl\xab\xd4>o\xf9\xd3\xea,q\x18-N\xe0\xeaND\xb0\xb4$K\xaa\xa4\xe4\xfb\x8f\xe5c\x19d\xfdLi\x1f\xe3a\xfb\x80#\x812\x8b\xf8\xf5\xfdm\n\x86\x89\x89\x1d\xf6\xbdxw\x92?\xfc-%\xd5\"\xe4\x0e\xa7\xcf\xcej.\x7f\xb7\xcb\b\"D\x04\xaf\x01\xc1H\x8a\x19\xf76\x1a\x83T\xce\x13\x8an\x90\x9d\xa0\xa5n\xeeE\xf4\xf4\aA\xf2\xb7\x8b\xda,\x13@\x8e\xb2%\xc7+\xb4\x04\xb5|\xcc\xd8O\xfc\xaeC\xae\xb8\x83\xf9+[\xcfo\xd7\xf0Mt^\xd7\xfc\xf3:\xc2ئ-}\x03\xdb\xc1\x89Vf\xe5jE\xbb\xa4tOY8\xccr\x80\xfa\x16\xb4\xe5\xbb*\xdd#\x11\b\xb3\x9cb| \xb1\a\xef\xa7W?_\xc37C\x1e\x1c8J*A\x1f\xe1\x15{\x9f\xc0\x1b\xa3ŷ\x13x\bz\xb0Q\x1e?\xf2Ie\xa5\x1d)Ъ\xdeĂ`M\xe04W\x94T\xd7EL\x10\x05<\xe1\x06\xf4\xf2\xc09ID\xac\x9a\b\x06\xad?\x9a$v|8n4\xfbYS\xfa\x9eg/!\x8bz\x96\xf5~\xb1\f䙜\b\xe5\xc2'p\xa2_j]\xc0\x89\xc7vAV\x91\xa7\xc0\f\xa1K\xc7|(\xc9x7\xd5k\xb2kIO\xd3'm\x1f\xa5Z\x15\xac\x8cE\x94\xba\x9b\x86\n~\xfaU\xf8\xe7ҋ\x87Z\xffSo?\xe8M|~\x16\xf0\xe9nz\t\aRv\xff\xfc\xd8u\x90\x0f\xf3.\xe1\x1c\xd3d\x9b\x7f\xaadY\xa5Z\xaf\xe7m\x1b\x14\xd1\x1d\xa3\xda|!\xdba>\xb7\x96\x11m\x8a\xaeUY\xa0\x12\xfc\x7f'\x9d\xe7\xf1K\x18\xdb\xcaOr.\xefo\xdf~I\x8bj\xe5%\x9e\xe4@\r\x13\xbf\x8f\xc5\x0eUѠ)\xe2j\xf4\xba\x91\xe5h5\xe7\U00037085\xb4\x94dO\xa4\x7f\xef\x06\x8bS\x82\x9a\xa9\x06\xb6k\xce\xca?=\xae2\t_\xbf\x8b{,-<ʯӪ\xf0\x80+\ah\t\x10\x1a4\xac\x11\x8f\xb4)b\xc6aPr\xba\xc0\x19\xc1\xb6k\x05hL\xcd1=f\x11\x19\x8a]\xfe۱\a]\xb8\xdf!\x86dE\x99\xbats\xf2^\xaa/Ȝ\xf7# \xbf/\xa3\xb6=\xccR\xab\xa5\\\xb56T\xa0\xfb\x9cRm]㢦\x19x\xdb\x1e\xaa\xb9\x8e2\xf2\x81\x97\x1c\xbf\xff\xfb\xdeҤ\xe1'\x1a\xae\xf9[\rڰ\xfb\x97!\xd56\xfbP\nx\xd4Fbfܒ\xf3{\xd6\xcb\x13\xd7\xd7\xe7\xd8XT\xcaKJ\xee\xeeq$S\x95v\x8a\xde%\xf0\xa92\xed\xf7óB?\xc37pu\xcf\xe5\xc8\x10w\x91o\x97\x8c\xd6p\xcd<\x1a2Z\x8cF\x86np49\xe8\xd9\xf7\x91\xee\xf7\x90\xc2S\xcc\x19]\xa4\xf8\xc4\xd4\xf14\x06G\x9f\x1e\x9e8\xed\xbe\xb4\x8fTj.\xbf\x06\xfd\xec\x8b\xda,\xfbdB\xff\u05ca\xce0d\xc3~\xa0\xf7J\xd5\x1d\x9ck\x04\xf5\xc9ŝ!Gaj$B\x19\xc5U\xde\x12eM\x02\xd2S\xe4\x99T\x16\xb4\xe4\x18\x1d\x8d45\":x\x87\v\x98\x87\x8a\xc0\x85n\xea\xd7nK\xb3u$B3/Ä\xfd\x88\xbdԶA\x1f\x1f\x06\n&q\x99\xf7\xca\xdalC\xce\xe1\xea\x94\xd1\xfe\x18W\xc5\xfeL\xb7\x05p\xa1[\xbfm\xd0\f\"\xd2\u05eeS\xb4\xf3zD\xd9\xd6\xc7P\xc7\xd1WI\xa5\x97m]\x87=}\xef\xb0{\xa6\x0e\xa8\x16\x94\xcf\xeb\x8e4\x88\x8e\x01\xacНb\xd5=\xaf\xc9Y\xdd֥\x1d5;8\xe2\xbe\xef\xe8)3\xba\xf7nܟ\xbcI&\x93\x99\xfb>X\xc3Y\xf7\xef\x0e\xba\xc4ܷM\xcdJ\xd7\xc9µ\xc7\x1aT\xdb,\xc82s\x16\x1bOn\xe4\xf8Q\x89>'s\xd5\xdfn\x7f\x12j\xa4\xd4u0\xba^l09\xafAHgj\xdcl\xef\x12rn\xb6\xaf|cz\xa7\xe4\xc9\xd2\r\x1d\xca!\x8e\xb7\x16\x03\xa6\xb7Z\x1d\xa8R\x93\x91K\xe5\xbf\xfbˑ\xa4]*O\xabQ\x18\xe9晝o\xf8\x94?\xe6\x84#9\x90Sh\\\xa5\xfd\xed\xdb\x13\xaa1\xdf.L&\xb2\xcb\xe7\x83C\fo\x1eݢN\x152Pw\x0e\xe7,\xfb\x1d\xfe\x19\xc3%Z<\x1fP8\x11\xaf\xba\xbf\xaa\xc8E\x859\x19\xb4\xec\x13\u008b\xda\xcd\xf8}\xf8\x058\x19\x1a\xe0\x9c\xed\xc6\xf4\xb7\xacP\xad\xb2\xfd\x11\xadB\x02\xa7\xed\xfe\xf3&\x9c\f@\xc3\v}\xceؓU\xa7\xbd\xc1\x80\\\xf4hw\x8fI\xfd\x91v\xb1}g\x9d\xc1\xaf\xbf]\xfd?\x00\x00\xff\xff\f\x19\xe2z\x0f%\x00\x00"), - []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Y_\x93۶\x11\x7fקع<$\x991\xa5\xdam3\x1d\xbd\xd9\xe7\xa6smr\xbd\xb1\xce~\xc9\xe4aE\xacHD$\x80\x00\xa0d5\xcdw\xef,@R\xfc'\xe9tnl\xbe\xdc\t\x00\x17?\xfc\xf6/\x96I\x92\xcc\xd0\xc8\x0fd\x9d\xd4j\th$}\xf4\xa4\xf8\x97\x9bo\xff\xe6\xe6R/v/g[\xa9\xc4\x12n+\xe7u\xf9\x8e\x9c\xaelJoi#\x95\xf4R\xabYI\x1e\x05z\\\xce\x00P)푇\x1d\xff\x04H\xb5\xf2V\x17\x05\xd9$#5\xdfVkZW\xb2\x10d\x83\xf0f\xebݟ\xe6/\xbf\x9b\xffu\x06\xa0\xb0\xa4%\x18-v\xba\xa8J\xb2伶\xe4\xe6;*\xc8\xea\xb9\xd43g(e\xe1\x99ՕY\xc2q\"\xbe\\o\x1cA?h\xf1!\xc8y\x17儩B:\xff\xaf\xc9\xe9\x1f\xa4\xf3a\x89)*\x8b\xc5\x04\x8e0\xeb\xa4ʪ\x02\xedx~\x06\xe0Rmh\t\xf7\f\xc5`Jb\x06P\x9f3@K\x00\x85\b\xcca\xf1`\xa5\xf2doYD\xc3X\x02\x82\\j\xa5\xf1\x81\x99V\x0e\xe8\r\xf8\x9cx\xcb\xc0*J%U\x16\x86\"\x04\xf0\x1a\xd6\x045\x12\x11\x84\x01\xfc\xe2\xb4z@\x9f/a\xce\xc4͍\x16s\xd5Ȭ\xd7D\xce\xef\a\xa3\xfe\xc0\xe7p\xdeJ\x95\x9dB\xf6\x7f\x06\xd5\xc3\xf3\xa0\xc5\x13\x91<\xe6\x14\xd64h*Sh\x14dy\xf3\x1c\x95(\b\xd8@\xc1[TnC\xf6\x04\x8a\xe6\xb5ǃ\xe9#y\xdf\xc8\xeb\xcc\\\xc3\xce5Tĵ\xbd\xed?t\x87.\xed\xfb\xa0E\xfd\x02\xd4F\rΣ\xaf\x1c\xb8*\xcd\x01\x1d\xdc\xd3~q\xa7\x1e\xac\xce,97\x01#,\x9f\x9b\x1c]\x1f\xc7*L\xfc\xb186ږ\xe8\x97 \x95\xff\xee/\xa7\xb1\xd5/ͽ\xf6X\xbc9xr=\xa4\x8f\xc3ሖ\x9d-\xab\xd5\xffE\xe0\xae\x19\xd2[\xad\xfa\xbc\xbe\x19\x8cN\x81\xed\bm\xe2\xed<\xb5\x14B\xed\xa3,\xc9y,MO\xea\xeb\xac/O\xa0\x8f\x03qz\xf72\x86\xb24\xa7\x12\x97\xf5JmH\xbd~\xb8\xfb\xf0\xe7Uo\x18\xc0Xm\xc8z\xd9D\xd7\xf8t\x92Gg\x14\xfa\xcc\xfe7\xe9\xcd\x01\xf0\x06\xf1-\x10\x9cE\xc8E'\x89c$jL\xd1y\xa4\x03Kƒ#\x15\xf3\n\x0f\xa3\x02\xbd\xfe\x85R?\x1f\x88^\x91e1\xe0r]\x15!\"\xed\xc8z\xb0\x94\xeaL\xc9\xff\xb4\xb2\x1d\xfb\"oZ\xa0'\xe7\x03\xd7Va\x01;,*z\x01\xa8\xc4@r\x89\a\xb0\xc4{B\xa5:\xf2\xc2\vn\x88\xe3G\xb6\x1f\xa96z\t\xb9\xf7\xc6-\x17\x8bL\xfa&\xa5\xa6\xba,+%\xfda\x11\xb2\xa3\\W^[\xb7\x10\xb4\xa3b\xe1d\x96\xa0Ms\xe9)\xf5\x95\xa5\x05\x1a\x99\x84\x83\xa8\x90V\xe7\xa5\xf8\xca\xd6I\xd8\xf5\xb6\x1dyd|B\"\xbcB=\x9c\x19A:\xc0ZT<\xe2Q\vM|\x7f\xf7\xf7\xd5#4H\xa2\xa6\xa2R\x8eKG\xbc4\xfaa6\xa5\xdap\x84\xe6\xf76V\x97A&)a\xb4T>\xfcH\vIʃ\xab֥\xf4l\x06\xbfV\xe4<\xabn(\xf66\x94\x1d\x1c\\+\xc3f.\x86\v\xee\x14\xdcbI\xc5-:\xfa̺b\xad\xb8\x84\x95\xf0$mu\x8b\xa9\xe1\xe2Hog\xa2\xa9\x84N\xa8vXݬ\f\xa5\xacY&\x97_\x95\x1b\x99F\x9f\xdah\v8Z\xdfgj:\x04\xf0\xb3\xc6t[\x99\x95\xd7\x163\xfaAG\x99\xc3E\x97̎\x9f7S\x82\x1aĪ\x93P\xe3\x8e\xe0\xe2J(\xea\xa5\x13\"\xf79Y\xea\xbec\xc9h'\xbd\xb6\a\x16\x1cS\xf1\xd0$Nj'\xf0\xa0Ņ\xb3q.\t\x0ediC\x96TJM\xb89W&M\x80\xefT\vc\x88\xa7\xf5\x01gB\xf3$\xe0\xd7\x0fwM\xf8m\x18\xae\xa1\x8f\"\xecEz\xf8\xd9H*D\xc8V\x97\xf7\x9e4\x04~\xee6\x11D\x88A^\x03\x82\x91\x14\xcb\xe06\xfe\x83T\xce\x13\x8az\x90\xdd\xceR=\xf7\"Ɩ\x93 \xf99\xe6\tV\t \xc7:)\xe0\x9f\xab\x7f\xdf/\xfe\xa1\xe39\x00Ӕ\x9c\v\xe5\x00\x95\xa4\xfc\x8b\xb6$\x10\xe4\xa4%\xc1u\x11\xcdKTrC\xce\xcfkid\xddO\xaf~\x9e\xe6\x0f\xe0{m\x81>bi\nz\x012rކ\xcf\xc6j\xa4\x8b\ao%\xc2^\xfa<\x005Z\xd4\a܇#x\xdc\x12\xe8\xfa\b\x15A!\xb74\xcd>\xc0M(4\x8f0\x7fc\xd7\xfa\xfd\x06\xbe\x89\xcer\xc3?o\"\x8c6Qv\xbd\xef\b\xc7\xe7\xe8\xc1[\x99et\xachG\xc6\u0081\x9dCⷠ-\x9fU鎈 \x98\xf5\x14\x03\x12\x89\x11\xbc\x9f^\xfd|\x03\xdf\xf498\xb1\x95T\x82>\xc2+\x90*rc\xb4\xf8v\x0e\x8f\xc1\x0e\x0e\xca\xe3G\xde)͵#\x05Z\x15\x87xA\xd8\x118]\x12\xec\xa9(\x92X\x92\b\xd8\xe3\x01\xf4\xe6\xc4>\x8d\x8a\xd84\x11\fZ\x7f\xb6,\xa9y8\xef4\xe3<\xdd?\x05\xbc\xbb[<\x87\x81\xa6\x9e|z\xee:\xc9ê\xaep\x862\xd9\xe7\xf7\xb9L\xf3\xe6vщ\xb6%\x8a\x18\x8eQ\x1d\xbe\x90\xef0ϕeD\x87\xa4n\x9e%\xa8\x04\xff\xef\xa4\xf3<\xfe\x1cb+\xf9I\xc1\xe5\xfd\xdd\xdb/\xe9Q\x95|N$9Q5\xc7\xe7crD\x95\x94h\x92\xb8\x1a\xbd.e:X\xcd5\xe3\x9d`%m$\xd9\v\xd5\u07fb\xde\xe2\xa6z\x9d\xa8>\xdb5W\x95\x9fN\xa1q\xb9\xf6wo/\xe0X\xb5\v\x1b\fG\x1d\xd6Eg#kЙ\xba\x0eO\xf0\xad\xfbӑ\xab\x0f\xaa\xbf\xbaA\xa6\xad\xcc$߿\xdb\xf0\x11\xae$\nK\xecv$\xbbO\x89\xc6H\x95]\x85\xb5i\xf0\xad\xc8\xf35v\xa2p\xee\xb6fϕ\xd7g\xed\xee\xb2K\xbd\x1f\x00\x01\xb4\x04\xc8gb\rm\xe9\x90\xc4*Π\xe4\x12\x8c\xab\xac\xbaT]\x13\xa01\x05\xd7I\xb12\x9b\xf2\xf5\xa6]\x99j\xb5\x91Ye\xc3\xe5h̔\xaa\x8a\x02\xd7\x05-\xc1\xdbj,\xe8\x8c\xfbt;\xa5\x174\xfe\xbe\xb3\xb4Q\xf7\x85^\xed\xf4\xa9z\x1d\xdc\xf1aHU\xe5\x18J\x02[m$N\x8c\xb3\xb1\x8f\x1c\x9d'nn\xae1\xa9\xe8I\x178\xa8\x1b\x8b\x13\x17\xd9\xda\x11벞G\xf8\xf2\x18\xdcq:;^렖~\xad\xf8\x8e\xd2G\x98L\xdf\xd9\ak\x8c\x16\xb3!i\xdd\xd86\x980\xf5\x9f\xf1ע\xc1|\xfb-\xec\x8f\xd9\xe1L\x99\xe05\x1btT\xcfF\x8a\x17a\xa2\x1e\xf2\a\xc6켕j\x9dB\xf1N\x96\x04\xa2\xb2A\xb5\xbc\xff\x9c\xc0o\xa4\x1b\xc3ۣc\x88և\x8d\xa7\xc1\x84y\x16\xe9<\x96f\x88\xaa\xf3j\x84%\xd0S\nԽ.MA\x9e\x04,\x0f\x9e\x9a\xad\xac\xb4-\xd1\xcfA*\xff\xfd_N\xf3Q\x136\r\xaf\xbeѪO\xcek\x1e\x85\xcepD\xc2\xdaZ\x93M2\xa4=\x16\x9f\x02ij\x80ם\xf7#\x92(\xb7;~\x11\n\x9b\x1e\xe8\x15\xf8\r\xc1k̷\x95\x81\x85\xd7\x16\xd7\x04?\xea<\xaap\xbf!Ka\xc52\xae`\x0f\x06ɺ\xd36\xa9:C\xf94\xae\xad\x855\xb2\x06\xfa\xeb\x7f\xe8\xb3\xd8Wn\t\x93\xf6Մ\xa2iX!\xb5J\x1b٫5=\xcb\xc0\xbaD*-\xa8\xc3\xda\b\x97t`\xac\xceɹ3\x86\xcfBzH\x1e\x8f\x03\x17)\xdaPX\xd3\x00\xaaL\xa1Q\x90\x05\xafa\x83J\x14\x14u\xe8-*\xb7\xaa-c\xac\xc2\xe6\xb5w\aӇ\xf2\xbe\x91י\x19a\x8aKw\xdf\xc50\x98o\xa8\xc4y\xbdV\x1bR\xaf\x9e\x1e>\xfcy\xd1\x1b\x06\xa6Ő\xf5\xb2\x89\xcc\xf1\xe9$\x9e\xce(\xf4\xf7\xfc\xbf\xac7\a\xc0\x1f\x88o\x81\xe0\fD.pQ\xc7W\x125\xa6ȑt`\xc9Xr\xa4bN\xe2aT\xa0\x97\xbfR\xee\xa7\x03\xd1\v\xb2,\x06\xdcFW\x85\xe0ĵ#\xeb\xc1R\xae\xd7J\xfe\xb7\x95\xed\x98p\xfeh\x81\x9e\x9c\x0f\x8eh\x15\x16\xb0â\xa2\x17\x80J\f$\x97x\x00K\xfcM\xa8TG^x\xc1\rq\xfc\x14\xacI\xad\xf4\x1c6\xde\x1b7\x9f\xcd\xd6\xd27\xe98\xd7eY)\xe9\x0f\xb3\x90Y\xe5\xb2\xf2ں\x99\xa0\x1d\x153'\xd7\x19\xda|#=徲4C#\xb3\xb0\x11\x15R\xf2\xb4\x14_\xd9:\x81\xbb\xdegG\x8a\x8eOH\xa2W\xa8\x87\xb3*{\x02֢\xe2\x16\x8fZ\xe0!\xa6\xee\xed\xdf\x17\xef\xa0A\x125\x15\x95r\\:\xe2\xa5\xd1\x0f\xb3)Պ\r\x9f\xdf[Y]\x06\x99\xa4\x84\xd1R\xf9\xf0#/$)\x0f\xaeZ\x96ҳ\x19\xfc\xa7\"\xe7YuC\xb1\xf7\xa1d\x81%;\x14\xc7\x011\\\xf0\xa0\xe0\x1eK*\xee\xd1\xd1\x1f\xac+֊\xcbX\t\xcf\xd2V\xb7\x10\x1b.\x8e\xf4v&\x9a*\xea\x84j\x87\xf1ma(g\xcd2\xb9\xfc\xaa\\\xc9:\x93\xac\xb4\x05\x1c\xad\xef3\x95\x0e\x01\xfc$3\xcap\xd1%\xb3\xe3\xe7uJP\x83Xu\x02y\x9d\xef\\\x9d\xa8\x8a~\xa2\xea>\xa3\x1ci\xc9h'\xbd\xb6\x87c\xa6\x1c\x9a\xc4I\xed\xf0\x93\xa3ʩ\xb8e{\xf7\xe1M\x90J0\xefԚ4\a\xa3(5\x00\xd5j\xad\xd9\xc9F\xea\x80\a\xcf\xeb\xd8\xce\x1d\xf9\xf4f\xd5\xc9\xcc&\x15\x1ckL\xe8֒\xc3m/\xb5.\b\x87l\x1a-.l\xfaIׁ\xc3Ҋ,\x85\xfc\x1fì\xd1!\x18{\x94\xaa\t\x1f\xb1\xe4\x06\xaf\x13\xfbXr\xb89\xa5\x9a\xd3v\bgRR\x12𫧇&\xed4\x96UC\x1fe\x96.?I\xb3\xe0g%\xa9\x10!Q_\xfev\xd2B\xf8yXE\x10!\xf6z\r\bFRN\xbd\xbc\aR9O(\xeaA\x0e7\x96\xea\xb9\x171\xa6\x9e\x04\xc9\xcf1?\xb2J\x009\xc6K\x01\xff\\\xfc\xfbq\xf6\x0f\x1d\xf7\x01\x98s%\x14\xce*T\x92\xf2/\xda\xf3\x8a '-\t>}дD%W\xe4\xfc\xb4\x96F\xd6\xfd\xfc\xf2\x974\x7f\x00?h\v\xf4\x11\xb9\xe8\x7f\x012rަ\x8d\xc6j\xa4\x8b\x1bo%\xc2^\xfaM\x00j\xb4\xa87\xb8\x0f[\xf0\xb8e\x8f\x89[\xa8\b\n\xb9\xa54\xfb\x00w\xa1x:\xc2\xfc\x8dC\xca\xefw\xf0M\f\x12w\xfc\xf3.\xc2h\v\x84n\xd49\xc2\xf1\x1b\xf4\xe0\xad\\\xaf\xe9Xh\x8f\x8c\x85\x13\x1a\xa7\x82oA[ޫ\xd2\x1d\x11A0\xeb)\x06b\x12#x?\xbf\xfc\xe5\x0e\xbe\xe9sp\xe2SR\t\xfa\b/\xd9\xc7\x037F\x8bo\xa7\xf0.\xd8\xc1Ay\xfc\xc8_\xca7ڑ\x02\xad\x8aC\xac7w\x04N\x97\x04{*\x8a,\x96b\x02\xf6x\x00\xbd:\xf1\x9dFEl\x9a\b\x06\xad?[\x8e\xd5<\x9cw\x9aq}\xd2<\xcf\xf3\x97P\xaf<\xcb{\xbfX\xae\x7f&\x13\xa10\xff\x04&\xbaG\x9d\x1b\x98\xd8VK\xb2\x8a<\x052\x84\xce\x1d\xf3\x90\x93\xf1n\xa6wdw\x92\xf6\xb3\xbd\xb6[\xa9\xd6\x19\x1bc\x16\xb5\xeef\xa1e3\xfb*\xfcs\xeb\xc6C\x9f\xe5Sw\x1f\x84|9\n\xf8\xebnv\v\x03M\x1d\xfd\xfc\xdcu\x92\x87E]\xd9\re\xb2\xcf\xef72\xdf4\xa7\xaaN\xb4-Q\xc4p\x8c\xea\xf0\x85|\x87y\xae,#:du\xc31C%\xf8\xffN:\xcf\xe3\xb7\x10[\xc9O\n.\xef\x1f\xde|I\x8f\xaa\xe4-\x91\xe4\xc4i!>\x1f\xb3#\xaa\xacD\x93\xc5\xd5\xe8u)\xf3\xc1j\xae\x95\x1f\x04+i%\xc9^\xa8\xfe\xde\xf6\x167U{\xa2\xean\xd7\\Uv;\x85\xc6m\xb4\x7fxs\x01Ǣ]\xd8`8\xea\xb0.:\x1bY\xec\x12gk\xcdsx\x82o=\x9e\x8e\\}P\xfd\xd5\r2m\xe5Z*,\x8e\x110\x1c\xc5\x14\x96\x18~%t_\xa21R\xad\xaf\xc2\xda\xf4\x8b\x16\xe4\xf9\xf8\x9e(\x9c\xbb\xed\xecs\xe5\xf5Y\xbb\xbb\xecR\xef\a@\x00-\x01\xf2\x9eXC[:d\xb1\x8a3(\xb9\x04\xe3*\xab.U\x97\x04hL\xc1uR\xac\xccR\xbe\xdet\xbfr\xadVr]w\"\xc7L\xa9\xaa(pY\xd0\x1c\xbc\xadN\x1d\x82\x92\xee\xd3m\xbc]\xd0\xf8\xfb\xce\xd2F\xdd\x17Z\x7f\xe9]\xf5\x1a\x82\xe3͐\xaa\xca1\x94\f\xb6\xdaHL\x8c\xb3\xb1\x8f\x1c\x9d'\xee\xee\xae1\xa9\xe8I\x178\x88g\xd0\xd4\x01\xbevĺ\xac\xaf\x8f\xac\xd1\x1d\xd3\xd9\xf1Z\a\xe5\xa35\x9fQ\xfa\b\xb3t\xafb\xb0\xc6h1\x19\x92֍m\x83\xc9cd\x1aN\xf4\x9d~0\x1b)xV\x9b'4\x9e\xafi\xf4\xc4륚\xf7\x98V}s\xe9\xc4\x05\xfbͭ\x1e>\x13\x1aO\xa2\xed\xc9\xdf\xd2\ay5\x14\x12\x1a\xb4V\xd4N\"Kj\x9b\x06\xb5\x9d\xd8c\x1b#\x86lc\xc9`\xd2\" 4\xd9]h4~\xed\xa24\xe9\xa0r$Bl\x1d}|$\xa1\xb9\xf3\x11\xe8)\xe3\xf7o\v \xe9\xe6Q\xbc\xee\xea\xdej\xdc\xd4I\x1a\x8b\x19s\x88-mᾥ\xb9hKQv\x94\xd7\x12\x16ő\bGX>a\xafP\x16$\xa0\xbd̽\x9a\xf9\x04\xe8qq\xf39\xc9/\xc99\\_\nZ?\xc5U\xb1\x93U\xbf\x02\xb8ԕ?a\x95_\xbbڵ\xae\xca\xc9J\x8bKH\x1e\xb5\b0\xd4\xe9+\xac1\x9a\x84Z\xba\xd7ZWa\fM\xc2KM?^\x93\n5-\xe4\xf3\xb1\x06\xce\xe4\xb0G\xda'F\x1b\x0fNL=\xd5a!15\xba_\xefN\xc6\xcel\xaa\xa6i\xe6\x922\xdb\xcb\xeb\xc4\xdc\x0f\xc1]\xaeb\xbb\xc6wK@h\xfb\xba\x1b]41 \\:\xab\xaa\\\x92eU\x84k\xedF'm\x05\x8cJt5\x97:\x9c\xb7\x12\x9a4\x1cE\xd5\xfd\xa5\xba!\x1d\xbc\xdck\x10ҙ\x02\x0f\xedf\u0089\x88]:ݞ?\xfaU\x13\xab8\xf3\x9c\xa8\xdb\xcew~\xdb?\x02H\x9f\xf7R7\xf9\xfdg|'?\x98o/\xf7?\xcf\x17\xceԝ\xfd?\xb6\xb8\xc5@\x16=\t\x97\x92E\xfd\xc7\x1f\xd7\xc7\xf8\xfeg\xfe\xc8\xf0\x9edo4\x18\x90\x8b\x8e\xec\xfa\n\xa9;R-\xdb\xfb\xd59\xfc\xf6\xfb\xe4\xff\x01\x00\x00\xff\xff\xec\xe3\xc3\ac%\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xdc=[s\xdb8w\xef\xf9\x15\x98\xf4a\xdb\x19\xcbi\xa6\x97\xe9\xf8\xcd\xf5:\x8d\xfb}\xebx\xec4\xfb\f\x91G\">\x83\x00\x17\x00\xa5h\xdb\xfe\xf7\x0e\x0e.$%\x90\x84d˛-^2\xa6\x80\x03\xe0\xdc\xcf\xc1\x01\xb2X,\xdeц}\x03\xa5\x99\x14W\x846\f\xbe\x1b\x10\xf6/}\xf9\xfco\xfa\x92\xc9\x0f\x9b\x8f\uf799(\xaf\xc8M\xab\x8d\xac\x1fA\xcbV\x15\xf03\xac\x98`\x86I\xf1\xae\x06CKj\xe8\xd5;B\xa8\x10\xd2P\xfbY\xdb?\t)\xa40Jr\x0ej\xb1\x06q\xf9\xdc.a\xd92^\x82B\xe0a\xea\xcd?^~\xfc\xd7\xcb\x7fyG\x88\xa05\\\x11\x05\xdaH\x05\xfar\x03\x1c\x94\xbcd\xf2\x9dn\xa0\xb00\xd7J\xb6\xcd\x15\xe9~pc\xfc|n\xad\x8fn8~\xe1L\x9b\xbf\xf4\xbf\xfe\x95i\x83\xbf4\xbcU\x94w\x93\xe1G\xcdĺ\xe5T\xc5\xcf\xef\bхl\xe0\x8a\xdc\xdbi\x1aZ@\xf9\x8e\x10\xbft\x9cv\xe1W\xbd\xf9\xe8@\x14\x15\xd4ԭ\x87\x10ـ\xb8~\xb8\xfb\xf6OO\x83τ\x94\xa0\v\xc5\x1a\x83\b\xf8\x9fE\xfcN\xc2B\tӄ\x92o\xb8Q\xbb\x1aD<1\x155DA\xa3@\x830\x9a\x98\n\bm\x1a\xce\n\xc4;\x91\xab\x1e\xa40J\x93\x95\x92u\amI\x8b\xe7\xb6!F\x12J\fUk0\xe4/\xed\x12\x94\x00\x03\x9a\x14\xbc\xd5\x06\xd4e\x04\xd4(ـ2,`ٵ\x1e\xef\xf4\xbeNm\xcc6\x8b\v7\x8a\x94\x96\x89\xc0m\xc1\xe3\x13J\x8f>\"W\xc4TLw[\r\xdb#T\x10\xb9\xfc\x1b\x14\xe6r\x0f\xf4\x13(\v\x86\xe8J\xb6\xbc\xb4\xbc\xb7\x01e\x91Uȵ`\xbfG\xd8\xdan\xdcNʩ\x01m\b\x13\x06\x94\xa0\x9cl(o\xe1\x82PQ\xeeA\xae\xe9\x8e(\xb0s\x92V\xf4\xe0\xe1\x00\xbd\xbf\x8e_\x90xb%\xafHeL\xa3\xaf>|X3\x13$\xaa\x90u\xdd\nfv\x1fP8ز5R\xe9\x0f%l\x80\x7f\xd0l\xbd\xa0\xaa\xa8\x98\x81´\n>І-p#\x02\xa5\xea\xb2.\xff.\x12u0\xad\xd9Y\x1e\xd5F1\xb1\xee\xfd\x80\x02q\x04y\xac\xa88\xc6s\xa0\xdc\x16;*\xd8O\x16u\x8f\xb7O_\xfbLɴ'J\x8f7\xc7\xe8c\xb1\xc9\xc4\n\x94\x1b\x87\xacia\x82(\x1bɄ\xc1?\n\xce@\x18\xa2\xdbe͌e\x83\xdfZЖ\xdf\xe5>\xd8\x1b\xd4:d\t\xa4mJj\xa0\xdc\xefp'\xc8\r\xad\x81\xdfP\roL+K\x15\xbd\xb0DȢV_\x97\xeewv\xe8\xed\xfd\x104\xe2\bi\xbd\x16yj\xa0\x18H\x9a\x1d\xc6VA]\xac\xa4\x1a(\x19;d\x88\xa3\xb4\xf0\xdb洈U\x8b\xfb\xbf\xccq\x99m\xff\x1eG[~\xb3+k\x05\xfb\xad\x05T\xa6N\xfc\xe1P_\xa9\x9ej\x1f6\xcbF\xfb\xd4\x1dE\xb4m\xf0\xbd\xe0m\te\xd4\xeb\a\x1b\xcc\xd9\xc6\xed\x01\x144z\x94\t+D\xd6\xfaؽ\x88\xeeWT\xe0T\x01\x11\xd2$\xe01\xe1\xe0\x11&\x10\x03I\x9a`G\x03ubœ[&D\xb4\x9c\xd3%\x87+bT{\x88F7\x96*Ew#\xd8\n\x1e\xc0\x8b\x90\x15\x81xU\xc3Y\x81$\x8f\n\x05\xf1\xf5\xe7E\x15\xd3VQ\x86]>HΊ\xdd\f\xben\x93\x83\x82\xb4z\xd9\xf5;$K\xa8\xe8\x86I\x95\x12\x03\xa9\xb0kϞwjZZ-\xe9\x81\xec۸\xcc\r'\x91UI\xf9<\xc7\x10\x9fm\x9f\xce:\x90\x02\x1dʸ\x15Omo\xbb\x97@\xe0;\x14\xadI,\x93\x90\xb2E\xd3$\x15i\xa46\xe3t\x1fW]\xa4\xef\x1c\xa5~\x9c`\x9a\x83\x9d%Y\xdd5\xaf\x84\x03Q-\x0e\x06\nY\n\xb0ۨ-Q\xbb\xbeJ\xb6\xae\xef(RȒj(\x89\x14\xa33#\xbb\xb4\x1c\xb4\x9f\xabD\xce\xe8\xf4\xd0E\xb7\x7f\xf4x\b\xa7K\xe0D\x03\x87\xc2Hu\x88\xcc\x1c\x94\xba\x96\xa3XGP\x99ЦC\t\xe860\x01\x92XN\xdfV\xac\xa8\x9c\x87a\xd9\x13\xe1\x90R\x82\xb6\xda\x04]\xe6\xdd\xd8&\xc9\x1c\xf9\xfd$Sڣk3b\xb5\x0f/\xa5Q\xba\x96\xa1\x86\xbb\x96Dm\xa7{\x0ft\x8b\xffn\xe4\xe4\xb6\xff\x7f\"6\x18\x93\x13\x98vB\xfe\t\xba\x9f\xd9<=ʷ\x18ၾ$w+\x02ucv\x17\x84\x99\xf0uN\x12(\xe7\xbd9\xfeĴ9\x9e\xe93I\x93#\x13g\"L\x9c\xe2OH\x174\x19O\xdebd\xd3\xe4\xaf\xfdQ\x17\x84\xad\"\xd2\xcb\v\xb2b܀\xda\xc3\xfeI\xaa>P\xe65\x90\x91c\xf5\b\xe6\tLQ\xdd~\xb7.\x8e\xee\x92`\x99x\xd9\x1f\xec|\xe3\x10A\f\xcd\xf3\f\\\x82\xf12SPc\x1cN\xbe\"6\xbb/\xe8T_\xdf\xff|\x18+\xef\xb7\f\xce;\xd8Ȍйv\xbd\xb7\xa3\xfe\xfa|T\x10~A\x1f(\x06U.\xe7rA(y\x86\x9ds]\xa8 \x96>4tΘ^\x01&\x7f\x90Ϟa\x87`\xd2ٜÖ\xcb\r\xae=C\xc2\xf5O\xb5\x01\x0e\xed\x9a|X\xec\xf0d? \"0\x86\xcfe\x03\u05fc($r'閩KB\v\xb8?a\x9bY\xacҟ\xa3\x9f\xfaD\x0e\xf8I;ZZ\x89\xa9\x98\xcfij@\x99\xc9%\xa8k\xdf(ge\x9c\xc8\xc9ȝ\xb8 \xf7\xd2\xd8\x7f0@\xd3\xc8(?K\xd0\xf7\xd2\xe0\x97\xb3`\xd4-\xfc\x9c\xf8t3\xa0\xa0\t\xa7\xe5-\xc2\xfa9?g\xd3,\xb7E\xdc3M\ue10dW\x1cJ2\xa7\xc2\xf4\xae\x9b\xceMT\xb7\x1a\xd3uB\x8a\x05\xda\xcc\xe4L\x1e\xdfR\r\xd0\xfd\xe2I\xfd\x84_\xad\xb1p\xbf\xb8$3\xa7\x05\x94!\xb2\xc4\xec'5\xb0fE\xe6|5\xa85\x90ƪ\xf0<\x8e\xc8T\xac~7DZO\x9e\xf5\xee\xb7\xef\x8b\xe7\x98/XX\x93\xb3\xf0\x10\x8c\xac3p\xe0uw9\xbf\x9f\x85\x95ٌ^\x81\x13f\xbb\x8e$Gǻ\xe6 \xe5\x05\xe8@+\x8e.\xce,uiY\xe2\x11\x1a\xe5\x0fGX\x94#x\xe1X\xd5\xd0[\xbb3\xc15m\xacZ\xf8okiQ\x9a\xfe\x974\x94)}I\xae\xf1\xa4\x8c\xc3\xe07\x9f\x87\xeb\x81ɘ\xb2\xb1SY\xfe\xd9Pnm\xbfU\xe0\x82\x00w\x9e\x80\\\x1d\xf8E\x17d[I\xed\xcc\xf6\x8a\x01\xc7\xf3\x8a\xf7ϰ{\x7fa\xa7\x9f\x9d\xb2\xafd\xde߉\xf7·8P\x18\xd1ᐂ\xef\xc8{\xfc\xed\xfdK\\\xa9LN\xcd\xec6`њ6y\x1c*\x92\xc9\xfa\xae\r8\xa6\x9f\x9b\xef\x92\xf2\xdeɞ\xdam\x16\x8b6R\x9b\xcf\xe9\xbc\xe1\xc8z\x1e\u0088\xa1g\x9cȱ\xcdF\f>\x8f\x16\xf5\xbdu\"W\x06\x94\xcf%:\x1b\x10\xe2\x8f\x17Ff\xa9S\x99\xfebc2\x90\xc6\xfc\xaeE\xf0\f7\xb9\x83\x9b\x9c%\x1e\xe3\xb0Z\xbc\x1c\xe9\xed\xdf~\xef\xe53\xad\xe4ڿ\xfb\x1bym\x87\xba\x90uM\xf7O5\xb3\x96z\xe3F\x06\x9e\xf6\x80\x1c\xf5պEyε\xc8\x1d\x0f\xe1\xf9喙\x8a\tB\x83\xda\x00\xe5\x19\x8a\x92F\xa6rةVQM\x96\x00\"\xa6\xe8\x7f\x04W\xa2f\xe2\x0e' \x1f\xcf\xe0zDt\x9d\xd3ٽ\x894\x89\x94\x8f\x1f\x9c\xc9jdI\xb6\x15(\x180\xc6a\xde\x1d=U!M/eq\x84C\xda\xc8\xf2'MVLi\xd3_\x82&\xadΥ\xf5\x91\xe4\xb3\xeb\xfe\xcaj\x90\xad9'\x82o\xbbi\x06g\xcd5\xfd\xce\xea\xb6&\xb4\x96\xad3\xe6\x86\xd5\xf1TףwK\x99\x89\xc7V\x98\xbf1Ғ\xa0\xe1`\x80,a\x95>\xefM\xb5B\n\xcdJP\xa1J\xc1\x91\x8dI+\x98+\xcax\x9b:%J\xb5c#`q\xab\xd4I\x01\xf0\x177\xb2\x97w\xac\xe4v\x88\xa0̽\xe3A\x1a\x10\xb6\"\xcc\x10\x10\x85\xc58(\xa7\x92q\n\x8f\fD\r\xcb\xd5sy\n\xdc6\x10m\x9d\x87\x80\x05\n$\x13\x93)\xb7~\xf7O\x94\xf1s\x90\xcdr\xde'\xa9\x1e\x81\x96\xa7\xe4h~\xed\r' t\xab\xf0\xf0\xdf\xe9\x8e-\xe3yk\xb6\x94#\x9c\xb6\xa2\xa8\x00\x95\x90\x18\xea\x06\a\x9e\tm\x80\xe6\xf2\x82\xf5\x8aZ!\x98X\xe7\xd1.;\x11\xda5\x87\ua954\x1c\xe8\xf8)d\xd7,\xae\xdf@\x13\xfd\xdaM\xf3BM\xd4\x11\xc1\x1d\x9b#\x1d\xb2)j\x95\x16\xa1\xc6@\xdd8\x91\x93D\xb5\xa2o]Π\x88\x8e\t\xc3\xfd*^3\xbef\x82e\xd0v@\xd7;\xc1L\xdfy\xb4 \xce\xea<\xda\t\xa2;pJ\x86\xedn\x00\xc0\nh\x88Cp\xed\x91k\x8ep$\x97@hYB\xe9r\x97\xd6\x15\xf1a\x89+|\x1b)nH\xee\xeexO0\x8b\xb2\xa1\r\x82N\xccê\r,Z\xf1,\xe4V,0\x18\xd7G\xeb\x90\x13\xb3T/\x9dޜ\xac\x8c\xe6\xf5K\xbe\x9a\x9e\xd3BC~\xcd\xe7\xa9\xe0?\x9dA\xcbd\xf3\xcdQ\t\x8f).\x98\xd3k\xae\x00{\xe4\xc7\xd9UL\xcd?1\xd8\x1fJ߸b\xe9\x17\x95\xc5ݥA\xf5\x9c\xc2m\x05\xa6\x02\x15J\xb3\x17X\x92^N\x9e\x90v\xc1K\xac\x93\xb3L\x15\\dW\xfe\xb9W9\x87\xd1M\xcb\xf9\x85\xe5m\xda\xf2d8l$\x8a\xd8!geՏ\xa5=\x86\x9c\xea\x8bl<\xf6+-\x86\xf5\x85\xb1\n\"\x14\x18\xca0\xb3\xa7qj\xbfXX\xda;\xdf\x1f\x96S`\xfe/,\xff\x0f/=̨\x94\xc8Gcn\x95fDb\x02V\x82\xc1zh\xec\xea+|?_\xe8\xfbc\xe1\xd4@\xfd\xa5\xf1\x123\xea\xc2f\xa05\x01g\xaf\xde\x04\xadA\xab\x9d+\x10\xed\x80\xcf\x19\xda\xf1ׅ\xbb\x05\x11\xc0\xa4\xf8\xf5k\x05A|}\xf5>\xd3\xe4\x9fI%\xdbDU\xdf\x04\xcaf\xaa;\xe67<(\xf4\xf0\a\n`\xe8\xe6\xe3\xe5\xf0\x17#}\xd9\af\xd1\x12\x800(\xea2\xb3L\x94l\xc3ʖ\xf2 \xb5\xdd\x1d\x02\xc7@\x1d\x9f%\xa0IE\x04\xe3\x8e\x01\xc3\xf8\x01Ñ/\x8d;\x969Z\xc5M\xfb\xa2y\xd5!'ׄ\fk>F\xac\xe1\xb1\xc7\x17\xafR\x05\xfb\x87\xd4z\x1c_\xe1\x91\x13I\xccTs\x9cPÑY,\xf6\xe2\xf3\x96\x9c*\x8dcb\xee\xb3Ud\xbc~\x1dF\x16~\xe6k.\x8e\xc1\xce\xd9\xeb+ް\xaa\xe2mj)2+(^\xaf\x142/\xfa<\xa9\x14`>`\x19\xaf\x82\x98\xad}xQ@sҖfk\x1a\x8e\xa9d\x98\xa5N\x9e\x98\xbdY\xad\u009bU(\xbcm]\xc2$\x17M\xfexL\xe5A\x8c\x93~\xa1M\xc3\xc4\xfa\x90)rYg\x92m\xe6Y\xe6~o!\x03\x9e\xe9\x873]t8\x12\xfa\xba\xeb҉H2\xa4-\x990\xf2\x92\\\x8b\x9d\x87\x9b\x80\xd3\v\x1f\x854\a\x17\xd9첶\x8c\xf3\xfem-\x04;\r\xcaߙԴv\xab\x1a\xf3\xf6\x93t\x95j\xe0\x94\x9f\x148~ك\xd1ώ\xbe\xa5\xe7_\xb7ܰ\x86\x83\xf5\xe86\xacL\xde!3\x15\xec\"\x92\xff&\xf1\x86\xd4r\x87\x90\xbe]\xf4\x9eQ\xec\x9eq\x926\xbf\xc9\x13\xb6\x97Q\xcc~\\\x11{\x06\xcdrE\xf1\r\x8b\xd5߰H\xfd\xad\x8b\xd3g8k\xe6\xe7\xe3\x8a\xd0O>\x81\tG\xfd\xf7\xb2\x84\a\xa9\xcc\\p\xf2\xb0\xdf?q\x92\xda\v\xd8$/\x89\b]\x13\xbb\xc4\x10Ç\x17\xa7m*}\xe8\x19\xdc\xe9_di\xd76w\xc6\xf2\xb8\xd7\xfd\xe0\xae\xf2\n\x14\b\xf7\xcc\xc7\x7f>}\xb9\x8f\xf0S>\xaf\xf7\x8c\xf7\x9e\x97p\x1eL\xe9\x91\xe3\x8f\xe6|1\x93\xc3\x16\xfa\x00\xaf|.B\x1b\xf6\x1f\xf8\xaa\xdb\v\xd2A\xd7\x0fw\b#\xf8i\xf8L\\\xac\xa2\x88'\x96K\xb0\x16+\xa2jT,\xeeV\x03\x88Ê\xdf\xfe3JP\xba'\xb3\x82\xc5d\xa1\xc6\xcb\n\xdeÝ[\xc7\xd8,\x9f\xac\xd3(vD:\x8e\xac\x98*\x17\rUf\x87l\xa3/\x06k\bff*\x9d3\xaaX\x0f\x9f\x01K\xa27\xbc\xfe\x85g\x91\xbbfxڻ\x8f\xbbS\xd61~\xffd\xf6\xe6\xc9+\xaec\xdcb/\x10S\x89\xcf\xc9\x02\x93WK\x93yM\xf4\xf0\xed\xa4\xb4\xcbc\x1c=\xad\xe7l\x14\x1dRM\t0v<\xaa:-h\xa3\xabēK/\xd3u\xf8\x1a\x99\xa1\xa6}\xc9&\x1d\x80\xc1>YQ\xf5\xb4\xd5\x16\x82>\v\xdbFi\xc5a)\xddnm\xb3\xab{a\xfc\xa2\x97)x\x9b#\xe1\xcc\xe7\\N~\xc8šgD\xfd`\xf6˪\xb6CL\x9dp\x18<\xeb\xdae\x14\x19O;\xb1\x99π\xe4\x19\x8c\x13\x9e\xfe@|\xe5\xe2\x8a$_\x04\xc9|\xf5\xe3\x0fE\xf4\x84V\xd3E\x05e\xcb\xe1\xd47\xff\x9ez\xe3\xe7_\xfd\v\xb3e\xbc\xfbg\x91\xdd3\xd0\xd6g\x1e\xbe/\xe8)\xe1!\xf7)9\xe6\xf0ap\xe0\x9e\x17+\xdcK\x94E\x01Z\xafZ\x1e\xaa\x94\n\x05\xd4@\x19\xba3\x1dW|T\x9dM\xdbpIKP7R\xacX\xe2\x84d\x80\xd6\xff\x1at\xde\xe3\xd9\x02?\xb6\xaa{\xdaq\xf2Y\xbc\x17i\xae\x86*\xca9\xf0O\x8c\x83\xfeYn\x85]W\x86@>\xa4\xc6\xf5\xeee\x15\xad\xb2f}GD[/\xad\x93\vƌ\a\x8b+\xa9\xa6+\xa4\x1dޙ0\xb0\x86T|\xbdU\xcc\xc0SC\x95\x06\\Q\xc6\x0e~\xdd\x1b\xe2\xa2\xcf\x15\xa7kW\nW\xb2\x82\x1a\x88\x06\x18g\x18[>\x8e\xd7\b\x8b\xef\xb02I\x8e$\xbd\xb2\x85z\xecJƨX\x8f=/\x9a0\xd5\xc9\aF\x9dE.hc\xf0\x02\f\xd2\x11\x89h<\f|\xb4w\xef\x8d\xd1\x01\xd8qN\xf3e̾`N\x1bZ'\xa2\x84y\xbdss\b\x06\x9f\x05Ve\xaf\xee\xae\xff\xc0b,\xb0#[\xaac1u\xd2\xf7\xee`;0\xe8\xaa[\xd0P\x12\u0600 V\x14)\xe3PNq\xeaWL$\xab\r\xa8\x9ft\x84\x83\x95\x80\x96ş\fU&.\xfdЏYIUSsEJj`aG\x9f溥\x9fIU\xea\xc4\xe3@\xbc\xd9\xe6ţ\b\xd7n\xac\xf5s\xf7\xd1jК\xaeC\x10\xba\x05\x05d\r\xc2\xe2=\xe6\x16\x93\x1eS\xb8\xd2\xe7\x8dE,-\xb5(\xa4\x85i\xa9\x9f\xc0\xb9p\xf1\xf44\xbcO\x8cQ\xeczTE\xa7U\x85\xbf<\xf8\bT\xef?w}\x80\x8bO\xfd\xbe>I\xecv\xec\xceF\xa8+\xf0\xc4\a\x8f\r\x8b\x91uJ\xa6\x8dę\x8f2'\x95\x94\xcfYn\xf6\xe7رK'1\xe1X\t\xafL.ekz~\x8eGxb\x99\xf8\xfc\xe7+\xdb\x17\x84y\xed.P\x8d\xe5V\xf3<\xbd\xcf\x03H1\xbc\x95\x86\xf2`d,_\xc6\x0e\xd5\xc4\x03\x02O\xe1\xf1d\xcew\x17\xfb\x90\xf7^e\xef`W\xddS\x9e^\x13t\xd7\xc7\xc7Ҫ>\xeb\x97\x04\x12_\x01\xed|\x92\xb17\x17\xe7\xec\x1fB\xfd\x84\x8b\xca\xc0\xf1\xe7\xae\xf7\x18\x1e\xdd2\x9d\xc3\f\"\x1di\x12\f>L\x15%ㄥOx\xa9ME\xf5\x9c{\xfa`\xfbD\xb7\xa3g\xae\xa2\x13\xfa8\"\x95\xe9{\xae\vr\x0f\xdb\xc4W\x87,<\xfdB\xa9Jt\xb9\x13\x0fJ\xae\x15\xe8C\xa6[\xe0}F&֟\xa4z\xe0횉/\xe3\x95\xdfS\x9d\x1f\xa82\xcc2\xad[Ob\xecM\xb0q\x89\xdf\xe6G\x8f\xff\xc0\x04\xe5\xec\xf7\x94.\xef\xff87Ä\xbek<\xf2N\xb1P\x01\xf1s\n\xd0k\xe8\x9ft\xcf\xfc\x84y/ɽL\x8a\xb1? fC\xa0L\x93%h\xb3\x80\xd5J*\xe3\xf2\xf7\x8b\x05a\xab\xe0 Y\r\x81q\xa2{͞\xb0T\xe2=\x1e\xbd\x05\x87e\xe5S\x89\n\xad\x0e\x86\x9c5ݹ\x8c$-\n\x1b\x13\xc0\amh*6y\x91\x9e\xc6P\xd5\xcbJ\x8e\n\xb9\xeb\xf7\x8f9\xbe\xa8>\x10\x9cC\x1d^gw\x06\x9d\x8f\x9di\r^\xcb \xdab\xef\x14eB\x9c\x1a\xbb\x1b\x0f\xbb\xf3L\xcd\xd7\beL=\xfa\xfd\r\x1e\xe2\xf6\a\xac\xbe\x93%[QQ\xb1\x1e\xbd\xd0V)ٮ\xab\xc0\x9bc\x0e\x11)[\x8c\x9c\x1bT\x05:\xfc\xc7!\xa6U\xa2wh\xe7k,ƴt\\\uee0f\xf2\x02E\xad\xba\x8b-\x9d\xaa\x9a\xb0\xf9\xd9Y\xc2\x11\x88\xb3\xb6?\x01\x91\xea\x9d(&\xaf\xe0\xf8@\x9bM\xdc՝\xc2P\x12\tQ\x1b\xbf\x1a\x12\"\xc41$\xf4}\x89.\xe2\xf9a02棜\x88\x8ei'\x06\xb78\rj~\xd3}'h\xe8\xee\x1c\x87\x0e=\b\xfeNJ\xbb\r \x1c\x13\xf9\xe2\xdc\xe9\xb8\xf7ǍX7\xd1ۺ=9v\xfd\xb6\ac\xef\n\xa4\x8db\xbbiB\xbc\xf9\xf7l\x95\x92\x17\xf7\xbf3-9\xfc\xc3\xc1\xafo|\x95qK\x95`b}\x12F~\xf5c\x13\xf1\xbc\a{Έ>\xac\xfc\xd5b\xfa\xa4Y:\xf8\x88\f^\xf6\xf0\xecg\xf2_\xfe/\x00\x00\xff\xffP\a\xb5\x16Cm\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xec=]s\x1c)\x92\xef\xfa\x15\x84\xeea?B\xdd^\xc7}ą\xde|\xb2gO\xb1\x1e[ai\xf4\xbctU\xb6\x9aQ\x15\xd4\x00\xd5r\xdf\xde\xfe\xf7\x8dL\xa0\xbe\xba\xe8\xa2Z-ygǼت\x86$\xc9L\xf2\x03\x12X,\x16g\xbc\x12\xf7\xa0\x8dP\xf2\x92\xf1J\xc0W\v\x12\xff2\xcb\xc7\xff6K\xa1\xdelߞ=\n\x99_\xb2\xab\xdaXU~\x01\xa3j\x9d\xc1{X\v)\xacP\xf2\xac\x04\xcbsn\xf9\xe5\x19c\\Je9~6\xf8'c\x99\x92V\xab\xa2\x00\xbdx\x00\xb9|\xacW\xb0\xaaE\x91\x83&\xe0\xa1\xebퟖo\xffk\xf9\x9fg\x8cI^\xc2%3\xd9\x06\xf2\xba\x00\xb3\xdcB\x01Z-\x85:3\x15d\b\xf4A\xab\xba\xbad\xed\x0f\xae\x91\xef\xd0!{\xeb\xdbӧB\x18\xfb\x97\xde\xe7\x8f\xc2X\xfa\xa9*j͋N\x7f\xf4\xd5\b\xf9P\x17\\\xb7\xdf\xcf\x183\x99\xaa\xe0\x92}®*\x9eA~Ƙǟ\xba^0\x9e\xe7D\x11^\xdch!-\xe8+U\xd4e\xa0Ă\xe5`2-*K#\xbe\xb5\xdcֆ\xa95\xb3\x1b\xe8\xf6\x83\xe5g\xa3\xe4\r\xb7\x9bK\xb64ToYm\xb8\t\xbf:\x129\x00\xfe\x93\xdd!n\xc6j!\x1f\xc6z{Ǯ\xb4\x92\f\xbeV\x1a\f\xa2\xccrb\xa0|`O\x1b\x90\xcc*\xa6kI\xa8\xfc\x0f\xcf\x1e\xebj\x04\x91\n\xb2\xe5\x00O\x8fI\xff\xe3\x14.w\x1b`\x057\x96YQ\x02\xe3\xbeC\xf6\xc4\r\xe1\xb0V\x9aٍ0\xd34A =l\x1d:\x1f\x87\x9f\x1dB9\xb7\xe0\xd1\xe9\x80\n»\xcc4\x90\xdcމ\x12\x8c\xe5e\x1f\xe6\xbb\aH\x00F$\xaaxmH8\xda\xd67\xddO\x0e\xc0J\xa9\x02\xb8\x80vX4\xb6\nu%\xa0\x80\xe6\f\xddN\x8d\x16FH\xb6\xae\xd1#]2\xd4\x12Q\x19\x11\xd2X\xe0\x11a>\x01\xef\xe0kV\xd49\xe4WEm,\xe8\xdbLU\x90\x87E\xa6Q͜\xca\xc3\x0f\a!\xfb\xf8\xa5\x10\x19 \x1f2WiA\x8b<1\xd1nC\x99]\x05n\xcd\tY\xed\x87\xd0\xc6(\x93\xbaŀņ\xe7\x7f<\xbf \t\xe8\xf7\xde\xef\xc70\xae\xa1!\xd3,\xddL\x16\x7f\xbc\x85\xb0PF\xa8;\xa9\xa3f\xf0\x9dk\xcdw\a\xb8\xde,\xa6\xbd\x00\xdfc\xb0\a\x9c\x97\xa1\xda7\xe2\xfd\xb0\xff\xdf\"\xf7O\xcboC\x8b\xce\\H\xe4s!\x8c\xed\xb1ٸU,$\xebX\b\xe9\t$\x1dLT\x93S\\\xfd'!\xe6I\xe7Nl\xb24\xb2\xe9'\xc0\xbf\x14%7J=\xa6P\xef\x7f\xb1^\xbb\x84\xc52\xda\x18a+\xd8\xf0\xadP\xda\f\x97I\xe1+d\xb5\x8dj\x16nY.\xd6k\xd0\b\x8b\x96\xf9\x9b]\x81C\xc4:\x1c\xbe\xb0\x8eʊV\x18\x8c\xabe:\xb2\x94\xa8\x11\x1b\n\x05\xa8Q\xa8\xce\xc1\xc1Ђ\x1c\x88\\lE^\xf3\x82|\t.37>\xde\xe0\x17\xd3j\x13\x02\xb1\x87\x7fT\xaa]q\x0eM\x18$2\xb1\xb7\xea\xa5$\xa0\x8f_bl\xb4_5N\x89\xb0\x94p\xb0od\xa6\xae\v0\xbe\xbb\x9c\xdc\xe4V']\xb4\xccrk\f\x05_A\xc1\f\x14\x90Y\xa5\xe3\x14J\x91\x03WR\x95n\x84\xb8#Z\xb6\x1fm\xb5\x83\x99\x00\xcb(\xc4݈l\xe3\xdcW\x144\x82\xc5r\x05\x86VExU\x15\x11\xd3ՖI\xe1\xf0\x9dM鍶$h\x90!ܘ.iK\xa2~n\xcb(\xd9۹٧\xfa\xf8:\xff(\xbe\xbf%\xa2\a\xabs\xa4\xb0Oh\x12F\xfb\x05\xc9\xf3!Jz\xa4\xb8\x00\xb3\xec\xac\xce\t\x1b\xbe\xa60\xb4\xe7?\xeem\xa5\xec\x11\xe5\xd7Ż\xe3&\xcc\f\xd6MΩ\x97e\\\xd3Ϳ\b\xdf\xc8d\xddz\x8b5\x8bg\x1f\xbb-/hW\xc03$\xbf`kQX \xa7j\nQ6\x83s\xa7$P\xaa\x05f\xb4Il\xb3͇f\xef(\xa1ŀVC\x00\xceA\x0fQ\x0e\xf1 \x01$k\\\v\xda4\x15\x1aJڌ\xa5H\xb2\xfb\x85\\\xc1w\x9f\xde\xc7c\xcfnI\x94ԽA%LZW\xde\r\x1c\xa3.\xae>T\t\xbf\x90\xbf\xd6\x04\x82n\x13\xfe\x82q\xf6\b;\xe7bqɐo)\x8b\xff|\xf8*\fv,s\xf6^\x81\xf9\xa4,}yQ*\xbbA\xbc\x06\x8d]O4A\xa5\xb3$H\xc4n\U00088ce5(\xa8\r?\x84a\xd7\x12C2G\xa2\x19\xddQ\xae\x90\xeb\xd2uVֆ\xb6Z\xa5\x92\v\xb7,6֛\xe7\x81\xd2=\x16\x9c\xa4c\xdf\xe9\x1d\x1a#\xf7\x8b\xcbZ*x\x06yآ\xa3t\x1an\xe1Ad3\xfa,A?\x00\xab\xd0,\xa4K\xcb\fE\xedG6_\xbc\xd2=\x87n\xf9\xbax\xacW\xa0%X0\v4k\v\x0fŪ2\x91.\xde&\x8c䜌\x95\x05\xce\xf5ĚAZ\x92\xaaG2r\x0eWO%\xd63\xc9D^\x04\xb9]IR\xd0Ml\x9dg\xbdf\xca\xcd1*\xa63\x16\xe7\x02\x94\x9c\xb6\xd6\xfe\x86\x96\x9ef\xe3\xdfYŅ6K\xf6\x8e2{\v\xe8\xfd\xe6\x17&;`\x12\xbb\xadh\x95\xfd\x97Zly\x81\xfe\a\x1a\bɠpވZ\xef\xf9j\x17\xeci\xa3\x8cs\x1b\x9aM\xbb\xf3Gع\x1d\xe5\xa4n\xbb\n\xeb\xfcZ\x9e;_fO\xf14\x8e\x8f\x92Ŏ\x9d\xd3o\xe7\xcfu\xeffH\xf4\x8c\xaa=Q.y\x95.ɔ7;'\xd0\xc0`=8DظI \xc5\x00a\x8a\x02ɢ\\)\x13I\x16\x89\xa0\x95 \xe87\xcaX\xb7\x0e\xd9\xf3\xf7G\x17*UX\x9cd|mA3c\x95\x0e)\x99\xa8\xf8S\x96\xe2\xbb\xe5n\x03\x06\xfc>\x94_\xf4t\x801\x8a=ou\x83\xb3*\xe7n/\x8c:\xe2\x19yOԶ\xd2*\x03\x13͋hK\xa2m\xeaQp\x9f\x0eͺ.w\xd1\xdf:Ik\xa7,J\x872ϑG\xd2\x1d\x11\x19}\xf8\xdaY\xa2F\xed\x82\x7f\xa7H\xeb182:\xafQ\x96|\x98\x0e\x9c\x8c\xee\x95k\x1d\xe6\x98\a\xe6\xc2-\xfdP\x93Ι\xe3u4\xa2\xfc\xcf\xe6ڔB^SG\xec\xed\v\xbaC^\x8b\xc7ң\xc6\xca\xf1N\xfaU\xe8\xac\xe5^\xf3\xc1\xe7\xd4)\xda\xf8\xd1\xd0c\xee\xfe\x9e\by\xd7R\xd9\xce2\xceL'\xbaR\xf9\xef\f[\vml\x17\rs \xb1j\x14\xd4\x11\xa1\xa7\xfc\xa0\xf5ё\xe7g\u05fa\xb3\xa0\xb8QO>qzN\xbc\x1dH\xba\xe1[\xf0\x99\xab 3UKZ\nC=\x80\xdd̀\xe8X\xe3\xac@\xa2\xbd\xeb4\x96u\x99N\x90\x05I\x92\x90\x93\xebf\xdd&?p\x91\xb6nŎc\xab=\x94\xc39V\x8e\x9fG!\xc1\xb3\x9bN_\U000af8acK\xc6K\xe4!\xb9\x1d\xa2\x84&\xa3ޱ\xbbI\xfb\xc4\x16d\xb4\xac\xc2YV\x15`\xc1\xa7m\xce\xc0#S҈\x1c\x1a\xd3\xefE@I\xc6ٚ\x8b\xa2\xd63\xb4\xeal\x92\xcf\r¼69}d\x95\x8eȂH\x94\xb8\xce>\xc3\v\x9e\xd6\xf8\x95\x9e\xe7Ǧ8\x8c\x1a\xe6\xfb\x8b\x95\x16\xca\x1d\x068\xbd\xcb\xe8ӎ\xb9\xdc}\xf7\x19\xbf\xfb\x8c\xdf}\xc69\x1d}\xf7\x19'\xcaw\x9f\xf1\xbb\xcfx\xb8|\xf7\x19S\xcaw\x9fq&\"\xdf\xcagL\xc1pAk\x9c\a*$a\x95\x98\n1\x85\xf6D_>\xe9ǟ\xd58I.\xf3\xf58ȑC<\x91\xe3\x171\xaf\xa35^Mr3\xce\xc00w\xdc)\xca\x04\x87\xf9\x04\xa7g\x02\x02\xa7?=s}\x10\xf2\tO\xcf\xf8!\xa4E\x18G\x9d\x9d\tD\x9a\x7fz\xe2\xc2'\x11\x95\xc0\xc3V\x8aK\xff\x88\x8d1&I\tx|\xe3\xe4\xf7\xbd\x8c\xc9\x17\x90\xa5W9\x913K\x9eFY\x7f\xfe\xc7\xf3_\a\x8bN˔(\x1b\xf6i\xeb\xd4xL?b,\xdfM\x8d\xecg\xa9\xfez\xa6\xc2Ie?\xf5DMC\xe4\b\xbc\xbeX\x0f\xa8\xfck\xd27\x16\xcaϕ\xb7\x96'8a\x7f=\x02/\xe9\x8c=7;\x99m\xb4\x92\xaa6~M\ba\xbd\xcbܽ\x03\x01dL\xd8G5\xc8\x7f\xb0\x8d\xaa#\xa76&H\x9b\x90E\x9bF\x90^R\xadO\x8c\x00˷o\x97\xfd_\xac\xf2)\xb6\xecI\xd8M\x04\x18\xddG\xc1\xf3\x1c\xe3\x82\u0381\x1e\xaf\a\xc2UIC\xa1\x8c\x00S\x9aIQ8\x89\r\x10z\xf2\xca>Wnu\xf0h\xbfiz\r+=\x11wn\xfam\x93-9\xed\xbe?#\xe9\xf6\xa4G\xa3\xbeYZ\xedqɴ\xa9+\x94\t\x89\xb3\xe9\xe9\xb2)lu%=I69BNM\x88\x9d\xbb\x02\xf1\xa2ɯ/\x93\xf2\x9aL\xb3\xb4\xf4ֹ\x14{\x95T\xd6WN`}\xbd\xb4\xd5\x19ɪ\xa7?\xf5\x92\xbe\x96~tveڲ\xcc\xe1\x84Ӥ4Ӥ\xa5\x9b\x94\x01\x1f5Ԥ\xf4ѹI\xa3I\x9cL\x9f\xae\xaf\x9a\x16\xfa\xaaɠ\xaf\x9f\x02:)m\x93\x15\xe6&y\x8e_r\x18ʴ\x03P|\v\xe1|.\x99\x94\xee\xb9\xe6ϊ;?\x0f`\xa1\xb0\x047\xf5\x15〲.\xac\xa8\x8a\xf6>\xb6X\xc0\xb9\x81]sY\xd1ϊ\x8e\xc8\xfb\x9b\xba>\x7fi$~9\x88j\xb8aOP\x14\x8c\xc7\xe6\xe6\x1e\x152w\x0fh\xa6\x16\x80\xb6\x11g\xb9\xbf\x8c\xc9_\x1ez\xe1\xa6\v\xdd\x06@\x16\xb6\x8c-\xf5qy\xf8\xa6\xaf\x83\x06,U\x8f\xedy\xe6.ޠo\xbfԠw\x8c\xee\x1dk|\xb3\xf6P\xa9\x9f\xe8\x06\x03Ӡ~\xbc:<\xb4g\xb2\x17\xe0\xb4ꁽ\x93\xce#\x18\xe2DmP\xef\xb4\x01\x1d*U\x8cӢ\xfdD@H\xd5@\x884Mq\xfe眲|\x89\xf0\xee\x14\x01^\x92\a4\xcf{\xfd\x86\xa7'\x8f=5\x99\x9e\x8c\x92tJ\xf2%½9\x01\xdf,\x7f5\xfd\x14\xe4\xfc\x8d\xe7\x17>\xf5\xf8R\xa7\x1dgP/\xf5t\xe3|ڽ\xd2i\xc6W?\xc5\xf8\x9a\xa7\x17g\x9dZLNϚ\x95q0'\xb5\xea\x19\xc7\xed\xd2r\t\xa6O!&\x9e>L\xcc4H\x1b\xfc\x91\xc3N<]8\xffTa\"\x7f\xe7L\xe9W>=\xf8ʧ\x06\xbf\xc5i\xc1\x04\tL\xa82\xffT\u0cf7\xa4\x94\xceAOn\xfb͑\xdaIyM\x8d\xe5\xfa\x88\r\xf6\xb5\xc2m\xb2X\xab\x17\x03\x90Y\xf2\x17\xf9ӣ\r\x87\xb6\xc1Q2;\x1eQo_\xb2u\xd7\xfa\x0e\xb1\x7f\xcd\xc1m]\x1a\xa88\x1a\x00\n\xdc(5+\xea*|\xe0\xd9f\xd0Æ\x1b\xb6V\xba䖝7\x9b\xc5o\\\a\xf8\xf7\xf9\x92\xb1\x1fT\x93\xabӽ/͈\xb2*v\x18\x89\xb1\xf3n\x83\xe7IIT:C\xcf7\xaa\x10Y\xc4\xe7\x1c\xbdW\xcf5ػl\x88n\xfe\xcb:\xd9\"\xb1\xc0\a\x9b\x8bp\xebb\xffJfw\x9f\xfb\x91k%\xbc\x12\x7f\xa6'\x95N\xb0\xea\xf6\xee\xe6\x9a`\x051\xa2\xb7\x9a\x9a\x04ņ\xe5+@\x97\xa1\x1d\xfb!}r\xbd\xeeA\xed\xe7\bw\x1f\xab\x80ܽL\x12\xdc\x16\xaf\x9a3\x85Z\xeb\xe6\xda\xe1r\xa8'\x94/.wL\xf9\xa7'\x84\xce\x17\x15\xd7v璉.zx\x04\xbb>\xb5jv\xd0Z\xed\xbf\xbc\xd2-=\xb2\x87GWh'{W\xf5\x93\a\x86\xf4|\x0eN\x87OUO\x9e\xa7~\x01\x9c\x0e\xbbP\v\xa2b\xe4\xa7h\x06\xe4\xc9W,\x8d\xbf\xa1\xffG\xb5\x85\xf7ѕ\xcb\xfe\xeb+\x83&#\xa9\x89\x01*]2\x1f\xa1`\x9b\x8fHw|?O\xed\xc5s\r\x03*\xfe\x8e\xf0\xe7,N\xde\xf6A\x8d?HB7\xa8\x87Nc^\x15=\xf5\xb4c7\xf7\x14\xb76\xaa\xd4O}\x1f\xb7\x86\xe5ɐ`\x10\x81%\xe4\xc17ZNEF\xab4\x7f\x80\x8fʽ\xad\x93\"&\xfd\x16\xbd\x97\x97\xbc\xe7\x16\xf2\xb5\xfd$\x8c)z?\xb6!\xc0\xf6|\xc6\xdeE\xff\x88\xed\x91O\x19X[\xba\x91ғ&\xef\xfd\xeb$\xa8\x8f\r \v\x02\x05\x1c\xb4\x15\xfew\xa3\x9e\xe8\x02\xfc\xf8\x1asx@\xa4\xf3\x86\x19\xd0A\x11J\xe1=j\x98uU(\x9e\x83\xbe\xa2GT\x12F\xfcS\xaf\xc1\xc0\x1d\xe8?\xc5\xe2\xedfd<\xa1\xe7\x17̒A\x8f\xae(\xa0\xf8A\x14`\x1c≦\xe1f\xbfec)\xear\xe5<\xd55\xfe\xd8tr\xc02\xbb\xa1\xd2\x06C\x05\x1a\xfdD\xb7\x15Q\x9b \xf9\x87\x89\xc1\x1a>\ni\xe1\x01\xc6c\xe8\t\x9b\xe0\xdeh \a (0\x8a\xf8\xfe\x12[y\xec\x11\xe4>\xdez \x03\xcdbdL\x8e\x95w\xabn\xee\xaf\f\xabeN\x1b\x00\xf7\x7f\xbe=J~\xb7\xbd\xf7e\x82NHQ\xef\xf7\xe3-;!BG;\x91O\x1fW\xe21X\xdc\x18\x95\t\x8a*\x9e\x84\xf5\xd79\xbe\xdc\x1d\xe2\x87\x02\xc4\x03\xd2Q\x1b\xf8\xfc$A\x7f\t\x16\xc8\\\xcbػ-\xd3\xda\xef\xa7=h\xd1\xf7Z\xac¾G`\f\x000\x15\xf6\xb9\x8c{\t(l\xaf\t\xd3H\x1c\xc4>\x01\xdcyG\xc8ik\x85\xb4\xe3\x9c!n\x9bVt\xd8tDCN\x8b\xed\xfd\x00\xc6 \x93\x9d\x1e}j\xaa\xb8Ӧ\x86\xfd^\x8cy\xa3\xb4c\x96\xe1@\xff\xb0\xf7kT\x83\x1f\xd4\xde1\xcd=\xaaF\xf6>\xd2CxyGr\xbc\x97\xde\xfdR\xaf\xda\a\x15\xd8\xdf\xfe~\xf6\x8f\x00\x00\x00\xff\xff)\x00\x87w>{\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcV\xcfo+5\x10\xbe\xe7\xaf\x18\x89+\xbb\xa1B \x94\x1b*\x1c*\xe0\xa9j\x9ezw\xbc\x93d\xa8\xd7^f\xc6)A\xfc\xf1\xc8\xf6n\x9b\xee:\xb4\x8f\x03\xbe\xad\xed\xf9\xe6\x9bo~x\x9b\xa6Y\x99\x81\x1e\x91\x85\x82߀\x19\b\xffT\xf4\xe9Kڧ\x1f\xa4\xa5\xb0>ݬ\x9e\xc8w\x1b\xb8\x8d\xa2\xa1\x7f@\t\x91-\xfe\x84{\xf2\xa4\x14\xfc\xaaG5\x9dQ\xb3Y\x01\x18\uf0da\xb4-\xe9\x13\xc0\x06\xaf\x1c\x9cCn\x0e\xe8ۧ\xb8\xc3]$\xd7!g\xf0\xc9\xf5\xe9\x9b\xf6\xe6\xfb\xf6\xbb\x15\x807=n@\x90ә\x1a\x8d\xc2\xf8GDQiO\xe8\x90CKa%\x03ڄ\x7f\xe0\x10\x87\r\xbc\x1e\x14\xfb\xd1w\xe1\xbd\xcdP\xdb\f\xf5P\xa0\xf2\xa9#\xd1_\xae\xdd\xf8\x95\xc6[\x83\x8bl\\\x9dP\xbe \xc7\xc0\xfa\xe9\xd5i\x03\"\\N\xc8\x1f\xa23\\5^\x01\x88\r\x03n \xdb\x0e\xc6b\xb7\x02\x18\x05\xc9Xͨ\xc5\xe9\xa6\xc0\xd9#\xf6\xa68\x01\b\x03\xfa\x1f\xef\xef\x1e\xbfݾ\xd9\x06\xe8P,ӠYֿ\x9b\x97}\xa8\x85\t$``\xa4\x04\x1a\xc0X\x8b\"`#3z\x85B\x19\xc8\xef\x03\xf79\xad`v!\xea\x05\xaa\x1e\x11\x1e\xb3\xfec\x98\xed\xcb\xe1\xc0a@V\x9a\xa4)\xeb\xa2\xe2.v\xff\x8dxZ)\xd6b\x05]*=\x94\xecy\xd4\v\xbbQ\x1e\b{\xd0#\t0\x0e\x8c\x82\xbe\x14c\xda6\x1e\xc2\xeew\xb4\xdaΠ\x8b.\x922\x19]\x97*\xf6\x84\xac\xc0h\xc3\xc1\xd3_/ؒ\x04JN\x9dѬ\x9dWdo\x1c\x9c\x8c\x8b\xf85\x18\xdf͐{s\x06\xc6\xe4\x13\xa2\xbf\xc0\xcb\x062\xe7\xf1[`\xccRo\xe0\xa8:\xc8f\xbd>\x90N}hC\xdfGOz^疢]\xd4\xc0\xb2\xee\xf0\x84n-th\f\xdb#)Z\x8d\x8ck3P\x93\x03\xf1\xb9\x17۾\xfb\x8a\xc7Ε7n\xf5\x9cjP\x94\xc9\x1f.\x0er\xeb|AzR#\x95b*P%\xc4\xd7,\xa4\xad$\xdd\xc3\xcf\xdb\xcf01)\x99*Iy\xbd\xba\xd0e\xcaOR\x93\xfc\x1e\xb9\xd8\xed9\xf4\x19\x13}7\x04\xf2\x9a?\xac\xa3\\\xb8qד\xcaT\xda)us\xd8\xdb<\xab`\x87\x10\x87\xce(v\xf3\vw\x1enM\x8f\xee\xd6\b\xfeϹJY\x91&%\xe1Cٺ\x9c\xc0\xf3\xcbEދ\x83iv^ImeJl\a\xb4)\xb9I\xdfdM{\xb2\xa5\xad\xf6\x81\xc1\xd4L\xda\x0f1\xc9\x16_\xc8e\x9cH\x85\xcdlN\xa5.\x7f\x9fM},哣\x11\x9co\xce8ݧ;s\xff\x8e\xf6h\xcf\xd6a\x81(S\bߧ\x92\x16\xfa\xd8/}6\xf0\t\x9f+\xbb\xf7\x1c҄\xc6\xf9\xa8\xb9Z\x1bP\x1e\xb1\x03\xf9E\xb8\xf3\xc8ʭ\xfc0.G~\x0eh\x04\x02\x8eާ\x96\x0e~\x01Yy\x11\x16wH\xb1\xaf\xb0\xa9\xf2\xb9\xf3\xfb\x90\xff\"Lrl\xb4\xb4\x13\x8e\xc9\x1e\xfd\x14^\x15\xc0\xeb\xb9.k9\xe7>$hY\xf9y\xfeo\xc6i.\x11c\xd5w\x93YU\x0f\x92ǚ\xe2\xf5\xfe\x1aYF\xe7\xcc\xce\xe1\x06\x94\xe3Һ\xd8\x1afs\x9eW\xcdTj\x9f\xa9GQ\xd3\x0f\xef\x14\xd0\xe2UH\xeb~\x81\x92\x9a\xe7\xf9\x88\xfeZ\x8b\xc0\xb3\x91W\xe7\x15\xc8\xdd\xf9\x9a\xe9\xed\xcb\xdf\xe6\xb2\xcfJ=o \xcd\xfaF\xa9\"䇔\xaa\xa6\xb4\xd4y\xf5\xb7f\xa1\xd2\xf6\xf2\xee4H\xde\xf4\xcb\xf4W\xb3\x8c\xe1*\x85j\x05,63|w\x11\x9eh`s\x98\x02\xfe'\x00\x00\xff\xff\xef\xf8\xa6>\x10\f\x00\x00"), diff --git a/pkg/apis/velero/v1/labels_annotations.go b/pkg/apis/velero/v1/labels_annotations.go index 78d231ba9..a50018dbe 100644 --- a/pkg/apis/velero/v1/labels_annotations.go +++ b/pkg/apis/velero/v1/labels_annotations.go @@ -101,6 +101,9 @@ const ( // ExcludeFromBackupLabel is the label to exclude k8s resource from backup, // even if the resource contains a matching selector label. ExcludeFromBackupLabel = "velero.io/exclude-from-backup" + + // PVRLabel is the label key used to identify the pvb for pvr pod + PVRLabel = "velero.io/pod-volume-restore" ) type AsyncOperationIDPrefix string diff --git a/pkg/apis/velero/v1/pod_volume_restore_type.go b/pkg/apis/velero/v1/pod_volume_restore_type.go index 34bc7e530..2d059a14a 100644 --- a/pkg/apis/velero/v1/pod_volume_restore_type.go +++ b/pkg/apis/velero/v1/pod_volume_restore_type.go @@ -54,15 +54,23 @@ type PodVolumeRestoreSpec struct { // +optional // +nullable UploaderSettings map[string]string `json:"uploaderSettings,omitempty"` + + // Cancel indicates request to cancel the ongoing PodVolumeRestore. It can be set + // when the PodVolumeRestore is in InProgress phase + Cancel bool `json:"cancel,omitempty"` } // PodVolumeRestorePhase represents the lifecycle phase of a PodVolumeRestore. -// +kubebuilder:validation:Enum=New;InProgress;Completed;Failed +// +kubebuilder:validation:Enum=New;Accepted;Prepared;InProgress;Canceling;Canceled;Completed;Failed type PodVolumeRestorePhase string const ( PodVolumeRestorePhaseNew PodVolumeRestorePhase = "New" + PodVolumeRestorePhaseAccepted PodVolumeRestorePhase = "Accepted" + PodVolumeRestorePhasePrepared PodVolumeRestorePhase = "Prepared" PodVolumeRestorePhaseInProgress PodVolumeRestorePhase = "InProgress" + PodVolumeRestorePhaseCanceling PodVolumeRestorePhase = "Canceling" + PodVolumeRestorePhaseCanceled PodVolumeRestorePhase = "Canceled" PodVolumeRestorePhaseCompleted PodVolumeRestorePhase = "Completed" PodVolumeRestorePhaseFailed PodVolumeRestorePhase = "Failed" ) @@ -95,6 +103,16 @@ type PodVolumeRestoreStatus struct { // about the restore operation. // +optional Progress shared.DataMoveOperationProgress `json:"progress,omitempty"` + + // AcceptedTimestamp records the time the pod volume restore is to be prepared. + // The server's time is used for AcceptedTimestamp + // +optional + // +nullable + AcceptedTimestamp *metav1.Time `json:"acceptedTimestamp,omitempty"` + + // Node is name of the node where the pod volume restore is processed. + // +optional + Node string `json:"node,omitempty"` } // TODO(2.0) After converting all resources to use the runtime-controller client, the genclient and k8s:deepcopy markers will no longer be needed and should be removed. @@ -103,14 +121,14 @@ type PodVolumeRestoreStatus struct { // +kubebuilder:object:generate=true // +kubebuilder:object:root=true // +kubebuilder:storageversion -// +kubebuilder:printcolumn:name="Namespace",type="string",JSONPath=".spec.pod.namespace",description="Namespace of the pod containing the volume to be restored" -// +kubebuilder:printcolumn:name="Pod",type="string",JSONPath=".spec.pod.name",description="Name of the pod containing the volume to be restored" +// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.phase",description="PodVolumeRestore status such as New/InProgress" +// +kubebuilder:printcolumn:name="Started",type="date",JSONPath=".status.startTimestamp",description="Time duration since this PodVolumeRestore was started" +// +kubebuilder:printcolumn:name="Bytes Done",type="integer",format="int64",JSONPath=".status.progress.bytesDone",description="Completed bytes" +// +kubebuilder:printcolumn:name="Total Bytes",type="integer",format="int64",JSONPath=".status.progress.totalBytes",description="Total bytes" +// +kubebuilder:printcolumn:name="Storage Location",type="string",JSONPath=".spec.backupStorageLocation",description="Name of the Backup Storage Location where the backup data is stored" +// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp",description="Time duration since this PodVolumeRestore was created" +// +kubebuilder:printcolumn:name="Node",type="string",JSONPath=".status.node",description="Name of the node where the PodVolumeRestore is processed" // +kubebuilder:printcolumn:name="Uploader Type",type="string",JSONPath=".spec.uploaderType",description="The type of the uploader to handle data transfer" -// +kubebuilder:printcolumn:name="Volume",type="string",JSONPath=".spec.volume",description="Name of the volume to be restored" -// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.phase",description="Pod Volume Restore status such as New/InProgress" -// +kubebuilder:printcolumn:name="TotalBytes",type="integer",format="int64",JSONPath=".status.progress.totalBytes",description="Pod Volume Restore status such as New/InProgress" -// +kubebuilder:printcolumn:name="BytesDone",type="integer",format="int64",JSONPath=".status.progress.bytesDone",description="Pod Volume Restore status such as New/InProgress" -// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp" type PodVolumeRestore struct { metav1.TypeMeta `json:",inline"` diff --git a/pkg/apis/velero/v1/zz_generated.deepcopy.go b/pkg/apis/velero/v1/zz_generated.deepcopy.go index 2fd13cfd2..47cc8b199 100644 --- a/pkg/apis/velero/v1/zz_generated.deepcopy.go +++ b/pkg/apis/velero/v1/zz_generated.deepcopy.go @@ -1149,6 +1149,10 @@ func (in *PodVolumeRestoreStatus) DeepCopyInto(out *PodVolumeRestoreStatus) { *out = (*in).DeepCopy() } out.Progress = in.Progress + if in.AcceptedTimestamp != nil { + in, out := &in.AcceptedTimestamp, &out.AcceptedTimestamp + *out = (*in).DeepCopy() + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PodVolumeRestoreStatus. diff --git a/pkg/builder/pod_builder.go b/pkg/builder/pod_builder.go index 50f8f5e51..c38acf66e 100644 --- a/pkg/builder/pod_builder.go +++ b/pkg/builder/pod_builder.go @@ -88,6 +88,11 @@ func (b *PodBuilder) InitContainers(containers ...*corev1api.Container) *PodBuil return b } +func (b *PodBuilder) InitContainerState(state corev1api.ContainerState) *PodBuilder { + b.object.Status.InitContainerStatuses = append(b.object.Status.InitContainerStatuses, corev1api.ContainerStatus{State: state}) + return b +} + func (b *PodBuilder) Containers(containers ...*corev1api.Container) *PodBuilder { for _, c := range containers { b.object.Spec.Containers = append(b.object.Spec.Containers, *c) diff --git a/pkg/builder/pod_volume_restore_builder.go b/pkg/builder/pod_volume_restore_builder.go index 3d4da94d6..965dd5a5d 100644 --- a/pkg/builder/pod_volume_restore_builder.go +++ b/pkg/builder/pod_volume_restore_builder.go @@ -97,3 +97,33 @@ func (b *PodVolumeRestoreBuilder) UploaderType(uploaderType string) *PodVolumeRe b.object.Spec.UploaderType = uploaderType return b } + +// Cancel sets the DataDownload's Cancel. +func (d *PodVolumeRestoreBuilder) Cancel(cancel bool) *PodVolumeRestoreBuilder { + d.object.Spec.Cancel = cancel + return d +} + +// AcceptedTimestamp sets the PodVolumeRestore's AcceptedTimestamp. +func (d *PodVolumeRestoreBuilder) AcceptedTimestamp(acceptedTimestamp *metav1.Time) *PodVolumeRestoreBuilder { + d.object.Status.AcceptedTimestamp = acceptedTimestamp + return d +} + +// Finalizers sets the PodVolumeRestore's Finalizers. +func (d *PodVolumeRestoreBuilder) Finalizers(finalizers []string) *PodVolumeRestoreBuilder { + d.object.Finalizers = finalizers + return d +} + +// Message sets the PodVolumeRestore's Message. +func (d *PodVolumeRestoreBuilder) Message(msg string) *PodVolumeRestoreBuilder { + d.object.Status.Message = msg + return d +} + +// Message sets the PodVolumeRestore's Node. +func (d *PodVolumeRestoreBuilder) Node(node string) *PodVolumeRestoreBuilder { + d.object.Status.Node = node + return d +} diff --git a/pkg/cmd/cli/nodeagent/server.go b/pkg/cmd/cli/nodeagent/server.go index ede158116..286ca7439 100644 --- a/pkg/cmd/cli/nodeagent/server.go +++ b/pkg/cmd/cli/nodeagent/server.go @@ -306,10 +306,6 @@ func (s *nodeAgentServer) run() { s.logger.Fatal(err, "unable to create controller", "controller", constant.ControllerPodVolumeBackup) } - if err = controller.NewPodVolumeRestoreReconciler(s.mgr.GetClient(), s.kubeClient, s.dataPathMgr, repoEnsurer, credentialGetter, s.logger).SetupWithManager(s.mgr); err != nil { - s.logger.WithError(err).Fatal("Unable to create the pod volume restore controller") - } - var loadAffinity *kube.LoadAffinity if s.dataPathConfigs != nil && len(s.dataPathConfigs.LoadAffinity) > 0 { loadAffinity = s.dataPathConfigs.LoadAffinity[0] @@ -332,6 +328,10 @@ func (s *nodeAgentServer) run() { } } + if err = controller.NewPodVolumeRestoreReconciler(s.mgr.GetClient(), s.mgr, s.kubeClient, s.dataPathMgr, s.nodeName, s.config.dataMoverPrepareTimeout, s.config.resourceTimeout, podResources, s.logger).SetupWithManager(s.mgr); err != nil { + s.logger.WithError(err).Fatal("Unable to create the pod volume restore controller") + } + dataUploadReconciler := controller.NewDataUploadReconciler( s.mgr.GetClient(), s.mgr, @@ -525,7 +525,7 @@ func (s *nodeAgentServer) markInProgressPVRsFailed(client ctrlclient.Client) { continue } - if err := controller.UpdatePVRStatusToFailed(s.ctx, client, &pvrs.Items[i], + if err := controller.UpdatePVRStatusToFailed(s.ctx, client, &pvrs.Items[i], errors.New("cannot survive from node-agent restart"), fmt.Sprintf("get a podvolumerestore with status %q during the server starting, mark it as %q", velerov1api.PodVolumeRestorePhaseInProgress, velerov1api.PodVolumeRestorePhaseFailed), time.Now(), s.logger); err != nil { s.logger.WithError(errors.WithStack(err)).Errorf("failed to patch podvolumerestore %q", pvr.GetName()) diff --git a/pkg/controller/data_download_controller.go b/pkg/controller/data_download_controller.go index 53d83d98e..75c04e377 100644 --- a/pkg/controller/data_download_controller.go +++ b/pkg/controller/data_download_controller.go @@ -198,9 +198,7 @@ func (r *DataDownloadReconciler) Reconcile(ctx context.Context, req ctrl.Request if time.Since(spotted) > delay { log.Infof("Data download %s is canceled in Phase %s but not handled in rasonable time", dd.GetName(), dd.Status.Phase) - if r.tryCancelDataDownload(ctx, dd, "") { - delete(r.cancelledDataDownload, dd.Name) - } + r.tryCancelDataDownload(ctx, dd, "") return ctrl.Result{}, nil } @@ -526,6 +524,7 @@ func (r *DataDownloadReconciler) tryCancelDataDownload(ctx context.Context, dd * // success update r.metrics.RegisterDataDownloadCancel(r.nodeName) r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(dd)) + delete(r.cancelledDataDownload, dd.Name) log.Warn("data download is canceled") diff --git a/pkg/controller/data_upload_controller.go b/pkg/controller/data_upload_controller.go index 6ab65f172..7dc19e855 100644 --- a/pkg/controller/data_upload_controller.go +++ b/pkg/controller/data_upload_controller.go @@ -226,9 +226,7 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request) if time.Since(spotted) > delay { log.Infof("Data upload %s is canceled in Phase %s but not handled in reasonable time", du.GetName(), du.Status.Phase) - if r.tryCancelDataUpload(ctx, du, "") { - delete(r.cancelledDataUpload, du.Name) - } + r.tryCancelDataUpload(ctx, du, "") return ctrl.Result{}, nil } @@ -566,6 +564,7 @@ func (r *DataUploadReconciler) tryCancelDataUpload(ctx context.Context, du *vele r.metrics.RegisterDataUploadCancel(r.nodeName) // cleans up any objects generated during the snapshot expose r.cleanUp(ctx, du, log) + delete(r.cancelledDataUpload, du.Name) log.Warn("data upload is canceled") diff --git a/pkg/controller/pod_volume_backup_controller.go b/pkg/controller/pod_volume_backup_controller.go index 254a39c91..16ad49c06 100644 --- a/pkg/controller/pod_volume_backup_controller.go +++ b/pkg/controller/pod_volume_backup_controller.go @@ -46,7 +46,10 @@ import ( "github.com/vmware-tanzu/velero/pkg/util/filesystem" ) -const pVBRRequestor string = "pod-volume-backup-restore" +const ( + pVBRRequestor = "pod-volume-backup-restore" + PodVolumeFinalizer = "velero.io/pod-volume-finalizer" +) // NewPodVolumeBackupReconciler creates the PodVolumeBackupReconciler instance func NewPodVolumeBackupReconciler(client client.Client, kubeClient kubernetes.Interface, dataPathMgr *datapath.Manager, ensurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter, diff --git a/pkg/controller/pod_volume_restore_controller.go b/pkg/controller/pod_volume_restore_controller.go index e65d1b606..46a00937d 100644 --- a/pkg/controller/pod_volume_restore_controller.go +++ b/pkg/controller/pod_volume_restore_controller.go @@ -19,8 +19,7 @@ package controller import ( "context" "fmt" - "os" - "path/filepath" + "strings" "time" "github.com/pkg/errors" @@ -30,49 +29,64 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes" clocks "k8s.io/utils/clock" ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + "sigs.k8s.io/controller-runtime/pkg/event" "sigs.k8s.io/controller-runtime/pkg/handler" + "sigs.k8s.io/controller-runtime/pkg/manager" + "sigs.k8s.io/controller-runtime/pkg/predicate" "sigs.k8s.io/controller-runtime/pkg/reconcile" - "github.com/vmware-tanzu/velero/internal/credentials" veleroapishared "github.com/vmware-tanzu/velero/pkg/apis/velero/shared" + velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/constant" "github.com/vmware-tanzu/velero/pkg/datapath" "github.com/vmware-tanzu/velero/pkg/exposer" - "github.com/vmware-tanzu/velero/pkg/podvolume" - "github.com/vmware-tanzu/velero/pkg/repository" + "github.com/vmware-tanzu/velero/pkg/nodeagent" "github.com/vmware-tanzu/velero/pkg/restorehelper" "github.com/vmware-tanzu/velero/pkg/uploader" - "github.com/vmware-tanzu/velero/pkg/util/boolptr" - "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util" + "github.com/vmware-tanzu/velero/pkg/util/kube" ) -func NewPodVolumeRestoreReconciler(client client.Client, kubeClient kubernetes.Interface, dataPathMgr *datapath.Manager, ensurer *repository.Ensurer, - credentialGetter *credentials.CredentialGetter, logger logrus.FieldLogger) *PodVolumeRestoreReconciler { +func NewPodVolumeRestoreReconciler(client client.Client, mgr manager.Manager, kubeClient kubernetes.Interface, dataPathMgr *datapath.Manager, + nodeName string, preparingTimeout time.Duration, resourceTimeout time.Duration, podResources corev1api.ResourceRequirements, + logger logrus.FieldLogger) *PodVolumeRestoreReconciler { return &PodVolumeRestoreReconciler{ - Client: client, - kubeClient: kubeClient, - logger: logger.WithField("controller", "PodVolumeRestore"), - repositoryEnsurer: ensurer, - credentialGetter: credentialGetter, - fileSystem: filesystem.NewFileSystem(), - clock: &clocks.RealClock{}, - dataPathMgr: dataPathMgr, + client: client, + mgr: mgr, + kubeClient: kubeClient, + logger: logger.WithField("controller", "PodVolumeRestore"), + nodeName: nodeName, + clock: &clocks.RealClock{}, + podResources: podResources, + dataPathMgr: dataPathMgr, + preparingTimeout: preparingTimeout, + resourceTimeout: resourceTimeout, + exposer: exposer.NewPodVolumeExposer(kubeClient, logger), + cancelledPVR: make(map[string]time.Time), } } type PodVolumeRestoreReconciler struct { - client.Client - kubeClient kubernetes.Interface - logger logrus.FieldLogger - repositoryEnsurer *repository.Ensurer - credentialGetter *credentials.CredentialGetter - fileSystem filesystem.Interface - clock clocks.WithTickerAndDelayedExecution - dataPathMgr *datapath.Manager + client client.Client + mgr manager.Manager + kubeClient kubernetes.Interface + logger logrus.FieldLogger + nodeName string + clock clocks.WithTickerAndDelayedExecution + podResources corev1api.ResourceRequirements + exposer exposer.PodVolumeExposer + dataPathMgr *datapath.Manager + preparingTimeout time.Duration + resourceTimeout time.Duration + cancelledPVR map[string]time.Time } // +kubebuilder:rbac:groups=velero.io,resources=podvolumerestores,verbs=get;list;watch;create;update;patch;delete @@ -81,123 +95,428 @@ type PodVolumeRestoreReconciler struct { // +kubebuilder:rbac:groups="",resources=persistentvolumes,verbs=get // +kubebuilder:rbac:groups="",resources=persistentvolumerclaims,verbs=get -func (c *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - log := c.logger.WithField("PodVolumeRestore", req.NamespacedName.String()) +func (r *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + log := r.logger.WithField("PodVolumeRestore", req.NamespacedName.String()) + log.Info("Reconciling PVR by advanced controller") pvr := &velerov1api.PodVolumeRestore{} - if err := c.Get(ctx, types.NamespacedName{Namespace: req.Namespace, Name: req.Name}, pvr); err != nil { + if err := r.client.Get(ctx, types.NamespacedName{Namespace: req.Namespace, Name: req.Name}, pvr); err != nil { if apierrors.IsNotFound(err) { - log.Warn("PodVolumeRestore not found, skip") + log.Warn("PVR not found, skip") return ctrl.Result{}, nil } - log.WithError(err).Error("Unable to get the PodVolumeRestore") + log.WithError(err).Error("Unable to get the PVR") return ctrl.Result{}, err } + log = log.WithField("pod", fmt.Sprintf("%s/%s", pvr.Spec.Pod.Namespace, pvr.Spec.Pod.Name)) if len(pvr.OwnerReferences) == 1 { log = log.WithField("restore", fmt.Sprintf("%s/%s", pvr.Namespace, pvr.OwnerReferences[0].Name)) } - shouldProcess, pod, err := c.shouldProcess(ctx, log, pvr) - if err != nil { - return ctrl.Result{}, err - } - if !shouldProcess { - return ctrl.Result{}, nil - } + // Logic for clear resources when pvr been deleted + if !isPVRInFinalState(pvr) { + if !controllerutil.ContainsFinalizer(pvr, PodVolumeFinalizer) { + if err := UpdatePVRWithRetry(ctx, r.client, req.NamespacedName, log, func(pvr *velerov1api.PodVolumeRestore) bool { + if controllerutil.ContainsFinalizer(pvr, PodVolumeFinalizer) { + return false + } - initContainerIndex := getInitContainerIndex(pod) - if initContainerIndex > 0 { - log.Warnf(`Init containers before the %s container may cause issues - if they interfere with volumes being restored: %s index %d`, restorehelper.WaitInitContainer, restorehelper.WaitInitContainer, initContainerIndex) - } + controllerutil.AddFinalizer(pvr, PodVolumeFinalizer) - log.Info("Restore starting") + return true + }); err != nil { + log.WithError(err).Errorf("failed to add finalizer for PVR %s/%s", pvr.Namespace, pvr.Name) + return ctrl.Result{}, err + } - callbacks := datapath.Callbacks{ - OnCompleted: c.OnDataPathCompleted, - OnFailed: c.OnDataPathFailed, - OnCancelled: c.OnDataPathCancelled, - OnProgress: c.OnDataPathProgress, - } + return ctrl.Result{}, nil + } - fsRestore, err := c.dataPathMgr.CreateFileSystemBR(pvr.Name, pVBRRequestor, ctx, c.Client, pvr.Namespace, callbacks, log) - if err != nil { - if err == datapath.ConcurrentLimitExceed { - return ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, nil - } else { - return c.errorOut(ctx, pvr, err, "error to create data path", log) + if !pvr.DeletionTimestamp.IsZero() { + if !pvr.Spec.Cancel { + log.Warnf("Cancel PVR under phase %s because it is being deleted", pvr.Status.Phase) + + if err := UpdatePVRWithRetry(ctx, r.client, req.NamespacedName, log, func(pvr *velerov1api.PodVolumeRestore) bool { + if pvr.Spec.Cancel { + return false + } + + pvr.Spec.Cancel = true + pvr.Status.Message = "Cancel PVR because it is being deleted" + + return true + }); err != nil { + log.WithError(err).Errorf("failed to set cancel flag for PVR %s/%s", pvr.Namespace, pvr.Name) + return ctrl.Result{}, err + } + + return ctrl.Result{}, nil + } + } + } else { + delete(r.cancelledPVR, pvr.Name) + + if controllerutil.ContainsFinalizer(pvr, PodVolumeFinalizer) { + if err := UpdatePVRWithRetry(ctx, r.client, req.NamespacedName, log, func(pvr *velerov1api.PodVolumeRestore) bool { + if !controllerutil.ContainsFinalizer(pvr, PodVolumeFinalizer) { + return false + } + + controllerutil.RemoveFinalizer(pvr, PodVolumeFinalizer) + + return true + }); err != nil { + log.WithError(err).Error("error to remove finalizer") + return ctrl.Result{}, err + } + + return ctrl.Result{}, nil } } - original := pvr.DeepCopy() - pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseInProgress - pvr.Status.StartTimestamp = &metav1.Time{Time: c.clock.Now()} - if err = c.Patch(ctx, pvr, client.MergeFrom(original)); err != nil { - c.closeDataPath(ctx, pvr.Name) - return c.errorOut(ctx, pvr, err, "error to update status to in progress", log) + if pvr.Spec.Cancel { + if spotted, found := r.cancelledPVR[pvr.Name]; !found { + r.cancelledPVR[pvr.Name] = r.clock.Now() + } else { + delay := cancelDelayOthers + if pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseInProgress { + delay = cancelDelayInProgress + } + + if time.Since(spotted) > delay { + log.Infof("PVR %s is canceled in Phase %s but not handled in rasonable time", pvr.GetName(), pvr.Status.Phase) + r.tryCancelPodVolumeRestore(ctx, pvr, "") + + return ctrl.Result{}, nil + } + } } - volumePath, err := exposer.GetPodVolumeHostPath(ctx, pod, pvr.Spec.Volume, c.kubeClient, c.fileSystem, log) - if err != nil { - c.closeDataPath(ctx, pvr.Name) - return c.errorOut(ctx, pvr, err, "error exposing host path for pod volume", log) + if pvr.Status.Phase == "" || pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseNew { + if pvr.Spec.Cancel { + log.Infof("PVR %s is canceled in Phase %s", pvr.GetName(), pvr.Status.Phase) + r.tryCancelPodVolumeRestore(ctx, pvr, "") + + return ctrl.Result{}, nil + } + + shouldProcess, pod, err := shouldProcess(ctx, r.client, log, pvr) + if err != nil { + return ctrl.Result{}, err + } + if !shouldProcess { + return ctrl.Result{}, nil + } + + log.Info("Accepting PVR") + + if err := r.acceptPodVolumeRestore(ctx, pvr); err != nil { + return ctrl.Result{}, errors.Wrapf(err, "error accepting PVR %s", pvr.Name) + } + + initContainerIndex := getInitContainerIndex(pod) + if initContainerIndex > 0 { + log.Warnf(`Init containers before the %s container may cause issues + if they interfere with volumes being restored: %s index %d`, restorehelper.WaitInitContainer, restorehelper.WaitInitContainer, initContainerIndex) + } + + log.Info("Exposing PVR") + + exposeParam := r.setupExposeParam(pvr) + if err := r.exposer.Expose(ctx, getPVROwnerObject(pvr), exposeParam); err != nil { + return r.errorOut(ctx, pvr, err, "error to expose PVR", log) + } + + log.Info("PVR is exposed") + + return ctrl.Result{}, nil + } else if pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseAccepted { + if peekErr := r.exposer.PeekExposed(ctx, getPVROwnerObject(pvr)); peekErr != nil { + log.Errorf("Cancel PVR %s/%s because of expose error %s", pvr.Namespace, pvr.Name, peekErr) + r.tryCancelPodVolumeRestore(ctx, pvr, fmt.Sprintf("found a PVR %s/%s with expose error: %s. mark it as cancel", pvr.Namespace, pvr.Name, peekErr)) + } else if pvr.Status.AcceptedTimestamp != nil { + if time.Since(pvr.Status.AcceptedTimestamp.Time) >= r.preparingTimeout { + r.onPrepareTimeout(ctx, pvr) + } + } + + return ctrl.Result{}, nil + } else if pvr.Status.Phase == velerov1api.PodVolumeRestorePhasePrepared { + log.Infof("PVR is prepared and should be processed by %s (%s)", pvr.Status.Node, r.nodeName) + + if pvr.Status.Node != r.nodeName { + return ctrl.Result{}, nil + } + + if pvr.Spec.Cancel { + log.Info("Prepared PVR is being cancelled") + r.OnDataPathCancelled(ctx, pvr.GetNamespace(), pvr.GetName()) + return ctrl.Result{}, nil + } + + asyncBR := r.dataPathMgr.GetAsyncBR(pvr.Name) + if asyncBR != nil { + log.Info("Cancellable data path is already started") + return ctrl.Result{}, nil + } + + res, err := r.exposer.GetExposed(ctx, getPVROwnerObject(pvr), r.client, r.nodeName, r.resourceTimeout) + if err != nil { + return r.errorOut(ctx, pvr, err, "exposed PVR is not ready", log) + } else if res == nil { + return r.errorOut(ctx, pvr, errors.New("no expose result is available for the current node"), "exposed PVR is not ready", log) + } + + log.Info("Exposed PVR is ready and creating data path routine") + + callbacks := datapath.Callbacks{ + OnCompleted: r.OnDataPathCompleted, + OnFailed: r.OnDataPathFailed, + OnCancelled: r.OnDataPathCancelled, + OnProgress: r.OnDataPathProgress, + } + + asyncBR, err = r.dataPathMgr.CreateMicroServiceBRWatcher(ctx, r.client, r.kubeClient, r.mgr, datapath.TaskTypeRestore, + pvr.Name, pvr.Namespace, res.ByPod.HostingPod.Name, res.ByPod.HostingContainer, pvr.Name, callbacks, false, log) + if err != nil { + if err == datapath.ConcurrentLimitExceed { + log.Info("Data path instance is concurrent limited requeue later") + return ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, nil + } else { + return r.errorOut(ctx, pvr, err, "error to create data path", log) + } + } + + if err := r.initCancelableDataPath(ctx, asyncBR, res, log); err != nil { + log.WithError(err).Errorf("Failed to init cancelable data path for %s", pvr.Name) + + r.closeDataPath(ctx, pvr.Name) + return r.errorOut(ctx, pvr, err, "error initializing data path", log) + } + + terminated := false + if err := UpdatePVRWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, log, func(pvr *velerov1api.PodVolumeRestore) bool { + if isPVRInFinalState(pvr) { + terminated = true + return false + } + + pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseInProgress + pvr.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + + return true + }); err != nil { + log.WithError(err).Warnf("Failed to update PVR %s to InProgress, will data path close and retry", pvr.Name) + + r.closeDataPath(ctx, pvr.Name) + return ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, nil + } + + if terminated { + log.Warnf("PVR %s is terminated during transition from prepared", pvr.Name) + r.closeDataPath(ctx, pvr.Name) + return ctrl.Result{}, nil + } + + log.Info("PVR is marked as in progress") + + if err := r.startCancelableDataPath(asyncBR, pvr, res, log); err != nil { + log.WithError(err).Errorf("Failed to start cancelable data path for %s", pvr.Name) + r.closeDataPath(ctx, pvr.Name) + + return r.errorOut(ctx, pvr, err, "error starting data path", log) + } + + return ctrl.Result{}, nil + } else if pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseInProgress { + if pvr.Spec.Cancel { + if pvr.Status.Node != r.nodeName { + return ctrl.Result{}, nil + } + + log.Info("PVR is being canceled") + + asyncBR := r.dataPathMgr.GetAsyncBR(pvr.Name) + if asyncBR == nil { + r.OnDataPathCancelled(ctx, pvr.GetNamespace(), pvr.GetName()) + return ctrl.Result{}, nil + } + + // Update status to Canceling + if err := UpdatePVRWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, log, func(pvr *velerov1api.PodVolumeRestore) bool { + if isPVRInFinalState(pvr) { + log.Warnf("PVR %s is terminated, abort setting it to cancelling", pvr.Name) + return false + } + + pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseCanceling + return true + }); err != nil { + log.WithError(err).Error("error updating PVR into canceling status") + return ctrl.Result{}, err + } + + asyncBR.Cancel() + return ctrl.Result{}, nil + } + return ctrl.Result{}, nil } - log.WithField("path", volumePath.ByPath).Debugf("Found host path") - - if err := fsRestore.Init(ctx, &datapath.FSBRInitParam{ - BSLName: pvr.Spec.BackupStorageLocation, - SourceNamespace: pvr.Spec.SourceNamespace, - UploaderType: pvr.Spec.UploaderType, - RepositoryType: podvolume.GetPvrRepositoryType(pvr), - RepoIdentifier: pvr.Spec.RepoIdentifier, - RepositoryEnsurer: c.repositoryEnsurer, - CredentialGetter: c.credentialGetter, - }); err != nil { - c.closeDataPath(ctx, pvr.Name) - return c.errorOut(ctx, pvr, err, "error to initialize data path", log) - } - - if err := fsRestore.StartRestore(pvr.Spec.SnapshotID, volumePath, pvr.Spec.UploaderSettings); err != nil { - c.closeDataPath(ctx, pvr.Name) - return c.errorOut(ctx, pvr, err, "error starting data path restore", log) - } - - log.WithField("path", volumePath.ByPath).Info("Async fs restore data path started") - return ctrl.Result{}, nil } -func (c *PodVolumeRestoreReconciler) errorOut(ctx context.Context, pvr *velerov1api.PodVolumeRestore, err error, msg string, log logrus.FieldLogger) (ctrl.Result, error) { - _ = UpdatePVRStatusToFailed(ctx, c.Client, pvr, errors.WithMessage(err, msg).Error(), c.clock.Now(), log) - return ctrl.Result{}, err +func (r *PodVolumeRestoreReconciler) acceptPodVolumeRestore(ctx context.Context, pvr *velerov1api.PodVolumeRestore) error { + return UpdatePVRWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, r.logger, func(pvr *velerov1api.PodVolumeRestore) bool { + pvr.Status.AcceptedTimestamp = &metav1.Time{Time: r.clock.Now()} + pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseAccepted + pvr.Status.Node = r.nodeName + + return true + }) } -func UpdatePVRStatusToFailed(ctx context.Context, c client.Client, pvb *velerov1api.PodVolumeRestore, errString string, time time.Time, log logrus.FieldLogger) error { - original := pvb.DeepCopy() - pvb.Status.Phase = velerov1api.PodVolumeRestorePhaseFailed - pvb.Status.Message = errString - pvb.Status.CompletionTimestamp = &metav1.Time{Time: time} +func (r *PodVolumeRestoreReconciler) tryCancelPodVolumeRestore(ctx context.Context, pvr *velerov1api.PodVolumeRestore, message string) bool { + log := r.logger.WithField("PVR", pvr.Name) + succeeded, err := funcExclusiveUpdatePodVolumeRestore(ctx, r.client, pvr, func(pvr *velerov1api.PodVolumeRestore) { + pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseCanceled + if pvr.Status.StartTimestamp.IsZero() { + pvr.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + } + pvr.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + + if message != "" { + pvr.Status.Message = message + } + }) - err := c.Patch(ctx, pvb, client.MergeFrom(original)) if err != nil { - log.WithError(err).Error("error updating PodVolumeRestore status") + log.WithError(err).Error("error updating PVR status") + return false + } else if !succeeded { + log.Warn("conflict in updating PVR status and will try it again later") + return false + } + + r.exposer.CleanUp(ctx, getPVROwnerObject(pvr)) + delete(r.cancelledPVR, pvr.Name) + + log.Warn("PVR is canceled") + + return true +} + +var funcExclusiveUpdatePodVolumeRestore = exclusiveUpdatePodVolumeRestore + +func exclusiveUpdatePodVolumeRestore(ctx context.Context, cli client.Client, pvr *velerov1api.PodVolumeRestore, + updateFunc func(*velerov1api.PodVolumeRestore)) (bool, error) { + updateFunc(pvr) + + err := cli.Update(ctx, pvr) + if err == nil { + return true, nil + } + + // warn we won't rollback pvr values in memory when error + if apierrors.IsConflict(err) { + return false, nil + } else { + return false, err + } +} + +func (r *PodVolumeRestoreReconciler) onPrepareTimeout(ctx context.Context, pvr *velerov1api.PodVolumeRestore) { + log := r.logger.WithField("PVR", pvr.Name) + + log.Info("Timeout happened for preparing PVR") + + succeeded, err := funcExclusiveUpdatePodVolumeRestore(ctx, r.client, pvr, func(pvr *velerov1api.PodVolumeRestore) { + pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseFailed + pvr.Status.Message = "timeout on preparing PVR" + }) + + if err != nil { + log.WithError(err).Warn("Failed to update PVR") + return + } + + if !succeeded { + log.Warn("PVR has been updated by others") + return + } + + diags := strings.Split(r.exposer.DiagnoseExpose(ctx, getPVROwnerObject(pvr)), "\n") + for _, diag := range diags { + log.Warnf("[Diagnose PVR expose]%s", diag) + } + + r.exposer.CleanUp(ctx, getPVROwnerObject(pvr)) + + log.Info("PVR has been cleaned up") +} + +func (r *PodVolumeRestoreReconciler) initCancelableDataPath(ctx context.Context, asyncBR datapath.AsyncBR, res *exposer.ExposeResult, log logrus.FieldLogger) error { + log.Info("Init cancelable PVR") + + if err := asyncBR.Init(ctx, nil); err != nil { + return errors.Wrap(err, "error initializing asyncBR") + } + + log.Infof("async data path init for pod %s, volume %s", res.ByPod.HostingPod.Name, res.ByPod.VolumeName) + + return nil +} + +func (r *PodVolumeRestoreReconciler) startCancelableDataPath(asyncBR datapath.AsyncBR, pvr *velerov1api.PodVolumeRestore, res *exposer.ExposeResult, log logrus.FieldLogger) error { + log.Info("Start cancelable PVR") + + if err := asyncBR.StartRestore(pvr.Spec.SnapshotID, datapath.AccessPoint{ + ByPath: res.ByPod.VolumeName, + }, pvr.Spec.UploaderSettings); err != nil { + return errors.Wrapf(err, "error starting async restore for pod %s, volume %s", res.ByPod.HostingPod.Name, res.ByPod.VolumeName) + } + + log.Infof("Async restore started for pod %s, volume %s", res.ByPod.HostingPod.Name, res.ByPod.VolumeName) + return nil +} + +func (r *PodVolumeRestoreReconciler) errorOut(ctx context.Context, pvr *velerov1api.PodVolumeRestore, err error, msg string, log logrus.FieldLogger) (ctrl.Result, error) { + r.exposer.CleanUp(ctx, getPVROwnerObject(pvr)) + + return ctrl.Result{}, UpdatePVRStatusToFailed(ctx, r.client, pvr, err, msg, r.clock.Now(), log) +} + +func UpdatePVRStatusToFailed(ctx context.Context, c client.Client, pvr *velerov1api.PodVolumeRestore, err error, msg string, time time.Time, log logrus.FieldLogger) error { + log.Info("update PVR status to Failed") + + if patchErr := UpdatePVRWithRetry(context.Background(), c, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, log, + func(pvr *velerov1api.PodVolumeRestore) bool { + if isPVRInFinalState(pvr) { + return false + } + + pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseFailed + pvr.Status.Message = errors.WithMessage(err, msg).Error() + pvr.Status.CompletionTimestamp = &metav1.Time{Time: time} + + return true + }); patchErr != nil { + log.WithError(patchErr).Warn("error updating PVR status") } return err } -func (c *PodVolumeRestoreReconciler) shouldProcess(ctx context.Context, log logrus.FieldLogger, pvr *velerov1api.PodVolumeRestore) (bool, *corev1api.Pod, error) { +func shouldProcess(ctx context.Context, client client.Client, log logrus.FieldLogger, pvr *velerov1api.PodVolumeRestore) (bool, *corev1api.Pod, error) { if !isPVRNew(pvr) { - log.Debug("PodVolumeRestore is not new, skip") + log.Debug("PVR is not new, skip") return false, nil, nil } // we filter the pods during the initialization of cache, if we can get a pod here, the pod must be in the same node with the controller // so we don't need to compare the node anymore pod := &corev1api.Pod{} - if err := c.Get(ctx, types.NamespacedName{Namespace: pvr.Spec.Pod.Namespace, Name: pvr.Spec.Pod.Name}, pod); err != nil { + if err := client.Get(ctx, types.NamespacedName{Namespace: pvr.Spec.Pod.Namespace, Name: pvr.Spec.Pod.Name}, pod); err != nil { if apierrors.IsNotFound(err) { log.WithError(err).Debug("Pod not found on this node, skip") return false, nil, nil @@ -214,39 +533,175 @@ func (c *PodVolumeRestoreReconciler) shouldProcess(ctx context.Context, log logr return true, pod, nil } -func (c *PodVolumeRestoreReconciler) SetupWithManager(mgr ctrl.Manager) error { - // The pod may not being scheduled at the point when its PVRs are initially reconciled. - // By watching the pods, we can trigger the PVR reconciliation again once the pod is finally scheduled on the node. - return ctrl.NewControllerManagedBy(mgr). - For(&velerov1api.PodVolumeRestore{}). - Watches(&corev1api.Pod{}, handler.EnqueueRequestsFromMapFunc(c.findVolumeRestoresForPod)). - Complete(c) +func (r *PodVolumeRestoreReconciler) closeDataPath(ctx context.Context, pvrName string) { + asyncBR := r.dataPathMgr.GetAsyncBR(pvrName) + if asyncBR != nil { + asyncBR.Close(ctx) + } + + r.dataPathMgr.RemoveAsyncBR(pvrName) } -func (c *PodVolumeRestoreReconciler) findVolumeRestoresForPod(ctx context.Context, pod client.Object) []reconcile.Request { +func (r *PodVolumeRestoreReconciler) SetupWithManager(mgr ctrl.Manager) error { + gp := kube.NewGenericEventPredicate(func(object client.Object) bool { + pvr := object.(*velerov1api.PodVolumeRestore) + if isLegacyPVR(pvr) { + return false + } + + if pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseAccepted { + return true + } + + if pvr.Spec.Cancel && !isPVRInFinalState(pvr) { + return true + } + + if isPVRInFinalState(pvr) && !pvr.DeletionTimestamp.IsZero() { + return true + } + + return false + }) + + s := kube.NewPeriodicalEnqueueSource(r.logger.WithField("controller", constant.ControllerPodVolumeRestore), r.client, &velerov1api.PodVolumeRestoreList{}, preparingMonitorFrequency, kube.PeriodicalEnqueueSourceOption{ + Predicates: []predicate.Predicate{gp}, + }) + + pred := kube.NewAllEventPredicate(func(obj client.Object) bool { + pvr := obj.(*velerov1.PodVolumeRestore) + return !isLegacyPVR(pvr) + }) + + return ctrl.NewControllerManagedBy(mgr). + For(&velerov1api.PodVolumeRestore{}, builder.WithPredicates(pred)). + WatchesRawSource(s). + Watches(&corev1api.Pod{}, handler.EnqueueRequestsFromMapFunc(r.findPVRForTargetPod)). + Watches(&corev1api.Pod{}, kube.EnqueueRequestsFromMapUpdateFunc(r.findPVRForRestorePod), + builder.WithPredicates(predicate.Funcs{ + UpdateFunc: func(ue event.UpdateEvent) bool { + newObj := ue.ObjectNew.(*corev1api.Pod) + + if _, ok := newObj.Labels[velerov1api.PVRLabel]; !ok { + return false + } + + if newObj.Spec.NodeName == "" { + return false + } + + return true + }, + CreateFunc: func(event.CreateEvent) bool { + return false + }, + DeleteFunc: func(de event.DeleteEvent) bool { + return false + }, + GenericFunc: func(ge event.GenericEvent) bool { + return false + }, + })). + Complete(r) +} + +func (r *PodVolumeRestoreReconciler) findPVRForTargetPod(ctx context.Context, pod client.Object) []reconcile.Request { list := &velerov1api.PodVolumeRestoreList{} options := &client.ListOptions{ LabelSelector: labels.Set(map[string]string{ velerov1api.PodUIDLabel: string(pod.GetUID()), }).AsSelector(), } - if err := c.List(context.TODO(), list, options); err != nil { - c.logger.WithField("pod", fmt.Sprintf("%s/%s", pod.GetNamespace(), pod.GetName())).WithError(err). + if err := r.client.List(context.TODO(), list, options); err != nil { + r.logger.WithField("pod", fmt.Sprintf("%s/%s", pod.GetNamespace(), pod.GetName())).WithError(err). Error("unable to list PodVolumeRestores") return []reconcile.Request{} } - requests := make([]reconcile.Request, len(list.Items)) - for i, item := range list.Items { - requests[i] = reconcile.Request{ + + requests := []reconcile.Request{} + for _, item := range list.Items { + if isLegacyPVR(&item) { + continue + } + + requests = append(requests, reconcile.Request{ NamespacedName: types.NamespacedName{ Namespace: item.GetNamespace(), Name: item.GetName(), }, - } + }) } return requests } +func (r *PodVolumeRestoreReconciler) findPVRForRestorePod(ctx context.Context, podObj client.Object) []reconcile.Request { + pod := podObj.(*corev1api.Pod) + pvr, err := findPVRByRestorePod(r.client, *pod) + + log := r.logger.WithField("pod", pod.Name) + if err != nil { + log.WithError(err).Error("unable to get PVR") + return []reconcile.Request{} + } else if pvr == nil { + log.Error("get empty PVR") + return []reconcile.Request{} + } + log = log.WithFields(logrus.Fields{ + "PVR": pvr.Name, + }) + + if pvr.Status.Phase != velerov1api.PodVolumeRestorePhaseAccepted { + return []reconcile.Request{} + } + + if pod.Status.Phase == corev1api.PodRunning { + log.Info("Preparing PVR") + + if err = UpdatePVRWithRetry(context.Background(), r.client, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, log, + func(pvr *velerov1api.PodVolumeRestore) bool { + if isPVRInFinalState(pvr) { + log.Warnf("PVR %s is terminated, abort setting it to prepared", pvr.Name) + return false + } + + pvr.Status.Phase = velerov1api.PodVolumeRestorePhasePrepared + return true + }); err != nil { + log.WithError(err).Warn("failed to update PVR, prepare will halt for this PVR") + return []reconcile.Request{} + } + } else if unrecoverable, reason := kube.IsPodUnrecoverable(pod, log); unrecoverable { + err := UpdatePVRWithRetry(context.Background(), r.client, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, log, + func(pvr *velerov1api.PodVolumeRestore) bool { + if pvr.Spec.Cancel { + return false + } + + pvr.Spec.Cancel = true + pvr.Status.Message = fmt.Sprintf("Cancel PVR because the exposing pod %s/%s is in abnormal status for reason %s", pod.Namespace, pod.Name, reason) + + return true + }) + + if err != nil { + log.WithError(err).Warn("failed to cancel PVR, and it will wait for prepare timeout") + return []reconcile.Request{} + } + + log.Infof("Exposed pod is in abnormal status(reason %s) and PVR is marked as cancel", reason) + } else { + return []reconcile.Request{} + } + + request := reconcile.Request{ + NamespacedName: types.NamespacedName{ + Namespace: pvr.Namespace, + Name: pvr.Name, + }, + } + return []reconcile.Request{request} +} + func isPVRNew(pvr *velerov1api.PodVolumeRestore) bool { return pvr.Status.Phase == "" || pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseNew } @@ -270,118 +725,186 @@ func getInitContainerIndex(pod *corev1api.Pod) int { return -1 } -func (c *PodVolumeRestoreReconciler) OnDataPathCompleted(ctx context.Context, namespace string, pvrName string, result datapath.Result) { - defer c.dataPathMgr.RemoveAsyncBR(pvrName) +func (r *PodVolumeRestoreReconciler) OnDataPathCompleted(ctx context.Context, namespace string, pvrName string, result datapath.Result) { + defer r.dataPathMgr.RemoveAsyncBR(pvrName) - log := c.logger.WithField("pvr", pvrName) + log := r.logger.WithField("PVR", pvrName) - log.WithField("PVR", pvrName).Info("Async fs restore data path completed") + log.WithField("PVR", pvrName).WithField("result", result.Restore).Info("Async fs restore data path completed") var pvr velerov1api.PodVolumeRestore - if err := c.Client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, &pvr); err != nil { + if err := r.client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, &pvr); err != nil { log.WithError(err).Warn("Failed to get PVR on completion") return } - volumePath := result.Restore.Target.ByPath - if volumePath == "" { - _, _ = c.errorOut(ctx, &pvr, errors.New("path is empty"), "invalid restore target", log) - return - } + log.Info("Cleaning up exposed environment") + r.exposer.CleanUp(ctx, getPVROwnerObject(&pvr)) - // Remove the .velero directory from the restored volume (it may contain done files from previous restores - // of this volume, which we don't want to carry over). If this fails for any reason, log and continue, since - // this is non-essential cleanup (the done files are named based on restore UID and the init container looks - // for the one specific to the restore being executed). - if err := os.RemoveAll(filepath.Join(volumePath, ".velero")); err != nil { - log.WithError(err).Warnf("error removing .velero directory from directory %s", volumePath) - } - - var restoreUID types.UID - for _, owner := range pvr.OwnerReferences { - if boolptr.IsSetToTrue(owner.Controller) { - restoreUID = owner.UID - break + if err := UpdatePVRWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, log, func(pvr *velerov1api.PodVolumeRestore) bool { + if isPVRInFinalState(pvr) { + return false } - } - // Create the .velero directory within the volume dir so we can write a done file - // for this restore. - if err := os.MkdirAll(filepath.Join(volumePath, ".velero"), 0755); err != nil { - _, _ = c.errorOut(ctx, &pvr, err, "error creating .velero directory for done file", log) - return - } + pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseCompleted + pvr.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} - // Write a done file with name= into the just-created .velero dir - // within the volume. The velero init container on the pod is waiting - // for this file to exist in each restored volume before completing. - if err := os.WriteFile(filepath.Join(volumePath, ".velero", string(restoreUID)), nil, 0644); err != nil { //nolint:gosec // Internal usage. No need to check. - _, _ = c.errorOut(ctx, &pvr, err, "error writing done file", log) - return + return true + }); err != nil { + log.WithError(err).Error("error updating PVR status") + } else { + log.Info("Restore completed") } - - original := pvr.DeepCopy() - pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseCompleted - pvr.Status.CompletionTimestamp = &metav1.Time{Time: c.clock.Now()} - if err := c.Patch(ctx, &pvr, client.MergeFrom(original)); err != nil { - log.WithError(err).Error("error updating PodVolumeRestore status") - } - - log.Info("Restore completed") } -func (c *PodVolumeRestoreReconciler) OnDataPathFailed(ctx context.Context, namespace string, pvrName string, err error) { - defer c.dataPathMgr.RemoveAsyncBR(pvrName) +func (r *PodVolumeRestoreReconciler) OnDataPathFailed(ctx context.Context, namespace string, pvrName string, err error) { + defer r.dataPathMgr.RemoveAsyncBR(pvrName) - log := c.logger.WithField("pvr", pvrName) + log := r.logger.WithField("PVR", pvrName) log.WithError(err).Error("Async fs restore data path failed") var pvr velerov1api.PodVolumeRestore - if getErr := c.Client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, &pvr); getErr != nil { + if getErr := r.client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, &pvr); getErr != nil { log.WithError(getErr).Warn("Failed to get PVR on failure") } else { - _, _ = c.errorOut(ctx, &pvr, err, "data path restore failed", log) + _, _ = r.errorOut(ctx, &pvr, err, "data path restore failed", log) } } -func (c *PodVolumeRestoreReconciler) OnDataPathCancelled(ctx context.Context, namespace string, pvrName string) { - defer c.dataPathMgr.RemoveAsyncBR(pvrName) +func (r *PodVolumeRestoreReconciler) OnDataPathCancelled(ctx context.Context, namespace string, pvrName string) { + defer r.dataPathMgr.RemoveAsyncBR(pvrName) - log := c.logger.WithField("pvr", pvrName) + log := r.logger.WithField("PVR", pvrName) log.Warn("Async fs restore data path canceled") var pvr velerov1api.PodVolumeRestore - if getErr := c.Client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, &pvr); getErr != nil { + if getErr := r.client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, &pvr); getErr != nil { log.WithError(getErr).Warn("Failed to get PVR on cancel") + return + } + // cleans up any objects generated during the snapshot expose + r.exposer.CleanUp(ctx, getPVROwnerObject(&pvr)) + + if err := UpdatePVRWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, log, func(pvr *velerov1api.PodVolumeRestore) bool { + if isPVRInFinalState(pvr) { + return false + } + + pvr.Status.Phase = velerov1api.PodVolumeRestorePhaseCanceled + if pvr.Status.StartTimestamp.IsZero() { + pvr.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + } + pvr.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + + return true + }); err != nil { + log.WithError(err).Error("error updating PVR status on cancel") } else { - _, _ = c.errorOut(ctx, &pvr, errors.New("PVR is canceled"), "data path restore canceled", log) + delete(r.cancelledPVR, pvr.Name) } } -func (c *PodVolumeRestoreReconciler) OnDataPathProgress(ctx context.Context, namespace string, pvrName string, progress *uploader.Progress) { - log := c.logger.WithField("pvr", pvrName) +func (r *PodVolumeRestoreReconciler) OnDataPathProgress(ctx context.Context, namespace string, pvrName string, progress *uploader.Progress) { + log := r.logger.WithField("PVR", pvrName) - var pvr velerov1api.PodVolumeRestore - if err := c.Client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, &pvr); err != nil { - log.WithError(err).Warn("Failed to get PVB on progress") - return - } - - original := pvr.DeepCopy() - pvr.Status.Progress = veleroapishared.DataMoveOperationProgress{TotalBytes: progress.TotalBytes, BytesDone: progress.BytesDone} - - if err := c.Client.Patch(ctx, &pvr, client.MergeFrom(original)); err != nil { + if err := UpdatePVRWithRetry(ctx, r.client, types.NamespacedName{Namespace: namespace, Name: pvrName}, log, func(pvr *velerov1api.PodVolumeRestore) bool { + pvr.Status.Progress = veleroapishared.DataMoveOperationProgress{TotalBytes: progress.TotalBytes, BytesDone: progress.BytesDone} + return true + }); err != nil { log.WithError(err).Error("Failed to update progress") } } -func (c *PodVolumeRestoreReconciler) closeDataPath(ctx context.Context, pvbName string) { - fsRestore := c.dataPathMgr.GetAsyncBR(pvbName) - if fsRestore != nil { - fsRestore.Close(ctx) +func (r *PodVolumeRestoreReconciler) setupExposeParam(pvr *velerov1api.PodVolumeRestore) exposer.PodVolumeExposeParam { + log := r.logger.WithField("PVR", pvr.Name) + + hostingPodLabels := map[string]string{velerov1api.PVRLabel: pvr.Name} + for _, k := range util.ThirdPartyLabels { + if v, err := nodeagent.GetLabelValue(context.Background(), r.kubeClient, pvr.Namespace, k, ""); err != nil { + if err != nodeagent.ErrNodeAgentLabelNotFound { + log.WithError(err).Warnf("Failed to check node-agent label, skip adding host pod label %s", k) + } + } else { + hostingPodLabels[k] = v + } } - c.dataPathMgr.RemoveAsyncBR(pvbName) + hostingPodAnnotation := map[string]string{} + for _, k := range util.ThirdPartyAnnotations { + if v, err := nodeagent.GetAnnotationValue(context.Background(), r.kubeClient, pvr.Namespace, k, ""); err != nil { + if err != nodeagent.ErrNodeAgentAnnotationNotFound { + log.WithError(err).Warnf("Failed to check node-agent annotation, skip adding host pod annotation %s", k) + } + } else { + hostingPodAnnotation[k] = v + } + } + + return exposer.PodVolumeExposeParam{ + Type: exposer.PodVolumeExposeTypeRestore, + ClientNamespace: pvr.Spec.Pod.Namespace, + ClientPodName: pvr.Spec.Pod.Name, + ClientPodVolume: pvr.Spec.Volume, + HostingPodLabels: hostingPodLabels, + HostingPodAnnotations: hostingPodAnnotation, + OperationTimeout: r.resourceTimeout, + Resources: r.podResources, + } +} + +func getPVROwnerObject(pvr *velerov1api.PodVolumeRestore) corev1api.ObjectReference { + return corev1api.ObjectReference{ + Kind: pvr.Kind, + Namespace: pvr.Namespace, + Name: pvr.Name, + UID: pvr.UID, + APIVersion: pvr.APIVersion, + } +} + +func findPVRByRestorePod(client client.Client, pod corev1api.Pod) (*velerov1api.PodVolumeRestore, error) { + if label, exist := pod.Labels[velerov1api.PVRLabel]; exist { + pvr := &velerov1api.PodVolumeRestore{} + err := client.Get(context.Background(), types.NamespacedName{ + Namespace: pod.Namespace, + Name: label, + }, pvr) + + if err != nil { + return nil, errors.Wrapf(err, "error to find PVR by pod %s/%s", pod.Namespace, pod.Name) + } + return pvr, nil + } + return nil, nil +} + +func isPVRInFinalState(pvr *velerov1api.PodVolumeRestore) bool { + return pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseFailed || + pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseCanceled || + pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseCompleted +} + +func UpdatePVRWithRetry(ctx context.Context, client client.Client, namespacedName types.NamespacedName, log logrus.FieldLogger, updateFunc func(*velerov1api.PodVolumeRestore) bool) error { + return wait.PollUntilContextCancel(ctx, time.Millisecond*100, true, func(ctx context.Context) (bool, error) { + pvr := &velerov1api.PodVolumeRestore{} + if err := client.Get(ctx, namespacedName, pvr); err != nil { + return false, errors.Wrap(err, "getting PVR") + } + + if updateFunc(pvr) { + err := client.Update(ctx, pvr) + if err != nil { + if apierrors.IsConflict(err) { + log.Warnf("failed to update PVR for %s/%s and will retry it", pvr.Namespace, pvr.Name) + return false, nil + } else { + return false, errors.Wrapf(err, "error updating PVR %s/%s", pvr.Namespace, pvr.Name) + } + } + } + + return true, nil + }) } diff --git a/pkg/controller/pod_volume_restore_controller_legacy.go b/pkg/controller/pod_volume_restore_controller_legacy.go new file mode 100644 index 000000000..03c8b4c56 --- /dev/null +++ b/pkg/controller/pod_volume_restore_controller_legacy.go @@ -0,0 +1,10 @@ +package controller + +import ( + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/uploader" +) + +func isLegacyPVR(pvr *velerov1api.PodVolumeRestore) bool { + return pvr.Spec.UploaderType == uploader.ResticType +} diff --git a/pkg/controller/pod_volume_restore_controller_test.go b/pkg/controller/pod_volume_restore_controller_test.go index f24aed653..5d4a3522e 100644 --- a/pkg/controller/pod_volume_restore_controller_test.go +++ b/pkg/controller/pod_volume_restore_controller_test.go @@ -18,21 +18,43 @@ package controller import ( "context" + "fmt" "testing" "time" + "github.com/pkg/errors" "github.com/sirupsen/logrus" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" "github.com/stretchr/testify/require" + appsv1api "k8s.io/api/apps/v1" corev1api "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/kubernetes" + clientgofake "k8s.io/client-go/kubernetes/fake" clocks "k8s.io/utils/clock" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + kbclient "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/manager" + "sigs.k8s.io/controller-runtime/pkg/reconcile" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/builder" + "github.com/vmware-tanzu/velero/pkg/datapath" + datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks" + "github.com/vmware-tanzu/velero/pkg/exposer" + exposermockes "github.com/vmware-tanzu/velero/pkg/exposer/mocks" "github.com/vmware-tanzu/velero/pkg/restorehelper" "github.com/vmware-tanzu/velero/pkg/test" + velerotest "github.com/vmware-tanzu/velero/pkg/test" + "github.com/vmware-tanzu/velero/pkg/uploader" + "github.com/vmware-tanzu/velero/pkg/util/boolptr" + "github.com/vmware-tanzu/velero/pkg/util/kube" ) func TestShouldProcess(t *testing.T) { @@ -195,11 +217,11 @@ func TestShouldProcess(t *testing.T) { c := &PodVolumeRestoreReconciler{ logger: logrus.New(), - Client: cli, + client: cli, clock: &clocks.RealClock{}, } - shouldProcess, _, _ := c.shouldProcess(ctx, c.logger, ts.obj) + shouldProcess, _, _ := shouldProcess(ctx, c.client, c.logger, ts.obj) require.Equal(t, ts.shouldProcessed, shouldProcess) }) } @@ -478,7 +500,7 @@ func TestGetInitContainerIndex(t *testing.T) { } } -func TestFindVolumeRestoresForPod(t *testing.T) { +func TestFindPVRForTargetPod(t *testing.T) { pod := &corev1api.Pod{} pod.UID = "uid" @@ -488,14 +510,14 @@ func TestFindVolumeRestoresForPod(t *testing.T) { // no matching PVR reconciler := &PodVolumeRestoreReconciler{ - Client: clientBuilder.Build(), + client: clientBuilder.Build(), logger: logrus.New(), } - requests := reconciler.findVolumeRestoresForPod(context.Background(), pod) + requests := reconciler.findPVRForTargetPod(context.Background(), pod) assert.Empty(t, requests) // contain one matching PVR - reconciler.Client = clientBuilder.WithLists(&velerov1api.PodVolumeRestoreList{ + reconciler.client = clientBuilder.WithLists(&velerov1api.PodVolumeRestoreList{ Items: []velerov1api.PodVolumeRestore{ { ObjectMeta: metav1.ObjectMeta{ @@ -515,6 +537,902 @@ func TestFindVolumeRestoresForPod(t *testing.T) { }, }, }).Build() - requests = reconciler.findVolumeRestoresForPod(context.Background(), pod) + requests = reconciler.findPVRForTargetPod(context.Background(), pod) assert.Len(t, requests, 1) } + +const pvrName string = "pvr-1" + +func pvrBuilder() *builder.PodVolumeRestoreBuilder { + return builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName). + BackupStorageLocation("bsl-loc"). + SnapshotID("test-snapshot-id") +} + +func initPodVolumeRestoreReconciler(objects []runtime.Object, cliObj []client.Object, needError ...bool) (*PodVolumeRestoreReconciler, error) { + var errs = make([]error, 6) + for k, isError := range needError { + if k == 0 && isError { + errs[0] = fmt.Errorf("Get error") + } else if k == 1 && isError { + errs[1] = fmt.Errorf("Create error") + } else if k == 2 && isError { + errs[2] = fmt.Errorf("Update error") + } else if k == 3 && isError { + errs[3] = fmt.Errorf("Patch error") + } else if k == 4 && isError { + errs[4] = apierrors.NewConflict(velerov1api.Resource("podvolumerestore"), pvrName, errors.New("conflict")) + } else if k == 5 && isError { + errs[5] = fmt.Errorf("List error") + } + } + return initPodVolumeRestoreReconcilerWithError(objects, cliObj, errs...) +} + +func initPodVolumeRestoreReconcilerWithError(objects []runtime.Object, cliObj []client.Object, needError ...error) (*PodVolumeRestoreReconciler, error) { + scheme := runtime.NewScheme() + err := velerov1api.AddToScheme(scheme) + if err != nil { + return nil, err + } + + err = corev1api.AddToScheme(scheme) + if err != nil { + return nil, err + } + + fakeClient := &FakeClient{ + Client: fake.NewClientBuilder().WithScheme(scheme).WithObjects(cliObj...).Build(), + } + + for k := range needError { + if k == 0 { + fakeClient.getError = needError[0] + } else if k == 1 { + fakeClient.createError = needError[1] + } else if k == 2 { + fakeClient.updateError = needError[2] + } else if k == 3 { + fakeClient.patchError = needError[3] + } else if k == 4 { + fakeClient.updateConflict = needError[4] + } else if k == 5 { + fakeClient.listError = needError[5] + } + } + + var fakeKubeClient *clientgofake.Clientset + if len(objects) != 0 { + fakeKubeClient = clientgofake.NewSimpleClientset(objects...) + } else { + fakeKubeClient = clientgofake.NewSimpleClientset() + } + + fakeFS := velerotest.NewFakeFileSystem() + pathGlob := fmt.Sprintf("/host_pods/%s/volumes/*/%s", "test-uid", "test-pvc") + _, err = fakeFS.Create(pathGlob) + if err != nil { + return nil, err + } + + dataPathMgr := datapath.NewManager(1) + + return NewPodVolumeRestoreReconciler(fakeClient, nil, fakeKubeClient, dataPathMgr, "test-node", time.Minute*5, time.Minute, corev1api.ResourceRequirements{}, velerotest.NewLogger()), nil +} + +func TestPodVolumeRestoreReconcile(t *testing.T) { + daemonSet := &appsv1api.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "velero", + Name: "node-agent", + }, + TypeMeta: metav1.TypeMeta{ + Kind: "DaemonSet", + APIVersion: appsv1api.SchemeGroupVersion.String(), + }, + Spec: appsv1api.DaemonSetSpec{ + Template: corev1api.PodTemplateSpec{ + Spec: corev1api.PodSpec{ + Containers: []corev1api.Container{ + { + Image: "fake-image", + }, + }, + }, + }, + }, + } + + node := builder.ForNode("fake-node").Labels(map[string]string{kube.NodeOSLabel: kube.NodeOSLinux}).Result() + + tests := []struct { + name string + pvr *velerov1api.PodVolumeRestore + notCreatePVR bool + targetPod *corev1api.Pod + dataMgr *datapath.Manager + needErrs []bool + needCreateFSBR bool + needDelete bool + sportTime *metav1.Time + mockExposeErr *bool + isGetExposeErr bool + isGetExposeNil bool + isPeekExposeErr bool + isNilExposer bool + notNilExpose bool + notMockCleanUp bool + mockInit bool + mockInitErr error + mockStart bool + mockStartErr error + mockCancel bool + mockClose bool + needExclusiveUpdateError error + expected *velerov1api.PodVolumeRestore + expectDeleted bool + expectCancelRecord bool + expectedResult *ctrl.Result + expectedErr string + expectDataPath bool + }{ + { + name: "pvr not found", + pvr: pvrBuilder().Result(), + notCreatePVR: true, + }, + { + name: "pvr not created in velero default namespace", + pvr: builder.ForPodVolumeRestore("test-ns", pvrName).Result(), + }, + { + name: "get dd fail", + pvr: builder.ForPodVolumeRestore("test-ns", pvrName).Result(), + needErrs: []bool{true, false, false, false}, + expectedErr: "Get error", + }, + { + name: "add finalizer to pvr", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Result(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "add finalizer to pvr failed", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Result(), + needErrs: []bool{false, false, true, false}, + expectedErr: "error updating PVR velero/pvr-1: Update error", + }, + { + name: "pvr is under deletion", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Result(), + needDelete: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Result(), + }, + { + name: "pvr is under deletion but cancel failed", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Result(), + needErrs: []bool{false, false, true, false}, + needDelete: true, + expectedErr: "error updating PVR velero/pvr-1: Update error", + }, + { + name: "pvr is under deletion and in terminal state", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeRestorePhaseFailed).Result(), + sportTime: &metav1.Time{Time: time.Now()}, + needDelete: true, + expectDeleted: true, + }, + { + name: "pvr is under deletion and in terminal state, but remove finalizer failed", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeRestorePhaseFailed).Result(), + needErrs: []bool{false, false, true, false}, + needDelete: true, + expectedErr: "error updating PVR velero/pvr-1: Update error", + }, + { + name: "delay cancel negative for others", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhasePrepared).Result(), + sportTime: &metav1.Time{Time: time.Now()}, + expectCancelRecord: true, + }, + { + name: "delay cancel negative for inProgress", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Result(), + sportTime: &metav1.Time{Time: time.Now().Add(-time.Minute * 58)}, + expectCancelRecord: true, + }, + { + name: "delay cancel affirmative for others", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhasePrepared).Result(), + sportTime: &metav1.Time{Time: time.Now().Add(-time.Minute * 5)}, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Result(), + }, + { + name: "delay cancel affirmative for inProgress", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Result(), + sportTime: &metav1.Time{Time: time.Now().Add(-time.Hour)}, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Result(), + }, + { + name: "delay cancel failed", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Result(), + needErrs: []bool{false, false, true, false}, + sportTime: &metav1.Time{Time: time.Now().Add(-time.Hour)}, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Result(), + expectCancelRecord: true, + }, + { + name: "Unknown pvr status", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase("Unknown").Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "new pvr but accept failed", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).PodNamespace("test-ns").PodName("test-pod").Result(), + targetPod: builder.ForPod("test-ns", "test-pod").InitContainers(&corev1api.Container{Name: restorehelper.WaitInitContainer}).InitContainerState(corev1api.ContainerState{Running: &corev1api.ContainerStateRunning{}}).Result(), + needErrs: []bool{false, false, true, false}, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Result(), + expectedErr: "error accepting PVR pvr-1: error updating PVR velero/pvr-1: Update error", + }, + { + name: "pvr is cancel on accepted", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Result(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Result(), + }, + { + name: "pvr expose failed", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).PodNamespace("test-ns").PodName("test-pod").Finalizers([]string{PodVolumeFinalizer}).Result(), + targetPod: builder.ForPod("test-ns", "test-pod").InitContainers(&corev1api.Container{Name: restorehelper.WaitInitContainer}).InitContainerState(corev1api.ContainerState{Running: &corev1api.ContainerStateRunning{}}).Result(), + mockExposeErr: boolptr.True(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeRestorePhaseFailed).Message("error to expose PVR").Result(), + expectedErr: "Error to expose restore exposer", + }, + { + name: "pvr succeeds for accepted", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).PodNamespace("test-ns").PodName("test-pod").Finalizers([]string{PodVolumeFinalizer}).Result(), + mockExposeErr: boolptr.False(), + notMockCleanUp: true, + targetPod: builder.ForPod("test-ns", "test-pod").InitContainers(&corev1api.Container{Name: restorehelper.WaitInitContainer}).InitContainerState(corev1api.ContainerState{Running: &corev1api.ContainerStateRunning{}}).Result(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeRestorePhaseAccepted).Result(), + }, + { + name: "prepare timeout on accepted", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseAccepted).Finalizers([]string{PodVolumeFinalizer}).AcceptedTimestamp(&metav1.Time{Time: time.Now().Add(-time.Minute * 30)}).Result(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeRestorePhaseFailed).Message("timeout on preparing PVR").Result(), + }, + { + name: "peek error on accepted", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseAccepted).Finalizers([]string{PodVolumeFinalizer}).Result(), + isPeekExposeErr: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Message("found a PVR velero/pvr-1 with expose error: fake-peek-error. mark it as cancel").Result(), + }, + { + name: "cancel on pvr", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Node("test-node").Result(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Result(), + }, + { + name: "Failed to get restore expose on prepared", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + isGetExposeErr: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Message("exposed PVR is not ready").Result(), + expectedErr: "Error to get PVR exposer", + }, + { + name: "Get nil restore expose on prepared", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + isGetExposeNil: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Message("exposed PVR is not ready").Result(), + expectedErr: "no expose result is available for the current node", + }, + { + name: "Error in data path is concurrent limited", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + dataMgr: datapath.NewManager(0), + notNilExpose: true, + notMockCleanUp: true, + expectedResult: &ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, + }, + { + name: "data path init error", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + mockInit: true, + mockInitErr: errors.New("fake-data-path-init-error"), + mockClose: true, + notNilExpose: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Message("error initializing data path").Result(), + expectedErr: "error initializing asyncBR: fake-data-path-init-error", + }, + { + name: "Unable to update status to in progress for pvr", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needErrs: []bool{false, false, true, false}, + mockInit: true, + mockClose: true, + notNilExpose: true, + notMockCleanUp: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "data path start error", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + mockInit: true, + mockStart: true, + mockStartErr: errors.New("fake-data-path-start-error"), + mockClose: true, + notNilExpose: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Message("error starting data path").Result(), + expectedErr: "error starting async restore for pod test-name, volume test-pvc: fake-data-path-start-error", + }, + { + name: "Prepare succeeds", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + mockInit: true, + mockStart: true, + notNilExpose: true, + notMockCleanUp: true, + expectDataPath: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "In progress pvr is not handled by the current node", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Result(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "In progress pvr is not set as cancel", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "Cancel pvr in progress with empty FSBR", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "Cancel pvr in progress and patch pvr error", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needErrs: []bool{false, false, true, false}, + needCreateFSBR: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Result(), + expectedErr: "error updating PVR velero/pvr-1: Update error", + expectCancelRecord: true, + expectDataPath: true, + }, + { + name: "Cancel pvr in progress succeeds", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseInProgress).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needCreateFSBR: true, + mockCancel: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Phase(velerov1api.PodVolumeRestorePhaseCanceling).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Result(), + expectDataPath: true, + expectCancelRecord: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + objs := []runtime.Object{daemonSet, node} + + ctlObj := []client.Object{} + if test.targetPod != nil { + ctlObj = append(ctlObj, test.targetPod) + } + + r, err := initPodVolumeRestoreReconciler(objs, ctlObj, test.needErrs...) + require.NoError(t, err) + + if !test.notCreatePVR { + err = r.client.Create(context.Background(), test.pvr) + require.NoError(t, err) + } + + if test.needDelete { + err = r.client.Delete(context.Background(), test.pvr) + require.NoError(t, err) + } + + if test.dataMgr != nil { + r.dataPathMgr = test.dataMgr + } else { + r.dataPathMgr = datapath.NewManager(1) + } + + if test.sportTime != nil { + r.cancelledPVR[test.pvr.Name] = test.sportTime.Time + } + + funcExclusiveUpdatePodVolumeRestore = exclusiveUpdatePodVolumeRestore + if test.needExclusiveUpdateError != nil { + funcExclusiveUpdatePodVolumeRestore = func(context.Context, kbclient.Client, *velerov1api.PodVolumeRestore, func(*velerov1api.PodVolumeRestore)) (bool, error) { + return false, test.needExclusiveUpdateError + } + } + + datapath.MicroServiceBRWatcherCreator = func(kbclient.Client, kubernetes.Interface, manager.Manager, string, string, + string, string, string, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { + asyncBR := datapathmockes.NewAsyncBR(t) + if test.mockInit { + asyncBR.On("Init", mock.Anything, mock.Anything).Return(test.mockInitErr) + } + + if test.mockStart { + asyncBR.On("StartRestore", mock.Anything, mock.Anything, mock.Anything).Return(test.mockStartErr) + } + + if test.mockCancel { + asyncBR.On("Cancel").Return() + } + + if test.mockClose { + asyncBR.On("Close", mock.Anything).Return() + } + + return asyncBR + } + + if test.mockExposeErr != nil || test.isGetExposeErr || test.isGetExposeNil || test.isPeekExposeErr || test.isNilExposer || test.notNilExpose { + if test.isNilExposer { + r.exposer = nil + } else { + r.exposer = func() exposer.PodVolumeExposer { + ep := exposermockes.NewPodVolumeExposer(t) + if test.mockExposeErr != nil { + if boolptr.IsSetToTrue(test.mockExposeErr) { + ep.On("Expose", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("Error to expose restore exposer")) + } else { + ep.On("Expose", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil) + } + } else if test.notNilExpose { + hostingPod := builder.ForPod("test-ns", "test-name").Volumes(&corev1api.Volume{Name: "test-pvc"}).Result() + hostingPod.ObjectMeta.SetUID("test-uid") + ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(&exposer.ExposeResult{ByPod: exposer.ExposeByPod{HostingPod: hostingPod, VolumeName: "test-pvc"}}, nil) + } else if test.isGetExposeErr { + ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil, errors.New("Error to get PVR exposer")) + } else if test.isGetExposeNil { + ep.On("GetExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(nil, nil) + } else if test.isPeekExposeErr { + ep.On("PeekExposed", mock.Anything, mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(errors.New("fake-peek-error")) + } + + if !test.notMockCleanUp { + ep.On("CleanUp", mock.Anything, mock.Anything).Return() + } + return ep + }() + } + } + + if test.needCreateFSBR { + if fsBR := r.dataPathMgr.GetAsyncBR(test.pvr.Name); fsBR == nil { + _, err := r.dataPathMgr.CreateMicroServiceBRWatcher(ctx, r.client, nil, nil, datapath.TaskTypeRestore, test.pvr.Name, pVBRRequestor, + velerov1api.DefaultNamespace, "", "", datapath.Callbacks{OnCancelled: r.OnDataPathCancelled}, false, velerotest.NewLogger()) + require.NoError(t, err) + } + } + + actualResult, err := r.Reconcile(ctx, ctrl.Request{ + NamespacedName: types.NamespacedName{ + Namespace: velerov1api.DefaultNamespace, + Name: test.pvr.Name, + }, + }) + + if test.expectedErr != "" { + assert.EqualError(t, err, test.expectedErr) + } else { + assert.NoError(t, err) + } + + if test.expectedResult != nil { + assert.Equal(t, test.expectedResult.Requeue, actualResult.Requeue) + assert.Equal(t, test.expectedResult.RequeueAfter, actualResult.RequeueAfter) + } + + if test.expected != nil || test.expectDeleted { + pvr := velerov1api.PodVolumeRestore{} + err = r.client.Get(ctx, kbclient.ObjectKey{ + Name: test.pvr.Name, + Namespace: test.pvr.Namespace, + }, &pvr) + + if test.expectDeleted { + assert.True(t, apierrors.IsNotFound(err)) + } else { + require.NoError(t, err) + + assert.Equal(t, test.expected.Status.Phase, pvr.Status.Phase) + assert.Contains(t, pvr.Status.Message, test.expected.Status.Message) + assert.Equal(t, test.expected.Finalizers, pvr.Finalizers) + assert.Equal(t, test.expected.Spec.Cancel, pvr.Spec.Cancel) + } + } + + if !test.expectDataPath { + assert.Nil(t, r.dataPathMgr.GetAsyncBR(test.pvr.Name)) + } else { + assert.NotNil(t, r.dataPathMgr.GetAsyncBR(test.pvr.Name)) + } + + if test.expectCancelRecord { + assert.Contains(t, r.cancelledPVR, test.pvr.Name) + } else { + assert.Empty(t, r.cancelledPVR) + } + }) + } +} + +func TestOnPodVolumeRestoreFailed(t *testing.T) { + for _, getErr := range []bool{true, false} { + ctx := context.TODO() + needErrs := []bool{getErr, false, false, false} + r, err := initPodVolumeRestoreReconciler(nil, []client.Object{}, needErrs...) + require.NoError(t, err) + + pvr := pvrBuilder().Result() + namespace := pvr.Namespace + pvrName := pvr.Name + + assert.NoError(t, r.client.Create(ctx, pvr)) + r.OnDataPathFailed(ctx, namespace, pvrName, fmt.Errorf("Failed to handle %v", pvrName)) + updatedPVR := &velerov1api.PodVolumeRestore{} + if getErr { + assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, updatedPVR)) + assert.NotEqual(t, velerov1api.PodVolumeRestorePhaseFailed, updatedPVR.Status.Phase) + assert.True(t, updatedPVR.Status.StartTimestamp.IsZero()) + } else { + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, updatedPVR)) + assert.Equal(t, velerov1api.PodVolumeRestorePhaseFailed, updatedPVR.Status.Phase) + assert.True(t, updatedPVR.Status.StartTimestamp.IsZero()) + } + } +} + +func TestOnPodVolumeRestoreCancelled(t *testing.T) { + for _, getErr := range []bool{true, false} { + ctx := context.TODO() + needErrs := []bool{getErr, false, false, false} + r, err := initPodVolumeRestoreReconciler(nil, nil, needErrs...) + require.NoError(t, err) + + pvr := pvrBuilder().Result() + namespace := pvr.Namespace + pvrName := pvr.Name + + assert.NoError(t, r.client.Create(ctx, pvr)) + r.OnDataPathCancelled(ctx, namespace, pvrName) + updatedPVR := &velerov1api.PodVolumeRestore{} + if getErr { + assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, updatedPVR)) + assert.NotEqual(t, velerov1api.PodVolumeRestorePhaseFailed, updatedPVR.Status.Phase) + assert.True(t, updatedPVR.Status.StartTimestamp.IsZero()) + } else { + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, updatedPVR)) + assert.Equal(t, velerov1api.PodVolumeRestorePhaseCanceled, updatedPVR.Status.Phase) + assert.False(t, updatedPVR.Status.StartTimestamp.IsZero()) + assert.False(t, updatedPVR.Status.CompletionTimestamp.IsZero()) + } + } +} + +func TestOnPodVolumeRestoreCompleted(t *testing.T) { + tests := []struct { + name string + emptyFSBR bool + isGetErr bool + rebindVolumeErr bool + }{ + { + name: "PVR complete", + emptyFSBR: false, + isGetErr: false, + rebindVolumeErr: false, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ctx := context.TODO() + needErrs := []bool{test.isGetErr, false, false, false} + r, err := initPodVolumeRestoreReconciler(nil, []client.Object{}, needErrs...) + r.exposer = func() exposer.PodVolumeExposer { + ep := exposermockes.NewPodVolumeExposer(t) + ep.On("CleanUp", mock.Anything, mock.Anything).Return() + return ep + }() + + require.NoError(t, err) + pvr := builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Result() + namespace := pvr.Namespace + ddName := pvr.Name + + assert.NoError(t, r.client.Create(ctx, pvr)) + r.OnDataPathCompleted(ctx, namespace, ddName, datapath.Result{}) + updatedDD := &velerov1api.PodVolumeRestore{} + if test.isGetErr { + assert.Error(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD)) + assert.Equal(t, velerov1api.PodVolumeRestorePhase(""), updatedDD.Status.Phase) + assert.True(t, updatedDD.Status.CompletionTimestamp.IsZero()) + } else { + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: ddName, Namespace: namespace}, updatedDD)) + assert.Equal(t, velerov1api.PodVolumeRestorePhaseCompleted, updatedDD.Status.Phase) + assert.False(t, updatedDD.Status.CompletionTimestamp.IsZero()) + } + }) + } +} + +func TestOnPodVolumeRestoreProgress(t *testing.T) { + totalBytes := int64(1024) + bytesDone := int64(512) + tests := []struct { + name string + pvr *velerov1api.PodVolumeRestore + progress uploader.Progress + needErrs []bool + }{ + { + name: "patch in progress phase success", + pvr: pvrBuilder().Result(), + progress: uploader.Progress{ + TotalBytes: totalBytes, + BytesDone: bytesDone, + }, + }, + { + name: "failed to get pvr", + pvr: pvrBuilder().Result(), + needErrs: []bool{true, false, false, false}, + }, + { + name: "failed to patch pvr", + pvr: pvrBuilder().Result(), + needErrs: []bool{false, false, true, false}, + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ctx := context.TODO() + + r, err := initPodVolumeRestoreReconciler(nil, []client.Object{}, test.needErrs...) + require.NoError(t, err) + defer func() { + r.client.Delete(ctx, test.pvr, &kbclient.DeleteOptions{}) + }() + + pvr := pvrBuilder().Result() + namespace := pvr.Namespace + pvrName := pvr.Name + + assert.NoError(t, r.client.Create(context.Background(), pvr)) + + // Create a Progress object + progress := &uploader.Progress{ + TotalBytes: totalBytes, + BytesDone: bytesDone, + } + + r.OnDataPathProgress(ctx, namespace, pvrName, progress) + if len(test.needErrs) != 0 && !test.needErrs[0] { + updatedPVR := &velerov1api.PodVolumeRestore{} + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: pvrName, Namespace: namespace}, updatedPVR)) + assert.Equal(t, test.progress.TotalBytes, updatedPVR.Status.Progress.TotalBytes) + assert.Equal(t, test.progress.BytesDone, updatedPVR.Status.Progress.BytesDone) + } + }) + } +} + +func TestFindPVBForRestorePod(t *testing.T) { + needErrs := []bool{false, false, false, false} + r, err := initPodVolumeRestoreReconciler(nil, []client.Object{}, needErrs...) + require.NoError(t, err) + tests := []struct { + name string + pvr *velerov1api.PodVolumeRestore + pod *corev1api.Pod + checkFunc func(*velerov1api.PodVolumeRestore, []reconcile.Request) + }{ + { + name: "find pvr for pod", + pvr: pvrBuilder().Phase(velerov1api.PodVolumeRestorePhaseAccepted).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, pvrName).Labels(map[string]string{velerov1api.PVRLabel: pvrName}).Status(corev1api.PodStatus{Phase: corev1api.PodRunning}).Result(), + checkFunc: func(pvr *velerov1api.PodVolumeRestore, requests []reconcile.Request) { + // Assert that the function returns a single request + assert.Len(t, requests, 1) + // Assert that the request contains the correct namespaced name + assert.Equal(t, pvr.Namespace, requests[0].Namespace) + assert.Equal(t, pvr.Name, requests[0].Name) + }, + }, { + name: "no selected label found for pod", + pvr: pvrBuilder().Phase(velerov1api.PodVolumeRestorePhaseAccepted).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, pvrName).Result(), + checkFunc: func(pvr *velerov1api.PodVolumeRestore, requests []reconcile.Request) { + // Assert that the function returns a single request + assert.Empty(t, requests) + }, + }, { + name: "no matched pod", + pvr: pvrBuilder().Phase(velerov1api.PodVolumeRestorePhaseAccepted).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, pvrName).Labels(map[string]string{velerov1api.PVRLabel: "non-existing-pvr"}).Result(), + checkFunc: func(pvr *velerov1api.PodVolumeRestore, requests []reconcile.Request) { + assert.Empty(t, requests) + }, + }, + { + name: "pvr not accept", + pvr: pvrBuilder().Phase(velerov1api.PodVolumeRestorePhaseInProgress).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, pvrName).Labels(map[string]string{velerov1api.PVRLabel: pvrName}).Result(), + checkFunc: func(pvr *velerov1api.PodVolumeRestore, requests []reconcile.Request) { + assert.Empty(t, requests) + }, + }, + } + for _, test := range tests { + ctx := context.Background() + assert.NoError(t, r.client.Create(ctx, test.pod)) + assert.NoError(t, r.client.Create(ctx, test.pvr)) + // Call the findSnapshotRestoreForPod function + requests := r.findPVRForRestorePod(context.Background(), test.pod) + test.checkFunc(test.pvr, requests) + r.client.Delete(ctx, test.pvr, &kbclient.DeleteOptions{}) + if test.pod != nil { + r.client.Delete(ctx, test.pod, &kbclient.DeleteOptions{}) + } + } +} + +func TestOnPVRPrepareTimeout(t *testing.T) { + tests := []struct { + name string + pvr *velerov1api.PodVolumeRestore + needErrs []error + expected *velerov1api.PodVolumeRestore + }{ + { + name: "update fail", + pvr: pvrBuilder().Result(), + needErrs: []error{nil, nil, fmt.Errorf("fake-update-error"), nil}, + expected: pvrBuilder().Result(), + }, + { + name: "update interrupted", + pvr: pvrBuilder().Result(), + needErrs: []error{nil, nil, &fakeAPIStatus{metav1.StatusReasonConflict}, nil}, + expected: pvrBuilder().Result(), + }, + { + name: "succeed", + pvr: pvrBuilder().Result(), + needErrs: []error{nil, nil, nil, nil}, + expected: pvrBuilder().Phase(velerov1api.PodVolumeRestorePhaseFailed).Result(), + }, + } + for _, test := range tests { + ctx := context.Background() + r, err := initPodVolumeRestoreReconcilerWithError(nil, []client.Object{}, test.needErrs...) + require.NoError(t, err) + + err = r.client.Create(ctx, test.pvr) + require.NoError(t, err) + + r.onPrepareTimeout(ctx, test.pvr) + + pvr := velerov1api.PodVolumeRestore{} + _ = r.client.Get(ctx, kbclient.ObjectKey{ + Name: test.pvr.Name, + Namespace: test.pvr.Namespace, + }, &pvr) + + assert.Equal(t, test.expected.Status.Phase, pvr.Status.Phase) + } +} + +func TestTryCancelPVR(t *testing.T) { + tests := []struct { + name string + pvr *velerov1api.PodVolumeRestore + needErrs []error + succeeded bool + expectedErr string + }{ + { + name: "update fail", + pvr: pvrBuilder().Result(), + needErrs: []error{nil, nil, fmt.Errorf("fake-update-error"), nil}, + }, + { + name: "cancel by others", + pvr: pvrBuilder().Result(), + needErrs: []error{nil, nil, &fakeAPIStatus{metav1.StatusReasonConflict}, nil}, + }, + { + name: "succeed", + pvr: pvrBuilder().Result(), + needErrs: []error{nil, nil, nil, nil}, + succeeded: true, + }, + } + for _, test := range tests { + ctx := context.Background() + r, err := initPodVolumeRestoreReconcilerWithError(nil, []client.Object{}, test.needErrs...) + require.NoError(t, err) + + err = r.client.Create(ctx, test.pvr) + require.NoError(t, err) + + r.tryCancelPodVolumeRestore(ctx, test.pvr, "") + + if test.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, test.expectedErr) + } + } +} + +func TestUpdatePVRWithRetry(t *testing.T) { + namespacedName := types.NamespacedName{ + Name: pvrName, + Namespace: "velero", + } + + // Define test cases + testCases := []struct { + Name string + needErrs []bool + noChange bool + ExpectErr bool + }{ + { + Name: "SuccessOnFirstAttempt", + }, + { + Name: "Error get", + needErrs: []bool{true, false, false, false, false}, + ExpectErr: true, + }, + { + Name: "Error update", + needErrs: []bool{false, false, true, false, false}, + ExpectErr: true, + }, + { + Name: "no change", + noChange: true, + needErrs: []bool{false, false, true, false, false}, + }, + { + Name: "Conflict with error timeout", + needErrs: []bool{false, false, false, false, true}, + ExpectErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.Name, func(t *testing.T) { + ctx, cancelFunc := context.WithTimeout(context.TODO(), time.Second*5) + defer cancelFunc() + r, err := initPodVolumeRestoreReconciler(nil, []client.Object{}, tc.needErrs...) + require.NoError(t, err) + err = r.client.Create(ctx, pvrBuilder().Result()) + require.NoError(t, err) + updateFunc := func(pvr *velerov1api.PodVolumeRestore) bool { + if tc.noChange { + return false + } + + pvr.Spec.Cancel = true + + return true + } + err = UpdatePVRWithRetry(ctx, r.client, namespacedName, velerotest.NewLogger().WithField("name", tc.Name), updateFunc) + if tc.ExpectErr { + assert.Error(t, err) + } else { + assert.NoError(t, err) + } + }) + } +} diff --git a/pkg/exposer/mocks/PodVolumeExposer.go b/pkg/exposer/mocks/PodVolumeExposer.go new file mode 100644 index 000000000..fbd5749ba --- /dev/null +++ b/pkg/exposer/mocks/PodVolumeExposer.go @@ -0,0 +1,125 @@ +// Code generated by mockery v2.39.1. DO NOT EDIT. + +package mocks + +import ( + context "context" + + client "sigs.k8s.io/controller-runtime/pkg/client" + + exposer "github.com/vmware-tanzu/velero/pkg/exposer" + + mock "github.com/stretchr/testify/mock" + + time "time" + + corev1api "k8s.io/api/core/v1" +) + +// PodVolumeExposer is an autogenerated mock type for the PodVolumeExposer type +type PodVolumeExposer struct { + mock.Mock +} + +// CleanUp provides a mock function with given fields: _a0, _a1 +func (_m *PodVolumeExposer) CleanUp(_a0 context.Context, _a1 corev1api.ObjectReference) { + _m.Called(_a0, _a1) +} + +// DiagnoseExpose provides a mock function with given fields: _a0, _a1 +func (_m *PodVolumeExposer) DiagnoseExpose(_a0 context.Context, _a1 corev1api.ObjectReference) string { + ret := _m.Called(_a0, _a1) + + if len(ret) == 0 { + panic("no return value specified for DiagnoseExpose") + } + + var r0 string + if rf, ok := ret.Get(0).(func(context.Context, corev1api.ObjectReference) string); ok { + r0 = rf(_a0, _a1) + } else { + r0 = ret.Get(0).(string) + } + + return r0 +} + +// Expose provides a mock function with given fields: _a0, _a1, _a2 +func (_m *PodVolumeExposer) Expose(_a0 context.Context, _a1 corev1api.ObjectReference, _a2 exposer.PodVolumeExposeParam) error { + ret := _m.Called(_a0, _a1, _a2) + + if len(ret) == 0 { + panic("no return value specified for Expose") + } + + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, corev1api.ObjectReference, exposer.PodVolumeExposeParam) error); ok { + r0 = rf(_a0, _a1, _a2) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// GetExposed provides a mock function with given fields: _a0, _a1, _a2, _a3, _a4 +func (_m *PodVolumeExposer) GetExposed(_a0 context.Context, _a1 corev1api.ObjectReference, _a2 client.Client, _a3 string, _a4 time.Duration) (*exposer.ExposeResult, error) { + ret := _m.Called(_a0, _a1, _a2, _a3, _a4) + + if len(ret) == 0 { + panic("no return value specified for GetExposed") + } + + var r0 *exposer.ExposeResult + var r1 error + if rf, ok := ret.Get(0).(func(context.Context, corev1api.ObjectReference, client.Client, string, time.Duration) (*exposer.ExposeResult, error)); ok { + return rf(_a0, _a1, _a2, _a3, _a4) + } + if rf, ok := ret.Get(0).(func(context.Context, corev1api.ObjectReference, client.Client, string, time.Duration) *exposer.ExposeResult); ok { + r0 = rf(_a0, _a1, _a2, _a3, _a4) + } else { + if ret.Get(0) != nil { + r0 = ret.Get(0).(*exposer.ExposeResult) + } + } + + if rf, ok := ret.Get(1).(func(context.Context, corev1api.ObjectReference, client.Client, string, time.Duration) error); ok { + r1 = rf(_a0, _a1, _a2, _a3, _a4) + } else { + r1 = ret.Error(1) + } + + return r0, r1 +} + +// PeekExposed provides a mock function with given fields: _a0, _a1 +func (_m *PodVolumeExposer) PeekExposed(_a0 context.Context, _a1 corev1api.ObjectReference) error { + ret := _m.Called(_a0, _a1) + + if len(ret) == 0 { + panic("no return value specified for PeekExposed") + } + + var r0 error + if rf, ok := ret.Get(0).(func(context.Context, corev1api.ObjectReference) error); ok { + r0 = rf(_a0, _a1) + } else { + r0 = ret.Error(0) + } + + return r0 +} + +// NewPodVolumeExposer creates a new instance of PodVolumeExposer. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations. +// The first argument is typically a *testing.T value. +func NewPodVolumeExposer(t interface { + mock.TestingT + Cleanup(func()) +}) *PodVolumeExposer { + mock := &PodVolumeExposer{} + mock.Mock.Test(t) + + t.Cleanup(func() { mock.AssertExpectations(t) }) + + return mock +} diff --git a/pkg/podvolume/restorer.go b/pkg/podvolume/restorer.go index 1c50b79d9..bebbeee3e 100644 --- a/pkg/podvolume/restorer.go +++ b/pkg/podvolume/restorer.go @@ -98,7 +98,7 @@ func newRestorer( return } - if pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseCompleted || pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseFailed { + if pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseCompleted || pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseFailed || pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseCanceled { r.resultsLock.Lock() defer r.resultsLock.Unlock() @@ -234,7 +234,7 @@ ForEachVolume: errs = append(errs, errors.New("timed out waiting for all PodVolumeRestores to complete")) break ForEachVolume case res := <-resultsChan: - if res.Status.Phase == velerov1api.PodVolumeRestorePhaseFailed { + if res.Status.Phase == velerov1api.PodVolumeRestorePhaseFailed || res.Status.Phase == velerov1api.PodVolumeRestorePhaseCanceled { errs = append(errs, errors.Errorf("pod volume restore failed: %s", res.Status.Message)) } tracker.TrackPodVolume(res) From ac4cf70d67036e78fdf450d999cde2b39e9a83f6 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Fri, 6 Jun 2025 15:53:34 +0800 Subject: [PATCH 24/31] vgdp ms pvr data path Signed-off-by: Lyndon-Li --- pkg/builder/pod_volume_restore_builder.go | 6 + pkg/cmd/cli/podvolume/restore.go | 13 ++ pkg/cmd/cli/podvolume/restore_test.go | 166 ++++++++++++++++++++ pkg/podvolume/restore_micro_service.go | 10 +- pkg/podvolume/restore_micro_service_test.go | 152 ++++++++++++++++++ 5 files changed, 344 insertions(+), 3 deletions(-) create mode 100644 pkg/cmd/cli/podvolume/restore_test.go diff --git a/pkg/builder/pod_volume_restore_builder.go b/pkg/builder/pod_volume_restore_builder.go index 3d4da94d6..47acd3cd0 100644 --- a/pkg/builder/pod_volume_restore_builder.go +++ b/pkg/builder/pod_volume_restore_builder.go @@ -97,3 +97,9 @@ func (b *PodVolumeRestoreBuilder) UploaderType(uploaderType string) *PodVolumeRe b.object.Spec.UploaderType = uploaderType return b } + +// OwnerReference sets the OwnerReference for this PodVolumeRestore. +func (b *PodVolumeRestoreBuilder) OwnerReference(ownerRef []metav1.OwnerReference) *PodVolumeRestoreBuilder { + b.object.OwnerReferences = ownerRef + return b +} diff --git a/pkg/cmd/cli/podvolume/restore.go b/pkg/cmd/cli/podvolume/restore.go index ba7a838d7..8d5924532 100644 --- a/pkg/cmd/cli/podvolume/restore.go +++ b/pkg/cmd/cli/podvolume/restore.go @@ -92,6 +92,19 @@ func NewRestoreCommand(f client.Factory) *cobra.Command { _ = command.MarkFlagRequired("pod-volume-restore") _ = command.MarkFlagRequired("resource-timeout") + command.PreRunE = func(cmd *cobra.Command, args []string) error { + if config.resourceTimeout <= 0 { + return errors.New("resource-timeout must be greater than 0") + } + if config.volumePath == "" { + return errors.New("volume-path cannot be empty") + } + if config.pvrName == "" { + return errors.New("pod-volume-restore name cannot be empty") + } + return nil + } + return command } diff --git a/pkg/cmd/cli/podvolume/restore_test.go b/pkg/cmd/cli/podvolume/restore_test.go new file mode 100644 index 000000000..74fbb8090 --- /dev/null +++ b/pkg/cmd/cli/podvolume/restore_test.go @@ -0,0 +1,166 @@ +/* +Copyright The Velero Contributors. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package podvolume + +import ( + "errors" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/mock" + + cacheMock "github.com/vmware-tanzu/velero/pkg/cmd/cli/datamover/mocks" + velerotest "github.com/vmware-tanzu/velero/pkg/test" +) + +func fakeCreateRestoreDataPathServiceWithErr(_ *podVolumeRestore) (dataPathService, error) { + return nil, errors.New("fake-create-data-path-error") +} + +func fakeCreateRestoreDataPathService(_ *podVolumeRestore) (dataPathService, error) { + return frHelper, nil +} + +func TestRunRestoreDataPath(t *testing.T) { + tests := []struct { + name string + pvrName string + createDataPathFail bool + initDataPathErr error + runCancelableDataPathErr error + runCancelableDataPathResult string + expectedMessage string + expectedSucceed bool + }{ + { + name: "create data path failed", + pvrName: "fake-name", + createDataPathFail: true, + expectedMessage: "Failed to create data path service for PVR fake-name: fake-create-data-path-error", + }, + { + name: "init data path failed", + pvrName: "fake-name", + initDataPathErr: errors.New("fake-init-data-path-error"), + expectedMessage: "Failed to init data path service for PVR fake-name: fake-init-data-path-error", + }, + { + name: "run data path failed", + pvrName: "fake-name", + runCancelableDataPathErr: errors.New("fake-run-data-path-error"), + expectedMessage: "Failed to run data path service for PVR fake-name: fake-run-data-path-error", + }, + { + name: "succeed", + pvrName: "fake-name", + runCancelableDataPathResult: "fake-run-data-path-result", + expectedMessage: "fake-run-data-path-result", + expectedSucceed: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + frHelper = &fakeRunHelper{ + initErr: test.initDataPathErr, + runCancelableDataPathErr: test.runCancelableDataPathErr, + runCancelableDataPathResult: test.runCancelableDataPathResult, + } + + if test.createDataPathFail { + funcCreateDataPathRestore = fakeCreateRestoreDataPathServiceWithErr + } else { + funcCreateDataPathRestore = fakeCreateRestoreDataPathService + } + + funcExitWithMessage = frHelper.ExitWithMessage + + s := &podVolumeRestore{ + logger: velerotest.NewLogger(), + cancelFunc: func() {}, + config: podVolumeRestoreConfig{ + pvrName: test.pvrName, + }, + } + + s.runDataPath() + + assert.Equal(t, test.expectedMessage, frHelper.exitMessage) + assert.Equal(t, test.expectedSucceed, frHelper.succeed) + }) + } +} + +func TestCreateRestoreDataPathService(t *testing.T) { + tests := []struct { + name string + fileStoreErr error + secretStoreErr error + mockGetInformer bool + getInformerErr error + expectedError string + }{ + { + name: "create credential file store error", + fileStoreErr: errors.New("fake-file-store-error"), + expectedError: "error to create credential file store: fake-file-store-error", + }, + { + name: "create credential secret store", + secretStoreErr: errors.New("fake-secret-store-error"), + expectedError: "error to create credential secret store: fake-secret-store-error", + }, + { + name: "get informer error", + mockGetInformer: true, + getInformerErr: errors.New("fake-get-informer-error"), + expectedError: "error to get controller-runtime informer from manager: fake-get-informer-error", + }, + { + name: "succeed", + mockGetInformer: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fcHelper := &fakeCreateDataPathServiceHelper{ + fileStoreErr: test.fileStoreErr, + secretStoreErr: test.secretStoreErr, + } + + funcNewCredentialFileStore = fcHelper.NewNamespacedFileStore + funcNewCredentialSecretStore = fcHelper.NewNamespacedSecretStore + + cache := cacheMock.NewCache(t) + if test.mockGetInformer { + cache.On("GetInformer", mock.Anything, mock.Anything).Return(nil, test.getInformerErr) + } + + funcExitWithMessage = frHelper.ExitWithMessage + + s := &podVolumeRestore{ + cache: cache, + } + + _, err := s.createDataPathService() + + if test.expectedError != "" { + assert.EqualError(t, err, test.expectedError) + } else { + assert.NoError(t, err) + } + }) + } +} diff --git a/pkg/podvolume/restore_micro_service.go b/pkg/podvolume/restore_micro_service.go index f7614bf7c..01f33a68a 100644 --- a/pkg/podvolume/restore_micro_service.go +++ b/pkg/podvolume/restore_micro_service.go @@ -304,6 +304,10 @@ func (r *RestoreMicroService) cancelPodVolumeRestore(pvr *velerov1api.PodVolumeR } } +var funcRemoveAll = os.RemoveAll +var funcMkdirAll = os.MkdirAll +var funcWriteFile = os.WriteFile + func writeCompletionMark(pvr *velerov1api.PodVolumeRestore, result datapath.RestoreResult, log logrus.FieldLogger) error { volumePath := result.Target.ByPath if volumePath == "" { @@ -314,7 +318,7 @@ func writeCompletionMark(pvr *velerov1api.PodVolumeRestore, result datapath.Rest // of this volume, which we don't want to carry over). If this fails for any reason, log and continue, since // this is non-essential cleanup (the done files are named based on restore UID and the init container looks // for the one specific to the restore being executed). - if err := os.RemoveAll(filepath.Join(volumePath, ".velero")); err != nil { + if err := funcRemoveAll(filepath.Join(volumePath, ".velero")); err != nil { log.WithError(err).Warnf("Failed to remove .velero directory from directory %s", volumePath) } @@ -326,14 +330,14 @@ func writeCompletionMark(pvr *velerov1api.PodVolumeRestore, result datapath.Rest // Create the .velero directory within the volume dir so we can write a done file // for this restore. - if err := os.MkdirAll(filepath.Join(volumePath, ".velero"), 0755); err != nil { + if err := funcMkdirAll(filepath.Join(volumePath, ".velero"), 0755); err != nil { return errors.Wrapf(err, "error creating .velero directory for done file") } // Write a done file with name= into the just-created .velero dir // within the volume. The velero init container on the pod is waiting // for this file to exist in each restored volume before completing. - if err := os.WriteFile(filepath.Join(volumePath, ".velero", string(restoreUID)), nil, 0644); err != nil { //nolint:gosec // Internal usage. No need to check. + if err := funcWriteFile(filepath.Join(volumePath, ".velero", string(restoreUID)), nil, 0644); err != nil { return errors.Wrapf(err, "error writing done file") } diff --git a/pkg/podvolume/restore_micro_service_test.go b/pkg/podvolume/restore_micro_service_test.go index 0968bfeec..dabbfb83f 100644 --- a/pkg/podvolume/restore_micro_service_test.go +++ b/pkg/podvolume/restore_micro_service_test.go @@ -19,6 +19,7 @@ package podvolume import ( "context" "fmt" + "os" "sync" "testing" "time" @@ -42,6 +43,8 @@ import ( kbclient "sigs.k8s.io/controller-runtime/pkg/client" datapathmockes "github.com/vmware-tanzu/velero/pkg/datapath/mocks" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) type restoreMsTestHelper struct { @@ -468,3 +471,152 @@ func TestRunCancelableDataPathRestore(t *testing.T) { cancel() } + +func TestWriteCompletionMark(t *testing.T) { + tests := []struct { + name string + pvr *velerov1api.PodVolumeRestore + result datapath.RestoreResult + funcRemoveAll func(string) error + funcMkdirAll func(string, os.FileMode) error + funcWriteFile func(string, []byte, os.FileMode) error + expectedErr string + expectedLog string + }{ + { + name: "no volume path", + result: datapath.RestoreResult{}, + expectedErr: "target volume is empty in restore result", + }, + { + name: "no owner reference", + result: datapath.RestoreResult{ + Target: datapath.AccessPoint{ + ByPath: "fake-volume-path", + }, + }, + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, "fake-pvr").Result(), + funcRemoveAll: func(string) error { + return nil + }, + expectedErr: "error finding restore UID", + }, + { + name: "mkdir fail", + result: datapath.RestoreResult{ + Target: datapath.AccessPoint{ + ByPath: "fake-volume-path", + }, + }, + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, "fake-pvr").OwnerReference([]metav1.OwnerReference{ + { + UID: "fake-uid", + }, + }).Result(), + funcRemoveAll: func(string) error { + return nil + }, + funcMkdirAll: func(string, os.FileMode) error { + return errors.New("fake-mk-dir-error") + }, + expectedErr: "error creating .velero directory for done file: fake-mk-dir-error", + }, + { + name: "write file fail", + result: datapath.RestoreResult{ + Target: datapath.AccessPoint{ + ByPath: "fake-volume-path", + }, + }, + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, "fake-pvr").OwnerReference([]metav1.OwnerReference{ + { + UID: "fake-uid", + }, + }).Result(), + funcRemoveAll: func(string) error { + return nil + }, + funcMkdirAll: func(string, os.FileMode) error { + return nil + }, + funcWriteFile: func(string, []byte, os.FileMode) error { + return errors.New("fake-write-file-error") + }, + expectedErr: "error writing done file: fake-write-file-error", + }, + { + name: "succeed", + result: datapath.RestoreResult{ + Target: datapath.AccessPoint{ + ByPath: "fake-volume-path", + }, + }, + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, "fake-pvr").OwnerReference([]metav1.OwnerReference{ + { + UID: "fake-uid", + }, + }).Result(), + funcRemoveAll: func(string) error { + return nil + }, + funcMkdirAll: func(string, os.FileMode) error { + return nil + }, + funcWriteFile: func(string, []byte, os.FileMode) error { + return nil + }, + }, + { + name: "succeed but previous dir is not removed", + result: datapath.RestoreResult{ + Target: datapath.AccessPoint{ + ByPath: "fake-volume-path", + }, + }, + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, "fake-pvr").OwnerReference([]metav1.OwnerReference{ + { + UID: "fake-uid", + }, + }).Result(), + funcRemoveAll: func(string) error { + return errors.New("fake-remove-dir-error") + }, + funcMkdirAll: func(string, os.FileMode) error { + return nil + }, + funcWriteFile: func(string, []byte, os.FileMode) error { + return nil + }, + expectedLog: "Failed to remove .velero directory from directory fake-volume-path", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + if test.funcRemoveAll != nil { + funcRemoveAll = test.funcRemoveAll + } + + if test.funcMkdirAll != nil { + funcMkdirAll = test.funcMkdirAll + } + + if test.funcWriteFile != nil { + funcWriteFile = test.funcWriteFile + } + + logBuffer := "" + err := writeCompletionMark(test.pvr, test.result, velerotest.NewSingleLogger(&logBuffer)) + + if test.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, test.expectedErr) + } + + if test.expectedLog != "" { + assert.Contains(t, logBuffer, test.expectedLog) + } + }) + } +} From fec271180d202b5ed5c06c5a2aef04191b667e28 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Tue, 10 Jun 2025 18:01:43 +0800 Subject: [PATCH 25/31] vgdp ms pvr controller Signed-off-by: Lyndon-Li --- changelogs/unreleased/9014-Lyndon-Li | 1 + pkg/builder/pod_volume_restore_builder.go | 30 +++++++++---------- pkg/controller/data_download_controller.go | 5 ++-- pkg/controller/data_upload_controller.go | 5 ++-- .../pod_volume_restore_controller.go | 16 +++++----- .../pod_volume_restore_controller_test.go | 7 +++-- 6 files changed, 34 insertions(+), 30 deletions(-) create mode 100644 changelogs/unreleased/9014-Lyndon-Li diff --git a/changelogs/unreleased/9014-Lyndon-Li b/changelogs/unreleased/9014-Lyndon-Li new file mode 100644 index 000000000..b36740f9d --- /dev/null +++ b/changelogs/unreleased/9014-Lyndon-Li @@ -0,0 +1 @@ +Fix issue #8959, add VGDP MS PVR controller \ No newline at end of file diff --git a/pkg/builder/pod_volume_restore_builder.go b/pkg/builder/pod_volume_restore_builder.go index 965dd5a5d..124f116fe 100644 --- a/pkg/builder/pod_volume_restore_builder.go +++ b/pkg/builder/pod_volume_restore_builder.go @@ -99,31 +99,31 @@ func (b *PodVolumeRestoreBuilder) UploaderType(uploaderType string) *PodVolumeRe } // Cancel sets the DataDownload's Cancel. -func (d *PodVolumeRestoreBuilder) Cancel(cancel bool) *PodVolumeRestoreBuilder { - d.object.Spec.Cancel = cancel - return d +func (b *PodVolumeRestoreBuilder) Cancel(cancel bool) *PodVolumeRestoreBuilder { + b.object.Spec.Cancel = cancel + return b } // AcceptedTimestamp sets the PodVolumeRestore's AcceptedTimestamp. -func (d *PodVolumeRestoreBuilder) AcceptedTimestamp(acceptedTimestamp *metav1.Time) *PodVolumeRestoreBuilder { - d.object.Status.AcceptedTimestamp = acceptedTimestamp - return d +func (b *PodVolumeRestoreBuilder) AcceptedTimestamp(acceptedTimestamp *metav1.Time) *PodVolumeRestoreBuilder { + b.object.Status.AcceptedTimestamp = acceptedTimestamp + return b } // Finalizers sets the PodVolumeRestore's Finalizers. -func (d *PodVolumeRestoreBuilder) Finalizers(finalizers []string) *PodVolumeRestoreBuilder { - d.object.Finalizers = finalizers - return d +func (b *PodVolumeRestoreBuilder) Finalizers(finalizers []string) *PodVolumeRestoreBuilder { + b.object.Finalizers = finalizers + return b } // Message sets the PodVolumeRestore's Message. -func (d *PodVolumeRestoreBuilder) Message(msg string) *PodVolumeRestoreBuilder { - d.object.Status.Message = msg - return d +func (b *PodVolumeRestoreBuilder) Message(msg string) *PodVolumeRestoreBuilder { + b.object.Status.Message = msg + return b } // Message sets the PodVolumeRestore's Node. -func (d *PodVolumeRestoreBuilder) Node(node string) *PodVolumeRestoreBuilder { - d.object.Status.Node = node - return d +func (b *PodVolumeRestoreBuilder) Node(node string) *PodVolumeRestoreBuilder { + b.object.Status.Node = node + return b } diff --git a/pkg/controller/data_download_controller.go b/pkg/controller/data_download_controller.go index 75c04e377..53d83d98e 100644 --- a/pkg/controller/data_download_controller.go +++ b/pkg/controller/data_download_controller.go @@ -198,7 +198,9 @@ func (r *DataDownloadReconciler) Reconcile(ctx context.Context, req ctrl.Request if time.Since(spotted) > delay { log.Infof("Data download %s is canceled in Phase %s but not handled in rasonable time", dd.GetName(), dd.Status.Phase) - r.tryCancelDataDownload(ctx, dd, "") + if r.tryCancelDataDownload(ctx, dd, "") { + delete(r.cancelledDataDownload, dd.Name) + } return ctrl.Result{}, nil } @@ -524,7 +526,6 @@ func (r *DataDownloadReconciler) tryCancelDataDownload(ctx context.Context, dd * // success update r.metrics.RegisterDataDownloadCancel(r.nodeName) r.restoreExposer.CleanUp(ctx, getDataDownloadOwnerObject(dd)) - delete(r.cancelledDataDownload, dd.Name) log.Warn("data download is canceled") diff --git a/pkg/controller/data_upload_controller.go b/pkg/controller/data_upload_controller.go index 7dc19e855..6ab65f172 100644 --- a/pkg/controller/data_upload_controller.go +++ b/pkg/controller/data_upload_controller.go @@ -226,7 +226,9 @@ func (r *DataUploadReconciler) Reconcile(ctx context.Context, req ctrl.Request) if time.Since(spotted) > delay { log.Infof("Data upload %s is canceled in Phase %s but not handled in reasonable time", du.GetName(), du.Status.Phase) - r.tryCancelDataUpload(ctx, du, "") + if r.tryCancelDataUpload(ctx, du, "") { + delete(r.cancelledDataUpload, du.Name) + } return ctrl.Result{}, nil } @@ -564,7 +566,6 @@ func (r *DataUploadReconciler) tryCancelDataUpload(ctx context.Context, du *vele r.metrics.RegisterDataUploadCancel(r.nodeName) // cleans up any objects generated during the snapshot expose r.cleanUp(ctx, du, log) - delete(r.cancelledDataUpload, du.Name) log.Warn("data upload is canceled") diff --git a/pkg/controller/pod_volume_restore_controller.go b/pkg/controller/pod_volume_restore_controller.go index 46a00937d..41362871a 100644 --- a/pkg/controller/pod_volume_restore_controller.go +++ b/pkg/controller/pod_volume_restore_controller.go @@ -43,7 +43,6 @@ import ( "sigs.k8s.io/controller-runtime/pkg/reconcile" veleroapishared "github.com/vmware-tanzu/velero/pkg/apis/velero/shared" - velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" "github.com/vmware-tanzu/velero/pkg/constant" "github.com/vmware-tanzu/velero/pkg/datapath" @@ -186,7 +185,9 @@ func (r *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Req if time.Since(spotted) > delay { log.Infof("PVR %s is canceled in Phase %s but not handled in rasonable time", pvr.GetName(), pvr.Status.Phase) - r.tryCancelPodVolumeRestore(ctx, pvr, "") + if r.tryCancelPodVolumeRestore(ctx, pvr, "") { + delete(r.cancelledPVR, pvr.Name) + } return ctrl.Result{}, nil } @@ -196,7 +197,7 @@ func (r *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Req if pvr.Status.Phase == "" || pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseNew { if pvr.Spec.Cancel { log.Infof("PVR %s is canceled in Phase %s", pvr.GetName(), pvr.Status.Phase) - r.tryCancelPodVolumeRestore(ctx, pvr, "") + _ = r.tryCancelPodVolumeRestore(ctx, pvr, "") return ctrl.Result{}, nil } @@ -234,7 +235,7 @@ func (r *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Req } else if pvr.Status.Phase == velerov1api.PodVolumeRestorePhaseAccepted { if peekErr := r.exposer.PeekExposed(ctx, getPVROwnerObject(pvr)); peekErr != nil { log.Errorf("Cancel PVR %s/%s because of expose error %s", pvr.Namespace, pvr.Name, peekErr) - r.tryCancelPodVolumeRestore(ctx, pvr, fmt.Sprintf("found a PVR %s/%s with expose error: %s. mark it as cancel", pvr.Namespace, pvr.Name, peekErr)) + _ = r.tryCancelPodVolumeRestore(ctx, pvr, fmt.Sprintf("found a PVR %s/%s with expose error: %s. mark it as cancel", pvr.Namespace, pvr.Name, peekErr)) } else if pvr.Status.AcceptedTimestamp != nil { if time.Since(pvr.Status.AcceptedTimestamp.Time) >= r.preparingTimeout { r.onPrepareTimeout(ctx, pvr) @@ -250,7 +251,7 @@ func (r *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Req } if pvr.Spec.Cancel { - log.Info("Prepared PVR is being cancelled") + log.Info("Prepared PVR is being canceled") r.OnDataPathCancelled(ctx, pvr.GetNamespace(), pvr.GetName()) return ctrl.Result{}, nil } @@ -346,7 +347,7 @@ func (r *PodVolumeRestoreReconciler) Reconcile(ctx context.Context, req ctrl.Req // Update status to Canceling if err := UpdatePVRWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvr.Namespace, Name: pvr.Name}, log, func(pvr *velerov1api.PodVolumeRestore) bool { if isPVRInFinalState(pvr) { - log.Warnf("PVR %s is terminated, abort setting it to cancelling", pvr.Name) + log.Warnf("PVR %s is terminated, abort setting it to canceling", pvr.Name) return false } @@ -399,7 +400,6 @@ func (r *PodVolumeRestoreReconciler) tryCancelPodVolumeRestore(ctx context.Conte } r.exposer.CleanUp(ctx, getPVROwnerObject(pvr)) - delete(r.cancelledPVR, pvr.Name) log.Warn("PVR is canceled") @@ -569,7 +569,7 @@ func (r *PodVolumeRestoreReconciler) SetupWithManager(mgr ctrl.Manager) error { }) pred := kube.NewAllEventPredicate(func(obj client.Object) bool { - pvr := obj.(*velerov1.PodVolumeRestore) + pvr := obj.(*velerov1api.PodVolumeRestore) return !isLegacyPVR(pvr) }) diff --git a/pkg/controller/pod_volume_restore_controller_test.go b/pkg/controller/pod_volume_restore_controller_test.go index 5d4a3522e..81800b805 100644 --- a/pkg/controller/pod_volume_restore_controller_test.go +++ b/pkg/controller/pod_volume_restore_controller_test.go @@ -774,9 +774,10 @@ func TestPodVolumeRestoreReconcile(t *testing.T) { expectedErr: "error accepting PVR pvr-1: error updating PVR velero/pvr-1: Update error", }, { - name: "pvr is cancel on accepted", - pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Result(), - expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Result(), + name: "pvr is cancel on accepted", + pvr: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Result(), + expectCancelRecord: true, + expected: builder.ForPodVolumeRestore(velerov1api.DefaultNamespace, pvrName).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeRestorePhaseCanceled).Result(), }, { name: "pvr expose failed", From 97a4d62d3c09ec10082945301a50e3807eeb932c Mon Sep 17 00:00:00 2001 From: Shubham Pampattiwar Date: Fri, 16 May 2025 12:11:34 -0700 Subject: [PATCH 26/31] Extend PVCAction itemblock plugin to support grouping PVCs under VolumeGroupSnapshot label Signed-off-by: Shubham Pampattiwar Add changelog file Signed-off-by: Shubham Pampattiwar Update VGS label key and address PR feedback Signed-off-by: Shubham Pampattiwar update log level to debug for edge cases Signed-off-by: Shubham Pampattiwar Change VGS label key constant location Signed-off-by: Shubham Pampattiwar run make update Signed-off-by: Shubham Pampattiwar --- .../unreleased/8944-shubham-pampattiwar | 1 + pkg/apis/velero/v1/labels_annotations.go | 3 + pkg/cmd/server/config/config.go | 9 +- pkg/controller/backup_controller_test.go | 6 +- pkg/itemblock/actions/pvc_action.go | 60 +++++++++- pkg/itemblock/actions/pvc_action_test.go | 109 ++++++++++++++++++ 6 files changed, 178 insertions(+), 10 deletions(-) create mode 100644 changelogs/unreleased/8944-shubham-pampattiwar diff --git a/changelogs/unreleased/8944-shubham-pampattiwar b/changelogs/unreleased/8944-shubham-pampattiwar new file mode 100644 index 000000000..f59ba9844 --- /dev/null +++ b/changelogs/unreleased/8944-shubham-pampattiwar @@ -0,0 +1 @@ +Extend PVCAction itemblock plugin to support grouping PVCs under VGS label key \ No newline at end of file diff --git a/pkg/apis/velero/v1/labels_annotations.go b/pkg/apis/velero/v1/labels_annotations.go index 78d231ba9..ca3f91b6d 100644 --- a/pkg/apis/velero/v1/labels_annotations.go +++ b/pkg/apis/velero/v1/labels_annotations.go @@ -101,6 +101,9 @@ const ( // ExcludeFromBackupLabel is the label to exclude k8s resource from backup, // even if the resource contains a matching selector label. ExcludeFromBackupLabel = "velero.io/exclude-from-backup" + + // defaultVGSLabelKey is the default label key used to group PVCs under a VolumeGroupSnapshot + DefaultVGSLabelKey = "velero.io/volume-group" ) type AsyncOperationIDPrefix string diff --git a/pkg/cmd/server/config/config.go b/pkg/cmd/server/config/config.go index b80a5c458..b9540bf2c 100644 --- a/pkg/cmd/server/config/config.go +++ b/pkg/cmd/server/config/config.go @@ -5,6 +5,8 @@ import ( "strings" "time" + velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/sirupsen/logrus" "github.com/spf13/pflag" @@ -37,9 +39,6 @@ const ( // the default TTL for a backup defaultBackupTTL = 30 * 24 * time.Hour - // defaultVGSLabelKey is the default label key used to group PVCs under a VolumeGroupSnapshot - defaultVGSLabelKey = "velero.io/volume-group-snapshot" - defaultCSISnapshotTimeout = 10 * time.Minute defaultItemOperationTimeout = 4 * time.Hour @@ -193,7 +192,7 @@ func GetDefaultConfig() *Config { DefaultVolumeSnapshotLocations: flag.NewMap().WithKeyValueDelimiter(':'), BackupSyncPeriod: defaultBackupSyncPeriod, DefaultBackupTTL: defaultBackupTTL, - DefaultVGSLabelKey: defaultVGSLabelKey, + DefaultVGSLabelKey: velerov1api.DefaultVGSLabelKey, DefaultCSISnapshotTimeout: defaultCSISnapshotTimeout, DefaultItemOperationTimeout: defaultItemOperationTimeout, ResourceTimeout: resourceTimeout, @@ -245,7 +244,7 @@ func (c *Config) BindFlags(flags *pflag.FlagSet) { flags.StringVar(&c.ProfilerAddress, "profiler-address", c.ProfilerAddress, "The address to expose the pprof profiler.") flags.DurationVar(&c.ResourceTerminatingTimeout, "terminating-resource-timeout", c.ResourceTerminatingTimeout, "How long to wait on persistent volumes and namespaces to terminate during a restore before timing out.") flags.DurationVar(&c.DefaultBackupTTL, "default-backup-ttl", c.DefaultBackupTTL, "How long to wait by default before backups can be garbage collected.") - flags.StringVar(&c.DefaultVGSLabelKey, "volume-group-snapshot-label-key", c.DefaultVGSLabelKey, "Label key for grouping PVCs into VolumeGroupSnapshot. Default value is 'velero.io/volume-group-snapshot'") + flags.StringVar(&c.DefaultVGSLabelKey, "volume-group-snapshot-label-key", c.DefaultVGSLabelKey, "Label key for grouping PVCs into VolumeGroupSnapshot. Default value is 'velero.io/volume-group'") flags.DurationVar(&c.RepoMaintenanceFrequency, "default-repo-maintain-frequency", c.RepoMaintenanceFrequency, "How often 'maintain' is run for backup repositories by default.") flags.DurationVar(&c.GarbageCollectionFrequency, "garbage-collection-frequency", c.GarbageCollectionFrequency, "How often garbage collection is run for expired backups.") flags.DurationVar(&c.ItemOperationSyncFrequency, "item-operation-sync-frequency", c.ItemOperationSyncFrequency, "How often to check status on backup/restore operations after backup/restore processing. Default is 10 seconds") diff --git a/pkg/controller/backup_controller_test.go b/pkg/controller/backup_controller_test.go index e8a28282d..8136010b3 100644 --- a/pkg/controller/backup_controller_test.go +++ b/pkg/controller/backup_controller_test.go @@ -490,8 +490,6 @@ func TestPrepareBackupRequest_SetsVGSLabelKey(t *testing.T) { require.NoError(t, err) now = now.Local() - defaultVGSLabelKey := "velero.io/volume-group-snapshot" - tests := []struct { name string backup *velerov1api.Backup @@ -515,8 +513,8 @@ func TestPrepareBackupRequest_SetsVGSLabelKey(t *testing.T) { { name: "backup with no spec or server flag, uses default", backup: builder.ForBackup("velero", "backup-3").Result(), - serverFlagKey: defaultVGSLabelKey, - expectedLabelKey: defaultVGSLabelKey, + serverFlagKey: velerov1api.DefaultVGSLabelKey, + expectedLabelKey: velerov1api.DefaultVGSLabelKey, }, } diff --git a/pkg/itemblock/actions/pvc_action.go b/pkg/itemblock/actions/pvc_action.go index eee4e2fb3..b5d7074af 100644 --- a/pkg/itemblock/actions/pvc_action.go +++ b/pkg/itemblock/actions/pvc_action.go @@ -105,9 +105,67 @@ func (a *PVCAction) GetRelatedItems(item runtime.Unstructured, backup *v1.Backup } } + // Gather groupedPVCs based on VGS label provided in the backup + groupedPVCs, err := a.getGroupedPVCs(context.Background(), pvc, backup) + if err != nil { + return nil, err + } + + // Add the groupedPVCs to relatedItems so that they processed in a single item block + relatedItems = append(relatedItems, groupedPVCs...) + return relatedItems, nil } func (a *PVCAction) Name() string { - return "PodItemBlockAction" + return "PVCItemBlockAction" +} + +// getGroupedPVCs returns other PVCs in the same group based on the VGS label key in the Backup spec. +func (a *PVCAction) getGroupedPVCs(ctx context.Context, pvc *corev1api.PersistentVolumeClaim, backup *v1.Backup) ([]velero.ResourceIdentifier, error) { + var related []velero.ResourceIdentifier + + vgsLabelKey := backup.Spec.VolumeGroupSnapshotLabelKey + if vgsLabelKey == "" { + a.log.Debug("No VolumeGroupSnapshotLabelKey provided in backup spec; skipping PVC grouping") + return nil, nil + } + + groupID, ok := pvc.Labels[vgsLabelKey] + if !ok || groupID == "" { + // PVC does not belong to any VGS group or groupID has empty value + a.log.Debug("PVC does not belong to any PVC group or group label value is empty; skipping PVC grouping") + return nil, nil + } + + pvcList := new(corev1api.PersistentVolumeClaimList) + if err := a.crClient.List( + ctx, + pvcList, + crclient.InNamespace(pvc.Namespace), + crclient.MatchingLabels{vgsLabelKey: groupID}, + ); err != nil { + return nil, errors.Wrapf(err, "failed to list PVCs for VGS grouping with label %s=%s in namespace %s", vgsLabelKey, groupID, pvc.Namespace) + } + + if len(pvcList.Items) <= 1 { + // Only the current PVC exists in this group + return nil, nil + } + + for _, groupPVC := range pvcList.Items { + if groupPVC.Name == pvc.Name { + continue + } + + a.log.Infof("Adding grouped PVC %s (group %s) to relatedItems for PVC %s", groupPVC.Name, groupID, pvc.Name) + + related = append(related, velero.ResourceIdentifier{ + GroupResource: kuberesource.PersistentVolumeClaims, + Namespace: groupPVC.Namespace, + Name: groupPVC.Name, + }) + } + + return related, nil } diff --git a/pkg/itemblock/actions/pvc_action_test.go b/pkg/itemblock/actions/pvc_action_test.go index c485dcd80..fcd54b022 100644 --- a/pkg/itemblock/actions/pvc_action_test.go +++ b/pkg/itemblock/actions/pvc_action_test.go @@ -124,6 +124,22 @@ func TestBackupPVAction(t *testing.T) { {GroupResource: kuberesource.Pods, Namespace: "velero", Name: "testPod2"}, }, }, + { + name: "Test with PVC grouping via VGS label", + pvc: builder.ForPersistentVolumeClaim("velero", "testPVC-1").ObjectMeta(builder.WithLabels("velero.io/group", "db")).VolumeName("testPV-1").Phase(corev1api.ClaimBound).Result(), + pods: []*corev1api.Pod{ + builder.ForPod("velero", "testPod-1"). + Volumes(builder.ForVolume("testPV-1").PersistentVolumeClaimSource("testPVC-1").Result()). + NodeName("node"). + Phase(corev1api.PodRunning).Result(), + }, + expectedErr: nil, + expectedRelated: []velero.ResourceIdentifier{ + {GroupResource: kuberesource.PersistentVolumes, Name: "testPV-1"}, + {GroupResource: kuberesource.Pods, Namespace: "velero", Name: "testPod-1"}, + {GroupResource: kuberesource.PersistentVolumeClaims, Namespace: "velero", Name: "groupedPVC"}, + }, + }, } backup := &v1.Backup{} @@ -152,6 +168,12 @@ func TestBackupPVAction(t *testing.T) { require.NoError(t, crClient.Create(context.Background(), pod)) } + if tc.name == "Test with PVC grouping via VGS label" { + groupedPVC := builder.ForPersistentVolumeClaim("velero", "groupedPVC").ObjectMeta(builder.WithLabels("velero.io/group", "db")).VolumeName("groupedPV").Phase(corev1api.ClaimBound).Result() + require.NoError(t, crClient.Create(context.Background(), groupedPVC)) + backup.Spec.VolumeGroupSnapshotLabelKey = "velero.io/group" + } + pvcMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&tc.pvc) require.NoError(t, err) @@ -165,3 +187,90 @@ func TestBackupPVAction(t *testing.T) { }) } } + +// Test_getGroupedPVCs verifies the PVC grouping logic for VolumeGroupSnapshots. +// This ensures only same-namespace PVCs with the same label key and value are included. +func Test_getGroupedPVCs(t *testing.T) { + tests := []struct { + name string + labelKey string + groupValue string + existingPVCs []*corev1api.PersistentVolumeClaim + targetPVC *corev1api.PersistentVolumeClaim + expectedRelated []velero.ResourceIdentifier + expectError bool + }{ + { + name: "No label key in spec", + labelKey: "", + targetPVC: builder.ForPersistentVolumeClaim("ns", "pvc-1").Result(), + expectError: false, + }, + { + name: "No group value", + labelKey: "velero.io/group", + groupValue: "", + targetPVC: builder.ForPersistentVolumeClaim("ns", "pvc-1").Result(), + expectError: false, + }, + { + name: "Target PVC does not have the label", + labelKey: "velero.io/group", + targetPVC: builder.ForPersistentVolumeClaim("ns", "pvc-1").Result(), + expectError: false, + }, + { + name: "Target PVC has label, but no group matches", + labelKey: "velero.io/group", + groupValue: "group-1", + targetPVC: builder.ForPersistentVolumeClaim("ns", "pvc-1").ObjectMeta(builder.WithLabels("velero.io/group", "group-1")).Result(), + existingPVCs: []*corev1api.PersistentVolumeClaim{ + builder.ForPersistentVolumeClaim("ns", "pvc-1").ObjectMeta(builder.WithLabels("velero.io/group", "group-1")).Result(), + }, + expectError: false, + expectedRelated: nil, + }, + { + name: "Multiple PVCs in the same group", + labelKey: "velero.io/group", + groupValue: "group-1", + targetPVC: builder.ForPersistentVolumeClaim("ns", "pvc-1").ObjectMeta(builder.WithLabels("velero.io/group", "group-1")).Result(), + existingPVCs: []*corev1api.PersistentVolumeClaim{ + builder.ForPersistentVolumeClaim("ns", "pvc-1").ObjectMeta(builder.WithLabels("velero.io/group", "group-1")).Result(), + builder.ForPersistentVolumeClaim("ns", "pvc-2").ObjectMeta(builder.WithLabels("velero.io/group", "group-1")).Result(), + builder.ForPersistentVolumeClaim("ns", "pvc-3").ObjectMeta(builder.WithLabels("velero.io/group", "group-1")).Result(), + }, + expectError: false, + expectedRelated: []velero.ResourceIdentifier{ + {GroupResource: kuberesource.PersistentVolumeClaims, Namespace: "ns", Name: "pvc-2"}, + {GroupResource: kuberesource.PersistentVolumeClaims, Namespace: "ns", Name: "pvc-3"}, + }, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + crClient := velerotest.NewFakeControllerRuntimeClient(t) + for _, pvc := range tc.existingPVCs { + require.NoError(t, crClient.Create(context.Background(), pvc)) + } + + logger := logrus.New() + a := &PVCAction{ + log: logger, + crClient: crClient, + } + + backup := builder.ForBackup("ns", "bkp").VolumeGroupSnapshotLabelKey(tc.labelKey).Result() + + related, err := a.getGroupedPVCs(context.Background(), tc.targetPVC, backup) + if tc.expectError { + require.Error(t, err) + } else { + require.NoError(t, err) + } + + assert.ElementsMatch(t, tc.expectedRelated, related) + }) + } +} From 99c699fcb1790fdc60af317f98a186db8ba13bf2 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Wed, 11 Jun 2025 11:25:37 +0800 Subject: [PATCH 27/31] vgdp ms pvb controller Signed-off-by: Lyndon-Li --- .../v1/bases/velero.io_podvolumebackups.yaml | 54 +- config/crd/v1/crds/crds.go | 2 +- pkg/apis/velero/v1/labels_annotations.go | 3 + pkg/apis/velero/v1/pod_volume_backup_types.go | 27 +- pkg/apis/velero/v1/zz_generated.deepcopy.go | 4 + pkg/builder/pod_volume_backup_builder.go | 30 + pkg/cmd/cli/nodeagent/server.go | 12 +- pkg/controller/data_upload_controller_test.go | 16 +- .../pod_volume_backup_controller.go | 909 +++++++++---- .../pod_volume_backup_controller_test.go | 1157 ++++++++++++----- pkg/podvolume/backupper.go | 8 +- 11 files changed, 1637 insertions(+), 585 deletions(-) diff --git a/config/crd/v1/bases/velero.io_podvolumebackups.yaml b/config/crd/v1/bases/velero.io_podvolumebackups.yaml index 0eadd8e59..f77c5df4a 100644 --- a/config/crd/v1/bases/velero.io_podvolumebackups.yaml +++ b/config/crd/v1/bases/velero.io_podvolumebackups.yaml @@ -15,38 +15,41 @@ spec: scope: Namespaced versions: - additionalPrinterColumns: - - description: Pod Volume Backup status such as New/InProgress + - description: PodVolumeBackup status such as New/InProgress jsonPath: .status.phase name: Status type: string - - description: Time when this backup was started + - description: Time duration since this PodVolumeBackup was started jsonPath: .status.startTimestamp - name: Created + name: Started type: date - - description: Namespace of the pod containing the volume to be backed up - jsonPath: .spec.pod.namespace - name: Namespace - type: string - - description: Name of the pod containing the volume to be backed up - jsonPath: .spec.pod.name - name: Pod - type: string - - description: Name of the volume to be backed up - jsonPath: .spec.volume - name: Volume - type: string - - description: The type of the uploader to handle data transfer - jsonPath: .spec.uploaderType - name: Uploader Type - type: string + - description: Completed bytes + format: int64 + jsonPath: .status.progress.bytesDone + name: Bytes Done + type: integer + - description: Total bytes + format: int64 + jsonPath: .status.progress.totalBytes + name: Total Bytes + type: integer - description: Name of the Backup Storage Location where this backup should be stored jsonPath: .spec.backupStorageLocation name: Storage Location type: string - - jsonPath: .metadata.creationTimestamp + - description: Time duration since this PodVolumeBackup was created + jsonPath: .metadata.creationTimestamp name: Age type: date + - description: Name of the node where the PodVolumeBackup is processed + jsonPath: .status.node + name: Node + type: string + - description: The type of the uploader to handle data transfer + jsonPath: .spec.uploaderType + name: Uploader + type: string name: v1 schema: openAPIV3Schema: @@ -170,6 +173,13 @@ spec: status: description: PodVolumeBackupStatus is the current status of a PodVolumeBackup. properties: + acceptedTimestamp: + description: |- + AcceptedTimestamp records the time the pod volume backup is to be prepared. + The server's time is used for AcceptedTimestamp + format: date-time + nullable: true + type: string completionTimestamp: description: |- CompletionTimestamp records the time a backup was completed. @@ -190,7 +200,11 @@ spec: description: Phase is the current state of the PodVolumeBackup. enum: - New + - Accepted + - Prepared - InProgress + - Canceling + - Canceled - Completed - Failed type: string diff --git a/config/crd/v1/crds/crds.go b/config/crd/v1/crds/crds.go index a412073f6..986184f6f 100644 --- a/config/crd/v1/crds/crds.go +++ b/config/crd/v1/crds/crds.go @@ -34,7 +34,7 @@ var rawCRDs = [][]byte{ []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xccYK\x8f\x1b\xb9\x11\xbe\xebW\x14v\x0f{ٖ\xec\x04\t\x02\xdd\xc6r\x02\x18\x19\xc7\x03k2\xb9.EVK\\\xb1\xc9\x0e\x1f\x92\x95\xc7\x7f\x0f\x8a\x0f\xa9\xd5\x0fK\xe3\x04\x9b\xe5eF|\x14\xeb\xf9U\x15\xbb\xaa\xaa\x19k\xe5\vZ'\x8d^\x02k%~\xf1\xa8闛\xef\xff\xe0\xe6\xd2,\x0eog{\xa9\xc5\x12V\xc1y\xd3|Fg\x82\xe5\xf8\x1ek\xa9\xa5\x97F\xcf\x1a\xf4L0ϖ3\x00\xa6\xb5\xf1\x8c\xa6\x1d\xfd\x04\xe0F{k\x94B[mQ\xcf\xf7a\x83\x9b \x95@\x1b\x89\x97\xab\x0fo\xe6o\x7f?\xff\xdd\f@\xb3\x06\x97\xb0a|\x1fZ\xe7\x8de[T\x86'\x92\xf3\x03*\xb4f.\xcd̵\xc8醭5\xa1]\xc2e!Qȷ'\xce\xdfEb\xebD\xec1\x13\x8b\xebJ:\xff\xe7\xe9=\x8f\xd2\xf9\xb8\xafU\xc125\xc5V\xdc\xe2v\xc6\xfa\xbf\\\xae\xae`\xe3TZ\x91z\x1b\x14\xb3\x13\xc7g\x00\x8e\x9b\x16\x97\x10O\xb7\x8c\xa3\x98\x01d\xd5Dj\x150!\xa2\xb2\x99z\xb2R{\xb4+\xa3B\xa3\xcfw\tt\xdc\xca\xd6Ge&Y \v\x03E\x1ap\x9e\xf9\xe0\xc0\x05\xbe\x03\xe6\xe0\xe1\xc0\xa4b\x1b\x85\x8b\xbfjV\xfe\x8f\xf4\x00~vF?1\xbf[\xc2<\x9d\x9a\xb7;\xe6\xcaj\xb2\xd1SgƟH\x00\xe7\xad\xd4\xdb1\x96\x1e\x99\xf3/LI\x119y\x96\r\x82t\xe0w\b\x8a9\x0f\x9e&\xe8W\xd2\x10\x90\x8a\x10\x8a\x86\xe0\xc8\\\xbe\a\xe0\x90\xa8D\x1d\x8ds\xaa\x06w]\xb1M\xac\xc0K\x8fJ\xe2\x9ff2\xf7\x1d\xb2ſ\xe7\xdc♤\xf3\xaci\xaf\xe8>lq\x8aؕ*\xdec͂\xf2]Q\xc9J\xaa\xeb\x97\xd7b\xb5\xc8\xe7\"\x9d\xba\xba\xf1\xfd\xd5\\\xbauc\x8cB\x96\xa8\xa4]\x87\xb7\xc9\v\xf9\x0e\x1b\xb6̛M\x8b\xfa\xe1\xe9\xc3\xcbo\xd7W\xd30\xe6H\xbd\xa0 ñ\x8emvh\x11^b\xfc%\xbb\xb9,ڙ&\x80\xd9\xfc\x8c\xdc_\x8c\xd8ZӢ\xf5\xb2\x04K\x1a\x1d,\xea\xcc\xf6x\xfaWu\xb5\x06@b\xa4S \b\x940\xf9U\x8e\x1f\x14Yr05\xf8\x9dt`\xb1\xb5\xe8P'\x98\xa2i\xa63\x83\xf3\x1e\xe95Z\"C\xb1\x1d\x94 ,;\xa0\xf5`\x91\x9b\xad\x96\xff8\xd3v\xe0Mvf\x8f\xceC\x8cP\xcd\x149k\xc0\x1f\x81iѣܰ\x13X\xa4;!\xe8\x0e\xbdx\xc0\xf5\xf9\xf8H\xd1 um\x96\xb0\xf3\xbeu\xcb\xc5b+}Ahn\x9a&h\xe9O\x8b\b\xb6r\x13\xbc\xb1n!\xf0\x80j\xe1\xe4\xb6b\x96\xef\xa4G\xee\x83\xc5\x05ke\x15\x05\xd1\tR\x1b\xf1\xbd͘\uebae\x1d\x84t\x1a\x11R_a\x1e\x82\xd7\xe42\x89T\x12\xf1b\x05\x9a\"\xd5}\xfe\xe3\xfa\x19\n'\xc9R\xc9(\x97\xad\x03\xbd\x14\xfb\x906\xa5\xaeѦs\xb55M\xa4\x89Z\xb4Fj\x1f\x7fp%Q{pa\xd3HOn\xf0\xf7\x80Γ\xe9\xfadW1\x8b\xc1\x06!\xb4\x11$\xfa\x1b>hX\xb1\x06Պ9\xfc\x85mEVq\x15\x19\xe1.kuss\x7fsRog\xa1\xe4\xd4\tӎ\xa2\xc1\xbaE~\x15w\x02\x9d\xb4\x14\x19\x9ey\x8c\xd1\xd5SP\x86\x8a\xe9\xa4\\\xc68H\xd0`\x9c\xa3s\x1f\x8d\xc0\xfeJ\x8f\xe5\x87\xf3\xc6+\x1e[\xb4\x8dt1\xbdBml?\xf3\xb03\x92wGA\xbc\xbe\xc1\x01P\x87f\xc8H\x05\x9f\x91\x89OZ\x9d&\x96\xfef\xa5\x1f^4aH\x1a\x89\xc5\xf5I\xf3'\xb4҈\x1b¿\xebm?\xab`g\x8ePG\xff\xd7^\x9d\b\xbb\xdcI\xf3!j\x97\xf1\xf0\xf4\xa1 x\x8a\xad\x1c\x98YWsx\xc8Amjx\x03B:*$\\$:T\x96\x0e*\x16\x1aK\xf06\xbcJ|nt-\xb7C\xa1\xbb\xb5є\xc7\xdc \xdd\xd3\xdc*\xdeD\xa8E\xde\xd1Zs\x90\x02mE\xf1!k\xc93'\xc1\xa6\fRKTb\x80M\x93Q\x16E\xb1((\xa8\x99\xbaa\xc3\xd5yc\xac\xa4\x99\xd4Ƀ/\x04\"\xd6\xd8&\xa7f\xedQ\v\xecg\x9bȍ\x89\x80\xe6P\xc0Q\xfa]BJ5\x16w\xf0\xd5أ\xb1\xc7\xd3\xd8t\x8f\xf7\xe7\x1d\xd2Δx\x11\x1cr\x8b>z\x1b*r\x1fr\xa59\xc0\xc7\xe0\"\xd6\xf6q\xa2\x8cX\xf0\x95\xd3{<\r\x15\r\xb7\x8c\x9bK\xa1\t\x96c\x11\xb5\x84ﾻ-\xd2 \xbb\x95A\xa5{\x11\xd4b\x8d\x16\xf5\xa0\x9a(\xe39\xe6(r\x1a\xf20\xack\xe4^\x1eP\x9dbN\"\xf0\xfc\x116\xc1\x83\b\x18\xad\xc6\xf8\xfeȬp\xc0M\xd32/7RI\x7f\x02\xe9&\xe83\xa5\xcc\x11E\xb686\xad?\xcd\xe1\x83v\x9ei\x8e\xee\\\a\x91ƒ+0\x9dv\xe5(\x8e\x05\x1d\xb3c\x18\x98\xc87\xc6y\xe0h\xc9\x1d\xd5\t\x8e\xd6\xe8픰#\xe9\x90z@\xab\xd1c̈\xc2pGɐc\xeb\xdd\xc2\x1c\xd0\x1e$\x1e\x17Gc\xf7Ro+b\xb0\xcaೈ\x9d\xdd\xe2\xfb\xf8\xe7[\xbc\xc0\xb4\t'\xeep\xdeu\x8c\xf5\x13\x95\xb7~\x87)E\xac\x93\x0f\x1a\vT@\x90k7\xd9w\x13\xb2\x8e\x85\xddX]\xde\x1d\xc5\xe4c\xf9c\x8f\xc3\xd4\xf1\x15P\x01\xf8R]t[5\xac\xad\xd2n\xe6M#\xf9\xac/m\xf2\xfb\xaf\xe3OiV\xa4\x16\x92Sq{\x8d\x1b\xa5\x89\x13W=͈\x1a\xfa]\xce\x14Z\x8e\xab)\x89\x9bk\x85\x1b\x1c\x7f\xea\uef74\xbe\t\xbas\xfew\xe8\xa9\xeet\xa0\x91\xea\x03f\x87z\x8e\x80ɍքT\xde\x00;\xa7\x81\x1f\\?\xff\xbd\x12=7\x81\xefqD\xf1\x03Q\xdeōE\xc7\xe9\x18\xf1\x12\x1c\xc6\xc4t\x8b\r\xb8\x1d\x11\x9c\xad\xd0\xde\xc3\xcb\xea\x816\x9eK\b\x06\xab\a\xd8\x04-\x14\x16\x8e\x8e;\xd4\xd4u\xc9\xfa4~\x17\x8d\xe7\xc7u\xd1j\xac\xber\xdfTt;.C\xcaoK\u061cF\xea\xa5;\x84l-\xd6\xf2\xcb\x1dB>ōE\xe1-\xf3;\x90\xdaI\x81\xc0Fԟ\n\xd9\tAϵѧ\x8c9\xdf`\x9e\xafaCb\xe75\xf0Pt|#~\x9e\xf2\xb6\xb3\x16\xca\xef\x9cݮ\xeb\xe4\xa98\x1e\x95\xe8p~\x94\xf9S\xaa>\xf9H\x19q\xc5\xcc\xcb\xf0\xc4W\xaa\xd8\xf244\x16\xccT3\x19kѵF\v\xea9\xef\xaba/,\xff\xef*\xd9q\xb3V\xd7(\xd7[+V\xb8\xab\x8d\x8b\xcf`\xafn\xe4\xd2\xe3`\xb7M2\x1bG\r\xf6\xa5\x97\xeb\xc9\xf8\x8b\xb4p\xa3%W\xa7\xaf\x93\x8eꗠce\x1b\xab\xaa\xf9l\xe4\xc4{l-R\x06\x13K\x92\xcdƃ\xda\x1c\xe9p\x87Z*ˌN\xf9\x9ez[\xa6E~U\xa0\xa5\x11\xcaG\xa9\x14\xd5\x00\x16\x1bCʢ\xb2\xdcR5\xc7b\xadu\xf8\xcd\xfc\xcd\xff\xafeT\xccy\xea\x00Q|ƃ\x1c>\xadݧ\xee\xc7\x01\x95\x82\x0e瘡\x1f?\x95׆\x85\xcd\xdb~\x82Z*\xaa\xff:\xd0qGu0\xf20\xfcn\xfd\xf8\x83\x8b=\x10j\xef\xe0H\x16t\x91%jzL~\xe1\t\xceS\x12\xb9i\xffn\x01\xae\r(\xa3\xb7h\xcbk\x0f\x15xɛ\x8c\x05\x81\x9er\x95\xde\x02\xdf1\xbd\xa5\xc8\x18\x83\xfc\xc8p\xe6\xbe\xcb'yϤ\x83H=\xe1\x1dw\x19\xf4Y\x8e\xb54\xaf1\xe6\xf43\xfc\x99\xffl\xd9\xcbkoO\xefSP[,\xd1_,\xa9\x9c\x14]\xf9\xcb\xd3\xfce|\xfb\xfb\xc0\xf0\xdd\xff[\xd5\xf3_}\xa9\x18|\xa1\xf8U(\xa7\xa1:\xf7f\xf1\xfc1\xedJ\xef\xb5\xf9\b\xb0\x8d\t~$\xf7w\x1c~4\xa6\xe3ǘ\xd7\xf0\x18?1\xdd*OhO\xb1\b\x0f\xd6\xc67\xdd\xf2\xd6\x18\x91b,+ݏ\xc0\x0f\xbd/aݵ\xe1w\xb2;\xe4\x1a\xcd҃ɔi;v\xcdJ\xee΄\xcd\xf9\xa5~\t\xff\xfc\xf7\xec?\x01\x00\x00\xff\xff\x03f\x86Y\xc0\x1d\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcVMo\x1b7\x10\xbd\xebW\f\xd0kwU\xa3hQ\xec\xadqr0\xda\x06\x82\x1d\xe4N\x91#-c.\xc9\xce\f\xe5\xba\x1f\xff\xbd \xb9+K\xab\x95\x93\\\xb27\x91Ù\xc7\xf7f\x1e\xd54\xcdJE\xfb\x11\x89m\xf0\x1d\xa8h\xf1/A\x9f\x7fq\xfb\xf8\v\xb76\xac\x0f7\xabG\xebM\a\xb7\x89%\f\xf7\xc8!\x91Ʒ\xb8\xb3ފ\r~5\xa0(\xa3Du+\x00\xe5}\x10\x95\x979\xff\x04\xd0\xc1\v\x05琚=\xfa\xf61mq\x9b\xac3H%\xf9T\xfa\xf0C{\xf3s\xfb\xd3\n\xc0\xab\x01;0\xe8Pp\xab\xf4c\x8a\x84\x7f&d\xe1\xf6\x80\x0e)\xb46\xac8\xa2\xce\xf9\xf7\x14R\xec\xe0e\xa3\x9e\x1fkW\xdcoK\xaa7%\xd5}MUv\x9de\xf9\xedZ\xc4\xefv\x8c\x8a.\x91rˀJ\x00[\xbfON\xd1b\xc8\n\x80u\x88\xd8\xc1\xfb\f+*\x8df\x050^\xbb\xc0l@\x19S\x88TnC\xd6\v\xd2mpi\x98\bl\xc0 k\xb2Q\nQ\x1fz,W\x84\xb0\x03\xe9\x11j9\x90\x00[\x1c\x11\x98r\x0e\xe0\x13\a\xbfQ\xd2w\xd0f\xbe\xda\x1a\x9a\x81\x8c\x01\x95\xea7\xf3ey\u0380Y\xc8\xfa\xfd5\b,J\x12O J]\x1b<\xd0\t\xbf\xe7\x00J|\x1b{\xc5\xe7\xd5\x1f\xcaƵ\xca5\xe6pS\x99\xd6=\x0e\xaa\x1bcCD\xff\xeb\xe6\xee\xe3\x8f\x0fg\xcbp\x8euAZ\xb0\fjB\x9a\x89\xab\xacA\xf0\b\x81`\b4\xb1\xca\xed1i\xa4\x10\x91\xc4N\xadU\xbf\x93\xe19Y\x9dA\xf8\xb79\xdb\x03Ȩ\xeb)0y\x8a\x90\v\x89cS\xa0\x19/Zɵ\f\x84\x91\x90\xd1\u05f9\xca\xcb\xcaC\xd8~B-\xed,\xf5\x03RN\x03܇\xe4L\x1e\xbe\x03\x92\x00\xa1\x0e{o\xff>\xe6\xe6|\xef\\\xd4))\x94\xe4\xb6\xf3\xca\xc1A\xb9\x84߃\xf2f\x96yP\xcf@\x98kB\xf2'\xf9\xca\x01\x9e\xe3\xf8#\x93h\xfd.tЋD\xee\xd6뽕\xc9Rt\x18\x86\xe4\xad<\xaf\x8b;\xd8m\x92@\xbc6x@\xb7f\xbbo\x14\xe9\xde\njI\x84k\x15mS.⋭\xb4\x83\xf9\x8eF\x13Ⳳ\x17\xddS\xbf\xe2\x02_!O\xf6\x84\xda#5U\xbd\xe2\x8b\ny)Sw\xff\xee\xe1\x03LH\xaaRU\x94\x97\xd0\v^&}2\x9b\xd6\xef\x90\xea\xb9\x1d\x85\xa1\xe4Dob\xb0^\xca\x0f\xed,z\x01N\xdb\xc1\nO\x1d\x9b\xa5\x9b\xa7\xbd-\xb6\x9b\x1d E\xa3\x04\xcd<\xe0\xceí\x1a\xd0\xdd*\xc6o\xacUV\x85\x9b,\xc2\x17\xa9u\xfa\x98̃+\xbd'\x1b\xd33pEڅ\xe1\x7f\x88\xa8\xb3\xb8\x99\xdf|\xda\ueb2ec\xb5\v\x04O\xbd\xd5\xfd4\xfc3\x9a\x8eFq\xce߲1\xe4\xef\xc5n\xe7;W/\x0fEdK8k\xd8\x06.\xbc\xfbu^\x8a\xa9~%3\xd5\xd1Gnt\"*\xcdw\xf4y\xb5t\xe8K\xb9@\xa2@\x17\xab3P\xefJP\xf9Ǡ\xacgP\xfey<\b\xd2+\x81'\xa4\r\x97\x95\x1ax\x8fO\v\xabw~CaO\xc8\xf3\x96ϛ\x9b\xca\x1e\xce߃WXZlʋE\xceVhNXd\t\xa4\xf6\xa7\xbcr\xda\x1e\x9d\xbe\x83\x7f\xfe[\xfd\x1f\x00\x00\xff\xff\xbeM\x1a\xea\xb1\n\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcWMo\xe36\x10\xbd\xfbW\f\xd0K\v\xac\xe4\x06E\x8b·\xd6\xd9C\xb0\xe96\x88\xb7\xb9S\xd4HbC\x91,9t6E\x7f|1\xa4\xe4\x0fYv\x9c\xcb\xea\xe6\xe1p\xf8\xe6\xcd\xcc#]\x14\xc5B8\xf5\x84>(kV \x9c¯\x84\x86\x7f\x85\xf2\xf9\xd7P*\xbb\xdc\xde,\x9e\x95\xa9W\xb0\x8e\x81l\xff\x88\xc1F/\xf1\x16\x1be\x14)k\x16=\x92\xa8\x05\x89\xd5\x02@\x18cI\xb09\xf0O\x00i\ry\xab5\xfa\xa2ES>\xc7\n\xab\xa8t\x8d>\x05\x1f\x8f\xde\xfeX\xde\xfcR\xfe\xbc\x000\xa2\xc7\x15\xd4\xf6\xc5h+j\x8f\xffD\f\x14\xca-j\xf4\xb6Tv\x11\x1cJ\x8e\xddz\x1b\xdd\n\xf6\vy\xefpn\xc6|;\x84y\xccaҊV\x81>ͭޫ\xc1\xc3\xe9\xe8\x85>\x05\x91\x16\x832m\xd4\u009f,/\x00\x82\xb4\x0eW\xf0\x99a8!\xb1^\x00\f)&XŐ\xdd\xf6&\x87\x92\x1d\xf6\"\xe3\x05\xb0\x0e\xcdo\x0fwO?m\x8e\xcc\x005\x06镣D\xd4\x7f\xc5\xce\x0e\xd3\x04@\x05\x100\xc0\x01\xb2;\x84 \f\bO\xaa\x11\x92\xa0\xf1\xb6\x87J\xc8\xe7\xe8\xc0V\x7f\xa3$\bd\xbdh\xf1\x03\x84(;\x10\x1c%;\x1c\x9c\xa5m\v\x8d\xd2X\xeel\xce[\x87\x9e\xd4Hy\xfe\x0e\x1a\xea\xc0z)\v\xfe8\xf1\xbc\vj\xee,\f@\x1d\x8e\xe4a=p\x05\xb6\x01\xeaT\x00\x8f\xcec@\x93{\x8d\xcd\xc2\fٔ\x93\xd0\x1b\xf4\x1c\x06Bg\xa3\xae\xb9!\xb7\xe8\t\x1aE\xaf\xcb41\xaa\x8ad}XָE\xbd\f\xaa-\x84\x97\x9d\"\x94\x14=.\x85SEJĤQ+\xfb\xfa;?\ff8:\x96^\xb9!\x03yeڃ\x854\x1d\xef(\x0f\xcfK\xee\xae\x1c*\xa7\xb8\xaf\x02\x9b\x98\xbaǏ\x9b/0\"ɕ\x1aZl\xe7z\xc2\xcbX\x1ffS\x99\x06}ޗڔc\xa2\xa9\x9dU\x86\xd2\x0f\xa9\x15\x1a\x82\x10\xab^Q\x18{\x9dK7\r\xbbNR\x04\x15Bt\xb5 \xac\xa7\x0ew\x06֢G\xbd\x16\x01\xbfq\xad\xb8*\xa1\xe0\"\\U\xadC\x81\x9d:gz\x0f\x16Fy&j^\x01\x128\xe1[\xa4\xa9u\x82\xe5Kr\xe2\xe3_:q,X\xdfcٖ\xac9a\x00\x92\xf5\xe8\x87i\xa1.a\x80\xd9F\x9fE2\xf67\xd3\xc0\xbc\xb2\xa0\xb0\xd8\x1db:=\x9a?4\xb1\x9f?\xa0\x80\xdf\x13\xe6{\xdb^\\_[C<\x17\x17\x9d\x9e\xac\x8e=n\x8cp\xa1\xb3o\xf8\xde\x11\xf6\x7f:\xf4\xf9\x1a\xbe\xe8:\xde滫\xef\x82c\xd4g\xcf}D\xbeA\xf0|\xa6\x83\xc3UQ\xae\xc04x^\x95\xe8zs\xf7\x1e\nϸ\xbf\xa3Hw\xa6\xb1o\xa4\xb8w\x9c\xf5;#\x03\xe3\x97\xde\x10o\xf74\xbfBƞ\xe6-\xf9\xeeD\xf8\x14+\xf4\x06\t\xc3^\xa9_\x14u\xb3\x11\x01^:%\xbb\xb41\r\x04_\x02!X\xa9\xe6$\xf5\n\xf8\xac#\xca\xe3\xccP\x16iXg\xcc\f\xfe\xc4|F\xfd\xce\x1dP\f\x8at\x95\x82\x92\xa0\x18ޡ\xa1\xc9\x7f\xa4ZF\xef\xd3\x15\x95\xad\xfc2\x99n\xb8VDG\xe5\xf9\xeb\xf1\xfe\r%\xbd\xdd{\xa6\x17\xb7P&\xa3q\x1e\x8b\xa0Z~A\xf1\x1akiҸS2\xf2w\xfc\xc2;&j\xb6\xa2\xf8թ<\x80o@\xfc\xb8ŝ\x8f&\xdf\xf3\xd37l\n\x88\x81\x9f[ \x85\x99\xc1X!Ԩ\x91\xb0\x86\xea5\xdf\\\xaf\x81\xb0?\xc5\xddX\xdf\vZ\x01\xdf\xff\x05\xa9\x9962QkQi\\\x01\xf9x\xae\xcbf\x13w\x9d\b3cx\x94\xf3\x03\xfb\xcc5\xc6n\x18/v\x06\x9c\xbd_\n\xf8\x8c/3\xd6\ao%\x86\x80\xa7ct6\x93\xd9!81\x06~\xa4\xd5\a,\r\x7f\x19\x06\xcb\xff\x01\x00\x00\xff\xffx\xae@\xbaJ\x0e\x00\x00"), - []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Z_s\x1b\xb7\x11\x7fק\xd8Q\x1e\x92\xcc\xf8\xc8\xdam3\x1d\xbe\xd9r\xd3Q\x9b\xa8\x1aS\xf6K&\x0f\xcbÒ\x87\xe8\x0e@\x01\x1ce6\xcdw\xef,p \xef\x8e )҉}/6\xf1g\xf1\xc3\xfe߅\x8a\xa2\xb8B#?\x90uR\xab\x19\xa0\x91\xf4ѓ\xe2_n\xf2\xf877\x91z\xba~y\xf5(\x95\x98\xc1M\xeb\xbcnޑӭ-\xe9--\xa5\x92^juՐG\x81\x1egW\x00\xa8\x94\xf6\xc8Î\x7f\x02\x94Zy\xab\xeb\x9al\xb1\"5yl\x17\xb4he-\xc8\x06\xe2\xe9\xe8\xf5\x9f&/\xbf\x9b\xfc\xf5\n@aC30Z\xacu\xdd6\xb4\xc0\xf2\xb15n\xb2\xa6\x9a\xac\x9eH}\xe5\f\x95L{eukf\xb0\x9b\x88{\xbbs#\xe6{->\x042o\x02\x990SK\xe7\xff\x95\x9b\xfdA:\x1fV\x98\xba\xb5X\xef\x83\b\x93N\xaaU[\xa3ݛ\xbe\x02p\xa564\x83;\x86a\xb0$q\x05\xd0]1\xc0*\x00\x85\bL\xc3\xfa\xdeJ\xe5\xc9\xde0\x85Ĭ\x02\x04\xb9\xd2J\xe3\x03S\ued40\b\x10\"Bp\x1e}\xeb\xc0\xb5e\x05\xe8\xe0\x8e\x9e\xa6\xb7\xea\xde\xea\x95%\x17\xe1\x01\xfcⴺG_\xcd`\x12\x97OL\x85\x8e\xba\xd9\xc8\xdey\x98\xe8\x86\xfc\x86A;o\xa5Z\xe5`<Ȇ\xe0\xa9\"\x05\xbe\x92\x0e\xe2m\xe1\t\x1dñ>\xdc2\x7fp\x98\xe7\xed\xcecc\x06\bn,\xe1nk\x84 \xd0S\x0e\xc0\x96\x9f\xa0\x97\xe0+b\xce\a\xc5B\xa9\xa4Z\x85\xa1(\t\xf0\x1a\x16\x14 \x92\x80\xd6d\x90\x19*'F\x8b\x89JD\a\xb0\xeeF\xa3\xa7x\xc3\xeb\x7foT\x03@\xf7Z\\\x00\xe5\xacs\xe3\xe2\xc1\xa9\x1f\xfaC'\xf5\xa3\xa2\xb0&\x1dޚZ\xa3 \xcb\xc7W\xa8DM,Y\x04oQ\xb9%\xd9\x030Ҷ\x87\x8d\x19\x82y\x9f\xe8\xf5f\xceaFg;s\xaf-\xae\b~\xd0epP\xacҖ\x06:\xed*\xdd\xd6\x02\x16\xe9\x14\x00\xe7\xb5\xcd*8#\x8e\xbb:\xba\x89\xec\xc8Άg\x1eFߣ\x9d\xfc\xe9\xa4d\x1b\x91Z\xe5-\xe8\xf5\x8a\xf2\xd6\x13\xa7\xd7/\xa3\xbb*+jp֭Ԇ\xd4\xeb\xfb\xdb\x0f\x7f\x9e\x0f\x86\x01\x8cՆ\xac\x97\xc9}Ư\x17\x1cz\xa30d\xf5\xff\x8a\xc1\x1c\x00\x1f\x10w\x81\xe0(A.\xead\x1c#\xd1a\x8a\xe2\x91\x0e,\x19K\x8eT\x8c\x1b<\x8c\n\xf4\xe2\x17*\xfddDzN\x96\xc9$A\x95Z\xad\xc9z\xb0Tꕒ\xff\xdd\xd2v\xac{|h\x8d\x9e\x9c\x87\xe0j\x15ְƺ\xa5\x17\x80J\x8c(7\xb8\x01K|&\xb4\xaaG/lpc\x1c?jK \xd5RϠ\xf2\u07b8\xd9t\xba\x92>\x85\xccR7M\xab\xa4\xdfLC\xf4\x93\x8b\xd6k릂\xd6TO\x9d\\\x15h\xcbJz*}ki\x8aF\x16\xe1\"*\x84\xcdI#\xbe\xb2]\x90u\x83c\xf7\xb4&~!ҝ!\x1e\x8e} \x1d`G*^q'\x85\xe4\xbb\xde\xfd}\xfe\x00\tI\x94T\x14\xcan\xe9\x1e_\x92|\x98\x9bR-\xd9\a\xf0\xbe\xa5\xd5M\xa0IJ\x18-\x95\x0f?\xcaZ\x92\xf2\xe0\xdaE#=\xab\xc1\x7fZr\x9eE7&{\x13\xd2\n\xf6e\xada5\x17\xe3\x05\xb7\nn\xb0\xa1\xfa\x06\x1d}fY\xb1T\\\xc1Bx\x96\xb4\xfa\xc9\xd2xqdoo\"\xa5:\aD;\xca_\xe6\x86J\x16,\xf3\x96wʥ\xec<\xddR[\xc0\xf1\xf2!\x9f\xf2\x0e\x80\xbf\xac\x97\x1b/:\xa5t\xfc\xbd\xc9\x11J\x80U\xcfa'o\xdc9\xcfz\xe8<\xfb_r\xe1\xdb=\x96\x8cv\xd2k\xbba\xc2\xd1{\x8f\x15\xe2\xa0l\xf8+Q\x95T_r\xbd\x9b\xb0\x13\xa4\x12\xccv\xda*4\xbb\xa2H5\x00\xd5j\xa5\xd9\xc4\xc6Ҁ[\xcf\xcbX\xc9\x1d\xf9\xfc]U\xa00\xda\xc9\x17\x95\nvy \xf4\xf3\xbd\xf1\xa5\x17Zׄc^*-\xe8ĝﴠ\x9c\xb0x+\xf8\n}\xc2Ƌl\xab\xd4>o\xf9\xd3\xea,q\x18-N\xe0\xeaND\xb0\xb4$K\xaa\xa4\xe4\xfb\x8f\xe5c\x19d\xfdLi\x1f\xe3a\xfb\x80#\x812\x8b\xf8\xf5\xfdm\n\x86\x89\x89\x1d\xf6\xbdxw\x92?\xfc-%\xd5\"\xe4\x0e\xa7\xcf\xcej.\x7f\xb7\xcb\b\"D\x04\xaf\x01\xc1H\x8a\x19\xf76\x1a\x83T\xce\x13\x8an\x90\x9d\xa0\xa5n\xeeE\xf4\xf4\aA\xf2\xb7\x8b\xda,\x13@\x8e\xb2%\xc7+\xb4\x04\xb5|\xcc\xd8O\xfc\xaeC\xae\xb8\x83\xf9+[\xcfo\xd7\xf0Mt^\xd7\xfc\xf3:\xc2ئ-}\x03\xdb\xc1\x89Vf\xe5jE\xbb\xa4tOY8\xccr\x80\xfa\x16\xb4\xe5\xbb*\xdd#\x11\b\xb3\x9cb| \xb1\a\xef\xa7W?_\xc37C\x1e\x1c8J*A\x1f\xe1\x15{\x9f\xc0\x1b\xa3ŷ\x13x\bz\xb0Q\x1e?\xf2Ie\xa5\x1d)Ъ\xdeĂ`M\xe04W\x94T\xd7EL\x10\x05<\xe1\x06\xf4\xf2\xc09ID\xac\x9a\b\x06\xad?\x9a$v|8n4\xfbYS\xfa\x9eg/!\x8bz\x96\xf5~\xb1\f䙜\b\xe5\xc2'p\xa2_j]\xc0\x89\xc7vAV\x91\xa7\xc0\f\xa1K\xc7|(\xc9x7\xd5k\xb2kIO\xd3'm\x1f\xa5Z\x15\xac\x8cE\x94\xba\x9b\x86\n~\xfaU\xf8\xe7ҋ\x87Z\xffSo?\xe8M|~\x16\xf0\xe9nz\t\aRv\xff\xfc\xd8u\x90\x0f\xf3.\xe1\x1c\xd3d\x9b\x7f\xaadY\xa5Z\xaf\xe7m\x1b\x14\xd1\x1d\xa3\xda|!\xdba>\xb7\x96\x11m\x8a\xaeUY\xa0\x12\xfc\x7f'\x9d\xe7\xf1K\x18\xdb\xcaOr.\xefo\xdf~I\x8bj\xe5%\x9e\xe4@\r\x13\xbf\x8f\xc5\x0eUѠ)\xe2j\xf4\xba\x91\xe5h5\xe7\U00037085\xb4\x94dO\xa4\x7f\xef\x06\x8bS\x82\x9a\xa9\x06\xb6k\xce\xca?=\xae2\t_\xbf\x8b{,-<ʯӪ\xf0\x80+\ah\t\x10\x1a4\xac\x11\x8f\xb4)b\xc6aPr\xba\xc0\x19\xc1\xb6k\x05hL\xcd1=f\x11\x19\x8a]\xfe۱\a]\xb8\xdf!\x86dE\x99\xbats\xf2^\xaa/Ȝ\xf7# \xbf/\xa3\xb6=\xccR\xab\xa5\\\xb56T\xa0\xfb\x9cRm]㢦\x19x\xdb\x1e\xaa\xb9\x8e2\xf2\x81\x97\x1c\xbf\xff\xfb\xdeҤ\xe1'\x1a\xae\xf9[\rڰ\xfb\x97!\xd56\xfbP\nx\xd4Fbfܒ\xf3{\xd6\xcb\x13\xd7\xd7\xe7\xd8XT\xcaKJ\xee\xeeq$S\x95v\x8a\xde%\xf0\xa92\xed\xf7óB?\xc37pu\xcf\xe5\xc8\x10w\x91o\x97\x8c\xd6p\xcd<\x1a2Z\x8cF\x86np49\xe8\xd9\xf7\x91\xee\xf7\x90\xc2S\xcc\x19]\xa4\xf8\xc4\xd4\xf14\x06G\x9f\x1e\x9e8\xed\xbe\xb4\x8fTj.\xbf\x06\xfd\xec\x8b\xda,\xfbdB\xff\u05ca\xce0d\xc3~\xa0\xf7J\xd5\x1d\x9ck\x04\xf5\xc9ŝ!Gaj$B\x19\xc5U\xde\x12eM\x02\xd2S\xe4\x99T\x16\xb4\xe4\x18\x1d\x8d45\":x\x87\v\x98\x87\x8a\xc0\x85n\xea\xd7nK\xb3u$B3/Ä\xfd\x88\xbdԶA\x1f\x1f\x06\n&q\x99\xf7\xca\xdalC\xce\xe1\xea\x94\xd1\xfe\x18W\xc5\xfeL\xb7\x05p\xa1[\xbfm\xd0\f\"\xd2\u05eeS\xb4\xf3zD\xd9\xd6\xc7P\xc7\xd1WI\xa5\x97m]\x87=}\xef\xb0{\xa6\x0e\xa8\x16\x94\xcf\xeb\x8e4\x88\x8e\x01\xacНb\xd5=\xaf\xc9Y\xdd֥\x1d5;8\xe2\xbe\xef\xe8)3\xba\xf7nܟ\xbcI&\x93\x99\xfb>X\xc3Y\xf7\xef\x0e\xba\xc4ܷM\xcdJ\xd7\xc9µ\xc7\x1aT\xdb,\xc82s\x16\x1bOn\xe4\xf8Q\x89>'s\xd5\xdfn\x7f\x12j\xa4\xd4u0\xba^l09\xafAHgj\xdcl\xef\x12rn\xb6\xaf|cz\xa7\xe4\xc9\xd2\r\x1d\xca!\x8e\xb7\x16\x03\xa6\xb7Z\x1d\xa8R\x93\x91K\xe5\xbf\xfbˑ\xa4]*O\xabQ\x18\xe9晝o\xf8\x94?\xe6\x84#9\x90Sh\\\xa5\xfd\xed\xdb\x13\xaa1\xdf.L&\xb2\xcb\xe7\x83C\fo\x1eݢN\x152Pw\x0e\xe7,\xfb\x1d\xfe\x19\xc3%Z<\x1fP8\x11\xaf\xba\xbf\xaa\xc8E\x859\x19\xb4\xec\x13\u008b\xda\xcd\xf8}\xf8\x058\x19\x1a\xe0\x9c\xed\xc6\xf4\xb7\xacP\xad\xb2\xfd\x11\xadB\x02\xa7\xed\xfe\xf3&\x9c\f@\xc3\v}\xceؓU\xa7\xbd\xc1\x80\\\xf4hw\x8fI\xfd\x91v\xb1}g\x9d\xc1\xaf\xbf]\xfd?\x00\x00\xff\xff\f\x19\xe2z\x0f%\x00\x00"), + []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4ZK\x93\x1b\xb7\x11\xbe\xef\xaf\xe8Z\x1flWi\xc8HI\\)ޤU\x9c\xda\xc4V\xb6ĕ..\x1f\xc0A\x93\x03\xef\f\x80\x00\x18R\x8c\xe3\xff\x9ej<\x86\xf3\x00\xc9]\xca\xf2\xe2\xb2K<\x1a\x8d\xaf\x1b\xfd\xc2\x14EqŴ\xf8\x88\xc6\n%\x17\xc0\xb4\xc0O\x0e%\xfd\xb2\xb3\x87\xbfٙP\xf3\xed˫\a!\xf9\x02nZ\xebT\xf3\x1e\xadjM\x89oq-\xa4pBɫ\x06\x1d\xe3̱\xc5\x15\x00\x93R9Fݖ~\x02\x94J:\xa3\xea\x1aM\xb1A9{hW\xb8jE\xcd\xd1x\xe2i\xeb\xed\x9ff/\xbf\x9b\xfd\xf5\n@\xb2\x06\x17\xa0\x15ߪ\xbamp\xc5ʇV\xdb\xd9\x16k4j&ԕ\xd5X\x12\xed\x8dQ\xad^\xc0a \xac\x8d\xfb\x06\x9e\xef\x14\xff\xe8ɼ\xf1d\xfcH-\xac\xfbWn\xf4\aa\x9d\x9f\xa1\xebְzʄ\x1f\xb4Bnښ\x99\xc9\xf0\x15\x80-\x95\xc6\x05\xbc#64+\x91_\x01\xc4#z\xb6\n`\x9c{\xd0X}g\x84thn\x88B\x02\xab\x00\x8e\xb64B;\x0fʈ?\xb0\x8e\xb9ւm\xcb\n\x98\x85w\xb8\x9b\xdf\xca;\xa36\x06m`\x0e\xe0\x17\xab\xe4\x1ds\xd5\x02fa\xfaLW\xccb\x1c\r\xe0.\xfd@\xecr{b\xd9:#\xe4&\xc7Ľh\x10xk\xbcP\xe9\xf4%\x82\xab\x84\x9dp\xb7c\x9684\xce\x1f;ϋ\x1f'\x8aֱF\x8f\x99\xea-\r\\q\xe60\xc7Ӎjt\x8d\x0e9\xac\xf6\x0e\xd3I\xd6\xca4\xcc-@H\xf7\xdd_\x8e\xc3\x11\xf1\x9a\xf9\xa5o\x95\x1cb\xf3\x86z\xa1\xd7\x1d8!Ym\xd0d\x01R\x8e՟È#\x02oz\xeb\x03'\x81n\xbf\xff,+\xa4x\xa0\xd6\xe0*\x84(\x95\xa5S\x86m\x10~Pe\x90\xe0\xaeB\x13%\xb8\x8ajU\xa9\xb6\xe6\xb0J'\x06\xb0N\x99\xac\x145\x96\xb3\xb0*\xd2MdG\xa2\x1c\xee\xf9%4\xad4Ȳ\x9a\x96\xac\xd1\xcc\xcf\x10J\xe6\xd5\xed\xf5\x06\x1f\xa5j}H\xa5\xe2\xd8\xe1\x87\x13\xb6\x84\x05mT\x89֞\xb8\x01Dc\xc0ȻC\xc7Y\x80*\xf4s\x12?\xad\xae\x15\xe3h\xc0)\xa8\x98\xe45\xd21\x188ä]G\x15\x99\n0-\xbb\xdf\xeb!+\x1f\xe2\xc01v¬\xed\xcb`\a\xcb\n\x1b\xb6\x88s\x95F\xf9\xfa\xee\xf6㟗\x83n D4\x1a'\x92]\x0e\xad\xe7uz\xbd0<\xee\xff\x8a\xc1\x18\x00m\x10V\x01'\xf7\x83\xd6\xc3\x10-,\xf2\xc8S\x80GX0\xa8\rZ\x94\xc1!Q7\x93\xa0V\xbf`\xe9f#\xd2K4D&݅R\xc9-\x1a\a\x06K\xb5\x91\xe2\xbf\x1dmKXӦ5sh\x9d\xbf\x8cF\xb2\x1a\xb6\xacn\xf1\x050\xc9G\x94\x1b\xb6\a\x83\xb4'\xb4\xb2G\xcf/\xb0c>~T\x06AȵZ@圶\x8b\xf9|#\\\xf2ťj\x9aV\n\xb7\x9f{\xb7*V\xadS\xc6\xce9n\xb1\x9e[\xb1)\x98)+\xe1\xb0t\xad\xc19Ӣ\xf0\a\x91\xde\x1f\xcf\x1a\xfe\x95\x89\xde\xdb\x0e\xb6\x9d\b:4\xefB\x9f \x1er\xaat\tX$\x15\x8ex\x90\x02u\x11t\xef\xff\xbe\xbc\x87\xc4I\x90T\x10\xcaa\xea\x04\x97$\x1fBS\xc85\xe9<\xad[\x1b\xd5x\x9a(\xb9VB:\xff\xa3\xac\x05J\a\xb6]5\u0091\x1a\xfc\xa7E\xebHtc\xb27>^\x81\x15\xdd%\xb2\x00|<\xe1V\xc2\rk\xb0\xbea\x16\xff`Y\x91TlABx\x94\xb4\xfaQ\xd8xr\x80\xb77\x90b\xa8#\xa2\x1dY\xb6\xa5ƒ\x04K\xd8\xd2J\xb1\x16љ\xac\x95\x016\x9e>\xc4)o\x00\xa8e\x1d\xc9x\xd29\xa5\xa3\xf6&G(1,{\x06<9\xbc\xe8\x9f\xea\xa1\x7f귃\x95\x8fk\fje\x85SfO\x84\x83\x83\x1c+\xc4Q\xd9P+\x99,\xb1\xbe\xe4x7~%\b\xc9\tv\xec\x14\x9aLQ\xa0\xea\x19Ur\xa3芍\xa5\x01\xb7\x8e\xa6\x91\x92[t\xf9\xb3\xcac\x0eMH8\x84\x98\xd0\x0f%LJ^)U#\x1bcI\xee\xee̙\xc9\x01\xe6\x84彭\xab\x98K\xbc\xd1$\xd3J9Ŗ\x9a\x92O\x12\x87V\xfc\f_qG\x06\x06\xd7h\xd0G#\xc1\xf6k\xe5=\x84cB&\x9b\x16\x12\x01p*\xc3\xd9*(\x11r\x18\xdf\r8y?\xe0\x84\xa3\xccr\xfc\xfa\xee69\xc3\x04b\xe4}\xe2\xef\xce\xe2Cm-\xb0\xe6>r8\xbfwVs\xa9ݮ\x03\x13\xde#8\x05\f\xb4\xc0\x12\a\xde\x18\x84\xb4\x0e\x19\x8f\x9dd\x04\rƱ\x17\xc1\xd2\x1fe\x92\xda\xc1k\x93L\x80\x91\xe7\x11\x1c\xfe\xb9\xfc\xf7\xbb\xf9?T8\a\xb0\x92B3\x9fDa\x83ҽ\xe8\x12)\x8eV\x18\xe4\x94\x16\xe1\xacaR\xacѺY\xa4\x86\xc6\xfe\xf4\xea\xe7<~\x00\xdf+\x03\xf8\x89Q:\xf2\x02D\xc0\xbcsfIm\x84\r\a\xef(\xc2N\xb8\xca3\xaa\x15\x8f\a\xdc\xf9#8\xf6@79\x1c\xa1E\xa8\xc5C\xe6\xfe\x84v\xed\xa3\xb9\x03\x9b\xbf\xd2\xed\xf9\xed\x1a\xbe\t\xc6\xeb\x9a~^\a6\xba\xb0\xa5\x7f\xc1\x0e\xec\x84[f\xc4f\x83\x87\xb8\x7f\xa2,\xe4f\xc9A}\v\xca\xd0Y\xa5\xea\x91\xf0\x84IN\xc1? \x9f\xb0\xf7ӫ\x9f\xaf\xe1\x9b!\x06G\xb6\x12\x92\xe3'xE\xd6\xc7c\xa3\x15\xffv\x06\xf7^\x0f\xf6ұO\xb4SY)\x8b\x12\x94\xac\xf7!\x00\xde\"X\xd5 찮\x8b\x10 rر=\xa8\xf5\x91}\x92\x88H5\x19hf\xdc\xc9 1\xe2p\xfa\xd2L\xa3\xa6\xd4\x1ew_|\x14\xf5\xa8\xdb\xfbl\x11\xc8#\x91\xf0\xe9\xc2g \xd1O\xbd.@\xe2\xa1]\xa1\x91\xe8Ѓ\xc1Ui\t\x87\x12\xb5\xb3s\xb5E\xb3\x15\xb8\x9b\xef\x94y\x10rS\x902\x16A\xeav\xee\xcbH\xf3\xaf\xfc\x9fK\x0f\xee\xeb?\x9f{zO\xe4\xf9 \xa0\xdd\xed\xfc\x12\x04Rt\xffx\xdfu\x14\x87e\f8\xc74\xe9\xce\xef*QV)\xd7\xebYۆ\xf1`\x8e\x99\xdc?\xd3\xdd!\x9c[C\x1c\xed\x8bX\x03-\x98\xe4\xf4\xbf\x15\xd6Q\xff%\xc0\xb6Ⳍˇ۷\xcfy\xa3Zq\x89%9\x92Ä\xf6\xa98pU4L\x17a6s\xaa\x11\xe5h6\xc5\U00037704\xb4\x16h΄\x7f\xef\a\x93S\x80\x9a\xc9\x06\xba9O\x8a?\x1d\xdbd\x02\xbe~y\xf8TXx\x12\xaf\xf3\xaap\xcf6\x16\x98A`\xd00M\x1a\xf1\x80\xfb\"D\x1c\x9a\t\n\x17(\"\xe8\n\x83\xc0\xb4\xaeɧ\x87(\"C1ƿ\x11\x1ef\xfd\xf9\x8e\x01\x92\x15e\xaaJ-\xd19!\x9f\x11\x9c\x0f#F~_\xa0\xba\x9a]\xa9\xe4Zlb\xb5s\x8a\x94l뚭j\\\x803\xed\xb1\x9c\xeb$\x90\xf74\xe5\xf4\xf9?\xf4\xa6&\r?S`̟jPv\x9c\x1e\x06e\xdbLY)\xe0Ai\xc12\xfd\x06\xad\x9b\xdc^\x1a\xb8\xbe~\xca\x1d\vJyI\xca\x1d\xd2\xe0\\V\x1a\x15=\x06\xf0)3u\xea\x90\xe5e\x85\xfe\x04\xdb@\xd9=\xa5#C\xbe\x8b|\xb9d4\xa7W]N]Z\xf1Q\xcf\xd0\f\x8e\x06\xc3\xf9\x1eUC\xf2\x05\xed'T\x91\xc2\xebU\xc448G\x97\u07b4(쾴\x8eD\x89\x9dvȻB\xff%\x12\x7f=&\xe2k\xbf\x86\xc7K!\x1a\xecR\xff\xa1\xad\v\xc9\xdd\nA\x1b\xd4,[\x15\x02_\xb9\xb7\xbe\x84\xf9\xb5\rĄ\x85\xd6\"\xf7\x15\xb4\xc9\xde\x13\n\xe9E\x893\x87\x05\xad\xbf\xcc^\xe4\vS\xe11\xad\xffRrQ\x95jJf\n!K\xa8\xf9'\x9c\xf4\x8a\x97C\xec@\xae\xc3+PC\xee\xb3PJ\x92\xd7L\xd4\xc8!=\x11?\x91\xca\n\xd7\x14\xe2\x04\x1b\x97\xea8\x91\xbd\xe3\xf9\xdfiIf@\x98\x06<_R\x98\rZ\xcb6\xe7lޏaV(o\xc5%\xc0V\xaauy%\xff\xda\xc6{\xfa\xb4\x12[\xb6r44\x11\xccU\xc9\"\xacۺ\xf6k\xfa\xc6\xf5\xf0\xf9\x80\xe7j\x85\xf9\xb0\xf8D}\xed\x14\x83\x15\xb3砺\xa399\xa3\xd5y\x84\x93V\vNx\xbfw\xb8\xcb\xf4&c\x90\x19\xba\x8b\x16&34\xf9\x0e\xa0?\x18\n\xc89\xe4\xd2X\x96f\xf7ʞ\x19\xfb\xde_\xbd'\x81\x1d\xf9\xbbĶt\x05\xe8J\xd5ɜ\xf8\xd7q\xd96+4$\t\xff\xfe>r\xd2L\xf2\xbe\xd8r\x99\xfaa}Ҡ@)V\x9bb\xdd\xdc\xdfo\xa7\x80\v\xabk\xb6\xef\xce\xe2\xf3#\xba\xcc\xf9G\x84ÍJfE\xe3\xb1x\xeft\x19\xb8\xfbV!\x9f\xfc\xe5>8\x18\xb6\xe9\xa7\x03\xa3\xf1\xee\x1b\x84/\xb3Éx\xd5J\xa6m\xa5\xdc\xed\xdb3\xaa\xb1\xec&\xa6\xfbxȽ\xbc\xf5\xf5\xefSqRT\x85\f\xab\a\xeb\xf6$c1\xfct\xe5\x12-^\x0e(\x9cq\x8e\xf1K\x9a\x9c\vZ\x92\x15 \x03\xe4_?oƟ9\xbc\xe8>\x9d`.V\x91ˊ\xc9M\xb6\x96\xa5\xa4\x0f\xb6\x95\x99>E\xc3Yo7<\xd0\x1f\xe9\xe8\xb2\xea4\xe9\xf4\x9c\xf3\x1e\xed\xf8\xf0\xd7\xefiWݛ\xf8\x02~\xfd\xed\xea\xff\x01\x00\x00\xff\xff\xd9H\xdbA\x14'\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Y_\x93۶\x11\x7fקع<$\x991\xa5\xdam3\x1d\xbd\xd9\xe7\xa6smr\xbd\xb1\xce~\xc9\xe4aE\xacHD$\x80\x00\xa0d5\xcdw\xef,@R\xfc'\xe9tnl\xbe\xdc\t\x00\x17?\xfc\xf6/\x96I\x92\xcc\xd0\xc8\x0fd\x9d\xd4j\th$}\xf4\xa4\xf8\x97\x9bo\xff\xe6\xe6R/v/g[\xa9\xc4\x12n+\xe7u\xf9\x8e\x9c\xaelJoi#\x95\xf4R\xabYI\x1e\x05z\\\xce\x00P)푇\x1d\xff\x04H\xb5\xf2V\x17\x05\xd9$#5\xdfVkZW\xb2\x10d\x83\xf0f\xebݟ\xe6/\xbf\x9b\xffu\x06\xa0\xb0\xa4%\x18-v\xba\xa8J\xb2伶\xe4\xe6;*\xc8\xea\xb9\xd43g(e\xe1\x99ՕY\xc2q\"\xbe\\o\x1cA?h\xf1!\xc8y\x17儩B:\xff\xaf\xc9\xe9\x1f\xa4\xf3a\x89)*\x8b\xc5\x04\x8e0\xeb\xa4ʪ\x02\xedx~\x06\xe0Rmh\t\xf7\f\xc5`Jb\x06P\x9f3@K\x00\x85\b\xcca\xf1`\xa5\xf2doYD\xc3X\x02\x82\\j\xa5\xf1\x81\x99V\x0e\xe8\r\xf8\x9cx\xcb\xc0*J%U\x16\x86\"\x04\xf0\x1a\xd6\x045\x12\x11\x84\x01\xfc\xe2\xb4z@\x9f/a\xce\xc4͍\x16s\xd5Ȭ\xd7D\xce\xef\a\xa3\xfe\xc0\xe7p\xdeJ\x95\x9dB\xf6\x7f\x06\xd5\xc3\xf3\xa0\xc5\x13\x91<\xe6\x14\xd64h*Sh\x14dy\xf3\x1c\x95(\b\xd8@\xc1[TnC\xf6\x04\x8a\xe6\xb5ǃ\xe9#y\xdf\xc8\xeb\xcc\\\xc3\xce5Tĵ\xbd\xed?t\x87.\xed\xfb\xa0E\xfd\x02\xd4F\rΣ\xaf\x1c\xb8*\xcd\x01\x1d\xdc\xd3~q\xa7\x1e\xac\xce,97\x01#,\x9f\x9b\x1c]\x1f\xc7*L\xfc\xb186ږ\xe8\x97 \x95\xff\xee/\xa7\xb1\xd5/ͽ\xf6X\xbc9xr=\xa4\x8f\xc3ሖ\x9d-\xab\xd5\xffE\xe0\xae\x19\xd2[\xad\xfa\xbc\xbe\x19\x8cN\x81\xed\bm\xe2\xed<\xb5\x14B\xed\xa3,\xc9y,MO\xea\xeb\xac/O\xa0\x8f\x03qz\xf72\x86\xb24\xa7\x12\x97\xf5JmH\xbd~\xb8\xfb\xf0\xe7Uo\x18\xc0Xm\xc8z\xd9D\xd7\xf8t\x92Gg\x14\xfa\xcc\xfe7\xe9\xcd\x01\xf0\x06\xf1-\x10\x9cE\xc8E'\x89c$jL\xd1y\xa4\x03Kƒ#\x15\xf3\n\x0f\xa3\x02\xbd\xfe\x85R?\x1f\x88^\x91e1\xe0r]\x15!\"\xed\xc8z\xb0\x94\xeaL\xc9\xff\xb4\xb2\x1d\xfb\"oZ\xa0'\xe7\x03\xd7Va\x01;,*z\x01\xa8\xc4@r\x89\a\xb0\xc4{B\xa5:\xf2\xc2\vn\x88\xe3G\xb6\x1f\xa96z\t\xb9\xf7\xc6-\x17\x8bL\xfa&\xa5\xa6\xba,+%\xfda\x11\xb2\xa3\\W^[\xb7\x10\xb4\xa3b\xe1d\x96\xa0Ms\xe9)\xf5\x95\xa5\x05\x1a\x99\x84\x83\xa8\x90V\xe7\xa5\xf8\xca\xd6I\xd8\xf5\xb6\x1dyd|B\"\xbcB=\x9c\x19A:\xc0ZT<\xe2Q\vM|\x7f\xf7\xf7\xd5#4H\xa2\xa6\xa2R\x8eKG\xbc4\xfaa6\xa5\xdap\x84\xe6\xf76V\x97A&)a\xb4T>\xfcH\vIʃ\xab֥\xf4l\x06\xbfV\xe4<\xabn(\xf66\x94\x1d\x1c\\+\xc3f.\x86\v\xee\x14\xdcbI\xc5-:\xfa̺b\xad\xb8\x84\x95\xf0$mu\x8b\xa9\xe1\xe2Hog\xa2\xa9\x84N\xa8vXݬ\f\xa5\xacY&\x97_\x95\x1b\x99F\x9f\xdah\v8Z\xdfgj:\x04\xf0\xb3\xc6t[\x99\x95\xd7\x163\xfaAG\x99\xc3E\x97̎\x9f7S\x82\x1aĪ\x93P\xe3\x8e\xe0\xe2J(\xea\xa5\x13\"\xf79Y\xea\xbec\xc9h'\xbd\xb6\a\x16\x1cS\xf1\xd0$Nj'\xf0\xa0Ņ\xb3q.\t\x0ediC\x96TJM\xb89W&M\x80\xefT\vc\x88\xa7\xf5\x01gB\xf3$\xe0\xd7\x0fwM\xf8m\x18\xae\xa1\x8f\"\xecEz\xf8\xd9H*D\xc8V\x97\xf7\x9e4\x04~\xee6\x11D\x88A^\x03\x82\x91\x14\xcb\xe06\xfe\x83T\xce\x13\x8az\x90\xdd\xceR=\xf7\"Ɩ\x93 \xf99\xe6\tV\t \xc7:)\xe0\x9f\xab\x7f\xdf/\xfe\xa1\xe39\x00Ӕ\x9c\v\xe5\x00\x95\xa4\xfc\x8b\xb6$\x10\xe4\xa4%\xc1u\x11\xcdKTrC\xce\xcfkid\xddO\xaf~\x9e\xe6\x0f\xe0{m\x81>bi\nz\x012rކ\xcf\xc6j\xa4\x8b\ao%\xc2^\xfa<\x005Z\xd4\a܇#x\xdc\x12\xe8\xfa\b\x15A!\xb74\xcd>\xc0M(4\x8f0\x7fc\xd7\xfa\xfd\x06\xbe\x89\xcer\xc3?o\"\x8c6Qv\xbd\xef\b\xc7\xe7\xe8\xc1[\x99et\xachG\xc6\u0081\x9dCⷠ-\x9fU鎈 \x98\xf5\x14\x03\x12\x89\x11\xbc\x9f^\xfd|\x03\xdf\xf498\xb1\x95T\x82>\xc2+\x90*rc\xb4\xf8v\x0e\x8f\xc1\x0e\x0e\xca\xe3G\xde)͵#\x05Z\x15\x87xA\xd8\x118]\x12\xec\xa9(\x92X\x92\b\xd8\xe3\x01\xf4\xe6\xc4>\x8d\x8a\xd84\x11\fZ\x7f\xb6,\xa9y8\xef4\xe3<\xdd?\x05\xbc\xbb[<\x87\x81\xa6\x9e|z\xee:\xc9ê\xaep\x862\xd9\xe7\xf7\xb9L\xf3\xe6vщ\xb6%\x8a\x18\x8eQ\x1d\xbe\x90\xef0ϕeD\x87\xa4n\x9e%\xa8\x04\xff\xef\xa4\xf3<\xfe\x1cb+\xf9I\xc1\xe5\xfd\xdd\xdb/\xe9Q\x95|N$9Q5\xc7\xe7crD\x95\x94h\x92\xb8\x1a\xbd.e:X\xcd5\xe3\x9d`%m$\xd9\v\xd5\u07fb\xde\xe2\xa6z\x9d\xa8>\xdb5W\x95\x9fN\xa1q\xb9\xf6wo/\xe0X\xb5\v\x1b\fG\x1d\xd6Eg#kЙ\xba\x0eO\xf0\xad\xfbӑ\xab\x0f\xaa\xbf\xbaA\xa6\xad\xcc$߿\xdb\xf0\x11\xae$\nK\xecv$\xbbO\x89\xc6H\x95]\x85\xb5i\xf0\xad\xc8\xf35v\xa2p\xee\xb6fϕ\xd7g\xed\xee\xb2K\xbd\x1f\x00\x01\xb4\x04\xc8gb\rm\xe9\x90\xc4*Π\xe4\x12\x8c\xab\xac\xbaT]\x13\xa01\x05\xd7I\xb12\x9b\xf2\xf5\xa6]\x99j\xb5\x91Ye\xc3\xe5h̔\xaa\x8a\x02\xd7\x05-\xc1\xdbj,\xe8\x8c\xfbt;\xa5\x174\xfe\xbe\xb3\xb4Q\xf7\x85^\xed\xf4\xa9z\x1d\xdc\xf1aHU\xe5\x18J\x02[m$N\x8c\xb3\xb1\x8f\x1c\x9d'nn\xae1\xa9\xe8I\x178\xa8\x1b\x8b\x13\x17\xd9\xda\x11벞G\xf8\xf2\x18\xdcq:;^렖~\xad\xf8\x8e\xd2G\x98L\xdf\xd9\ak\x8c\x16\xb3!i\xdd\xd86\x980\xf5\x9f\xf1ע\xc1|\xfb-\xec\x8f\xd9\xe1L\x99\xe0\x83\x00\x17\x00\xa5h\xdb\xfe\xf7\x0e\x0e.$%\x90\x84d˛-^2\xa6\x80\x03\xe0\xdc\xcf\xc1\x01\xb2X,\xdeц}\x03\xa5\x99\x14W\x846\f\xbe\x1b\x10\xf6/}\xf9\xfco\xfa\x92\xc9\x0f\x9b\x8f\uf799(\xaf\xc8M\xab\x8d\xac\x1fA\xcbV\x15\xf03\xac\x98`\x86I\xf1\xae\x06CKj\xe8\xd5;B\xa8\x10\xd2P\xfbY\xdb?\t)\xa40Jr\x0ej\xb1\x06q\xf9\xdc.a\xd92^\x82B\xe0a\xea\xcd?^~\xfc\xd7\xcb\x7fyG\x88\xa05\\\x11\x05\xdaH\x05\xfar\x03\x1c\x94\xbcd\xf2\x9dn\xa0\xb00\xd7J\xb6\xcd\x15\xe9~pc\xfc|n\xad\x8fn8~\xe1L\x9b\xbf\xf4\xbf\xfe\x95i\x83\xbf4\xbcU\x94w\x93\xe1G\xcdĺ\xe5T\xc5\xcf\xef\bхl\xe0\x8a\xdc\xdbi\x1aZ@\xf9\x8e\x10\xbft\x9cv\xe1W\xbd\xf9\xe8@\x14\x15\xd4ԭ\x87\x10ـ\xb8~\xb8\xfb\xf6OO\x83τ\x94\xa0\v\xc5\x1a\x83\b\xf8\x9fE\xfcN\xc2B\tӄ\x92o\xb8Q\xbb\x1aD<1\x155DA\xa3@\x830\x9a\x98\n\bm\x1a\xce\n\xc4;\x91\xab\x1e\xa40J\x93\x95\x92u\amI\x8b\xe7\xb6!F\x12J\fUk0\xe4/\xed\x12\x94\x00\x03\x9a\x14\xbc\xd5\x06\xd4e\x04\xd4(ـ2,`ٵ\x1e\xef\xf4\xbeNm\xcc6\x8b\v7\x8a\x94\x96\x89\xc0m\xc1\xe3\x13J\x8f>\"W\xc4TLw[\r\xdb#T\x10\xb9\xfc\x1b\x14\xe6r\x0f\xf4\x13(\v\x86\xe8J\xb6\xbc\xb4\xbc\xb7\x01e\x91Uȵ`\xbfG\xd8\xdan\xdcNʩ\x01m\b\x13\x06\x94\xa0\x9cl(o\xe1\x82PQ\xeeA\xae\xe9\x8e(\xb0s\x92V\xf4\xe0\xe1\x00\xbd\xbf\x8e_\x90xb%\xafHeL\xa3\xaf>|X3\x13$\xaa\x90u\xdd\nfv\x1fP8ز5R\xe9\x0f%l\x80\x7f\xd0l\xbd\xa0\xaa\xa8\x98\x81´\n>І-p#\x02\xa5\xea\xb2.\xff.\x12u0\xad\xd9Y\x1e\xd5F1\xb1\xee\xfd\x80\x02q\x04y\xac\xa88\xc6s\xa0\xdc\x16;*\xd8O\x16u\x8f\xb7O_\xfbLɴ'J\x8f7\xc7\xe8c\xb1\xc9\xc4\n\x94\x1b\x87\xacia\x82(\x1bɄ\xc1?\n\xce@\x18\xa2\xdbe͌e\x83\xdfZЖ\xdf\xe5>\xd8\x1b\xd4:d\t\xa4mJj\xa0\xdc\xefp'\xc8\r\xad\x81\xdfP\roL+K\x15\xbd\xb0DȢV_\x97\xeewv\xe8\xed\xfd\x104\xe2\bi\xbd\x16yj\xa0\x18H\x9a\x1d\xc6VA]\xac\xa4\x1a(\x19;d\x88\xa3\xb4\xf0\xdb洈U\x8b\xfb\xbf\xccq\x99m\xff\x1eG[~\xb3+k\x05\xfb\xad\x05T\xa6N\xfc\xe1P_\xa9\x9ej\x1f6\xcbF\xfb\xd4\x1dE\xb4m\xf0\xbd\xe0m\te\xd4\xeb\a\x1b\xcc\xd9\xc6\xed\x01\x144z\x94\t+D\xd6\xfaؽ\x88\xeeWT\xe0T\x01\x11\xd2$\xe01\xe1\xe0\x11&\x10\x03I\x9a`G\x03ubœ[&D\xb4\x9c\xd3%\x87+bT{\x88F7\x96*Ew#\xd8\n\x1e\xc0\x8b\x90\x15\x81xU\xc3Y\x81$\x8f\n\x05\xf1\xf5\xe7E\x15\xd3VQ\x86]>HΊ\xdd\f\xben\x93\x83\x82\xb4z\xd9\xf5;$K\xa8\xe8\x86I\x95\x12\x03\xa9\xb0kϞwjZZ-\xe9\x81\xec۸\xcc\r'\x91UI\xf9<\xc7\x10\x9fm\x9f\xce:\x90\x02\x1dʸ\x15Omo\xbb\x97@\xe0;\x14\xadI,\x93\x90\xb2E\xd3$\x15i\xa46\xe3t\x1fW]\xa4\xef\x1c\xa5~\x9c`\x9a\x83\x9d%Y\xdd5\xaf\x84\x03Q-\x0e\x06\nY\n\xb0ۨ-Q\xbb\xbeJ\xb6\xae\xef(RȒj(\x89\x14\xa33#\xbb\xb4\x1c\xb4\x9f\xabD\xce\xe8\xf4\xd0E\xb7\x7f\xf4x\b\xa7K\xe0D\x03\x87\xc2Hu\x88\xcc\x1c\x94\xba\x96\xa3XGP\x99ЦC\t\xe860\x01\x92XN\xdfV\xac\xa8\x9c\x87a\xd9\x13\xe1\x90R\x82\xb6\xda\x04]\xe6\xdd\xd8&\xc9\x1c\xf9\xfd$Sڣk3b\xb5\x0f/\xa5Q\xba\x96\xa1\x86\xbb\x96Dm\xa7{\x0ft\x8b\xffn\xe4\xe4\xb6\xff\x7f\"6\x18\x93\x13\x98vB\xfe\t\xba\x9f\xd9<=ʷ\x18ၾ$w+\x02ucv\x17\x84\x99\xf0uN\x12(\xe7\xbd9\xfeĴ9\x9e\xe93I\x93#\x13g\"L\x9c\xe2OH\x174\x19O\xdebd\xd3\xe4\xaf\xfdQ\x17\x84\xad\"\xd2\xcb\v\xb2b܀\xda\xc3\xfeI\xaa>P\xe65\x90\x91c\xf5\b\xe6\tLQ\xdd~\xb7.\x8e\xee\x92`\x99x\xd9\x1f\xec|\xe3\x10A\f\xcd\xf3\f\\\x82\xf12SPc\x1cN\xbe\"6\xbb/\xe8T_\xdf\xff|\x18+\xef\xb7\f\xce;\xd8Ȍйv\xbd\xb7\xa3\xfe\xfa|T\x10~A\x1f(\x06U.\xe7rA(y\x86\x9ds]\xa8 \x96>4tΘ^\x01&\x7f\x90Ϟa\x87`\xd2ٜÖ\xcb\r\xae=C\xc2\xf5O\xb5\x01\x0e\xed\x9a|X\xec\xf0d? \"0\x86\xcfe\x03\u05fc($r'閩KB\v\xb8?a\x9bY\xacҟ\xa3\x9f\xfaD\x0e\xf8I;ZZ\x89\xa9\x98\xcfij@\x99\xc9%\xa8k\xdf(ge\x9c\xc8\xc9ȝ\xb8 \xf7\xd2\xd8\x7f0@\xd3\xc8(?K\xd0\xf7\xd2\xe0\x97\xb3`\xd4-\xfc\x9c\xf8t3\xa0\xa0\t\xa7\xe5-\xc2\xfa9?g\xd3,\xb7E\xdc3M\ue10dW\x1cJ2\xa7\xc2\xf4\xae\x9b\xceMT\xb7\x1a\xd3uB\x8a\x05\xda\xcc\xe4L\x1e\xdfR\r\xd0\xfd\xe2I\xfd\x84_\xad\xb1p\xbf\xb8$3\xa7\x05\x94!\xb2\xc4\xec'5\xb0fE\xe6|5\xa85\x90ƪ\xf0<\x8e\xc8T\xac~7DZO\x9e\xf5\xee\xb7\xef\x8b\xe7\x98/XX\x93\xb3\xf0\x10\x8c\xac3p\xe0uw9\xbf\x9f\x85\x95ٌ^\x81\x13f\xbb\x8e$Gǻ\xe6 \xe5\x05\xe8@+\x8e.\xce,uiY\xe2\x11\x1a\xe5\x0fGX\x94#x\xe1X\xd5\xd0[\xbb3\xc15m\xacZ\xf8okiQ\x9a\xfe\x974\x94)}I\xae\xf1\xa4\x8c\xc3\xe07\x9f\x87\xeb\x81ɘ\xb2\xb1SY\xfe\xd9Pnm\xbfU\xe0\x82\x00w\x9e\x80\\\x1d\xf8E\x17d[I\xed\xcc\xf6\x8a\x01\xc7\xf3\x8a\xf7ϰ{\x7fa\xa7\x9f\x9d\xb2\xafd\xde߉\xf7·8P\x18\xd1ᐂ\xef\xc8{\xfc\xed\xfdK\\\xa9LN\xcd\xec6`њ6y\x1c*\x92\xc9\xfa\xae\r8\xa6\x9f\x9b\xef\x92\xf2\xdeɞ\xdam\x16\x8b6R\x9b\xcf\xe9\xbc\xe1\xc8z\x1e\u0088\xa1g\x9cȱ\xcdF\f>\x8f\x16\xf5\xbdu\"W\x06\x94\xcf%:\x1b\x10\xe2\x8f\x17Ff\xa9S\x99\xfebc2\x90\xc6\xfc\xaeE\xf0\f7\xb9\x83\x9b\x9c%\x1e\xe3\xb0Z\xbc\x1c\xe9\xed\xdf~\xef\xe53\xad\xe4ڿ\xfb\x1bym\x87\xba\x90uM\xf7O5\xb3\x96z\xe3F\x06\x9e\xf6\x80\x1c\xf5պEyε\xc8\x1d\x0f\xe1\xf9喙\x8a\tB\x83\xda\x00\xe5\x19\x8a\x92F\xa6rةVQM\x96\x00\"\xa6\xe8\x7f\x04W\xa2f\xe2\x0e' \x1f\xcf\xe0zDt\x9d\xd3ٽ\x894\x89\x94\x8f\x1f\x9c\xc9jdI\xb6\x15(\x180\xc6a\xde\x1d=U!M/eq\x84C\xda\xc8\xf2'MVLi\xd3_\x82&\xadΥ\xf5\x91\xe4\xb3\xeb\xfe\xcaj\x90\xad9'\x82o\xbbi\x06g\xcd5\xfd\xce\xea\xb6&\xb4\x96\xad3\xe6\x86\xd5\xf1TףwK\x99\x89\xc7V\x98\xbf1Ғ\xa0\xe1`\x80,a\x95>\xefM\xb5B\n\xcdJP\xa1J\xc1\x91\x8dI+\x98+\xcax\x9b:%J\xb5c#`q\xab\xd4I\x01\xf0\x177\xb2\x97w\xac\xe4v\x88\xa0̽\xe3A\x1a\x10\xb6\"\xcc\x10\x10\x85\xc58(\xa7\x92q\n\x8f\fD\r\xcb\xd5sy\n\xdc6\x10m\x9d\x87\x80\x05\n$\x13\x93)\xb7~\xf7O\x94\xf1s\x90\xcdr\xde'\xa9\x1e\x81\x96\xa7\xe4h~\xed\r' t\xab\xf0\xf0\xdf\xe9\x8e-\xe3yk\xb6\x94#\x9c\xb6\xa2\xa8\x00\x95\x90\x18\xea\x06\a\x9e\tm\x80\xe6\xf2\x82\xf5\x8aZ!\x98X\xe7\xd1.;\x11\xda5\x87\ua954\x1c\xe8\xf8)d\xd7,\xae\xdf@\x13\xfd\xdaM\xf3BM\xd4\x11\xc1\x1d\x9b#\x1d\xb2)j\x95\x16\xa1\xc6@\xdd8\x91\x93D\xb5\xa2o]Π\x88\x8e\t\xc3\xfd*^3\xbef\x82e\xd0v@\xd7;\xc1L\xdfy\xb4 \xce\xea<\xda\t\xa2;pJ\x86\xedn\x00\xc0\nh\x88Cp\xed\x91k\x8ep$\x97@hYB\xe9r\x97\xd6\x15\xf1a\x89+|\x1b)nH\xee\xeexO0\x8b\xb2\xa1\r\x82N\xccê\r,Z\xf1,\xe4V,0\x18\xd7G\xeb\x90\x13\xb3T/\x9dޜ\xac\x8c\xe6\xf5K\xbe\x9a\x9e\xd3BC~\xcd\xe7\xa9\xe0?\x9dA\xcbd\xf3\xcdQ\t\x8f).\x98\xd3k\xae\x00{\xe4\xc7\xd9UL\xcd?1\xd8\x1fJ߸b\xe9\x17\x95\xc5ݥA\xf5\x9c\xc2m\x05\xa6\x02\x15J\xb3\x17X\x92^N\x9e\x90v\xc1K\xac\x93\xb3L\x15\\dW\xfe\xb9W9\x87\xd1M\xcb\xf9\x85\xe5m\xda\xf2d8l$\x8a\xd8!geՏ\xa5=\x86\x9c\xea\x8bl<\xf6+-\x86\xf5\x85\xb1\n\"\x14\x18\xca0\xb3\xa7qj\xbfXX\xda;\xdf\x1f\x96S`\xfe/,\xff\x0f/=̨\x94\xc8Gcn\x95fDb\x02V\x82\xc1zh\xec\xea+|?_\xe8\xfbc\xe1\xd4@\xfd\xa5\xf1\x123\xea\xc2f\xa05\x01g\xaf\xde\x04\xadA\xab\x9d+\x10\xed\x80\xcf\x19\xda\xf1ׅ\xbb\x05\x11\xc0\xa4\xf8\xf5k\x05A|}\xf5>\xd3\xe4\x9fI%\xdbDU\xdf\x04\xcaf\xaa;\xe67<(\xf4\xf0\a\n`\xe8\xe6\xe3\xe5\xf0\x17#}\xd9\af\xd1\x12\x800(\xea2\xb3L\x94l\xc3ʖ\xf2 \xb5\xdd\x1d\x02\xc7@\x1d\x9f%\xa0IE\x04\xe3\x8e\x01\xc3\xf8\x01Ñ/\x8d;\x969Z\xc5M\xfb\xa2y\xd5!'ׄ\fk>F\xac\xe1\xb1\xc7\x17\xafR\x05\xfb\x87\xd4z\x1c_\xe1\x91\x13I\xccTs\x9cPÑY,\xf6\xe2\xf3\x96\x9c*\x8dcb\xee\xb3Ud\xbc~\x1dF\x16~\xe6k.\x8e\xc1\xce\xd9\xeb+ް\xaa\xe2mj)2+(^\xaf\x142/\xfa<\xa9\x14`>`\x19\xaf\x82\x98\xad}xQ@sҖfk\x1a\x8e\xa9d\x98\xa5N\x9e\x98\xbdY\xad\u009bU(\xbcm]\xc2$\x17M\xfexL\xe5A\x8c\x93~\xa1M\xc3\xc4\xfa\x90)rYg\x92m\xe6Y\xe6~o!\x03\x9e\xe9\x873]t8\x12\xfa\xba\xeb҉H2\xa4-\x990\xf2\x92\\\x8b\x9d\x87\x9b\x80\xd3\v\x1f\x854\a\x17\xd9첶\x8c\xf3\xfem-\x04;\r\xcaߙԴv\xab\x1a\xf3\xf6\x93t\x95j\xe0\x94\x9f\x148~ك\xd1ώ\xbe\xa5\xe7_\xb7ܰ\x86\x83\xf5\xe86\xacL\xde!3\x15\xec\"\x92\xff&\xf1\x86\xd4r\x87\x90\xbe]\xf4\x9eQ\xec\x9eq\x926\xbf\xc9\x13\xb6\x97Q\xcc~\\\x11{\x06\xcdrE\xf1\r\x8b\xd5߰H\xfd\xad\x8b\xd3g8k\xe6\xe7\xe3\x8a\xd0O>\x81\tG\xfd\xf7\xb2\x84\a\xa9\xcc\\p\xf2\xb0\xdf?q\x92\xda\v\xd8$/\x89\b]\x13\xbb\xc4\x10Ç\x17\xa7m*}\xe8\x19\xdc\xe9_di\xd76w\xc6\xf2\xb8\xd7\xfd\xe0\xae\xf2\n\x14\b\xf7\xcc\xc7\x7f>}\xb9\x8f\xf0S>\xaf\xf7\x8c\xf7\x9e\x97p\x1eL\xe9\x91\xe3\x8f\xe6|1\x93\xc3\x16\xfa\x00\xaf|.B\x1b\xf6\x1f\xf8\xaa\xdb\v\xd2A\xd7\x0fw\b#\xf8i\xf8L\\\xac\xa2\x88'\x96K\xb0\x16+\xa2jT,\xeeV\x03\x88Ê\xdf\xfe3JP\xba'\xb3\x82\xc5d\xa1\xc6\xcb\n\xdeÝ[\xc7\xd8,\x9f\xac\xd3(vD:\x8e\xac\x98*\x17\rUf\x87l\xa3/\x06k\bff*\x9d3\xaaX\x0f\x9f\x01K\xa27\xbc\xfe\x85g\x91\xbbfxڻ\x8f\xbbS\xd61~\xffd\xf6\xe6\xc9+\xaec\xdcb/\x10S\x89\xcf\xc9\x02\x93WK\x93yM\xf4\xf0\xed\xa4\xb4\xcbc\x1c=\xad\xe7l\x14\x1dRM\t0v<\xaa:-h\xa3\xabēK/\xd3u\xf8\x1a\x99\xa1\xa6}\xc9&\x1d\x80\xc1>YQ\xf5\xb4\xd5\x16\x82>\v\xdbFi\xc5a)\xddnm\xb3\xab{a\xfc\xa2\x97)x\x9b#\xe1\xcc\xe7\\N~\xc8šgD\xfd`\xf6˪\xb6CL\x9dp\x18<\xeb\xdae\x14\x19O;\xb1\x99π\xe4\x19\x8c\x13\x9e\xfe@|\xe5\xe2\x8a$_\x04\xc9|\xf5\xe3\x0fE\xf4\x84V\xd3E\x05e\xcb\xe1\xd47\xff\x9ez\xe3\xe7_\xfd\v\xb3e\xbc\xfbg\x91\xdd3\xd0\xd6g\x1e\xbe/\xe8)\xe1!\xf7)9\xe6\xf0ap\xe0\x9e\x17+\xdcK\x94E\x01Z\xafZ\x1e\xaa\x94\n\x05\xd4@\x19\xba3\x1dW|T\x9dM\xdbpIKP7R\xacX\xe2\x84d\x80\xd6\xff\x1at\xde\xe3\xd9\x02?\xb6\xaa{\xdaq\xf2Y\xbc\x17i\xae\x86*\xca9\xf0O\x8c\x83\xfeYn\x85]W\x86@>\xa4\xc6\xf5\xeee\x15\xad\xb2f}GD[/\xad\x93\vƌ\a\x8b+\xa9\xa6+\xa4\x1dޙ0\xb0\x86T|\xbdU\xcc\xc0SC\x95\x06\\Q\xc6\x0e~\xdd\x1b\xe2\xa2\xcf\x15\xa7kW\nW\xb2\x82\x1a\x88\x06\x18g\x18[>\x8e\xd7\b\x8b\xef\xb02I\x8e$\xbd\xb2\x85z\xecJƨX\x8f=/\x9a0\xd5\xc9\aF\x9dE.hc\xf0\x02\f\xd2\x11\x89h<\f|\xb4w\xef\x8d\xd1\x01\xd8qN\xf3e̾`N\x1bZ'\xa2\x84y\xbdss\b\x06\x9f\x05Ve\xaf\xee\xae\xff\xc0b,\xb0#[\xaac1u\xd2\xf7\xee`;0\xe8\xaa[\xd0P\x12\u0600 V\x14)\xe3PNq\xeaWL$\xab\r\xa8\x9ft\x84\x83\x95\x80\x96ş\fU&.\xfdЏYIUSsEJj`aG\x9f溥\x9fIU\xea\xc4\xe3@\xbc\xd9\xe6ţ\b\xd7n\xac\xf5s\xf7\xd1jК\xaeC\x10\xba\x05\x05d\r\xc2\xe2=\xe6\x16\x93\x1eS\xb8\xd2\xe7\x8dE,-\xb5(\xa4\x85i\xa9\x9f\xc0\xb9p\xf1\xf44\xbcO\x8cQ\xeczTE\xa7U\x85\xbf<\xf8\bT\xef?w}\x80\x8bO\xfd\xbe>I\xecv\xec\xceF\xa8+\xf0\xc4\a\x8f\r\x8b\x91uJ\xa6\x8dę\x8f2'\x95\x94\xcfYn\xf6\xe7رK'1\xe1X\t\xafL.ekz~\x8eGxb\x99\xf8\xfc\xe7+\xdb\x17\x84y\xed.P\x8d\xe5V\xf3<\xbd\xcf\x03H1\xbc\x95\x86\xf2`d,_\xc6\x0e\xd5\xc4\x03\x02O\xe1\xf1d\xcew\x17\xfb\x90\xf7^e\xef`W\xddS\x9e^\x13t\xd7\xc7\xc7Ҫ>\xeb\x97\x04\x12_\x01\xed|\x92\xb17\x17\xe7\xec\x1fB\xfd\x84\x8b\xca\xc0\xf1\xe7\xae\xf7\x18\x1e\xdd2\x9d\xc3\f\"\x1di\x12\f>L\x15%ㄥOx\xa9ME\xf5\x9c{\xfa`\xfbD\xb7\xa3g\xae\xa2\x13\xfa8\"\x95\xe9{\xae\vr\x0f\xdb\xc4W\x87,<\xfdB\xa9Jt\xb9\x13\x0fJ\xae\x15\xe8C\xa6[\xe0}F&֟\xa4z\xe0횉/\xe3\x95\xdfS\x9d\x1f\xa82\xcc2\xad[Ob\xecM\xb0q\x89\xdf\xe6G\x8f\xff\xc0\x04\xe5\xec\xf7\x94.\xef\xff87Ä\xbek<\xf2N\xb1P\x01\xf1s\n\xd0k\xe8\x9ft\xcf\xfc\x84y/ɽL\x8a\xb1? fC\xa0L\x93%h\xb3\x80\xd5J*\xe3\xf2\xf7\x8b\x05a\xab\xe0 Y\r\x81q\xa2{͞\xb0T\xe2=\x1e\xbd\x05\x87e\xe5S\x89\n\xad\x0e\x86\x9c5ݹ\x8c$-\n\x1b\x13\xc0\amh*6y\x91\x9e\xc6P\xd5\xcbJ\x8e\n\xb9\xeb\xf7\x8f9\xbe\xa8>\x10\x9cC\x1d^gw\x06\x9d\x8f\x9di\r^\xcb \xdab\xef\x14eB\x9c\x1a\xbb\x1b\x0f\xbb\xf3L\xcd\xd7\beL=\xfa\xfd\r\x1e\xe2\xf6\a\xac\xbe\x93%[QQ\xb1\x1e\xbd\xd0V)ٮ\xab\xc0\x9bc\x0e\x11)[\x8c\x9c\x1bT\x05:\xfc\xc7!\xa6U\xa2wh\xe7k,ƴt\\\uee0f\xf2\x02E\xad\xba\x8b-\x9d\xaa\x9a\xb0\xf9\xd9Y\xc2\x11\x88\xb3\xb6?\x01\x91\xea\x9d(&\xaf\xe0\xf8@\x9bM\xdc՝\xc2P\x12\tQ\x1b\xbf\x1a\x12\"\xc41$\xf4}\x89.\xe2\xf9a02棜\x88\x8ei'\x06\xb78\rj~\xd3}'h\xe8\xee\x1c\x87\x0e=\b\xfeNJ\xbb\r \x1c\x13\xf9\xe2\xdc\xe9\xb8\xf7ǍX7\xd1ۺ=9v\xfd\xb6\ac\xef\n\xa4\x8db\xbbiB\xbc\xf9\xf7l\x95\x92\x17\xf7\xbf3-9\xfc\xc3\xc1\xafo|\x95qK\x95`b}\x12F~\xf5c\x13\xf1\xbc\a{Έ>\xac\xfc\xd5b\xfa\xa4Y:\xf8\x88\f^\xf6\xf0\xecg\xf2_\xfe/\x00\x00\xff\xffP\a\xb5\x16Cm\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xec=]s\x1c)\x92\xef\xfa\x15\x84\xeea?B\xdd^\xc7}ą\xde|\xb2gO\xb1\x1e[ai\xf4\xbctU\xb6\x9aQ\x15\xd4\x00\xd5r\xdf\xde\xfe\xf7\x8dL\xa0\xbe\xba\xe8\xa2Z-ygǼت\x86$\xc9L\xf2\x03\x12X,\x16g\xbc\x12\xf7\xa0\x8dP\xf2\x92\xf1J\xc0W\v\x12\xff2\xcb\xc7\xff6K\xa1\xdelߞ=\n\x99_\xb2\xab\xdaXU~\x01\xa3j\x9d\xc1{X\v)\xacP\xf2\xac\x04\xcbsn\xf9\xe5\x19c\\Je9~6\xf8'c\x99\x92V\xab\xa2\x00\xbdx\x00\xb9|\xacW\xb0\xaaE\x91\x83&\xe0\xa1\xebퟖo\xffk\xf9\x9fg\x8cI^\xc2%3\xd9\x06\xf2\xba\x00\xb3\xdcB\x01Z-\x85:3\x15d\b\xf4A\xab\xba\xbad\xed\x0f\xae\x91\xef\xd0!{\xeb\xdbӧB\x18\xfb\x97\xde\xe7\x8f\xc2X\xfa\xa9*j͋N\x7f\xf4\xd5\b\xf9P\x17\\\xb7\xdf\xcf\x183\x99\xaa\xe0\x92}®*\x9eA~Ƙǟ\xba^0\x9e\xe7D\x11^\xdch!-\xe8+U\xd4e\xa0Ă\xe5`2-*K#\xbe\xb5\xdcֆ\xa95\xb3\x1b\xe8\xf6\x83\xe5g\xa3\xe4\r\xb7\x9bK\xb64ToYm\xb8\t\xbf:\x129\x00\xfe\x93\xdd!n\xc6j!\x1f\xc6z{Ǯ\xb4\x92\f\xbeV\x1a\f\xa2\xccrb\xa0|`O\x1b\x90\xcc*\xa6kI\xa8\xfc\x0f\xcf\x1e\xebj\x04\x91\n\xb2\xe5\x00O\x8fI\xff\xe3\x14.w\x1b`\x057\x96YQ\x02\xe3\xbeC\xf6\xc4\r\xe1\xb0V\x9aٍ0\xd34A =l\x1d:\x1f\x87\x9f\x1dB9\xb7\xe0\xd1\xe9\x80\n»\xcc4\x90\xdcމ\x12\x8c\xe5e\x1f\xe6\xbb\aH\x00F$\xaaxmH8\xda\xd67\xddO\x0e\xc0J\xa9\x02\xb8\x80vX4\xb6\nu%\xa0\x80\xe6\f\xddN\x8d\x16FH\xb6\xae\xd1#]2\xd4\x12Q\x19\x11\xd2X\xe0\x11a>\x01\xef\xe0kV\xd49\xe4WEm,\xe8\xdbLU\x90\x87E\xa6Q͜\xca\xc3\x0f\a!\xfb\xf8\xa5\x10\x19 \x1f2WiA\x8b<1\xd1nC\x99]\x05n\xcd\tY\xed\x87\xd0\xc6(\x93\xbaŀņ\xe7\x7f<\xbf \t\xe8\xf7\xde\xef\xc70\xae\xa1!\xd3,\xddL\x16\x7f\xbc\x85\xb0PF\xa8;\xa9\xa3f\xf0\x9dk\xcdw\a\xb8\xde,\xa6\xbd\x00\xdfc\xb0\a\x9c\x97\xa1\xda7\xe2\xfd\xb0\xff\xdf\"\xf7O\xcboC\x8b\xce\\H\xe4s!\x8c\xed\xb1ٸU,$\xebX\b\xe9\t$\x1dLT\x93S\\\xfd'!\xe6I\xe7Nl\xb24\xb2\xe9'\xc0\xbf\x14%7J=\xa6P\xef\x7f\xb1^\xbb\x84\xc52\xda\x18a+\xd8\xf0\xadP\xda\f\x97I\xe1+d\xb5\x8dj\x16nY.\xd6k\xd0\b\x8b\x96\xf9\x9b]\x81C\xc4:\x1c\xbe\xb0\x8eʊV\x18\x8c\xabe:\xb2\x94\xa8\x11\x1b\n\x05\xa8Q\xa8\xce\xc1\xc1Ђ\x1c\x88\\lE^\xf3\x82|\t.37>\xde\xe0\x17\xd3j\x13\x02\xb1\x87\x7fT\xaa]q\x0eM\x18$2\xb1\xb7\xea\xa5$\xa0\x8f_bl\xb4_5N\x89\xb0\x94p\xb0od\xa6\xae\v0\xbe\xbb\x9c\xdc\xe4V']\xb4\xccrk\f\x05_A\xc1\f\x14\x90Y\xa5\xe3\x14J\x91\x03WR\x95n\x84\xb8#Z\xb6\x1fm\xb5\x83\x99\x00\xcb(\xc4݈l\xe3\xdcW\x144\x82\xc5r\x05\x86VExU\x15\x11\xd3ՖI\xe1\xf0\x9dM鍶$h\x90!ܘ.iK\xa2~n\xcb(\xd9۹٧\xfa\xf8:\xff(\xbe\xbf%\xa2\a\xabs\xa4\xb0Oh\x12F\xfb\x05\xc9\xf3!Jz\xa4\xb8\x00\xb3\xec\xac\xce\t\x1b\xbe\xa60\xb4\xe7?\xeem\xa5\xec\x11\xe5\xd7Ż\xe3&\xcc\f\xd6MΩ\x97e\\\xd3Ϳ\b\xdf\xc8d\xddz\x8b5\x8bg\x1f\xbb-/hW\xc03$\xbf`kQX \xa7j\nQ6\x83s\xa7$P\xaa\x05f\xb4Il\xb3͇f\xef(\xa1ŀVC\x00\xceA\x0fQ\x0e\xf1 \x01$k\\\v\xda4\x15\x1aJڌ\xa5H\xb2\xfb\x85\\\xc1w\x9f\xde\xc7c\xcfnI\x94ԽA%LZW\xde\r\x1c\xa3.\xae>T\t\xbf\x90\xbf\xd6\x04\x82n\x13\xfe\x82q\xf6\b;\xe7bqɐo)\x8b\xff|\xf8*\fv,s\xf6^\x81\xf9\xa4,}yQ*\xbbA\xbc\x06\x8d]O4A\xa5\xb3$H\xc4n\U00088ce5(\xa8\r?\x84a\xd7\x12C2G\xa2\x19\xddQ\xae\x90\xeb\xd2uVֆ\xb6Z\xa5\x92\v\xb7,6֛\xe7\x81\xd2=\x16\x9c\xa4c\xdf\xe9\x1d\x1a#\xf7\x8b\xcbZ*x\x06yآ\xa3t\x1an\xe1Ad3\xfa,A?\x00\xab\xd0,\xa4K\xcb\fE\xedG6_\xbc\xd2=\x87n\xf9\xbax\xacW\xa0%X0\v4k\v\x0fŪ2\x91.\xde&\x8c䜌\x95\x05\xce\xf5ĚAZ\x92\xaaG2r\x0eWO%\xd63\xc9D^\x04\xb9]IR\xd0Ml\x9dg\xbdf\xca\xcd1*\xa63\x16\xe7\x02\x94\x9c\xb6\xd6\xfe\x86\x96\x9ef\xe3\xdfYŅ6K\xf6\x8e2{\v\xe8\xfd\xe6\x17&;`\x12\xbb\xadh\x95\xfd\x97Zly\x81\xfe\a\x1a\bɠpވZ\xef\xf9j\x17\xeci\xa3\x8cs\x1b\x9aM\xbb\xf3Gع\x1d\xe5\xa4n\xbb\n\xeb\xfcZ\x9e;_fO\xf14\x8e\x8f\x92Ŏ\x9d\xd3o\xe7\xcfu\xeffH\xf4\x8c\xaa=Q.y\x95.ɔ7;'\xd0\xc0`=8DظI \xc5\x00a\x8a\x02ɢ\\)\x13I\x16\x89\xa0\x95 \xe87\xcaX\xb7\x0e\xd9\xf3\xf7G\x17*UX\x9cd|mA3c\x95\x0e)\x99\xa8\xf8S\x96\xe2\xbb\xe5n\x03\x06\xfc>\x94_\xf4t\x801\x8a=ou\x83\xb3*\xe7n/\x8c:\xe2\x19yOԶ\xd2*\x03\x13͋hK\xa2m\xeaQp\x9f\x0eͺ.w\xd1\xdf:Ik\xa7,J\x872ϑG\xd2\x1d\x11\x19}\xf8\xdaY\xa2F\xed\x82\x7f\xa7H\xeb182:\xafQ\x96|\x98\x0e\x9c\x8c\xee\x95k\x1d\xe6\x98\a\xe6\xc2-\xfdP\x93Ι\xe3u4\xa2\xfc\xcf\xe6ڔB^SG\xec\xed\v\xbaC^\x8b\xc7ң\xc6\xca\xf1N\xfaU\xe8\xac\xe5^\xf3\xc1\xe7\xd4)\xda\xf8\xd1\xd0c\xee\xfe\x9e\by\xd7R\xd9\xce2\xceL'\xbaR\xf9\xef\f[\vml\x17\rs \xb1j\x14\xd4\x11\xa1\xa7\xfc\xa0\xf5ё\xe7g\u05fa\xb3\xa0\xb8QO>qzN\xbc\x1dH\xba\xe1[\xf0\x99\xab 3UKZ\nC=\x80\xdd̀\xe8X\xe3\xac@\xa2\xbd\xeb4\x96u\x99N\x90\x05I\x92\x90\x93\xebf\xdd&?p\x91\xb6nŎc\xab=\x94\xc39V\x8e\x9fG!\xc1\xb3\x9bN_\U000af8acK\xc6K\xe4!\xb9\x1d\xa2\x84&\xa3ޱ\xbbI\xfb\xc4\x16d\xb4\xac\xc2YV\x15`\xc1\xa7m\xce\xc0#S҈\x1c\x1a\xd3\xefE@I\xc6ٚ\x8b\xa2\xd63\xb4\xeal\x92\xcf\r¼69}d\x95\x8eȂH\x94\xb8\xce>\xc3\v\x9e\xd6\xf8\x95\x9e\xe7Ǧ8\x8c\x1a\xe6\xfb\x8b\x95\x16\xca\x1d\x068\xbd\xcb\xe8ӎ\xb9\xdc}\xf7\x19\xbf\xfb\x8c\xdf}\xc69\x1d}\xf7\x19'\xcaw\x9f\xf1\xbb\xcfx\xb8|\xf7\x19S\xcaw\x9fq&\"\xdf\xcagL\xc1pAk\x9c\a*$a\x95\x98\n1\x85\xf6D_>\xe9ǟ\xd58I.\xf3\xf58ȑC<\x91\xe3\x171\xaf\xa35^Mr3\xce\xc00w\xdc)\xca\x04\x87\xf9\x04\xa7g\x02\x02\xa7?=s}\x10\xf2\tO\xcf\xf8!\xa4E\x18G\x9d\x9d\tD\x9a\x7fz\xe2\xc2'\x11\x95\xc0\xc3V\x8aK\xff\x88\x8d1&I\tx|\xe3\xe4\xf7\xbd\x8c\xc9\x17\x90\xa5W9\x913K\x9eFY\x7f\xfe\xc7\xf3_\a\x8bN˔(\x1b\xf6i\xeb\xd4xL?b,\xdfM\x8d\xecg\xa9\xfez\xa6\xc2Ie?\xf5DMC\xe4\b\xbc\xbeX\x0f\xa8\xfck\xd27\x16\xcaϕ\xb7\x96'8a\x7f=\x02/\xe9\x8c=7;\x99m\xb4\x92\xaa6~M\ba\xbd\xcbܽ\x03\x01dL\xd8G5\xc8\x7f\xb0\x8d\xaa#\xa76&H\x9b\x90E\x9bF\x90^R\xadO\x8c\x00˷o\x97\xfd_\xac\xf2)\xb6\xecI\xd8M\x04\x18\xddG\xc1\xf3\x1c\xe3\x82\u0381\x1e\xaf\a\xc2UIC\xa1\x8c\x00S\x9aIQ8\x89\r\x10z\xf2\xca>Wnu\xf0h\xbfiz\r+=\x11wn\xfam\x93-9\xed\xbe?#\xe9\xf6\xa4G\xa3\xbeYZ\xedqɴ\xa9+\x94\t\x89\xb3\xe9\xe9\xb2)lu%=I69BNM\x88\x9d\xbb\x02\xf1\xa2ɯ/\x93\xf2\x9aL\xb3\xb4\xf4ֹ\x14{\x95T\xd6WN`}\xbd\xb4\xd5\x19ɪ\xa7?\xf5\x92\xbe\x96~tveڲ\xcc\xe1\x84Ӥ4Ӥ\xa5\x9b\x94\x01\x1f5Ԥ\xf4ѹI\xa3I\x9cL\x9f\xae\xaf\x9a\x16\xfa\xaaɠ\xaf\x9f\x02:)m\x93\x15\xe6&y\x8e_r\x18ʴ\x03P|\v\xe1|.\x99\x94\xee\xb9\xe6ϊ;?\x0f`\xa1\xb0\x047\xf5\x15〲.\xac\xa8\x8a\xf6>\xb6X\xc0\xb9\x81]sY\xd1ϊ\x8e\xc8\xfb\x9b\xba>\x7fi$~9\x88j\xb8aOP\x14\x8c\xc7\xe6\xe6\x1e\x152w\x0fh\xa6\x16\x80\xb6\x11g\xb9\xbf\x8c\xc9_\x1ez\xe1\xa6\v\xdd\x06@\x16\xb6\x8c-\xf5qy\xf8\xa6\xaf\x83\x06,U\x8f\xedy\xe6.ޠo\xbfԠw\x8c\xee\x1dk|\xb3\xf6P\xa9\x9f\xe8\x06\x03Ӡ~\xbc:<\xb4g\xb2\x17\xe0\xb4ꁽ\x93\xce#\x18\xe2DmP\xef\xb4\x01\x1d*U\x8cӢ\xfdD@H\xd5@\x884Mq\xfe眲|\x89\xf0\xee\x14\x01^\x92\a4\xcf{\xfd\x86\xa7'\x8f=5\x99\x9e\x8c\x92tJ\xf2%½9\x01\xdf,\x7f5\xfd\x14\xe4\xfc\x8d\xe7\x17>\xf5\xf8R\xa7\x1dgP/\xf5t\xe3|ڽ\xd2i\xc6W?\xc5\xf8\x9a\xa7\x17g\x9dZLNϚ\x95q0'\xb5\xea\x19\xc7\xed\xd2r\t\xa6O!&\x9e>L\xcc4H\x1b\xfc\x91\xc3N<]8\xffTa\"\x7f\xe7L\xe9W>=\xf8ʧ\x06\xbf\xc5i\xc1\x04\tL\xa82\xffT\u0cf7\xa4\x94\xceAOn\xfb͑\xdaIyM\x8d\xe5\xfa\x88\r\xf6\xb5\xc2m\xb2X\xab\x17\x03\x90Y\xf2\x17\xf9ӣ\r\x87\xb6\xc1Q2;\x1eQo_\xb2u\xd7\xfa\x0e\xb1\x7f\xcd\xc1m]\x1a\xa88\x1a\x00\n\xdc(5+\xea*|\xe0\xd9f\xd0Æ\x1b\xb6V\xba䖝7\x9b\xc5o\\\a\xf8\xf7\xf9\x92\xb1\x1fT\x93\xabӽ/͈\xb2*v\x18\x89\xb1\xf3n\x83\xe7IIT:C\xcf7\xaa\x10Y\xc4\xe7\x1c\xbdW\xcf5ػl\x88n\xfe\xcb:\xd9\"\xb1\xc0\a\x9b\x8bp\xebb\xffJfw\x9f\xfb\x91k%\xbc\x12\x7f\xa6'\x95N\xb0\xea\xf6\xee\xe6\x9a`\x051\xa2\xb7\x9a\x9a\x04ņ\xe5+@\x97\xa1\x1d\xfb!}r\xbd\xeeA\xed\xe7\bw\x1f\xab\x80ܽL\x12\xdc\x16\xaf\x9a3\x85Z\xeb\xe6\xda\xe1r\xa8'\x94/.wL\xf9\xa7'\x84\xce\x17\x15\xd7v璉.zx\x04\xbb>\xb5jv\xd0Z\xed\xbf\xbc\xd2-=\xb2\x87GWh'{W\xf5\x93\a\x86\xf4|\x0eN\x87OUO\x9e\xa7~\x01\x9c\x0e\xbbP\v\xa2b\xe4\xa7h\x06\xe4\xc9W,\x8d\xbf\xa1\xffG\xb5\x85\xf7ѕ\xcb\xfe\xeb+\x83&#\xa9\x89\x01*]2\x1f\xa1`\x9b\x8fHw|?O\xed\xc5s\r\x03*\xfe\x8e\xf0\xe7,N\xde\xf6A\x8d?HB7\xa8\x87Nc^\x15=\xf5\xb4c7\xf7\x14\xb76\xaa\xd4O}\x1f\xb7\x86\xe5ɐ`\x10\x81%\xe4\xc17ZNEF\xab4\x7f\x80\x8fʽ\xad\x93\"&\xfd\x16\xbd\x97\x97\xbc\xe7\x16\xf2\xb5\xfd$\x8c)z?\xb6!\xc0\xf6|\xc6\xdeE\xff\x88\xed\x91O\x19X[\xba\x91ғ&\xef\xfd\xeb$\xa8\x8f\r \v\x02\x05\x1c\xb4\x15\xfew\xa3\x9e\xe8\x02\xfc\xf8\x1asx@\xa4\xf3\x86\x19\xd0A\x11J\xe1=j\x98uU(\x9e\x83\xbe\xa2GT\x12F\xfcS\xaf\xc1\xc0\x1d\xe8?\xc5\xe2\xedfd<\xa1\xe7\x17̒A\x8f\xae(\xa0\xf8A\x14`\x1c≦\xe1f\xbfec)\xear\xe5<\xd55\xfe\xd8tr\xc02\xbb\xa1\xd2\x06C\x05\x1a\xfdD\xb7\x15Q\x9b \xf9\x87\x89\xc1\x1a>\ni\xe1\x01\xc6c\xe8\t\x9b\xe0\xdeh \a (0\x8a\xf8\xfe\x12[y\xec\x11\xe4>\xdez \x03\xcdbdL\x8e\x95w\xabn\xee\xaf\f\xabeN\x1b\x00\xf7\x7f\xbe=J~\xb7\xbd\xf7e\x82NHQ\xef\xf7\xe3-;!BG;\x91O\x1fW\xe21X\xdc\x18\x95\t\x8a*\x9e\x84\xf5\xd79\xbe\xdc\x1d\xe2\x87\x02\xc4\x03\xd2Q\x1b\xf8\xfc$A\x7f\t\x16\xc8\\\xcbػ-\xd3\xda\xef\xa7=h\xd1\xf7Z\xac¾G`\f\x000\x15\xf6\xb9\x8c{\t(l\xaf\t\xd3H\x1c\xc4>\x01\xdcyG\xc8ik\x85\xb4\xe3\x9c!n\x9bVt\xd8tDCN\x8b\xed\xfd\x00\xc6 \x93\x9d\x1e}j\xaa\xb8Ӧ\x86\xfd^\x8cy\xa3\xb4c\x96\xe1@\xff\xb0\xf7kT\x83\x1f\xd4\xde1\xcd=\xaaF\xf6>\xd2CxyGr\xbc\x97\xde\xfdR\xaf\xda\a\x15\xd8\xdf\xfe~\xf6\x8f\x00\x00\x00\xff\xff)\x00\x87w>{\x00\x00"), diff --git a/pkg/apis/velero/v1/labels_annotations.go b/pkg/apis/velero/v1/labels_annotations.go index 78d231ba9..0f0f9438a 100644 --- a/pkg/apis/velero/v1/labels_annotations.go +++ b/pkg/apis/velero/v1/labels_annotations.go @@ -101,6 +101,9 @@ const ( // ExcludeFromBackupLabel is the label to exclude k8s resource from backup, // even if the resource contains a matching selector label. ExcludeFromBackupLabel = "velero.io/exclude-from-backup" + + // PVBLabel is the label key used to identify the pvb for pvb pod + PVBLabel = "velero.io/pod-volume-backup" ) type AsyncOperationIDPrefix string diff --git a/pkg/apis/velero/v1/pod_volume_backup_types.go b/pkg/apis/velero/v1/pod_volume_backup_types.go index ced436a82..546616c5a 100644 --- a/pkg/apis/velero/v1/pod_volume_backup_types.go +++ b/pkg/apis/velero/v1/pod_volume_backup_types.go @@ -64,12 +64,16 @@ type PodVolumeBackupSpec struct { } // PodVolumeBackupPhase represents the lifecycle phase of a PodVolumeBackup. -// +kubebuilder:validation:Enum=New;InProgress;Completed;Failed +// +kubebuilder:validation:Enum=New;Accepted;Prepared;InProgress;Canceling;Canceled;Completed;Failed type PodVolumeBackupPhase string const ( PodVolumeBackupPhaseNew PodVolumeBackupPhase = "New" + PodVolumeBackupPhaseAccepted PodVolumeBackupPhase = "Accepted" + PodVolumeBackupPhasePrepared PodVolumeBackupPhase = "Prepared" PodVolumeBackupPhaseInProgress PodVolumeBackupPhase = "InProgress" + PodVolumeBackupPhaseCanceling PodVolumeBackupPhase = "Canceling" + PodVolumeBackupPhaseCanceled PodVolumeBackupPhase = "Canceled" PodVolumeBackupPhaseCompleted PodVolumeBackupPhase = "Completed" PodVolumeBackupPhaseFailed PodVolumeBackupPhase = "Failed" ) @@ -113,20 +117,27 @@ type PodVolumeBackupStatus struct { // about the backup operation. // +optional Progress shared.DataMoveOperationProgress `json:"progress,omitempty"` + + // AcceptedTimestamp records the time the pod volume backup is to be prepared. + // The server's time is used for AcceptedTimestamp + // +optional + // +nullable + AcceptedTimestamp *metav1.Time `json:"acceptedTimestamp,omitempty"` } // TODO(2.0) After converting all resources to use the runttime-controller client, // the genclient and k8s:deepcopy markers will no longer be needed and should be removed. // +genclient // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object -// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.phase",description="Pod Volume Backup status such as New/InProgress" -// +kubebuilder:printcolumn:name="Created",type="date",JSONPath=".status.startTimestamp",description="Time when this backup was started" -// +kubebuilder:printcolumn:name="Namespace",type="string",JSONPath=".spec.pod.namespace",description="Namespace of the pod containing the volume to be backed up" -// +kubebuilder:printcolumn:name="Pod",type="string",JSONPath=".spec.pod.name",description="Name of the pod containing the volume to be backed up" -// +kubebuilder:printcolumn:name="Volume",type="string",JSONPath=".spec.volume",description="Name of the volume to be backed up" -// +kubebuilder:printcolumn:name="Uploader Type",type="string",JSONPath=".spec.uploaderType",description="The type of the uploader to handle data transfer" +// +kubebuilder:storageversion +// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.phase",description="PodVolumeBackup status such as New/InProgress" +// +kubebuilder:printcolumn:name="Started",type="date",JSONPath=".status.startTimestamp",description="Time duration since this PodVolumeBackup was started" +// +kubebuilder:printcolumn:name="Bytes Done",type="integer",format="int64",JSONPath=".status.progress.bytesDone",description="Completed bytes" +// +kubebuilder:printcolumn:name="Total Bytes",type="integer",format="int64",JSONPath=".status.progress.totalBytes",description="Total bytes" // +kubebuilder:printcolumn:name="Storage Location",type="string",JSONPath=".spec.backupStorageLocation",description="Name of the Backup Storage Location where this backup should be stored" -// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp" +// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp",description="Time duration since this PodVolumeBackup was created" +// +kubebuilder:printcolumn:name="Node",type="string",JSONPath=".status.node",description="Name of the node where the PodVolumeBackup is processed" +// +kubebuilder:printcolumn:name="Uploader",type="string",JSONPath=".spec.uploaderType",description="The type of the uploader to handle data transfer" // +kubebuilder:object:root=true // +kubebuilder:object:generate=true diff --git a/pkg/apis/velero/v1/zz_generated.deepcopy.go b/pkg/apis/velero/v1/zz_generated.deepcopy.go index 2fd13cfd2..1cf3aaff8 100644 --- a/pkg/apis/velero/v1/zz_generated.deepcopy.go +++ b/pkg/apis/velero/v1/zz_generated.deepcopy.go @@ -1043,6 +1043,10 @@ func (in *PodVolumeBackupStatus) DeepCopyInto(out *PodVolumeBackupStatus) { *out = (*in).DeepCopy() } out.Progress = in.Progress + if in.AcceptedTimestamp != nil { + in, out := &in.AcceptedTimestamp, &out.AcceptedTimestamp + *out = (*in).DeepCopy() + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PodVolumeBackupStatus. diff --git a/pkg/builder/pod_volume_backup_builder.go b/pkg/builder/pod_volume_backup_builder.go index ac89f21de..e16ed223f 100644 --- a/pkg/builder/pod_volume_backup_builder.go +++ b/pkg/builder/pod_volume_backup_builder.go @@ -119,3 +119,33 @@ func (b *PodVolumeBackupBuilder) Annotations(annotations map[string]string) *Pod b.object.Annotations = annotations return b } + +// Cancel sets the PodVolumeBackup's Cancel. +func (b *PodVolumeBackupBuilder) Cancel(cancel bool) *PodVolumeBackupBuilder { + b.object.Spec.Cancel = cancel + return b +} + +// AcceptedTimestamp sets the PodVolumeBackup's AcceptedTimestamp. +func (b *PodVolumeBackupBuilder) AcceptedTimestamp(acceptedTimestamp *metav1.Time) *PodVolumeBackupBuilder { + b.object.Status.AcceptedTimestamp = acceptedTimestamp + return b +} + +// Finalizers sets the PodVolumeBackup's Finalizers. +func (b *PodVolumeBackupBuilder) Finalizers(finalizers []string) *PodVolumeBackupBuilder { + b.object.Finalizers = finalizers + return b +} + +// Message sets the PodVolumeBackup's Message. +func (b *PodVolumeBackupBuilder) Message(msg string) *PodVolumeBackupBuilder { + b.object.Status.Message = msg + return b +} + +// OwnerReference sets the PodVolumeBackup's OwnerReference. +func (b *PodVolumeBackupBuilder) OwnerReference(ref metav1.OwnerReference) *PodVolumeBackupBuilder { + b.object.OwnerReferences = append(b.object.OwnerReferences, ref) + return b +} diff --git a/pkg/cmd/cli/nodeagent/server.go b/pkg/cmd/cli/nodeagent/server.go index ede158116..4044b8897 100644 --- a/pkg/cmd/cli/nodeagent/server.go +++ b/pkg/cmd/cli/nodeagent/server.go @@ -299,12 +299,6 @@ func (s *nodeAgentServer) run() { credentialGetter := &credentials.CredentialGetter{FromFile: credentialFileStore, FromSecret: credSecretStore} repoEnsurer := repository.NewEnsurer(s.mgr.GetClient(), s.logger, s.config.resourceTimeout) - pvbReconciler := controller.NewPodVolumeBackupReconciler(s.mgr.GetClient(), s.kubeClient, s.dataPathMgr, repoEnsurer, - credentialGetter, s.nodeName, s.mgr.GetScheme(), s.metrics, s.logger) - - if err := pvbReconciler.SetupWithManager(s.mgr); err != nil { - s.logger.Fatal(err, "unable to create controller", "controller", constant.ControllerPodVolumeBackup) - } if err = controller.NewPodVolumeRestoreReconciler(s.mgr.GetClient(), s.kubeClient, s.dataPathMgr, repoEnsurer, credentialGetter, s.logger).SetupWithManager(s.mgr); err != nil { s.logger.WithError(err).Fatal("Unable to create the pod volume restore controller") @@ -332,6 +326,12 @@ func (s *nodeAgentServer) run() { } } + pvbReconciler := controller.NewPodVolumeBackupReconciler(s.mgr.GetClient(), s.mgr, s.kubeClient, s.dataPathMgr, s.nodeName, s.config.dataMoverPrepareTimeout, s.config.resourceTimeout, podResources, s.metrics, s.logger) + + if err := pvbReconciler.SetupWithManager(s.mgr); err != nil { + s.logger.Fatal(err, "unable to create controller", "controller", constant.ControllerPodVolumeBackup) + } + dataUploadReconciler := controller.NewDataUploadReconciler( s.mgr.GetClient(), s.mgr, diff --git a/pkg/controller/data_upload_controller_test.go b/pkg/controller/data_upload_controller_test.go index 11c0e8da4..c22819e58 100644 --- a/pkg/controller/data_upload_controller_test.go +++ b/pkg/controller/data_upload_controller_test.go @@ -312,30 +312,29 @@ func (f *fakeSnapshotExposer) DiagnoseExpose(context.Context, corev1api.ObjectRe func (f *fakeSnapshotExposer) CleanUp(context.Context, corev1api.ObjectReference, string, string) { } -type fakeDataUploadFSBR struct { - du *velerov2alpha1api.DataUpload +type fakeFSBR struct { kubeClient kbclient.Client clock clock.WithTickerAndDelayedExecution initErr error startErr error } -func (f *fakeDataUploadFSBR) Init(ctx context.Context, param any) error { +func (f *fakeFSBR) Init(ctx context.Context, param any) error { return f.initErr } -func (f *fakeDataUploadFSBR) StartBackup(source datapath.AccessPoint, uploaderConfigs map[string]string, param any) error { +func (f *fakeFSBR) StartBackup(source datapath.AccessPoint, uploaderConfigs map[string]string, param any) error { return f.startErr } -func (f *fakeDataUploadFSBR) StartRestore(snapshotID string, target datapath.AccessPoint, uploaderConfigs map[string]string) error { +func (f *fakeFSBR) StartRestore(snapshotID string, target datapath.AccessPoint, uploaderConfigs map[string]string) error { return nil } -func (b *fakeDataUploadFSBR) Cancel() { +func (b *fakeFSBR) Cancel() { } -func (b *fakeDataUploadFSBR) Close(ctx context.Context) { +func (b *fakeFSBR) Close(ctx context.Context) { } func TestReconcile(t *testing.T) { @@ -651,8 +650,7 @@ func TestReconcile(t *testing.T) { } datapath.MicroServiceBRWatcherCreator = func(kbclient.Client, kubernetes.Interface, manager.Manager, string, string, string, string, string, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { - return &fakeDataUploadFSBR{ - du: test.du, + return &fakeFSBR{ kubeClient: r.client, clock: r.Clock, initErr: test.fsBRInitErr, diff --git a/pkg/controller/pod_volume_backup_controller.go b/pkg/controller/pod_volume_backup_controller.go index 254a39c91..3231efda6 100644 --- a/pkg/controller/pod_volume_backup_controller.go +++ b/pkg/controller/pod_volume_backup_controller.go @@ -27,58 +27,72 @@ import ( corev1api "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes" clocks "k8s.io/utils/clock" ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/manager" + "sigs.k8s.io/controller-runtime/pkg/predicate" + "sigs.k8s.io/controller-runtime/pkg/reconcile" - "github.com/vmware-tanzu/velero/internal/credentials" veleroapishared "github.com/vmware-tanzu/velero/pkg/apis/velero/shared" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + "github.com/vmware-tanzu/velero/pkg/constant" "github.com/vmware-tanzu/velero/pkg/datapath" "github.com/vmware-tanzu/velero/pkg/exposer" "github.com/vmware-tanzu/velero/pkg/metrics" - "github.com/vmware-tanzu/velero/pkg/podvolume" - "github.com/vmware-tanzu/velero/pkg/repository" + "github.com/vmware-tanzu/velero/pkg/nodeagent" "github.com/vmware-tanzu/velero/pkg/uploader" - "github.com/vmware-tanzu/velero/pkg/util/filesystem" + "github.com/vmware-tanzu/velero/pkg/util" + "github.com/vmware-tanzu/velero/pkg/util/kube" ) -const pVBRRequestor string = "pod-volume-backup-restore" +const ( + pVBRRequestor = "pod-volume-backup-restore" + PodVolumeFinalizer = "velero.io/pod-volume-finalizer" +) // NewPodVolumeBackupReconciler creates the PodVolumeBackupReconciler instance -func NewPodVolumeBackupReconciler(client client.Client, kubeClient kubernetes.Interface, dataPathMgr *datapath.Manager, ensurer *repository.Ensurer, credentialGetter *credentials.CredentialGetter, - nodeName string, scheme *runtime.Scheme, metrics *metrics.ServerMetrics, logger logrus.FieldLogger) *PodVolumeBackupReconciler { +func NewPodVolumeBackupReconciler(client client.Client, mgr manager.Manager, kubeClient kubernetes.Interface, dataPathMgr *datapath.Manager, + nodeName string, preparingTimeout time.Duration, resourceTimeout time.Duration, podResources corev1api.ResourceRequirements, + metrics *metrics.ServerMetrics, logger logrus.FieldLogger) *PodVolumeBackupReconciler { return &PodVolumeBackupReconciler{ - Client: client, - kubeClient: kubeClient, - logger: logger.WithField("controller", "PodVolumeBackup"), - repositoryEnsurer: ensurer, - credentialGetter: credentialGetter, - nodeName: nodeName, - fileSystem: filesystem.NewFileSystem(), - clock: &clocks.RealClock{}, - scheme: scheme, - metrics: metrics, - dataPathMgr: dataPathMgr, + client: client, + mgr: mgr, + kubeClient: kubeClient, + logger: logger.WithField("controller", "PodVolumeBackup"), + nodeName: nodeName, + clock: &clocks.RealClock{}, + metrics: metrics, + podResources: podResources, + dataPathMgr: dataPathMgr, + preparingTimeout: preparingTimeout, + resourceTimeout: resourceTimeout, + exposer: exposer.NewPodVolumeExposer(kubeClient, logger), + cancelledPVB: make(map[string]time.Time), } } // PodVolumeBackupReconciler reconciles a PodVolumeBackup object type PodVolumeBackupReconciler struct { - client.Client - kubeClient kubernetes.Interface - scheme *runtime.Scheme - clock clocks.WithTickerAndDelayedExecution - metrics *metrics.ServerMetrics - credentialGetter *credentials.CredentialGetter - repositoryEnsurer *repository.Ensurer - nodeName string - fileSystem filesystem.Interface - logger logrus.FieldLogger - dataPathMgr *datapath.Manager + client client.Client + mgr manager.Manager + kubeClient kubernetes.Interface + clock clocks.WithTickerAndDelayedExecution + exposer exposer.PodVolumeExposer + metrics *metrics.ServerMetrics + nodeName string + logger logrus.FieldLogger + podResources corev1api.ResourceRequirements + dataPathMgr *datapath.Manager + preparingTimeout time.Duration + resourceTimeout time.Duration + cancelledPVB map[string]time.Time } // +kubebuilder:rbac:groups=velero.io,resources=podvolumebackups,verbs=get;list;watch;create;update;patch;delete @@ -90,13 +104,13 @@ func (r *PodVolumeBackupReconciler) Reconcile(ctx context.Context, req ctrl.Requ "podvolumebackup": req.NamespacedName, }) - var pvb velerov1api.PodVolumeBackup - if err := r.Client.Get(ctx, req.NamespacedName, &pvb); err != nil { + var pvb = &velerov1api.PodVolumeBackup{} + if err := r.client.Get(ctx, req.NamespacedName, pvb); err != nil { if apierrors.IsNotFound(err) { - log.Debug("Unable to find PodVolumeBackup") + log.Warn("Unable to find PVB, skip") return ctrl.Result{}, nil } - return ctrl.Result{}, errors.Wrap(err, "getting PodVolumeBackup") + return ctrl.Result{}, errors.Wrap(err, "getting PVB") } if len(pvb.OwnerReferences) == 1 { log = log.WithField( @@ -105,157 +119,420 @@ func (r *PodVolumeBackupReconciler) Reconcile(ctx context.Context, req ctrl.Requ ) } - // Only process items for this node. - if pvb.Spec.Node != r.nodeName { + if !isPVBInFinalState(pvb) { + if !controllerutil.ContainsFinalizer(pvb, PodVolumeFinalizer) { + if err := UpdatePVBWithRetry(ctx, r.client, req.NamespacedName, log, func(pvb *velerov1api.PodVolumeBackup) bool { + if controllerutil.ContainsFinalizer(pvb, PodVolumeFinalizer) { + return false + } + + controllerutil.AddFinalizer(pvb, PodVolumeFinalizer) + + return true + }); err != nil { + log.WithError(err).Errorf("Failed to add finalizer for PVB %s/%s", pvb.Namespace, pvb.Name) + return ctrl.Result{}, err + } + + return ctrl.Result{}, nil + } + + if !pvb.DeletionTimestamp.IsZero() { + if !pvb.Spec.Cancel { + log.Warnf("Cancel PVB under phase %s because it is being deleted", pvb.Status.Phase) + + if err := UpdatePVBWithRetry(ctx, r.client, req.NamespacedName, log, func(pvb *velerov1api.PodVolumeBackup) bool { + if pvb.Spec.Cancel { + return false + } + + pvb.Spec.Cancel = true + pvb.Status.Message = "Cancel PVB because it is being deleted" + + return true + }); err != nil { + log.WithError(err).Errorf("Failed to set cancel flag for PVB %s/%s", pvb.Namespace, pvb.Name) + return ctrl.Result{}, err + } + + return ctrl.Result{}, nil + } + } + } else { + delete(r.cancelledPVB, pvb.Name) + + if controllerutil.ContainsFinalizer(pvb, PodVolumeFinalizer) { + if err := UpdatePVBWithRetry(ctx, r.client, req.NamespacedName, log, func(pvb *velerov1api.PodVolumeBackup) bool { + if !controllerutil.ContainsFinalizer(pvb, PodVolumeFinalizer) { + return false + } + + controllerutil.RemoveFinalizer(pvb, PodVolumeFinalizer) + + return true + }); err != nil { + log.WithError(err).Error("error to remove finalizer") + return ctrl.Result{}, err + } + + return ctrl.Result{}, nil + } + } + + if pvb.Spec.Cancel { + if spotted, found := r.cancelledPVB[pvb.Name]; !found { + r.cancelledPVB[pvb.Name] = r.clock.Now() + } else { + delay := cancelDelayOthers + if pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseInProgress { + delay = cancelDelayInProgress + } + + if time.Since(spotted) > delay { + log.Infof("PVB %s is canceled in Phase %s but not handled in reasonable time", pvb.GetName(), pvb.Status.Phase) + if r.tryCancelPodVolumeBackup(ctx, pvb, "") { + delete(r.cancelledPVB, pvb.Name) + } + + return ctrl.Result{}, nil + } + } + } + + if pvb.Status.Phase == "" || pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseNew { + if pvb.Spec.Cancel { + log.Infof("PVB %s is canceled in Phase %s", pvb.GetName(), pvb.Status.Phase) + r.tryCancelPodVolumeBackup(ctx, pvb, "") + + return ctrl.Result{}, nil + } + + // Only process items for this node. + if pvb.Spec.Node != r.nodeName { + return ctrl.Result{}, nil + } + + log.Info("Accepting PVB") + + if err := r.acceptPodVolumeBackup(ctx, pvb); err != nil { + return ctrl.Result{}, errors.Wrapf(err, "error accepting PVB %s", pvb.Name) + } + + log.Info("Exposing PVB") + + exposeParam := r.setupExposeParam(pvb) + if err := r.exposer.Expose(ctx, getPVBOwnerObject(pvb), exposeParam); err != nil { + return r.errorOut(ctx, pvb, err, "error to expose PVB", log) + } + + log.Info("PVB is exposed") + return ctrl.Result{}, nil - } + } else if pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseAccepted { + if peekErr := r.exposer.PeekExposed(ctx, getPVBOwnerObject(pvb)); peekErr != nil { + log.Errorf("Cancel PVB %s/%s because of expose error %s", pvb.Namespace, pvb.Name, peekErr) + r.tryCancelPodVolumeBackup(ctx, pvb, fmt.Sprintf("found a PVB %s/%s with expose error: %s. mark it as cancel", pvb.Namespace, pvb.Name, peekErr)) + } else if pvb.Status.AcceptedTimestamp != nil { + if time.Since(pvb.Status.AcceptedTimestamp.Time) >= r.preparingTimeout { + r.onPrepareTimeout(ctx, pvb) + } + } - switch pvb.Status.Phase { - case "", velerov1api.PodVolumeBackupPhaseNew: - // Only process new items. - default: - log.Debug("PodVolumeBackup is not new, not processing") return ctrl.Result{}, nil - } + } else if pvb.Status.Phase == velerov1api.PodVolumeBackupPhasePrepared { + log.Infof("PVB is prepared and should be processed by %s (%s)", pvb.Spec.Node, r.nodeName) - log.Info("PodVolumeBackup starting") + if pvb.Spec.Node != r.nodeName { + return ctrl.Result{}, nil + } - callbacks := datapath.Callbacks{ - OnCompleted: r.OnDataPathCompleted, - OnFailed: r.OnDataPathFailed, - OnCancelled: r.OnDataPathCancelled, - OnProgress: r.OnDataPathProgress, - } + if pvb.Spec.Cancel { + log.Info("Prepared PVB is being cancelled") + r.OnDataPathCancelled(ctx, pvb.GetNamespace(), pvb.GetName()) + return ctrl.Result{}, nil + } - fsBackup, err := r.dataPathMgr.CreateFileSystemBR(pvb.Name, pVBRRequestor, ctx, r.Client, pvb.Namespace, callbacks, log) + asyncBR := r.dataPathMgr.GetAsyncBR(pvb.Name) + if asyncBR != nil { + log.Info("Cancellable data path is already started") + return ctrl.Result{}, nil + } - if err != nil { - if err == datapath.ConcurrentLimitExceed { + res, err := r.exposer.GetExposed(ctx, getPVBOwnerObject(pvb), r.client, r.nodeName, r.resourceTimeout) + if err != nil { + return r.errorOut(ctx, pvb, err, "exposed PVB is not ready", log) + } else if res == nil { + return r.errorOut(ctx, pvb, errors.New("no expose result is available for the current node"), "exposed PVB is not ready", log) + } + + log.Info("Exposed PVB is ready and creating data path routine") + + callbacks := datapath.Callbacks{ + OnCompleted: r.OnDataPathCompleted, + OnFailed: r.OnDataPathFailed, + OnCancelled: r.OnDataPathCancelled, + OnProgress: r.OnDataPathProgress, + } + + asyncBR, err = r.dataPathMgr.CreateMicroServiceBRWatcher(ctx, r.client, r.kubeClient, r.mgr, datapath.TaskTypeBackup, + pvb.Name, pvb.Namespace, res.ByPod.HostingPod.Name, res.ByPod.HostingContainer, pvb.Name, callbacks, false, log) + if err != nil { + if err == datapath.ConcurrentLimitExceed { + log.Info("Data path instance is concurrent limited requeue later") + return ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, nil + } else { + return r.errorOut(ctx, pvb, err, "error to create data path", log) + } + } + + r.metrics.RegisterPodVolumeBackupEnqueue(r.nodeName) + + if err := r.initCancelableDataPath(ctx, asyncBR, res, log); err != nil { + log.WithError(err).Errorf("Failed to init cancelable data path for %s", pvb.Name) + + r.closeDataPath(ctx, pvb.Name) + return r.errorOut(ctx, pvb, err, "error initializing data path", log) + } + + terminated := false + if err := UpdatePVBWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, log, func(pvb *velerov1api.PodVolumeBackup) bool { + if isPVBInFinalState(pvb) { + terminated = true + return false + } + + pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseInProgress + pvb.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + + return true + }); err != nil { + log.WithError(err).Warnf("Failed to update PVB %s to InProgress, will data path close and retry", pvb.Name) + + r.closeDataPath(ctx, pvb.Name) return ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, nil - } else { - return r.errorOut(ctx, &pvb, err, "error to create data path", log) } - } - r.metrics.RegisterPodVolumeBackupEnqueue(r.nodeName) - - // Update status to InProgress. - original := pvb.DeepCopy() - pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseInProgress - pvb.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} - if err := r.Client.Patch(ctx, &pvb, client.MergeFrom(original)); err != nil { - r.closeDataPath(ctx, pvb.Name) - return r.errorOut(ctx, &pvb, err, "error updating PodVolumeBackup status", log) - } - - var pod corev1api.Pod - podNamespacedName := client.ObjectKey{ - Namespace: pvb.Spec.Pod.Namespace, - Name: pvb.Spec.Pod.Name, - } - if err := r.Client.Get(ctx, podNamespacedName, &pod); err != nil { - r.closeDataPath(ctx, pvb.Name) - return r.errorOut(ctx, &pvb, err, fmt.Sprintf("getting pod %s/%s", pvb.Spec.Pod.Namespace, pvb.Spec.Pod.Name), log) - } - - path, err := exposer.GetPodVolumeHostPath(ctx, &pod, pvb.Spec.Volume, r.kubeClient, r.fileSystem, log) - if err != nil { - r.closeDataPath(ctx, pvb.Name) - return r.errorOut(ctx, &pvb, err, "error exposing host path for pod volume", log) - } - - log.WithField("path", path.ByPath).Debugf("Found host path") - - if err := fsBackup.Init(ctx, &datapath.FSBRInitParam{ - BSLName: pvb.Spec.BackupStorageLocation, - SourceNamespace: pvb.Spec.Pod.Namespace, - UploaderType: pvb.Spec.UploaderType, - RepositoryType: podvolume.GetPvbRepositoryType(&pvb), - RepoIdentifier: pvb.Spec.RepoIdentifier, - RepositoryEnsurer: r.repositoryEnsurer, - CredentialGetter: r.credentialGetter, - }); err != nil { - r.closeDataPath(ctx, pvb.Name) - return r.errorOut(ctx, &pvb, err, "error to initialize data path", log) - } - - // If this is a PVC, look for the most recent completed pod volume backup for it and get - // its snapshot ID to do new backup based on it. Without this, - // if the pod using the PVC (and therefore the directory path under /host_pods/) has - // changed since the PVC's last backup, for backup, it will not be able to identify a suitable - // parent snapshot to use, and will have to do a full rescan of the contents of the PVC. - var parentSnapshotID string - if pvcUID, ok := pvb.Labels[velerov1api.PVCUIDLabel]; ok { - parentSnapshotID = r.getParentSnapshot(ctx, log, pvcUID, &pvb) - if parentSnapshotID == "" { - log.Info("No parent snapshot found for PVC, not based on parent snapshot for this backup") - } else { - log.WithField("parentSnapshotID", parentSnapshotID).Info("Based on parent snapshot for this backup") + if terminated { + log.Warnf("PVB %s is terminated during transition from prepared", pvb.Name) + r.closeDataPath(ctx, pvb.Name) + return ctrl.Result{}, nil } - } - if err := fsBackup.StartBackup(path, pvb.Spec.UploaderSettings, &datapath.FSBRStartParam{ - RealSource: "", - ParentSnapshot: parentSnapshotID, - ForceFull: false, - Tags: pvb.Spec.Tags, - }); err != nil { - r.closeDataPath(ctx, pvb.Name) - return r.errorOut(ctx, &pvb, err, "error starting data path backup", log) - } + log.Info("PVB is marked as in progress") - log.WithField("path", path.ByPath).Info("Async fs backup data path started") + if err := r.startCancelableDataPath(asyncBR, pvb, res, log); err != nil { + log.WithError(err).Errorf("Failed to start cancelable data path for %s", pvb.Name) + r.closeDataPath(ctx, pvb.Name) + + return r.errorOut(ctx, pvb, err, "error starting data path", log) + } + + return ctrl.Result{}, nil + } else if pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseInProgress { + if pvb.Spec.Cancel { + if pvb.Spec.Node != r.nodeName { + return ctrl.Result{}, nil + } + + log.Info("In progress PVB is being canceled") + + asyncBR := r.dataPathMgr.GetAsyncBR(pvb.Name) + if asyncBR == nil { + r.OnDataPathCancelled(ctx, pvb.GetNamespace(), pvb.GetName()) + return ctrl.Result{}, nil + } + + // Update status to Canceling + if err := UpdatePVBWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, log, func(pvb *velerov1api.PodVolumeBackup) bool { + if isPVBInFinalState(pvb) { + log.Warnf("PVB %s is terminated, abort setting it to cancelling", pvb.Name) + return false + } + + pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseCanceling + return true + }); err != nil { + log.WithError(err).Error("error updating PVB into canceling status") + return ctrl.Result{}, err + } + + asyncBR.Cancel() + return ctrl.Result{}, nil + } + return ctrl.Result{}, nil + } return ctrl.Result{}, nil } +func (r *PodVolumeBackupReconciler) acceptPodVolumeBackup(ctx context.Context, pvb *velerov1api.PodVolumeBackup) error { + return UpdatePVBWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, r.logger, func(pvb *velerov1api.PodVolumeBackup) bool { + pvb.Status.AcceptedTimestamp = &metav1.Time{Time: r.clock.Now()} + pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseAccepted + + return true + }) +} + +func (r *PodVolumeBackupReconciler) tryCancelPodVolumeBackup(ctx context.Context, pvb *velerov1api.PodVolumeBackup, message string) bool { + log := r.logger.WithField("PVB", pvb.Name) + succeeded, err := funcExclusiveUpdatePodVolumeBackup(ctx, r.client, pvb, func(pvb *velerov1api.PodVolumeBackup) { + pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseCanceled + if pvb.Status.StartTimestamp.IsZero() { + pvb.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + } + pvb.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + + if message != "" { + pvb.Status.Message = message + } + }) + + if err != nil { + log.WithError(err).Error("error updating PVB status") + return false + } else if !succeeded { + log.Warn("conflict in updating PVB status and will try it again later") + return false + } + + r.exposer.CleanUp(ctx, getPVBOwnerObject(pvb)) + + log.Warn("PVB is canceled") + + return true +} + +var funcExclusiveUpdatePodVolumeBackup = exclusiveUpdatePodVolumeBackup + +func exclusiveUpdatePodVolumeBackup(ctx context.Context, cli client.Client, pvb *velerov1api.PodVolumeBackup, updateFunc func(*velerov1api.PodVolumeBackup)) (bool, error) { + updateFunc(pvb) + + err := cli.Update(ctx, pvb) + if err == nil { + return true, nil + } + + if apierrors.IsConflict(err) { + return false, nil + } else { + return false, err + } +} + +func (r *PodVolumeBackupReconciler) onPrepareTimeout(ctx context.Context, pvb *velerov1api.PodVolumeBackup) { + log := r.logger.WithField("PVB", pvb.Name) + + log.Info("Timeout happened for preparing PVB") + + succeeded, err := funcExclusiveUpdatePodVolumeBackup(ctx, r.client, pvb, func(pvb *velerov1api.PodVolumeBackup) { + pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseFailed + pvb.Status.Message = "timeout on preparing PVB" + }) + + if err != nil { + log.WithError(err).Warn("Failed to update PVB") + return + } + + if !succeeded { + log.Warn("PVB has been updated by others") + return + } + + diags := strings.Split(r.exposer.DiagnoseExpose(ctx, getPVBOwnerObject(pvb)), "\n") + for _, diag := range diags { + log.Warnf("[Diagnose PVB expose]%s", diag) + } + + r.exposer.CleanUp(ctx, getPVBOwnerObject(pvb)) + + log.Info("PVB has been cleaned up") +} + +func (r *PodVolumeBackupReconciler) initCancelableDataPath(ctx context.Context, asyncBR datapath.AsyncBR, res *exposer.ExposeResult, log logrus.FieldLogger) error { + log.Info("Init cancelable PVB") + + if err := asyncBR.Init(ctx, nil); err != nil { + return errors.Wrap(err, "error initializing asyncBR") + } + + log.Infof("async data path init for pod %s, volume %s", res.ByPod.HostingPod.Name, res.ByPod.VolumeName) + + return nil +} + +func (r *PodVolumeBackupReconciler) startCancelableDataPath(asyncBR datapath.AsyncBR, pvb *velerov1api.PodVolumeBackup, res *exposer.ExposeResult, log logrus.FieldLogger) error { + log.Info("Start cancelable PVB") + + if err := asyncBR.StartBackup(datapath.AccessPoint{ + ByPath: res.ByPod.VolumeName, + }, pvb.Spec.UploaderSettings, nil); err != nil { + return errors.Wrapf(err, "error starting async backup for pod %s, volume %s", res.ByPod.HostingPod.Name, res.ByPod.VolumeName) + } + + log.Infof("Async backup started for pod %s, volume %s", res.ByPod.HostingPod.Name, res.ByPod.VolumeName) + return nil +} + func (r *PodVolumeBackupReconciler) OnDataPathCompleted(ctx context.Context, namespace string, pvbName string, result datapath.Result) { defer r.dataPathMgr.RemoveAsyncBR(pvbName) - log := r.logger.WithField("pvb", pvbName) + log := r.logger.WithField("PVB", pvbName) log.WithField("PVB", pvbName).Info("Async fs backup data path completed") - var pvb velerov1api.PodVolumeBackup - if err := r.Client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, &pvb); err != nil { + pvb := &velerov1api.PodVolumeBackup{} + if err := r.client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, pvb); err != nil { log.WithError(err).Warn("Failed to get PVB on completion") return } + log.Info("Cleaning up exposed environment") + r.exposer.CleanUp(ctx, getPVBOwnerObject(pvb)) + // Update status to Completed with path & snapshot ID. - original := pvb.DeepCopy() - pvb.Status.Path = result.Backup.Source.ByPath - pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseCompleted - pvb.Status.SnapshotID = result.Backup.SnapshotID - pvb.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} - if result.Backup.EmptySnapshot { - pvb.Status.Message = "volume was empty so no snapshot was taken" + var completionTime metav1.Time + if err := UpdatePVBWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, log, func(pvb *velerov1api.PodVolumeBackup) bool { + completionTime = metav1.Time{Time: r.clock.Now()} + + if isPVBInFinalState(pvb) { + return false + } + + pvb.Status.Path = result.Backup.Source.ByPath + pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseCompleted + pvb.Status.SnapshotID = result.Backup.SnapshotID + pvb.Status.CompletionTimestamp = &completionTime + if result.Backup.EmptySnapshot { + pvb.Status.Message = "volume was empty so no snapshot was taken" + } + + return true + }); err != nil { + log.WithError(err).Error("error updating PVB status") + } else { + latencyDuration := completionTime.Time.Sub(pvb.Status.StartTimestamp.Time) + latencySeconds := float64(latencyDuration / time.Second) + backupName := fmt.Sprintf("%s/%s", pvb.Namespace, pvb.OwnerReferences[0].Name) + generateOpName := fmt.Sprintf("%s-%s-%s-%s-backup", pvb.Name, pvb.Spec.BackupStorageLocation, pvb.Spec.Pod.Namespace, pvb.Spec.UploaderType) + r.metrics.ObservePodVolumeOpLatency(r.nodeName, pvb.Name, generateOpName, backupName, latencySeconds) + r.metrics.RegisterPodVolumeOpLatencyGauge(r.nodeName, pvb.Name, generateOpName, backupName, latencySeconds) + r.metrics.RegisterPodVolumeBackupDequeue(r.nodeName) + + log.Info("PVB completed") } - - if err := r.Client.Patch(ctx, &pvb, client.MergeFrom(original)); err != nil { - log.WithError(err).Error("error updating PodVolumeBackup status") - } - - latencyDuration := pvb.Status.CompletionTimestamp.Time.Sub(pvb.Status.StartTimestamp.Time) - latencySeconds := float64(latencyDuration / time.Second) - backupName := fmt.Sprintf("%s/%s", pvb.Namespace, pvb.OwnerReferences[0].Name) - generateOpName := fmt.Sprintf("%s-%s-%s-%s-backup", pvb.Name, pvb.Spec.BackupStorageLocation, pvb.Spec.Pod.Namespace, pvb.Spec.UploaderType) - r.metrics.ObservePodVolumeOpLatency(r.nodeName, pvb.Name, generateOpName, backupName, latencySeconds) - r.metrics.RegisterPodVolumeOpLatencyGauge(r.nodeName, pvb.Name, generateOpName, backupName, latencySeconds) - r.metrics.RegisterPodVolumeBackupDequeue(r.nodeName) - - log.Info("PodVolumeBackup completed") } func (r *PodVolumeBackupReconciler) OnDataPathFailed(ctx context.Context, namespace, pvbName string, err error) { defer r.dataPathMgr.RemoveAsyncBR(pvbName) - log := r.logger.WithField("pvb", pvbName) + log := r.logger.WithField("PVB", pvbName) log.WithError(err).Error("Async fs backup data path failed") var pvb velerov1api.PodVolumeBackup - if getErr := r.Client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, &pvb); getErr != nil { + if getErr := r.client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, &pvb); getErr != nil { log.WithError(getErr).Warn("Failed to get PVB on failure") } else { _, _ = r.errorOut(ctx, &pvb, err, "data path backup failed", log) @@ -265,131 +542,305 @@ func (r *PodVolumeBackupReconciler) OnDataPathFailed(ctx context.Context, namesp func (r *PodVolumeBackupReconciler) OnDataPathCancelled(ctx context.Context, namespace string, pvbName string) { defer r.dataPathMgr.RemoveAsyncBR(pvbName) - log := r.logger.WithField("pvb", pvbName) + log := r.logger.WithField("PVB", pvbName) log.Warn("Async fs backup data path canceled") var pvb velerov1api.PodVolumeBackup - if getErr := r.Client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, &pvb); getErr != nil { + if getErr := r.client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, &pvb); getErr != nil { log.WithError(getErr).Warn("Failed to get PVB on cancel") + return + } + // cleans up any objects generated during the snapshot expose + r.exposer.CleanUp(ctx, getPVBOwnerObject(&pvb)) + + if err := UpdatePVBWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, log, func(pvb *velerov1api.PodVolumeBackup) bool { + if isPVBInFinalState(pvb) { + return false + } + + pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseCanceled + if pvb.Status.StartTimestamp.IsZero() { + pvb.Status.StartTimestamp = &metav1.Time{Time: r.clock.Now()} + } + pvb.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now()} + + return true + }); err != nil { + log.WithError(err).Error("error updating PVB status on cancel") } else { - _, _ = r.errorOut(ctx, &pvb, errors.New("PVB is canceled"), "data path backup canceled", log) + delete(r.cancelledPVB, pvb.Name) } } func (r *PodVolumeBackupReconciler) OnDataPathProgress(ctx context.Context, namespace string, pvbName string, progress *uploader.Progress) { log := r.logger.WithField("pvb", pvbName) - var pvb velerov1api.PodVolumeBackup - if err := r.Client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, &pvb); err != nil { - log.WithError(err).Warn("Failed to get PVB on progress") - return - } - - original := pvb.DeepCopy() - pvb.Status.Progress = veleroapishared.DataMoveOperationProgress{TotalBytes: progress.TotalBytes, BytesDone: progress.BytesDone} - - if err := r.Client.Patch(ctx, &pvb, client.MergeFrom(original)); err != nil { + if err := UpdatePVBWithRetry(ctx, r.client, types.NamespacedName{Namespace: namespace, Name: pvbName}, log, func(pvb *velerov1api.PodVolumeBackup) bool { + pvb.Status.Progress = veleroapishared.DataMoveOperationProgress{TotalBytes: progress.TotalBytes, BytesDone: progress.BytesDone} + return true + }); err != nil { log.WithError(err).Error("Failed to update progress") } } // SetupWithManager registers the PVB controller. func (r *PodVolumeBackupReconciler) SetupWithManager(mgr ctrl.Manager) error { + gp := kube.NewGenericEventPredicate(func(object client.Object) bool { + pvb := object.(*velerov1api.PodVolumeBackup) + if pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseAccepted { + return true + } + + if pvb.Spec.Cancel && !isPVBInFinalState(pvb) { + return true + } + + if isPVBInFinalState(pvb) && !pvb.DeletionTimestamp.IsZero() { + return true + } + + return false + }) + + s := kube.NewPeriodicalEnqueueSource(r.logger.WithField("controller", constant.ControllerPodVolumeBackup), r.client, &velerov1api.PodVolumeBackupList{}, preparingMonitorFrequency, kube.PeriodicalEnqueueSourceOption{ + Predicates: []predicate.Predicate{gp}, + }) + return ctrl.NewControllerManagedBy(mgr). For(&velerov1api.PodVolumeBackup{}). + WatchesRawSource(s). + Watches(&corev1api.Pod{}, kube.EnqueueRequestsFromMapUpdateFunc(r.findPVBForPod), + builder.WithPredicates(predicate.Funcs{ + UpdateFunc: func(ue event.UpdateEvent) bool { + newObj := ue.ObjectNew.(*corev1api.Pod) + + if _, ok := newObj.Labels[velerov1api.PVBLabel]; !ok { + return false + } + + if newObj.Spec.NodeName == "" { + return false + } + + return true + }, + CreateFunc: func(event.CreateEvent) bool { + return false + }, + DeleteFunc: func(de event.DeleteEvent) bool { + return false + }, + GenericFunc: func(ge event.GenericEvent) bool { + return false + }, + })). Complete(r) } -// getParentSnapshot finds the most recent completed PodVolumeBackup for the -// specified PVC and returns its snapshot ID. Any errors encountered are -// logged but not returned since they do not prevent a backup from proceeding. -func (r *PodVolumeBackupReconciler) getParentSnapshot(ctx context.Context, log logrus.FieldLogger, pvcUID string, podVolumeBackup *velerov1api.PodVolumeBackup) string { - log = log.WithField("pvcUID", pvcUID) - log.Infof("Looking for most recent completed PodVolumeBackup for this PVC") +func (r *PodVolumeBackupReconciler) findPVBForPod(ctx context.Context, podObj client.Object) []reconcile.Request { + pod := podObj.(*corev1api.Pod) + pvb, err := findPVBByPod(r.client, *pod) - listOpts := &client.ListOptions{ - Namespace: podVolumeBackup.Namespace, + log := r.logger.WithField("pod", pod.Name) + if err != nil { + log.WithError(err).Error("unable to get PVB") + return []reconcile.Request{} + } else if pvb == nil { + log.Error("get empty PVB") + return []reconcile.Request{} } - matchingLabels := client.MatchingLabels(map[string]string{velerov1api.PVCUIDLabel: pvcUID}) - matchingLabels.ApplyToList(listOpts) + log = log.WithFields(logrus.Fields{ + "PVB": pvb.Name, + }) - var pvbList velerov1api.PodVolumeBackupList - if err := r.Client.List(ctx, &pvbList, listOpts); err != nil { - log.WithError(errors.WithStack(err)).Error("getting list of podvolumebackups for this PVC") + if pvb.Status.Phase != velerov1api.PodVolumeBackupPhaseAccepted { + return []reconcile.Request{} } - // Go through all the podvolumebackups for the PVC and look for the most - // recent completed one to use as the parent. - var mostRecentPVB velerov1api.PodVolumeBackup - for _, pvb := range pvbList.Items { - if pvb.Spec.UploaderType != podVolumeBackup.Spec.UploaderType { - continue - } - if pvb.Status.Phase != velerov1api.PodVolumeBackupPhaseCompleted { - continue + if pod.Status.Phase == corev1api.PodRunning { + log.Info("Preparing PVB") + + if err = UpdatePVBWithRetry(context.Background(), r.client, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, log, + func(pvb *velerov1api.PodVolumeBackup) bool { + if isPVBInFinalState(pvb) { + log.Warnf("PVB %s is terminated, abort setting it to prepared", pvb.Name) + return false + } + + pvb.Status.Phase = velerov1api.PodVolumeBackupPhasePrepared + return true + }); err != nil { + log.WithError(err).Warn("Failed to update PVB, prepare will halt for this PVB") + return []reconcile.Request{} } + } else if unrecoverable, reason := kube.IsPodUnrecoverable(pod, log); unrecoverable { + err := UpdatePVBWithRetry(context.Background(), r.client, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, log, + func(pvb *velerov1api.PodVolumeBackup) bool { + if pvb.Spec.Cancel { + return false + } - if podVolumeBackup.Spec.BackupStorageLocation != pvb.Spec.BackupStorageLocation { - // Check the backup storage location is the same as spec in order to - // support backup to multiple backup-locations. Otherwise, there exists - // a case that backup volume snapshot to the second location would - // failed, since the founded parent ID is only valid for the first - // backup location, not the second backup location. Also, the second - // backup should not use the first backup parent ID since its for the - // first backup location only. - continue - } + pvb.Spec.Cancel = true + pvb.Status.Message = fmt.Sprintf("Cancel PVB because the exposing pod %s/%s is in abnormal status for reason %s", pod.Namespace, pod.Name, reason) - if mostRecentPVB.Status == (velerov1api.PodVolumeBackupStatus{}) || pvb.Status.StartTimestamp.After(mostRecentPVB.Status.StartTimestamp.Time) { - mostRecentPVB = pvb + return true + }) + + if err != nil { + log.WithError(err).Warn("failed to cancel PVB, and it will wait for prepare timeout") + return []reconcile.Request{} } + log.Infof("Exposed pod is in abnormal status(reason %s) and PVB is marked as cancel", reason) + } else { + return []reconcile.Request{} } - if mostRecentPVB.Status == (velerov1api.PodVolumeBackupStatus{}) { - log.Info("No completed PodVolumeBackup found for PVC") - return "" + request := reconcile.Request{ + NamespacedName: types.NamespacedName{ + Namespace: pvb.Namespace, + Name: pvb.Name, + }, } - - log.WithFields(map[string]any{ - "parentPodVolumeBackup": mostRecentPVB.Name, - "parentSnapshotID": mostRecentPVB.Status.SnapshotID, - }).Info("Found most recent completed PodVolumeBackup for PVC") - - return mostRecentPVB.Status.SnapshotID -} - -func (r *PodVolumeBackupReconciler) closeDataPath(ctx context.Context, pvbName string) { - fsBackup := r.dataPathMgr.GetAsyncBR(pvbName) - if fsBackup != nil { - fsBackup.Close(ctx) - } - - r.dataPathMgr.RemoveAsyncBR(pvbName) + return []reconcile.Request{request} } func (r *PodVolumeBackupReconciler) errorOut(ctx context.Context, pvb *velerov1api.PodVolumeBackup, err error, msg string, log logrus.FieldLogger) (ctrl.Result, error) { - _ = UpdatePVBStatusToFailed(ctx, r.Client, pvb, err, msg, r.clock.Now(), log) + r.exposer.CleanUp(ctx, getPVBOwnerObject(pvb)) + + _ = UpdatePVBStatusToFailed(ctx, r.client, pvb, err, msg, r.clock.Now(), log) return ctrl.Result{}, err } func UpdatePVBStatusToFailed(ctx context.Context, c client.Client, pvb *velerov1api.PodVolumeBackup, errOut error, msg string, time time.Time, log logrus.FieldLogger) error { - original := pvb.DeepCopy() - pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseFailed - pvb.Status.CompletionTimestamp = &metav1.Time{Time: time} - if dataPathError, ok := errOut.(datapath.DataPathError); ok { - pvb.Status.SnapshotID = dataPathError.GetSnapshotID() - } - if len(strings.TrimSpace(msg)) == 0 { - pvb.Status.Message = errOut.Error() - } else { - pvb.Status.Message = errors.WithMessage(errOut, msg).Error() - } - err := c.Patch(ctx, pvb, client.MergeFrom(original)) - if err != nil { - log.WithError(err).Error("error updating PodVolumeBackup status") + log.Info("update PVB status to Failed") + + if patchErr := UpdatePVBWithRetry(context.Background(), c, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, log, + func(pvb *velerov1api.PodVolumeBackup) bool { + if isPVBInFinalState(pvb) { + return false + } + + pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseFailed + pvb.Status.CompletionTimestamp = &metav1.Time{Time: time} + if dataPathError, ok := errOut.(datapath.DataPathError); ok { + pvb.Status.SnapshotID = dataPathError.GetSnapshotID() + } + if len(strings.TrimSpace(msg)) == 0 { + pvb.Status.Message = errOut.Error() + } else { + pvb.Status.Message = errors.WithMessage(errOut, msg).Error() + } + if pvb.Status.StartTimestamp.IsZero() { + pvb.Status.StartTimestamp = &metav1.Time{Time: time} + } + + return true + }); patchErr != nil { + log.WithError(patchErr).Warn("error updating PVB status") } - return err + return errOut +} + +func (r *PodVolumeBackupReconciler) closeDataPath(ctx context.Context, pvbName string) { + asyncBR := r.dataPathMgr.GetAsyncBR(pvbName) + if asyncBR != nil { + asyncBR.Close(ctx) + } + + r.dataPathMgr.RemoveAsyncBR(pvbName) +} + +func (r *PodVolumeBackupReconciler) setupExposeParam(pvb *velerov1api.PodVolumeBackup) exposer.PodVolumeExposeParam { + log := r.logger.WithField("PVB", pvb.Name) + + hostingPodLabels := map[string]string{velerov1api.PVBLabel: pvb.Name} + for _, k := range util.ThirdPartyLabels { + if v, err := nodeagent.GetLabelValue(context.Background(), r.kubeClient, pvb.Namespace, k, ""); err != nil { + if err != nodeagent.ErrNodeAgentLabelNotFound { + log.WithError(err).Warnf("Failed to check node-agent label, skip adding host pod label %s", k) + } + } else { + hostingPodLabels[k] = v + } + } + + hostingPodAnnotation := map[string]string{} + for _, k := range util.ThirdPartyAnnotations { + if v, err := nodeagent.GetAnnotationValue(context.Background(), r.kubeClient, pvb.Namespace, k, ""); err != nil { + if err != nodeagent.ErrNodeAgentAnnotationNotFound { + log.WithError(err).Warnf("Failed to check node-agent annotation, skip adding host pod annotation %s", k) + } + } else { + hostingPodAnnotation[k] = v + } + } + + return exposer.PodVolumeExposeParam{ + Type: exposer.PodVolumeExposeTypeBackup, + ClientNamespace: pvb.Spec.Pod.Namespace, + ClientPodName: pvb.Spec.Pod.Name, + ClientPodVolume: pvb.Spec.Volume, + HostingPodLabels: hostingPodLabels, + HostingPodAnnotations: hostingPodAnnotation, + OperationTimeout: r.resourceTimeout, + Resources: r.podResources, + } +} + +func getPVBOwnerObject(pvb *velerov1api.PodVolumeBackup) corev1api.ObjectReference { + return corev1api.ObjectReference{ + Kind: pvb.Kind, + Namespace: pvb.Namespace, + Name: pvb.Name, + UID: pvb.UID, + APIVersion: pvb.APIVersion, + } +} + +func findPVBByPod(client client.Client, pod corev1api.Pod) (*velerov1api.PodVolumeBackup, error) { + if label, exist := pod.Labels[velerov1api.PVBLabel]; exist { + pvb := &velerov1api.PodVolumeBackup{} + err := client.Get(context.Background(), types.NamespacedName{ + Namespace: pod.Namespace, + Name: label, + }, pvb) + + if err != nil { + return nil, errors.Wrapf(err, "error to find PVB by pod %s/%s", pod.Namespace, pod.Name) + } + return pvb, nil + } + return nil, nil +} + +func isPVBInFinalState(pvb *velerov1api.PodVolumeBackup) bool { + return pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseFailed || + pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseCanceled || + pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseCompleted +} + +func UpdatePVBWithRetry(ctx context.Context, client client.Client, namespacedName types.NamespacedName, log logrus.FieldLogger, updateFunc func(*velerov1api.PodVolumeBackup) bool) error { + return wait.PollUntilContextCancel(ctx, time.Millisecond*100, true, func(ctx context.Context) (bool, error) { + pvb := &velerov1api.PodVolumeBackup{} + if err := client.Get(ctx, namespacedName, pvb); err != nil { + return false, errors.Wrap(err, "getting PVB") + } + + if updateFunc(pvb) { + err := client.Update(ctx, pvb) + if err != nil { + if apierrors.IsConflict(err) { + log.Warnf("failed to update PVB for %s/%s and will retry it", pvb.Namespace, pvb.Name) + return false, nil + } else { + return false, errors.Wrapf(err, "error updating PVB with error %s/%s", pvb.Namespace, pvb.Name) + } + } + } + + return true, nil + }) } diff --git a/pkg/controller/pod_volume_backup_controller_test.go b/pkg/controller/pod_volume_backup_controller_test.go index d5f1e3471..aa9a398ed 100644 --- a/pkg/controller/pod_volume_backup_controller_test.go +++ b/pkg/controller/pod_volume_backup_controller_test.go @@ -19,183 +19,487 @@ package controller import ( "context" "fmt" + "testing" "time" - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" + "github.com/pkg/errors" "github.com/sirupsen/logrus" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + appsv1api "k8s.io/api/apps/v1" corev1api "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" - "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/kubernetes" + clientgofake "k8s.io/client-go/kubernetes/fake" "k8s.io/utils/clock" - testclocks "k8s.io/utils/clock/testing" ctrl "sigs.k8s.io/controller-runtime" - kbclient "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/manager" + "sigs.k8s.io/controller-runtime/pkg/reconcile" - "github.com/vmware-tanzu/velero/internal/credentials" velerov1api "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" "github.com/vmware-tanzu/velero/pkg/builder" "github.com/vmware-tanzu/velero/pkg/datapath" + "github.com/vmware-tanzu/velero/pkg/exposer" "github.com/vmware-tanzu/velero/pkg/metrics" velerotest "github.com/vmware-tanzu/velero/pkg/test" + "github.com/vmware-tanzu/velero/pkg/uploader" + "github.com/vmware-tanzu/velero/pkg/util/kube" ) -const name = "pvb-1" +const pvbName = "pvb-1" + +func initPVBReconciler(needError ...bool) (*PodVolumeBackupReconciler, error) { + var errs = make([]error, 6) + for k, isError := range needError { + if k == 0 && isError { + errs[0] = fmt.Errorf("Get error") + } else if k == 1 && isError { + errs[1] = fmt.Errorf("Create error") + } else if k == 2 && isError { + errs[2] = fmt.Errorf("Update error") + } else if k == 3 && isError { + errs[3] = fmt.Errorf("Patch error") + } else if k == 4 && isError { + errs[4] = apierrors.NewConflict(velerov1api.Resource("podvolumebackup"), pvbName, errors.New("conflict")) + } else if k == 5 && isError { + errs[5] = fmt.Errorf("List error") + } + } + + return initPVBReconcilerWithError(errs...) +} + +func initPVBReconcilerWithError(needError ...error) (*PodVolumeBackupReconciler, error) { + daemonSet := &appsv1api.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "velero", + Name: "node-agent", + }, + TypeMeta: metav1.TypeMeta{ + Kind: "DaemonSet", + APIVersion: appsv1api.SchemeGroupVersion.String(), + }, + Spec: appsv1api.DaemonSetSpec{ + Template: corev1api.PodTemplateSpec{ + Spec: corev1api.PodSpec{ + Containers: []corev1api.Container{ + { + Image: "fake-image", + }, + }, + }, + }, + }, + } + + node := builder.ForNode("fake-node").Labels(map[string]string{kube.NodeOSLabel: kube.NodeOSLinux}).Result() + + dataPathMgr := datapath.NewManager(1) + + scheme := runtime.NewScheme() + err := velerov1api.AddToScheme(scheme) + if err != nil { + return nil, err + } + + err = corev1api.AddToScheme(scheme) + if err != nil { + return nil, err + } + + fakeClient := &FakeClient{ + Client: fake.NewClientBuilder().WithScheme(scheme).Build(), + } + + for k := range needError { + if k == 0 { + fakeClient.getError = needError[0] + } else if k == 1 { + fakeClient.createError = needError[1] + } else if k == 2 { + fakeClient.updateError = needError[2] + } else if k == 3 { + fakeClient.patchError = needError[3] + } else if k == 4 { + fakeClient.updateConflict = needError[4] + } else if k == 5 { + fakeClient.listError = needError[5] + } + } + + fakeKubeClient := clientgofake.NewSimpleClientset(daemonSet, node) + + return NewPodVolumeBackupReconciler( + fakeClient, + nil, + fakeKubeClient, + dataPathMgr, + "test-node", + time.Minute*5, + time.Minute, + corev1api.ResourceRequirements{}, + metrics.NewServerMetrics(), + velerotest.NewLogger(), + ), nil +} func pvbBuilder() *builder.PodVolumeBackupBuilder { - return builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, name). - PodNamespace(velerov1api.DefaultNamespace). - PodName(name). - Volume("pvb-1-volume"). - BackupStorageLocation("bsl-loc"). - ObjectMeta( - func(obj metav1.Object) { - obj.SetOwnerReferences([]metav1.OwnerReference{{Name: name}}) - }, - ) + return builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, pvbName).BackupStorageLocation("bsl-loc") } -func podBuilder() *builder.PodBuilder { - return builder. - ForPod(velerov1api.DefaultNamespace, name). - Volumes(&corev1api.Volume{Name: "pvb-1-volume"}) +type fakePvbExposer struct { + kubeClient client.Client + clock clock.WithTickerAndDelayedExecution + peekErr error + exposeErr error + getErr error + getNil bool } -func bslBuilder() *builder.BackupStorageLocationBuilder { - return builder. - ForBackupStorageLocation(velerov1api.DefaultNamespace, "bsl-loc") -} - -func buildBackupRepo() *velerov1api.BackupRepository { - return &velerov1api.BackupRepository{ - Spec: velerov1api.BackupRepositorySpec{ResticIdentifier: ""}, - TypeMeta: metav1.TypeMeta{ - APIVersion: velerov1api.SchemeGroupVersion.String(), - Kind: "BackupRepository", - }, - ObjectMeta: metav1.ObjectMeta{ - Namespace: velerov1api.DefaultNamespace, - Name: fmt.Sprintf("%s-bsl-loc-restic-dn24h", velerov1api.DefaultNamespace), - Labels: map[string]string{ - velerov1api.StorageLocationLabel: "bsl-loc", - velerov1api.VolumeNamespaceLabel: velerov1api.DefaultNamespace, - velerov1api.RepositoryTypeLabel: "restic", - }, - }, - Status: velerov1api.BackupRepositoryStatus{ - Phase: velerov1api.BackupRepositoryPhaseReady, - }, - } -} - -type fakeFSBR struct { - pvb *velerov1api.PodVolumeBackup - client kbclient.Client - clock clock.WithTickerAndDelayedExecution -} - -func (b *fakeFSBR) Init(ctx context.Context, param any) error { - return nil -} - -func (b *fakeFSBR) StartBackup(source datapath.AccessPoint, uploaderConfigs map[string]string, param any) error { - pvb := b.pvb - - original := b.pvb.DeepCopy() - pvb.Status.Phase = velerov1api.PodVolumeBackupPhaseCompleted - pvb.Status.CompletionTimestamp = &metav1.Time{Time: b.clock.Now()} - - b.client.Patch(ctx, pvb, kbclient.MergeFrom(original)) - - return nil -} - -func (b *fakeFSBR) StartRestore(snapshotID string, target datapath.AccessPoint, uploaderConfigs map[string]string) error { - return nil -} - -func (b *fakeFSBR) Cancel() { -} - -func (b *fakeFSBR) Close(ctx context.Context) { -} - -var _ = Describe("PodVolumeBackup Reconciler", func() { - type request struct { - pvb *velerov1api.PodVolumeBackup - pod *corev1api.Pod - bsl *velerov1api.BackupStorageLocation - backupRepo *velerov1api.BackupRepository - expectedProcessed bool - expected *velerov1api.PodVolumeBackup - expectedRequeue ctrl.Result - expectedErrMsg string - dataMgr *datapath.Manager +func (f *fakePvbExposer) Expose(ctx context.Context, ownerObject corev1api.ObjectReference, param exposer.PodVolumeExposeParam) error { + if f.exposeErr != nil { + return f.exposeErr } - // `now` will be used to set the fake clock's time; capture - // it here so it can be referenced in the test case defs. - now, err := time.Parse(time.RFC1123, time.RFC1123) - Expect(err).ToNot(HaveOccurred()) - now = now.Local() + return nil +} - DescribeTable("a pod volume backup", - func(test request) { - ctx := context.Background() +func (f *fakePvbExposer) GetExposed(context.Context, corev1api.ObjectReference, client.Client, string, time.Duration) (*exposer.ExposeResult, error) { + if f.getErr != nil { + return nil, f.getErr + } - fakeClient := fake.NewClientBuilder().WithScheme(scheme.Scheme).Build() - err = fakeClient.Create(ctx, test.pvb) - Expect(err).ToNot(HaveOccurred()) + if f.getNil { + return nil, nil + } - err = fakeClient.Create(ctx, test.pod) - Expect(err).ToNot(HaveOccurred()) + pod := &corev1api.Pod{} - err = fakeClient.Create(ctx, test.bsl) - Expect(err).ToNot(HaveOccurred()) + nodeOS := "linux" + pNodeOS := &nodeOS - err = fakeClient.Create(ctx, test.backupRepo) - Expect(err).ToNot(HaveOccurred()) + return &exposer.ExposeResult{ByPod: exposer.ExposeByPod{HostingPod: pod, VolumeName: pvbName, NodeOS: pNodeOS}}, nil +} - fakeFS := velerotest.NewFakeFileSystem() - pathGlob := fmt.Sprintf("/host_pods/%s/volumes/*/%s", "", "pvb-1-volume") - _, err = fakeFS.Create(pathGlob) - Expect(err).ToNot(HaveOccurred()) +func (f *fakePvbExposer) PeekExposed(ctx context.Context, ownerObject corev1api.ObjectReference) error { + return f.peekErr +} - credentialFileStore, err := credentials.NewNamespacedFileStore( - fakeClient, - velerov1api.DefaultNamespace, - "/tmp/credentials", - fakeFS, - ) +func (f *fakePvbExposer) DiagnoseExpose(context.Context, corev1api.ObjectReference) string { + return "" +} - Expect(err).ToNot(HaveOccurred()) +func (f *fakePvbExposer) CleanUp(context.Context, corev1api.ObjectReference) { +} - if test.dataMgr == nil { - test.dataMgr = datapath.NewManager(1) +func TestPVBReconcile(t *testing.T) { + tests := []struct { + name string + pvb *velerov1api.PodVolumeBackup + notCreatePvb bool + needDelete bool + sportTime *metav1.Time + pod *corev1api.Pod + dataMgr *datapath.Manager + needCreateFSBR bool + needExclusiveUpdateError error + needMockExposer bool + expected *velerov1api.PodVolumeBackup + expectDeleted bool + expectCancelRecord bool + needErrs []bool + peekErr error + exposeErr error + getExposeErr error + getExposeNil bool + fsBRInitErr error + fsBRStartErr error + expectedErr string + expectedResult *ctrl.Result + expectDataPath bool + }{ + { + name: "pvb not found", + pvb: pvbBuilder().Result(), + notCreatePvb: true, + }, + { + name: "pvb not created in velero default namespace", + pvb: builder.ForPodVolumeBackup("test-ns", pvbName).Result(), + }, + { + name: "get pvb fail", + pvb: pvbBuilder().Result(), + needErrs: []bool{true, false, false, false}, + expectedErr: "getting PVB: Get error", + }, + { + name: "add finalizer to pvb", + pvb: pvbBuilder().Result(), + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "add finalizer to pvb failed", + pvb: pvbBuilder().Result(), + needErrs: []bool{false, false, true, false}, + expectedErr: "error updating PVB with error velero/pvb-1: Update error", + }, + { + name: "pvb is under deletion", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Result(), + needDelete: true, + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Result(), + }, + { + name: "pvb is under deletion but cancel failed", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Result(), + needErrs: []bool{false, false, true, false}, + needDelete: true, + expectedErr: "error updating PVB with error velero/pvb-1: Update error", + }, + { + name: "pvb is under deletion and in terminal state", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeBackupPhaseFailed).Result(), + sportTime: &metav1.Time{Time: time.Now()}, + needDelete: true, + expectDeleted: true, + }, + { + name: "pvb is under deletion and in terminal state, but remove finalizer failed", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeBackupPhaseFailed).Result(), + needErrs: []bool{false, false, true, false}, + needDelete: true, + expectedErr: "error updating PVB with error velero/pvb-1: Update error", + }, + { + name: "delay cancel negative for others", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhasePrepared).Result(), + sportTime: &metav1.Time{Time: time.Now()}, + expectCancelRecord: true, + }, + { + name: "delay cancel negative for inProgress", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhaseInProgress).Result(), + sportTime: &metav1.Time{Time: time.Now().Add(-time.Minute * 58)}, + expectCancelRecord: true, + }, + { + name: "delay cancel affirmative for others", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhasePrepared).Result(), + sportTime: &metav1.Time{Time: time.Now().Add(-time.Minute * 5)}, + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhaseCanceled).Result(), + }, + { + name: "delay cancel affirmative for inProgress", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhaseInProgress).Result(), + sportTime: &metav1.Time{Time: time.Now().Add(-time.Hour)}, + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhaseCanceled).Result(), + }, + { + name: "delay cancel failed", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhaseInProgress).Result(), + needErrs: []bool{false, false, true, false}, + sportTime: &metav1.Time{Time: time.Now().Add(-time.Hour)}, + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhaseInProgress).Result(), + expectCancelRecord: true, + }, + { + name: "Unknown pvb status", + pvb: pvbBuilder().Phase("Unknown").Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "new pvb but accept failed", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needErrs: []bool{false, false, true, false}, + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Result(), + expectedErr: "error accepting PVB pvb-1: error updating PVB with error velero/pvb-1: Update error", + }, + { + name: "pvb is cancel on accepted", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Cancel(true).Result(), + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhaseCanceled).Result(), + expectCancelRecord: true, + }, + { + name: "pvb expose failed", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + exposeErr: errors.New("fake-expose-error"), + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeBackupPhaseFailed).Message("error to expose PVB").Result(), + expectedErr: "fake-expose-error", + }, + { + name: "pvb succeeds for accepted", + pvb: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + expected: pvbBuilder().Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeBackupPhaseAccepted).Result(), + }, + { + name: "prepare timeout on accepted", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseAccepted).Finalizers([]string{PodVolumeFinalizer}).AcceptedTimestamp(&metav1.Time{Time: time.Now().Add(-time.Minute * 30)}).Result(), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeBackupPhaseFailed).Message("timeout on preparing PVB").Result(), + }, + { + name: "peek error on accepted", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseAccepted).Finalizers([]string{PodVolumeFinalizer}).Result(), + needMockExposer: true, + peekErr: errors.New("fake-peak-error"), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseCanceled).Finalizers([]string{PodVolumeFinalizer}).Phase(velerov1api.PodVolumeBackupPhaseCanceled).Message("found a PVB velero/pvb-1 with expose error: fake-peak-error. mark it as cancel").Result(), + }, + { + name: "cancel on prepared", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Cancel(true).Result(), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseCanceled).Finalizers([]string{PodVolumeFinalizer}).Cancel(true).Phase(velerov1api.PodVolumeBackupPhaseCanceled).Result(), + }, + { + name: "Failed to get pvb expose on prepared", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + getExposeErr: errors.New("fake-get-error"), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Message("exposed PVB is not ready: fake-get-error").Result(), + expectedErr: "fake-get-error", + }, + { + name: "Get nil restore expose on prepared", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + getExposeNil: true, + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Message("exposed PVB is not ready").Result(), + expectedErr: "no expose result is available for the current node", + }, + { + name: "Error in data path is concurrent limited", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + dataMgr: datapath.NewManager(0), + expectedResult: &ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, + }, + { + name: "data path init error", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + fsBRInitErr: errors.New("fake-data-path-init-error"), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Message("error initializing data path").Result(), + expectedErr: "error initializing asyncBR: fake-data-path-init-error", + }, + { + name: "Unable to update status to in progress for data upload", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + needErrs: []bool{false, false, true, false}, + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Result(), + expectedResult: &ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, + }, + { + name: "data path start error", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + fsBRStartErr: errors.New("fake-data-path-start-error"), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseFailed).Finalizers([]string{PodVolumeFinalizer}).Message("error starting data path").Result(), + expectedErr: "error starting async backup for pod , volume pvb-1: fake-data-path-start-error", + }, + { + name: "Prepare succeeds", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhasePrepared).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needMockExposer: true, + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Result(), + expectDataPath: true, + }, + { + name: "In progress pvb is not handled by the current node", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Result(), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "In progress pvb is not set as cancel", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "Cancel pvb in progress with empty FSBR", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseCanceled).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Result(), + }, + { + name: "Cancel pvb in progress and patch pvb error", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needErrs: []bool{false, false, true, false}, + needCreateFSBR: true, + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Result(), + expectedErr: "error updating PVB with error velero/pvb-1: Update error", + expectCancelRecord: true, + expectDataPath: true, + }, + { + name: "Cancel pvb in progress succeeds", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Node("test-node").Result(), + needCreateFSBR: true, + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseCanceling).Cancel(true).Finalizers([]string{PodVolumeFinalizer}).Result(), + expectDataPath: true, + expectCancelRecord: true, + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + r, err := initPVBReconciler(test.needErrs...) + require.NoError(t, err) + + if !test.notCreatePvb { + err = r.client.Create(context.Background(), test.pvb) + require.NoError(t, err) } - datapath.FSBRCreator = func(string, string, kbclient.Client, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { - return &fakeFSBR{ - pvb: test.pvb, - client: fakeClient, - clock: testclocks.NewFakeClock(now), + if test.needDelete { + err = r.client.Delete(context.Background(), test.pvb) + require.NoError(t, err) + } + + if test.pod != nil { + err = r.client.Create(ctx, test.pod) + require.NoError(t, err) + } + + if test.dataMgr != nil { + r.dataPathMgr = test.dataMgr + } else { + r.dataPathMgr = datapath.NewManager(1) + } + + if test.sportTime != nil { + r.cancelledPVB[test.pvb.Name] = test.sportTime.Time + } + + if test.needMockExposer { + r.exposer = &fakePvbExposer{r.client, r.clock, test.peekErr, test.exposeErr, test.getExposeErr, test.getExposeNil} + } + + funcExclusiveUpdatePodVolumeBackup = exclusiveUpdatePodVolumeBackup + if test.needExclusiveUpdateError != nil { + funcExclusiveUpdatePodVolumeBackup = func(context.Context, client.Client, *velerov1api.PodVolumeBackup, func(*velerov1api.PodVolumeBackup)) (bool, error) { + return false, test.needExclusiveUpdateError } } - // Setup reconciler - Expect(velerov1api.AddToScheme(scheme.Scheme)).To(Succeed()) - r := PodVolumeBackupReconciler{ - Client: fakeClient, - clock: testclocks.NewFakeClock(now), - metrics: metrics.NewNodeMetrics(), - credentialGetter: &credentials.CredentialGetter{FromFile: credentialFileStore}, - nodeName: "test_node", - fileSystem: fakeFS, - logger: velerotest.NewLogger(), - dataPathMgr: test.dataMgr, + datapath.MicroServiceBRWatcherCreator = func(client.Client, kubernetes.Interface, manager.Manager, string, string, string, string, string, string, datapath.Callbacks, logrus.FieldLogger) datapath.AsyncBR { + return &fakeFSBR{ + kubeClient: r.client, + clock: r.clock, + initErr: test.fsBRInitErr, + startErr: test.fsBRStartErr, + } + } + + if test.needCreateFSBR { + if fsBR := r.dataPathMgr.GetAsyncBR(test.pvb.Name); fsBR == nil { + _, err := r.dataPathMgr.CreateMicroServiceBRWatcher(ctx, r.client, nil, nil, datapath.TaskTypeBackup, test.pvb.Name, velerov1api.DefaultNamespace, "", "", "", datapath.Callbacks{OnCancelled: r.OnDataPathCancelled}, false, velerotest.NewLogger()) + require.NoError(t, err) + } } actualResult, err := r.Reconcile(ctx, ctrl.Request{ @@ -204,184 +508,419 @@ var _ = Describe("PodVolumeBackup Reconciler", func() { Name: test.pvb.Name, }, }) - Expect(actualResult).To(BeEquivalentTo(test.expectedRequeue)) - if test.expectedErrMsg == "" { - Expect(err).ToNot(HaveOccurred()) + + if test.expectedErr != "" { + assert.EqualError(t, err, test.expectedErr) } else { - Expect(err.Error()).To(BeEquivalentTo(test.expectedErrMsg)) + assert.NoError(t, err) } - pvb := velerov1api.PodVolumeBackup{} - err = r.Client.Get(ctx, kbclient.ObjectKey{ - Name: test.pvb.Name, - Namespace: test.pvb.Namespace, - }, &pvb) - // Assertions - if test.expected == nil { - Expect(apierrors.IsNotFound(err)).To(BeTrue()) + if test.expectedResult != nil { + assert.Equal(t, test.expectedResult.Requeue, actualResult.Requeue) + assert.Equal(t, test.expectedResult.RequeueAfter, actualResult.RequeueAfter) + } + + if test.expected != nil || test.expectDeleted { + pvb := velerov1api.PodVolumeBackup{} + err = r.client.Get(ctx, client.ObjectKey{ + Name: test.pvb.Name, + Namespace: test.pvb.Namespace, + }, &pvb) + + if test.expectDeleted { + assert.True(t, apierrors.IsNotFound(err)) + } else { + require.NoError(t, err) + + assert.Equal(t, test.expected.Status.Phase, pvb.Status.Phase) + assert.Contains(t, pvb.Status.Message, test.expected.Status.Message) + assert.Equal(t, pvb.Finalizers, test.expected.Finalizers) + assert.Equal(t, pvb.Spec.Cancel, test.expected.Spec.Cancel) + } + } + + if !test.expectDataPath { + assert.Nil(t, r.dataPathMgr.GetAsyncBR(test.pvb.Name)) } else { - Expect(err).ToNot(HaveOccurred()) - Eventually(pvb.Status.Phase).Should(Equal(test.expected.Status.Phase)) + assert.NotNil(t, r.dataPathMgr.GetAsyncBR(test.pvb.Name)) } - // Processed PVBs will have completion timestamps. - if test.expectedProcessed == true { - Expect(pvb.Status.CompletionTimestamp).ToNot(BeNil()) + if test.expectCancelRecord { + assert.Contains(t, r.cancelledPVB, test.pvb.Name) + } else { + assert.Empty(t, r.cancelledPVB) } + }) + } +} - // Unprocessed PVBs will not have completion timestamps. - if test.expectedProcessed == false { - Expect(pvb.Status.CompletionTimestamp).To(BeNil()) - } +func TestOnPVBCancelled(t *testing.T) { + ctx := context.TODO() + r, err := initPVBReconciler() + require.NoError(t, err) + pvb := pvbBuilder().Result() + namespace := pvb.Namespace + pvbName := pvb.Name + + assert.NoError(t, r.client.Create(ctx, pvb)) + + r.OnDataPathCancelled(ctx, namespace, pvbName) + updatedPvb := &velerov1api.PodVolumeBackup{} + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, updatedPvb)) + assert.Equal(t, velerov1api.PodVolumeBackupPhaseCanceled, updatedPvb.Status.Phase) + assert.False(t, updatedPvb.Status.CompletionTimestamp.IsZero()) + assert.False(t, updatedPvb.Status.StartTimestamp.IsZero()) +} + +func TestOnPVBProgress(t *testing.T) { + totalBytes := int64(1024) + bytesDone := int64(512) + tests := []struct { + name string + pvb *velerov1api.PodVolumeBackup + progress uploader.Progress + needErrs []bool + }{ + { + name: "patch in progress phase success", + pvb: pvbBuilder().Result(), + progress: uploader.Progress{ + TotalBytes: totalBytes, + BytesDone: bytesDone, + }, }, - Entry("empty phase pvb on same node should be processed", request{ - pvb: pvbBuilder().Phase("").Node("test_node").Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: true, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseCompleted). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("new phase pvb on same node should be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseNew). - Node("test_node"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: true, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseCompleted). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("in progress phase pvb on same node should not be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseInProgress). - Node("test_node"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseInProgress). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("completed phase pvb on same node should not be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseCompleted). - Node("test_node"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseCompleted). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("failed phase pvb on same node should not be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseFailed). - Node("test_node"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseFailed). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("empty phase pvb on different node should not be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseFailed). - Node("test_node_2"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseFailed). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("new phase pvb on different node should not be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseNew). - Node("test_node_2"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseNew). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("in progress phase pvb on different node should not be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseInProgress). - Node("test_node_2"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseInProgress). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("completed phase pvb on different node should not be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseCompleted). - Node("test_node_2"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseCompleted). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("failed phase pvb on different node should not be processed", request{ - pvb: pvbBuilder(). - Phase(velerov1api.PodVolumeBackupPhaseFailed). - Node("test_node_2"). - Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(velerov1api.PodVolumeBackupPhaseFailed). - Result(), - expectedRequeue: ctrl.Result{}, - }), - Entry("pvb should be requeued when exceeding max concurrent number", request{ - pvb: pvbBuilder().Phase("").Node("test_node").Result(), - pod: podBuilder().Result(), - bsl: bslBuilder().Result(), - backupRepo: buildBackupRepo(), - dataMgr: datapath.NewManager(0), - expectedProcessed: false, - expected: builder.ForPodVolumeBackup(velerov1api.DefaultNamespace, "pvb-1"). - Phase(""). - Result(), - expectedRequeue: ctrl.Result{Requeue: true, RequeueAfter: time.Second * 5}, - }), - ) -}) + { + name: "failed to get pvb", + pvb: pvbBuilder().Result(), + needErrs: []bool{true, false, false, false}, + }, + { + name: "failed to patch pvb", + pvb: pvbBuilder().Result(), + needErrs: []bool{false, false, true, false}, + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + ctx := context.TODO() + + r, err := initPVBReconciler(test.needErrs...) + require.NoError(t, err) + defer func() { + r.client.Delete(ctx, test.pvb, &client.DeleteOptions{}) + }() + + pvb := pvbBuilder().Result() + namespace := pvb.Namespace + pvbName := pvb.Name + + assert.NoError(t, r.client.Create(context.Background(), pvb)) + + // Create a Progress object + progress := &uploader.Progress{ + TotalBytes: totalBytes, + BytesDone: bytesDone, + } + + r.OnDataPathProgress(ctx, namespace, pvbName, progress) + if len(test.needErrs) != 0 && !test.needErrs[0] { + + updatedPvb := &velerov1api.PodVolumeBackup{} + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, updatedPvb)) + assert.Equal(t, test.progress.TotalBytes, updatedPvb.Status.Progress.TotalBytes) + assert.Equal(t, test.progress.BytesDone, updatedPvb.Status.Progress.BytesDone) + } + }) + } +} + +func TestOnPvbFailed(t *testing.T) { + ctx := context.TODO() + r, err := initPVBReconciler() + require.NoError(t, err) + + pvb := pvbBuilder().Result() + namespace := pvb.Namespace + pvbName := pvb.Name + + assert.NoError(t, r.client.Create(ctx, pvb)) + + r.OnDataPathFailed(ctx, namespace, pvbName, fmt.Errorf("Failed to handle %v", pvbName)) + updatedPvb := &velerov1api.PodVolumeBackup{} + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, updatedPvb)) + assert.Equal(t, velerov1api.PodVolumeBackupPhaseFailed, updatedPvb.Status.Phase) + assert.False(t, updatedPvb.Status.CompletionTimestamp.IsZero()) + assert.False(t, updatedPvb.Status.StartTimestamp.IsZero()) +} + +func TestOnPvbCompleted(t *testing.T) { + ctx := context.TODO() + r, err := initPVBReconciler() + require.NoError(t, err) + + now := time.Now() + pvb := pvbBuilder().StartTimestamp(&metav1.Time{Time: now.Add(-time.Minute)}).CompletionTimestamp(&metav1.Time{Time: now}).OwnerReference(metav1.OwnerReference{Name: "test-backup"}).Result() + namespace := pvb.Namespace + pvbName := pvb.Name + + assert.NoError(t, r.client.Create(ctx, pvb)) + + r.OnDataPathCompleted(ctx, namespace, pvbName, datapath.Result{}) + updatedPvb := &velerov1api.PodVolumeBackup{} + assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, updatedPvb)) + assert.Equal(t, velerov1api.PodVolumeBackupPhaseCompleted, updatedPvb.Status.Phase) + assert.False(t, updatedPvb.Status.CompletionTimestamp.IsZero()) +} + +func TestFindPvbForPod(t *testing.T) { + r, err := initPVBReconciler() + require.NoError(t, err) + tests := []struct { + name string + pvb *velerov1api.PodVolumeBackup + pod *corev1api.Pod + checkFunc func(*velerov1api.PodVolumeBackup, []reconcile.Request) + }{ + { + name: "find pvb for pod", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseAccepted).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, pvbName).Labels(map[string]string{velerov1api.PVBLabel: pvbName}).Status(corev1api.PodStatus{Phase: corev1api.PodRunning}).Result(), + checkFunc: func(pvb *velerov1api.PodVolumeBackup, requests []reconcile.Request) { + // Assert that the function returns a single request + assert.Len(t, requests, 1) + // Assert that the request contains the correct namespaced name + assert.Equal(t, pvb.Namespace, requests[0].Namespace) + assert.Equal(t, pvb.Name, requests[0].Name) + }, + }, { + name: "no selected label found for pod", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseAccepted).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, pvbName).Result(), + checkFunc: func(pvb *velerov1api.PodVolumeBackup, requests []reconcile.Request) { + // Assert that the function returns a single request + assert.Empty(t, requests) + }, + }, { + name: "no matched pod", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseAccepted).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, pvbName).Labels(map[string]string{velerov1api.PVBLabel: "non-existing-pvb"}).Result(), + checkFunc: func(pvb *velerov1api.PodVolumeBackup, requests []reconcile.Request) { + assert.Empty(t, requests) + }, + }, + { + name: "pvb not accepte", + pvb: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseInProgress).Result(), + pod: builder.ForPod(velerov1api.DefaultNamespace, pvbName).Labels(map[string]string{velerov1api.PVBLabel: pvbName}).Result(), + checkFunc: func(pvb *velerov1api.PodVolumeBackup, requests []reconcile.Request) { + assert.Empty(t, requests) + }, + }, + } + for _, test := range tests { + ctx := context.Background() + assert.NoError(t, r.client.Create(ctx, test.pod)) + assert.NoError(t, r.client.Create(ctx, test.pvb)) + + requests := r.findPVBForPod(context.Background(), test.pod) + test.checkFunc(test.pvb, requests) + r.client.Delete(ctx, test.pvb, &client.DeleteOptions{}) + if test.pod != nil { + r.client.Delete(ctx, test.pod, &client.DeleteOptions{}) + } + } +} + +func TestAcceptPvb(t *testing.T) { + tests := []struct { + name string + pvb *velerov1api.PodVolumeBackup + needErrs []error + expectedErr string + }{ + { + name: "update fail", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, fmt.Errorf("fake-update-error"), nil}, + expectedErr: "fake-update-error", + }, + { + name: "accepted by others", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, &fakeAPIStatus{metav1.StatusReasonConflict}, nil}, + }, + { + name: "succeed", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, nil, nil}, + }, + } + for _, test := range tests { + ctx := context.Background() + r, err := initPVBReconcilerWithError(test.needErrs...) + require.NoError(t, err) + + err = r.client.Create(ctx, test.pvb) + require.NoError(t, err) + + err = r.acceptPodVolumeBackup(ctx, test.pvb) + if test.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, test.expectedErr) + } + } +} + +func TestOnPvbPrepareTimeout(t *testing.T) { + tests := []struct { + name string + pvb *velerov1api.PodVolumeBackup + needErrs []error + expected *velerov1api.PodVolumeBackup + }{ + { + name: "update fail", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, fmt.Errorf("fake-update-error"), nil}, + expected: pvbBuilder().Result(), + }, + { + name: "update interrupted", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, &fakeAPIStatus{metav1.StatusReasonConflict}, nil}, + expected: pvbBuilder().Result(), + }, + { + name: "succeed", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, nil, nil}, + expected: pvbBuilder().Phase(velerov1api.PodVolumeBackupPhaseFailed).Result(), + }, + } + for _, test := range tests { + ctx := context.Background() + r, err := initPVBReconcilerWithError(test.needErrs...) + require.NoError(t, err) + + err = r.client.Create(ctx, test.pvb) + require.NoError(t, err) + + r.onPrepareTimeout(ctx, test.pvb) + + pvb := velerov1api.PodVolumeBackup{} + _ = r.client.Get(ctx, client.ObjectKey{ + Name: test.pvb.Name, + Namespace: test.pvb.Namespace, + }, &pvb) + + assert.Equal(t, test.expected.Status.Phase, pvb.Status.Phase) + } +} + +func TestTryCancelPvb(t *testing.T) { + tests := []struct { + name string + pvb *velerov1api.PodVolumeBackup + needErrs []error + succeeded bool + expectedErr string + }{ + { + name: "update fail", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, fmt.Errorf("fake-update-error"), nil}, + }, + { + name: "cancel by others", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, &fakeAPIStatus{metav1.StatusReasonConflict}, nil}, + }, + { + name: "succeed", + pvb: pvbBuilder().Result(), + needErrs: []error{nil, nil, nil, nil}, + succeeded: true, + }, + } + for _, test := range tests { + ctx := context.Background() + r, err := initPVBReconcilerWithError(test.needErrs...) + require.NoError(t, err) + + err = r.client.Create(ctx, test.pvb) + require.NoError(t, err) + + r.tryCancelPodVolumeBackup(ctx, test.pvb, "") + + if test.expectedErr == "" { + assert.NoError(t, err) + } else { + assert.EqualError(t, err, test.expectedErr) + } + } +} + +func TestUpdatePvbWithRetry(t *testing.T) { + namespacedName := types.NamespacedName{ + Name: pvbName, + Namespace: "velero", + } + + // Define test cases + testCases := []struct { + Name string + needErrs []bool + noChange bool + ExpectErr bool + }{ + { + Name: "SuccessOnFirstAttempt", + }, + { + Name: "Error get", + needErrs: []bool{true, false, false, false, false}, + ExpectErr: true, + }, + { + Name: "Error update", + needErrs: []bool{false, false, true, false, false}, + ExpectErr: true, + }, + { + Name: "no change", + noChange: true, + needErrs: []bool{false, false, true, false, false}, + }, + { + Name: "Conflict with error timeout", + needErrs: []bool{false, false, false, false, true}, + ExpectErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.Name, func(t *testing.T) { + ctx, cancelFunc := context.WithTimeout(context.TODO(), time.Second*5) + defer cancelFunc() + r, err := initPVBReconciler(tc.needErrs...) + require.NoError(t, err) + err = r.client.Create(ctx, pvbBuilder().Result()) + require.NoError(t, err) + updateFunc := func(pvb *velerov1api.PodVolumeBackup) bool { + if tc.noChange { + return false + } + + pvb.Spec.Cancel = true + return true + } + err = UpdatePVBWithRetry(ctx, r.client, namespacedName, velerotest.NewLogger().WithField("name", tc.Name), updateFunc) + if tc.ExpectErr { + assert.Error(t, err) + } else { + assert.NoError(t, err) + } + }) + } +} diff --git a/pkg/podvolume/backupper.go b/pkg/podvolume/backupper.go index 776cdc332..c989a363a 100644 --- a/pkg/podvolume/backupper.go +++ b/pkg/podvolume/backupper.go @@ -169,7 +169,8 @@ func newBackupper( } if pvb.Status.Phase != velerov1api.PodVolumeBackupPhaseCompleted && - pvb.Status.Phase != velerov1api.PodVolumeBackupPhaseFailed { + pvb.Status.Phase != velerov1api.PodVolumeBackupPhaseFailed && + pvb.Status.Phase != velerov1api.PodVolumeBackupPhaseCanceled { return } @@ -179,7 +180,8 @@ func newBackupper( existPVB, ok := existObj.(*velerov1api.PodVolumeBackup) // the PVB in the indexer is already in final status, no need to call WaitGroup.Done() if ok && (existPVB.Status.Phase == velerov1api.PodVolumeBackupPhaseCompleted || - existPVB.Status.Phase == velerov1api.PodVolumeBackupPhaseFailed) { + existPVB.Status.Phase == velerov1api.PodVolumeBackupPhaseFailed || + pvb.Status.Phase != velerov1api.PodVolumeBackupPhaseCanceled) { statusChangedToFinal = false } } @@ -428,7 +430,7 @@ func (b *backupper) WaitAllPodVolumesProcessed(log logrus.FieldLogger) []*velero continue } podVolumeBackups = append(podVolumeBackups, pvb) - if pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseFailed { + if pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseFailed || pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseCanceled { log.Errorf("pod volume backup failed: %s", pvb.Status.Message) } } From c3a8c89ae32f413905ca037e1364cba14d8f5ff2 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Wed, 11 Jun 2025 09:30:20 +0000 Subject: [PATCH 28/31] vgdp ms pvb controller Signed-off-by: Lyndon-Li --- changelogs/unreleased/9015-Lyndon-Li | 1 + config/crd/v1/crds/crds.go | 2 +- pkg/controller/pod_volume_backup_controller.go | 4 ++-- pkg/controller/pod_volume_backup_controller_test.go | 12 +++--------- pkg/podvolume/backupper.go | 2 +- 5 files changed, 8 insertions(+), 13 deletions(-) create mode 100644 changelogs/unreleased/9015-Lyndon-Li diff --git a/changelogs/unreleased/9015-Lyndon-Li b/changelogs/unreleased/9015-Lyndon-Li new file mode 100644 index 000000000..51021f8c0 --- /dev/null +++ b/changelogs/unreleased/9015-Lyndon-Li @@ -0,0 +1 @@ +Fix issue #8958, add VGDP MS PVB controller \ No newline at end of file diff --git a/config/crd/v1/crds/crds.go b/config/crd/v1/crds/crds.go index 986184f6f..b93ec3a2f 100644 --- a/config/crd/v1/crds/crds.go +++ b/config/crd/v1/crds/crds.go @@ -35,7 +35,7 @@ var rawCRDs = [][]byte{ []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcVMo\x1b7\x10\xbd\xebW\f\xd0kwU\xa3hQ\xec\xadqr0\xda\x06\x82\x1d\xe4N\x91#-c.\xc9\xce\f\xe5\xba\x1f\xff\xbd \xb9+K\xab\x95\x93\\\xb27\x91Ù\xc7\xf7f\x1e\xd54\xcdJE\xfb\x11\x89m\xf0\x1d\xa8h\xf1/A\x9f\x7fq\xfb\xf8\v\xb76\xac\x0f7\xabG\xebM\a\xb7\x89%\f\xf7\xc8!\x91Ʒ\xb8\xb3ފ\r~5\xa0(\xa3Du+\x00\xe5}\x10\x95\x979\xff\x04\xd0\xc1\v\x05琚=\xfa\xf61mq\x9b\xac3H%\xf9T\xfa\xf0C{\xf3s\xfb\xd3\n\xc0\xab\x01;0\xe8Pp\xab\xf4c\x8a\x84\x7f&d\xe1\xf6\x80\x0e)\xb46\xac8\xa2\xce\xf9\xf7\x14R\xec\xe0e\xa3\x9e\x1fkW\xdcoK\xaa7%\xd5}MUv\x9de\xf9\xedZ\xc4\xefv\x8c\x8a.\x91rˀJ\x00[\xbfON\xd1b\xc8\n\x80u\x88\xd8\xc1\xfb\f+*\x8df\x050^\xbb\xc0l@\x19S\x88TnC\xd6\v\xd2mpi\x98\bl\xc0 k\xb2Q\nQ\x1fz,W\x84\xb0\x03\xe9\x11j9\x90\x00[\x1c\x11\x98r\x0e\xe0\x13\a\xbfQ\xd2w\xd0f\xbe\xda\x1a\x9a\x81\x8c\x01\x95\xea7\xf3ey\u0380Y\xc8\xfa\xfd5\b,J\x12O J]\x1b<\xd0\t\xbf\xe7\x00J|\x1b{\xc5\xe7\xd5\x1f\xcaƵ\xca5\xe6pS\x99\xd6=\x0e\xaa\x1bcCD\xff\xeb\xe6\xee\xe3\x8f\x0fg\xcbp\x8euAZ\xb0\fjB\x9a\x89\xab\xacA\xf0\b\x81`\b4\xb1\xca\xed1i\xa4\x10\x91\xc4N\xadU\xbf\x93\xe19Y\x9dA\xf8\xb79\xdb\x03Ȩ\xeb)0y\x8a\x90\v\x89cS\xa0\x19/Zɵ\f\x84\x91\x90\xd1\u05f9\xca\xcb\xcaC\xd8~B-\xed,\xf5\x03RN\x03܇\xe4L\x1e\xbe\x03\x92\x00\xa1\x0e{o\xff>\xe6\xe6|\xef\\\xd4))\x94\xe4\xb6\xf3\xca\xc1A\xb9\x84߃\xf2f\x96yP\xcf@\x98kB\xf2'\xf9\xca\x01\x9e\xe3\xf8#\x93h\xfd.tЋD\xee\xd6뽕\xc9Rt\x18\x86\xe4\xad<\xaf\x8b;\xd8m\x92@\xbc6x@\xb7f\xbbo\x14\xe9\xde\njI\x84k\x15mS.⋭\xb4\x83\xf9\x8eF\x13Ⳳ\x17\xddS\xbf\xe2\x02_!O\xf6\x84\xda#5U\xbd\xe2\x8b\ny)Sw\xff\xee\xe1\x03LH\xaaRU\x94\x97\xd0\v^&}2\x9b\xd6\xef\x90\xea\xb9\x1d\x85\xa1\xe4Dob\xb0^\xca\x0f\xed,z\x01N\xdb\xc1\nO\x1d\x9b\xa5\x9b\xa7\xbd-\xb6\x9b\x1d E\xa3\x04\xcd<\xe0\xceí\x1a\xd0\xdd*\xc6o\xacUV\x85\x9b,\xc2\x17\xa9u\xfa\x98̃+\xbd'\x1b\xd33pEڅ\xe1\x7f\x88\xa8\xb3\xb8\x99\xdf|\xda\ueb2ec\xb5\v\x04O\xbd\xd5\xfd4\xfc3\x9a\x8eFq\xce߲1\xe4\xef\xc5n\xe7;W/\x0fEdK8k\xd8\x06.\xbc\xfbu^\x8a\xa9~%3\xd5\xd1Gnt\"*\xcdw\xf4y\xb5t\xe8K\xb9@\xa2@\x17\xab3P\xefJP\xf9Ǡ\xacgP\xfey<\b\xd2+\x81'\xa4\r\x97\x95\x1ax\x8fO\v\xabw~CaO\xc8\xf3\x96ϛ\x9b\xca\x1e\xce߃WXZlʋE\xceVhNXd\t\xa4\xf6\xa7\xbcr\xda\x1e\x9d\xbe\x83\x7f\xfe[\xfd\x1f\x00\x00\xff\xff\xbeM\x1a\xea\xb1\n\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcWMo\xe36\x10\xbd\xfbW\f\xd0K\v\xac\xe4\x06E\x8b·\xd6\xd9C\xb0\xe96\x88\xb7\xb9S\xd4HbC\x91,9t6E\x7f|1\xa4\xe4\x0fYv\x9c\xcb\xea\xe6\xe1p\xf8\xe6\xcd\xcc#]\x14\xc5B8\xf5\x84>(kV \x9c¯\x84\x86\x7f\x85\xf2\xf9\xd7P*\xbb\xdc\xde,\x9e\x95\xa9W\xb0\x8e\x81l\xff\x88\xc1F/\xf1\x16\x1be\x14)k\x16=\x92\xa8\x05\x89\xd5\x02@\x18cI\xb09\xf0O\x00i\ry\xab5\xfa\xa2ES>\xc7\n\xab\xa8t\x8d>\x05\x1f\x8f\xde\xfeX\xde\xfcR\xfe\xbc\x000\xa2\xc7\x15\xd4\xf6\xc5h+j\x8f\xffD\f\x14\xca-j\xf4\xb6Tv\x11\x1cJ\x8e\xddz\x1b\xdd\n\xf6\vy\xefpn\xc6|;\x84y\xccaҊV\x81>ͭޫ\xc1\xc3\xe9\xe8\x85>\x05\x91\x16\x832m\xd4\u009f,/\x00\x82\xb4\x0eW\xf0\x99a8!\xb1^\x00\f)&XŐ\xdd\xf6&\x87\x92\x1d\xf6\"\xe3\x05\xb0\x0e\xcdo\x0fwO?m\x8e\xcc\x005\x06镣D\xd4\x7f\xc5\xce\x0e\xd3\x04@\x05\x100\xc0\x01\xb2;\x84 \f\bO\xaa\x11\x92\xa0\xf1\xb6\x87J\xc8\xe7\xe8\xc0V\x7f\xa3$\bd\xbdh\xf1\x03\x84(;\x10\x1c%;\x1c\x9c\xa5m\v\x8d\xd2X\xeel\xce[\x87\x9e\xd4Hy\xfe\x0e\x1a\xea\xc0z)\v\xfe8\xf1\xbc\vj\xee,\f@\x1d\x8e\xe4a=p\x05\xb6\x01\xeaT\x00\x8f\xcec@\x93{\x8d\xcd\xc2\fٔ\x93\xd0\x1b\xf4\x1c\x06Bg\xa3\xae\xb9!\xb7\xe8\t\x1aE\xaf\xcb41\xaa\x8ad}XָE\xbd\f\xaa-\x84\x97\x9d\"\x94\x14=.\x85SEJĤQ+\xfb\xfa;?\ff8:\x96^\xb9!\x03yeڃ\x854\x1d\xef(\x0f\xcfK\xee\xae\x1c*\xa7\xb8\xaf\x02\x9b\x98\xbaǏ\x9b/0\"ɕ\x1aZl\xe7z\xc2\xcbX\x1ffS\x99\x06}ޗڔc\xa2\xa9\x9dU\x86\xd2\x0f\xa9\x15\x1a\x82\x10\xab^Q\x18{\x9dK7\r\xbbNR\x04\x15Bt\xb5 \xac\xa7\x0ew\x06֢G\xbd\x16\x01\xbfq\xad\xb8*\xa1\xe0\"\\U\xadC\x81\x9d:gz\x0f\x16Fy&j^\x01\x128\xe1[\xa4\xa9u\x82\xe5Kr\xe2\xe3_:q,X\xdfcٖ\xac9a\x00\x92\xf5\xe8\x87i\xa1.a\x80\xd9F\x9fE2\xf67\xd3\xc0\xbc\xb2\xa0\xb0\xd8\x1db:=\x9a?4\xb1\x9f?\xa0\x80\xdf\x13\xe6{\xdb^\\_[C<\x17\x17\x9d\x9e\xac\x8e=n\x8cp\xa1\xb3o\xf8\xde\x11\xf6\x7f:\xf4\xf9\x1a\xbe\xe8:\xde滫\xef\x82c\xd4g\xcf}D\xbeA\xf0|\xa6\x83\xc3UQ\xae\xc04x^\x95\xe8zs\xf7\x1e\nϸ\xbf\xa3Hw\xa6\xb1o\xa4\xb8w\x9c\xf5;#\x03\xe3\x97\xde\x10o\xf74\xbfBƞ\xe6-\xf9\xeeD\xf8\x14+\xf4\x06\t\xc3^\xa9_\x14u\xb3\x11\x01^:%\xbb\xb41\r\x04_\x02!X\xa9\xe6$\xf5\n\xf8\xac#\xca\xe3\xccP\x16iXg\xcc\f\xfe\xc4|F\xfd\xce\x1dP\f\x8at\x95\x82\x92\xa0\x18ޡ\xa1\xc9\x7f\xa4ZF\xef\xd3\x15\x95\xad\xfc2\x99n\xb8VDG\xe5\xf9\xeb\xf1\xfe\r%\xbd\xdd{\xa6\x17\xb7P&\xa3q\x1e\x8b\xa0Z~A\xf1\x1akiҸS2\xf2w\xfc\xc2;&j\xb6\xa2\xf8թ<\x80o@\xfc\xb8ŝ\x8f&\xdf\xf3\xd37l\n\x88\x81\x9f[ \x85\x99\xc1X!Ԩ\x91\xb0\x86\xea5\xdf\\\xaf\x81\xb0?\xc5\xddX\xdf\vZ\x01\xdf\xff\x05\xa9\x9962QkQi\\\x01\xf9x\xae\xcbf\x13w\x9d\b3cx\x94\xf3\x03\xfb\xcc5\xc6n\x18/v\x06\x9c\xbd_\n\xf8\x8c/3\xd6\ao%\x86\x80\xa7ct6\x93\xd9!81\x06~\xa4\xd5\a,\r\x7f\x19\x06\xcb\xff\x01\x00\x00\xff\xffx\xae@\xbaJ\x0e\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4ZK\x93\x1b\xb7\x11\xbe\xef\xaf\xe8Z\x1flWi\xc8HI\\)ޤU\x9c\xda\xc4V\xb6ĕ..\x1f\xc0A\x93\x03\xef\f\x80\x00\x18R\x8c\xe3\xff\x9ej<\x86\xf3\x00\xc9]\xca\xf2\xe2\xb2K<\x1a\x8d\xaf\x1b\xfd\xc2\x14EqŴ\xf8\x88\xc6\n%\x17\xc0\xb4\xc0O\x0e%\xfd\xb2\xb3\x87\xbfٙP\xf3\xed˫\a!\xf9\x02nZ\xebT\xf3\x1e\xadjM\x89oq-\xa4pBɫ\x06\x1d\xe3̱\xc5\x15\x00\x93R9Fݖ~\x02\x94J:\xa3\xea\x1aM\xb1A9{hW\xb8jE\xcd\xd1x\xe2i\xeb\xed\x9ff/\xbf\x9b\xfd\xf5\n@\xb2\x06\x17\xa0\x15ߪ\xbamp\xc5ʇV\xdb\xd9\x16k4j&ԕ\xd5X\x12\xed\x8dQ\xad^\xc0a \xac\x8d\xfb\x06\x9e\xef\x14\xff\xe8ɼ\xf1d\xfcH-\xac\xfbWn\xf4\aa\x9d\x9f\xa1\xebְzʄ\x1f\xb4Bnښ\x99\xc9\xf0\x15\x80-\x95\xc6\x05\xbc#64+\x91_\x01\xc4#z\xb6\n`\x9c{\xd0X}g\x84thn\x88B\x02\xab\x00\x8e\xb64B;\x0fʈ?\xb0\x8e\xb9ւm\xcb\n\x98\x85w\xb8\x9b\xdf\xca;\xa36\x06m`\x0e\xe0\x17\xab\xe4\x1ds\xd5\x02fa\xfaLW\xccb\x1c\r\xe0.\xfd@\xecr{b\xd9:#\xe4&\xc7Ľh\x10xk\xbcP\xe9\xf4%\x82\xab\x84\x9dp\xb7c\x9684\xce\x1f;ϋ\x1f'\x8aֱF\x8f\x99\xea-\r\\q\xe60\xc7Ӎjt\x8d\x0e9\xac\xf6\x0e\xd3I\xd6\xca4\xcc-@H\xf7\xdd_\x8e\xc3\x11\xf1\x9a\xf9\xa5o\x95\x1cb\xf3\x86z\xa1\xd7\x1d8!Ym\xd0d\x01R\x8e՟È#\x02oz\xeb\x03'\x81n\xbf\xff,+\xa4x\xa0\xd6\xe0*\x84(\x95\xa5S\x86m\x10~Pe\x90\xe0\xaeB\x13%\xb8\x8ajU\xa9\xb6\xe6\xb0J'\x06\xb0N\x99\xac\x145\x96\xb3\xb0*\xd2MdG\xa2\x1c\xee\xf9%4\xad4Ȳ\x9a\x96\xac\xd1\xcc\xcf\x10J\xe6\xd5\xed\xf5\x06\x1f\xa5j}H\xa5\xe2\xd8\xe1\x87\x13\xb6\x84\x05mT\x89֞\xb8\x01Dc\xc0ȻC\xc7Y\x80*\xf4s\x12?\xad\xae\x15\xe3h\xc0)\xa8\x98\xe45\xd21\x188ä]G\x15\x99\n0-\xbb\xdf\xeb!+\x1f\xe2\xc01v¬\xed\xcb`\a\xcb\n\x1b\xb6\x88s\x95F\xf9\xfa\xee\xf6㟗\x83n D4\x1a'\x92]\x0e\xad\xe7uz\xbd0<\xee\xff\x8a\xc1\x18\x00m\x10V\x01'\xf7\x83\xd6\xc3\x10-,\xf2\xc8S\x80GX0\xa8\rZ\x94\xc1!Q7\x93\xa0V\xbf`\xe9f#\xd2K4D&݅R\xc9-\x1a\a\x06K\xb5\x91\xe2\xbf\x1dmKXӦ5sh\x9d\xbf\x8cF\xb2\x1a\xb6\xacn\xf1\x050\xc9G\x94\x1b\xb6\a\x83\xb4'\xb4\xb2G\xcf/\xb0c>~T\x06AȵZ@圶\x8b\xf9|#\\\xf2ťj\x9aV\n\xb7\x9f{\xb7*V\xadS\xc6\xce9n\xb1\x9e[\xb1)\x98)+\xe1\xb0t\xad\xc19Ӣ\xf0\a\x91\xde\x1f\xcf\x1a\xfe\x95\x89\xde\xdb\x0e\xb6\x9d\b:4\xefB\x9f \x1er\xaat\tX$\x15\x8ex\x90\x02u\x11t\xef\xff\xbe\xbc\x87\xc4I\x90T\x10\xcaa\xea\x04\x97$\x1fBS\xc85\xe9<\xad[\x1b\xd5x\x9a(\xb9VB:\xff\xa3\xac\x05J\a\xb6]5\u0091\x1a\xfc\xa7E\xebHtc\xb27>^\x81\x15\xdd%\xb2\x00|<\xe1V\xc2\rk\xb0\xbea\x16\xff`Y\x91TlABx\x94\xb4\xfaQ\xd8xr\x80\xb77\x90b\xa8#\xa2\x1dY\xb6\xa5ƒ\x04K\xd8\xd2J\xb1\x16љ\xac\x95\x016\x9e>\xc4)o\x00\xa8e\x1d\xc9x\xd29\xa5\xa3\xf6&G(1,{\x06<9\xbc\xe8\x9f\xea\xa1\x7f귃\x95\x8fk\fje\x85SfO\x84\x83\x83\x1c+\xc4Q\xd9P+\x99,\xb1\xbe\xe4x7~%\b\xc9\tv\xec\x14\x9aLQ\xa0\xea\x19Ur\xa3芍\xa5\x01\xb7\x8e\xa6\x91\x92[t\xf9\xb3\xcac\x0eMH8\x84\x98\xd0\x0f%LJ^)U#\x1bcI\xee\xee̙\xc9\x01\xe6\x84彭\xab\x98K\xbc\xd1$\xd3J9Ŗ\x9a\x92O\x12\x87V\xfc\f_qG\x06\x06\xd7h\xd0G#\xc1\xf6k\xe5=\x84cB&\x9b\x16\x12\x01p*\xc3\xd9*(\x11r\x18\xdf\r8y?\xe0\x84\xa3\xccr\xfc\xfa\xee69\xc3\x04b\xe4}\xe2\xef\xce\xe2Cm-\xb0\xe6>r8\xbfwVs\xa9ݮ\x03\x13\xde#8\x05\f\xb4\xc0\x12\a\xde\x18\x84\xb4\x0e\x19\x8f\x9dd\x04\rƱ\x17\xc1\xd2\x1fe\x92\xda\xc1k\x93L\x80\x91\xe7\x11\x1c\xfe\xb9\xfc\xf7\xbb\xf9?T8\a\xb0\x92B3\x9fDa\x83ҽ\xe8\x12)\x8eV\x18\xe4\x94\x16\xe1\xacaR\xacѺY\xa4\x86\xc6\xfe\xf4\xea\xe7<~\x00\xdf+\x03\xf8\x89Q:\xf2\x02D\xc0\xbcsfIm\x84\r\a\xef(\xc2N\xb8\xca3\xaa\x15\x8f\a\xdc\xf9#8\xf6@79\x1c\xa1E\xa8\xc5C\xe6\xfe\x84v\xed\xa3\xb9\x03\x9b\xbf\xd2\xed\xf9\xed\x1a\xbe\t\xc6\xeb\x9a~^\a6\xba\xb0\xa5\x7f\xc1\x0e\xec\x84[f\xc4f\x83\x87\xb8\x7f\xa2,\xe4f\xc9A}\v\xca\xd0Y\xa5\xea\x91\xf0\x84IN\xc1? \x9f\xb0\xf7ӫ\x9f\xaf\xe1\x9b!\x06G\xb6\x12\x92\xe3'xE\xd6\xc7c\xa3\x15\xffv\x06\xf7^\x0f\xf6ұO\xb4SY)\x8b\x12\x94\xac\xf7!\x00\xde\"X\xd5 찮\x8b\x10 rر=\xa8\xf5\x91}\x92\x88H5\x19hf\xdc\xc9 1\xe2p\xfa\xd2L\xa3\xa6\xd4\x1ew_|\x14\xf5\xa8\xdb\xfbl\x11\xc8#\x91\xf0\xe9\xc2g \xd1O\xbd.@\xe2\xa1]\xa1\x91\xe8Ѓ\xc1Ui\t\x87\x12\xb5\xb3s\xb5E\xb3\x15\xb8\x9b\xef\x94y\x10rS\x902\x16A\xeav\xee\xcbH\xf3\xaf\xfc\x9fK\x0f\xee\xeb?\x9f{zO\xe4\xf9 \xa0\xdd\xed\xfc\x12\x04Rt\xffx\xdfu\x14\x87e\f8\xc74\xe9\xce\xef*QV)\xd7\xebYۆ\xf1`\x8e\x99\xdc?\xd3\xdd!\x9c[C\x1c\xed\x8bX\x03-\x98\xe4\xf4\xbf\x15\xd6Q\xff%\xc0\xb6Ⳍˇ۷\xcfy\xa3Zq\x89%9\x92Ä\xf6\xa98pU4L\x17a6s\xaa\x11\xe5h6\xc5\U00037704\xb4\x16h΄\x7f\xef\a\x93S\x80\x9a\xc9\x06\xba9O\x8a?\x1d\xdbd\x02\xbe~y\xf8TXx\x12\xaf\xf3\xaap\xcf6\x16\x98A`\xd00M\x1a\xf1\x80\xfb\"D\x1c\x9a\t\n\x17(\"\xe8\n\x83\xc0\xb4\xaeɧ\x87(\"C1ƿ\x11\x1ef\xfd\xf9\x8e\x01\x92\x15e\xaaJ-\xd19!\x9f\x11\x9c\x0f#F~_\xa0\xba\x9a]\xa9\xe4Zlb\xb5s\x8a\x94l뚭j\\\x803\xed\xb1\x9c\xeb$\x90\xf74\xe5\xf4\xf9?\xf4\xa6&\r?S`̟jPv\x9c\x1e\x06e\xdbLY)\xe0Ai\xc12\xfd\x06\xad\x9b\xdc^\x1a\xb8\xbe~\xca\x1d\vJyI\xca\x1d\xd2\xe0\\V\x1a\x15=\x06\xf0)3u\xea\x90\xe5e\x85\xfe\x04\xdb@\xd9=\xa5#C\xbe\x8b|\xb9d4\xa7W]N]Z\xf1Q\xcf\xd0\f\x8e\x06\xc3\xf9\x1eUC\xf2\x05\xed'T\x91\xc2\xebU\xc448G\x97\u07b4(쾴\x8eD\x89\x9dvȻB\xff%\x12\x7f=&\xe2k\xbf\x86\xc7K!\x1a\xecR\xff\xa1\xad\v\xc9\xdd\nA\x1b\xd4,[\x15\x02_\xb9\xb7\xbe\x84\xf9\xb5\rĄ\x85\xd6\"\xf7\x15\xb4\xc9\xde\x13\n\xe9E\x893\x87\x05\xad\xbf\xcc^\xe4\vS\xe11\xad\xffRrQ\x95jJf\n!K\xa8\xf9'\x9c\xf4\x8a\x97C\xec@\xae\xc3+PC\xee\xb3PJ\x92\xd7L\xd4\xc8!=\x11?\x91\xca\n\xd7\x14\xe2\x04\x1b\x97\xea8\x91\xbd\xe3\xf9\xdfiIf@\x98\x06<_R\x98\rZ\xcb6\xe7lޏaV(o\xc5%\xc0V\xaauy%\xff\xda\xc6{\xfa\xb4\x12[\xb6r44\x11\xccU\xc9\"\xacۺ\xf6k\xfa\xc6\xf5\xf0\xf9\x80\xe7j\x85\xf9\xb0\xf8D}\xed\x14\x83\x15\xb3砺\xa399\xa3\xd5y\x84\x93V\vNx\xbfw\xb8\xcb\xf4&c\x90\x19\xba\x8b\x16&34\xf9\x0e\xa0?\x18\n\xc89\xe4\xd2X\x96f\xf7ʞ\x19\xfb\xde_\xbd'\x81\x1d\xf9\xbbĶt\x05\xe8J\xd5ɜ\xf8\xd7q\xd96+4$\t\xff\xfe>r\xd2L\xf2\xbe\xd8r\x99\xfaa}Ҡ@)V\x9bb\xdd\xdc\xdfo\xa7\x80\v\xabk\xb6\xef\xce\xe2\xf3#\xba\xcc\xf9G\x84ÍJfE\xe3\xb1x\xeft\x19\xb8\xfbV!\x9f\xfc\xe5>8\x18\xb6\xe9\xa7\x03\xa3\xf1\xee\x1b\x84/\xb3Éx\xd5J\xa6m\xa5\xdc\xed\xdb3\xaa\xb1\xec&\xa6\xfbxȽ\xbc\xf5\xf5\xefSqRT\x85\f\xab\a\xeb\xf6$c1\xfct\xe5\x12-^\x0e(\x9cq\x8e\xf1K\x9a\x9c\vZ\x92\x15 \x03\xe4_?oƟ9\xbc\xe8>\x9d`.V\x91ˊ\xc9M\xb6\x96\xa5\xa4\x0f\xb6\x95\x99>E\xc3Yo7<\xd0\x1f\xe9\xe8\xb2\xea4\xe9\xf4\x9c\xf3\x1e\xed\xf8\xf0\xd7\xefiWݛ\xf8\x02~\xfd\xed\xea\xff\x01\x00\x00\xff\xff\xd9H\xdbA\x14'\x00\x00"), - []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Y_\x93۶\x11\x7fקع<$\x991\xa5\xdam3\x1d\xbd\xd9\xe7\xa6smr\xbd\xb1\xce~\xc9\xe4aE\xacHD$\x80\x00\xa0d5\xcdw\xef,@R\xfc'\xe9tnl\xbe\xdc\t\x00\x17?\xfc\xf6/\x96I\x92\xcc\xd0\xc8\x0fd\x9d\xd4j\th$}\xf4\xa4\xf8\x97\x9bo\xff\xe6\xe6R/v/g[\xa9\xc4\x12n+\xe7u\xf9\x8e\x9c\xaelJoi#\x95\xf4R\xabYI\x1e\x05z\\\xce\x00P)푇\x1d\xff\x04H\xb5\xf2V\x17\x05\xd9$#5\xdfVkZW\xb2\x10d\x83\xf0f\xebݟ\xe6/\xbf\x9b\xffu\x06\xa0\xb0\xa4%\x18-v\xba\xa8J\xb2伶\xe4\xe6;*\xc8\xea\xb9\xd43g(e\xe1\x99ՕY\xc2q\"\xbe\\o\x1cA?h\xf1!\xc8y\x17儩B:\xff\xaf\xc9\xe9\x1f\xa4\xf3a\x89)*\x8b\xc5\x04\x8e0\xeb\xa4ʪ\x02\xedx~\x06\xe0Rmh\t\xf7\f\xc5`Jb\x06P\x9f3@K\x00\x85\b\xcca\xf1`\xa5\xf2doYD\xc3X\x02\x82\\j\xa5\xf1\x81\x99V\x0e\xe8\r\xf8\x9cx\xcb\xc0*J%U\x16\x86\"\x04\xf0\x1a\xd6\x045\x12\x11\x84\x01\xfc\xe2\xb4z@\x9f/a\xce\xc4͍\x16s\xd5Ȭ\xd7D\xce\xef\a\xa3\xfe\xc0\xe7p\xdeJ\x95\x9dB\xf6\x7f\x06\xd5\xc3\xf3\xa0\xc5\x13\x91<\xe6\x14\xd64h*Sh\x14dy\xf3\x1c\x95(\b\xd8@\xc1[TnC\xf6\x04\x8a\xe6\xb5ǃ\xe9#y\xdf\xc8\xeb\xcc\\\xc3\xce5Tĵ\xbd\xed?t\x87.\xed\xfb\xa0E\xfd\x02\xd4F\rΣ\xaf\x1c\xb8*\xcd\x01\x1d\xdc\xd3~q\xa7\x1e\xac\xce,97\x01#,\x9f\x9b\x1c]\x1f\xc7*L\xfc\xb186ږ\xe8\x97 \x95\xff\xee/\xa7\xb1\xd5/ͽ\xf6X\xbc9xr=\xa4\x8f\xc3ሖ\x9d-\xab\xd5\xffE\xe0\xae\x19\xd2[\xad\xfa\xbc\xbe\x19\x8cN\x81\xed\bm\xe2\xed<\xb5\x14B\xed\xa3,\xc9y,MO\xea\xeb\xac/O\xa0\x8f\x03qz\xf72\x86\xb24\xa7\x12\x97\xf5JmH\xbd~\xb8\xfb\xf0\xe7Uo\x18\xc0Xm\xc8z\xd9D\xd7\xf8t\x92Gg\x14\xfa\xcc\xfe7\xe9\xcd\x01\xf0\x06\xf1-\x10\x9cE\xc8E'\x89c$jL\xd1y\xa4\x03Kƒ#\x15\xf3\n\x0f\xa3\x02\xbd\xfe\x85R?\x1f\x88^\x91e1\xe0r]\x15!\"\xed\xc8z\xb0\x94\xeaL\xc9\xff\xb4\xb2\x1d\xfb\"oZ\xa0'\xe7\x03\xd7Va\x01;,*z\x01\xa8\xc4@r\x89\a\xb0\xc4{B\xa5:\xf2\xc2\vn\x88\xe3G\xb6\x1f\xa96z\t\xb9\xf7\xc6-\x17\x8bL\xfa&\xa5\xa6\xba,+%\xfda\x11\xb2\xa3\\W^[\xb7\x10\xb4\xa3b\xe1d\x96\xa0Ms\xe9)\xf5\x95\xa5\x05\x1a\x99\x84\x83\xa8\x90V\xe7\xa5\xf8\xca\xd6I\xd8\xf5\xb6\x1dyd|B\"\xbcB=\x9c\x19A:\xc0ZT<\xe2Q\vM|\x7f\xf7\xf7\xd5#4H\xa2\xa6\xa2R\x8eKG\xbc4\xfaa6\xa5\xdap\x84\xe6\xf76V\x97A&)a\xb4T>\xfcH\vIʃ\xab֥\xf4l\x06\xbfV\xe4<\xabn(\xf66\x94\x1d\x1c\\+\xc3f.\x86\v\xee\x14\xdcbI\xc5-:\xfa̺b\xad\xb8\x84\x95\xf0$mu\x8b\xa9\xe1\xe2Hog\xa2\xa9\x84N\xa8vXݬ\f\xa5\xacY&\x97_\x95\x1b\x99F\x9f\xdah\v8Z\xdfgj:\x04\xf0\xb3\xc6t[\x99\x95\xd7\x163\xfaAG\x99\xc3E\x97̎\x9f7S\x82\x1aĪ\x93P\xe3\x8e\xe0\xe2J(\xea\xa5\x13\"\xf79Y\xea\xbec\xc9h'\xbd\xb6\a\x16\x1cS\xf1\xd0$Nj'\xf0\xa0Ņ\xb3q.\t\x0ediC\x96TJM\xb89W&M\x80\xefT\vc\x88\xa7\xf5\x01gB\xf3$\xe0\xd7\x0fwM\xf8m\x18\xae\xa1\x8f\"\xecEz\xf8\xd9H*D\xc8V\x97\xf7\x9e4\x04~\xee6\x11D\x88A^\x03\x82\x91\x14\xcb\xe06\xfe\x83T\xce\x13\x8az\x90\xdd\xceR=\xf7\"Ɩ\x93 \xf99\xe6\tV\t \xc7:)\xe0\x9f\xab\x7f\xdf/\xfe\xa1\xe39\x00Ӕ\x9c\v\xe5\x00\x95\xa4\xfc\x8b\xb6$\x10\xe4\xa4%\xc1u\x11\xcdKTrC\xce\xcfkid\xddO\xaf~\x9e\xe6\x0f\xe0{m\x81>bi\nz\x012rކ\xcf\xc6j\xa4\x8b\ao%\xc2^\xfa<\x005Z\xd4\a܇#x\xdc\x12\xe8\xfa\b\x15A!\xb74\xcd>\xc0M(4\x8f0\x7fc\xd7\xfa\xfd\x06\xbe\x89\xcer\xc3?o\"\x8c6Qv\xbd\xef\b\xc7\xe7\xe8\xc1[\x99et\xachG\xc6\u0081\x9dCⷠ-\x9fU鎈 \x98\xf5\x14\x03\x12\x89\x11\xbc\x9f^\xfd|\x03\xdf\xf498\xb1\x95T\x82>\xc2+\x90*rc\xb4\xf8v\x0e\x8f\xc1\x0e\x0e\xca\xe3G\xde)͵#\x05Z\x15\x87xA\xd8\x118]\x12\xec\xa9(\x92X\x92\b\xd8\xe3\x01\xf4\xe6\xc4>\x8d\x8a\xd84\x11\fZ\x7f\xb6,\xa9y8\xef4\xe3<\xdd?\x05\xbc\xbb[<\x87\x81\xa6\x9e|z\xee:\xc9ê\xaep\x862\xd9\xe7\xf7\xb9L\xf3\xe6vщ\xb6%\x8a\x18\x8eQ\x1d\xbe\x90\xef0ϕeD\x87\xa4n\x9e%\xa8\x04\xff\xef\xa4\xf3<\xfe\x1cb+\xf9I\xc1\xe5\xfd\xdd\xdb/\xe9Q\x95|N$9Q5\xc7\xe7crD\x95\x94h\x92\xb8\x1a\xbd.e:X\xcd5\xe3\x9d`%m$\xd9\v\xd5\u07fb\xde\xe2\xa6z\x9d\xa8>\xdb5W\x95\x9fN\xa1q\xb9\xf6wo/\xe0X\xb5\v\x1b\fG\x1d\xd6Eg#kЙ\xba\x0eO\xf0\xad\xfbӑ\xab\x0f\xaa\xbf\xbaA\xa6\xad\xcc$߿\xdb\xf0\x11\xae$\nK\xecv$\xbbO\x89\xc6H\x95]\x85\xb5i\xf0\xad\xc8\xf35v\xa2p\xee\xb6fϕ\xd7g\xed\xee\xb2K\xbd\x1f\x00\x01\xb4\x04\xc8gb\rm\xe9\x90\xc4*Π\xe4\x12\x8c\xab\xac\xbaT]\x13\xa01\x05\xd7I\xb12\x9b\xf2\xf5\xa6]\x99j\xb5\x91Ye\xc3\xe5h̔\xaa\x8a\x02\xd7\x05-\xc1\xdbj,\xe8\x8c\xfbt;\xa5\x174\xfe\xbe\xb3\xb4Q\xf7\x85^\xed\xf4\xa9z\x1d\xdc\xf1aHU\xe5\x18J\x02[m$N\x8c\xb3\xb1\x8f\x1c\x9d'nn\xae1\xa9\xe8I\x178\xa8\x1b\x8b\x13\x17\xd9\xda\x11벞G\xf8\xf2\x18\xdcq:;^렖~\xad\xf8\x8e\xd2G\x98L\xdf\xd9\ak\x8c\x16\xb3!i\xdd\xd86\x980\xf5\x9f\xf1ע\xc1|\xfb-\xec\x8f\xd9\xe1L\x99\xe039\xba>\x8eeX\xf8cq\xac\xb5-\xd1/@*\xff\xdd_\x8ec\xab7ͼ\xf6X\xbc\xd9{r=\xa4O\xc3鈖\x9d-\xab\xd5\xffE\xe0\xae\x18\xd2[\xad\xfar}3\x98\x9d\x02\xdba\xda\xc4\xdbYj)\x84\xda'Y\x92\xf3X\x9a\x1e\xd7\xd7Y\x9f\x9f@\x1f'\xe2\xf2\xf6e\feiN%.jJmH\xbd~\xbc\xff\xf0\xe7eo\x1a\xc0Xm\xc8z\xd9D\xd78:ɣ3\v}\xc9\xfe/\xe9\xad\x01\xf0\x01q\x17\b\xce\"䢓\xc49\x125\xa6\xe8<ҁ%cɑ\x8ay\x85\xa7Q\x81^\xfdB\xa9\x9f\rX/\xc92\x1bp\xb9\xae\x8a\x10\x91\xb6d=XJu\xa6\xe4\x7f[ގ}\x91\x0f-Г\xf3A\xd6Va\x01[,*z\x01\xa8Ās\x89{\xb0\xc4gB\xa5:\xfc\xc2\x067\xc4\xf1#ۏTk\xbd\x80\xdc{\xe3\x16\xf3y&}\x93RS]\x96\x95\x92~?\x0f\xd9Q\xae*\xaf\xad\x9b\v\xdaR1w2KЦ\xb9\xf4\x94\xfa\xca\xd2\x1c\x8dL\xc2ETH\xab\xb3R|e\xeb$\xeczǎ<2\x8e\x90\b/P\x0fgF\x90\x0e\xb0f\x15\xafx\xd0B\x13\xdf\xdf\xfd}\xf9\x04\r\x92\xa8\xa9\xa8\x94\x03\xe9H.\x8d~X\x9aR\xad9B\xf3\xbe\xb5\xd5e\xe0IJ\x18-\x95\x0f?\xd2B\x92\xf2\xe0\xaaU)=\x9b\xc1\x7f*r\x9eU7d{\x17\xca\x0e\x0e\xae\x95a3\x17C\x82{\x05wXRq\x87\x8e>\xb3\xaeX+.a%\xca,]\xf9L\x9a\x05\x8f\xb5\xa4B\x84,}\xfe\xecI\v\xe1q\xbf\x8e B\xec\xf5\x1a\x10\x8c\xa4X\xfe\xb7y\x0f\xa4r\x9ePԓ\x1cn,\xd5k/bL=\n\x92\xc7!?\xb2J\x009\xc6K\x01\xff\\\xfe\xfba\xfe\x0f\x1d\xef\x01\x98\xa6l)\\\xc3PIʿhK!ANZ\x12\\\x0fҬD%\xd7\xe4\xfc\xac\xe6F\xd6\xfd\xf4\xea\xe7i\xf9\x01|\xaf-\xd0G,MA/@F\x99\xb7i\xa3\xb1\x1a\xe9\xe2\xc5[\x8e\xb0\x93>\x0f@\x8d\x16\xf5\x05w\xe1\n\x1e7\xec1\xf1\n\x15A!74-}\x80\xdbP`\x1f`\xfe\xca!\xe5\xb7[\xf8&\x06\x89[\xfey\x1ba\xb4\x05B7\xea\x1c\xe0\xf8\x1c=x+\xb3\x8c\x0e\x95\xfc\xc8X8\xa1q*\xf8\x16\xb4\xe5\xbb*\xdda\x11\x18\xb3\x9eb &1\x82\xf7ӫ\x9fo\u16fe\f\x8e\x1c%\x95\xa0\x8f\xf0\x8a}<\xc8\xc6h\xf1\xed\f\x9e\x82\x1d\xec\x95Ǐ|R\x9akG\n\xb4*\xf6\xf1a\xb4%p\xba$\xd8QQ$\xb1\x14\x13\xb0\xc3=\xe8\xf5\x91s\x1a\x15\xb1i\"\x18\xb4\xfed9V\xcb\xe1\xb4ӌ\xeb\x93f<\xcf_B\xbd\xf2,\xef\xfdb\xb9\xfe\x99\x92\b\x85\xf9'H\xa2\xfb\xe4\xbcB\x12\x9bjEV\x91\xa7 \f\xa1S\xc7rH\xc9x7\xd7[\xb2[I\xbb\xf9NۍTY\xc2ƘD\xad\xbby\xe8'̿\n\x7f\xae\xbdx\xe8<|\xea\xed{\x8d\x92\xcf/\x02>\xddͯ\x91@SG??w\x1d\x95ò\xae\xec\x86<\xd9\xe7w\xb9L\xf3\xe6UՉ\xb6%\x8a\x18\x8eQ\xed\xbf\x90ﰜ+ˈ\xf6I\xdd4LP\t\xfe\xdfI\xe7y\xfe\x1a\xc1V\xf2\x93\x82\xcb\xfb\xfb\xb7_ң*yM$9\xf2Z\x88\xe3cr@\x95\x94h\x92H\x8d^\x972\x1dPs\xad|/XIkI\xf6L\xf5\xf7\xaeG\xdcT\xed\x13UwKsQ\xd9\xed\x14\x1a\x97k\x7f\xff\xf6\f\x8eeK\xd8`8\xe8\xb0.:\x1b^\x83\x8e\xdcex\x82o=\x1c\x8f\\}P}\xea\x06\x99\xb62\x93\n\x8bC\x04\fO1\x85%v;\xb1\xddQ\xa21Re\x17am\x1a\x9bK\xf2\xfc|\x9f(\x9c\xbb-\xe9S\xe5\xf5I\xbb;\xefR\xef\a@\x00-\x01\xf2\x9dXC\x1b\xda'\xb1\x8a3(\xb9\x04\xe3*\xab.UW\x04hL\xc1uR\xac̦|\xbdiӦZ\xadeV\xd9\xf0(\x1cKJUE\x81\xab\x82\x16\xe0mu\xec\x114\xe9>\xdd\x0e\xf1\x19\x8d\xbf\xef\x906\xea>ӣ\x9e\xbeU\xafs=\xbe\f\xa9\xaa\x1cCI`\xa3\x8dĉy6\xf6\x91\xa3\xf3\xc2\xed\xed%&\x15=\xe9\x8c\f\xea\x86\xea\xc4\x03\xbevĺ\xac\xaf\x9f\xac\xd1\x1d\xa7\xb3\xe3\xa5\x0e\xcaOk~\xa3\xf4\x11&ӽ\x8a\x01\x8d\xd1\xe2f(\xb4nl\x1b,\x1e\"\xd3p\xa1\xef\xf4\x83\xd5^\xa3\xbf{\x9bq\x9b't\x91/i\xf4\xc4\xceu-\xf7\x98V}\xd3\xcf\xe6\x82\xfd\xeaVO\xaa\xf9\xe5\xd6k:_\xd5\t\x19\xb3\tMZ+jG\x91%\x85>B\xec^\xec\xd05'O\x19A\x97_\xdc\x1a\xea\x1bfG\"<\xc1\xf8\x85\xb8FY\x90\x80\xf6\x83\xe2\x04\x9b\xa7\x9c\xc0\x85\x16\xe5\u05eeeT9\x12!*O\x80\x1e'\xe7惀@O\t\xb3\xb8.\xfaL\xfa\\I\xceav\xce\xe9~\x8cT\xb1\x13So\x01\\\xe9ʷ\xad\x98\xda\xfbjQ|\xedjӸ(\xa7\x84\x06ҹ\x86\x10\xd3L\x99a\x1b\aN\xdb!\x9c\x88o\x0f\xb4\x9b\x98\x1d}\xa0\xe9.\xde5&4\xb1\xf6}\xb0\x8e\x8b\x04P\x1ft\x8d\xfd\xb7m\xb8\\\x17\x8d\xc9k\xcfu@U\xaeȲt§\xa2FLm\xc1\x82Jt\x859\xf5\x96j94Q3\xb2\xaa\xdb\x01u\xff0\x18\xb5\xd7 \xa43\x05\xee\xdb˄\x02\x96-x\xba\x9bz0\xa3\xc659P\x1cI\xb3\xa7\x1bu\xed\xa7\xb0\xe9\xf2|\xea\xc3Z\x7f\x8c\xbf\x92\r\xd6\xdbo\x80\x7f\xcc\t'\xca\x04\xe7\xd1\xfaO\n\x90\xcb\x1e\x87s\xb11\x9c7\x1d\x19O\x87\xb4\xfe1\x9f3\x9aMJo4\x19\x90\x8b\x0e\xef\xba\xe3ߝ\xa9V\xed\xe7\xb0\x05\xfc\xfa\xdb\xcd\xff\x03\x00\x00\xff\xff\xfa\x1dp/\xd6\"\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xdc=[s\xdb8w\xef\xf9\x15\x98\xf4a\xdb\x19\xcbi\xa6\x97\xe9\xf8\xcd\xf5:\x8d\xfb}\xebx\xec4\xfb\f\x91G\">\x83\x00\x17\x00\xa5h\xdb\xfe\xf7\x0e\x0e.$%\x90\x84d˛-^2\xa6\x80\x03\xe0\xdc\xcf\xc1\x01\xb2X,\xdeц}\x03\xa5\x99\x14W\x846\f\xbe\x1b\x10\xf6/}\xf9\xfco\xfa\x92\xc9\x0f\x9b\x8f\uf799(\xaf\xc8M\xab\x8d\xac\x1fA\xcbV\x15\xf03\xac\x98`\x86I\xf1\xae\x06CKj\xe8\xd5;B\xa8\x10\xd2P\xfbY\xdb?\t)\xa40Jr\x0ej\xb1\x06q\xf9\xdc.a\xd92^\x82B\xe0a\xea\xcd?^~\xfc\xd7\xcb\x7fyG\x88\xa05\\\x11\x05\xdaH\x05\xfar\x03\x1c\x94\xbcd\xf2\x9dn\xa0\xb00\xd7J\xb6\xcd\x15\xe9~pc\xfc|n\xad\x8fn8~\xe1L\x9b\xbf\xf4\xbf\xfe\x95i\x83\xbf4\xbcU\x94w\x93\xe1G\xcdĺ\xe5T\xc5\xcf\xef\bхl\xe0\x8a\xdc\xdbi\x1aZ@\xf9\x8e\x10\xbft\x9cv\xe1W\xbd\xf9\xe8@\x14\x15\xd4ԭ\x87\x10ـ\xb8~\xb8\xfb\xf6OO\x83τ\x94\xa0\v\xc5\x1a\x83\b\xf8\x9fE\xfcN\xc2B\tӄ\x92o\xb8Q\xbb\x1aD<1\x155DA\xa3@\x830\x9a\x98\n\bm\x1a\xce\n\xc4;\x91\xab\x1e\xa40J\x93\x95\x92u\amI\x8b\xe7\xb6!F\x12J\fUk0\xe4/\xed\x12\x94\x00\x03\x9a\x14\xbc\xd5\x06\xd4e\x04\xd4(ـ2,`ٵ\x1e\xef\xf4\xbeNm\xcc6\x8b\v7\x8a\x94\x96\x89\xc0m\xc1\xe3\x13J\x8f>\"W\xc4TLw[\r\xdb#T\x10\xb9\xfc\x1b\x14\xe6r\x0f\xf4\x13(\v\x86\xe8J\xb6\xbc\xb4\xbc\xb7\x01e\x91Uȵ`\xbfG\xd8\xdan\xdcNʩ\x01m\b\x13\x06\x94\xa0\x9cl(o\xe1\x82PQ\xeeA\xae\xe9\x8e(\xb0s\x92V\xf4\xe0\xe1\x00\xbd\xbf\x8e_\x90xb%\xafHeL\xa3\xaf>|X3\x13$\xaa\x90u\xdd\nfv\x1fP8ز5R\xe9\x0f%l\x80\x7f\xd0l\xbd\xa0\xaa\xa8\x98\x81´\n>І-p#\x02\xa5\xea\xb2.\xff.\x12u0\xad\xd9Y\x1e\xd5F1\xb1\xee\xfd\x80\x02q\x04y\xac\xa88\xc6s\xa0\xdc\x16;*\xd8O\x16u\x8f\xb7O_\xfbLɴ'J\x8f7\xc7\xe8c\xb1\xc9\xc4\n\x94\x1b\x87\xacia\x82(\x1bɄ\xc1?\n\xce@\x18\xa2\xdbe͌e\x83\xdfZЖ\xdf\xe5>\xd8\x1b\xd4:d\t\xa4mJj\xa0\xdc\xefp'\xc8\r\xad\x81\xdfP\roL+K\x15\xbd\xb0DȢV_\x97\xeewv\xe8\xed\xfd\x104\xe2\bi\xbd\x16yj\xa0\x18H\x9a\x1d\xc6VA]\xac\xa4\x1a(\x19;d\x88\xa3\xb4\xf0\xdb洈U\x8b\xfb\xbf\xccq\x99m\xff\x1eG[~\xb3+k\x05\xfb\xad\x05T\xa6N\xfc\xe1P_\xa9\x9ej\x1f6\xcbF\xfb\xd4\x1dE\xb4m\xf0\xbd\xe0m\te\xd4\xeb\a\x1b\xcc\xd9\xc6\xed\x01\x144z\x94\t+D\xd6\xfaؽ\x88\xeeWT\xe0T\x01\x11\xd2$\xe01\xe1\xe0\x11&\x10\x03I\x9a`G\x03ubœ[&D\xb4\x9c\xd3%\x87+bT{\x88F7\x96*Ew#\xd8\n\x1e\xc0\x8b\x90\x15\x81xU\xc3Y\x81$\x8f\n\x05\xf1\xf5\xe7E\x15\xd3VQ\x86]>HΊ\xdd\f\xben\x93\x83\x82\xb4z\xd9\xf5;$K\xa8\xe8\x86I\x95\x12\x03\xa9\xb0kϞwjZZ-\xe9\x81\xec۸\xcc\r'\x91UI\xf9<\xc7\x10\x9fm\x9f\xce:\x90\x02\x1dʸ\x15Omo\xbb\x97@\xe0;\x14\xadI,\x93\x90\xb2E\xd3$\x15i\xa46\xe3t\x1fW]\xa4\xef\x1c\xa5~\x9c`\x9a\x83\x9d%Y\xdd5\xaf\x84\x03Q-\x0e\x06\nY\n\xb0ۨ-Q\xbb\xbeJ\xb6\xae\xef(RȒj(\x89\x14\xa33#\xbb\xb4\x1c\xb4\x9f\xabD\xce\xe8\xf4\xd0E\xb7\x7f\xf4x\b\xa7K\xe0D\x03\x87\xc2Hu\x88\xcc\x1c\x94\xba\x96\xa3XGP\x99ЦC\t\xe860\x01\x92XN\xdfV\xac\xa8\x9c\x87a\xd9\x13\xe1\x90R\x82\xb6\xda\x04]\xe6\xdd\xd8&\xc9\x1c\xf9\xfd$Sڣk3b\xb5\x0f/\xa5Q\xba\x96\xa1\x86\xbb\x96Dm\xa7{\x0ft\x8b\xffn\xe4\xe4\xb6\xff\x7f\"6\x18\x93\x13\x98vB\xfe\t\xba\x9f\xd9<=ʷ\x18ၾ$w+\x02ucv\x17\x84\x99\xf0uN\x12(\xe7\xbd9\xfeĴ9\x9e\xe93I\x93#\x13g\"L\x9c\xe2OH\x174\x19O\xdebd\xd3\xe4\xaf\xfdQ\x17\x84\xad\"\xd2\xcb\v\xb2b܀\xda\xc3\xfeI\xaa>P\xe65\x90\x91c\xf5\b\xe6\tLQ\xdd~\xb7.\x8e\xee\x92`\x99x\xd9\x1f\xec|\xe3\x10A\f\xcd\xf3\f\\\x82\xf12SPc\x1cN\xbe\"6\xbb/\xe8T_\xdf\xff|\x18+\xef\xb7\f\xce;\xd8Ȍйv\xbd\xb7\xa3\xfe\xfa|T\x10~A\x1f(\x06U.\xe7rA(y\x86\x9ds]\xa8 \x96>4tΘ^\x01&\x7f\x90Ϟa\x87`\xd2ٜÖ\xcb\r\xae=C\xc2\xf5O\xb5\x01\x0e\xed\x9a|X\xec\xf0d? \"0\x86\xcfe\x03\u05fc($r'閩KB\v\xb8?a\x9bY\xacҟ\xa3\x9f\xfaD\x0e\xf8I;ZZ\x89\xa9\x98\xcfij@\x99\xc9%\xa8k\xdf(ge\x9c\xc8\xc9ȝ\xb8 \xf7\xd2\xd8\x7f0@\xd3\xc8(?K\xd0\xf7\xd2\xe0\x97\xb3`\xd4-\xfc\x9c\xf8t3\xa0\xa0\t\xa7\xe5-\xc2\xfa9?g\xd3,\xb7E\xdc3M\ue10dW\x1cJ2\xa7\xc2\xf4\xae\x9b\xceMT\xb7\x1a\xd3uB\x8a\x05\xda\xcc\xe4L\x1e\xdfR\r\xd0\xfd\xe2I\xfd\x84_\xad\xb1p\xbf\xb8$3\xa7\x05\x94!\xb2\xc4\xec'5\xb0fE\xe6|5\xa85\x90ƪ\xf0<\x8e\xc8T\xac~7DZO\x9e\xf5\xee\xb7\xef\x8b\xe7\x98/XX\x93\xb3\xf0\x10\x8c\xac3p\xe0uw9\xbf\x9f\x85\x95ٌ^\x81\x13f\xbb\x8e$Gǻ\xe6 \xe5\x05\xe8@+\x8e.\xce,uiY\xe2\x11\x1a\xe5\x0fGX\x94#x\xe1X\xd5\xd0[\xbb3\xc15m\xacZ\xf8okiQ\x9a\xfe\x974\x94)}I\xae\xf1\xa4\x8c\xc3\xe07\x9f\x87\xeb\x81ɘ\xb2\xb1SY\xfe\xd9Pnm\xbfU\xe0\x82\x00w\x9e\x80\\\x1d\xf8E\x17d[I\xed\xcc\xf6\x8a\x01\xc7\xf3\x8a\xf7ϰ{\x7fa\xa7\x9f\x9d\xb2\xafd\xde߉\xf7·8P\x18\xd1ᐂ\xef\xc8{\xfc\xed\xfdK\\\xa9LN\xcd\xec6`њ6y\x1c*\x92\xc9\xfa\xae\r8\xa6\x9f\x9b\xef\x92\xf2\xdeɞ\xdam\x16\x8b6R\x9b\xcf\xe9\xbc\xe1\xc8z\x1e\u0088\xa1g\x9cȱ\xcdF\f>\x8f\x16\xf5\xbdu\"W\x06\x94\xcf%:\x1b\x10\xe2\x8f\x17Ff\xa9S\x99\xfebc2\x90\xc6\xfc\xaeE\xf0\f7\xb9\x83\x9b\x9c%\x1e\xe3\xb0Z\xbc\x1c\xe9\xed\xdf~\xef\xe53\xad\xe4ڿ\xfb\x1bym\x87\xba\x90uM\xf7O5\xb3\x96z\xe3F\x06\x9e\xf6\x80\x1c\xf5պEyε\xc8\x1d\x0f\xe1\xf9喙\x8a\tB\x83\xda\x00\xe5\x19\x8a\x92F\xa6rةVQM\x96\x00\"\xa6\xe8\x7f\x04W\xa2f\xe2\x0e' \x1f\xcf\xe0zDt\x9d\xd3ٽ\x894\x89\x94\x8f\x1f\x9c\xc9jdI\xb6\x15(\x180\xc6a\xde\x1d=U!M/eq\x84C\xda\xc8\xf2'MVLi\xd3_\x82&\xadΥ\xf5\x91\xe4\xb3\xeb\xfe\xcaj\x90\xad9'\x82o\xbbi\x06g\xcd5\xfd\xce\xea\xb6&\xb4\x96\xad3\xe6\x86\xd5\xf1TףwK\x99\x89\xc7V\x98\xbf1Ғ\xa0\xe1`\x80,a\x95>\xefM\xb5B\n\xcdJP\xa1J\xc1\x91\x8dI+\x98+\xcax\x9b:%J\xb5c#`q\xab\xd4I\x01\xf0\x177\xb2\x97w\xac\xe4v\x88\xa0̽\xe3A\x1a\x10\xb6\"\xcc\x10\x10\x85\xc58(\xa7\x92q\n\x8f\fD\r\xcb\xd5sy\n\xdc6\x10m\x9d\x87\x80\x05\n$\x13\x93)\xb7~\xf7O\x94\xf1s\x90\xcdr\xde'\xa9\x1e\x81\x96\xa7\xe4h~\xed\r' t\xab\xf0\xf0\xdf\xe9\x8e-\xe3yk\xb6\x94#\x9c\xb6\xa2\xa8\x00\x95\x90\x18\xea\x06\a\x9e\tm\x80\xe6\xf2\x82\xf5\x8aZ!\x98X\xe7\xd1.;\x11\xda5\x87\ua954\x1c\xe8\xf8)d\xd7,\xae\xdf@\x13\xfd\xdaM\xf3BM\xd4\x11\xc1\x1d\x9b#\x1d\xb2)j\x95\x16\xa1\xc6@\xdd8\x91\x93D\xb5\xa2o]Π\x88\x8e\t\xc3\xfd*^3\xbef\x82e\xd0v@\xd7;\xc1L\xdfy\xb4 \xce\xea<\xda\t\xa2;pJ\x86\xedn\x00\xc0\nh\x88Cp\xed\x91k\x8ep$\x97@hYB\xe9r\x97\xd6\x15\xf1a\x89+|\x1b)nH\xee\xeexO0\x8b\xb2\xa1\r\x82N\xccê\r,Z\xf1,\xe4V,0\x18\xd7G\xeb\x90\x13\xb3T/\x9dޜ\xac\x8c\xe6\xf5K\xbe\x9a\x9e\xd3BC~\xcd\xe7\xa9\xe0?\x9dA\xcbd\xf3\xcdQ\t\x8f).\x98\xd3k\xae\x00{\xe4\xc7\xd9UL\xcd?1\xd8\x1fJ߸b\xe9\x17\x95\xc5ݥA\xf5\x9c\xc2m\x05\xa6\x02\x15J\xb3\x17X\x92^N\x9e\x90v\xc1K\xac\x93\xb3L\x15\\dW\xfe\xb9W9\x87\xd1M\xcb\xf9\x85\xe5m\xda\xf2d8l$\x8a\xd8!geՏ\xa5=\x86\x9c\xea\x8bl<\xf6+-\x86\xf5\x85\xb1\n\"\x14\x18\xca0\xb3\xa7qj\xbfXX\xda;\xdf\x1f\x96S`\xfe/,\xff\x0f/=̨\x94\xc8Gcn\x95fDb\x02V\x82\xc1zh\xec\xea+|?_\xe8\xfbc\xe1\xd4@\xfd\xa5\xf1\x123\xea\xc2f\xa05\x01g\xaf\xde\x04\xadA\xab\x9d+\x10\xed\x80\xcf\x19\xda\xf1ׅ\xbb\x05\x11\xc0\xa4\xf8\xf5k\x05A|}\xf5>\xd3\xe4\x9fI%\xdbDU\xdf\x04\xcaf\xaa;\xe67<(\xf4\xf0\a\n`\xe8\xe6\xe3\xe5\xf0\x17#}\xd9\af\xd1\x12\x800(\xea2\xb3L\x94l\xc3ʖ\xf2 \xb5\xdd\x1d\x02\xc7@\x1d\x9f%\xa0IE\x04\xe3\x8e\x01\xc3\xf8\x01Ñ/\x8d;\x969Z\xc5M\xfb\xa2y\xd5!'ׄ\fk>F\xac\xe1\xb1\xc7\x17\xafR\x05\xfb\x87\xd4z\x1c_\xe1\x91\x13I\xccTs\x9cPÑY,\xf6\xe2\xf3\x96\x9c*\x8dcb\xee\xb3Ud\xbc~\x1dF\x16~\xe6k.\x8e\xc1\xce\xd9\xeb+ް\xaa\xe2mj)2+(^\xaf\x142/\xfa<\xa9\x14`>`\x19\xaf\x82\x98\xad}xQ@sҖfk\x1a\x8e\xa9d\x98\xa5N\x9e\x98\xbdY\xad\u009bU(\xbcm]\xc2$\x17M\xfexL\xe5A\x8c\x93~\xa1M\xc3\xc4\xfa\x90)rYg\x92m\xe6Y\xe6~o!\x03\x9e\xe9\x873]t8\x12\xfa\xba\xeb҉H2\xa4-\x990\xf2\x92\\\x8b\x9d\x87\x9b\x80\xd3\v\x1f\x854\a\x17\xd9첶\x8c\xf3\xfem-\x04;\r\xcaߙԴv\xab\x1a\xf3\xf6\x93t\x95j\xe0\x94\x9f\x148~ك\xd1ώ\xbe\xa5\xe7_\xb7ܰ\x86\x83\xf5\xe86\xacL\xde!3\x15\xec\"\x92\xff&\xf1\x86\xd4r\x87\x90\xbe]\xf4\x9eQ\xec\x9eq\x926\xbf\xc9\x13\xb6\x97Q\xcc~\\\x11{\x06\xcdrE\xf1\r\x8b\xd5߰H\xfd\xad\x8b\xd3g8k\xe6\xe7\xe3\x8a\xd0O>\x81\tG\xfd\xf7\xb2\x84\a\xa9\xcc\\p\xf2\xb0\xdf?q\x92\xda\v\xd8$/\x89\b]\x13\xbb\xc4\x10Ç\x17\xa7m*}\xe8\x19\xdc\xe9_di\xd76w\xc6\xf2\xb8\xd7\xfd\xe0\xae\xf2\n\x14\b\xf7\xcc\xc7\x7f>}\xb9\x8f\xf0S>\xaf\xf7\x8c\xf7\x9e\x97p\x1eL\xe9\x91\xe3\x8f\xe6|1\x93\xc3\x16\xfa\x00\xaf|.B\x1b\xf6\x1f\xf8\xaa\xdb\v\xd2A\xd7\x0fw\b#\xf8i\xf8L\\\xac\xa2\x88'\x96K\xb0\x16+\xa2jT,\xeeV\x03\x88Ê\xdf\xfe3JP\xba'\xb3\x82\xc5d\xa1\xc6\xcb\n\xdeÝ[\xc7\xd8,\x9f\xac\xd3(vD:\x8e\xac\x98*\x17\rUf\x87l\xa3/\x06k\bff*\x9d3\xaaX\x0f\x9f\x01K\xa27\xbc\xfe\x85g\x91\xbbfxڻ\x8f\xbbS\xd61~\xffd\xf6\xe6\xc9+\xaec\xdcb/\x10S\x89\xcf\xc9\x02\x93WK\x93yM\xf4\xf0\xed\xa4\xb4\xcbc\x1c=\xad\xe7l\x14\x1dRM\t0v<\xaa:-h\xa3\xabēK/\xd3u\xf8\x1a\x99\xa1\xa6}\xc9&\x1d\x80\xc1>YQ\xf5\xb4\xd5\x16\x82>\v\xdbFi\xc5a)\xddnm\xb3\xab{a\xfc\xa2\x97)x\x9b#\xe1\xcc\xe7\\N~\xc8šgD\xfd`\xf6˪\xb6CL\x9dp\x18<\xeb\xdae\x14\x19O;\xb1\x99π\xe4\x19\x8c\x13\x9e\xfe@|\xe5\xe2\x8a$_\x04\xc9|\xf5\xe3\x0fE\xf4\x84V\xd3E\x05e\xcb\xe1\xd47\xff\x9ez\xe3\xe7_\xfd\v\xb3e\xbc\xfbg\x91\xdd3\xd0\xd6g\x1e\xbe/\xe8)\xe1!\xf7)9\xe6\xf0ap\xe0\x9e\x17+\xdcK\x94E\x01Z\xafZ\x1e\xaa\x94\n\x05\xd4@\x19\xba3\x1dW|T\x9dM\xdbpIKP7R\xacX\xe2\x84d\x80\xd6\xff\x1at\xde\xe3\xd9\x02?\xb6\xaa{\xdaq\xf2Y\xbc\x17i\xae\x86*\xca9\xf0O\x8c\x83\xfeYn\x85]W\x86@>\xa4\xc6\xf5\xeee\x15\xad\xb2f}GD[/\xad\x93\vƌ\a\x8b+\xa9\xa6+\xa4\x1dޙ0\xb0\x86T|\xbdU\xcc\xc0SC\x95\x06\\Q\xc6\x0e~\xdd\x1b\xe2\xa2\xcf\x15\xa7kW\nW\xb2\x82\x1a\x88\x06\x18g\x18[>\x8e\xd7\b\x8b\xef\xb02I\x8e$\xbd\xb2\x85z\xecJƨX\x8f=/\x9a0\xd5\xc9\aF\x9dE.hc\xf0\x02\f\xd2\x11\x89h<\f|\xb4w\xef\x8d\xd1\x01\xd8qN\xf3e̾`N\x1bZ'\xa2\x84y\xbdss\b\x06\x9f\x05Ve\xaf\xee\xae\xff\xc0b,\xb0#[\xaac1u\xd2\xf7\xee`;0\xe8\xaa[\xd0P\x12\u0600 V\x14)\xe3PNq\xeaWL$\xab\r\xa8\x9ft\x84\x83\x95\x80\x96ş\fU&.\xfdЏYIUSsEJj`aG\x9f溥\x9fIU\xea\xc4\xe3@\xbc\xd9\xe6ţ\b\xd7n\xac\xf5s\xf7\xd1jК\xaeC\x10\xba\x05\x05d\r\xc2\xe2=\xe6\x16\x93\x1eS\xb8\xd2\xe7\x8dE,-\xb5(\xa4\x85i\xa9\x9f\xc0\xb9p\xf1\xf44\xbcO\x8cQ\xeczTE\xa7U\x85\xbf<\xf8\bT\xef?w}\x80\x8bO\xfd\xbe>I\xecv\xec\xceF\xa8+\xf0\xc4\a\x8f\r\x8b\x91uJ\xa6\x8dę\x8f2'\x95\x94\xcfYn\xf6\xe7رK'1\xe1X\t\xafL.ekz~\x8eGxb\x99\xf8\xfc\xe7+\xdb\x17\x84y\xed.P\x8d\xe5V\xf3<\xbd\xcf\x03H1\xbc\x95\x86\xf2`d,_\xc6\x0e\xd5\xc4\x03\x02O\xe1\xf1d\xcew\x17\xfb\x90\xf7^e\xef`W\xddS\x9e^\x13t\xd7\xc7\xc7Ҫ>\xeb\x97\x04\x12_\x01\xed|\x92\xb17\x17\xe7\xec\x1fB\xfd\x84\x8b\xca\xc0\xf1\xe7\xae\xf7\x18\x1e\xdd2\x9d\xc3\f\"\x1di\x12\f>L\x15%ㄥOx\xa9ME\xf5\x9c{\xfa`\xfbD\xb7\xa3g\xae\xa2\x13\xfa8\"\x95\xe9{\xae\vr\x0f\xdb\xc4W\x87,<\xfdB\xa9Jt\xb9\x13\x0fJ\xae\x15\xe8C\xa6[\xe0}F&֟\xa4z\xe0횉/\xe3\x95\xdfS\x9d\x1f\xa82\xcc2\xad[Ob\xecM\xb0q\x89\xdf\xe6G\x8f\xff\xc0\x04\xe5\xec\xf7\x94.\xef\xff87Ä\xbek<\xf2N\xb1P\x01\xf1s\n\xd0k\xe8\x9ft\xcf\xfc\x84y/ɽL\x8a\xb1? fC\xa0L\x93%h\xb3\x80\xd5J*\xe3\xf2\xf7\x8b\x05a\xab\xe0 Y\r\x81q\xa2{͞\xb0T\xe2=\x1e\xbd\x05\x87e\xe5S\x89\n\xad\x0e\x86\x9c5ݹ\x8c$-\n\x1b\x13\xc0\amh*6y\x91\x9e\xc6P\xd5\xcbJ\x8e\n\xb9\xeb\xf7\x8f9\xbe\xa8>\x10\x9cC\x1d^gw\x06\x9d\x8f\x9di\r^\xcb \xdab\xef\x14eB\x9c\x1a\xbb\x1b\x0f\xbb\xf3L\xcd\xd7\beL=\xfa\xfd\r\x1e\xe2\xf6\a\xac\xbe\x93%[QQ\xb1\x1e\xbd\xd0V)ٮ\xab\xc0\x9bc\x0e\x11)[\x8c\x9c\x1bT\x05:\xfc\xc7!\xa6U\xa2wh\xe7k,ƴt\\\uee0f\xf2\x02E\xad\xba\x8b-\x9d\xaa\x9a\xb0\xf9\xd9Y\xc2\x11\x88\xb3\xb6?\x01\x91\xea\x9d(&\xaf\xe0\xf8@\x9bM\xdc՝\xc2P\x12\tQ\x1b\xbf\x1a\x12\"\xc41$\xf4}\x89.\xe2\xf9a02棜\x88\x8ei'\x06\xb78\rj~\xd3}'h\xe8\xee\x1c\x87\x0e=\b\xfeNJ\xbb\r \x1c\x13\xf9\xe2\xdc\xe9\xb8\xf7ǍX7\xd1ۺ=9v\xfd\xb6\ac\xef\n\xa4\x8db\xbbiB\xbc\xf9\xf7l\x95\x92\x17\xf7\xbf3-9\xfc\xc3\xc1\xafo|\x95qK\x95`b}\x12F~\xf5c\x13\xf1\xbc\a{Έ>\xac\xfc\xd5b\xfa\xa4Y:\xf8\x88\f^\xf6\xf0\xecg\xf2_\xfe/\x00\x00\xff\xffP\a\xb5\x16Cm\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xec=]s\x1c)\x92\xef\xfa\x15\x84\xeea?B\xdd^\xc7}ą\xde|\xb2gO\xb1\x1e[ai\xf4\xbctU\xb6\x9aQ\x15\xd4\x00\xd5r\xdf\xde\xfe\xf7\x8dL\xa0\xbe\xba\xe8\xa2Z-ygǼت\x86$\xc9L\xf2\x03\x12X,\x16g\xbc\x12\xf7\xa0\x8dP\xf2\x92\xf1J\xc0W\v\x12\xff2\xcb\xc7\xff6K\xa1\xdelߞ=\n\x99_\xb2\xab\xdaXU~\x01\xa3j\x9d\xc1{X\v)\xacP\xf2\xac\x04\xcbsn\xf9\xe5\x19c\\Je9~6\xf8'c\x99\x92V\xab\xa2\x00\xbdx\x00\xb9|\xacW\xb0\xaaE\x91\x83&\xe0\xa1\xebퟖo\xffk\xf9\x9fg\x8cI^\xc2%3\xd9\x06\xf2\xba\x00\xb3\xdcB\x01Z-\x85:3\x15d\b\xf4A\xab\xba\xbad\xed\x0f\xae\x91\xef\xd0!{\xeb\xdbӧB\x18\xfb\x97\xde\xe7\x8f\xc2X\xfa\xa9*j͋N\x7f\xf4\xd5\b\xf9P\x17\\\xb7\xdf\xcf\x183\x99\xaa\xe0\x92}®*\x9eA~Ƙǟ\xba^0\x9e\xe7D\x11^\xdch!-\xe8+U\xd4e\xa0Ă\xe5`2-*K#\xbe\xb5\xdcֆ\xa95\xb3\x1b\xe8\xf6\x83\xe5g\xa3\xe4\r\xb7\x9bK\xb64ToYm\xb8\t\xbf:\x129\x00\xfe\x93\xdd!n\xc6j!\x1f\xc6z{Ǯ\xb4\x92\f\xbeV\x1a\f\xa2\xccrb\xa0|`O\x1b\x90\xcc*\xa6kI\xa8\xfc\x0f\xcf\x1e\xebj\x04\x91\n\xb2\xe5\x00O\x8fI\xff\xe3\x14.w\x1b`\x057\x96YQ\x02\xe3\xbeC\xf6\xc4\r\xe1\xb0V\x9aٍ0\xd34A =l\x1d:\x1f\x87\x9f\x1dB9\xb7\xe0\xd1\xe9\x80\n»\xcc4\x90\xdcމ\x12\x8c\xe5e\x1f\xe6\xbb\aH\x00F$\xaaxmH8\xda\xd67\xddO\x0e\xc0J\xa9\x02\xb8\x80vX4\xb6\nu%\xa0\x80\xe6\f\xddN\x8d\x16FH\xb6\xae\xd1#]2\xd4\x12Q\x19\x11\xd2X\xe0\x11a>\x01\xef\xe0kV\xd49\xe4WEm,\xe8\xdbLU\x90\x87E\xa6Q͜\xca\xc3\x0f\a!\xfb\xf8\xa5\x10\x19 \x1f2WiA\x8b<1\xd1nC\x99]\x05n\xcd\tY\xed\x87\xd0\xc6(\x93\xbaŀņ\xe7\x7f<\xbf \t\xe8\xf7\xde\xef\xc70\xae\xa1!\xd3,\xddL\x16\x7f\xbc\x85\xb0PF\xa8;\xa9\xa3f\xf0\x9dk\xcdw\a\xb8\xde,\xa6\xbd\x00\xdfc\xb0\a\x9c\x97\xa1\xda7\xe2\xfd\xb0\xff\xdf\"\xf7O\xcboC\x8b\xce\\H\xe4s!\x8c\xed\xb1ٸU,$\xebX\b\xe9\t$\x1dLT\x93S\\\xfd'!\xe6I\xe7Nl\xb24\xb2\xe9'\xc0\xbf\x14%7J=\xa6P\xef\x7f\xb1^\xbb\x84\xc52\xda\x18a+\xd8\xf0\xadP\xda\f\x97I\xe1+d\xb5\x8dj\x16nY.\xd6k\xd0\b\x8b\x96\xf9\x9b]\x81C\xc4:\x1c\xbe\xb0\x8eʊV\x18\x8c\xabe:\xb2\x94\xa8\x11\x1b\n\x05\xa8Q\xa8\xce\xc1\xc1Ђ\x1c\x88\\lE^\xf3\x82|\t.37>\xde\xe0\x17\xd3j\x13\x02\xb1\x87\x7fT\xaa]q\x0eM\x18$2\xb1\xb7\xea\xa5$\xa0\x8f_bl\xb4_5N\x89\xb0\x94p\xb0od\xa6\xae\v0\xbe\xbb\x9c\xdc\xe4V']\xb4\xccrk\f\x05_A\xc1\f\x14\x90Y\xa5\xe3\x14J\x91\x03WR\x95n\x84\xb8#Z\xb6\x1fm\xb5\x83\x99\x00\xcb(\xc4݈l\xe3\xdcW\x144\x82\xc5r\x05\x86VExU\x15\x11\xd3ՖI\xe1\xf0\x9dM鍶$h\x90!ܘ.iK\xa2~n\xcb(\xd9۹٧\xfa\xf8:\xff(\xbe\xbf%\xa2\a\xabs\xa4\xb0Oh\x12F\xfb\x05\xc9\xf3!Jz\xa4\xb8\x00\xb3\xec\xac\xce\t\x1b\xbe\xa60\xb4\xe7?\xeem\xa5\xec\x11\xe5\xd7Ż\xe3&\xcc\f\xd6MΩ\x97e\\\xd3Ϳ\b\xdf\xc8d\xddz\x8b5\x8bg\x1f\xbb-/hW\xc03$\xbf`kQX \xa7j\nQ6\x83s\xa7$P\xaa\x05f\xb4Il\xb3͇f\xef(\xa1ŀVC\x00\xceA\x0fQ\x0e\xf1 \x01$k\\\v\xda4\x15\x1aJڌ\xa5H\xb2\xfb\x85\\\xc1w\x9f\xde\xc7c\xcfnI\x94ԽA%LZW\xde\r\x1c\xa3.\xae>T\t\xbf\x90\xbf\xd6\x04\x82n\x13\xfe\x82q\xf6\b;\xe7bqɐo)\x8b\xff|\xf8*\fv,s\xf6^\x81\xf9\xa4,}yQ*\xbbA\xbc\x06\x8d]O4A\xa5\xb3$H\xc4n\U00088ce5(\xa8\r?\x84a\xd7\x12C2G\xa2\x19\xddQ\xae\x90\xeb\xd2uVֆ\xb6Z\xa5\x92\v\xb7,6֛\xe7\x81\xd2=\x16\x9c\xa4c\xdf\xe9\x1d\x1a#\xf7\x8b\xcbZ*x\x06yآ\xa3t\x1an\xe1Ad3\xfa,A?\x00\xab\xd0,\xa4K\xcb\fE\xedG6_\xbc\xd2=\x87n\xf9\xbax\xacW\xa0%X0\v4k\v\x0fŪ2\x91.\xde&\x8c䜌\x95\x05\xce\xf5ĚAZ\x92\xaaG2r\x0eWO%\xd63\xc9D^\x04\xb9]IR\xd0Ml\x9dg\xbdf\xca\xcd1*\xa63\x16\xe7\x02\x94\x9c\xb6\xd6\xfe\x86\x96\x9ef\xe3\xdfYŅ6K\xf6\x8e2{\v\xe8\xfd\xe6\x17&;`\x12\xbb\xadh\x95\xfd\x97Zly\x81\xfe\a\x1a\bɠpވZ\xef\xf9j\x17\xeci\xa3\x8cs\x1b\x9aM\xbb\xf3Gع\x1d\xe5\xa4n\xbb\n\xeb\xfcZ\x9e;_fO\xf14\x8e\x8f\x92Ŏ\x9d\xd3o\xe7\xcfu\xeffH\xf4\x8c\xaa=Q.y\x95.ɔ7;'\xd0\xc0`=8DظI \xc5\x00a\x8a\x02ɢ\\)\x13I\x16\x89\xa0\x95 \xe87\xcaX\xb7\x0e\xd9\xf3\xf7G\x17*UX\x9cd|mA3c\x95\x0e)\x99\xa8\xf8S\x96\xe2\xbb\xe5n\x03\x06\xfc>\x94_\xf4t\x801\x8a=ou\x83\xb3*\xe7n/\x8c:\xe2\x19yOԶ\xd2*\x03\x13͋hK\xa2m\xeaQp\x9f\x0eͺ.w\xd1\xdf:Ik\xa7,J\x872ϑG\xd2\x1d\x11\x19}\xf8\xdaY\xa2F\xed\x82\x7f\xa7H\xeb182:\xafQ\x96|\x98\x0e\x9c\x8c\xee\x95k\x1d\xe6\x98\a\xe6\xc2-\xfdP\x93Ι\xe3u4\xa2\xfc\xcf\xe6ڔB^SG\xec\xed\v\xbaC^\x8b\xc7ң\xc6\xca\xf1N\xfaU\xe8\xac\xe5^\xf3\xc1\xe7\xd4)\xda\xf8\xd1\xd0c\xee\xfe\x9e\by\xd7R\xd9\xce2\xceL'\xbaR\xf9\xef\f[\vml\x17\rs \xb1j\x14\xd4\x11\xa1\xa7\xfc\xa0\xf5ё\xe7g\u05fa\xb3\xa0\xb8QO>qzN\xbc\x1dH\xba\xe1[\xf0\x99\xab 3UKZ\nC=\x80\xdd̀\xe8X\xe3\xac@\xa2\xbd\xeb4\x96u\x99N\x90\x05I\x92\x90\x93\xebf\xdd&?p\x91\xb6nŎc\xab=\x94\xc39V\x8e\x9fG!\xc1\xb3\x9bN_\U000af8acK\xc6K\xe4!\xb9\x1d\xa2\x84&\xa3ޱ\xbbI\xfb\xc4\x16d\xb4\xac\xc2YV\x15`\xc1\xa7m\xce\xc0#S҈\x1c\x1a\xd3\xefE@I\xc6ٚ\x8b\xa2\xd63\xb4\xeal\x92\xcf\r¼69}d\x95\x8eȂH\x94\xb8\xce>\xc3\v\x9e\xd6\xf8\x95\x9e\xe7Ǧ8\x8c\x1a\xe6\xfb\x8b\x95\x16\xca\x1d\x068\xbd\xcb\xe8ӎ\xb9\xdc}\xf7\x19\xbf\xfb\x8c\xdf}\xc69\x1d}\xf7\x19'\xcaw\x9f\xf1\xbb\xcfx\xb8|\xf7\x19S\xcaw\x9fq&\"\xdf\xcagL\xc1pAk\x9c\a*$a\x95\x98\n1\x85\xf6D_>\xe9ǟ\xd58I.\xf3\xf58ȑC<\x91\xe3\x171\xaf\xa35^Mr3\xce\xc00w\xdc)\xca\x04\x87\xf9\x04\xa7g\x02\x02\xa7?=s}\x10\xf2\tO\xcf\xf8!\xa4E\x18G\x9d\x9d\tD\x9a\x7fz\xe2\xc2'\x11\x95\xc0\xc3V\x8aK\xff\x88\x8d1&I\tx|\xe3\xe4\xf7\xbd\x8c\xc9\x17\x90\xa5W9\x913K\x9eFY\x7f\xfe\xc7\xf3_\a\x8bN˔(\x1b\xf6i\xeb\xd4xL?b,\xdfM\x8d\xecg\xa9\xfez\xa6\xc2Ie?\xf5DMC\xe4\b\xbc\xbeX\x0f\xa8\xfck\xd27\x16\xcaϕ\xb7\x96'8a\x7f=\x02/\xe9\x8c=7;\x99m\xb4\x92\xaa6~M\ba\xbd\xcbܽ\x03\x01dL\xd8G5\xc8\x7f\xb0\x8d\xaa#\xa76&H\x9b\x90E\x9bF\x90^R\xadO\x8c\x00˷o\x97\xfd_\xac\xf2)\xb6\xecI\xd8M\x04\x18\xddG\xc1\xf3\x1c\xe3\x82\u0381\x1e\xaf\a\xc2UIC\xa1\x8c\x00S\x9aIQ8\x89\r\x10z\xf2\xca>Wnu\xf0h\xbfiz\r+=\x11wn\xfam\x93-9\xed\xbe?#\xe9\xf6\xa4G\xa3\xbeYZ\xedqɴ\xa9+\x94\t\x89\xb3\xe9\xe9\xb2)lu%=I69BNM\x88\x9d\xbb\x02\xf1\xa2ɯ/\x93\xf2\x9aL\xb3\xb4\xf4ֹ\x14{\x95T\xd6WN`}\xbd\xb4\xd5\x19ɪ\xa7?\xf5\x92\xbe\x96~tveڲ\xcc\xe1\x84Ӥ4Ӥ\xa5\x9b\x94\x01\x1f5Ԥ\xf4ѹI\xa3I\x9cL\x9f\xae\xaf\x9a\x16\xfa\xaaɠ\xaf\x9f\x02:)m\x93\x15\xe6&y\x8e_r\x18ʴ\x03P|\v\xe1|.\x99\x94\xee\xb9\xe6ϊ;?\x0f`\xa1\xb0\x047\xf5\x15〲.\xac\xa8\x8a\xf6>\xb6X\xc0\xb9\x81]sY\xd1ϊ\x8e\xc8\xfb\x9b\xba>\x7fi$~9\x88j\xb8aOP\x14\x8c\xc7\xe6\xe6\x1e\x152w\x0fh\xa6\x16\x80\xb6\x11g\xb9\xbf\x8c\xc9_\x1ez\xe1\xa6\v\xdd\x06@\x16\xb6\x8c-\xf5qy\xf8\xa6\xaf\x83\x06,U\x8f\xedy\xe6.ޠo\xbfԠw\x8c\xee\x1dk|\xb3\xf6P\xa9\x9f\xe8\x06\x03Ӡ~\xbc:<\xb4g\xb2\x17\xe0\xb4ꁽ\x93\xce#\x18\xe2DmP\xef\xb4\x01\x1d*U\x8cӢ\xfdD@H\xd5@\x884Mq\xfe眲|\x89\xf0\xee\x14\x01^\x92\a4\xcf{\xfd\x86\xa7'\x8f=5\x99\x9e\x8c\x92tJ\xf2%½9\x01\xdf,\x7f5\xfd\x14\xe4\xfc\x8d\xe7\x17>\xf5\xf8R\xa7\x1dgP/\xf5t\xe3|ڽ\xd2i\xc6W?\xc5\xf8\x9a\xa7\x17g\x9dZLNϚ\x95q0'\xb5\xea\x19\xc7\xed\xd2r\t\xa6O!&\x9e>L\xcc4H\x1b\xfc\x91\xc3N<]8\xffTa\"\x7f\xe7L\xe9W>=\xf8ʧ\x06\xbf\xc5i\xc1\x04\tL\xa82\xffT\u0cf7\xa4\x94\xceAOn\xfb͑\xdaIyM\x8d\xe5\xfa\x88\r\xf6\xb5\xc2m\xb2X\xab\x17\x03\x90Y\xf2\x17\xf9ӣ\r\x87\xb6\xc1Q2;\x1eQo_\xb2u\xd7\xfa\x0e\xb1\x7f\xcd\xc1m]\x1a\xa88\x1a\x00\n\xdc(5+\xea*|\xe0\xd9f\xd0Æ\x1b\xb6V\xba䖝7\x9b\xc5o\\\a\xf8\xf7\xf9\x92\xb1\x1fT\x93\xabӽ/͈\xb2*v\x18\x89\xb1\xf3n\x83\xe7IIT:C\xcf7\xaa\x10Y\xc4\xe7\x1c\xbdW\xcf5ػl\x88n\xfe\xcb:\xd9\"\xb1\xc0\a\x9b\x8bp\xebb\xffJfw\x9f\xfb\x91k%\xbc\x12\x7f\xa6'\x95N\xb0\xea\xf6\xee\xe6\x9a`\x051\xa2\xb7\x9a\x9a\x04ņ\xe5+@\x97\xa1\x1d\xfb!}r\xbd\xeeA\xed\xe7\bw\x1f\xab\x80ܽL\x12\xdc\x16\xaf\x9a3\x85Z\xeb\xe6\xda\xe1r\xa8'\x94/.wL\xf9\xa7'\x84\xce\x17\x15\xd7v璉.zx\x04\xbb>\xb5jv\xd0Z\xed\xbf\xbc\xd2-=\xb2\x87GWh'{W\xf5\x93\a\x86\xf4|\x0eN\x87OUO\x9e\xa7~\x01\x9c\x0e\xbbP\v\xa2b\xe4\xa7h\x06\xe4\xc9W,\x8d\xbf\xa1\xffG\xb5\x85\xf7ѕ\xcb\xfe\xeb+\x83&#\xa9\x89\x01*]2\x1f\xa1`\x9b\x8fHw|?O\xed\xc5s\r\x03*\xfe\x8e\xf0\xe7,N\xde\xf6A\x8d?HB7\xa8\x87Nc^\x15=\xf5\xb4c7\xf7\x14\xb76\xaa\xd4O}\x1f\xb7\x86\xe5ɐ`\x10\x81%\xe4\xc17ZNEF\xab4\x7f\x80\x8fʽ\xad\x93\"&\xfd\x16\xbd\x97\x97\xbc\xe7\x16\xf2\xb5\xfd$\x8c)z?\xb6!\xc0\xf6|\xc6\xdeE\xff\x88\xed\x91O\x19X[\xba\x91ғ&\xef\xfd\xeb$\xa8\x8f\r \v\x02\x05\x1c\xb4\x15\xfew\xa3\x9e\xe8\x02\xfc\xf8\x1asx@\xa4\xf3\x86\x19\xd0A\x11J\xe1=j\x98uU(\x9e\x83\xbe\xa2GT\x12F\xfcS\xaf\xc1\xc0\x1d\xe8?\xc5\xe2\xedfd<\xa1\xe7\x17̒A\x8f\xae(\xa0\xf8A\x14`\x1c≦\xe1f\xbfec)\xear\xe5<\xd55\xfe\xd8tr\xc02\xbb\xa1\xd2\x06C\x05\x1a\xfdD\xb7\x15Q\x9b \xf9\x87\x89\xc1\x1a>\ni\xe1\x01\xc6c\xe8\t\x9b\xe0\xdeh \a (0\x8a\xf8\xfe\x12[y\xec\x11\xe4>\xdez \x03\xcdbdL\x8e\x95w\xabn\xee\xaf\f\xabeN\x1b\x00\xf7\x7f\xbe=J~\xb7\xbd\xf7e\x82NHQ\xef\xf7\xe3-;!BG;\x91O\x1fW\xe21X\xdc\x18\x95\t\x8a*\x9e\x84\xf5\xd79\xbe\xdc\x1d\xe2\x87\x02\xc4\x03\xd2Q\x1b\xf8\xfc$A\x7f\t\x16\xc8\\\xcbػ-\xd3\xda\xef\xa7=h\xd1\xf7Z\xac¾G`\f\x000\x15\xf6\xb9\x8c{\t(l\xaf\t\xd3H\x1c\xc4>\x01\xdcyG\xc8ik\x85\xb4\xe3\x9c!n\x9bVt\xd8tDCN\x8b\xed\xfd\x00\xc6 \x93\x9d\x1e}j\xaa\xb8Ӧ\x86\xfd^\x8cy\xa3\xb4c\x96\xe1@\xff\xb0\xf7kT\x83\x1f\xd4\xde1\xcd=\xaaF\xf6>\xd2CxyGr\xbc\x97\xde\xfdR\xaf\xda\a\x15\xd8\xdf\xfe~\xf6\x8f\x00\x00\x00\xff\xff)\x00\x87w>{\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcV\xcfo+5\x10\xbe\xe7\xaf\x18\x89+\xbb\xa1B \x94\x1b*\x1c*\xe0\xa9j\x9ezw\xbc\x93d\xa8\xd7^f\xc6)A\xfc\xf1\xc8\xf6n\x9b\xee:\xb4\x8f\x03\xbe\xad\xed\xf9\xe6\x9bo~x\x9b\xa6Y\x99\x81\x1e\x91\x85\x82߀\x19\b\xffT\xf4\xe9Kڧ\x1f\xa4\xa5\xb0>ݬ\x9e\xc8w\x1b\xb8\x8d\xa2\xa1\x7f@\t\x91-\xfe\x84{\xf2\xa4\x14\xfc\xaaG5\x9dQ\xb3Y\x01\x18\uf0da\xb4-\xe9\x13\xc0\x06\xaf\x1c\x9cCn\x0e\xe8ۧ\xb8\xc3]$\xd7!g\xf0\xc9\xf5\xe9\x9b\xf6\xe6\xfb\xf6\xbb\x15\x807=n@\x90ә\x1a\x8d\xc2\xf8GDQiO\xe8\x90CKa%\x03ڄ\x7f\xe0\x10\x87\r\xbc\x1e\x14\xfb\xd1w\xe1\xbd\xcdP\xdb\f\xf5P\xa0\xf2\xa9#\xd1_\xae\xdd\xf8\x95\xc6[\x83\x8bl\\\x9dP\xbe \xc7\xc0\xfa\xe9\xd5i\x03\"\\N\xc8\x1f\xa23\\5^\x01\x88\r\x03n \xdb\x0e\xc6b\xb7\x02\x18\x05\xc9Xͨ\xc5\xe9\xa6\xc0\xd9#\xf6\xa68\x01\b\x03\xfa\x1f\xef\xef\x1e\xbfݾ\xd9\x06\xe8P,ӠYֿ\x9b\x97}\xa8\x85\t$``\xa4\x04\x1a\xc0X\x8b\"`#3z\x85B\x19\xc8\xef\x03\xf79\xad`v!\xea\x05\xaa\x1e\x11\x1e\xb3\xfec\x98\xed\xcb\xe1\xc0a@V\x9a\xa4)\xeb\xa2\xe2.v\xff\x8dxZ)\xd6b\x05]*=\x94\xecy\xd4\v\xbbQ\x1e\b{\xd0#\t0\x0e\x8c\x82\xbe\x14c\xda6\x1e\xc2\xeew\xb4\xdaΠ\x8b.\x922\x19]\x97*\xf6\x84\xac\xc0h\xc3\xc1\xd3_/ؒ\x04JN\x9dѬ\x9dWdo\x1c\x9c\x8c\x8b\xf85\x18\xdf͐{s\x06\xc6\xe4\x13\xa2\xbf\xc0\xcb\x062\xe7\xf1[`\xccRo\xe0\xa8:\xc8f\xbd>\x90N}hC\xdfGOz^疢]\xd4\xc0\xb2\xee\xf0\x84n-th\f\xdb#)Z\x8d\x8ck3P\x93\x03\xf1\xb9\x17۾\xfb\x8a\xc7Ε7n\xf5\x9cjP\x94\xc9\x1f.\x0er\xeb|AzR#\x95b*P%\xc4\xd7,\xa4\xad$\xdd\xc3\xcf\xdb\xcf01)\x99*Iy\xbd\xba\xd0e\xcaOR\x93\xfc\x1e\xb9\xd8\xed9\xf4\x19\x13}7\x04\xf2\x9a?\xac\xa3\\\xb8qד\xcaT\xda)us\xd8\xdb<\xab`\x87\x10\x87\xce(v\xf3\vw\x1enM\x8f\xee\xd6\b\xfeϹJY\x91&%\xe1Cٺ\x9c\xc0\xf3\xcbEދ\x83iv^ImeJl\a\xb4)\xb9I\xdfdM{\xb2\xa5\xad\xf6\x81\xc1\xd4L\xda\x0f1\xc9\x16_\xc8e\x9cH\x85\xcdlN\xa5.\x7f\x9fM},哣\x11\x9co\xce8ݧ;s\xff\x8e\xf6h\xcf\xd6a\x81(S\bߧ\x92\x16\xfa\xd8/}6\xf0\t\x9f+\xbb\xf7\x1c҄\xc6\xf9\xa8\xb9Z\x1bP\x1e\xb1\x03\xf9E\xb8\xf3\xc8ʭ\xfc0.G~\x0eh\x04\x02\x8eާ\x96\x0e~\x01Yy\x11\x16wH\xb1\xaf\xb0\xa9\xf2\xb9\xf3\xfb\x90\xff\"Lrl\xb4\xb4\x13\x8e\xc9\x1e\xfd\x14^\x15\xc0\xeb\xb9.k9\xe7>$hY\xf9y\xfeo\xc6i.\x11c\xd5w\x93YU\x0f\x92ǚ\xe2\xf5\xfe\x1aYF\xe7\xcc\xce\xe1\x06\x94\xe3Һ\xd8\x1afs\x9eW\xcdTj\x9f\xa9GQ\xd3\x0f\xef\x14\xd0\xe2UH\xeb~\x81\x92\x9a\xe7\xf9\x88\xfeZ\x8b\xc0\xb3\x91W\xe7\x15\xc8\xdd\xf9\x9a\xe9\xed\xcb\xdf\xe6\xb2\xcfJ=o \xcd\xfaF\xa9\"䇔\xaa\xa6\xb4\xd4y\xf5\xb7f\xa1\xd2\xf6\xf2\xee4H\xde\xf4\xcb\xf4W\xb3\x8c\xe1*\x85j\x05,63|w\x11\x9eh`s\x98\x02\xfe'\x00\x00\xff\xff\xef\xf8\xa6>\x10\f\x00\x00"), diff --git a/pkg/controller/pod_volume_backup_controller.go b/pkg/controller/pod_volume_backup_controller.go index 3231efda6..6fde05784 100644 --- a/pkg/controller/pod_volume_backup_controller.go +++ b/pkg/controller/pod_volume_backup_controller.go @@ -247,7 +247,7 @@ func (r *PodVolumeBackupReconciler) Reconcile(ctx context.Context, req ctrl.Requ } if pvb.Spec.Cancel { - log.Info("Prepared PVB is being cancelled") + log.Info("Prepared PVB is being canceled") r.OnDataPathCancelled(ctx, pvb.GetNamespace(), pvb.GetName()) return ctrl.Result{}, nil } @@ -345,7 +345,7 @@ func (r *PodVolumeBackupReconciler) Reconcile(ctx context.Context, req ctrl.Requ // Update status to Canceling if err := UpdatePVBWithRetry(ctx, r.client, types.NamespacedName{Namespace: pvb.Namespace, Name: pvb.Name}, log, func(pvb *velerov1api.PodVolumeBackup) bool { if isPVBInFinalState(pvb) { - log.Warnf("PVB %s is terminated, abort setting it to cancelling", pvb.Name) + log.Warnf("PVB %s is terminated, abort setting it to canceling", pvb.Name) return false } diff --git a/pkg/controller/pod_volume_backup_controller_test.go b/pkg/controller/pod_volume_backup_controller_test.go index aa9a398ed..38249bfa0 100644 --- a/pkg/controller/pod_volume_backup_controller_test.go +++ b/pkg/controller/pod_volume_backup_controller_test.go @@ -624,7 +624,6 @@ func TestOnPVBProgress(t *testing.T) { r.OnDataPathProgress(ctx, namespace, pvbName, progress) if len(test.needErrs) != 0 && !test.needErrs[0] { - updatedPvb := &velerov1api.PodVolumeBackup{} assert.NoError(t, r.client.Get(ctx, types.NamespacedName{Name: pvbName, Namespace: namespace}, updatedPvb)) assert.Equal(t, test.progress.TotalBytes, updatedPvb.Status.Progress.TotalBytes) @@ -740,18 +739,13 @@ func TestAcceptPvb(t *testing.T) { }{ { name: "update fail", - pvb: pvbBuilder().Result(), + pvb: pvbBuilder().Node("test-node").Result(), needErrs: []error{nil, nil, fmt.Errorf("fake-update-error"), nil}, - expectedErr: "fake-update-error", - }, - { - name: "accepted by others", - pvb: pvbBuilder().Result(), - needErrs: []error{nil, nil, &fakeAPIStatus{metav1.StatusReasonConflict}, nil}, + expectedErr: "error updating PVB with error velero/pvb-1: fake-update-error", }, { name: "succeed", - pvb: pvbBuilder().Result(), + pvb: pvbBuilder().Node("test-node").Result(), needErrs: []error{nil, nil, nil, nil}, }, } diff --git a/pkg/podvolume/backupper.go b/pkg/podvolume/backupper.go index c989a363a..6a96617f9 100644 --- a/pkg/podvolume/backupper.go +++ b/pkg/podvolume/backupper.go @@ -181,7 +181,7 @@ func newBackupper( // the PVB in the indexer is already in final status, no need to call WaitGroup.Done() if ok && (existPVB.Status.Phase == velerov1api.PodVolumeBackupPhaseCompleted || existPVB.Status.Phase == velerov1api.PodVolumeBackupPhaseFailed || - pvb.Status.Phase != velerov1api.PodVolumeBackupPhaseCanceled) { + pvb.Status.Phase == velerov1api.PodVolumeBackupPhaseCanceled) { statusChangedToFinal = false } } From c001eee0888530947f86967f7d9babed7667fe07 Mon Sep 17 00:00:00 2001 From: Lyndon-Li Date: Thu, 12 Jun 2025 07:54:02 +0000 Subject: [PATCH 29/31] vgdp ms pvb controller Signed-off-by: Lyndon-Li --- config/crd/v1/crds/crds.go | 2 +- pkg/cmd/cli/nodeagent/server.go | 7 +++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/config/crd/v1/crds/crds.go b/config/crd/v1/crds/crds.go index b93ec3a2f..9f8a5b689 100644 --- a/config/crd/v1/crds/crds.go +++ b/config/crd/v1/crds/crds.go @@ -35,7 +35,7 @@ var rawCRDs = [][]byte{ []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcVMo\x1b7\x10\xbd\xebW\f\xd0kwU\xa3hQ\xec\xadqr0\xda\x06\x82\x1d\xe4N\x91#-c.\xc9\xce\f\xe5\xba\x1f\xff\xbd \xb9+K\xab\x95\x93\\\xb27\x91Ù\xc7\xf7f\x1e\xd54\xcdJE\xfb\x11\x89m\xf0\x1d\xa8h\xf1/A\x9f\x7fq\xfb\xf8\v\xb76\xac\x0f7\xabG\xebM\a\xb7\x89%\f\xf7\xc8!\x91Ʒ\xb8\xb3ފ\r~5\xa0(\xa3Du+\x00\xe5}\x10\x95\x979\xff\x04\xd0\xc1\v\x05琚=\xfa\xf61mq\x9b\xac3H%\xf9T\xfa\xf0C{\xf3s\xfb\xd3\n\xc0\xab\x01;0\xe8Pp\xab\xf4c\x8a\x84\x7f&d\xe1\xf6\x80\x0e)\xb46\xac8\xa2\xce\xf9\xf7\x14R\xec\xe0e\xa3\x9e\x1fkW\xdcoK\xaa7%\xd5}MUv\x9de\xf9\xedZ\xc4\xefv\x8c\x8a.\x91rˀJ\x00[\xbfON\xd1b\xc8\n\x80u\x88\xd8\xc1\xfb\f+*\x8df\x050^\xbb\xc0l@\x19S\x88TnC\xd6\v\xd2mpi\x98\bl\xc0 k\xb2Q\nQ\x1fz,W\x84\xb0\x03\xe9\x11j9\x90\x00[\x1c\x11\x98r\x0e\xe0\x13\a\xbfQ\xd2w\xd0f\xbe\xda\x1a\x9a\x81\x8c\x01\x95\xea7\xf3ey\u0380Y\xc8\xfa\xfd5\b,J\x12O J]\x1b<\xd0\t\xbf\xe7\x00J|\x1b{\xc5\xe7\xd5\x1f\xcaƵ\xca5\xe6pS\x99\xd6=\x0e\xaa\x1bcCD\xff\xeb\xe6\xee\xe3\x8f\x0fg\xcbp\x8euAZ\xb0\fjB\x9a\x89\xab\xacA\xf0\b\x81`\b4\xb1\xca\xed1i\xa4\x10\x91\xc4N\xadU\xbf\x93\xe19Y\x9dA\xf8\xb79\xdb\x03Ȩ\xeb)0y\x8a\x90\v\x89cS\xa0\x19/Zɵ\f\x84\x91\x90\xd1\u05f9\xca\xcb\xcaC\xd8~B-\xed,\xf5\x03RN\x03܇\xe4L\x1e\xbe\x03\x92\x00\xa1\x0e{o\xff>\xe6\xe6|\xef\\\xd4))\x94\xe4\xb6\xf3\xca\xc1A\xb9\x84߃\xf2f\x96yP\xcf@\x98kB\xf2'\xf9\xca\x01\x9e\xe3\xf8#\x93h\xfd.tЋD\xee\xd6뽕\xc9Rt\x18\x86\xe4\xad<\xaf\x8b;\xd8m\x92@\xbc6x@\xb7f\xbbo\x14\xe9\xde\njI\x84k\x15mS.⋭\xb4\x83\xf9\x8eF\x13Ⳳ\x17\xddS\xbf\xe2\x02_!O\xf6\x84\xda#5U\xbd\xe2\x8b\ny)Sw\xff\xee\xe1\x03LH\xaaRU\x94\x97\xd0\v^&}2\x9b\xd6\xef\x90\xea\xb9\x1d\x85\xa1\xe4Dob\xb0^\xca\x0f\xed,z\x01N\xdb\xc1\nO\x1d\x9b\xa5\x9b\xa7\xbd-\xb6\x9b\x1d E\xa3\x04\xcd<\xe0\xceí\x1a\xd0\xdd*\xc6o\xacUV\x85\x9b,\xc2\x17\xa9u\xfa\x98̃+\xbd'\x1b\xd33pEڅ\xe1\x7f\x88\xa8\xb3\xb8\x99\xdf|\xda\ueb2ec\xb5\v\x04O\xbd\xd5\xfd4\xfc3\x9a\x8eFq\xce߲1\xe4\xef\xc5n\xe7;W/\x0fEdK8k\xd8\x06.\xbc\xfbu^\x8a\xa9~%3\xd5\xd1Gnt\"*\xcdw\xf4y\xb5t\xe8K\xb9@\xa2@\x17\xab3P\xefJP\xf9Ǡ\xacgP\xfey<\b\xd2+\x81'\xa4\r\x97\x95\x1ax\x8fO\v\xabw~CaO\xc8\xf3\x96ϛ\x9b\xca\x1e\xce߃WXZlʋE\xceVhNXd\t\xa4\xf6\xa7\xbcr\xda\x1e\x9d\xbe\x83\x7f\xfe[\xfd\x1f\x00\x00\xff\xff\xbeM\x1a\xea\xb1\n\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcWMo\xe36\x10\xbd\xfbW\f\xd0K\v\xac\xe4\x06E\x8b·\xd6\xd9C\xb0\xe96\x88\xb7\xb9S\xd4HbC\x91,9t6E\x7f|1\xa4\xe4\x0fYv\x9c\xcb\xea\xe6\xe1p\xf8\xe6\xcd\xcc#]\x14\xc5B8\xf5\x84>(kV \x9c¯\x84\x86\x7f\x85\xf2\xf9\xd7P*\xbb\xdc\xde,\x9e\x95\xa9W\xb0\x8e\x81l\xff\x88\xc1F/\xf1\x16\x1be\x14)k\x16=\x92\xa8\x05\x89\xd5\x02@\x18cI\xb09\xf0O\x00i\ry\xab5\xfa\xa2ES>\xc7\n\xab\xa8t\x8d>\x05\x1f\x8f\xde\xfeX\xde\xfcR\xfe\xbc\x000\xa2\xc7\x15\xd4\xf6\xc5h+j\x8f\xffD\f\x14\xca-j\xf4\xb6Tv\x11\x1cJ\x8e\xddz\x1b\xdd\n\xf6\vy\xefpn\xc6|;\x84y\xccaҊV\x81>ͭޫ\xc1\xc3\xe9\xe8\x85>\x05\x91\x16\x832m\xd4\u009f,/\x00\x82\xb4\x0eW\xf0\x99a8!\xb1^\x00\f)&XŐ\xdd\xf6&\x87\x92\x1d\xf6\"\xe3\x05\xb0\x0e\xcdo\x0fwO?m\x8e\xcc\x005\x06镣D\xd4\x7f\xc5\xce\x0e\xd3\x04@\x05\x100\xc0\x01\xb2;\x84 \f\bO\xaa\x11\x92\xa0\xf1\xb6\x87J\xc8\xe7\xe8\xc0V\x7f\xa3$\bd\xbdh\xf1\x03\x84(;\x10\x1c%;\x1c\x9c\xa5m\v\x8d\xd2X\xeel\xce[\x87\x9e\xd4Hy\xfe\x0e\x1a\xea\xc0z)\v\xfe8\xf1\xbc\vj\xee,\f@\x1d\x8e\xe4a=p\x05\xb6\x01\xeaT\x00\x8f\xcec@\x93{\x8d\xcd\xc2\fٔ\x93\xd0\x1b\xf4\x1c\x06Bg\xa3\xae\xb9!\xb7\xe8\t\x1aE\xaf\xcb41\xaa\x8ad}XָE\xbd\f\xaa-\x84\x97\x9d\"\x94\x14=.\x85SEJĤQ+\xfb\xfa;?\ff8:\x96^\xb9!\x03yeڃ\x854\x1d\xef(\x0f\xcfK\xee\xae\x1c*\xa7\xb8\xaf\x02\x9b\x98\xbaǏ\x9b/0\"ɕ\x1aZl\xe7z\xc2\xcbX\x1ffS\x99\x06}ޗڔc\xa2\xa9\x9dU\x86\xd2\x0f\xa9\x15\x1a\x82\x10\xab^Q\x18{\x9dK7\r\xbbNR\x04\x15Bt\xb5 \xac\xa7\x0ew\x06֢G\xbd\x16\x01\xbfq\xad\xb8*\xa1\xe0\"\\U\xadC\x81\x9d:gz\x0f\x16Fy&j^\x01\x128\xe1[\xa4\xa9u\x82\xe5Kr\xe2\xe3_:q,X\xdfcٖ\xac9a\x00\x92\xf5\xe8\x87i\xa1.a\x80\xd9F\x9fE2\xf67\xd3\xc0\xbc\xb2\xa0\xb0\xd8\x1db:=\x9a?4\xb1\x9f?\xa0\x80\xdf\x13\xe6{\xdb^\\_[C<\x17\x17\x9d\x9e\xac\x8e=n\x8cp\xa1\xb3o\xf8\xde\x11\xf6\x7f:\xf4\xf9\x1a\xbe\xe8:\xde滫\xef\x82c\xd4g\xcf}D\xbeA\xf0|\xa6\x83\xc3UQ\xae\xc04x^\x95\xe8zs\xf7\x1e\nϸ\xbf\xa3Hw\xa6\xb1o\xa4\xb8w\x9c\xf5;#\x03\xe3\x97\xde\x10o\xf74\xbfBƞ\xe6-\xf9\xeeD\xf8\x14+\xf4\x06\t\xc3^\xa9_\x14u\xb3\x11\x01^:%\xbb\xb41\r\x04_\x02!X\xa9\xe6$\xf5\n\xf8\xac#\xca\xe3\xccP\x16iXg\xcc\f\xfe\xc4|F\xfd\xce\x1dP\f\x8at\x95\x82\x92\xa0\x18ޡ\xa1\xc9\x7f\xa4ZF\xef\xd3\x15\x95\xad\xfc2\x99n\xb8VDG\xe5\xf9\xeb\xf1\xfe\r%\xbd\xdd{\xa6\x17\xb7P&\xa3q\x1e\x8b\xa0Z~A\xf1\x1akiҸS2\xf2w\xfc\xc2;&j\xb6\xa2\xf8թ<\x80o@\xfc\xb8ŝ\x8f&\xdf\xf3\xd37l\n\x88\x81\x9f[ \x85\x99\xc1X!Ԩ\x91\xb0\x86\xea5\xdf\\\xaf\x81\xb0?\xc5\xddX\xdf\vZ\x01\xdf\xff\x05\xa9\x9962QkQi\\\x01\xf9x\xae\xcbf\x13w\x9d\b3cx\x94\xf3\x03\xfb\xcc5\xc6n\x18/v\x06\x9c\xbd_\n\xf8\x8c/3\xd6\ao%\x86\x80\xa7ct6\x93\xd9!81\x06~\xa4\xd5\a,\r\x7f\x19\x06\xcb\xff\x01\x00\x00\xff\xffx\xae@\xbaJ\x0e\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4ZK\x93\x1b\xb7\x11\xbe\xef\xaf\xe8Z\x1flWi\xc8HI\\)ޤU\x9c\xda\xc4V\xb6ĕ..\x1f\xc0A\x93\x03\xef\f\x80\x00\x18R\x8c\xe3\xff\x9ej<\x86\xf3\x00\xc9]\xca\xf2\xe2\xb2K<\x1a\x8d\xaf\x1b\xfd\xc2\x14EqŴ\xf8\x88\xc6\n%\x17\xc0\xb4\xc0O\x0e%\xfd\xb2\xb3\x87\xbfٙP\xf3\xed˫\a!\xf9\x02nZ\xebT\xf3\x1e\xadjM\x89oq-\xa4pBɫ\x06\x1d\xe3̱\xc5\x15\x00\x93R9Fݖ~\x02\x94J:\xa3\xea\x1aM\xb1A9{hW\xb8jE\xcd\xd1x\xe2i\xeb\xed\x9ff/\xbf\x9b\xfd\xf5\n@\xb2\x06\x17\xa0\x15ߪ\xbamp\xc5ʇV\xdb\xd9\x16k4j&ԕ\xd5X\x12\xed\x8dQ\xad^\xc0a \xac\x8d\xfb\x06\x9e\xef\x14\xff\xe8ɼ\xf1d\xfcH-\xac\xfbWn\xf4\aa\x9d\x9f\xa1\xebְzʄ\x1f\xb4Bnښ\x99\xc9\xf0\x15\x80-\x95\xc6\x05\xbc#64+\x91_\x01\xc4#z\xb6\n`\x9c{\xd0X}g\x84thn\x88B\x02\xab\x00\x8e\xb64B;\x0fʈ?\xb0\x8e\xb9ւm\xcb\n\x98\x85w\xb8\x9b\xdf\xca;\xa36\x06m`\x0e\xe0\x17\xab\xe4\x1ds\xd5\x02fa\xfaLW\xccb\x1c\r\xe0.\xfd@\xecr{b\xd9:#\xe4&\xc7Ľh\x10xk\xbcP\xe9\xf4%\x82\xab\x84\x9dp\xb7c\x9684\xce\x1f;ϋ\x1f'\x8aֱF\x8f\x99\xea-\r\\q\xe60\xc7Ӎjt\x8d\x0e9\xac\xf6\x0e\xd3I\xd6\xca4\xcc-@H\xf7\xdd_\x8e\xc3\x11\xf1\x9a\xf9\xa5o\x95\x1cb\xf3\x86z\xa1\xd7\x1d8!Ym\xd0d\x01R\x8e՟È#\x02oz\xeb\x03'\x81n\xbf\xff,+\xa4x\xa0\xd6\xe0*\x84(\x95\xa5S\x86m\x10~Pe\x90\xe0\xaeB\x13%\xb8\x8ajU\xa9\xb6\xe6\xb0J'\x06\xb0N\x99\xac\x145\x96\xb3\xb0*\xd2MdG\xa2\x1c\xee\xf9%4\xad4Ȳ\x9a\x96\xac\xd1\xcc\xcf\x10J\xe6\xd5\xed\xf5\x06\x1f\xa5j}H\xa5\xe2\xd8\xe1\x87\x13\xb6\x84\x05mT\x89֞\xb8\x01Dc\xc0ȻC\xc7Y\x80*\xf4s\x12?\xad\xae\x15\xe3h\xc0)\xa8\x98\xe45\xd21\x188ä]G\x15\x99\n0-\xbb\xdf\xeb!+\x1f\xe2\xc01v¬\xed\xcb`\a\xcb\n\x1b\xb6\x88s\x95F\xf9\xfa\xee\xf6㟗\x83n D4\x1a'\x92]\x0e\xad\xe7uz\xbd0<\xee\xff\x8a\xc1\x18\x00m\x10V\x01'\xf7\x83\xd6\xc3\x10-,\xf2\xc8S\x80GX0\xa8\rZ\x94\xc1!Q7\x93\xa0V\xbf`\xe9f#\xd2K4D&݅R\xc9-\x1a\a\x06K\xb5\x91\xe2\xbf\x1dmKXӦ5sh\x9d\xbf\x8cF\xb2\x1a\xb6\xacn\xf1\x050\xc9G\x94\x1b\xb6\a\x83\xb4'\xb4\xb2G\xcf/\xb0c>~T\x06AȵZ@圶\x8b\xf9|#\\\xf2ťj\x9aV\n\xb7\x9f{\xb7*V\xadS\xc6\xce9n\xb1\x9e[\xb1)\x98)+\xe1\xb0t\xad\xc19Ӣ\xf0\a\x91\xde\x1f\xcf\x1a\xfe\x95\x89\xde\xdb\x0e\xb6\x9d\b:4\xefB\x9f \x1er\xaat\tX$\x15\x8ex\x90\x02u\x11t\xef\xff\xbe\xbc\x87\xc4I\x90T\x10\xcaa\xea\x04\x97$\x1fBS\xc85\xe9<\xad[\x1b\xd5x\x9a(\xb9VB:\xff\xa3\xac\x05J\a\xb6]5\u0091\x1a\xfc\xa7E\xebHtc\xb27>^\x81\x15\xdd%\xb2\x00|<\xe1V\xc2\rk\xb0\xbea\x16\xff`Y\x91TlABx\x94\xb4\xfaQ\xd8xr\x80\xb77\x90b\xa8#\xa2\x1dY\xb6\xa5ƒ\x04K\xd8\xd2J\xb1\x16љ\xac\x95\x016\x9e>\xc4)o\x00\xa8e\x1d\xc9x\xd29\xa5\xa3\xf6&G(1,{\x06<9\xbc\xe8\x9f\xea\xa1\x7f귃\x95\x8fk\fje\x85SfO\x84\x83\x83\x1c+\xc4Q\xd9P+\x99,\xb1\xbe\xe4x7~%\b\xc9\tv\xec\x14\x9aLQ\xa0\xea\x19Ur\xa3芍\xa5\x01\xb7\x8e\xa6\x91\x92[t\xf9\xb3\xcac\x0eMH8\x84\x98\xd0\x0f%LJ^)U#\x1bcI\xee\xee̙\xc9\x01\xe6\x84彭\xab\x98K\xbc\xd1$\xd3J9Ŗ\x9a\x92O\x12\x87V\xfc\f_qG\x06\x06\xd7h\xd0G#\xc1\xf6k\xe5=\x84cB&\x9b\x16\x12\x01p*\xc3\xd9*(\x11r\x18\xdf\r8y?\xe0\x84\xa3\xccr\xfc\xfa\xee69\xc3\x04b\xe4}\xe2\xef\xce\xe2Cm-\xb0\xe6>r8\xbfwVs\xa9ݮ\x03\x13\xde#8\x05\f\xb4\xc0\x12\a\xde\x18\x84\xb4\x0e\x19\x8f\x9dd\x04\rƱ\x17\xc1\xd2\x1fe\x92\xda\xc1k\x93L\x80\x91\xe7\x11\x1c\xfe\xb9\xfc\xf7\xbb\xf9?T8\a\xb0\x92B3\x9fDa\x83ҽ\xe8\x12)\x8eV\x18\xe4\x94\x16\xe1\xacaR\xacѺY\xa4\x86\xc6\xfe\xf4\xea\xe7<~\x00\xdf+\x03\xf8\x89Q:\xf2\x02D\xc0\xbcsfIm\x84\r\a\xef(\xc2N\xb8\xca3\xaa\x15\x8f\a\xdc\xf9#8\xf6@79\x1c\xa1E\xa8\xc5C\xe6\xfe\x84v\xed\xa3\xb9\x03\x9b\xbf\xd2\xed\xf9\xed\x1a\xbe\t\xc6\xeb\x9a~^\a6\xba\xb0\xa5\x7f\xc1\x0e\xec\x84[f\xc4f\x83\x87\xb8\x7f\xa2,\xe4f\xc9A}\v\xca\xd0Y\xa5\xea\x91\xf0\x84IN\xc1? \x9f\xb0\xf7ӫ\x9f\xaf\xe1\x9b!\x06G\xb6\x12\x92\xe3'xE\xd6\xc7c\xa3\x15\xffv\x06\xf7^\x0f\xf6ұO\xb4SY)\x8b\x12\x94\xac\xf7!\x00\xde\"X\xd5 찮\x8b\x10 rر=\xa8\xf5\x91}\x92\x88H5\x19hf\xdc\xc9 1\xe2p\xfa\xd2L\xa3\xa6\xd4\x1ew_|\x14\xf5\xa8\xdb\xfbl\x11\xc8#\x91\xf0\xe9\xc2g \xd1O\xbd.@\xe2\xa1]\xa1\x91\xe8Ѓ\xc1Ui\t\x87\x12\xb5\xb3s\xb5E\xb3\x15\xb8\x9b\xef\x94y\x10rS\x902\x16A\xeav\xee\xcbH\xf3\xaf\xfc\x9fK\x0f\xee\xeb?\x9f{zO\xe4\xf9 \xa0\xdd\xed\xfc\x12\x04Rt\xffx\xdfu\x14\x87e\f8\xc74\xe9\xce\xef*QV)\xd7\xebYۆ\xf1`\x8e\x99\xdc?\xd3\xdd!\x9c[C\x1c\xed\x8bX\x03-\x98\xe4\xf4\xbf\x15\xd6Q\xff%\xc0\xb6Ⳍˇ۷\xcfy\xa3Zq\x89%9\x92Ä\xf6\xa98pU4L\x17a6s\xaa\x11\xe5h6\xc5\U00037704\xb4\x16h΄\x7f\xef\a\x93S\x80\x9a\xc9\x06\xba9O\x8a?\x1d\xdbd\x02\xbe~y\xf8TXx\x12\xaf\xf3\xaap\xcf6\x16\x98A`\xd00M\x1a\xf1\x80\xfb\"D\x1c\x9a\t\n\x17(\"\xe8\n\x83\xc0\xb4\xaeɧ\x87(\"C1ƿ\x11\x1ef\xfd\xf9\x8e\x01\x92\x15e\xaaJ-\xd19!\x9f\x11\x9c\x0f#F~_\xa0\xba\x9a]\xa9\xe4Zlb\xb5s\x8a\x94l뚭j\\\x803\xed\xb1\x9c\xeb$\x90\xf74\xe5\xf4\xf9?\xf4\xa6&\r?S`̟jPv\x9c\x1e\x06e\xdbLY)\xe0Ai\xc12\xfd\x06\xad\x9b\xdc^\x1a\xb8\xbe~\xca\x1d\vJyI\xca\x1d\xd2\xe0\\V\x1a\x15=\x06\xf0)3u\xea\x90\xe5e\x85\xfe\x04\xdb@\xd9=\xa5#C\xbe\x8b|\xb9d4\xa7W]N]Z\xf1Q\xcf\xd0\f\x8e\x06\xc3\xf9\x1eUC\xf2\x05\xed'T\x91\xc2\xebU\xc448G\x97\u07b4(쾴\x8eD\x89\x9dvȻB\xff%\x12\x7f=&\xe2k\xbf\x86\xc7K!\x1a\xecR\xff\xa1\xad\v\xc9\xdd\nA\x1b\xd4,[\x15\x02_\xb9\xb7\xbe\x84\xf9\xb5\rĄ\x85\xd6\"\xf7\x15\xb4\xc9\xde\x13\n\xe9E\x893\x87\x05\xad\xbf\xcc^\xe4\vS\xe11\xad\xffRrQ\x95jJf\n!K\xa8\xf9'\x9c\xf4\x8a\x97C\xec@\xae\xc3+PC\xee\xb3PJ\x92\xd7L\xd4\xc8!=\x11?\x91\xca\n\xd7\x14\xe2\x04\x1b\x97\xea8\x91\xbd\xe3\xf9\xdfiIf@\x98\x06<_R\x98\rZ\xcb6\xe7lޏaV(o\xc5%\xc0V\xaauy%\xff\xda\xc6{\xfa\xb4\x12[\xb6r44\x11\xccU\xc9\"\xacۺ\xf6k\xfa\xc6\xf5\xf0\xf9\x80\xe7j\x85\xf9\xb0\xf8D}\xed\x14\x83\x15\xb3砺\xa399\xa3\xd5y\x84\x93V\vNx\xbfw\xb8\xcb\xf4&c\x90\x19\xba\x8b\x16&34\xf9\x0e\xa0?\x18\n\xc89\xe4\xd2X\x96f\xf7ʞ\x19\xfb\xde_\xbd'\x81\x1d\xf9\xbbĶt\x05\xe8J\xd5ɜ\xf8\xd7q\xd96+4$\t\xff\xfe>r\xd2L\xf2\xbe\xd8r\x99\xfaa}Ҡ@)V\x9bb\xdd\xdc\xdfo\xa7\x80\v\xabk\xb6\xef\xce\xe2\xf3#\xba\xcc\xf9G\x84ÍJfE\xe3\xb1x\xeft\x19\xb8\xfbV!\x9f\xfc\xe5>8\x18\xb6\xe9\xa7\x03\xa3\xf1\xee\x1b\x84/\xb3Éx\xd5J\xa6m\xa5\xdc\xed\xdb3\xaa\xb1\xec&\xa6\xfbxȽ\xbc\xf5\xf5\xefSqRT\x85\f\xab\a\xeb\xf6$c1\xfct\xe5\x12-^\x0e(\x9cq\x8e\xf1K\x9a\x9c\vZ\x92\x15 \x03\xe4_?oƟ9\xbc\xe8>\x9d`.V\x91ˊ\xc9M\xb6\x96\xa5\xa4\x0f\xb6\x95\x99>E\xc3Yo7<\xd0\x1f\xe9\xe8\xb2\xea4\xe9\xf4\x9c\xf3\x1e\xed\xf8\xf0\xd7\xefiWݛ\xf8\x02~\xfd\xed\xea\xff\x01\x00\x00\xff\xff\xd9H\xdbA\x14'\x00\x00"), - []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Y_\x93۶\x11\x7f\xbfO\xb1syH2cJ\xb5\xdbf:z\xb3\xcfM\xe7\xda\xe4zc\x9d\xfd\x92\xc9ÊX\x91\x88H\x00\x05@\xc9j\x9a\xef\xdeY\x80\xa4\xf8O\xd2INl\xbc\xdc\tX,~\xd8\xffX&Ir\x83F~ \xeb\xa4V\v@#\xe9\xa3'ſ\xdcl\xf377\x93z\xbe}y\xb3\x91J,\xe0\xaer^\x97\xef\xc8\xe9ʦ\xf4\x96\xd6RI/\xb5\xba)ɣ@\x8f\x8b\x1b\x00TJ{\xe4i\xc7?\x01R\xad\xbc\xd5EA6\xc9H\xcd6ՊV\x95,\x04\xd9\xc0\xbc9z\xfb\xa7\xd9\xcb\xeff\x7f\xbd\x01PX\xd2\x02\x8c\x16[]T%Yr^[r\xb3-\x15d\xf5L\xea\x1bg(e\xe6\x99ՕY\xc0a!n\xae\x0f\x8e\xa0\x1f\xb5\xf8\x10\xf8\xbc\x8b|\xc2R!\x9d\xff\xd7\xe4\xf2\x0f\xd2\xf9@b\x8a\xcab1\x81#\xac:\xa9\xb2\xaa@;^\xbf\x01p\xa96\xb4\x80\a\x86b0%q\x03P\xdf3@K\x00\x85\b\x92\xc3\xe2\xd1J\xe5\xc9\xde1\x8bFb\t\br\xa9\x95\xc6\aɴ|@\xaf\xc1\xe7\xc4G\x06\xa9\xa2TRea*B\x00\xafaEP#\x11\x81\x19\xc0/N\xabG\xf4\xf9\x02f,\xb8\x99\xd1b\xa6\x1a\x9e5M\x94\xf9\xc3`\xd6\xef\xf9\x1e\xce[\xa9\xb2c\xc8~gP=<\x8fZ<\x13\xc9SN\x81\xa6AS\x99B\xa3 ˇ\xe7\xa8DA\xc0\x06\nޢrk\xb2GP4۞\xf6\xa6\x8f\xe4}ï\xb3r\x89t.\x11E\xa4\xed\x1d\xff\xa1;u\xee\xdcG-\xea\rP\x1b58\x8f\xber\xe0\xaa4\at\xf0@\xbb\xf9\xbdz\xb4:\xb3\xe4\xdc\x04\x8c@>39\xba>\x8eeX\xf8cq\xac\xb5-\xd1/@*\xff\xdd_\x8ec\xab7ͼ\xf6X\xbc\xd9{r=\xa4O\xc3鈖\x9d-\xab\xd5\xffE\xe0\xae\x18\xd2[\xad\xfar}3\x98\x9d\x02\xdba\xda\xc4\xdbYj)\x84\xda'Y\x92\xf3X\x9a\x1e\xd7\xd7Y\x9f\x9f@\x1f'\xe2\xf2\xf6e\feiN%.jJmH\xbd~\xbc\xff\xf0\xe7eo\x1a\xc0Xm\xc8z\xd9D\xd78:ɣ3\v}\xc9\xfe/\xe9\xad\x01\xf0\x01q\x17\b\xce\"䢓\xc49\x125\xa6\xe8<ҁ%cɑ\x8ay\x85\xa7Q\x81^\xfdB\xa9\x9f\rX/\xc92\x1bp\xb9\xae\x8a\x10\x91\xb6d=XJu\xa6\xe4\x7f[ގ}\x91\x0f-Г\xf3A\xd6Va\x01[,*z\x01\xa8Ās\x89{\xb0\xc4gB\xa5:\xfc\xc2\x067\xc4\xf1#ۏTk\xbd\x80\xdc{\xe3\x16\xf3y&}\x93RS]\x96\x95\x92~?\x0f\xd9Q\xae*\xaf\xad\x9b\v\xdaR1w2KЦ\xb9\xf4\x94\xfa\xca\xd2\x1c\x8dL\xc2ETH\xab\xb3R|e\xeb$\xeczǎ<2\x8e\x90\b/P\x0fgF\x90\x0e\xb0f\x15\xafx\xd0B\x13\xdf\xdf\xfd}\xf9\x04\r\x92\xa8\xa9\xa8\x94\x03\xe9H.\x8d~X\x9aR\xad9B\xf3\xbe\xb5\xd5e\xe0IJ\x18-\x95\x0f?\xd2B\x92\xf2\xe0\xaaU)=\x9b\xc1\x7f*r\x9eU7d{\x17\xca\x0e\x0e\xae\x95a3\x17C\x82{\x05wXRq\x87\x8e>\xb3\xaeX+.a%\xca,]\xf9L\x9a\x05\x8f\xb5\xa4B\x84,}\xfe\xecI\v\xe1q\xbf\x8e B\xec\xf5\x1a\x10\x8c\xa4X\xfe\xb7y\x0f\xa4r\x9ePԓ\x1cn,\xd5k/bL=\n\x92\xc7!?\xb2J\x009\xc6K\x01\xff\\\xfe\xfba\xfe\x0f\x1d\xef\x01\x98\xa6l)\\\xc3PIʿhK!ANZ\x12\\\x0fҬD%\xd7\xe4\xfc\xac\xe6F\xd6\xfd\xf4\xea\xe7i\xf9\x01|\xaf-\xd0G,MA/@F\x99\xb7i\xa3\xb1\x1a\xe9\xe2\xc5[\x8e\xb0\x93>\x0f@\x8d\x16\xf5\x05w\xe1\n\x1e7\xec1\xf1\n\x15A!74-}\x80\xdbP`\x1f`\xfe\xca!\xe5\xb7[\xf8&\x06\x89[\xfey\x1ba\xb4\x05B7\xea\x1c\xe0\xf8\x1c=x+\xb3\x8c\x0e\x95\xfc\xc8X8\xa1q*\xf8\x16\xb4\xe5\xbb*\xdda\x11\x18\xb3\x9eb &1\x82\xf7ӫ\x9fo\u16fe\f\x8e\x1c%\x95\xa0\x8f\xf0\x8a}<\xc8\xc6h\xf1\xed\f\x9e\x82\x1d\xec\x95Ǐ|R\x9akG\n\xb4*\xf6\xf1a\xb4%p\xba$\xd8QQ$\xb1\x14\x13\xb0\xc3=\xe8\xf5\x91s\x1a\x15\xb1i\"\x18\xb4\xfed9V\xcb\xe1\xb4ӌ\xeb\x93f<\xcf_B\xbd\xf2,\xef\xfdb\xb9\xfe\x99\x92\b\x85\xf9'H\xa2\xfb\xe4\xbcB\x12\x9bjEV\x91\xa7 \f\xa1S\xc7rH\xc9x7\xd7[\xb2[I\xbb\xf9NۍTY\xc2ƘD\xad\xbby\xe8'̿\n\x7f\xae\xbdx\xe8<|\xea\xed{\x8d\x92\xcf/\x02>\xddͯ\x91@SG??w\x1d\x95ò\xae\xec\x86<\xd9\xe7w\xb9L\xf3\xe6UՉ\xb6%\x8a\x18\x8eQ\xed\xbf\x90ﰜ+ˈ\xf6I\xdd4LP\t\xfe\xdfI\xe7y\xfe\x1a\xc1V\xf2\x93\x82\xcb\xfb\xfb\xb7_ң*yM$9\xf2Z\x88\xe3cr@\x95\x94h\x92H\x8d^\x972\x1dPs\xad|/XIkI\xf6L\xf5\xf7\xaeG\xdcT\xed\x13UwKsQ\xd9\xed\x14\x1a\x97k\x7f\xff\xf6\f\x8eeK\xd8`8\xe8\xb0.:\x1b^\x83\x8e\xdcex\x82o=\x1c\x8f\\}P}\xea\x06\x99\xb62\x93\n\x8bC\x04\fO1\x85%v;\xb1\xddQ\xa21Re\x17am\x1a\x9bK\xf2\xfc|\x9f(\x9c\xbb-\xe9S\xe5\xf5I\xbb;\xefR\xef\a@\x00-\x01\xf2\x9dXC\x1b\xda'\xb1\x8a3(\xb9\x04\xe3*\xab.UW\x04hL\xc1uR\xac̦|\xbdiӦZ\xadeV\xd9\xf0(\x1cKJUE\x81\xab\x82\x16\xe0mu\xec\x114\xe9>\xdd\x0e\xf1\x19\x8d\xbf\xef\x906\xea>ӣ\x9e\xbeU\xafs=\xbe\f\xa9\xaa\x1cCI`\xa3\x8dĉy6\xf6\x91\xa3\xf3\xc2\xed\xed%&\x15=\xe9\x8c\f\xea\x86\xea\xc4\x03\xbevĺ\xac\xaf\x9f\xac\xd1\x1d\xa7\xb3\xe3\xa5\x0e\xcaOk~\xa3\xf4\x11&ӽ\x8a\x01\x8d\xd1\xe2f(\xb4nl\x1b,\x1e\"\xd3p\xa1\xef\xf4\x83\xd5^\xa3\xbf{\x9bq\x9b't\x91/i\xf4\xc4\xceu-\xf7\x98V}\xd3\xcf\xe6\x82\xfd\xeaVO\xaa\xf9\xe5\xd6k:_\xd5\t\x19\xb3\tMZ+jG\x91%\x85>B\xec^\xec\xd05'O\x19A\x97_\xdc\x1a\xea\x1bfG\"<\xc1\xf8\x85\xb8FY\x90\x80\xf6\x83\xe2\x04\x9b\xa7\x9c\xc0\x85\x16\xe5\u05eeeT9\x12!*O\x80\x1e'\xe7惀@O\t\xb3\xb8.\xfaL\xfa\\I\xceav\xce\xe9~\x8cT\xb1\x13So\x01\\\xe9ʷ\xad\x98\xda\xfbjQ|\xedjӸ(\xa7\x84\x06ҹ\x86\x10\xd3L\x99a\x1b\aN\xdb!\x9c\x88o\x0f\xb4\x9b\x98\x1d}\xa0\xe9.\xde5&4\xb1\xf6}\xb0\x8e\x8b\x04P\x1ft\x8d\xfd\xb7m\xb8\\\x17\x8d\xc9k\xcfu@U\xaeȲt§\xa2FLm\xc1\x82Jt\x859\xf5\x96j94Q3\xb2\xaa\xdb\x01u\xff0\x18\xb5\xd7 \xa43\x05\xee\xdb˄\x02\x96-x\xba\x9bz0\xa3\xc659P\x1cI\xb3\xa7\x1bu\xed\xa7\xb0\xe9\xf2|\xea\xc3Z\x7f\x8c\xbf\x92\r\xd6\xdbo\x80\x7f\xcc\t'\xca\x04\xe7\xd1\xfaO\n\x90\xcb\x1e\x87s\xb11\x9c7\x1d\x19O\x87\xb4\xfe1\x9f3\x9aMJo4\x19\x90\x8b\x0e\xef\xba\xe3ߝ\xa9V\xed\xe7\xb0\x05\xfc\xfa\xdb\xcd\xff\x03\x00\x00\xff\xff\xfa\x1dp/\xd6\"\x00\x00"), + []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xc4Z_\x93۶\x11\x7fקع<$\x991\xa5\xc6m3\x1d\xbd\xd9\xe7\xa6smr\xbd\xb1l\xbfd\xf2\xb0\"V\x12\"\x12@\x01P\xb2\x9a\xe6\xbbw\x16 )\xfe\x81\xa4\x93<\x8e\xf9b\v\x00\x97?\xfc\xf6/\x16\x97e\xd9\x04\x8d\xfc@\xd6I\xad\xe6\x80F\xd2GO\x8a\x7f\xb9\xe9\xf6on*\xf5l\xf7\xddd+\x95\x98\xc3}\xe5\xbc.ߒӕ\xcd\xe9\r\xad\xa4\x92^j5)ɣ@\x8f\xf3\t\x00*\xa5=\xf2\xb0\xe3\x9f\x00\xb9V\xde\xea\xa2 \x9b\xadIM\xb7Ւ\x96\x95,\x04\xd9 \xbc\xf9\xf4\xeeO\xd3ﾟ\xfeu\x02\xa0\xb0\xa49\x18-v\xba\xa8J\xb2伶\xe4\xa6;*\xc8\xea\xa9\xd4\x13g(g\xe1k\xab+3\x87\xe3D|\xb9\xfep\x04\xfd\xa4Ň \xe7m\x94\x13\xa6\n\xe9\xfc\xbf\x92\xd3?J\xe7\xc3\x12ST\x16\x8b\x04\x8e0\xeb\xa4ZW\x05\xda\xf1\xfc\x04\xc0\xe5\xda\xd0\x1c\x1e\x19\x8a\xc1\x9c\xc4\x04\xa0\xdeg\x80\x96\x01\n\x11\x98\xc3\xe2\xc9J\xe5\xc9\u07b3\x88\x86\xb1\f\x04\xb9\xdcJ\xe3\x033C\x88\xe0<\xfaʁ\xab\xf2\r\xa0\x83G\xda\xcf\x1eԓ\xd5kK.\xc2\x03\xf8\xd5i\xf5\x84~3\x87i\\>5\x1btT\xcfF\x8a\x17a\xa2\x1e\xf2\a\xc6켕j\x9dB\xf1N\x96\x04\xa2\xb2A\xb5\xbc\xff\x9c\xc0o\xa4\x1b\xc3ۣc\x88և\x8d\xa7\xc1\x84y\x16\xe9<\x96f\x88\xaa\xf3j\x84%\xd0S\nԽ.MA\x9e\x04,\x0f\x9e\x9a\xad\xac\xb4-\xd1\xcfA*\xff\xfd_N\xf3Q\x136\r\xaf\xbeѪO\xcek\x1e\x85\xcepD\xc2\xdaZ\x93M2\xa4=\x16\x9f\x02ij\x80ם\xf7#\x92(\xb7;~\x11\n\x9b\x1e\xe8\x15\xf8\r\xc1k̷\x95\x81\x85\xd7\x16\xd7\x04?\xea<\xaap\xbf!Ka\xc52\xae`\x0f\x06ɺ\xd36\xa9:C\xf94\xae\xad\x855\xb2\x06\xfa\xeb\x7f\xe8\xb3\xd8Wn\t\x93\xf6Մ\xa2iX!\xb5J\x1b٫5=\xcb\xc0\xbaD*-\xa8\xc3\xda\b\x97t`\xac\xceɹ3\x86\xcfBzH\x1e\x8f\x03\x17)\xdaPX\xd3\x00\xaaL\xa1Q\x90\x05\xafa\x83J\x14\x14u\xe8-*\xb7\xaa-c\xac\xc2\xe6\xb5w\aӇ\xf2\xbe\x91י\x19a\x8aKw\xdf\xc50\x98o\xa8\xc4y\xbdV\x1bR\xaf\x9e\x1e>\xfcy\xd1\x1b\x06\xa6Ő\xf5\xb2\x89\xcc\xf1\xe9$\x9e\xce(\xf4\xf7\xfc\xbf\xac7\a\xc0\x1f\x88o\x81\xe0\fD.pQ\xc7W\x125\xa6ȑt`\xc9Xr\xa4bN\xe2aT\xa0\x97\xbfR\xee\xa7\x03\xd1\v\xb2,\x06\xdcFW\x85\xe0ĵ#\xeb\xc1R\xae\xd7J\xfe\xb7\x95\xed\x98p\xfeh\x81\x9e\x9c\x0f\x8eh\x15\x16\xb0â\xa2\x17\x80J\f$\x97x\x00K\xfcM\xa8TG^x\xc1\rq\xfc\x14\xacI\xad\xf4\x1c6\xde\x1b7\x9f\xcd\xd6\xd27\xe98\xd7eY)\xe9\x0f\xb3\x90Y\xe5\xb2\xf2ں\x99\xa0\x1d\x153'\xd7\x19\xda|#=徲4C#\xb3\xb0\x11\x15R\xf2\xb4\x14_\xd9:\x81\xbb\xdegG\x8a\x8eOH\xa2W\xa8\x87\xb3*{\x02֢\xe2\x16\x8fZ\xe0!\xa6\xee\xed\xdf\x17\xef\xa0A\x125\x15\x95r\\:\xe2\xa5\xd1\x0f\xb3)Պ\r\x9f\xdf[Y]\x06\x99\xa4\x84\xd1R\xf9\xf0#/$)\x0f\xaeZ\x96ҳ\x19\xfc\xa7\"\xe7YuC\xb1\xf7\xa1d\x81%;\x14\xc7\x011\\\xf0\xa0\xe0\x1eK*\xee\xd1\xd1\x1f\xac+֊\xcbX\t\xcf\xd2V\xb7\x10\x1b.\x8e\xf4v&\x9a*\xea\x84j\x87\xf1ma(g\xcd2\xb9\xfc\xaa\\\xc9:\x93\xac\xb4\x05\x1c\xad\xef3\x95\x0e\x01\xfc$3\xcap\xd1%\xb3\xe3\xe7uJP\x83Xu\x02y\x9d\xef\\\x9d\xa8\x8a~\xa2\xea>\xa3\x1ci\xc9h'\xbd\xb6\x87c\xa6\x1c\x9a\xc4I\xed\xf0\x93\xa3ʩ\xb8e{\xf7\xe1M\x90J0\xefԚ4\a\xa3(5\x00\xd5j\xad\xd9\xc9F\xea\x80\a\xcf\xeb\xd8\xce\x1d\xf9\xf4f\xd5\xc9\xcc&\x15\x1ckL\xe8֒\xc3m/\xb5.\b\x87l\x1a-.l\xfaIׁ\xc3Ҋ,\x85\xfc\x1fì\xd1!\x18{\x94\xaa\t\x1f\xb1\xe4\x06\xaf\x13\xfbXr\xb89\xa5\x9a\xd3v\bgRR\x12𫧇&\xed4\x96UC\x1fe\x96.?I\xb3\xe0g%\xa9\x10!Q_\xfev\xd2B\xf8yXE\x10!\xf6z\r\bFRN\xbd\xbc\aR9O(\xeaA\x0e7\x96\xea\xb9\x171\xa6\x9e\x04\xc9\xcf1?\xb2J\x009\xc6K\x01\xff\\\xfc\xfbq\xf6\x0f\x1d\xf7\x01\x98s%\x14\xce*T\x92\xf2/\xda\xf3\x8a '-\t>}дD%W\xe4\xfc\xb4\x96F\xd6\xfd\xfc\xf2\x974\x7f\x00?h\v\xf4\x11\xb9\xe8\x7f\x012rަ\x8d\xc6j\xa4\x8b\x1bo%\xc2^\xfaM\x00j\xb4\xa87\xb8\x0f[\xf0\xb8e\x8f\x89[\xa8\b\n\xb9\xa54\xfb\x00w\xa1x:\xc2\xfc\x8dC\xca\xefw\xf0M\f\x12w\xfc\xf3.\xc2h\v\x84n\xd49\xc2\xf1\x1b\xf4\xe0\xad\\\xaf\xe9Xh\x8f\x8c\x85\x13\x1a\xa7\x82oA[ޫ\xd2\x1d\x11A0\xeb)\x06b\x12#x?\xbf\xfc\xe5\x0e\xbe\xe9sp\xe2SR\t\xfa\b/\xd9\xc7\x037F\x8bo\xa7\xf0.\xd8\xc1Ay\xfc\xc8_\xca7ڑ\x02\xad\x8aC\xac7w\x04N\x97\x04{*\x8a,\x96b\x02\xf6x\x00\xbd:\xf1\x9dFEl\x9a\b\x06\xad?[\x8e\xd5<\x9cw\x9aq}\xd2<\xcf\xf3\x97P\xaf<\xcb{\xbfX\xae\x7f&\x13\xa10\xff\x04&\xbaG\x9d\x1b\x98\xd8VK\xb2\x8a<\x052\x84\xce\x1d\xf3\x90\x93\xf1n\xa6wdw\x92\xf6\xb3\xbd\xb6[\xa9\xd6\x19\x1bc\x16\xb5\xeef\xa1e3\xfb*\xfcs\xeb\xc6C\x9f\xe5Sw\x1f\x84|9\n\xf8\xebnv\v\x03M\x1d\xfd\xfc\xdcu\x92\x87E]\xd9\re\xb2\xcf\xef72\xdf4\xa7\xaaN\xb4-Q\xc4p\x8c\xea\xf0\x85|\x87y\xae,#:du\xc31C%\xf8\xffN:\xcf\xe3\xb7\x10[\xc9O\n.\xef\x1f\xde|I\x8f\xaa\xe4-\x91\xe4\xc4i!>\x1f\xb3#\xaa\xacD\x93\xc5\xd5\xe8u)\xf3\xc1j\xae\x95\x1f\x04+i%\xc9^\xa8\xfe\xde\xf6\x167U{\xa2\xean\xd7\\Uv;\x85\xc6m\xb4\x7fxs\x01Ǣ]\xd8`8\xea\xb0.:\x1bY\xec\x12gk\xcdsx\x82o=\x9e\x8e\\}P\xfd\xd5\r2m\xe5Z*,\x8e\x110\x1c\xc5\x14\x96\x18~%t_\xa21R\xad\xaf\xc2\xda\xf4\x8b\x16\xe4\xf9\xf8\x9e(\x9c\xbb\xed\xecs\xe5\xf5Y\xbb\xbb\xecR\xef\a@\x00-\x01\xf2\x9eXC[:d\xb1\x8a3(\xb9\x04\xe3*\xab.U\x97\x04hL\xc1uR\xac\xccR\xbe\xdet\xbfr\xadVr]w\"\xc7L\xa9\xaa(pY\xd0\x1c\xbc\xadN\x1d\x82\x92\xee\xd3m\xbc]\xd0\xf8\xfb\xce\xd2F\xdd\x17Z\x7f\xe9]\xf5\x1a\x82\xe3͐\xaa\xca1\x94\f\xb6\xdaHL\x8c\xb3\xb1\x8f\x1c\x9d'\xee\xee\xae1\xa9\xe8I\x178\x88g\xd0\xd4\x01\xbevĺ\xac\xaf\x8f\xac\xd1\x1d\xd3\xd9\xf1Z\a\xe5\xa35\x9fQ\xfa\b\xb3t\xafb\xb0\xc6h1\x19\x92֍m\x83\xc9cd\x1aN\xf4\x9d~0\x1b)xV\x9b'4\x9e\xafi\xf4\xc4륚\xf7\x98V}s\xe9\xc4\x05\xfbͭ\x1e>\x13\x1aO\xa2\xed\xc9\xdf\xd2\ay5\x14\x12\x1a\xb4V\xd4N\"Kj\x9b\x06\xb5\x9d\xd8c\x1b#\x86lc\xc9`\xd2\" 4\xd9]h4~\xed\xa24\xe9\xa0r$Bl\x1d}|$\xa1\xb9\xf3\x11\xe8)\xe3\xf7o\v \xe9\xe6Q\xbc\xee\xea\xdej\xdc\xd4I\x1a\x8b\x19s\x88-mᾥ\xb9hKQv\x94\xd7\x12\x16ő\bGX>a\xafP\x16$\xa0\xbd̽\x9a\xf9\x04\xe8qq\xf39\xc9/\xc99\\_\nZ?\xc5U\xb1\x93U\xbf\x02\xb8ԕ?a\x95_\xbbڵ\xae\xca\xc9J\x8bKH\x1e\xb5\b0\xd4\xe9+\xac1\x9a\x84Z\xba\xd7ZWa\fM\xc2KM?^\x93\n5-\xe4\xf3\xb1\x06\xce\xe4\xb0G\xda'F\x1b\x0fNL=\xd5a!15\xba_\xefN\xc6\xcel\xaa\xa6i\xe6\x922\xdb\xcb\xeb\xc4\xdc\x0f\xc1]\xaeb\xbb\xc6wK@h\xfb\xba\x1b]41 \\:\xab\xaa\\\x92eU\x84k\xedF'm\x05\x8cJt5\x97:\x9c\xb7\x12\x9a4\x1cE\xd5\xfd\xa5\xba!\x1d\xbc\xdck\x10ҙ\x02\x0f\xedf\u0089\x88]:ݞ?\xfaU\x13\xab8\xf3\x9c\xa8\xdb\xcew~\xdb?\x02H\x9f\xf7R7\xf9\xfdg|'?\x98o/\xf7?\xcf\x17\xceԝ\xfd?\xb6\xb8\xc5@\x16=\t\x97\x92E\xfd\xc7\x1f\xd7\xc7\xf8\xfeg\xfe\xc8\xf0\x9edo4\x18\x90\x8b\x8e\xec\xfa\n\xa9;R-\xdb\xfb\xd59\xfc\xf6\xfb\xe4\xff\x01\x00\x00\xff\xff\xec\xe3\xc3\ac%\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xdc=[s\xdb8w\xef\xf9\x15\x98\xf4a\xdb\x19\xcbi\xa6\x97\xe9\xf8\xcd\xf5:\x8d\xfb}\xebx\xec4\xfb\f\x91G\">\x83\x00\x17\x00\xa5h\xdb\xfe\xf7\x0e\x0e.$%\x90\x84d˛-^2\xa6\x80\x03\xe0\xdc\xcf\xc1\x01\xb2X,\xdeц}\x03\xa5\x99\x14W\x846\f\xbe\x1b\x10\xf6/}\xf9\xfco\xfa\x92\xc9\x0f\x9b\x8f\uf799(\xaf\xc8M\xab\x8d\xac\x1fA\xcbV\x15\xf03\xac\x98`\x86I\xf1\xae\x06CKj\xe8\xd5;B\xa8\x10\xd2P\xfbY\xdb?\t)\xa40Jr\x0ej\xb1\x06q\xf9\xdc.a\xd92^\x82B\xe0a\xea\xcd?^~\xfc\xd7\xcb\x7fyG\x88\xa05\\\x11\x05\xdaH\x05\xfar\x03\x1c\x94\xbcd\xf2\x9dn\xa0\xb00\xd7J\xb6\xcd\x15\xe9~pc\xfc|n\xad\x8fn8~\xe1L\x9b\xbf\xf4\xbf\xfe\x95i\x83\xbf4\xbcU\x94w\x93\xe1G\xcdĺ\xe5T\xc5\xcf\xef\bхl\xe0\x8a\xdc\xdbi\x1aZ@\xf9\x8e\x10\xbft\x9cv\xe1W\xbd\xf9\xe8@\x14\x15\xd4ԭ\x87\x10ـ\xb8~\xb8\xfb\xf6OO\x83τ\x94\xa0\v\xc5\x1a\x83\b\xf8\x9fE\xfcN\xc2B\tӄ\x92o\xb8Q\xbb\x1aD<1\x155DA\xa3@\x830\x9a\x98\n\bm\x1a\xce\n\xc4;\x91\xab\x1e\xa40J\x93\x95\x92u\amI\x8b\xe7\xb6!F\x12J\fUk0\xe4/\xed\x12\x94\x00\x03\x9a\x14\xbc\xd5\x06\xd4e\x04\xd4(ـ2,`ٵ\x1e\xef\xf4\xbeNm\xcc6\x8b\v7\x8a\x94\x96\x89\xc0m\xc1\xe3\x13J\x8f>\"W\xc4TLw[\r\xdb#T\x10\xb9\xfc\x1b\x14\xe6r\x0f\xf4\x13(\v\x86\xe8J\xb6\xbc\xb4\xbc\xb7\x01e\x91Uȵ`\xbfG\xd8\xdan\xdcNʩ\x01m\b\x13\x06\x94\xa0\x9cl(o\xe1\x82PQ\xeeA\xae\xe9\x8e(\xb0s\x92V\xf4\xe0\xe1\x00\xbd\xbf\x8e_\x90xb%\xafHeL\xa3\xaf>|X3\x13$\xaa\x90u\xdd\nfv\x1fP8ز5R\xe9\x0f%l\x80\x7f\xd0l\xbd\xa0\xaa\xa8\x98\x81´\n>І-p#\x02\xa5\xea\xb2.\xff.\x12u0\xad\xd9Y\x1e\xd5F1\xb1\xee\xfd\x80\x02q\x04y\xac\xa88\xc6s\xa0\xdc\x16;*\xd8O\x16u\x8f\xb7O_\xfbLɴ'J\x8f7\xc7\xe8c\xb1\xc9\xc4\n\x94\x1b\x87\xacia\x82(\x1bɄ\xc1?\n\xce@\x18\xa2\xdbe͌e\x83\xdfZЖ\xdf\xe5>\xd8\x1b\xd4:d\t\xa4mJj\xa0\xdc\xefp'\xc8\r\xad\x81\xdfP\roL+K\x15\xbd\xb0DȢV_\x97\xeewv\xe8\xed\xfd\x104\xe2\bi\xbd\x16yj\xa0\x18H\x9a\x1d\xc6VA]\xac\xa4\x1a(\x19;d\x88\xa3\xb4\xf0\xdb洈U\x8b\xfb\xbf\xccq\x99m\xff\x1eG[~\xb3+k\x05\xfb\xad\x05T\xa6N\xfc\xe1P_\xa9\x9ej\x1f6\xcbF\xfb\xd4\x1dE\xb4m\xf0\xbd\xe0m\te\xd4\xeb\a\x1b\xcc\xd9\xc6\xed\x01\x144z\x94\t+D\xd6\xfaؽ\x88\xeeWT\xe0T\x01\x11\xd2$\xe01\xe1\xe0\x11&\x10\x03I\x9a`G\x03ubœ[&D\xb4\x9c\xd3%\x87+bT{\x88F7\x96*Ew#\xd8\n\x1e\xc0\x8b\x90\x15\x81xU\xc3Y\x81$\x8f\n\x05\xf1\xf5\xe7E\x15\xd3VQ\x86]>HΊ\xdd\f\xben\x93\x83\x82\xb4z\xd9\xf5;$K\xa8\xe8\x86I\x95\x12\x03\xa9\xb0kϞwjZZ-\xe9\x81\xec۸\xcc\r'\x91UI\xf9<\xc7\x10\x9fm\x9f\xce:\x90\x02\x1dʸ\x15Omo\xbb\x97@\xe0;\x14\xadI,\x93\x90\xb2E\xd3$\x15i\xa46\xe3t\x1fW]\xa4\xef\x1c\xa5~\x9c`\x9a\x83\x9d%Y\xdd5\xaf\x84\x03Q-\x0e\x06\nY\n\xb0ۨ-Q\xbb\xbeJ\xb6\xae\xef(RȒj(\x89\x14\xa33#\xbb\xb4\x1c\xb4\x9f\xabD\xce\xe8\xf4\xd0E\xb7\x7f\xf4x\b\xa7K\xe0D\x03\x87\xc2Hu\x88\xcc\x1c\x94\xba\x96\xa3XGP\x99ЦC\t\xe860\x01\x92XN\xdfV\xac\xa8\x9c\x87a\xd9\x13\xe1\x90R\x82\xb6\xda\x04]\xe6\xdd\xd8&\xc9\x1c\xf9\xfd$Sڣk3b\xb5\x0f/\xa5Q\xba\x96\xa1\x86\xbb\x96Dm\xa7{\x0ft\x8b\xffn\xe4\xe4\xb6\xff\x7f\"6\x18\x93\x13\x98vB\xfe\t\xba\x9f\xd9<=ʷ\x18ၾ$w+\x02ucv\x17\x84\x99\xf0uN\x12(\xe7\xbd9\xfeĴ9\x9e\xe93I\x93#\x13g\"L\x9c\xe2OH\x174\x19O\xdebd\xd3\xe4\xaf\xfdQ\x17\x84\xad\"\xd2\xcb\v\xb2b܀\xda\xc3\xfeI\xaa>P\xe65\x90\x91c\xf5\b\xe6\tLQ\xdd~\xb7.\x8e\xee\x92`\x99x\xd9\x1f\xec|\xe3\x10A\f\xcd\xf3\f\\\x82\xf12SPc\x1cN\xbe\"6\xbb/\xe8T_\xdf\xff|\x18+\xef\xb7\f\xce;\xd8Ȍйv\xbd\xb7\xa3\xfe\xfa|T\x10~A\x1f(\x06U.\xe7rA(y\x86\x9ds]\xa8 \x96>4tΘ^\x01&\x7f\x90Ϟa\x87`\xd2ٜÖ\xcb\r\xae=C\xc2\xf5O\xb5\x01\x0e\xed\x9a|X\xec\xf0d? \"0\x86\xcfe\x03\u05fc($r'閩KB\v\xb8?a\x9bY\xacҟ\xa3\x9f\xfaD\x0e\xf8I;ZZ\x89\xa9\x98\xcfij@\x99\xc9%\xa8k\xdf(ge\x9c\xc8\xc9ȝ\xb8 \xf7\xd2\xd8\x7f0@\xd3\xc8(?K\xd0\xf7\xd2\xe0\x97\xb3`\xd4-\xfc\x9c\xf8t3\xa0\xa0\t\xa7\xe5-\xc2\xfa9?g\xd3,\xb7E\xdc3M\ue10dW\x1cJ2\xa7\xc2\xf4\xae\x9b\xceMT\xb7\x1a\xd3uB\x8a\x05\xda\xcc\xe4L\x1e\xdfR\r\xd0\xfd\xe2I\xfd\x84_\xad\xb1p\xbf\xb8$3\xa7\x05\x94!\xb2\xc4\xec'5\xb0fE\xe6|5\xa85\x90ƪ\xf0<\x8e\xc8T\xac~7DZO\x9e\xf5\xee\xb7\xef\x8b\xe7\x98/XX\x93\xb3\xf0\x10\x8c\xac3p\xe0uw9\xbf\x9f\x85\x95ٌ^\x81\x13f\xbb\x8e$Gǻ\xe6 \xe5\x05\xe8@+\x8e.\xce,uiY\xe2\x11\x1a\xe5\x0fGX\x94#x\xe1X\xd5\xd0[\xbb3\xc15m\xacZ\xf8okiQ\x9a\xfe\x974\x94)}I\xae\xf1\xa4\x8c\xc3\xe07\x9f\x87\xeb\x81ɘ\xb2\xb1SY\xfe\xd9Pnm\xbfU\xe0\x82\x00w\x9e\x80\\\x1d\xf8E\x17d[I\xed\xcc\xf6\x8a\x01\xc7\xf3\x8a\xf7ϰ{\x7fa\xa7\x9f\x9d\xb2\xafd\xde߉\xf7·8P\x18\xd1ᐂ\xef\xc8{\xfc\xed\xfdK\\\xa9LN\xcd\xec6`њ6y\x1c*\x92\xc9\xfa\xae\r8\xa6\x9f\x9b\xef\x92\xf2\xdeɞ\xdam\x16\x8b6R\x9b\xcf\xe9\xbc\xe1\xc8z\x1e\u0088\xa1g\x9cȱ\xcdF\f>\x8f\x16\xf5\xbdu\"W\x06\x94\xcf%:\x1b\x10\xe2\x8f\x17Ff\xa9S\x99\xfebc2\x90\xc6\xfc\xaeE\xf0\f7\xb9\x83\x9b\x9c%\x1e\xe3\xb0Z\xbc\x1c\xe9\xed\xdf~\xef\xe53\xad\xe4ڿ\xfb\x1bym\x87\xba\x90uM\xf7O5\xb3\x96z\xe3F\x06\x9e\xf6\x80\x1c\xf5պEyε\xc8\x1d\x0f\xe1\xf9喙\x8a\tB\x83\xda\x00\xe5\x19\x8a\x92F\xa6rةVQM\x96\x00\"\xa6\xe8\x7f\x04W\xa2f\xe2\x0e' \x1f\xcf\xe0zDt\x9d\xd3ٽ\x894\x89\x94\x8f\x1f\x9c\xc9jdI\xb6\x15(\x180\xc6a\xde\x1d=U!M/eq\x84C\xda\xc8\xf2'MVLi\xd3_\x82&\xadΥ\xf5\x91\xe4\xb3\xeb\xfe\xcaj\x90\xad9'\x82o\xbbi\x06g\xcd5\xfd\xce\xea\xb6&\xb4\x96\xad3\xe6\x86\xd5\xf1TףwK\x99\x89\xc7V\x98\xbf1Ғ\xa0\xe1`\x80,a\x95>\xefM\xb5B\n\xcdJP\xa1J\xc1\x91\x8dI+\x98+\xcax\x9b:%J\xb5c#`q\xab\xd4I\x01\xf0\x177\xb2\x97w\xac\xe4v\x88\xa0̽\xe3A\x1a\x10\xb6\"\xcc\x10\x10\x85\xc58(\xa7\x92q\n\x8f\fD\r\xcb\xd5sy\n\xdc6\x10m\x9d\x87\x80\x05\n$\x13\x93)\xb7~\xf7O\x94\xf1s\x90\xcdr\xde'\xa9\x1e\x81\x96\xa7\xe4h~\xed\r' t\xab\xf0\xf0\xdf\xe9\x8e-\xe3yk\xb6\x94#\x9c\xb6\xa2\xa8\x00\x95\x90\x18\xea\x06\a\x9e\tm\x80\xe6\xf2\x82\xf5\x8aZ!\x98X\xe7\xd1.;\x11\xda5\x87\ua954\x1c\xe8\xf8)d\xd7,\xae\xdf@\x13\xfd\xdaM\xf3BM\xd4\x11\xc1\x1d\x9b#\x1d\xb2)j\x95\x16\xa1\xc6@\xdd8\x91\x93D\xb5\xa2o]Π\x88\x8e\t\xc3\xfd*^3\xbef\x82e\xd0v@\xd7;\xc1L\xdfy\xb4 \xce\xea<\xda\t\xa2;pJ\x86\xedn\x00\xc0\nh\x88Cp\xed\x91k\x8ep$\x97@hYB\xe9r\x97\xd6\x15\xf1a\x89+|\x1b)nH\xee\xeexO0\x8b\xb2\xa1\r\x82N\xccê\r,Z\xf1,\xe4V,0\x18\xd7G\xeb\x90\x13\xb3T/\x9dޜ\xac\x8c\xe6\xf5K\xbe\x9a\x9e\xd3BC~\xcd\xe7\xa9\xe0?\x9dA\xcbd\xf3\xcdQ\t\x8f).\x98\xd3k\xae\x00{\xe4\xc7\xd9UL\xcd?1\xd8\x1fJ߸b\xe9\x17\x95\xc5ݥA\xf5\x9c\xc2m\x05\xa6\x02\x15J\xb3\x17X\x92^N\x9e\x90v\xc1K\xac\x93\xb3L\x15\\dW\xfe\xb9W9\x87\xd1M\xcb\xf9\x85\xe5m\xda\xf2d8l$\x8a\xd8!geՏ\xa5=\x86\x9c\xea\x8bl<\xf6+-\x86\xf5\x85\xb1\n\"\x14\x18\xca0\xb3\xa7qj\xbfXX\xda;\xdf\x1f\x96S`\xfe/,\xff\x0f/=̨\x94\xc8Gcn\x95fDb\x02V\x82\xc1zh\xec\xea+|?_\xe8\xfbc\xe1\xd4@\xfd\xa5\xf1\x123\xea\xc2f\xa05\x01g\xaf\xde\x04\xadA\xab\x9d+\x10\xed\x80\xcf\x19\xda\xf1ׅ\xbb\x05\x11\xc0\xa4\xf8\xf5k\x05A|}\xf5>\xd3\xe4\x9fI%\xdbDU\xdf\x04\xcaf\xaa;\xe67<(\xf4\xf0\a\n`\xe8\xe6\xe3\xe5\xf0\x17#}\xd9\af\xd1\x12\x800(\xea2\xb3L\x94l\xc3ʖ\xf2 \xb5\xdd\x1d\x02\xc7@\x1d\x9f%\xa0IE\x04\xe3\x8e\x01\xc3\xf8\x01Ñ/\x8d;\x969Z\xc5M\xfb\xa2y\xd5!'ׄ\fk>F\xac\xe1\xb1\xc7\x17\xafR\x05\xfb\x87\xd4z\x1c_\xe1\x91\x13I\xccTs\x9cPÑY,\xf6\xe2\xf3\x96\x9c*\x8dcb\xee\xb3Ud\xbc~\x1dF\x16~\xe6k.\x8e\xc1\xce\xd9\xeb+ް\xaa\xe2mj)2+(^\xaf\x142/\xfa<\xa9\x14`>`\x19\xaf\x82\x98\xad}xQ@sҖfk\x1a\x8e\xa9d\x98\xa5N\x9e\x98\xbdY\xad\u009bU(\xbcm]\xc2$\x17M\xfexL\xe5A\x8c\x93~\xa1M\xc3\xc4\xfa\x90)rYg\x92m\xe6Y\xe6~o!\x03\x9e\xe9\x873]t8\x12\xfa\xba\xeb҉H2\xa4-\x990\xf2\x92\\\x8b\x9d\x87\x9b\x80\xd3\v\x1f\x854\a\x17\xd9첶\x8c\xf3\xfem-\x04;\r\xcaߙԴv\xab\x1a\xf3\xf6\x93t\x95j\xe0\x94\x9f\x148~ك\xd1ώ\xbe\xa5\xe7_\xb7ܰ\x86\x83\xf5\xe86\xacL\xde!3\x15\xec\"\x92\xff&\xf1\x86\xd4r\x87\x90\xbe]\xf4\x9eQ\xec\x9eq\x926\xbf\xc9\x13\xb6\x97Q\xcc~\\\x11{\x06\xcdrE\xf1\r\x8b\xd5߰H\xfd\xad\x8b\xd3g8k\xe6\xe7\xe3\x8a\xd0O>\x81\tG\xfd\xf7\xb2\x84\a\xa9\xcc\\p\xf2\xb0\xdf?q\x92\xda\v\xd8$/\x89\b]\x13\xbb\xc4\x10Ç\x17\xa7m*}\xe8\x19\xdc\xe9_di\xd76w\xc6\xf2\xb8\xd7\xfd\xe0\xae\xf2\n\x14\b\xf7\xcc\xc7\x7f>}\xb9\x8f\xf0S>\xaf\xf7\x8c\xf7\x9e\x97p\x1eL\xe9\x91\xe3\x8f\xe6|1\x93\xc3\x16\xfa\x00\xaf|.B\x1b\xf6\x1f\xf8\xaa\xdb\v\xd2A\xd7\x0fw\b#\xf8i\xf8L\\\xac\xa2\x88'\x96K\xb0\x16+\xa2jT,\xeeV\x03\x88Ê\xdf\xfe3JP\xba'\xb3\x82\xc5d\xa1\xc6\xcb\n\xdeÝ[\xc7\xd8,\x9f\xac\xd3(vD:\x8e\xac\x98*\x17\rUf\x87l\xa3/\x06k\bff*\x9d3\xaaX\x0f\x9f\x01K\xa27\xbc\xfe\x85g\x91\xbbfxڻ\x8f\xbbS\xd61~\xffd\xf6\xe6\xc9+\xaec\xdcb/\x10S\x89\xcf\xc9\x02\x93WK\x93yM\xf4\xf0\xed\xa4\xb4\xcbc\x1c=\xad\xe7l\x14\x1dRM\t0v<\xaa:-h\xa3\xabēK/\xd3u\xf8\x1a\x99\xa1\xa6}\xc9&\x1d\x80\xc1>YQ\xf5\xb4\xd5\x16\x82>\v\xdbFi\xc5a)\xddnm\xb3\xab{a\xfc\xa2\x97)x\x9b#\xe1\xcc\xe7\\N~\xc8šgD\xfd`\xf6˪\xb6CL\x9dp\x18<\xeb\xdae\x14\x19O;\xb1\x99π\xe4\x19\x8c\x13\x9e\xfe@|\xe5\xe2\x8a$_\x04\xc9|\xf5\xe3\x0fE\xf4\x84V\xd3E\x05e\xcb\xe1\xd47\xff\x9ez\xe3\xe7_\xfd\v\xb3e\xbc\xfbg\x91\xdd3\xd0\xd6g\x1e\xbe/\xe8)\xe1!\xf7)9\xe6\xf0ap\xe0\x9e\x17+\xdcK\x94E\x01Z\xafZ\x1e\xaa\x94\n\x05\xd4@\x19\xba3\x1dW|T\x9dM\xdbpIKP7R\xacX\xe2\x84d\x80\xd6\xff\x1at\xde\xe3\xd9\x02?\xb6\xaa{\xdaq\xf2Y\xbc\x17i\xae\x86*\xca9\xf0O\x8c\x83\xfeYn\x85]W\x86@>\xa4\xc6\xf5\xeee\x15\xad\xb2f}GD[/\xad\x93\vƌ\a\x8b+\xa9\xa6+\xa4\x1dޙ0\xb0\x86T|\xbdU\xcc\xc0SC\x95\x06\\Q\xc6\x0e~\xdd\x1b\xe2\xa2\xcf\x15\xa7kW\nW\xb2\x82\x1a\x88\x06\x18g\x18[>\x8e\xd7\b\x8b\xef\xb02I\x8e$\xbd\xb2\x85z\xecJƨX\x8f=/\x9a0\xd5\xc9\aF\x9dE.hc\xf0\x02\f\xd2\x11\x89h<\f|\xb4w\xef\x8d\xd1\x01\xd8qN\xf3e̾`N\x1bZ'\xa2\x84y\xbdss\b\x06\x9f\x05Ve\xaf\xee\xae\xff\xc0b,\xb0#[\xaac1u\xd2\xf7\xee`;0\xe8\xaa[\xd0P\x12\u0600 V\x14)\xe3PNq\xeaWL$\xab\r\xa8\x9ft\x84\x83\x95\x80\x96ş\fU&.\xfdЏYIUSsEJj`aG\x9f溥\x9fIU\xea\xc4\xe3@\xbc\xd9\xe6ţ\b\xd7n\xac\xf5s\xf7\xd1jК\xaeC\x10\xba\x05\x05d\r\xc2\xe2=\xe6\x16\x93\x1eS\xb8\xd2\xe7\x8dE,-\xb5(\xa4\x85i\xa9\x9f\xc0\xb9p\xf1\xf44\xbcO\x8cQ\xeczTE\xa7U\x85\xbf<\xf8\bT\xef?w}\x80\x8bO\xfd\xbe>I\xecv\xec\xceF\xa8+\xf0\xc4\a\x8f\r\x8b\x91uJ\xa6\x8dę\x8f2'\x95\x94\xcfYn\xf6\xe7رK'1\xe1X\t\xafL.ekz~\x8eGxb\x99\xf8\xfc\xe7+\xdb\x17\x84y\xed.P\x8d\xe5V\xf3<\xbd\xcf\x03H1\xbc\x95\x86\xf2`d,_\xc6\x0e\xd5\xc4\x03\x02O\xe1\xf1d\xcew\x17\xfb\x90\xf7^e\xef`W\xddS\x9e^\x13t\xd7\xc7\xc7Ҫ>\xeb\x97\x04\x12_\x01\xed|\x92\xb17\x17\xe7\xec\x1fB\xfd\x84\x8b\xca\xc0\xf1\xe7\xae\xf7\x18\x1e\xdd2\x9d\xc3\f\"\x1di\x12\f>L\x15%ㄥOx\xa9ME\xf5\x9c{\xfa`\xfbD\xb7\xa3g\xae\xa2\x13\xfa8\"\x95\xe9{\xae\vr\x0f\xdb\xc4W\x87,<\xfdB\xa9Jt\xb9\x13\x0fJ\xae\x15\xe8C\xa6[\xe0}F&֟\xa4z\xe0횉/\xe3\x95\xdfS\x9d\x1f\xa82\xcc2\xad[Ob\xecM\xb0q\x89\xdf\xe6G\x8f\xff\xc0\x04\xe5\xec\xf7\x94.\xef\xff87Ä\xbek<\xf2N\xb1P\x01\xf1s\n\xd0k\xe8\x9ft\xcf\xfc\x84y/ɽL\x8a\xb1? fC\xa0L\x93%h\xb3\x80\xd5J*\xe3\xf2\xf7\x8b\x05a\xab\xe0 Y\r\x81q\xa2{͞\xb0T\xe2=\x1e\xbd\x05\x87e\xe5S\x89\n\xad\x0e\x86\x9c5ݹ\x8c$-\n\x1b\x13\xc0\amh*6y\x91\x9e\xc6P\xd5\xcbJ\x8e\n\xb9\xeb\xf7\x8f9\xbe\xa8>\x10\x9cC\x1d^gw\x06\x9d\x8f\x9di\r^\xcb \xdab\xef\x14eB\x9c\x1a\xbb\x1b\x0f\xbb\xf3L\xcd\xd7\beL=\xfa\xfd\r\x1e\xe2\xf6\a\xac\xbe\x93%[QQ\xb1\x1e\xbd\xd0V)ٮ\xab\xc0\x9bc\x0e\x11)[\x8c\x9c\x1bT\x05:\xfc\xc7!\xa6U\xa2wh\xe7k,ƴt\\\uee0f\xf2\x02E\xad\xba\x8b-\x9d\xaa\x9a\xb0\xf9\xd9Y\xc2\x11\x88\xb3\xb6?\x01\x91\xea\x9d(&\xaf\xe0\xf8@\x9bM\xdc՝\xc2P\x12\tQ\x1b\xbf\x1a\x12\"\xc41$\xf4}\x89.\xe2\xf9a02棜\x88\x8ei'\x06\xb78\rj~\xd3}'h\xe8\xee\x1c\x87\x0e=\b\xfeNJ\xbb\r \x1c\x13\xf9\xe2\xdc\xe9\xb8\xf7ǍX7\xd1ۺ=9v\xfd\xb6\ac\xef\n\xa4\x8db\xbbiB\xbc\xf9\xf7l\x95\x92\x17\xf7\xbf3-9\xfc\xc3\xc1\xafo|\x95qK\x95`b}\x12F~\xf5c\x13\xf1\xbc\a{Έ>\xac\xfc\xd5b\xfa\xa4Y:\xf8\x88\f^\xf6\xf0\xecg\xf2_\xfe/\x00\x00\xff\xffP\a\xb5\x16Cm\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xec=]s\x1c)\x92\xef\xfa\x15\x84\xeea?B\xdd^\xc7}ą\xde|\xb2gO\xb1\x1e[ai\xf4\xbctU\xb6\x9aQ\x15\xd4\x00\xd5r\xdf\xde\xfe\xf7\x8dL\xa0\xbe\xba\xe8\xa2Z-ygǼت\x86$\xc9L\xf2\x03\x12X,\x16g\xbc\x12\xf7\xa0\x8dP\xf2\x92\xf1J\xc0W\v\x12\xff2\xcb\xc7\xff6K\xa1\xdelߞ=\n\x99_\xb2\xab\xdaXU~\x01\xa3j\x9d\xc1{X\v)\xacP\xf2\xac\x04\xcbsn\xf9\xe5\x19c\\Je9~6\xf8'c\x99\x92V\xab\xa2\x00\xbdx\x00\xb9|\xacW\xb0\xaaE\x91\x83&\xe0\xa1\xebퟖo\xffk\xf9\x9fg\x8cI^\xc2%3\xd9\x06\xf2\xba\x00\xb3\xdcB\x01Z-\x85:3\x15d\b\xf4A\xab\xba\xbad\xed\x0f\xae\x91\xef\xd0!{\xeb\xdbӧB\x18\xfb\x97\xde\xe7\x8f\xc2X\xfa\xa9*j͋N\x7f\xf4\xd5\b\xf9P\x17\\\xb7\xdf\xcf\x183\x99\xaa\xe0\x92}®*\x9eA~Ƙǟ\xba^0\x9e\xe7D\x11^\xdch!-\xe8+U\xd4e\xa0Ă\xe5`2-*K#\xbe\xb5\xdcֆ\xa95\xb3\x1b\xe8\xf6\x83\xe5g\xa3\xe4\r\xb7\x9bK\xb64ToYm\xb8\t\xbf:\x129\x00\xfe\x93\xdd!n\xc6j!\x1f\xc6z{Ǯ\xb4\x92\f\xbeV\x1a\f\xa2\xccrb\xa0|`O\x1b\x90\xcc*\xa6kI\xa8\xfc\x0f\xcf\x1e\xebj\x04\x91\n\xb2\xe5\x00O\x8fI\xff\xe3\x14.w\x1b`\x057\x96YQ\x02\xe3\xbeC\xf6\xc4\r\xe1\xb0V\x9aٍ0\xd34A =l\x1d:\x1f\x87\x9f\x1dB9\xb7\xe0\xd1\xe9\x80\n»\xcc4\x90\xdcމ\x12\x8c\xe5e\x1f\xe6\xbb\aH\x00F$\xaaxmH8\xda\xd67\xddO\x0e\xc0J\xa9\x02\xb8\x80vX4\xb6\nu%\xa0\x80\xe6\f\xddN\x8d\x16FH\xb6\xae\xd1#]2\xd4\x12Q\x19\x11\xd2X\xe0\x11a>\x01\xef\xe0kV\xd49\xe4WEm,\xe8\xdbLU\x90\x87E\xa6Q͜\xca\xc3\x0f\a!\xfb\xf8\xa5\x10\x19 \x1f2WiA\x8b<1\xd1nC\x99]\x05n\xcd\tY\xed\x87\xd0\xc6(\x93\xbaŀņ\xe7\x7f<\xbf \t\xe8\xf7\xde\xef\xc70\xae\xa1!\xd3,\xddL\x16\x7f\xbc\x85\xb0PF\xa8;\xa9\xa3f\xf0\x9dk\xcdw\a\xb8\xde,\xa6\xbd\x00\xdfc\xb0\a\x9c\x97\xa1\xda7\xe2\xfd\xb0\xff\xdf\"\xf7O\xcboC\x8b\xce\\H\xe4s!\x8c\xed\xb1ٸU,$\xebX\b\xe9\t$\x1dLT\x93S\\\xfd'!\xe6I\xe7Nl\xb24\xb2\xe9'\xc0\xbf\x14%7J=\xa6P\xef\x7f\xb1^\xbb\x84\xc52\xda\x18a+\xd8\xf0\xadP\xda\f\x97I\xe1+d\xb5\x8dj\x16nY.\xd6k\xd0\b\x8b\x96\xf9\x9b]\x81C\xc4:\x1c\xbe\xb0\x8eʊV\x18\x8c\xabe:\xb2\x94\xa8\x11\x1b\n\x05\xa8Q\xa8\xce\xc1\xc1Ђ\x1c\x88\\lE^\xf3\x82|\t.37>\xde\xe0\x17\xd3j\x13\x02\xb1\x87\x7fT\xaa]q\x0eM\x18$2\xb1\xb7\xea\xa5$\xa0\x8f_bl\xb4_5N\x89\xb0\x94p\xb0od\xa6\xae\v0\xbe\xbb\x9c\xdc\xe4V']\xb4\xccrk\f\x05_A\xc1\f\x14\x90Y\xa5\xe3\x14J\x91\x03WR\x95n\x84\xb8#Z\xb6\x1fm\xb5\x83\x99\x00\xcb(\xc4݈l\xe3\xdcW\x144\x82\xc5r\x05\x86VExU\x15\x11\xd3ՖI\xe1\xf0\x9dM鍶$h\x90!ܘ.iK\xa2~n\xcb(\xd9۹٧\xfa\xf8:\xff(\xbe\xbf%\xa2\a\xabs\xa4\xb0Oh\x12F\xfb\x05\xc9\xf3!Jz\xa4\xb8\x00\xb3\xec\xac\xce\t\x1b\xbe\xa60\xb4\xe7?\xeem\xa5\xec\x11\xe5\xd7Ż\xe3&\xcc\f\xd6MΩ\x97e\\\xd3Ϳ\b\xdf\xc8d\xddz\x8b5\x8bg\x1f\xbb-/hW\xc03$\xbf`kQX \xa7j\nQ6\x83s\xa7$P\xaa\x05f\xb4Il\xb3͇f\xef(\xa1ŀVC\x00\xceA\x0fQ\x0e\xf1 \x01$k\\\v\xda4\x15\x1aJڌ\xa5H\xb2\xfb\x85\\\xc1w\x9f\xde\xc7c\xcfnI\x94ԽA%LZW\xde\r\x1c\xa3.\xae>T\t\xbf\x90\xbf\xd6\x04\x82n\x13\xfe\x82q\xf6\b;\xe7bqɐo)\x8b\xff|\xf8*\fv,s\xf6^\x81\xf9\xa4,}yQ*\xbbA\xbc\x06\x8d]O4A\xa5\xb3$H\xc4n\U00088ce5(\xa8\r?\x84a\xd7\x12C2G\xa2\x19\xddQ\xae\x90\xeb\xd2uVֆ\xb6Z\xa5\x92\v\xb7,6֛\xe7\x81\xd2=\x16\x9c\xa4c\xdf\xe9\x1d\x1a#\xf7\x8b\xcbZ*x\x06yآ\xa3t\x1an\xe1Ad3\xfa,A?\x00\xab\xd0,\xa4K\xcb\fE\xedG6_\xbc\xd2=\x87n\xf9\xbax\xacW\xa0%X0\v4k\v\x0fŪ2\x91.\xde&\x8c䜌\x95\x05\xce\xf5ĚAZ\x92\xaaG2r\x0eWO%\xd63\xc9D^\x04\xb9]IR\xd0Ml\x9dg\xbdf\xca\xcd1*\xa63\x16\xe7\x02\x94\x9c\xb6\xd6\xfe\x86\x96\x9ef\xe3\xdfYŅ6K\xf6\x8e2{\v\xe8\xfd\xe6\x17&;`\x12\xbb\xadh\x95\xfd\x97Zly\x81\xfe\a\x1a\bɠpވZ\xef\xf9j\x17\xeci\xa3\x8cs\x1b\x9aM\xbb\xf3Gع\x1d\xe5\xa4n\xbb\n\xeb\xfcZ\x9e;_fO\xf14\x8e\x8f\x92Ŏ\x9d\xd3o\xe7\xcfu\xeffH\xf4\x8c\xaa=Q.y\x95.ɔ7;'\xd0\xc0`=8DظI \xc5\x00a\x8a\x02ɢ\\)\x13I\x16\x89\xa0\x95 \xe87\xcaX\xb7\x0e\xd9\xf3\xf7G\x17*UX\x9cd|mA3c\x95\x0e)\x99\xa8\xf8S\x96\xe2\xbb\xe5n\x03\x06\xfc>\x94_\xf4t\x801\x8a=ou\x83\xb3*\xe7n/\x8c:\xe2\x19yOԶ\xd2*\x03\x13͋hK\xa2m\xeaQp\x9f\x0eͺ.w\xd1\xdf:Ik\xa7,J\x872ϑG\xd2\x1d\x11\x19}\xf8\xdaY\xa2F\xed\x82\x7f\xa7H\xeb182:\xafQ\x96|\x98\x0e\x9c\x8c\xee\x95k\x1d\xe6\x98\a\xe6\xc2-\xfdP\x93Ι\xe3u4\xa2\xfc\xcf\xe6ڔB^SG\xec\xed\v\xbaC^\x8b\xc7ң\xc6\xca\xf1N\xfaU\xe8\xac\xe5^\xf3\xc1\xe7\xd4)\xda\xf8\xd1\xd0c\xee\xfe\x9e\by\xd7R\xd9\xce2\xceL'\xbaR\xf9\xef\f[\vml\x17\rs \xb1j\x14\xd4\x11\xa1\xa7\xfc\xa0\xf5ё\xe7g\u05fa\xb3\xa0\xb8QO>qzN\xbc\x1dH\xba\xe1[\xf0\x99\xab 3UKZ\nC=\x80\xdd̀\xe8X\xe3\xac@\xa2\xbd\xeb4\x96u\x99N\x90\x05I\x92\x90\x93\xebf\xdd&?p\x91\xb6nŎc\xab=\x94\xc39V\x8e\x9fG!\xc1\xb3\x9bN_\U000af8acK\xc6K\xe4!\xb9\x1d\xa2\x84&\xa3ޱ\xbbI\xfb\xc4\x16d\xb4\xac\xc2YV\x15`\xc1\xa7m\xce\xc0#S҈\x1c\x1a\xd3\xefE@I\xc6ٚ\x8b\xa2\xd63\xb4\xeal\x92\xcf\r¼69}d\x95\x8eȂH\x94\xb8\xce>\xc3\v\x9e\xd6\xf8\x95\x9e\xe7Ǧ8\x8c\x1a\xe6\xfb\x8b\x95\x16\xca\x1d\x068\xbd\xcb\xe8ӎ\xb9\xdc}\xf7\x19\xbf\xfb\x8c\xdf}\xc69\x1d}\xf7\x19'\xcaw\x9f\xf1\xbb\xcfx\xb8|\xf7\x19S\xcaw\x9fq&\"\xdf\xcagL\xc1pAk\x9c\a*$a\x95\x98\n1\x85\xf6D_>\xe9ǟ\xd58I.\xf3\xf58ȑC<\x91\xe3\x171\xaf\xa35^Mr3\xce\xc00w\xdc)\xca\x04\x87\xf9\x04\xa7g\x02\x02\xa7?=s}\x10\xf2\tO\xcf\xf8!\xa4E\x18G\x9d\x9d\tD\x9a\x7fz\xe2\xc2'\x11\x95\xc0\xc3V\x8aK\xff\x88\x8d1&I\tx|\xe3\xe4\xf7\xbd\x8c\xc9\x17\x90\xa5W9\x913K\x9eFY\x7f\xfe\xc7\xf3_\a\x8bN˔(\x1b\xf6i\xeb\xd4xL?b,\xdfM\x8d\xecg\xa9\xfez\xa6\xc2Ie?\xf5DMC\xe4\b\xbc\xbeX\x0f\xa8\xfck\xd27\x16\xcaϕ\xb7\x96'8a\x7f=\x02/\xe9\x8c=7;\x99m\xb4\x92\xaa6~M\ba\xbd\xcbܽ\x03\x01dL\xd8G5\xc8\x7f\xb0\x8d\xaa#\xa76&H\x9b\x90E\x9bF\x90^R\xadO\x8c\x00˷o\x97\xfd_\xac\xf2)\xb6\xecI\xd8M\x04\x18\xddG\xc1\xf3\x1c\xe3\x82\u0381\x1e\xaf\a\xc2UIC\xa1\x8c\x00S\x9aIQ8\x89\r\x10z\xf2\xca>Wnu\xf0h\xbfiz\r+=\x11wn\xfam\x93-9\xed\xbe?#\xe9\xf6\xa4G\xa3\xbeYZ\xedqɴ\xa9+\x94\t\x89\xb3\xe9\xe9\xb2)lu%=I69BNM\x88\x9d\xbb\x02\xf1\xa2ɯ/\x93\xf2\x9aL\xb3\xb4\xf4ֹ\x14{\x95T\xd6WN`}\xbd\xb4\xd5\x19ɪ\xa7?\xf5\x92\xbe\x96~tveڲ\xcc\xe1\x84Ӥ4Ӥ\xa5\x9b\x94\x01\x1f5Ԥ\xf4ѹI\xa3I\x9cL\x9f\xae\xaf\x9a\x16\xfa\xaaɠ\xaf\x9f\x02:)m\x93\x15\xe6&y\x8e_r\x18ʴ\x03P|\v\xe1|.\x99\x94\xee\xb9\xe6ϊ;?\x0f`\xa1\xb0\x047\xf5\x15〲.\xac\xa8\x8a\xf6>\xb6X\xc0\xb9\x81]sY\xd1ϊ\x8e\xc8\xfb\x9b\xba>\x7fi$~9\x88j\xb8aOP\x14\x8c\xc7\xe6\xe6\x1e\x152w\x0fh\xa6\x16\x80\xb6\x11g\xb9\xbf\x8c\xc9_\x1ez\xe1\xa6\v\xdd\x06@\x16\xb6\x8c-\xf5qy\xf8\xa6\xaf\x83\x06,U\x8f\xedy\xe6.ޠo\xbfԠw\x8c\xee\x1dk|\xb3\xf6P\xa9\x9f\xe8\x06\x03Ӡ~\xbc:<\xb4g\xb2\x17\xe0\xb4ꁽ\x93\xce#\x18\xe2DmP\xef\xb4\x01\x1d*U\x8cӢ\xfdD@H\xd5@\x884Mq\xfe眲|\x89\xf0\xee\x14\x01^\x92\a4\xcf{\xfd\x86\xa7'\x8f=5\x99\x9e\x8c\x92tJ\xf2%½9\x01\xdf,\x7f5\xfd\x14\xe4\xfc\x8d\xe7\x17>\xf5\xf8R\xa7\x1dgP/\xf5t\xe3|ڽ\xd2i\xc6W?\xc5\xf8\x9a\xa7\x17g\x9dZLNϚ\x95q0'\xb5\xea\x19\xc7\xed\xd2r\t\xa6O!&\x9e>L\xcc4H\x1b\xfc\x91\xc3N<]8\xffTa\"\x7f\xe7L\xe9W>=\xf8ʧ\x06\xbf\xc5i\xc1\x04\tL\xa82\xffT\u0cf7\xa4\x94\xceAOn\xfb͑\xdaIyM\x8d\xe5\xfa\x88\r\xf6\xb5\xc2m\xb2X\xab\x17\x03\x90Y\xf2\x17\xf9ӣ\r\x87\xb6\xc1Q2;\x1eQo_\xb2u\xd7\xfa\x0e\xb1\x7f\xcd\xc1m]\x1a\xa88\x1a\x00\n\xdc(5+\xea*|\xe0\xd9f\xd0Æ\x1b\xb6V\xba䖝7\x9b\xc5o\\\a\xf8\xf7\xf9\x92\xb1\x1fT\x93\xabӽ/͈\xb2*v\x18\x89\xb1\xf3n\x83\xe7IIT:C\xcf7\xaa\x10Y\xc4\xe7\x1c\xbdW\xcf5ػl\x88n\xfe\xcb:\xd9\"\xb1\xc0\a\x9b\x8bp\xebb\xffJfw\x9f\xfb\x91k%\xbc\x12\x7f\xa6'\x95N\xb0\xea\xf6\xee\xe6\x9a`\x051\xa2\xb7\x9a\x9a\x04ņ\xe5+@\x97\xa1\x1d\xfb!}r\xbd\xeeA\xed\xe7\bw\x1f\xab\x80ܽL\x12\xdc\x16\xaf\x9a3\x85Z\xeb\xe6\xda\xe1r\xa8'\x94/.wL\xf9\xa7'\x84\xce\x17\x15\xd7v璉.zx\x04\xbb>\xb5jv\xd0Z\xed\xbf\xbc\xd2-=\xb2\x87GWh'{W\xf5\x93\a\x86\xf4|\x0eN\x87OUO\x9e\xa7~\x01\x9c\x0e\xbbP\v\xa2b\xe4\xa7h\x06\xe4\xc9W,\x8d\xbf\xa1\xffG\xb5\x85\xf7ѕ\xcb\xfe\xeb+\x83&#\xa9\x89\x01*]2\x1f\xa1`\x9b\x8fHw|?O\xed\xc5s\r\x03*\xfe\x8e\xf0\xe7,N\xde\xf6A\x8d?HB7\xa8\x87Nc^\x15=\xf5\xb4c7\xf7\x14\xb76\xaa\xd4O}\x1f\xb7\x86\xe5ɐ`\x10\x81%\xe4\xc17ZNEF\xab4\x7f\x80\x8fʽ\xad\x93\"&\xfd\x16\xbd\x97\x97\xbc\xe7\x16\xf2\xb5\xfd$\x8c)z?\xb6!\xc0\xf6|\xc6\xdeE\xff\x88\xed\x91O\x19X[\xba\x91ғ&\xef\xfd\xeb$\xa8\x8f\r \v\x02\x05\x1c\xb4\x15\xfew\xa3\x9e\xe8\x02\xfc\xf8\x1asx@\xa4\xf3\x86\x19\xd0A\x11J\xe1=j\x98uU(\x9e\x83\xbe\xa2GT\x12F\xfcS\xaf\xc1\xc0\x1d\xe8?\xc5\xe2\xedfd<\xa1\xe7\x17̒A\x8f\xae(\xa0\xf8A\x14`\x1c≦\xe1f\xbfec)\xear\xe5<\xd55\xfe\xd8tr\xc02\xbb\xa1\xd2\x06C\x05\x1a\xfdD\xb7\x15Q\x9b \xf9\x87\x89\xc1\x1a>\ni\xe1\x01\xc6c\xe8\t\x9b\xe0\xdeh \a (0\x8a\xf8\xfe\x12[y\xec\x11\xe4>\xdez \x03\xcdbdL\x8e\x95w\xabn\xee\xaf\f\xabeN\x1b\x00\xf7\x7f\xbe=J~\xb7\xbd\xf7e\x82NHQ\xef\xf7\xe3-;!BG;\x91O\x1fW\xe21X\xdc\x18\x95\t\x8a*\x9e\x84\xf5\xd79\xbe\xdc\x1d\xe2\x87\x02\xc4\x03\xd2Q\x1b\xf8\xfc$A\x7f\t\x16\xc8\\\xcbػ-\xd3\xda\xef\xa7=h\xd1\xf7Z\xac¾G`\f\x000\x15\xf6\xb9\x8c{\t(l\xaf\t\xd3H\x1c\xc4>\x01\xdcyG\xc8ik\x85\xb4\xe3\x9c!n\x9bVt\xd8tDCN\x8b\xed\xfd\x00\xc6 \x93\x9d\x1e}j\xaa\xb8Ӧ\x86\xfd^\x8cy\xa3\xb4c\x96\xe1@\xff\xb0\xf7kT\x83\x1f\xd4\xde1\xcd=\xaaF\xf6>\xd2CxyGr\xbc\x97\xde\xfdR\xaf\xda\a\x15\xd8\xdf\xfe~\xf6\x8f\x00\x00\x00\xff\xff)\x00\x87w>{\x00\x00"), []byte("\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xbcV\xcfo+5\x10\xbe\xe7\xaf\x18\x89+\xbb\xa1B \x94\x1b*\x1c*\xe0\xa9j\x9ezw\xbc\x93d\xa8\xd7^f\xc6)A\xfc\xf1\xc8\xf6n\x9b\xee:\xb4\x8f\x03\xbe\xad\xed\xf9\xe6\x9bo~x\x9b\xa6Y\x99\x81\x1e\x91\x85\x82߀\x19\b\xffT\xf4\xe9Kڧ\x1f\xa4\xa5\xb0>ݬ\x9e\xc8w\x1b\xb8\x8d\xa2\xa1\x7f@\t\x91-\xfe\x84{\xf2\xa4\x14\xfc\xaaG5\x9dQ\xb3Y\x01\x18\uf0da\xb4-\xe9\x13\xc0\x06\xaf\x1c\x9cCn\x0e\xe8ۧ\xb8\xc3]$\xd7!g\xf0\xc9\xf5\xe9\x9b\xf6\xe6\xfb\xf6\xbb\x15\x807=n@\x90ә\x1a\x8d\xc2\xf8GDQiO\xe8\x90CKa%\x03ڄ\x7f\xe0\x10\x87\r\xbc\x1e\x14\xfb\xd1w\xe1\xbd\xcdP\xdb\f\xf5P\xa0\xf2\xa9#\xd1_\xae\xdd\xf8\x95\xc6[\x83\x8bl\\\x9dP\xbe \xc7\xc0\xfa\xe9\xd5i\x03\"\\N\xc8\x1f\xa23\\5^\x01\x88\r\x03n \xdb\x0e\xc6b\xb7\x02\x18\x05\xc9Xͨ\xc5\xe9\xa6\xc0\xd9#\xf6\xa68\x01\b\x03\xfa\x1f\xef\xef\x1e\xbfݾ\xd9\x06\xe8P,ӠYֿ\x9b\x97}\xa8\x85\t$``\xa4\x04\x1a\xc0X\x8b\"`#3z\x85B\x19\xc8\xef\x03\xf79\xad`v!\xea\x05\xaa\x1e\x11\x1e\xb3\xfec\x98\xed\xcb\xe1\xc0a@V\x9a\xa4)\xeb\xa2\xe2.v\xff\x8dxZ)\xd6b\x05]*=\x94\xecy\xd4\v\xbbQ\x1e\b{\xd0#\t0\x0e\x8c\x82\xbe\x14c\xda6\x1e\xc2\xeew\xb4\xdaΠ\x8b.\x922\x19]\x97*\xf6\x84\xac\xc0h\xc3\xc1\xd3_/ؒ\x04JN\x9dѬ\x9dWdo\x1c\x9c\x8c\x8b\xf85\x18\xdf͐{s\x06\xc6\xe4\x13\xa2\xbf\xc0\xcb\x062\xe7\xf1[`\xccRo\xe0\xa8:\xc8f\xbd>\x90N}hC\xdfGOz^疢]\xd4\xc0\xb2\xee\xf0\x84n-th\f\xdb#)Z\x8d\x8ck3P\x93\x03\xf1\xb9\x17۾\xfb\x8a\xc7Ε7n\xf5\x9cjP\x94\xc9\x1f.\x0er\xeb|AzR#\x95b*P%\xc4\xd7,\xa4\xad$\xdd\xc3\xcf\xdb\xcf01)\x99*Iy\xbd\xba\xd0e\xcaOR\x93\xfc\x1e\xb9\xd8\xed9\xf4\x19\x13}7\x04\xf2\x9a?\xac\xa3\\\xb8qד\xcaT\xda)us\xd8\xdb<\xab`\x87\x10\x87\xce(v\xf3\vw\x1enM\x8f\xee\xd6\b\xfeϹJY\x91&%\xe1Cٺ\x9c\xc0\xf3\xcbEދ\x83iv^ImeJl\a\xb4)\xb9I\xdfdM{\xb2\xa5\xad\xf6\x81\xc1\xd4L\xda\x0f1\xc9\x16_\xc8e\x9cH\x85\xcdlN\xa5.\x7f\x9fM},哣\x11\x9co\xce8ݧ;s\xff\x8e\xf6h\xcf\xd6a\x81(S\bߧ\x92\x16\xfa\xd8/}6\xf0\t\x9f+\xbb\xf7\x1c҄\xc6\xf9\xa8\xb9Z\x1bP\x1e\xb1\x03\xf9E\xb8\xf3\xc8ʭ\xfc0.G~\x0eh\x04\x02\x8eާ\x96\x0e~\x01Yy\x11\x16wH\xb1\xaf\xb0\xa9\xf2\xb9\xf3\xfb\x90\xff\"Lrl\xb4\xb4\x13\x8e\xc9\x1e\xfd\x14^\x15\xc0\xeb\xb9.k9\xe7>$hY\xf9y\xfeo\xc6i.\x11c\xd5w\x93YU\x0f\x92ǚ\xe2\xf5\xfe\x1aYF\xe7\xcc\xce\xe1\x06\x94\xe3Һ\xd8\x1afs\x9eW\xcdTj\x9f\xa9GQ\xd3\x0f\xef\x14\xd0\xe2UH\xeb~\x81\x92\x9a\xe7\xf9\x88\xfeZ\x8b\xc0\xb3\x91W\xe7\x15\xc8\xdd\xf9\x9a\xe9\xed\xcb\xdf\xe6\xb2\xcfJ=o \xcd\xfaF\xa9\"䇔\xaa\xa6\xb4\xd4y\xf5\xb7f\xa1\xd2\xf6\xf2\xee4H\xde\xf4\xcb\xf4W\xb3\x8c\xe1*\x85j\x05,63|w\x11\x9eh`s\x98\x02\xfe'\x00\x00\xff\xff\xef\xf8\xa6>\x10\f\x00\x00"), diff --git a/pkg/cmd/cli/nodeagent/server.go b/pkg/cmd/cli/nodeagent/server.go index 7bf632807..c7e987ddc 100644 --- a/pkg/cmd/cli/nodeagent/server.go +++ b/pkg/cmd/cli/nodeagent/server.go @@ -303,12 +303,11 @@ func (s *nodeAgentServer) run() { } pvbReconciler := controller.NewPodVolumeBackupReconciler(s.mgr.GetClient(), s.mgr, s.kubeClient, s.dataPathMgr, s.nodeName, s.config.dataMoverPrepareTimeout, s.config.resourceTimeout, podResources, s.metrics, s.logger) - if err := pvbReconciler.SetupWithManager(s.mgr); err != nil { s.logger.Fatal(err, "unable to create controller", "controller", constant.ControllerPodVolumeBackup) } - if err = controller.NewPodVolumeRestoreReconciler(s.mgr.GetClient(), s.mgr, s.kubeClient, s.dataPathMgr, s.nodeName, s.config.dataMoverPrepareTimeout, s.config.resourceTimeout, podResources, s.logger).SetupWithManager(s.mgr); err != nil { + if err := controller.NewPodVolumeRestoreReconciler(s.mgr.GetClient(), s.mgr, s.kubeClient, s.dataPathMgr, s.nodeName, s.config.dataMoverPrepareTimeout, s.config.resourceTimeout, podResources, s.logger).SetupWithManager(s.mgr); err != nil { s.logger.WithError(err).Fatal("Unable to create the pod volume restore controller") } @@ -327,7 +326,7 @@ func (s *nodeAgentServer) run() { s.logger, s.metrics, ) - if err = dataUploadReconciler.SetupWithManager(s.mgr); err != nil { + if err := dataUploadReconciler.SetupWithManager(s.mgr); err != nil { s.logger.WithError(err).Fatal("Unable to create the data upload controller") } @@ -338,7 +337,7 @@ func (s *nodeAgentServer) run() { } dataDownloadReconciler := controller.NewDataDownloadReconciler(s.mgr.GetClient(), s.mgr, s.kubeClient, s.dataPathMgr, restorePVCConfig, podResources, s.nodeName, s.config.dataMoverPrepareTimeout, s.logger, s.metrics) - if err = dataDownloadReconciler.SetupWithManager(s.mgr); err != nil { + if err := dataDownloadReconciler.SetupWithManager(s.mgr); err != nil { s.logger.WithError(err).Fatal("Unable to create the data download controller") } From daff6ab6859b60df4e3802ec324f47de894c84bc Mon Sep 17 00:00:00 2001 From: Xun Jiang Date: Thu, 22 May 2025 14:27:08 +0800 Subject: [PATCH 30/31] The modification of VGDP affinity design. Signed-off-by: Xun Jiang --- design/Implemented/node-agent-affinity.md | 2 +- .../repo_maintenance_job_config.md | 6 +- design/vgdp-affinity-enhancement.md | 299 ++++++++++++++++++ 3 files changed, 303 insertions(+), 4 deletions(-) create mode 100644 design/vgdp-affinity-enhancement.md diff --git a/design/Implemented/node-agent-affinity.md b/design/Implemented/node-agent-affinity.md index 140f6477f..4c62e358b 100644 --- a/design/Implemented/node-agent-affinity.md +++ b/design/Implemented/node-agent-affinity.md @@ -128,5 +128,5 @@ Once this problem happens, the backupPod stays in `Pending` phase, and the corre On the other hand, the backupPod is deleted after the prepare timeout, so there is no way to tell the cause is one of the above problems or others. To help the troubleshooting, we can add some diagnostic mechanism to discover the status of the backupPod and node-agent in the same node before deleting it as a result of the prepare timeout. -[1]: Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md +[1]: unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md [2]: volume-snapshot-data-movement/volume-snapshot-data-movement.md \ No newline at end of file diff --git a/design/Implemented/repo_maintenance_job_config.md b/design/Implemented/repo_maintenance_job_config.md index 7b4068a74..195b5b9dc 100644 --- a/design/Implemented/repo_maintenance_job_config.md +++ b/design/Implemented/repo_maintenance_job_config.md @@ -4,7 +4,7 @@ Add this design to make the repository maintenance job can read configuration from a dedicate ConfigMap and make the Job's necessary parts configurable, e.g. `PodSpec.Affinity` and `PodSpec.Resources`. ## Background -Repository maintenance is split from the Velero server to a k8s Job in v1.14 by design [repository maintenance job](Implemented/repository-maintenance.md). +Repository maintenance is split from the Velero server to a k8s Job in v1.14 by design [repository maintenance job](repository-maintenance.md). The repository maintenance Job configuration was read from the Velero server CLI parameter, and it inherits the most of Velero server's Deployment's PodSpec to fill un-configured fields. This design introduces a new way to let the user to customize the repository maintenance behavior instead of inheriting from the Velero server Deployment or reading from `velero server` CLI parameters. @@ -13,7 +13,7 @@ It's possible new configurations are introduced in future releases based on this For the node selection, the repository maintenance Job also inherits from the Velero server deployment before, but the Job may last for a while and cost noneligible resources, especially memory. The users have the need to choose which k8s node to run the maintenance Job. -This design reuses the data structure introduced by design [node-agent affinity configuration](Implemented/node-agent-affinity.md) to make the repository maintenance job can choose which node running on. +This design reuses the data structure introduced by design [Velero Generic Data Path affinity configuration](node-agent-affinity.md) to make the repository maintenance job can choose which node running on. ## Goals - Unify the repository maintenance Job configuration at one place. @@ -118,7 +118,7 @@ For example, the following BackupRepository's key should be `test-default-kopia` volumeNamespace: test ``` -The `LoadAffinity` structure is reused from design [node-agent affinity configuration](Implemented/node-agent-affinity.md). +The `LoadAffinity` structure is reused from design [Velero Generic Data Path affinity configuration](node-agent-affinity.md). It's possible that the users want to choose nodes that match condition A or condition B to run the job. For example, the user want to let the nodes is in a specified machine type or the nodes locate in the us-central1-x zones to run the job. This can be done by adding multiple entries in the `LoadAffinity` array. diff --git a/design/vgdp-affinity-enhancement.md b/design/vgdp-affinity-enhancement.md new file mode 100644 index 000000000..bbac325b0 --- /dev/null +++ b/design/vgdp-affinity-enhancement.md @@ -0,0 +1,299 @@ +# Velero Generic Data Path Load Affinity Enhancement Design + +## Glossary & Abbreviation + +**Velero Generic Data Path (VGDP)**: VGDP is the collective modules that is introduced in [Unified Repository design][1]. Velero uses these modules to finish data transfer for various purposes (i.e., PodVolume backup/restore, Volume Snapshot Data Movement). VGDP modules include uploaders and the backup repository. + +**Exposer**: Exposer is a module that is introduced in [Volume Snapshot Data Movement Design][1]. Velero uses this module to expose the volume snapshots to Velero node-agent pods or node-agent associated pods so as to complete the data movement from the snapshots. + +## Background + +The implemented [VGDP LoadAffinity design][3] already defined the a structure `LoadAffinity` in `--node-agent-configmap` parameter. The parameter is used to set the affinity of the backupPod of VGDP. + +There are still some limitations of this design: +* The affinity setting is global. Say there are two StorageClasses and the underlying storage can only provision volumes to part of the cluster nodes. The supported nodes don't have intersection. Then the affinity will definitely not work in some cases. +* The old design only take the first element of the []*LoadAffinity array. By this way, it cannot support the or logic between Affinity selectors. +* The old design focuses on the backupPod affinity, but the restorePod also needs the affinity setting. + +As a result, create this design to address the limitations. + +## Goals + +- Enhance the node affinity of VGDP instances for volume snapshot data movement: add per StorageClass node affinity. +- Enhance the node affinity of VGDP instances for volume snapshot data movement: support the or logic between affinity selectors. +- Define the behaviors of node affinity of VGDP instances in node-agent for volume snapshot data movement restore, when the PVC restore doesn't require delay binding. + +## Non-Goals + +- It is also beneficial to support VGDP instances affinity for PodVolume backup/restore, this will be implemented after the PodVolume micro service completes. + +## Solution + +This design still uses the ConfigMap specified by `velero node-agent` CLI's parameter `--node-agent-configmap` to host the node affinity configurations. + +Upon the implemented [VGDP LoadAffinity design][3] introduced `[]*LoadAffinity` structure, this design add a new field `StorageClass`. This field is optional. +* If the `LoadAffinity` element's `StorageClass` doesn't have value, it means this element is applied to global, just as the old design. +* If the `LoadAffinity` element's `StorageClass` has value, it means this element is applied to the VGDP instances' PVCs use the specified StorageClass. +* To support the or logic between LoadAffinity elements, this design allows multiple instances of `LoadAffinity` whose `StorageClass` field have the same value. +* The `LoadAffinity` element whose `StorageClass` has value has higher priority than the `LoadAffinity` element whose `StorageClass` doesn't have value. + + +```go +type Configs struct { + // LoadConcurrency is the config for load concurrency per node. + LoadConcurrency *LoadConcurrency `json:"loadConcurrency,omitempty"` + + // LoadAffinity is the config for data path load affinity. + LoadAffinity []*LoadAffinity `json:"loadAffinity,omitempty"` +} + +type LoadAffinity struct { + // NodeSelector specifies the label selector to match nodes + NodeSelector metav1.LabelSelector `json:"nodeSelector"` +} +``` + +``` go +type LoadAffinity struct { + // NodeSelector specifies the label selector to match nodes + NodeSelector metav1.LabelSelector `json:"nodeSelector"` + + // StorageClass specifies the VGDPs the LoadAffinity applied to. If the StorageClass doesn't have value, it applies to all. If not, it applies to only the VGDPs that use this StorageClass. + StorageClass string `json:"storageClass"` +} +``` + +### Decision Tree + +```mermaid +flowchart TD + A[VGDP Pod Needs Scheduling] --> B{Is this a restore operation?} + + B -->|Yes| C{StorageClass has volumeBindingMode: WaitForFirstConsumer?} + B -->|No| D[Backup Operation] + + C -->|Yes| E{restorePVC.ignoreDelayBinding = true?} + C -->|No| F[StorageClass binding mode: Immediate] + + E -->|No| G[Wait for target Pod scheduling
Use Pod's selected node
⚠️ Affinity rules ignored] + E -->|Yes| H[Apply affinity rules
despite WaitForFirstConsumer] + + F --> I{Check StorageClass in loadAffinity by StorageClass field} + H --> I + D --> J{Using backupPVC with different StorageClass?} + + J -->|Yes| K[Use final StorageClass
for affinity lookup] + J -->|No| L[Use original PVC StorageClass
for affinity lookup] + + K --> I + L --> I + + I -->|StorageClass found| N[Filter the LoadAffinity by
the StorageClass
🎯 and apply the LoadAffinity HIGHEST PRIORITY] + I -->|StorageClass not found| O{Check loadAffinity element without StorageClass field} + + O -->|No loadAffinity configured| R[No affinity constraints
Schedule on any available node
🌐 DEFAULT] + + N --> S{Multiple rules in array?} + S -->|Yes| T[Apply all rules as OR conditions
Pod scheduled on nodes matching ANY rule] + S -->|No| U[Apply single rule
Pod scheduled on nodes matching this rule] + + O --> S + + T --> V[Validate node-agent availability
⚠️ Ensure node-agent pods exist on target nodes] + U --> V + + V --> W{Node-agent available on selected nodes?} + W -->|Yes| X[✅ VGDP Pod scheduled successfully] + W -->|No| Y[❌ Pod stays in Pending state
Timeout after 30min
Check node-agent DaemonSet coverage] + + R --> Z[Schedule on any node
✅ Basic scheduling] + + %% Styling + classDef successNode fill:#d4edda,stroke:#155724,color:#155724 + classDef warningNode fill:#fff3cd,stroke:#856404,color:#856404 + classDef errorNode fill:#f8d7da,stroke:#721c24,color:#721c24 + classDef priorityHigh fill:#e7f3ff,stroke:#0066cc,color:#0066cc + classDef priorityMedium fill:#f0f8ff,stroke:#4d94ff,color:#4d94ff + classDef priorityDefault fill:#f8f9fa,stroke:#6c757d,color:#6c757d + + class X,Z successNode + class G,V,Y warningNode + class Y errorNode + class N,T,U priorityHigh + class P,Q priorityMedium + class R priorityDefault +``` + +### Examples + +#### Multiple LoadAffinities + +``` json +{ + "loadAffinity": [ + { + "nodeSelector": { + "matchLabels": { + "beta.kubernetes.io/instance-type": "Standard_B4ms" + } + } + }, + { + "nodeSelector": { + "matchExpressions": [ + { + "key": "topology.kubernetes.io/zone", + "operator": "In", + "values": [ + "us-central1-a" + ] + } + ] + } + } + ] +} +``` + +This sample demonstrates how to use multiple affinities in `loadAffinity`. That can support more complicated scenarios, e.g. need to filter nodes satisfied either of two conditions, instead of satisfied both of two conditions. + +In this example, the VGDP pods will be assigned to nodes, which instance type is `Standard_B4ms` or which zone is `us-central1-a`. + + +#### LoadAffinity interacts with LoadAffinityPerStorageClass + +``` json +{ + "loadAffinity": [ + { + "nodeSelector": { + "matchLabels": { + "beta.kubernetes.io/instance-type": "Standard_B4ms" + } + } + }, + { + "nodeSelector": { + "matchExpressions": [ + { + "key": "kubernetes.io/os", + "values": [ + "linux" + ], + "operator": "In" + } + ] + }, + "storageClass": "kibishii-storage-class" + }, + { + "nodeSelector": { + "matchLabels": { + "beta.kubernetes.io/instance-type": "Standard_B8ms" + } + }, + "storageClass": "kibishii-storage-class" + } + ] +} +``` + +This sample demonstrates how the `loadAffinity` elements with `StorageClass` field and without `StorageClass` field setting work together. +If the VGDP mounting volume is created from StorageClass `kibishii-storage-class`, its pod will run Linux nodes or instance type as `Standard_B8ms`. + +The other VGDP instances will run on nodes, which instance type is `Standard_B4ms`. + +#### LoadAffinity interacts with BackupPVC + +``` json +{ + "loadAffinity": [ + { + "nodeSelector": { + "matchLabels": { + "beta.kubernetes.io/instance-type": "Standard_B4ms" + } + }, + "storageClass": "kibishii-storage-class" + }, + { + "nodeSelector": { + "matchLabels": { + "beta.kubernetes.io/instance-type": "Standard_B2ms" + } + }, + "storageClass": "worker-storagepolicy" + } + ], + "backupPVC": { + "kibishii-storage-class": { + "storageClass": "worker-storagepolicy" + } + } +} +``` + +Velero data mover supports to use different StorageClass to create backupPVC by [design](https://github.com/vmware-tanzu/velero/pull/7982). + +In this example, if the backup target PVC's StorageClass is `kibishii-storage-class`, its backupPVC should use StorageClass `worker-storagepolicy`. Because the final StorageClass is `worker-storagepolicy`, the backupPod uses the loadAffinity specified by `loadAffinity`'s elements with `StorageClass` field set to `worker-storagepolicy`. backupPod will be assigned to nodes, which instance type is `Standard_B2ms`. + + +#### LoadAffinity interacts with RestorePVC + +``` json +{ + "loadAffinity": [ + { + "nodeSelector": { + "matchLabels": { + "beta.kubernetes.io/instance-type": "Standard_B4ms" + } + }, + "storageClass": "kibishii-storage-class" + } + ], + "restorePVC": { + "ignoreDelayBinding": false + } +} +``` + +##### StorageClass's bind mode is WaitForFirstConsumer + +``` yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: kibishii-storage-class +parameters: + svStorageClass: worker-storagepolicy +provisioner: csi.vsphere.vmware.com +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +``` + +If restorePVC should be created from StorageClass `kibishii-storage-class`, and it's volumeBindingMode is `WaitForFirstConsumer`. +Although `loadAffinityPerStorageClass` has a section matches the StorageClass, the `ignoreDelayBinding` is set `false`, the Velero exposer will wait until the target Pod scheduled to a node, and returns the node as SelectedNode for the restorePVC. +As a result, the `loadAffinityPerStorageClass` will not take affect. + +##### StorageClass's bind mode is Immediate + +``` yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: kibishii-storage-class +parameters: + svStorageClass: worker-storagepolicy +provisioner: csi.vsphere.vmware.com +reclaimPolicy: Delete +volumeBindingMode: Immediate +``` + +Because the StorageClass volumeBindingMode is `Immediate`, although `ignoreDelayBinding` is set to `false`, restorePVC will not be created according to the target Pod. + +The restorePod will be assigned to nodes, which instance type is `Standard_B4ms`. + +[1]: Implemented/unified-repo-and-kopia-integration/unified-repo-and-kopia-integration.md +[2]: Implemented/volume-snapshot-data-movement/volume-snapshot-data-movement.md +[3]: Implemented/node-agent-affinity.md \ No newline at end of file From d12b700b90b20e06c4aba8b935d11d45e01f55ae Mon Sep 17 00:00:00 2001 From: iiriix <1596129+iiriix@users.noreply.github.com> Date: Thu, 12 Jun 2025 09:35:11 -0700 Subject: [PATCH 31/31] chore: bump golang.org/x/oauth2 to fix restic CVE-2025-22868 Signed-off-by: Iiriix <1596129+iiriix@users.noreply.github.com> --- hack/fix_restic_cve.txt | 56 ++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/hack/fix_restic_cve.txt b/hack/fix_restic_cve.txt index b4e42bee2..c05992097 100644 --- a/hack/fix_restic_cve.txt +++ b/hack/fix_restic_cve.txt @@ -1,8 +1,8 @@ diff --git a/go.mod b/go.mod -index 5f939c481..5c5db077f 100644 +index 5f939c481..6ae17f4a1 100644 --- a/go.mod +++ b/go.mod -@@ -24,32 +24,32 @@ require ( +@@ -24,32 +24,31 @@ require ( github.com/restic/chunker v0.4.0 github.com/spf13/cobra v1.6.1 github.com/spf13/pflag v1.0.5 @@ -16,7 +16,7 @@ index 5f939c481..5c5db077f 100644 - google.golang.org/api v0.106.0 + golang.org/x/crypto v0.36.0 + golang.org/x/net v0.38.0 -+ golang.org/x/oauth2 v0.7.0 ++ golang.org/x/oauth2 v0.28.0 + golang.org/x/sync v0.12.0 + golang.org/x/sys v0.31.0 + golang.org/x/term v0.30.0 @@ -27,10 +27,10 @@ index 5f939c481..5c5db077f 100644 require ( - cloud.google.com/go v0.108.0 // indirect - cloud.google.com/go/compute v1.15.1 // indirect -+ cloud.google.com/go v0.110.0 // indirect -+ cloud.google.com/go/compute v1.19.1 // indirect - cloud.google.com/go/compute/metadata v0.2.3 // indirect +- cloud.google.com/go/compute/metadata v0.2.3 // indirect - cloud.google.com/go/iam v0.10.0 // indirect ++ cloud.google.com/go v0.110.0 // indirect ++ cloud.google.com/go/compute/metadata v0.3.0 // indirect + cloud.google.com/go/iam v0.13.0 // indirect github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 // indirect github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect @@ -49,7 +49,7 @@ index 5f939c481..5c5db077f 100644 github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/cpuid/v2 v2.2.3 // indirect -@@ -63,11 +63,13 @@ require ( +@@ -63,11 +62,13 @@ require ( go.opencensus.io v0.24.0 // indirect golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect google.golang.org/appengine v1.6.7 // indirect @@ -68,24 +68,24 @@ index 5f939c481..5c5db077f 100644 + +toolchain go1.23.7 diff --git a/go.sum b/go.sum -index 026e1d2fa..836a9b274 100644 +index 026e1d2fa..805792055 100644 --- a/go.sum +++ b/go.sum -@@ -1,23 +1,26 @@ +@@ -1,23 +1,24 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= -cloud.google.com/go v0.108.0 h1:xntQwnfn8oHGX0crLVinvHM+AhXvi3QHQIEcX/2hiWk= -cloud.google.com/go v0.108.0/go.mod h1:lNUfQqusBJp0bgAg6qrHgYFYbTB+dOiob1itwnlD33Q= -cloud.google.com/go/compute v1.15.1 h1:7UGq3QknM33pw5xATlpzeoomNxsacIVvTqTTvbfajmE= -cloud.google.com/go/compute v1.15.1/go.mod h1:bjjoF/NtFUrkD/urWfdHaKuOPDR5nWIs63rR+SXhcpA= -+cloud.google.com/go v0.110.0 h1:Zc8gqp3+a9/Eyph2KDmcGaPtbKRIoqq4YTlL4NMD0Ys= -+cloud.google.com/go v0.110.0/go.mod h1:SJnCLqQ0FCFGSZMUNUf84MV3Aia54kn7pi8st7tMzaY= -+cloud.google.com/go/compute v1.19.1 h1:am86mquDUgjGNWxiGn+5PGLbmgiWXlE/yNWpIpNvuXY= -+cloud.google.com/go/compute v1.19.1/go.mod h1:6ylj3a05WF8leseCdIf77NK0g1ey+nj5IKd5/kvShxE= - cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY= - cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA= +-cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY= +-cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA= -cloud.google.com/go/iam v0.10.0 h1:fpP/gByFs6US1ma53v7VxhvbJpO2Aapng6wabJ99MuI= -cloud.google.com/go/iam v0.10.0/go.mod h1:nXAECrMt2qHpF6RZUZseteD6QyanL68reN4OXPw0UWM= -cloud.google.com/go/longrunning v0.3.0 h1:NjljC+FYPV3uh5/OwWT6pVU+doBqMg2x/rZlE+CamDs= ++cloud.google.com/go v0.110.0 h1:Zc8gqp3+a9/Eyph2KDmcGaPtbKRIoqq4YTlL4NMD0Ys= ++cloud.google.com/go v0.110.0/go.mod h1:SJnCLqQ0FCFGSZMUNUf84MV3Aia54kn7pi8st7tMzaY= ++cloud.google.com/go/compute/metadata v0.3.0 h1:Tz+eQXMEqDIKRsmY3cHTL6FVaynIjX2QxYC4trgAKZc= ++cloud.google.com/go/compute/metadata v0.3.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k= +cloud.google.com/go/iam v0.13.0 h1:+CmB+K0J/33d0zSQ9SlFWUeCCEn5XJA0ZMZ3pHE9u8k= +cloud.google.com/go/iam v0.13.0/go.mod h1:ljOg+rcNfzZ5d6f1nAUJ8ZIxOaZUVoS14bKCtaLZ/D0= +cloud.google.com/go/longrunning v0.4.1 h1:v+yFJOfKC3yZdY6ZUI933pIYdhyhV8S3NpWrXWmg7jM= @@ -105,7 +105,7 @@ index 026e1d2fa..836a9b274 100644 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/Julusian/godocdown v0.0.0-20170816220326-6d19f8ff2df8/go.mod h1:INZr5t32rG59/5xeltqoCJoNY7e5x/3xoY9WSWVWg74= github.com/anacrolix/fuse v0.2.0 h1:pc+To78kI2d/WUjIyrsdqeJQAesuwpGxlI3h1nAv3Do= -@@ -54,6 +57,7 @@ github.com/felixge/fgprof v0.9.3/go.mod h1:RdbpDgzqYVh/T9fPELJyV7EYJuHB55UTEULNu +@@ -54,6 +55,7 @@ github.com/felixge/fgprof v0.9.3/go.mod h1:RdbpDgzqYVh/T9fPELJyV7EYJuHB55UTEULNu github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY= github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0= github.com/golang-jwt/jwt v3.2.1+incompatible h1:73Z+4BJcrTC+KczS6WvTPvRGOp1WmfEP4Q1lOd9Z/+c= @@ -113,7 +113,7 @@ index 026e1d2fa..836a9b274 100644 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE= -@@ -70,8 +74,8 @@ github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvq +@@ -70,8 +72,8 @@ github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvq github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= @@ -124,7 +124,7 @@ index 026e1d2fa..836a9b274 100644 github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= -@@ -82,17 +86,18 @@ github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/ +@@ -82,17 +84,18 @@ github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/ github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= @@ -148,7 +148,7 @@ index 026e1d2fa..836a9b274 100644 github.com/hashicorp/golang-lru/v2 v2.0.1 h1:5pv5N1lT1fjLg2VQ5KWc7kmucp2x/kvFOnxuVTqZ6x4= github.com/hashicorp/golang-lru/v2 v2.0.1/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= github.com/ianlancetaylor/demangle v0.0.0-20210905161508-09a460cdf81d/go.mod h1:aYm2/VgdVmcIU8iMfdMvDMsRAQjcfZSKFby6HOFvi/w= -@@ -114,6 +119,7 @@ github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg= +@@ -114,6 +117,7 @@ github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg= github.com/kurin/blazer v0.5.4-0.20211030221322-ba894c124ac6 h1:nz7i1au+nDzgExfqW5Zl6q85XNTvYoGnM5DHiQC0yYs= github.com/kurin/blazer v0.5.4-0.20211030221322-ba894c124ac6/go.mod h1:4FCXMUWo9DllR2Do4TtBd377ezyAJ51vB5uTBjt0pGU= github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= @@ -156,7 +156,7 @@ index 026e1d2fa..836a9b274 100644 github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= github.com/minio/minio-go/v7 v7.0.46 h1:Vo3tNmNXuj7ME5qrvN4iadO7b4mzu/RSFdUkUhaPldk= -@@ -129,6 +135,7 @@ github.com/modocache/gover v0.0.0-20171022184752-b58185e213c5/go.mod h1:caMODM3P +@@ -129,6 +133,7 @@ github.com/modocache/gover v0.0.0-20171022184752-b58185e213c5/go.mod h1:caMODM3P github.com/ncw/swift/v2 v2.0.1 h1:q1IN8hNViXEv8Zvg3Xdis4a3c4IlIGezkYz09zQL5J0= github.com/ncw/swift/v2 v2.0.1/go.mod h1:z0A9RVdYPjNjXVo2pDOPxZ4eu3oarO1P91fTItcb+Kg= github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 h1:Qj1ukM4GlMWXNdMBuXcXfz/Kw9s1qm0CLY32QxuSImI= @@ -164,7 +164,7 @@ index 026e1d2fa..836a9b274 100644 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/profile v1.7.0 h1:hnbDkaNWPCLMO9wGLdBFTIZvzDrDfBM2072E1S9gJkA= -@@ -172,8 +179,8 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk +@@ -172,8 +177,8 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= @@ -175,7 +175,7 @@ index 026e1d2fa..836a9b274 100644 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= -@@ -189,17 +196,17 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL +@@ -189,17 +194,17 @@ golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLL golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= @@ -186,8 +186,8 @@ index 026e1d2fa..836a9b274 100644 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= -golang.org/x/oauth2 v0.4.0 h1:NF0gk8LVPg1Ml7SSbGyySuoxdsXitj7TvgvuRxIMc/M= -golang.org/x/oauth2 v0.4.0/go.mod h1:RznEsdpjGAINPTOF0UH/t+xJ75L18YO3Ho6Pyn+uRec= -+golang.org/x/oauth2 v0.7.0 h1:qe6s0zUXlPX80/dITx3440hWZ7GwMwgDDyrSGTPJG/g= -+golang.org/x/oauth2 v0.7.0/go.mod h1:hPLQkd9LyjfXTiRohC/41GhcFqxisoUQ99sCUOHO9x4= ++golang.org/x/oauth2 v0.28.0 h1:CrgCKl8PPAVtLnU3c+EDw6x11699EWlsDeWNWKdIOkc= ++golang.org/x/oauth2 v0.28.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -199,7 +199,7 @@ index 026e1d2fa..836a9b274 100644 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -@@ -214,17 +221,17 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc +@@ -214,17 +219,17 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.0.0-20220408201424-a24fb2fb8a0f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220704084225-05e143d24a9e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -223,7 +223,7 @@ index 026e1d2fa..836a9b274 100644 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= -@@ -237,8 +244,8 @@ golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8T +@@ -237,8 +242,8 @@ golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk= golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= @@ -234,7 +234,7 @@ index 026e1d2fa..836a9b274 100644 google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c= -@@ -246,15 +253,15 @@ google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCID +@@ -246,15 +251,15 @@ google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCID google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= @@ -254,7 +254,7 @@ index 026e1d2fa..836a9b274 100644 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= -@@ -266,14 +273,15 @@ google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpAD +@@ -266,14 +271,15 @@ google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpAD google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=