diff --git a/design/CLI/PoC/base/CRDs.yaml b/design/CLI/PoC/base/CRDs.yaml new file mode 100644 index 000000000..5a62b8d96 --- /dev/null +++ b/design/CLI/PoC/base/CRDs.yaml @@ -0,0 +1,1182 @@ +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + labels: + component: velero + name: backups.velero.io +spec: + group: velero.io + names: + kind: Backup + listKind: BackupList + plural: backups + singular: backup + scope: "" + validation: + openAPIV3Schema: + description: Backup is a Velero resource that respresents the capture of Kubernetes + cluster state at a point in time (API objects and associated volume state). + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: BackupSpec defines the specification for a Velero backup. + properties: + excludedNamespaces: + description: ExcludedNamespaces contains a list of namespaces that + are not included in the backup. + items: + type: string + nullable: true + type: array + excludedResources: + description: ExcludedResources is a slice of resource names that are + not included in the backup. + items: + type: string + nullable: true + type: array + hooks: + description: Hooks represent custom behaviors that should be executed + at different phases of the backup. + properties: + resources: + description: Resources are hooks that should be executed when + backing up individual instances of a resource. + items: + description: BackupResourceHookSpec defines one or more BackupResourceHooks + that should be executed based on the rules defined for namespaces, + resources, and label selector. + properties: + excludedNamespaces: + description: ExcludedNamespaces specifies the namespaces + to which this hook spec does not apply. + items: + type: string + nullable: true + type: array + excludedResources: + description: ExcludedResources specifies the resources to + which this hook spec does not apply. + items: + type: string + nullable: true + type: array + includedNamespaces: + description: IncludedNamespaces specifies the namespaces + to which this hook spec applies. If empty, it applies + to all namespaces. + items: + type: string + nullable: true + type: array + includedResources: + description: IncludedResources specifies the resources to + which this hook spec applies. If empty, it applies to + all resources. + items: + type: string + nullable: true + type: array + labelSelector: + description: LabelSelector, if specified, filters the resources + to which this hook spec applies. + nullable: true + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector + that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are In, + NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. + If the operator is In or NotIn, the values array + must be non-empty. If the operator is Exists + or DoesNotExist, the values array must be empty. + This array is replaced during a strategic merge + patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. + A single {key,value} in the matchLabels map is equivalent + to an element of matchExpressions, whose key field + is "key", the operator is "In", and the values array + contains only "value". The requirements are ANDed. + type: object + type: object + name: + description: Name is the name of this hook. + type: string + post: + description: PostHooks is a list of BackupResourceHooks + to execute after storing the item in the backup. These + are executed after all "additional items" from item actions + are processed. + items: + description: BackupResourceHook defines a hook for a resource. + properties: + exec: + description: Exec defines an exec hook. + properties: + command: + description: Command is the command and arguments + to execute. + items: + type: string + minItems: 1 + type: array + container: + description: Container is the container in the + pod where the command should be executed. If + not specified, the pod's first container is + used. + type: string + onError: + description: OnError specifies how Velero should + behave if it encounters an error executing this + hook. + items: + enum: + - Continue + - Fail + type: string + timeout: + description: Timeout defines the maximum amount + of time Velero should wait for the hook to complete + before considering the execution a failure. + type: string + required: + - command + type: object + required: + - exec + type: object + type: array + pre: + description: PreHooks is a list of BackupResourceHooks to + execute prior to storing the item in the backup. These + are executed before any "additional items" from item actions + are processed. + items: + description: BackupResourceHook defines a hook for a resource. + properties: + exec: + description: Exec defines an exec hook. + properties: + command: + description: Command is the command and arguments + to execute. + items: + type: string + minItems: 1 + type: array + container: + description: Container is the container in the + pod where the command should be executed. If + not specified, the pod's first container is + used. + type: string + onError: + description: OnError specifies how Velero should + behave if it encounters an error executing this + hook. + items: + enum: + - Continue + - Fail + type: string + timeout: + description: Timeout defines the maximum amount + of time Velero should wait for the hook to complete + before considering the execution a failure. + type: string + required: + - command + type: object + required: + - exec + type: object + type: array + required: + - name + type: object + nullable: true + type: array + type: object + includeClusterResources: + description: IncludeClusterResources specifies whether cluster-scoped + resources should be included for consideration in the backup. + nullable: true + type: boolean + includedNamespaces: + description: IncludedNamespaces is a slice of namespace names to include + objects from. If empty, all namespaces are included. + items: + type: string + nullable: true + type: array + includedResources: + description: IncludedResources is a slice of resource names to include + in the backup. If empty, all resources are included. + items: + type: string + nullable: true + type: array + labelSelector: + description: LabelSelector is a metav1.LabelSelector to filter with + when adding individual objects to the backup. If empty or nil, all + objects are included. Optional. + nullable: true + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. + The requirements are ANDed. + items: + description: A label selector requirement is a selector that + contains values, a key, and an operator that relates the key + and values. + properties: + key: + description: key is the label key that the selector applies + to. + type: string + operator: + description: operator represents a key's relationship to + a set of values. Valid operators are In, NotIn, Exists + and DoesNotExist. + type: string + values: + description: values is an array of string values. If the + operator is In or NotIn, the values array must be non-empty. + If the operator is Exists or DoesNotExist, the values + array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single + {key,value} in the matchLabels map is equivalent to an element + of matchExpressions, whose key field is "key", the operator + is "In", and the values array contains only "value". The requirements + are ANDed. + type: object + type: object + snapshotVolumes: + description: SnapshotVolumes specifies whether to take cloud snapshots + of any PV's referenced in the set of objects included in the Backup. + nullable: true + type: boolean + storageLocation: + description: StorageLocation is a string containing the name of a + BackupStorageLocation where the backup should be stored. + type: string + ttl: + description: TTL is a time.Duration-parseable string describing how + long the Backup should be retained for. + type: string + volumeSnapshotLocations: + description: VolumeSnapshotLocations is a list containing names of + VolumeSnapshotLocations associated with this backup. + items: + type: string + type: array + type: object + status: + description: BackupStatus captures the current status of a Velero backup. + properties: + completionTimestamp: + description: CompletionTimestamp records the time a backup was completed. Completion time is recorded even on failed backups. Completion time is recorded before uploading the backup object. The server's time is used for CompletionTimestamps + format: date-time + nullable: true + type: string + errors: + description: Errors is a count of all error messages that were generated during execution of the backup. The actual errors are in the backup's log file in object storage. + type: integer + expiration: + description: Expiration is when this Backup is eligible for garbage-collection. + format: date-time + nullable: true + type: string + phase: + description: Phase is the current state of the Backup. + items: + enum: + - New + - FailedValidation + - InProgress + - Completed + - PartiallyFailed + - Failed + - Deleting + type: string + startTimestamp: + description: StartTimestamp records the time a backup was started. + Separate from CreationTimestamp, since that value changes on restores. + The server's time is used for StartTimestamps + format: date-time + nullable: true + type: string + validationErrors: + description: ValidationErrors is a slice of all validation errors + (if applicable). + items: + type: string + nullable: true + type: array + version: + description: Version is the backup format version. + type: integer + volumeSnapshotsAttempted: + description: VolumeSnapshotsAttempted is the total number of attempted + volume snapshots for this backup. + type: integer + volumeSnapshotsCompleted: + description: VolumeSnapshotsCompleted is the total number of successfully + completed volume snapshots for this backup. + type: integer + warnings: + description: Warnings is a count of all warning messages that were + generated during execution of the backup. The actual warnings are + in the backup's log file in object storage. + type: integer + type: object + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + labels: + component: velero + name: deletebackuprequests.velero.io +spec: + group: velero.io + names: + kind: DeleteBackupRequest + listKind: DeleteBackupRequestList + plural: deletebackuprequests + singular: deletebackuprequest + scope: "" + validation: + openAPIV3Schema: + description: DeleteBackupRequest is a request to delete one or more backups. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: DeleteBackupRequestSpec is the specification for which backups + to delete. + properties: + backupName: + type: string + required: + - backupName + type: object + status: + description: DeleteBackupRequestStatus is the current status of a DeleteBackupRequest. + properties: + errors: + description: Errors contains any errors that were encountered during + the deletion process. + items: + type: string + nullable: true + type: array + phase: + description: Phase is the current state of the DeleteBackupRequest. + items: + enum: + - New + - InProgress + - Processed + type: string + type: object + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + labels: + component: velero + name: downloadrequests.velero.io +spec: + group: velero.io + names: + kind: DownloadRequest + listKind: DownloadRequestList + plural: downloadrequests + singular: downloadrequest + scope: "" + validation: + openAPIV3Schema: + description: DownloadRequest is a request to download an artifact from backup + object storage, such as a backup log file. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: DownloadRequestSpec is the specification for a download request. + properties: + target: + description: Target is what to download (e.g. logs for a backup). + properties: + kind: + description: Kind is the type of file to download. + items: + enum: + - BackupLog + - BackupContents + - BackupVolumeSnapshot + - BackupResourceList + - RestoreLog + - RestoreResults + type: string + name: + description: Name is the name of the kubernetes resource with + which the file is associated. + type: string + required: + - kind + - name + type: object + required: + - target + type: object + status: + description: DownloadRequestStatus is the current status of a DownloadRequest. + properties: + downloadURL: + description: DownloadURL contains the pre-signed URL for the target + file. + type: string + expiration: + description: Expiration is when this DownloadRequest expires and can + be deleted by the system. + format: date-time + nullable: true + type: string + phase: + description: Phase is the current state of the DownloadRequest. + items: + enum: + - New + - Processed + type: string + type: object + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + labels: + component: velero + name: restores.velero.io +spec: + group: velero.io + names: + kind: Restore + listKind: RestoreList + plural: restores + singular: restore + scope: "" + validation: + openAPIV3Schema: + description: Restore is a Velero resource that represents the application + of resources from a Velero backup to a target Kubernetes cluster. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: RestoreSpec defines the specification for a Velero restore. + properties: + backupName: + description: BackupName is the unique name of the Velero backup to + restore from. + type: string + excludedNamespaces: + description: ExcludedNamespaces contains a list of namespaces that + are not included in the restore. + items: + type: string + nullable: true + type: array + excludedResources: + description: ExcludedResources is a slice of resource names that are + not included in the restore. + items: + type: string + nullable: true + type: array + includeClusterResources: + description: IncludeClusterResources specifies whether cluster-scoped + resources should be included for consideration in the restore. If + null, defaults to true. + nullable: true + type: boolean + includedNamespaces: + description: IncludedNamespaces is a slice of namespace names to include + objects from. If empty, all namespaces are included. + items: + type: string + nullable: true + type: array + includedResources: + description: IncludedResources is a slice of resource names to include + in the restore. If empty, all resources in the backup are included. + items: + type: string + nullable: true + type: array + labelSelector: + description: LabelSelector is a metav1.LabelSelector to filter with + when restoring individual objects from the backup. If empty or nil, + all objects are included. Optional. + nullable: true + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. + The requirements are ANDed. + items: + description: A label selector requirement is a selector that + contains values, a key, and an operator that relates the key + and values. + properties: + key: + description: key is the label key that the selector applies + to. + type: string + operator: + description: operator represents a key's relationship to + a set of values. Valid operators are In, NotIn, Exists + and DoesNotExist. + type: string + values: + description: values is an array of string values. If the + operator is In or NotIn, the values array must be non-empty. + If the operator is Exists or DoesNotExist, the values + array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single + {key,value} in the matchLabels map is equivalent to an element + of matchExpressions, whose key field is "key", the operator + is "In", and the values array contains only "value". The requirements + are ANDed. + type: object + type: object + namespaceMapping: + additionalProperties: + type: string + description: NamespaceMapping is a map of source namespace names to + target namespace names to restore into. Any source namespaces not + included in the map will be restored into namespaces of the same + name. + type: object + restorePVs: + description: RestorePVs specifies whether to restore all included + PVs from snapshot (via the cloudprovider). + nullable: true + type: boolean + scheduleName: + description: ScheduleName is the unique name of the Velero schedule + to restore from. If specified, and BackupName is empty, Velero will + restore from the most recent successful backup created from this + schedule. + type: string + required: + - backupName + type: object + status: + description: RestoreStatus captures the current status of a Velero restore + properties: + errors: + description: Errors is a count of all error messages that were generated + during execution of the restore. The actual errors are stored in + object storage. + type: integer + failureReason: + description: FailureReason is an error that caused the entire restore + to fail. + type: string + phase: + description: Phase is the current state of the Restore + items: + enum: + - New + - FailedValidation + - InProgress + - Completed + - PartiallyFailed + - Failed + type: string + validationErrors: + description: ValidationErrors is a slice of all validation errors + (if applicable) + items: + type: string + nullable: true + type: array + warnings: + description: Warnings is a count of all warning messages that were + generated during execution of the restore. The actual warnings are + stored in object storage. + type: integer + type: object + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + labels: + component: velero + name: schedules.velero.io +spec: + group: velero.io + names: + kind: Schedule + listKind: ScheduleList + plural: schedules + singular: schedule + scope: "" + validation: + openAPIV3Schema: + description: Schedule is a Velero resource that represents a pre-scheduled + or periodic Backup that should be run. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: ScheduleSpec defines the specification for a Velero schedule + properties: + schedule: + description: Schedule is a Cron expression defining when to run the + Backup. + type: string + template: + description: Template is the definition of the Backup to be run on + the provided schedule + properties: + excludedNamespaces: + description: ExcludedNamespaces contains a list of namespaces + that are not included in the backup. + items: + type: string + nullable: true + type: array + excludedResources: + description: ExcludedResources is a slice of resource names that + are not included in the backup. + items: + type: string + nullable: true + type: array + hooks: + description: Hooks represent custom behaviors that should be executed + at different phases of the backup. + properties: + resources: + description: Resources are hooks that should be executed when + backing up individual instances of a resource. + items: + description: BackupResourceHookSpec defines one or more + BackupResourceHooks that should be executed based on the + rules defined for namespaces, resources, and label selector. + properties: + excludedNamespaces: + description: ExcludedNamespaces specifies the namespaces + to which this hook spec does not apply. + items: + type: string + nullable: true + type: array + excludedResources: + description: ExcludedResources specifies the resources + to which this hook spec does not apply. + items: + type: string + nullable: true + type: array + includedNamespaces: + description: IncludedNamespaces specifies the namespaces + to which this hook spec applies. If empty, it applies + to all namespaces. + items: + type: string + nullable: true + type: array + includedResources: + description: IncludedResources specifies the resources + to which this hook spec applies. If empty, it applies + to all resources. + items: + type: string + nullable: true + type: array + labelSelector: + description: LabelSelector, if specified, filters the + resources to which this hook spec applies. + nullable: true + properties: + matchExpressions: + description: matchExpressions is a list of label + selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a + selector that contains values, a key, and an + operator that relates the key and values. + properties: + key: + description: key is the label key that the + selector applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are + In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string + values. If the operator is In or NotIn, + the values array must be non-empty. If the + operator is Exists or DoesNotExist, the + values array must be empty. This array is + replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} + pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, + whose key field is "key", the operator is "In", + and the values array contains only "value". The + requirements are ANDed. + type: object + type: object + name: + description: Name is the name of this hook. + type: string + post: + description: PostHooks is a list of BackupResourceHooks + to execute after storing the item in the backup. These + are executed after all "additional items" from item + actions are processed. + items: + description: BackupResourceHook defines a hook for + a resource. + properties: + exec: + description: Exec defines an exec hook. + properties: + command: + description: Command is the command and arguments + to execute. + items: + type: string + minItems: 1 + type: array + container: + description: Container is the container in + the pod where the command should be executed. + If not specified, the pod's first container + is used. + type: string + onError: + description: OnError specifies how Velero + should behave if it encounters an error + executing this hook. + items: + enum: + - Continue + - Fail + type: string + timeout: + description: Timeout defines the maximum amount + of time Velero should wait for the hook + to complete before considering the execution + a failure. + type: string + required: + - command + type: object + required: + - exec + type: object + type: array + pre: + description: PreHooks is a list of BackupResourceHooks + to execute prior to storing the item in the backup. + These are executed before any "additional items" from + item actions are processed. + items: + description: BackupResourceHook defines a hook for + a resource. + properties: + exec: + description: Exec defines an exec hook. + properties: + command: + description: Command is the command and arguments + to execute. + items: + type: string + minItems: 1 + type: array + container: + description: Container is the container in + the pod where the command should be executed. + If not specified, the pod's first container + is used. + type: string + onError: + description: OnError specifies how Velero + should behave if it encounters an error + executing this hook. + items: + enum: + - Continue + - Fail + type: string + timeout: + description: Timeout defines the maximum amount + of time Velero should wait for the hook + to complete before considering the execution + a failure. + type: string + required: + - command + type: object + required: + - exec + type: object + type: array + required: + - name + type: object + nullable: true + type: array + type: object + includeClusterResources: + description: IncludeClusterResources specifies whether cluster-scoped + resources should be included for consideration in the backup. + nullable: true + type: boolean + includedNamespaces: + description: IncludedNamespaces is a slice of namespace names + to include objects from. If empty, all namespaces are included. + items: + type: string + nullable: true + type: array + includedResources: + description: IncludedResources is a slice of resource names to + include in the backup. If empty, all resources are included. + items: + type: string + nullable: true + type: array + labelSelector: + description: LabelSelector is a metav1.LabelSelector to filter + with when adding individual objects to the backup. If empty + or nil, all objects are included. Optional. + nullable: true + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector + that contains values, a key, and an operator that relates + the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: operator represents a key's relationship + to a set of values. Valid operators are In, NotIn, + Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If + the operator is In or NotIn, the values array must + be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced + during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A + single {key,value} in the matchLabels map is equivalent + to an element of matchExpressions, whose key field is "key", + the operator is "In", and the values array contains only + "value". The requirements are ANDed. + type: object + type: object + snapshotVolumes: + description: SnapshotVolumes specifies whether to take cloud snapshots + of any PV's referenced in the set of objects included in the + Backup. + nullable: true + type: boolean + storageLocation: + description: StorageLocation is a string containing the name of + a BackupStorageLocation where the backup should be stored. + type: string + ttl: + description: TTL is a time.Duration-parseable string describing + how long the Backup should be retained for. + type: string + volumeSnapshotLocations: + description: VolumeSnapshotLocations is a list containing names + of VolumeSnapshotLocations associated with this backup. + items: + type: string + type: array + type: object + required: + - schedule + - template + type: object + status: + description: ScheduleStatus captures the current state of a Velero schedule + properties: + lastBackup: + description: LastBackup is the last time a Backup was run for this Schedule schedule + format: date-time + nullable: true + type: string + phase: + description: Phase is the current phase of the Schedule + items: + enum: + - New + - Enabled + - FailedValidation + type: string + validationErrors: + description: ValidationErrors is a slice of all validation errors (if applicable) + items: + type: string + type: array + type: object + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + labels: + component: velero + name: serverstatusrequests.velero.io +spec: + group: velero.io + names: + kind: ServerStatusRequest + listKind: ServerStatusRequestList + plural: serverstatusrequests + singular: serverstatusrequest + scope: "" + validation: + openAPIV3Schema: + description: ServerStatusRequest is a request to access current status information + about the Velero server. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: ServerStatusRequestSpec is the specification for a ServerStatusRequest. + type: object + status: + description: ServerStatusRequestStatus is the current status of a ServerStatusRequest. + properties: + phase: + description: Phase is the current lifecycle phase of the ServerStatusRequest. + items: + enum: + - New + - Processed + type: string + plugins: + description: Plugins list information about the plugins running on + the Velero server + items: + description: PluginInfo contains attributes of a Velero plugin + properties: + kind: + type: string + name: + type: string + required: + - kind + - name + type: object + nullable: true + type: array + processedTimestamp: + description: ProcessedTimestamp is when the ServerStatusRequest was + processed by the ServerStatusRequestController. + format: date-time + nullable: true + type: string + serverVersion: + description: ServerVersion is the Velero server version. + type: string + type: object + type: object + version: v1beta1 + versions: + - name: v1beta1 + served: true + storage: true \ No newline at end of file diff --git a/design/CLI/PoC/base/backupstoragelocations.yaml b/design/CLI/PoC/base/backupstoragelocations.yaml new file mode 100644 index 000000000..fbeed58cb --- /dev/null +++ b/design/CLI/PoC/base/backupstoragelocations.yaml @@ -0,0 +1,131 @@ +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + labels: + component: velero + name: backupstoragelocations.velero.io +spec: + group: velero.io + names: + kind: BackupStorageLocation + listKind: BackupStorageLocationList + plural: backupstoragelocations + singular: backupstoragelocation + scope: "" + validation: + openAPIV3Schema: + description: BackupStorageLocation is a location where Velero stores backup + objects. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: BackupStorageLocationSpec defines the specification for a + Velero BackupStorageLocation. + properties: + accessMode: + description: AccessMode defines the permissions for the backup storage + location. + enum: + - ReadOnly + - ReadWrite + type: string + backupSyncPeriod: + description: BackupSyncPeriod defines how frequently to sync backup + API objects from object storage. A value of 0 disables sync. + nullable: true + type: string + config: + additionalProperties: + type: string + description: Config is for provider-specific configuration fields. + type: object + objectStorage: + description: ObjectStorageLocation specifies the settings necessary + to connect to a provider's object storage. + properties: + bucket: + description: Bucket is the bucket to use for object storage. + type: string + prefix: + description: Prefix is the path inside a bucket to use for Velero + storage. Optional. + type: string + required: + - bucket + type: object + provider: + description: Provider is the provider of the backup storage. + type: string + required: + - objectStorage + - provider + type: object + status: + description: BackupStorageLocationStatus describes the current status + of a Velero BackupStorageLocation. + properties: + accessMode: + description: "AccessMode is an unused field. \n Deprecated: there + is now an AccessMode field on the Spec and this field will be removed + entirely as of v2.0." + enum: + - ReadOnly + - ReadWrite + type: string + lastSyncedRevision: + description: "LastSyncedRevision is the value of the `metadata/revision` + file in the backup storage location the last time the BSL's contents + were synced into the cluster. \n Deprecated: this field is no longer + updated or used for detecting changes to the location's contents + and will be removed entirely in v2.0." + type: string + lastSyncedTime: + description: LastSyncedTime is the last time the contents of the location + were synced into the cluster. + format: date-time + nullable: true + type: string + phase: + description: Phase is the current state of the BackupStorageLocation. + enum: + - Available + - Unavailable + type: string + type: object + type: object + version: v1 + versions: + - name: v1 + served: true + storage: true +--- +apiVersion: velero.io/v1 +kind: BackupStorageLocation +metadata: + creationTimestamp: null + labels: + component: velero + name: default + namespace: velero +spec: + config: + region: minio + s3ForcePathStyle: "true" + s3Url: http://minio.velero.svc:9000 + objectStorage: + bucket: velero + provider: aws diff --git a/design/CLI/PoC/base/deployment.yaml b/design/CLI/PoC/base/deployment.yaml new file mode 100644 index 000000000..36cf4d462 --- /dev/null +++ b/design/CLI/PoC/base/deployment.yaml @@ -0,0 +1,89 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + component: velero + name: velero + namespace: velero +spec: + selector: + matchLabels: + deploy: velero + strategy: {} + template: + metadata: + annotations: + prometheus.io/path: /metrics + prometheus.io/port: "8085" + prometheus.io/scrape: "true" + labels: + component: velero + deploy: velero + spec: + containers: + - args: + - server + command: + - /velero + env: + - name: VELERO_SCRATCH_DIR + value: /scratch + - name: VELERO_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: LD_LIBRARY_PATH + value: /plugins + name: velero + image: velero/velero:latest + imagePullPolicy: Always + ports: + - containerPort: 8085 + name: metrics + resources: + limits: + cpu: "1" + memory: 256Mi + requests: + cpu: 500m + memory: 128Mi + volumeMounts: + - mountPath: /scratch + name: scratch + restartPolicy: Always + serviceAccountName: velero + volumes: + - emptyDir: {} + name: scratch +--- +apiVersion: rbac.authorization.k8s.io/v1beta1 +kind: ClusterRoleBinding +metadata: + labels: + component: velero + name: velero +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin +subjects: + - kind: ServiceAccount + name: velero + namespace: velero +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + labels: + component: velero + name: velero + namespace: velero +--- +apiVersion: v1 +kind: Namespace +metadata: + labels: + component: velero + name: velero +spec: {} \ No newline at end of file diff --git a/design/CLI/PoC/base/kustomization.yaml b/design/CLI/PoC/base/kustomization.yaml new file mode 100644 index 000000000..989aa0e1d --- /dev/null +++ b/design/CLI/PoC/base/kustomization.yaml @@ -0,0 +1,12 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - deployment.yaml + - CRDs.yaml + - backupstoragelocations.yaml + - volumesnapshotlocations.yaml # including so the velero server can run + - resticrepository.yaml # including so the velero server can runl + - podvolumes.yaml # including so the velero server can runl + - minio.yaml + diff --git a/design/CLI/PoC/base/minio.yaml b/design/CLI/PoC/base/minio.yaml new file mode 100644 index 000000000..24cb46402 --- /dev/null +++ b/design/CLI/PoC/base/minio.yaml @@ -0,0 +1,107 @@ +# Copyright 2017 the Velero contributors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + namespace: velero + name: minio + labels: + component: minio +spec: + strategy: + type: Recreate + selector: + matchLabels: + component: minio + template: + metadata: + labels: + component: minio + spec: + volumes: + - name: storage + emptyDir: {} + - name: config + emptyDir: {} + containers: + - name: minio + image: minio/minio:latest + imagePullPolicy: IfNotPresent + args: + - server + - /storage + - --config-dir=/config + env: + - name: MINIO_ACCESS_KEY + value: "minio" + - name: MINIO_SECRET_KEY + value: "minio123" + ports: + - containerPort: 9000 + volumeMounts: + - name: storage + mountPath: "/storage" + - name: config + mountPath: "/config" + +--- +apiVersion: v1 +kind: Service +metadata: + namespace: velero + name: minio + labels: + component: minio +spec: + # ClusterIP is recommended for production environments. + # Change to NodePort if needed per documentation, + # but only if you run Minio in a test/trial environment, for example with Minikube. + type: ClusterIP + ports: + - port: 9000 + targetPort: 9000 + protocol: TCP + selector: + component: minio + +--- +apiVersion: batch/v1 +kind: Job +metadata: + namespace: velero + name: minio-setup + labels: + component: minio +spec: + template: + metadata: + name: minio-setup + spec: + restartPolicy: OnFailure + volumes: + - name: config + emptyDir: {} + containers: + - name: mc + image: minio/mc:latest + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero" + volumeMounts: + - name: config + mountPath: "/config" diff --git a/design/CLI/PoC/base/podvolumes.yaml b/design/CLI/PoC/base/podvolumes.yaml new file mode 100644 index 000000000..0a909fa41 --- /dev/null +++ b/design/CLI/PoC/base/podvolumes.yaml @@ -0,0 +1,297 @@ +--- + apiVersion: apiextensions.k8s.io/v1beta1 + kind: CustomResourceDefinition + metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + creationTimestamp: null + labels: + component: velero + name: podvolumebackups.velero.io + spec: + group: velero.io + names: + kind: PodVolumeBackup + listKind: PodVolumeBackupList + plural: podvolumebackups + singular: podvolumebackup + scope: "" + validation: + openAPIV3Schema: + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: PodVolumeBackupSpec is the specification for a PodVolumeBackup. + properties: + backupStorageLocation: + description: BackupStorageLocation is the name of the backup storage + location where the restic repository is stored. + type: string + node: + description: Node is the name of the node that the Pod is running + on. + type: string + pod: + description: Pod is a reference to the pod containing the volume to + be backed up. + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: 'If referring to a piece of an object instead of + an entire object, this string should contain a valid JSON/Go + field access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within + a pod, this would take on a value like: "spec.containers{name}" + (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" + (container with index 2 in this pod). This syntax is chosen + only to have some well-defined way of referencing a part of + an object. TODO: this design is not final and this field is + subject to change in the future.' + type: string + kind: + description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + namespace: + description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/' + type: string + resourceVersion: + description: 'Specific resourceVersion to which this reference + is made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency' + type: string + uid: + description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids' + type: string + type: object + repoIdentifier: + description: RepoIdentifier is the restic repository identifier. + type: string + tags: + additionalProperties: + type: string + description: Tags are a map of key-value pairs that should be applied + to the volume backup as tags. + type: object + volume: + description: Volume is the name of the volume within the Pod to be + backed up. + type: string + required: + - backupStorageLocation + - node + - pod + - repoIdentifier + - volume + type: object + status: + description: PodVolumeBackupStatus is the current status of a PodVolumeBackup. + properties: + completionTimestamp: + description: CompletionTimestamp records the time a backup was completed. + Completion time is recorded even on failed backups. Completion time + is recorded before uploading the backup object. The server's time + is used for CompletionTimestamps + format: date-time + nullable: true + type: string + message: + description: Message is a message about the pod volume backup's status. + type: string + path: + description: Path is the full path within the controller pod being + backed up. + type: string + phase: + description: Phase is the current state of the PodVolumeBackup. + enum: + - New + - InProgress + - Completed + - Failed + type: string + progress: + description: Progress holds the total number of bytes of the volume + and the current number of backed up bytes. This can be used to display + progress information about the backup operation. + properties: + bytesDone: + format: int64 + type: integer + totalBytes: + format: int64 + type: integer + type: object + snapshotID: + description: SnapshotID is the identifier for the snapshot of the + pod volume. + type: string + startTimestamp: + description: StartTimestamp records the time a backup was started. + Separate from CreationTimestamp, since that value changes on restores. + The server's time is used for StartTimestamps + format: date-time + nullable: true + type: string + type: object + type: object + version: v1 + versions: + - name: v1 + served: true + storage: true +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + creationTimestamp: null + labels: + component: velero + name: podvolumerestores.velero.io +spec: + group: velero.io + names: + kind: PodVolumeRestore + listKind: PodVolumeRestoreList + plural: podvolumerestores + singular: podvolumerestore + scope: "" + validation: + openAPIV3Schema: + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: PodVolumeRestoreSpec is the specification for a PodVolumeRestore. + properties: + backupStorageLocation: + description: BackupStorageLocation is the name of the backup storage + location where the restic repository is stored. + type: string + pod: + description: Pod is a reference to the pod containing the volume to + be restored. + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: 'If referring to a piece of an object instead of + an entire object, this string should contain a valid JSON/Go + field access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within + a pod, this would take on a value like: "spec.containers{name}" + (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" + (container with index 2 in this pod). This syntax is chosen + only to have some well-defined way of referencing a part of + an object. TODO: this design is not final and this field is + subject to change in the future.' + type: string + kind: + description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + namespace: + description: 'Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/' + type: string + resourceVersion: + description: 'Specific resourceVersion to which this reference + is made, if any. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency' + type: string + uid: + description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids' + type: string + type: object + repoIdentifier: + description: RepoIdentifier is the restic repository identifier. + type: string + snapshotID: + description: SnapshotID is the ID of the volume snapshot to be restored. + type: string + volume: + description: Volume is the name of the volume within the Pod to be + restored. + type: string + required: + - backupStorageLocation + - pod + - repoIdentifier + - snapshotID + - volume + type: object + status: + description: PodVolumeRestoreStatus is the current status of a PodVolumeRestore. + properties: + completionTimestamp: + description: CompletionTimestamp records the time a restore was completed. + Completion time is recorded even on failed restores. The server's + time is used for CompletionTimestamps + format: date-time + nullable: true + type: string + message: + description: Message is a message about the pod volume restore's status. + type: string + phase: + description: Phase is the current state of the PodVolumeRestore. + enum: + - New + - InProgress + - Completed + - Failed + type: string + progress: + description: Progress holds the total number of bytes of the snapshot + and the current number of restored bytes. This can be used to display + progress information about the restore operation. + properties: + bytesDone: + format: int64 + type: integer + totalBytes: + format: int64 + type: integer + type: object + startTimestamp: + description: StartTimestamp records the time a restore was started. + The server's time is used for StartTimestamps + format: date-time + nullable: true + type: string + type: object + type: object + version: v1 + versions: + - name: v1 + served: true + storage: true \ No newline at end of file diff --git a/design/CLI/PoC/base/resticrepository.yaml b/design/CLI/PoC/base/resticrepository.yaml new file mode 100644 index 000000000..674c0d5c5 --- /dev/null +++ b/design/CLI/PoC/base/resticrepository.yaml @@ -0,0 +1,85 @@ +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + creationTimestamp: null + labels: + component: velero + name: resticrepositories.velero.io +spec: + group: velero.io + names: + kind: ResticRepository + listKind: ResticRepositoryList + plural: resticrepositories + singular: resticrepository + scope: "" + validation: + openAPIV3Schema: + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: ResticRepositorySpec is the specification for a ResticRepository. + properties: + backupStorageLocation: + description: BackupStorageLocation is the name of the BackupStorageLocation + that should contain this repository. + type: string + maintenanceFrequency: + description: MaintenanceFrequency is how often maintenance should + be run. + type: string + resticIdentifier: + description: ResticIdentifier is the full restic-compatible string + for identifying this repository. + type: string + volumeNamespace: + description: VolumeNamespace is the namespace this restic repository + contains pod volume backups for. + type: string + required: + - backupStorageLocation + - maintenanceFrequency + - resticIdentifier + - volumeNamespace + type: object + status: + description: ResticRepositoryStatus is the current status of a ResticRepository. + properties: + lastMaintenanceTime: + description: LastMaintenanceTime is the last time maintenance was + run. + format: date-time + nullable: true + type: string + message: + description: Message is a message about the current status of the + ResticRepository. + type: string + phase: + description: Phase is the current state of the ResticRepository. + enum: + - New + - Ready + - NotReady + type: string + type: object + type: object + version: v1 + versions: + - name: v1 + served: true + storage: true \ No newline at end of file diff --git a/design/CLI/PoC/base/volumesnapshotlocations.yaml b/design/CLI/PoC/base/volumesnapshotlocations.yaml new file mode 100644 index 000000000..c1db58501 --- /dev/null +++ b/design/CLI/PoC/base/volumesnapshotlocations.yaml @@ -0,0 +1,80 @@ +--- +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: (unknown) + labels: + component: velero + name: volumesnapshotlocations.velero.io +spec: + group: velero.io + names: + kind: VolumeSnapshotLocation + listKind: VolumeSnapshotLocationList + plural: volumesnapshotlocations + singular: volumesnapshotlocation + scope: "" + validation: + openAPIV3Schema: + description: VolumeSnapshotLocation is a location where Velero stores volume + snapshots. + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: VolumeSnapshotLocationSpec defines the specification for + a Velero VolumeSnapshotLocation. + properties: + config: + additionalProperties: + type: string + description: Config is for provider-specific configuration fields. + type: object + provider: + description: Provider is the provider of the volume storage. + type: string + required: + - provider + type: object + status: + description: VolumeSnapshotLocationStatus describes the current status + of a Velero VolumeSnapshotLocation. + properties: + phase: + description: VolumeSnapshotLocationPhase is the lifecyle phase of + a Velero VolumeSnapshotLocation. + enum: + - Available + - Unavailable + type: string + type: object + type: object + version: v1 + versions: + - name: v1 + served: true + storage: true +--- +apiVersion: velero.io/v1 +kind: VolumeSnapshotLocation +metadata: + creationTimestamp: null + labels: + component: velero + name: default + namespace: velero +spec: + config: + region: us-east-2 + provider: aws \ No newline at end of file diff --git a/design/CLI/PoC/overlays/plugins/aws-plugin.yaml b/design/CLI/PoC/overlays/plugins/aws-plugin.yaml new file mode 100644 index 000000000..d2374b7ca --- /dev/null +++ b/design/CLI/PoC/overlays/plugins/aws-plugin.yaml @@ -0,0 +1,40 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: velero +spec: + selector: + matchLabels: + deploy: velero + template: + metadata: + labels: + component: velero + deploy: velero + spec: + containers: + - args: + - server + name: velero + env: + - name: AWS_SHARED_CREDENTIALS_FILE + value: /credentials/cloud + volumeMounts: + - mountPath: /plugins + name: plugins + - mountPath: /credentials + name: cloud-credential-aws + initContainers: + - image: velero/velero-plugin-for-aws:v1.0.1 + imagePullPolicy: Always + name: velero-plugin-for-aws + volumeMounts: + - mountPath: /target + name: plugins + volumes: + - emptyDir: {} + name: plugins + - name: cloud-credential-aws + secret: + secretName: cloud-credential-aws diff --git a/design/CLI/PoC/overlays/plugins/azure-plugin.yaml b/design/CLI/PoC/overlays/plugins/azure-plugin.yaml new file mode 100644 index 000000000..d9442c783 --- /dev/null +++ b/design/CLI/PoC/overlays/plugins/azure-plugin.yaml @@ -0,0 +1,40 @@ +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: velero +spec: + selector: + matchLabels: + deploy: velero + template: + metadata: + labels: + component: velero + deploy: velero + spec: + containers: + - args: + - server + name: velero + env: + - name: AZURE_SHARED_CREDENTIALS_FILE + value: /credentials/cloud + volumeMounts: + - mountPath: /plugins + name: plugins + - mountPath: /credentials + name: cloud-credential-azure + initContainers: + - image: velero/velero-plugin-for-microsoft-azure:v1.0.1 + imagePullPolicy: Always + name: velero-plugin-for-microsoft-azure + volumeMounts: + - mountPath: /target + name: plugins + volumes: + - emptyDir: {} + name: plugins + - name: cloud-credential-azure + secret: + secretName: cloud-credential-azure diff --git a/design/CLI/PoC/overlays/plugins/cloud b/design/CLI/PoC/overlays/plugins/cloud new file mode 100644 index 000000000..7546f354f --- /dev/null +++ b/design/CLI/PoC/overlays/plugins/cloud @@ -0,0 +1,3 @@ +[default] +aws_access_key_id = minio +aws_secret_access_key = minio123 \ No newline at end of file diff --git a/design/CLI/PoC/overlays/plugins/kustomization.yaml b/design/CLI/PoC/overlays/plugins/kustomization.yaml new file mode 100644 index 000000000..c27ac51d3 --- /dev/null +++ b/design/CLI/PoC/overlays/plugins/kustomization.yaml @@ -0,0 +1,24 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +bases: + - ../../base + +patchesStrategicMerge: + - aws-plugin.yaml # this patches the Velero deployment + # - azure-plugin.yaml # this patches the Velero deployment + +generatorOptions: + disableNameSuffixHash: true + labels: + component: velero + +secretGenerator: +- name: cloud-credentials + files: + - "cloud" + + + + + diff --git a/design/CLI/PoC/overlays/plugins/restic.yaml b/design/CLI/PoC/overlays/plugins/restic.yaml new file mode 100644 index 000000000..576ea2ff5 --- /dev/null +++ b/design/CLI/PoC/overlays/plugins/restic.yaml @@ -0,0 +1,68 @@ +--- +apiVersion: apps/v1 +kind: DaemonSet +metadata: + creationTimestamp: null + labels: + component: velero + name: restic + namespace: velero +spec: + selector: + matchLabels: + name: restic + template: + metadata: + creationTimestamp: null + labels: + component: velero + name: restic + spec: + containers: + - args: + - restic + - server + command: + - /velero + env: + - name: NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: VELERO_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + - name: VELERO_SCRATCH_DIR + value: /scratch + - name: GOOGLE_APPLICATION_CREDENTIALS + value: /credentials/cloud + - name: AWS_SHARED_CREDENTIALS_FILE + value: /credentials/cloud + - name: AZURE_CREDENTIALS_FILE + value: /credentials/cloud + image: velero/velero:latest + imagePullPolicy: Always + name: restic + resources: {} + volumeMounts: + - mountPath: /host_pods + mountPropagation: HostToContainer + name: host-pods + - mountPath: /scratch + name: scratch + - mountPath: /credentials + name: cloud-credentials + securityContext: + runAsUser: 0 + serviceAccountName: velero + volumes: + - hostPath: + path: /var/lib/kubelet/pods + name: host-pods + - emptyDir: {} + name: scratch + - name: cloud-credentials + secret: + secretName: cloud-credentials + updateStrategy: {} diff --git a/design/cli-install-changes.md b/design/cli-install-changes.md new file mode 100644 index 000000000..11ba6b6ea --- /dev/null +++ b/design/cli-install-changes.md @@ -0,0 +1,373 @@ +# Proposal for a more intuitive CLI to install and configure Velero + +Currently, the Velero CLI tool has a `install` command that configures numerous major and minor aspects of Velero. As a result, the combined set of flags for this `install` command makes it hard to intuit and reason about the different Velero components. This document proposes changes to improve the UX for installation and configuration in a way that would make it easier for the user to discover what needs to be configured by looking at what is available in the CLI rather then having to rely heavily on our documentation for the usage. At the same time, it is expected that the documentation update to reflect these changes will also make the documentation flow easier to follow. + +This proposal prioritizes discoverability and self-documentation over minimizing length or number of commands and flags. + +## Goals + +- Split flags currently under the `velero install` command into multiple commands, and group flags under commands in a way that allows a good level of discovery and self-documentation +- Maintain compatibility with gitops practices (i.e. ability to generate a full set of yaml for install that can be stored in source control) +- Have a clear path for deprecating commands + +## Non Goals + +- Introduce new CLI features +- Propose changes to the CLI that go beyond the functionality of install and configure +- Optimize for shorter length or number of commands/flags + +## Background + +This document proposes users could benefit from a more intuitive and self-documenting CLI setup as compared to our existing CLI UX. Ultimately, it is proposed that a recipe-style CLI flow for installation, configuration and use would greatly contribute to this purpose. + +Also, the `install` command currently can be reused to update Velero deployment configurations. For server and restic related install and configurations, settings will be moved to under `velero config`. + +## High-Level Design + +The naming and organization of the proposed new CLI commands below have been inspired on the `kubectl` commands, particularly `kubectl set` and `kubectl config`. + +#### General CLI improvements + +These are improvements that are part of this proposal: +- Go over all flags and document what is optional, what is required, and default values. +- Capitalize all help messages + +#### Commands + +The organization of the commands follows this format: + +``` +velero [resource] [operation] [flags] +``` + +To conform with Velero's current practice: +- commands will also work by swapping the operation/resource. +- the "object" of a command is an argument, and flags are strictly for modifiers (example: `backup get my-backup` and not `backup get --name my-backup`) + +All commands will include the `--dry-run` flag, which can be used to output yaml files containing the commands' configuration for resource creation or patching. + +`--dry-run generate resources, but don't send them to the cluster. Use with -o. Optional.` + +The `--help` and `--output` flags will also be included for all commands, omitted below for brevity. + +Below is the proposed set of new commands to setup and configure Velero. + +1) `velero config` + +``` + server Configure up the namespace, RBAC, deployment, etc., but does not add any external plugins, BSL/VSL definitions. This would be the minimum set of commands to get the Velero server up and running and ready to accept other configurations. + --label-columns stringArray a comma-separated list of labels to be displayed as columns + --show-labels show labels in the last column + --image string image to use for the Velero and restic server pods. Optional. (default "velero/velero:latest") + --pod-annotations mapStringString annotations to add to the Velero and restic pods. Optional. Format is key1=value1,key2=value2 + --restore-only run the server in restore-only mode. Optional. + --pod-cpu-limit string CPU limit for Velero pod. A value of "0" is treated as unbounded. Optional. (default "1000m") + --pod-cpu-request string CPU request for Velero pod. A value of "0" is treated as unbounded. Optional. (default "500m") + --pod-mem-limit string memory limit for Velero pod. A value of "0" is treated as unbounded. Optional. (default "256Mi") + --pod-mem-request string memory request for Velero pod. A value of "0" is treated as unbounded. Optional. (default "128Mi") + --client-burst int maximum number of requests by the server to the Kubernetes API in a short period of time (default 30) + --client-qps float32 maximum number of requests per second by the server to the Kubernetes API once the burst limit has been reached (default 20) + --default-backup-ttl duration how long to wait by default before backups can be garbage collected (default 720h0m0s) + --disable-controllers strings list of controllers to disable on startup. Valid values are backup,backup-sync,schedule,gc,backup-deletion,restore,download-request,restic-repo,server-status-request + --log-format the format for log output. Valid values are text, json. (default text) + --log-level the level at which to log. Valid values are debug, info, warning, error, fatal, panic. (default info) + --metrics-address string the address to expose prometheus metrics (default ":8085") + --plugin-dir string directory containing Velero plugins (default "/plugins") + --profiler-address string the address to expose the pprof profiler (default "localhost:6060") + --restore-only run in a mode where only restores are allowed; backups, schedules, and garbage-collection are all disabled. DEPRECATED: this flag will be removed in v2.0. Use read-only backup storage locations instead. + --restore-resource-priorities strings desired order of resource restores; any resource not in the list will be restored alphabetically after the prioritized resources (default [namespaces,storageclasses,persistentvolumes,persistentvolumeclaims,secrets,configmaps,serviceaccounts,limitranges,pods,replicaset,customresourcedefinitions]) + --terminating-resource-timeout duration how long to wait on persistent volumes and namespaces to terminate during a restore before timing out (default 10m0s) + + restic Configuration for restic operations. + --default-prune-frequency duration how often 'restic prune' is run for restic repositories by default. Optional. + --pod-annotations mapStringString annotations to add to the Velero and restic pods. Optional. Format is key1=value1,key2=value2 + --pod-cpu-limit string CPU limit for restic pod. A value of "0" is treated as unbounded. Optional. (default "0") + --pod-cpu-request string CPU request for restic pod. A value of "0" is treated as unbounded. Optional. (default "0") + --pod-mem-limit string memory limit for restic pod. A value of "0" is treated as unbounded. Optional. (default "0") + --pod-mem-request string memory request for restic pod. A value of "0" is treated as unbounded. Optional. (default "0") + --timeout duration how long backups/restores of pod volumes should be allowed to run before timing out (default 1h0m0s) + repo + get Get restic repositories +``` +The `velero config server` command will create the following resources: + +``` +Namespace +Deployment +backups.velero.io +backupstoragelocations.velero.io +deletebackuprequests.velero.io +downloadrequests.velero.io +podvolumebackups.velero.io +podvolumerestores.velero.io +resticrepositories.velero.io +restores.velero.io +schedules.velero.io +serverstatusrequests.velero.io +volumesnapshotlocations.velero.io +``` + +Note: Velero will maintain the `velero server` command run by the Velero pod, which starts the Velero server deployment. + +2) `velero backup-location` +Commands/flags for backup locations. + +``` + set + --default string sets the default backup storage location (default "default") (NEW, -- was `server --default-backup-storage-location; could be set as an annotation on the BSL) + --credentials mapStringString sets the name of the corresponding credentials secret for a provider. Format is provider:credentials-secret-name. (NEW) + --cacert-file mapStringString configuration to use for creating a secret containing a custom certificate for an S3 location of a plugin provider. Format is provider:path-to-file. (NEW) + + create NAME [flags] + --default Sets this new location to be the new default backup location. Default is false. (NEW) + --access-mode access mode for the backup storage location. Valid values are ReadWrite,ReadOnly (default ReadWrite) + --backup-sync-period 0s how often to ensure all Velero backups in object storage exist as Backup API objects in the cluster. Optional. Set this to 0s to disable sync + --bucket string name of the object storage bucket where backups should be stored. Required. + --config mapStringString configuration to use for creating a backup storage location. Format is key1=value1,key2=value2 (was also in `velero install --backup-location-config`). Required for Azure. + --provider string provider name for backup storage. Required. + --label-columns stringArray a comma-separated list of labels to be displayed as columns + --labels mapStringString labels to apply to the backup storage location + --prefix string prefix under which all Velero data should be stored within the bucket. Optional. + --provider string name of the backup storage provider (e.g. aws, azure, gcp) + --show-labels show labels in the last column + --credentials mapStringString sets the name of the corresponding credentials secret for a provider. Format is provider:credentials-secret-name. (NEW) + --cacert-file mapStringString configuration to use for creating a secret containing a custom certificate for an S3 location of a plugin provider. Format is provider:path-to-file. (NEW) + + get Display backup storage locations + --default displays the current default backup storage location (NEW) + --label-columns stringArray a comma-separated list of labels to be displayed as columns + -l, --selector string only show items matching this label selector + --show-labels show labels in the last column + +``` + +3) `velero snapshot-location` +Commands/flags for snapshot locations. + +``` + set + --default mapStringString sets the list of unique volume providers and default volume snapshot location (provider1:location-01,provider2:location-02,...) (NEW, -- was `server --default-volume-snapshot-locations; could be set as an annotation on the VSL) + --credentials mapStringString sets the list of name of the corresponding credentials secret for providers. Format is (provider1:credentials-secret-name1,provider2:credentials-secret-name2,...) (NEW) + + create NAME [flags] + --default Sets these new locations to be the new default snapshot locations. Default is false. (NEW) + --config mapStringString configuration to use for creating a volume snapshot location. Format is key1=value1,key2=value2 (was also in `velero install --`snapshot-location-config`). Required. + --provider string provider name for volume storage. Required. + --label-columns stringArray a comma-separated list of labels to be displayed as columns + --labels mapStringString labels to apply to the volume snapshot location + --provider string name of the volume snapshot provider (e.g. aws, azure, gcp) + --show-labels show labels in the last column + --credentials mapStringString sets the list of name of the corresponding credentials secret for providers. Format is (provider1:credentials-secret-name1,provider2:credentials-secret-name2,...) (NEW) + + get Display snapshot locations + --default list of unique volume providers and default volume snapshot location (provider1:location-01,provider2:location-02,...) (NEW -- was `server --default-volume-snapshot-locations`)) +``` + +4) `velero plugin` +Configuration for plugins. + +``` + add stringArray IMAGES [flags] - add plugin container images to install into the Velero Deployment + + get get information for all plugins on the velero server (was `get`) + --timeout duration maximum time to wait for plugin information to be reported (default 5s) + + remove Remove a plugin [NAME | IMAGE] + + set + --credentials-file mapStringString configuration to use for creating a secret containing the AIM credentials for a plugin provider. Format is provider:path-to-file. (was `secret-file`) + --no-secret flag indicating if a secret should be created. Must be used as confirmation if create --secret-file is not provided. Optional. (MOVED FROM install -- not sure we need it?) + --sa-annotations mapStringString annotations to add to the Velero ServiceAccount for GKE. Add iam.gke.io/gcp-service-account=[GSA_NAME]@[PROJECT_NAME].iam.gserviceaccount.com for workload identity. Optional. Format is key1=value1,key2=value2 +``` + +#### Example + +Considering this proposal, let's consider what a high-level documentation for getting Velero ready to do backups could look like for Velero users: + +After installing the Velero CLI: +``` +velero config server [flags] (required) +velero config restic [flags] +velero plugin add IMAGES [flags] (add/config provider plugins) +velero backup-location/snapshot-location create NAME [flags] (run `velero plugin --get` to see what kind of plugins are available; create locations) +velero backup/restore/schedule create/get/delete NAME [flags] +``` + +The above recipe-style documentation should highlight 1) the main components of Velero, and, 2) the relationship/dependency between the main components + +### Deprecation + +#### Timeline + +In order to maintain compatibility with the current Velero version for a sufficient amount of time, and give users a chance to upgrade any install scripts they might have, we will keep the current `velero install` command in parallel with the new commands until the next major Velero version, which will be Velero 2.0. In the mean time, ia deprecation warning will be added to the `velero install` command. + +#### Commands/flags deprecated or moved + +##### Velero Install +`velero install (DEPRECATED)` + +Flags moved to... + +...`velero config server`: +``` + --image string image to use for the Velero and restic server pods. Optional. (default "velero/velero:latest") + --label-columns stringArray a comma-separated list of labels to be displayed as columns + --pod-annotations mapStringString annotations to add to the Velero and restic pods. Optional. Format is key1=value1,key2=value2 + --show-labels show labels in the last column + --pod-cpu-limit string CPU limit for Velero pod. A value of "0" is treated as unbounded. Optional. (default "1000m") + --pod-cpu-request string CPU request for Velero pod. A value of "0" is treated as unbounded. Optional. (default "500m") + --pod-mem-limit string memory limit for Velero pod. A value of "0" is treated as unbounded. Optional. (default "256Mi") + --pod-mem-request string memory request for Velero pod. A value of "0" is treated as unbounded. Optional. (default "128Mi") +``` + +...`velero config restic` +``` + --default-prune-frequency duration how often 'restic prune' is run for restic repositories by default. Optional. + --pod-cpu-limit string CPU limit for restic pod. A value of "0" is treated as unbounded. Optional. (default "0") + --pod-cpu-request string CPU request for restic pod. A value of "0" is treated as unbounded. Optional. (default "0") + --pod-mem-limit string memory limit for restic pod. A value of "0" is treated as unbounded. Optional. (default "0") + --pod-mem-request string memory request for restic pod. A value of "0" is treated as unbounded. Optional. (default "0") +``` + +...`backup-location create` +``` + --backup-location-config mapStringString configuration to use for the backup storage location. Format is key1=value1,key2=value2 + --bucket string name of the object storage bucket where backups should be stored + --prefix string prefix under which all Velero data should be stored within the bucket. Optional. +``` + +...`snapshot-location create` +``` + --snapshot-location-config mapStringString configuration to use for the volume snapshot location. Format is key1=value1,key2=value2 +``` + +...both `backup-location create` and `snapshot-location create` +``` + --provider string provider name for backup and volume storage +``` + +...`plugin` +``` + --plugins stringArray Plugin container images to install into the Velero Deployment + --sa-annotations mapStringString annotations to add to the Velero ServiceAccount. Add iam.gke.io/gcp-service-account=[GSA_NAME]@[PROJECT_NAME].iam.gserviceaccount.com for workload identity. Optional. Format is key1=value1,key2=value2 + --no-secret flag indicating if a secret should be created. Must be used as confirmation if --secret-file is not provided. Optional. + --secret-file string (renamed `credentials-file`) file containing credentials for backup and volume provider. If not specified, --no-secret must be used for confirmation. Optional. +``` + +Flags to deprecate: +``` + --no-default-backup-location flag indicating if a default backup location should be created. Must be used as confirmation if --bucket or --provider are not provided. Optional. + --use-volume-snapshots whether or not to create snapshot location automatically. Set to false if you do not plan to create volume snapshots via a storage provider. (default true) + --wait wait for Velero deployment to be ready. Optional. + --use-restic (obsolete since now we have `velero config restic`) +``` + +##### Velero Server + +These flags will be moved to under `velero config server`: + +`velero server --default-backup-storage-location (DEPRECATED)` changed to `velero backup-location set --default` + +`velero server --default-volume-snapshot-locations (DEPRECATED)` changed to `velero snapshot-location set --default` + +The value for these flags will be stored as annotations. + +## Detailed Design + +#### Handling CA certs + +In anticipation of a new configuration implementation to handle custom CA certs (as per design doc https://github.com/vmware-tanzu/velero/blob/master/design/custom-ca-support.md), a new flag `velero storage-location create/set --cacert-file mapStringString` is proposed. It sets the configuration to use for creating a secret containing a custom certificate for an S3 location of a plugin provider. Format is provider:path-to-file. + +See discussion https://github.com/vmware-tanzu/velero/pull/2259#discussion_r384700723 for more clarification. + +#### Renaming "provider" to "location-plugin" + +As part of this change, we should change to use the term `location-plugin` instead of `provider`. The reasoning: in practice, we usually have 1 plugin per provider, and if there is an implementation for both object store and volume snapshotter for that provider, it will all be contained in the same plugin. When we handle plugins, we follow this logic. In other words, there's a plugin name (ex: `velero.io/aws`) and it can contain implementations of kind `ObjectStore` and/or `VolumeSnapshotter`. + +But when we handle BSL or VSL (and the CLI commands/flags that configure them), we use the term `provider`, which can cause ambiguity as if that is a kind of thing different from a plugin. If the plugin is the "thing" that contains the implementation for the desired provider, we should make it easier for the user to guess that and change BackupStorageLocation/VolumeSnapshotLocation `Spec.Provider` field to be called `Spec.Location-Plugin` and all related CLI command flags to `location-plugin`, and update the docs accordingly. + +This change will require a CRD version bump and deprecation cycle. + +#### GitOps Compatibility + +To maintain compatibility with gitops practices, each of the new commands will generate `yaml` output that can be stored in source control. + +For content examples, please refer to the files here: + +https://github.com/carlisia/velero/tree/c-cli-design/design/CLI/PoC + +Note: actual `yaml` file names are defined by the user. + +`velero config server` - base/deployment.yaml + +`velero config restic` - overlays/plugins/restic.yaml + +`velero backup-location create` - base/backupstoragelocations.yaml + +`velero snapshot-location create` - base/volumasnapshotlocations.yaml + +`velero plugin add velero/velero-plugin-for-aws:v1.0.1` - overlays/plugins/aws-plugin.yaml + +`velero plugin add velero/velero-plugin-for-microsoft-azure:v1.0.1` - overlay/plugins/azure-plugin.yaml + +These resources can be deployed/deleted using the included kustomize setup and running: + +``` +kubectl apply -k design/CLI/PoC/overlays/plugins/ + +kubectl delete -k design/CLI/PoC/overlays/plugins/ +``` + +Note: All CRDs, including the `ResticRepository`, may continue to be deployed at startup as it is now, or together with their respective instantiation. + + +#### Changes to startup behavior + +To recap, this proposal redesigns the Velero CLI to make `velero install` obsolete, and instead breaks down the installation and configuration into separate commands. These are the major highlights: + +- Plugins will only be installed separately via `velero plugin add` +- BSL/VSL will be continue to be configured separately, and now each will have an associated secret + +Since each BSL/VSL will have its own association with a secret, the user will no longer need to upload a new secret whenever changing to, or adding, a BSL/VSL for a provider that is different from the one in use. This will be done at setup time. This will make it easier to support any number of BSL/VSL combinations, with different providers each. + +The user will start up the Velero server on a cluster by using the command `velero config server`. This will create the Velero deployment resource with default values or values overwritten with flags, create the Velero CRDs, and anything else that is not specific to plugins or BSL/VSL. + +The Velero server will start up, verify that the deployment is running, that all CRDs were found, and log a message that it is waiting for a BSL to be configured. at this point, other operations, such as configuring restic, will be allowed. Velero should keep track of its status, ie, if it is ready to create backups or not. This could be a field `ServerStatus` added to `ServerStatusRequest`. Possible values could be [ready|waiting]. "ready" would mean there is at least 1 valid BSL, and "waiting" would be anything but that. + +When adding/configuring a BSL or VSL, we will allow creating locations, and continuously verify if there is a corresponding, valid plugin. When a valid match is found, mark the BSL/VSL as "ready". This would require adding a field to the BSL/VSL, or using the existing `Phase` field, and keep track of its status, possibly: [ready|waiting]. + +With the first approach: the server would transition into "ready" (to create backups) as soon as there is one BSL. It would require a set sequence of actions, ie, first install the plugin, only then the user can successfully configure a BSL. + +With the second approach, the Velero server would continue looping and checking all existing BSLs for at least 1 with a "ready" status. Once it found that, it would set itself to "ready" also. + +Another new behavior that must be added: the server needs to identify when there no longer exists a valid BSL. At this point, it should change its status from "ready" to one that indicates it is not ready, maybe "waiting". With the first approach above, this would mean checking if there is still at least one BSL. With the second approach, it would require checking the status of all BSLs to find at least one with the status of "ready". + +As it is today, a valid VSL would not be required to create backups, unless the backup included a PV. + +To make it easier for the user to identify if their Velero server is ready to create backups or not, a `velero status` command should be added. This issue has been created some time ago for this purpose: https://github.com/vmware-tanzu/velero/issues/1094. + +## Alternatives Considered + +It seems that the vast majority of tools document their usage with `kubectl` and `yaml` files to install and configure their Kubernetes resources. Many of them also make use of Helm, and to a lesser extent some of them have their own CLI tools. + +Amongst the tools that have their own CLI, not enough examples were found to establish a clear pattern of usage. It seems the most relevant priority should be to have output in `yaml` format. + +Any set of `yaml` files can also be arranged to use with Kustomize by creating/updating resources, and patching them using Kustomize functionalities. + +The way the Velero commands were arranged in this proposal with the ability to output corresponding `yaml` files, and the included Kustomize examples, makes it in line with the widely used practices for installation and configuration. + +Some CLI tools do not document their usage with Kustomize, one could assume it is because anyone with knowledge of Kustomize and `yaml` files would know how to use it. + +Here are some examples: + +https://github.com/jetstack/kustomize-cert-manager-demo + +https://github.com/istio/installer/tree/master/kustomize + +https://github.com/weaveworks/flagger/tree/master/kustomize + +https://github.com/jpeach/contour/tree/1c575c772e9fd747fba72ae41ab99bdae7a01864/kustomize (RFC) + +## Security Considerations + +N/A \ No newline at end of file