Add a test to verify that the kube-cert-agent recovers when a pod becomes unhealthy.

This required some small adjustments to the produciton code to make it more feasible to test.

The new test takes an existing agent pod and terminates the `sleep` process, causing the pod to go into an `Error` status.
The agent controllers _should_ respond to this by deleting and recreating that failed pod, but the current code just gets stuck.

This is meant to replicate the situation when a cluster is suspended and resumed, which also causes the agent pod to be in this terminal error state.

Signed-off-by: Matt Moyer <moyerm@vmware.com>

internal/controller/kubecertagent/kubecertagent.go

@@ -129,7 +129,7 @@ func (c *AgentPodConfig) newAgentPod(controllerManagerPod *corev1.Pod) *corev1.P
 					Name:            "sleeper",
 					Image:           c.ContainerImage,
 					ImagePullPolicy: corev1.PullIfNotPresent,
-					Command:         []string{"/bin/sleep", "infinity"},
+					Command:         []string{"/bin/sh", "-c", "/bin/sleep infinity"},
 					VolumeMounts:    controllerManagerPod.Spec.Containers[0].VolumeMounts,
 					Resources: corev1.ResourceRequirements{
 						Limits: corev1.ResourceList{

internal/controller/kubecertagent/kubecertagent_test.go

@@ -1,4 +1,4 @@
-// Copyright 2020 the Pinniped contributors. All Rights Reserved.
+// Copyright 2020-2021 the Pinniped contributors. All Rights Reserved.
 // SPDX-License-Identifier: Apache-2.0
 
 package kubecertagent
@@ -101,7 +101,7 @@ func exampleControllerManagerAndAgentPods(
 					Image:           "some-agent-image",
 					ImagePullPolicy: corev1.PullIfNotPresent,
 					VolumeMounts:    controllerManagerPod.Spec.Containers[0].VolumeMounts,
-					Command:         []string{"/bin/sleep", "infinity"},
+					Command:         []string{"/bin/sh", "-c", "/bin/sleep infinity"},
 					Resources: corev1.ResourceRequirements{
 						Limits: corev1.ResourceList{
 							corev1.ResourceMemory: resource.MustParse("16Mi"),