Fix deadlock during shutdown which prevented leader election cleanup

Before this fix, the deadlock would prevent the leader pod from giving
up its lease, which would make it take several minutes for new pods to
be allowed to elect a new leader. During that time, no Pinniped
controllers could write to the Kube API, so important resources were not
being updated during that window. It would also make pod shutdown take
about 1 minute.

After this fix, the leader gives up its lease immediately, and pod
shutdown takes about 1 second. This improves restart/upgrade time and
also fixes the problem where there was no leader for several minutes
after a restart/upgrade.

The deadlock was between the post-start hook and the pre-shutdown hook.
The pre-shutdown hook blocked until a certain background goroutine in
the post-start hook finished, but that goroutine could not finish until
the pre-shutdown hook finished. Thus, they were both blocked, waiting
for each other infinitely. Eventually the process would be externally
killed.

This deadlock was most likely introduced by some change in Kube's
generic api server package related to how the many complex channels used
during server shutdown interact with each other, and was not noticed
when we upgraded to the version which introduced the change.

internal/controllerlib/controller.go

@@ -1,4 +1,4 @@
-// Copyright 2020 the Pinniped contributors. All Rights Reserved.
+// Copyright 2020-2023 the Pinniped contributors. All Rights Reserved.
 // SPDX-License-Identifier: Apache-2.0
 
 package controllerlib
@@ -102,6 +102,7 @@ func (c *controller) Run(ctx context.Context, workers int) {
 	workerContext, workerContextCancel := context.WithCancel(context.Background())
 
 	defer func() {
+		plog.Debug("starting to shut down controller workers", "controller", c.Name(), "workers", workers)
 		c.queue.ShutDown()    // shutdown the controller queue first
 		workerContextCancel() // cancel the worker context, which tell workers to initiate shutdown
 
@@ -126,7 +127,9 @@ func (c *controller) Run(ctx context.Context, workers int) {
 		}()
 	}
 
+	plog.Debug("controller started", "controller", c.Name(), "workers", workers)
 	<-ctx.Done() // wait for controller context to be cancelled
+	plog.Debug("controller context cancelled, next will terminate workers", "controller", c.Name(), "workers", workers)
 }
 
 func (c *controller) invokeAllRunOpts() {