# kas shutdown

kas reacts to `SIGTERM` by starting a graceful shutdown process.
The shutdown process should satisfy these requirements:

## Requirements

1. Do not disrupt any in-flight processing to the maximum extent possible.
2. Shutdown as quickly as possible without violating the first requirement.

## TL;DR

kas shuts down as fast as it can.
There are no agentk -> kas connections that block shutdown.
What blocks the shutdown process are the user's requests that are being proxied.
It'd be very unfortunate to immediately drop them as that is bad UX.

We cannot tell people to "just retry" because that's a
["pretty horrible" UX](https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/671#note_924882799) (user quote).
The linked issue was caused by a bug, not unexpected shutdown, but the outcome/UX is identical from the user's perspective.

## Implementation details

kas is a modular monolith.
Conceptually it consists of a base layer, a modules framework, and [modules](modules.md).
Base layer implements most of the shared functionality that is exposed to modules via the modules framework.
Modules framework starts and stops modules and gRPC servers in stages.
Shutdown of stages is done in reverse order.
If stage `B` depends on stage `A` then the ordered shutdown ensures that `B` gets the shutdown signal first,
all goroutines of `B` finish, and only then `A` gets a signal to shut down.
The application exits once all stages have shut down.

As of this writing, base layer owns all gRPC servers.
Some modules need certain functionality that uses those servers (e.g. request routing uses the in-memory private API server).
So, modules can let the framework know that they need to be shut down before or after those gRPC servers.
This ensures gRPC servers are not shutdown while still in use.
This approach is followed throughout the whole application.

A module may expose a gRPC API on the existing servers and/or a new API endpoint, like an HTTP server.
As of this writing, we have two modules that expose HTTP servers: Kubernetes API proxy and Observability (metrics, etc).

### Maximum connection age

kas is usually not exposed to the agents directly on the network.
There likely are reverse proxies, load balancers, and other "network boxes" in front of kas that handle the incoming traffic.
Agentk might also be making connections to kas via a forward proxy, etc.
On GitLab.com we have something like
CloudFlare -> GCP load balancer -> HAProxy -> GCP load balancer for the in-cluster traffic routing -> kas.
L4/L7 hops often have and enforce a maximum time a TCP connection/HTTP request can stay open for.
E.g. HAProxy would reset the connection after a certain time.
Obviously, this would disrupt the in-flight requests on the connection.

To reduce the risk of running into the above issue, kas and agentk use
[gRPC's max connection age configuration knobs](https://github.com/grpc/proposal/blob/master/A9-server-side-conn-mgt.md)
to ensure clients wrap up and gracefully disconnect before a proxy/LB kills a connection.

The code that does this lives in [max_conn_age.go](../internal/tool/grpctool/max_conn_age.go).
See comments there for some more info. It's messy but it works really well.
gRPC developers resist to
[provide a proper mechanism](https://github.com/grpc/grpc/issues/26703)
to react to max connection age having been reached.

A context is available to all gRPC handlers to receive a signal that maximum connection age has been reached.
kas has a few gRPC handlers that poll GitLab/Gitaly and only return a response to agentk when there is something
to respond with.
Such handlers react to the signal about maximum connection age and return cleanly.
Client recognizes a clean return and retries immediately.
When maximum connection age is reached, gRPC sends a `GOAWAY` frame - see the
[spec doc](https://github.com/grpc/proposal/blob/master/A9-server-side-conn-mgt.md), linked above, and the
[HTTP/2 spec section on `GOAWAY`](https://httpwg.org/specs/rfc7540.html#rfc.section.6.8).
The client stops using the TCP connection for new requests.
Because the context signals after maximum connection age has been reached, and `GOAWAY` has been sent,
new requests are guaranteed to use a different TCP connection.
When this happens while kas is shutting down, the load balancer will route all new TCP connections to kas replicas
that are not shutting down.

Max connection age is configurable per listener and, as of this writing, defaults to 2 hours.
On GitLab.com, HAProxy is configured appropriately too.

### gRPC servers shutdown

gRPC servers react to the shutdown signal the following way:

1. Sleep for 5 seconds (configurable per listener). This ensures that, if kas is deployed to Kubernetes, `Service` / `Ingress`
   have time to respond to `Pod` shutdown and stop routing new requests to that replica.
   If kas immediately stopped accepting new connections, they would be dropped and not sent to a different `Pod`.
   See https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/299+ for background.
2. Close the listener to stop accepting new connections.
3. Send `GOAWAY` on all open TCP connections to let the clients know that the server doesn't want to talk anymore.
4. After 1 second signal all the still running RPCs that the server is shutting down. This is done using the same
   context as max connection age. For a handler there is no difference why it needs to stop - due to max connection age
   or because the server is shutting down. There is [no way to know](https://github.com/grpc/grpc-go/issues/6830) when
   gRPC had sent `GOAWAY`, so we just wait 1 second (in code this step runs concurrently with the previous and
   the next steps).
5. Waiting for all in-flight RPCs to finish.

Once all RPCs finish, the stage where the server was running finishes too.
All dependant stages can shut down now.

### Kubernetes API proxy shutdown

1. Sleep for 5 seconds (configurable). Same as for gRPC servers, this is to work correctly when deployed to Kubernetes.
2. Close the listener to stop accepting new connections.
3. Wait for all in-flight requests to finish.

Proxy reacts to the same "kas is shutting down" context as gRPC servers.
At the moment only the [aggregated watch API](watch-aggregator-api.md) respects it though.
All other requests block server shutdown.
This is because proxy is not aware of the semantics of the request being handled - there is no safe moment to abort an in-flight request.
One exception is watch requests.
~~They are safe to abort at any moment because clients are designed and equipped to correctly and efficiently re-establish a watch.
We could abort the watch requests to speed up kas shutdown to aid goal 2.
This is tracked in https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/262.~~

It turned out even `kubectl` is not (always) robust enough to restore broken watch connections.
We've decided to strive to provide better UX than Kubernetes that drops watch requests immediately on shutdown.
