# `kas` to `agentk` connectivity

This is a technical design for https://gitlab.com/groups/gitlab-org/-/epics/12180+.
It extends the system defined in [kas_request_routing.md](kas_request_routing.md).

> As a self-managed GitLab user, I need to deploy to my in-cloud clusters that can not access GitLab directly.
> On the other hand, GitLab can access these clusters.

To make it possible to use GitLab Agent for Kubernetes in the above use case, we have to establish the tunnel connection
from `kas` to `agentk`.
Then `agentk` can use such connection to make requests to `kas`.

**Assumption**: this feature is only needed on self-managed GitLab installations and will not be used on GitLab.com.
This is an important constraint because we don't have to worry about abuse that much.
If it was enabled on GitLab.com, users would be able to use GitLab as a mechanism to DoS unrelated URLs by specifying
those URLs many times (as distinct agents).

At the same time, both modes should be supported by self-managed installations.

All kas instances would connect to all registered agents.
This is not a scalability issue since self-managed installations run 1-2 kas instances only as this is more than enough.

## UI / UX

`kas` needs a list of URLs of `agentk`s to connect to.
We need a UI that allows to register agents in both modes - the existing mode (`agentk` connects to `kas`) and
the new mode (`kas` connects to `agentk`).

The existing UI provides agent installation instructions with the `kas` URL and a token to use.

New UI should allow to specify URL of the agent (required). Additional **optional** inputs:

- Certificate authority (CA) certificate to validate `agentk`'s certificate.
- Client certificate and client key to configure mutual TLS.
- Host name for certificate validation.

We'll use JWT tokens for authentication with EdDSA for signing.
The new UI should provide JWT public key the agent should use to validate `kas`'s JWT signature.
The private key should be stored in the database in an encrypted form.

Agent's URL is persisted in the DB. `kas` periodically gets information about agents to connect to.
For each agent, a separate connection manager is started.
It maintains an autoscaling pool of reverse tunnel connections that are used by the agent to talk to `kas`.

User should be able to update the information they provided when registering an agent.
TLS certificate rotation is one of the use cases that requires this capability.

## `agentk` -> `kas` connectivity

This section documents the case when `agentk` connects to `kas`.

### `agentk` -> `kas` request flow

When `agentk` needs to make a request to `kas`, it just makes a request since it has direct connectivity.
`Client` is a component inside `agentk`, such as the code that fetches configuration.

```mermaid
graph TB
  Client -- request --> agentk
  agentk -- request --> kas
```

### `kas` -> `agentk` request flow

When `kas` needs to make a request to `agentk`, it goes via the reverse tunnel, established by `agentk` to a `kas` instance.
Arrows show the direction of the TCP connection establishment, from client to server.
The `Client` can be something outside of `kas` (e.g. the monolith) or inside of `kas` (e.g. Kubernetes API reverse proxy).

```mermaid
graph TB
  routing-kas[Routing kas]
  gateway-kas[Gateway kas]

  Client -- request --> routing-kas
  routing-kas --> gateway-kas
  agentk -- reverse tunnel --> gateway-kas
```

The `Gateway kas` and `Routing kas` can be the same or different `kas` instances.

## `kas` -> `agentk` connectivity

This section documents the case when `kas` connects to `agentk`.

### `agentk` -> `kas` request flow

When `agentk` needs to make a request to `kas`, it goes via the reverse tunnel, established by
`kas` to an `agentk` instance.
Arrows show the direction of the TCP connection establishment, from client to server.
The `Client` is a component inside `agentk`, such as the code that fetches configuration.

```mermaid
graph TB
  routing-agentk[Routing agentk]
  gateway-agentk[Gateway agentk]

  Client -- request --> routing-agentk
  routing-agentk --> gateway-agentk
  kas -- reverse tunnel --> gateway-agentk
```

The `Gateway agentk` and `Routing agentk` can be the same or different `agentk` instances.

### `kas` -> `agentk` request flow

When `kas` needs to make a request to `agentk`, it just makes a request since it has direct connectivity.
The `Client` can be something outside of `kas` (e.g. the monolith) or inside of `kas` (e.g. Kubernetes API reverse proxy).

```mermaid
graph TB
  Client -- request --> kas
  kas -- request --> agentk
```

## Implementation notes

When a connection is established from `agentk` to `kas`, `kas` uses Redis to store information about which
`kas` has connections from which agents (by agent id).

When a connection is established from `kas` to `agentk`, `agentk` discovers other `agentk` `Pod`s by looking for them
in the same namespace, filtering by labels set in `Deployment`'s `spec.selector.matchLabels`.
They are passed to the `Pod` in an environment variable `POD_SELECTOR_LABELS`.
The variable contains a
[label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors)
in a string form. Examples:

- Equality-based: `environment=production,tier=frontend`
- Set-based: `environment in (production),tier in (frontend)`

Note that there is no need to do URL escaping here, `agentk` will do it once it puts the string into a URL to make a request.

We assume that all `Pod`s use the same configuration and hence use the listen port from configuration.
It's a new port for internal `agentk`->`agentk` communication, similar to `kas`' private API port.
TLS on the port can be configured via command line flags, similar to observability port configuration.

Current routing and tunnel finding code in `kas` will be refactored to make it possible to reuse it in `agentk`.
There are three roles in the code:

- Router. This is the instance of `kas` (or `agentk`) that has a client that needs to make a request to `agentk` (or `kas`). It looks for a gateway `kas` (or `agentk`) instance that has a connection from the `agentk` (or `kas`) it wants to talk to and send a request to it.
- Gateway. This is an instance of `kas` (or `agentk`) that has a connection from an `agentk` (or `kas`). It accepts incoming requests from Router instance and, if there is a matching tunnel, routes that request through it.
- Tunnel client. This is `agentk` (or `kas`) that keeps some number of connections to the Gateway. It re-establishes connections after they had been used, scales the number of connections (from 2 to 500 at the moment).

### `/api/v4/internal/kubernetes/receptive_agents` API in the monolith

`kas` calls this API to get the list of agents it needs to keep a connection to.
This API endpoint is called every 5 minutes (configurable via `kas` configuration file).

Request:

```text
GET /api/v4/internal/kubernetes/receptive_agents
Accept: application/json
Gitlab-Kas-Api-Request: JWT token
```

Response on success:

```text
HTTP/1.1 200 OK
Content-Type: application/json

{
  "agents": [
    {
      "id": 42,
      "url": "grpcs://company.example.com:port/some/path",
      "ca_cert": "...",
      "tls_host": "...",
      "jwt": {
        "private_key": "..."
      },
      "mtls": {
        "client_cert": "...",
        "client_key": "..."
      }
    }
  ]
}
```

Notes:

- Response JSON contains the top-level `agents` field so that we can extend the object with more fields later if needed. `kas` ignores unknown fields here.
- `id` is the agent ID.
- Scheme of the URL can be `grpc` or `grpcs` (TLS). Port and path parts are optional. Port `80` is used for `grpc` connections and `443` for `grpcs`. No path (i.e. `/`) is used unless specified.
- `ca_cert` specifies certificate authority certificate to validate the `agentk`'s TLS certificate. Optional.
- `tls_host` is the host name to use for TLS certificate validation. If specified, it's used instead of the domain that is part of the `url`.
- `jwt.private_key` is the base64-encoded EdDSA private key to generate a JWT signature for each request.
- `mtls.client_key` and `mtls.client_cert` specify client key and certificate to use for mutual TLS.
- Either `jwt` or `mtls` must be present, but not both.

When `kas` generates a JWT token to call `agentk`:
- Audience is set to `gitlab-agent`.
- Issuer to `gitlab-kas`.
- Token is valid for 5 seconds.
- Not before is now-5 seconds.

When `agentk` generates a JWT token to call the same or another `agentk` `Pod`:
- Audience is set to `gitlab-agent`.
- Issuer to `gitlab-agent`.
- Token is valid for 5 seconds.
- Not before is now-5 seconds.
