# Watch Aggregator API

The Kubernetes API module in KAS supports a special `/watch`
endpoint. This endpoint is able to aggregate watches on multiple
resources within a single WebSocket connection.
Individual watches can be started and stopped by sending
messages on the WebSocket connection.

This document serves as a specification for the `gitlab-agent-watch-api`
WebSocket subprotocol supported at `/watch`.

## Protocol

The Watch Aggregator API is based on a standard [WebSocket] communication.
The message formats for the request and response messages are defined
by the `gitlab-agent-watch-api` [subprotocol].
The `/watch` API endpoint only accepts this subprotocol and no other
(not even the empty `""` subprotocol).

### `gitlab-agent-watch-api` subprotocol

#### Requests

The follow messages can be sent by the client to request starting or
stopping a watch.

##### Start a Watch

The following message can be used to start an individual watch:

```json
{
  "type": "watch",
  "watch_id": "<arbitrary-id>",
  "watch_params": {
    "group": "<k8s-group>",
    "version": "<k8s-version>",
    "resource": "<k8s-resource>"
  }
}
```

Fields:

- **type**: use `watch` to start a new watch.
- **watch_id**: a unique watch id to later identify watch events and to stop it.
- **watch_params**: object with information for the Kubernetes Watch API.
    - **group** (optional): the name of the Kubernetes group (maybe empty for the built-in resources).
    - **version**: the version of the Kubernetes resource.
    - **resource**: the name of the Kubernetes resource.
    - **resource_version**: see [upstream API reference for `resourceVersion`](https://kubernetes.io/docs/reference/kubernetes-api/common-parameters/common-parameters/#resourceVersion).
    - **allow_watch_bookmarks**: see [upstream API reference for `allowWatchBookmarks`](https://kubernetes.io/docs/reference/kubernetes-api/common-parameters/common-parameters/#allowWatchBookmarks).
    - **send_initial_events**: see [upstream API reference for `sendInitialEvents`](https://kubernetes.io/docs/reference/kubernetes-api/common-parameters/common-parameters/#sendInitialEvents).
    - **field_selector**: see [upstream documentation for `fieldSelector`](https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/)
    - **label_selector**: see [upstream documentation for `labelSelector`](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#list-and-watch-filtering)

Example:

```json
{
  "type": "watch",
  "watch_id": "all-pods",
  "watch_params": {
    "version": "v1",
    "resource": "pods"
  }
}
```

This message will start a watch on the Kubernetes API server
loosely translating to a `GET /api/v1/pods?watch=true` request.

##### Stop a Watch

The following message can be used to stop an individual watch:

```json
{
  "type": "unwatch",
  "watch_id": "<known-watch-id>"
}
```

Fields:

- **type**: use `unwatch` to stop a watch.
- **watch_id**: a watch id that was previously used with a [start request](#start-a-watch).

Example:

```json
{
  "type": "unwatch",
  "watch_id": "all-pods"
}
```

#### Responses

The following messages are sent by the server and should be handled by the client.

##### Watch Event

Once a [watch has been started successfully](#start-a-watch) the server will sent
watch events when they happen. A watch event has the following format:

```json
{
  "type": "event",
  "watch_id": "<watch-id>",
  "event": { ... }
}
```

Fields:

- **type**: `event` indicates that the enclosed `event` field contains a Kubernetes watch event object.
- **watch_id**: the watch id for which this event happened.
- **event**: the standard unmodified Kubernetes watch event object.

Example

```json
{
  "type": "event",
  "watch_id": "all-pods",
  "event": {
    "type": "ADDED",
    "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "8467", "name": "foo"}, ...}
  }
}
```

##### Watch Stop Event

The `stop` event is sent when the watch stopped "normally", for example after
the client requests to [`unwatch`](#stop-a-watch) an established watch.

```json
{
  "type": "stop",
  "watch_id": "<watch-id>"
}
```

Fields:

- **type**: `stop` indicates that the specific watch has been stopped.
- **watch_id**: the watch id of the watch to stop.

##### Errors

Errors messages can be sent from the server under many circumstances,
for example if the watch cannot be established
or if a watch is abruptly closed
or if a watch event cannot be decoded.

An error message has the following format:

```json
{
  "type": "error",
  "watch_id": "<watch-id>",
  "error": {
    "type": "<error-type",
    "message": "<error-message>",
    "object": { ... }
  }
}
```

Fields:

- **type**: `error` indicates that an error has happened.
- **watch_id**: if set, contains the watch id for which the error has happened.
- **error**: object with error details.
- **type**: the error type, either `WATCH_REQUEST_FAILED` or `WATCH_FAILED`.
  the `WATCH_REQUEST_FAILED` indicates that the watch request itself was not valid or
  the watch could not be established at all. The `WATCH_FAILED` type indicates an error
  that happened once the watch has already been established.
- **object**: if set, contains a Kubernetes object with additional error details
  from the Kubernetes API server. This is most likely a
  [`Status`](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Status) object,
  but in very rare cases may be an arbitrary [`runtime.Object`].
  Please check on the return object type.

If an error occurred, the Watch Aggregator API will emit the aforementioned error message(s)
and close the connection. The client may restart the watch aggregation request.

#### Behavioral Notes

- The server to terminate all watches when it shuts down.
  The client may immediately try to re-establish the connection.
  In a future iteration (and some enhancements in the upstream websocket package)
  the server will close the connection with a dedicates status and reason for this situation.
- Started watches may not be stopped explicitly before the client closes the connection.
- There is an arbitrary and hard-coded limit of `100` watches per aggregated watch request.
- An open watch will block the receiving agents from a shutdown.
  This behavior is the same if a direct watch is requests.
  See [issue to fix this](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/262).
- The server does not retry any failed watches.

[Websocket]: https://datatracker.ietf.org/doc/html/rfc6455
[subprotocol]: https://datatracker.ietf.org/doc/html/rfc6455#section-1.9
