# GitLab-managed Kubernetes resources

## Problem to solve

As a Platform Engineer, I want to provide self-service capabilities for developers when they start a new project
so they can deploy their applications without manual action from my end and with keeping other applications safe.

As a Platform Engineer, I want to enable developers to deploy testing/development versions of the applications
either to dedicated or to a shared Kubernetes namespace with a dedicated or default service account without giving
the developers authorization to create namespaces and service accounts, so they can test and share their
version easily for testing without compromising the cluster.

### Idea

Allow the platform engineer to provide a mechanism for application developers to self-provision a Kubernetes
namespaces without allowing application developers administrative rights in the cluster.

Operate the mechanism from environment "events" to support both GitLab CI and UI/API-based workflows.
After the namespace is created, allow CI jobs to have restricted access to the namespace.

## Requirements

- May create zero, one or more Kubernetes namespace(s) and service account(s) for every new/restarted GitLab environment or per project depending on configuration.
- Remove the related resources when the GitLab environment is stopped.
  - Allow an environment to be marked as ephemeral/static. Do not remove the associated resources when the environment is stopped.
  - Otherwise, remove the related resources. If a namespace is shared by multiple environments, only remove shared resources when no environments refer to them.
- Ensure the namespace is allowed according to the agent configuration, to prevent the impersonated user (static identity or CI job) specifying a namespace they shouldn't have access to.
- Allow the identity the CI job uses (static identity or CI job impersonation) to access the newly created namespace.
- Creation and removal of the Kubernetes resources should not consume [compute minutes](https://docs.gitlab.com/ee/ci/pipelines/compute_minutes.html).
- Make it easy to find GitLab environment(s) knowing the cluster namespace.
- Make it easy to find the cluster namespace knowing the GitLab environment.
- Allow multiple environments to share the same namespace.
- Environment deletion should not delete the namespace/resources if another environment uses that namespace.

## Proposal

Extend KAS and Rails to support templated generation of namespaces and related Kubernetes resources.

The feature should be available in ~"GitLab Premium" and ~"GitLab Ultimate"

### Environment state model

For reference, here are the various states of an environment.

[![](https://mermaid.ink/img/pako:eNpNT8luwyAQ_ZXRnFqJ_IAPlWK4pJdGTdtDTQ4IxouEAWFQZcX59-I4qTqXmbfMdkHtDWGFrfU_ulcxwYeQDkrsG36A17cafITPwxl2u5eFR1KJFqif3rNzg-ueN299U0_JhwV4s-ZA5vxfO0YKKhKDvdY0TQy-KA7tzEBQsH4uIzc3v7kFWVr3iGarHrP4fU-5896BDEeKoxpMeeKyMhJTTyNJrEppqFXZJonSXYtV5eRPs9NYpZiJYfS567FqlZ0KysGU78SguqjGPzYo9-39A19_AWYaYnc?type=png)](https://mermaid-js.github.io/mermaid-live-editor/edit#pako:eNpNT8luwyAQ_ZXRnFqJ_IAPlWK4pJdGTdtDTQ4IxouEAWFQZcX59-I4qTqXmbfMdkHtDWGFrfU_ulcxwYeQDkrsG36A17cafITPwxl2u5eFR1KJFqif3rNzg-ueN299U0_JhwV4s-ZA5vxfO0YKKhKDvdY0TQy-KA7tzEBQsH4uIzc3v7kFWVr3iGarHrP4fU-5896BDEeKoxpMeeKyMhJTTyNJrEppqFXZJonSXYtV5eRPs9NYpZiJYfS567FqlZ0KysGU78SguqjGPzYo9-39A19_AWYaYnc)

### User flow - creation

Platform engineer configures and installs one or more Agents for Kubernetes.
The agent is set up with `ci_access` sharing to application projects.
Agent configuration project can have environment templates under
`.gitlab/agents/<agent-name>/environment_templates/<template name>.yaml`.

Resources management must be explicitly enabled in the agent configuration file to be used:

```yaml
ci_access:
  projects:
    - id: group2/project1
      resource_management:
        enabled: true
  groups:
    - id: group1
      resource_management:
        enabled: true
```

App dev engineer uses an agent in their CI job with the `environment.kubernetes.agent` syntax.
They can optionally specify a template to use with `environment.kubernetes.template: "<template name>"`.

Before the related job starts:

- Rails reaches out to KAS to retrieve the specified template. If no template was specified in CI job, Rails asks for the default template.
- KAS fetches the requested template from Gitaly. The default template is hard-coded in KAS.
  The default template can be overridden in the agent configuration project by having a template with the name `default`.
- Rails asks KAS to render the template, supplying all required variables. KAS returns the rendered template.
  KAS handles certain object kinds if they are specified in the `objects` list (see below).
- Rails makes an RPC to KAS to apply the template to the cluster.
  It specifies the rendered template, the agent id, and other parameters required to apply it.
- KAS ensures the requested resources exist.

Environment, project, and agent identifying information has to be in the names and keys to allow
multiple environments to share a namespace.
Hence, `{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}` is used as the default name (or part of it) of objects.

We have to be very conservative in what we put into the label's name since we want to allow as much growing room as
practical. We only have 63 characters to work with (prefix `agent.gitlab.com/` is not counted towards that limit).
We could use e.g. base32 encoding for integers but that would make it harder for humans to recognize/scan for the numbers.
See [Labels' syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set) for limits.

Labels to force-set on the created resources:

```yaml
agent.gitlab.com/env-{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}: "" # template
#                    012345678901234567890123456789012345678901234567890123456789123
agent.gitlab.com/env-012345678901234567891234-{{ .project.id }}-{{ .agent.id }}: "" # example of longest slug of 24 chars
agent.gitlab.com/id-{{ .agent.id }}: ""
agent.gitlab.com/project_id-{{ .project.id }}: ""
agent.gitlab.com/environment_slug-{{ .environment.slug }}: ""
```

See [Annotations' syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set) for limits.
Annotations to force-set on the created resources:

```yaml
agent.gitlab.com/env-{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}: "<JSON object>" # template
```

It's impractical to try to cram all the information that might be required into individual labels/annotations since the
keys must contain `{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}` for uniqueness - not enough space remains to work with.
Labels are not even an option since labels' values have strict charset limitations.
Instead, we use the same name for both the label and the annotation and put a proper JSON object into the annotation's
value.

JSON object in the annotation value is:

```json
{
  "environment_id": 123,
  "environment_name": "<GitLab environment name>",
  "environment_slug": "<GitLab environment slug>",
  "environment_url": "<environment URL>",
  "environment_page_url": "<GitLab environment page URL>",
  "environment_tier": "<GitLab environment tier>",
  "agent_id": 42,
  "agent_name": "<GitLab Agent name>",
  "agent_url": "https://gitlab.example.com/<user>/<project>/-/cluster_agents/<agent name>",
  "project_id": 4242,
  "project_slug": "project1",
  "project_path": "group1/group2/project1",
  "project_url": "https://gitlab.example.com/group1/group2/project1",
  "template_name": "<template name>"
}
```

Kubernetes Resource Model (KRM) -based approach.
Think "trimmed down Kustomize that only supports a limited list of object kinds and no layers".

Any number of objects of any supported kind can be specified, nothing is required.
Only certain fields support templating (depending on the object but labels, annotations, names in general).
Only the following objects (group+version+kind) are supported.

The user may provide any number or none of these objects.

#### `v1/Namespace`

User can create namespaces.

#### `v1/ServiceAccount`

User can create service accounts and configure them to use certain image pull secrets.

#### `rbac.authorization.k8s.io/v1/RoleBinding`

User can create role bindings for identities (service accounts and anything else).

#### `agent.gitlab.com/v1/ImagePullSecretGenerator`

User can create `Secret`s with image pull secrets.
This object kind is handled by KAS and the monolith.
Monolith sends credentials along with the rendered template when calling to KAS RPC.

#### `agent.gitlab.com/v1/CredentialsGenerator`

User can create `Secret`s with access tokens.
This object kind is handled by KAS and the monolith.
Monolith sends credentials along with the rendered template when calling to KAS RPC.

#### `agent.gitlab.com/v1/DefaultServiceAccount`

User can specify image pull secrets and other fields for the default `ServiceAccount`.
This object can be specified at most once.

#### Flux objects

Most Flux objects are supported too.

#### Template functions

A limited set of templating functions should be supported to augment the template variables
to match the desired pattern of e.g. a namespace and to comply with Kubernetes rules for
naming things (like RFC1123 for labels and other names).

The template functions supported follow the same name and syntax as Helm (with sprig)
if available here.

Currently supported functions:

- `lower <string>`: convert to lower case. `lower "HELLO"` -> `"hello"`
- `substr <start> <end> <string>`: take a substring from a string. `substr 0 5 "hello world"` -> `"hello"`
- `replace <to-replace> <replacement> <string>`: replace all occurences of a substring in the string. `replace "_" "-" "foo_bar"` -> `"foo-bar"`
- `trimPrefix <prefix> <string>`: remove the prefix from the string. `trimPrefix "-" "-hello"` -> `"hello"`
- `trimSuffix <suffix> <string>`: remove the suffix from the string. `trimSuffix "- "hello-"` -> `"hello"`

#### Example template

```yaml
objects:
  - apiVersion: v1
    kind: Namespace
    metadata:
      name: '{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}'
  - apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: {{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}
      namespace: '{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}'
    imagePullSecrets:
      - name: pull-{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}
  - apiVersion: agent.gitlab.com/v1
    kind: DefaultServiceAccount
    metadata:
      name: "" # must be empty or not specified.
      namespace: '{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}'
    imagePullSecrets:
      - name: pull-{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}
  - apiVersion: agent.gitlab.com/v1
    kind: ImagePullSecretGenerator
    metadata:
      name: pull-{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}
      namespace: '{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}'
    type: kubernetes.io/dockerconfigjson # 'kubernetes.io/dockercfg' or 'kubernetes.io/dockerconfigjson' (default)
    scopes:
      - read_registry
  - apiVersion: agent.gitlab.com/v1
    kind: CredentialsGenerator
    metadata:
      name: token-{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}
      namespace: '{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}'
    secrets: # generates a Secret with type=Opaque and two tokens in keys token1 and token2
      - key: token1
        scopes:
          - read_registry
      - key: token2
        scopes:
          - read_registry
          - write_registry
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: bind-{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}
      namespace: '{{ .environment.slug }}-{{ .project.id }}-{{ .agent.id }}'
    subjects:
      - kind: Group # this is the default value for impersonation == CI job
        apiGroup: rbac.authorization.k8s.io
        name: "gitlab:project_env:{{ .project.id }}:{{ .environment.slug }}"
      - kind: "" # this can be used for impersonation == static identity. Empty to let Kubernetes use the right kind. See https://github.com/kubernetes/api/blob/v0.31.3/rbac/v1/types.go#L84-L85.
        apiGroup: "<detect from username of static identity>" # ServiceAccount if username starts from 'system:serviceaccount:', 'User' otherwise.
        name: "<username of static identity"
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: admin

apply_resources: on_start
delete_resources: on_stop
```

### User flow - update/upsert

The resources are re-applied based on the `apply_resources` property:
  - `on_create`: when the environment is created
  - `on_start`: when the environment is started or restarted.

### User flow - deletion

The resources are deleted based on the `delete_resources` property:
  - `never`: GitLab will never delete these resources
  - `on_stop`: when the environment is stopped and no other environments reference the related resources
  - `on_delete`: when the environment is deleted and no other environments reference the related resources

Later we might extend the logic with
  - option to scale to zero and back, instead of purging

Provisioning an environment creates objects.
Rails persists group, version, kind, namespace, name of the provisioned objects that KAS returns.
Deletion RPC requires information about the provisioned objects so that they can be found and deleted.

When an object, that was created by a template, no longer has any environment-related labels
(i.e. doesn't belong to any environments anymore), it is deleted.
If the object is a Namespace, everything in it is deleted since it is considered to be part of the environment.

Deletion is done using
[foreground cascading deletion](https://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion).

If an object in a template uses a pre-existing namespace, that namespace is never deleted or altered in any way by
the feature.

### User flow - errors

Show errors on the environment page and/or agent activity page.
CI job with an environment links to the environment page.

### Default template

See [`default_template.yaml`](../internal/module/managed_resources/server/default_template.yaml).

### Migration from GitLab managed clusters

We need to be able to provide seamless migration from GitLab managed clusters to GitLab managed cluster resources
as some of the managed namespaces might serve production traffic.
Based on user calls, GitLab managed clusters are used often as a self-service mechanism for long-running
(production) environments.

Migrations are supported with the following templates.

For environment-level namespaces:

```yaml
objects:
  - apiVersion: v1
    kind: Namespace
    metadata:
      name: '{{ .project.slug }}-{{ .project.id }}-{{ .environment.slug }}'
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: bind-{{ .agent.id }}-{{ .project.id }}-{{ .environment.slug }}
      namespace: {{ .project.slug }}-{{ .project.id }}-{{ .environment.slug }}
    subjects:
      - kind: Group # this is the default value for impersonation == CI job
        apiGroup: rbac.authorization.k8s.io
        name: "gitlab:project_env:{{ .project.id }}:{{ .environment.slug }}"
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: admin
```

For project-level namespaces:

```yaml
objects:
  - apiVersion: v1
    kind: Namespace
    metadata:
      name: '{{ .project.slug }}-{{ .project.id }}'
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: bind-{{ .agent.id }}-{{ .project.id }}-{{ .environment.slug }}
      namespace: {{ .project.slug }}-{{ .project.id }}
    subjects:
      - kind: Group # this is the default value for impersonation == CI job
        apiGroup: rbac.authorization.k8s.io
        name: "gitlab:project_env:{{ .project.id }}:{{ .environment.slug }}"
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: admin
```

The user needs to install and configure the agent with `ci_access` and CI job impersonation.

### To be discussed

- Error handling: What happens when the "pre-job" Kubernetes-related calls fail?
  - Should it fail the job? Yes
  - What is the timeout for these "pre" tasks?
  - Does re-running the job re-runs the "pre" tasks too? Yes
- Should we make this functionality available outside of CI? Yes. We should make it bound to environment lifecycle events, not CI jobs.
- What values, functions are available for the template?

## Intended users

* [Priyanka (Platform Engineer)](https://handbook.gitlab.com/handbook/product/personas/#priyanka-platform-engineer)

## Feature Usage Metrics

Events:
- `agent_applied_cluster_resource` with attributes:
   - environment name
   - environment tier
   - `group+version+kind` list

Metrics:

- MAU creating/updating/removing resources in the past month - broken down by
   - pricing tier
   - environment tier
- Number of environments creating/updating resources in the past month - broken down by
   - pricing tier
   - environment tier
- Number of environments removing resources in the past month - broken down by
   - pricing tier
   - environment tier
- Number of `group+version+kind` resources created/updated in the past month - broken down by
   - pricing tier
   - environment tier
- Number of `group+version+kind` resources removed in the past month - broken down by
   - pricing tier
   - environment tier

## Does this feature require an audit event?

- we should re-use the `deployment_started` event when a deployment to a protected environment starts
- we should add audit events for `environment_stopped` and `environment_created` when protected environments are stopped or created

- [ ] separate issues opened

## References

- A [collection of requirements](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/db78cf56faa7bf29cd9366946240b8fa81625efd/doc/gitlab_managed_clusters.md) when we still remembered the issues of the GitLab managed resources from 2022
- Related documentation pages:
  1. [GitLab managed clusters](https://docs.gitlab.com/ee/user/project/clusters/gitlab_managed_clusters.html)
  1. [List of resources created by GitLab managed clusters](https://docs.gitlab.com/ee/user/project/clusters/cluster_access.html)
  1. [Related environment variables](https://docs.gitlab.com/ee/user/project/clusters/deploy_to_cluster.html#deployment-variables)
  1. [Auto Review Apps](https://docs.gitlab.com/ee/topics/autodevops/stages.html#auto-review-apps)
- [Environment CRD issue](https://gitlab.com/gitlab-org/gitlab/-/issues/352186) (closed)
