# Building multi-node KAS environment using Omnibus

This document describes how to build multi-node KAS environment using Omnibus.
It is mainly for testing purpose.

[[_TOC_]]

## Overview

It creates total of five VM instances in GCP (Google Cloud Platform).
- Two nodes with GitLab monolith + KAS
- One database node
- One Redis node
- One Gitaly node

## Building the environment

### Create VM instances

Go to GCP and create total of five instances.
We can follow steps described in [Installing GitLab on Google Cloud Platform](https://docs.gitlab.com/ee/install/google_cloud_platform/).
In this document, we use Debian as an OS installed in the VM instances.
- `n2-standard-2` for machine type
- `50GB` for boot disk size

Next, let's create new firewall rule to allow all ports.
Go to "Create firewall rule" and set the following:
- `0.0.0.0/0` for Source IP ranges
- `tcp:0-65535` for Protocols and ports
- Ingress for Direction of traffic

After creating the firewall rule, update the VM instances to allow all ports.
Go to "Edit" and add the firewall rule to the "Network tags".
Now, all the VM instances can communicate with each other.

### Database node setup

Follow these steps to setup PostgreSQL instance: [Standalone PostgreSQL using the Linux package](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#standalone-postgresql-using-the-linux-package).

### Redis node setup

Follow [Standalone Redis using the Linux package](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#standalone-redis-using-the-linux-package)

### Gitaly node

Follow [Configure Gitaly](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitaly).

### GitLab monolith + KAS nodes setup

First, we follow [Configure GitLab Rails](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitlab-rails)
and install Omnibus package in the two nodes.

#### `gitlab.rb` configuration

Create `/etc/gitlab/gitlab.rb` file and copy the configurations below.
This is an example setup used for verifying KAS->KAS TLS communication.

You need to replace the host and password for the database and Redis:
- `gitlab_rails['db_host']`
- `gitlab_rails['db_password']`
- `gitlab_rails['redis_host']`
- `gitlab_rails['redis_password']`
- `gitlab_kas['api_secret_key']`
- `gitlab_kas['private_api_secret_key']`
- URLs from `gitlab.example.com` to your domain.

```ruby
external_url 'https://gitlab.example.com'

## PostgreSQL connection details
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '<Database node IP address>' # IP/hostname of database server
gitlab_rails['db_password'] = '<Database password>'

### GitLab Redis settings
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '<Redis node IP address>' # IP/hostname of Redis server
gitlab_rails['redis_password'] = '<Redis password>'

##! Settings used by the GitLab application
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'wss://gitlab.example.com/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://gitlab.example.com:8153'
gitlab_rails['gitlab_kas_external_k8s_proxy_url'] = 'https://gitlab.example.com/-/kubernetes-agent/k8s-proxy/'

##! Define to enable GitLab KAS
gitlab_kas_external_url "wss://gitlab.example.com/-/kubernetes-agent/"
gitlab_kas['enable'] = true

##! Shared secret used for authentication between KAS and GitLab
gitlab_kas['api_secret_key'] = '<32_bytes_long_base64_encoded_value>'

##! Shared secret used for authentication between different KAS instances in a multi-node setup
gitlab_kas['private_api_secret_key'] = '<32_bytes_long_base64_encoded_value>'

##! Listen configuration for GitLab KAS
gitlab_kas['listen_address'] = '0.0.0.0:8150'
gitlab_kas['internal_api_listen_address'] = '0.0.0.0:8153'
gitlab_kas['private_api_listen_address'] = '0.0.0.0:8155'
gitlab_kas['private_api_certificate_file'] = "/etc/gitlab/ssl/gitlab.example.com.crt" # Necessary for KAS->KAS TLS communication
gitlab_kas['private_api_key_file'] = "/etc/gitlab/ssl/gitlab.example.com.key" # Necessary for KAS->KAS TLS communication
gitlab_kas['log_level'] = 'debug'

##! Environment variables for GitLab KAS
gitlab_kas['env'] = {
  'SSL_CERT_DIR' => "/etc/gitlab/ssl",
  'OWN_PRIVATE_API_URL' => 'grpcs://<IP address of this instance>:8155', # Use `grpcs` for KAS->KAS TLS communication
  'OWN_PRIVATE_API_HOST' => 'gitlab.example.com',
  'GRPC_GO_LOG_VERBOSITY_LEVEL' => '99',
  'GRPC_GO_LOG_SEVERITY_LEVEL' => 'debug'
}

##! Redis settings for GitLab KAS
gitlab_kas['redis_host'] = '<Redis node IP address>'
gitlab_kas['redis_port'] = '6379'
gitlab_kas['redis_password'] = '<Redis password>'

# Gitaly
gitlab_rails['gitaly_token'] = 'gitaly-secret'
gitlab_shell['secret_token'] = 'secret-token'

git_data_dirs({
  'default' => { 'gitaly_address' => 'tcp://<IP address>' },
})

gitaly['enable'] = false
```

For more information, refer to configurations for Rails from [Configure GitLab Rails](https://docs.gitlab.com/ee/administration/reference_architectures/2k_users.html#configure-gitlab-rails)
and [KAS: Enable on multiple nodes](https://docs.gitlab.com/administration/clusters/kas/#turn-on-kas-on-multiple-nodes).

#### SSL Cert file

Omnibus will create SSL cert files using Let'sEncrypt automatically unless you add `letsencrypt['enable'] = false`.

So, after you set up your first Rails monolith + KAS instance, copy cert files from the first instance
and copy them to the second instance.

```shell
$ gcloud compute scp --zone "us-central1-a" --project "<YOUR-PROJECT>" \
--recurse your-1st-rails-node:/etc/gitlab/ssl ./
$ gcloud compute scp --zone "us-central1-a" --project "<YOUR-PROJECT>" \
--recurse ./ssl your-2nd-rails-node:/etc/gitlab/ssl
```

In the second instance, you can add these settings to disable Let'sEncrypt and use copied cert files.
This is necessary if you want to use the same domain for multiple instances.
Otherwise, Let'sEncrypt will fail to issue a certificate for the same domain.

```ruby
letsencrypt['enable'] = false
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.example.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.example.com.key"
```

## Troubleshooting

### Database connection

If your Gitlab Rails + KAS node fails to talk to the database, then check `/var/opt/gitlab/postgresql/data/pg_hba.conf`
in the database node.

The Gitlab Rails + KAS node IP address needs be allowed to communicate to the database.

### Check Gitaly connection

```shell
$ gitlab-rake gitlab:gitaly:check
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
```

### Check generated KAS config

The generated KAS config is located at `/var/opt/gitlab/gitlab-kas/gitlab-kas-config.yml` in the KAS node.

