# Shoot Control Plane Migration

## Motivation

Currently moving the control plane of a shoot cluster can only be done manually and requires deep knowledge of how exactly to transfer the resources and state from one seed to another. This can make it slow and prone to errors.

Automatic migration can be very useful in a couple of scenarios:
- Seed goes down and can't be repaired (fast enough or at all) and it's control planes need to be brought to another seed
- Seed needs to be changed, but this operation requires the recreation of the seed (e.g. turn a single-AZ seed into a multi-AZ seed)
- Seeds need to be rebalanced
- New seeds become available in a region closer to/in the region of the workers and the control plane should be moved there to improve latency
- Gardener ring, which is a self-supporting setup/underlay for a highly available (usually cross-region) Gardener deployment


## Goals

* Provide a mechanism to migrate the control plane of a shoot cluster from one seed to another
* The mechanism should support migration from a seed which is no longer reachable (Disaster Recovery)
* The shoot cluster nodes are preserved and continue to run the workload, but will talk to the new control plane after the migration completes
* Extension controllers implement a mechanism which allows them to store their state or to be restored from an already existing state on a different seed cluster.
* The already existing shoot reconciliation flow is reused for migration with minimum changes


## Terminology

__Source Seed__ is the seed which currently hosts the control plane of a Shoot Cluster

__Destination Seed__ is the seed to which the control plane is being migrated


## Resources and controller state which have to be migrated between two seeds:

**Note:** The following lists are just FYI and are meant to show the current resources which need to be moved to the __Destination Seed__

### Secrets

Gardener has preconfigured lists of needed secrets which are generated when a shoot is created and deployed in the seed. Following is a minimum set of secrets which must be migrated to the __Destination Seed__. Other secrets can be regenerated from them.

- ca
- ca-front-proxy
- static-token
- ca-kubelet
- ca-metrics-server
- etcd-encryption-secret
- kube-aggregator
- kube-apiserver-basic-auth
- kube-apiserver
- service-account-key
- ssh-keypair

### Custom Resources and state of extension controllers

Gardenlet deploys custom resources in the __Source Seed__ cluster during shoot reconciliation which are reconciled by extension controllers. The state of these controllers and any additional resources they create is independent of the gardenlet and must also be migrated to the __Destination Seed__. Following is a list of custom resources, and the state which is generated by them that has to be migrated.

- **BackupBucket**: nothing relevant for migration
- **BackupEntry**: nothing relevant for migration
- **ControlPlane**: nothing relevant for migration
- **DNSProvider**/DNSEntry: nothing relevant for migration
- **Extensions**: migration of state needs to be handled individually
- **Infrastructure**: terraform state
- **Network**: nothing relevant for migration
- **OperatingSystemConfig**: nothing relevant for migration
- **Worker**: Machine-Controller-Manager related objects: machineclasses, machinedeployments, machinesets, machines

This list depends on the currently installed extensions and can change in the future

## Proposal

### Custom Resource on the garden cluster

The Garden cluster has a new Custom Resource which is stored in the project namespace of the Shoot called `ShootState`. It contains all the required data described above so that the control plane can be recreated on the __Destination Seed__.

This data is separated into two sections. The first is generated by the gardenlet and then either used to generate new resources (e.g secrets) or is directly deployed to the Shoot's control plane on the __Destination Seed__.

The second is generated by the extension controllers in the seed.

```yaml
apiVersion: core.gardener.cloud/v1alpha1
kind: ShootState
metadata:
  name: my-shoot
  namespace: garden-core
  ownerReference:
    apiVersion: core.gardener.cloud/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Shoot
    name: my-shoot
    uid: ...
  finalizers:
  - gardener
gardenlet:
  secrets:
  - name: ca
    data:
      ca.crt: ...
      ca.key: ...
  - name: ssh-keypair
    data:
      id_rsa: ...
  - name:
...
extensions:
- kind: Infrastructure
  state: ... (Terraform state)
- kind: ControlPlane
  purpose: normal
  state: ... (Certificates generated by the extension)
- kind: Worker
  state: ... (Machine objects)
```

The state data is saved as a `runtime.RawExtension` type, which can be encoded/decoded by the corresponding extension controller.

There can be sensitive data in the `ShootState` which has to be hidden from the end-users. Hence, it will be recommended to provide an etcd encryption configuration to the Gardener API server in order to encrypt the `ShootState` resource.

#### Size limitations

There are limits on the size of the request bodies sent to the kubernetes API server when creating or updating resources: by default ETCD can only accept request bodies which do not exceed 1.5 MiB (this can be configured with the `--max-request-bytes` flag); the kubernetes API Server has a request body limit of 3 MiB which cannot be set from the outside (with a command line flag); the gRPC configuration used by the API server to talk to ETCD has a limit of 2 MiB per request body which cannot be configured from the outside; and `watch` requests have a 16 MiB limit on the buffer used to stream resources.

This means that if `ShootState` is bigger than 1.5 MiB, the ETCD max request bytes will have to be increased. However, there is still an upper limit of 2 MiB imposed by the gRPC configuration.

If `ShootState` exceeds this size limitation it must make use of configmap/secret references to store the state of extension controllers. This is an implementation detail of Gardener and can be done at a later time if necessary as extensions will not be affected.

Splitting the `ShootState` into multiple resources could have a positive benefit on performance as the Gardener API Server and Gardener Controller Manager would handle multiple small resources instead of one big resource.

### Gardener extensions changes

All extension controllers which require state migration must save their state in a new `status.state` field and act on an annotation `gardener.cloud/operation=restore` in the respective Custom Resources which should trigger a restoration operation instead of reconciliation. A restoration operation means that the extension has to restore its state in the Shoot's namespace on the __Destination Seed__ from the `status.state` field.

As an example: the `Infrastructure` resource must save the terraform state.

```
apiVersion: extensions.gardener.cloud/v1alpha1
kind: Infrastructure
metadata:
  name: infrastructure
  namespace: shoot--foo--bar
spec:
  type: azure
  region: eu-west-1
  secretRef:
    name: cloudprovider
    namespace: shoot--foo--bar
  providerConfig:
    apiVersion: azure.provider.extensions.gardener.cloud/v1alpha1
    kind: InfrastructureConfig
    resourceGroup:
      name: mygroup
    networks:
      vnet: # specify either 'name' or 'cidr'
      # name: my-vnet
        cidr: 10.250.0.0/16
      workers: 10.250.0.0/19
status:
  state: |
      {
          "version": 3,
          "terraform_version": "0.11.14",
          "serial": 2,
          "lineage": "3a1e2faa-e7b6-f5f0-5043-368dd8ea6c10",
          "modules": [
              {
              }
          ]
          ...
      }
```

Extensions which do not require state migration should set `status.state=nil` in their Custom Resources and trigger a normal reconciliation operation if the CR contains the `core.gardener.cloud/operation=restore` annotation.

Similar to the contract for the [reconcile operation](https://github.com/gardener/gardener/blob/master/docs/extensions/reconcile-trigger.md), the extension controller has to remove the `restore` annotation after the restoration operation has finished.

An additional annotation `gardener.cloud/operation=migrate` is added to the Custom Resources. It is used to tell the extension controllers in the __Source Seed__ that they must stop reconciling resources (in case they are requeued due to errors) and should perform cleanup activities in the Shoot's control plane. These cleanup activities involve removing the finalizers on Custom Resources and deleting them without actually deleting any infrastructure resources.

**Note:** The same size limitations from the previous section are relevant here as well.

### Shoot reconciliation flow changes

The only data which must be stored in the `ShootState` by the gardenlet is secrets (e.g ca for the API server). Therefore the `botanist.DeploySecrets` step is changed. It is split into two functions which take a list of secrets that have to be generated.
- `botanist.GenerateSecretState` Generates certificate authorities and other secrets which have to be persisted in the ShootState and must not be regenerated on the __Destination Seed__.
- `botanist.DeploySecrets` Takes secret data from the `ShootState`, generates new ones (e.g. client tls certificates from the saved certificate authorities) and deploys everything in the Shoot's control plane on the __Destination Seed__


### ShootState synchronization controller

The ShootState synchronization controller will become part of the gardenlet. It syncs the state of extension custom resources from the shoot namespace to the garden cluster and updates the corresponding `spec.extension.state` field in the `ShootState` resource. The controller can `watch` Custom Resources used by the extensions and update the `ShootState` only when changes occur.


### Migration workflow
1. Starting migration
    - Migration can only be started after a Shoot cluster has been successfully created so that the `status.seed` field in the `Shoot` resource has been set
    - The `Shoot` resource's field `spec.seedName="new-seed"` is edited to hold the name of the __Destination Seed__ and reconciliation is automatically triggered
    - The Garden Controller Manager checks if the equality between `spec.seedName` and `status.seed`, detects that they are different and triggers migration.
1. The Garden Controller Manager waits for the __Destination Seed__ to be ready
1. Shoot's API server is stopped
1. Backup the Shoot's ETCD.
1. Extension resources in the __Source Seed__ are annotated with `gardener.cloud/operation=migrate`
1. Scale Down the Shoot's control plane in the __Source Seed__.
1. The gardenlet in the __Destination Seed__ fetches the state of extension resources from the `ShootState` resource in the garden cluster.
1. Normal reconciliation flow is resumed in the __Destination Seed__. Extension resources are annotated with `gardener.cloud/operation=restore` to instruct the extension controllers to reconstruct their state.
1. The Shoot's namespace in __Source Seed__ is deleted.


### Leader Election and Control Plane Termination

During migration "split brain" scenario must be avoided. This means that a Shoot's control plane in the __Source Seed__ must be scaled down before it is scaled up in the __Destination Seed__.

**Note**: This section is still under discussion. The plan is to first implement the `ShootState` and modify the reconciliation flow and extensions accordingly. Additionally multiple scenarios need to be considered depending on the reachability to the Garden cluster from the __Source Seed__ components, the __Source Seed's__ API server from the Garden cluster and the __Source Seed's__ API server from the controllers running on the seed. **The initial implementation will only cover the case where everything is running**.

Extension controllers do not need leader election functionality because they only reconcile the extension resources if the reconcile operation annotation is specified in the resource. Since this is set only during a reconciliation triggered by the Garden Controller Manager it cannot happen during migration.

For other controllers in the controlplane (e.g MCM, etcd backup restore, kube apiserver) leader election has to be implemented. We plan to introduce gardenlet soon therefore the garden cluster is expected to be reachable from all seeds.

Another flag might be needed to tell the gardenlet in the __Destination Seed__ that the control plane in the __Source Seed__ has been scaled down and the Shoot reconciliation can begin.

Horizontal Pod Autoscalers also need to be considered and removed.

#### Garden cluster and __Source Seed__ are healthy and there are no network problems

If both the Garden cluster and __Source Seed__ cluster are healthy, the Garden Controller Manager (or gardenlet after checking the `spec.seedName`) can directly scale down the Shoot's control plane as part of the migration flow.

#### If components in the __Source Seed__ cannot reliably read who the leader is from the Garden cluster

Currently we have come up with two ideas to handle this case:

**DNS leader election:** A DNS TXT entry with TTL=60s and value seed='__Source Seed__' is used. The record is created and maintained by the Gardener Controller Manager (by using the DNS Controller Manager and its DNSEntry resource). When a control plane migration is detected, the Gardener Controller Manager changes the value of the DNS Entry to seed='__Destination Seed__' and waits for 2*TTL + 1 = 121 seconds to ensure that the change is propagated to all controllers in the old seed. We rely on the fact that DNS is highly available (100% for AWS Route53) and that control plane components in the __Source Seed__ can see the changes.

Control plane components have to be shut down even when there is no access to the __Source Seed's__ API server. To be able to do that a daemonset is deployed in each Seed cluster. When the daemonset in the __Source Seed__ sees that it is nolonger the leader (by checking the DNS record) and there is no connection to the __Source Seed's__ API server, the daemonset will kill the Shoot's control plane pods by directly talking to the Kubelet's API server. If the __Source Seed's__ API Server comes back up, then gardenlet should take care of scaling down the deployments and statefulsets in the Shoot's control plane. This could be problematic if the gardenlet is in a crashloop backoff or takes too much time to do the scaling.

As an alternative to the daemonset, a sidecar container can be added to each control plane component. The sidecar checks the DNS Entry to see if it is still the leader. If it is not, it shuts down the entire pod. This way we do not run the risk of deploymnets and statefulsets recreating the control plane pods after the seed's apiserver comes back up.

The problems with using DNS as leader election is caching. Additionally, not all DNS servers respect TTL settings.

**Using timestamps in the ETCD backup entries:** Once a Shoot is successfully created on the __Source Seed__ a timestamp is saved in the Cluster Resource and/or saved in the ETCD backup restore sidecar (either as an environment variable or additional configuration). The timestamps must not be modified afterwards and is used when the backup restore container writes data to the backup entry of the Shoot in the following way:
- If there is no timestamp in the backup entry, the current timestamp is uploaded.
- If there is a timestamp in the backup entry it is compared to the backup restore container's timestamp:
    1. If it is the same, nothing is done
    2. If it is older, it is replaced with the timestamp of the current backuprestore container
    3. If it is newer - the current backuprestore container does not have ownership of the backup entry

When case 3 happens it means that the shoot has been migrated and the backuprestore container in the  __Destination Seed__ has started using the Shoot's backup entry. The backuprestore on the __Source Seed__ should be configured to shut itself and the etcd container down once it sees the newer timestamp. Shutting down the etcd will cause the kubernetes control plane components to go into crashloop backoff and the MCM will not be able to do anything as it will not be able to list nodes (this has to be verified with MCM).

For this approach to work backups must be enabled for the Shoot which is migrated. Additionally, synchronization based on the timestamps in the backup entry depend on the frequency of backups made by the backuprestore container.
