Cluster O&M operations

Introduction

info

OBClusterOperation is a feature that is available in ob-operator v2.2.2 and later.

In order to simplify the operation and maintenance of OceanBase clusters and keep the operation traces in short term, ob-operator provides the cluster O&M resource OBClusterOperation for you to perform cluster O&M operations.

By creating the OBClusterOperation resource, you can perform the following cluster O&M operations:

• AddZones: Add zones to the cluster.
• DeleteZones: Delete zones from the cluster.
• AdjustReplicas: Adjust the number of replicas of zones.
• Upgrade: Upgrade the version of oceanbase cluster.
• RestartOBServers: Restart the specific oceanbase servers.
• DeleteOBServers: Delete the specific oceanbase servers.
• ModifyOBServers: Modify the configuration of specific oceanbase servers, including cpu, memory, storage class, storage capacity, monitor deployment and NFS backup volume mount.
• SetParameters: Set the parameters of oceanbase cluster.

The OBClusterOperation resource is a custom resource with the following fields:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: <op-name>- # The name of the OBClusterOperation resource will be automatically generated by `kubectl create`.
  namespace: <namespace>
spec:
  obcluster: <obcluster-name> # The name of the OBCluster resource to be operated.
  type: <operation-type> # The type of the operation, including AddZones, DeleteZones, AdjustReplicas, Upgrade, RestartOBServers, DeleteOBServers, ModifyOBServers, SetParameters.
  force: <force> # Whether to force the operation, default is false.
  ttlDays: <ttlDays> # The number of days to keep the operation traces, default is 7.
  <configuration-for-operation>: # The configuration for the operation, which is different for different operation types. The field name is the same as the operation type while the first capital letter is replaced with lowercase letter. For example, the configuration field for AddZones operation is addZones.
    field1: value1
    field2: value2
    # ...

What needs to be noted is that only the specific configuration that matches the operation type will take effect. That is to say, if the operation type is AddZones, only the addZones field will take effect, and other specific configuration fields will be ignored.

The OBClusterOperation resource is a one-time resource, which means that the resource will be deleted automatically after the operation is completed. The operation traces will be kept for a period of time specified by the ttlDays field.

We recommend that you use the kubectl create command to create the OBClusterOperation resource to avoid applying resources with duplicated name, which can automatically generate the name of the resource with the generateName field. For example,

kubectl create -f path/to/obclusteroperation.yaml

Operations

AddZones

The configuration for the AddZones operation is as follows:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: op-add-zones-
spec:
  obcluster: test
  type: AddZones
  addZones:
    - zone: zone2
      replica: 1
    - zone: zone3
      replica: 1

DeleteZones

The configuration for the DeleteZones operation is as follows:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: op-delete-zones-
spec:
  obcluster: test
  type: DeleteZones
  deleteZones:
    - zone2

AdjustReplicas

The configuration for the AdjustReplicas operation is as follows:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: op-adjust-replicas-
spec:
  obcluster: test
  type: AdjustReplicas
  adjustReplicas:
    - zones: [zone1]
      to: 2

Upgrade

The configuration for the Upgrade operation is as follows:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: op-upgrade-
spec:
  obcluster: test
  type: AdjustReplicas
  upgrade:
    image: xxx/xxxxx

RestartOBServers

The configuration for the RestartOBServers operation is as follows:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: op-restart-observers-
spec:
  obcluster: test
  type: RestartOBServers
  restartOBServers:
    observers: # The servers to be restarted, default is empty.
      - observer-xxx-1
      - observer-xxx-5
    obzones: # The zones to which the servers belong, default is empty.
      - zone1
      - zone2
    all: false # Whether to restart all the servers in the cluster, default is false.

DeleteOBServers

The configuration for the DeleteOBServers operation is as follows:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: op-delete-observers-
spec:
  obcluster: test
  type: AdjustReplicas
  observers:
    - observer-xxx-1
    - observer-xxx-5

ModifyOBServers

note

ModifyOBServers operation will rolling replace the servers in the cluster one by one. The operation will be completed after all the servers are replaced. The next observer will be replaced only after the previous observer is successfully replaced.

The configuration for the ModifyOBServers operation is as follows:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: op-modify-observers-
spec:
  obcluster: test
  type: ModifyOBServers
  modifyOBServers:
    resource: # The resource configuration to be modified, default is empty.
      cpu: 3
      memory: 13Gi
    expandStorageSize: # The storage capacity to be expanded, default is empty.
      dataStorage: 100Gi
      logStorage: 50Gi
      redoLogStorage: 100Gi
    modifyStorageClass: # The storage class to be modified, default is empty.
      dataStorage: new-storage-class
      logStorage: new-storage-class
      redoLogStorage: new-storage-class
    addingMonitor: # The monitor to be added, default is empty.
      image: xxx/obagent:xxx
      resource:
        cpu: 1
        memory: 1Gi
    removeMonitor: true # Whether to remove the monitor, default is false.
    addingBackupVolume: # The backup volume to be added, default is empty.
      volume:
        name: backup
        nfs:
          server: 1.2.3.4
          path: /opt/nfs
          readOnly: false
    removeBackupVolume: true # Whether to remove the backup volume, default is false.

SetParameters

The configuration for the SetParameters operation is as follows:

apiVersion: oceanbase.oceanbase.com/v1alpha1
kind: OBClusterOperation
metadata:
  generateName: op-set-parameters-
spec:
  obcluster: test
  type: SetParameters
  setParameters: # The parameters to be set
    - name: __min_full_resource_pool_memory
      value: "3221225472"
    - name: enable_syslog_recycle
      value: "True"