About This Upgrade

If you installed using the standard cluster installation process, and the inventory file that was used is available, you can use upgrade playbooks to automate the OpenShift cluster upgrade process.

OpenShift Origin 3.10 involves signficant changes to the architecture of the cluster, which are discussed further in the OpenShift Origin 3.10 Release Notes. The following sections detail how these notable technical changes affect the upgrade process from OpenShift Origin 3.9.

Defining Node Groups and Host Mappings

Starting in OpenShift Origin 3.10, node configurations are now bootstrapped from the master. When the node boot and services are started, the node checks if a kubeconfig and other node configuration files exist before joining the cluster. If they do not, the node pulls the configuration from the master, then joins the cluster.

This process replaces administrators having to manually maintain the node configuration uniquely on each node host. Instead, the contents of a node host’s /etc/origin/node/node-config.yaml file are now provided by ConfigMaps sourced from the master.

To map which ConfigMap to use for which node host, node groups must be defined and then set for each host in the inventory file. These procedures are handled during the Preparing for an Automated Upgrade section.

Node ConfigMaps

The Configmaps for defining the node configurations must be available in the openshift-node project. ConfigMaps are also now the authoritative definition for node labels; the old openshift_node_labels value is effectively ignored.

The installer provides a playbook for creating the following default ConfigMaps:

  • node-config-master

  • node-config-infra

  • node-config-compute

The following ConfigMaps are also created, which label nodes into multiple roles:

  • node-config-all-in-one

  • node-config-master-infra

Changes should not be made to a node host’s /etc/origin/node/node-config.yaml file. They can be overwritten by the configuration defined in the ConfigMap used by the node.

Node Group Defintions

After updating the openshift-ansible package, you can view what the default set of node group definitions looks like in YAML format in the ~/openshift-ansible/roles/openshift_facts/defaults/main.yml file:

  - name: node-config-master
      - 'node-role.kubernetes.io/master=true'
    edits: []
  - name: node-config-infra
      - 'node-role.kubernetes.io/infra=true'
    edits: []
  - name: node-config-compute
      - 'node-role.kubernetes.io/compute=true'
    edits: []
  - name: node-config-master-infra
      - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true'
    edits: []
  - name: node-config-all-in-one
      - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true,node-role.kubernetes.io/compute=true'
    edits: []

If you do not set the openshift_node_groups variable in your inventory file’s [OSEv3:vars] group, the defaults defined above will be used. However, if you want to deviate from these defaults, you must define the entire openshift_node_groups structure (including all desired node groups) in your inventory file.

The openshift_node_groups value is not merged with the defaults, and the YAML definition must first be translated into a Python dictionary. You can then use the edits field to modify any node configuration variables as desired by specifying the key-value pairs.

See Master and Node Configuration Files for reference on configurable node variables.

For example, the following entry in an inventory file defines groups named node-config-master, node-config-infra, and node-config-compute, and edits the ConfigMap for node-config-compute to set kubeletArguments.pods-per-core to 20:

openshift_node_groups=[{'name': 'node-config-master', 'labels': ['node-role.kubernetes.io/master=true']}, {'name': 'node-config-infra', 'labels': ['node-role.kubernetes.io/infra=true',]}, {'name': 'node-config-compute', 'labels': ['node-role.kubernetes.io/compute=true'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]

Whenever the openshift_node_group.yml playbook is run, the changes defined in the edits field will update the related ConfigMap (node-config-compute in this example), which will ultimately affect the node’s configuration file on the host.

Upgrade Workflow

The 3.9 to 3.10 control plane upgrade performs the following steps for you:

  • A backup of all etcd data is taken for recovery purposes.

  • The API and controllers are updated from 3.9 to 3.10.

  • Internal data structures are updated to 3.10.

  • The default router, if one exists, is updated from 3.9 to 3.10.

  • The default registry, if one exists, is updated from 3.9 to 3.10.

  • The default image streams and InstantApp templates are updated.

The 3.9 to 3.10 node upgrade performs a rolling update of nodes, which:

  • Marks a subset of nodes unschedulable and drains them of pods.

  • Updates node components from 3.9 to 3.10.

  • Converts from local configuration to bootstrapped TLS and Config.

  • Converts SDN components from systemd services to DaemonSets.

  • Returns those nodes to service.

Automated upgrade playbooks are run via Ansible directly using the ansible-playbook command with an inventory file, similar to the advanced installation method. The same v3_10 upgrade playbooks can be used for either of the following scenarios:

  • Upgrading existing OpenShift Origin 3.9 clusters to 3.10

  • Upgrading existing OpenShift Origin 3.10 clusters to the latest asynchronous errata updates

Running Upgrade Playbooks

Ensure that you have the latest openshift-ansible code checked out:

# cd ~/openshift-ansible
# git pull https://github.com/openshift/openshift-ansible master

Then run one of the following upgrade playbooks utilizing the inventory file you used during the advanced installation. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location.

Upgrading OpenShift Origin Minor Versions

To upgrade to a new OpenShift Origin minor version run the following playbook:

# ansible-playbook \
    -i </path/to/inventory/file> \

Upgrading to OpenShift Origin z-Stream Releases

To upgrade an existing OpenShift Origin latest z-stream of a minor release (e.g., 3.10.z), run the following playbook:

# ansible-playbook \
    -i </path/to/inventory/file> \

After rebooting, continue to Verifying the Upgrade.