Release notes

Release notes to version 2.0.0

• Upgrade Kubernetes to version 1.22.6 (kubespray 2.18.1)
• Use containerd as default container runtime in Kubernetes
• Upgrade Terraform to version 1.1.5
• Upgrade Velero to version 1.8.0 with multibackend and snapshots support
• Remove BOSH fully from Cloudboostr
• Fix OpenSearch service to aggregate logs
• Fix https redirections for OpsControl services
• Improve VMs OS upgrades

Required steps

• New efk_deployment_enabled parameter added to Opscontrol terraform.tfvars file. It allows to disable the ELK (OpenSearch) installation in Opscontrol. Default value is true.
• New velero_snapshot_volumes parameter in k8s-deployment.json which can be used to enable PersistentVolume snapshot feautre in Velero backup service.

Important notes

• In this version default container runtime in Kubernetes is changed to containerd. There is still an option to use dockershim and it requires usage of extensions. During upgrade process all pods should be moved from docker to containerd automatically but there might be some unexpected issues.
• There is a new Kubernetes version in this upgrade. There are some APIs removed, see more details here before running upgrade: https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/#api-changes
• After upgrade you may have to manually remove BOSH Director from OpsControl if you have not removed that yet after 2.0.0 upgrade.
• Please make sure you have a backup of your data before running an update.

Release notes to version 2.0.0

• Replace Credhub with Vault v1.9.2
• Replace UAA with Keycloak v15.0.2
• Replace Opendistro with Opensearch
• Move Prometheus/Grafana to Kubernetes
• Removal BOSH from Environment deployment
• Add multiple users support on jumpbox
• Add concourse extensions and log retention settings
• Add variable with CB version
• vSphere: Add LB active monitor in NSXT Policy API
• AWS: update LoadBalancers in Terraform

Required steps

• In case you have configured Kubernetes with OIDC authentication with UAA it will still work but it is already deprecated and you have to migrate to Keycloak. UAA will be completly removed in Cloudboostr 2.1.0 release
• New ansible_strategy parameter added to Opscontrol terraform.tfvars file. You can check possible values here: https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html
• Removed concourse_ui_certificate_name and grafana_certificate_name from terraform.tfvars
• New users parameter in config.json that can be used to add custom users to all jumpboxes (Opscontrol and all Environments). This is an array of object with two parameters:
  • name – string with username
  • ssh_key – base64 encoded public key. Note: public key should include in a comment valid user email address

# Example of config.json file with users
{
    "envs": [
         {
            "name": "test",
            "backend_type": "aws",
            "config_repo_url": "...",
            "config_repo_branch": "..."
        }
    ],
    "users": [
        {
            "name": "test",
            "ssh_key": "<base64_encoded_public_key>"
        }
    ]
}

• New pipeline available to update jumpbox users in env.json:

(...)
    {
      "name": "update_users",
      "file": "ci/pipelines/update-users.yml",
      "vars": [
        {"name": "timer_interval", "value": "24h"}
      ]
    },
(...)

• Removed bosh_ variables from env.json
• BOSH installation was removed from Environments. In order to fully remove BOSH director you have to delete it manually from Environment

bosh delete-env -n \
    --state /etc/bosh-state/state.json \
    --vars-store /etc/bosh-state/creds.yml \
    ~/configure_jumpbox_bosh_workspace/manifest.yml

• New ansible_strategy parameter in k8s-deployment.json which can be used to modify strategy for kubespray deployment. See above example from terraform.tfvars to see possible values. Note: you can use "opscontrol_var": "ansible_strategy" to reuse value from Opscontrol.

Important notes

• This upgrade is a major upgrade that replaces core components from Cloudboostr: Credhub and UAA with Vault and Keycloak. Please make sure to update to new toolset your custom scripts and extensions before upgrade.
• After changing kube-apiserver (if there is no Kubernetes version change) you have to manually reinit kubeadm from one of the master nodes:

sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml phase control-plane all

• Please make sure you have a backup of your data before running an update.
• This version removes completly BOSH from Environment, which affects CloudFoundry deployment. It should change in next Cloudboostr 2.1.0 release.
• If you are going to change from in-tree to external cloud provider in OpsControl you have to manaully migrate the volumes to new CSI or remove them completly and recreate.
• This upgrade does not include new Kubernetes version. It is still v1.21.5 (the same as in Cloudboostr v1.7.0).

Release notes to version 1.7.0

• Upgrade Kubernetes to version 1.21.5 (kubespray 2.17.0)
• Upgrade Concourse to version 7.5.0
• Upgrade Ansible to version 3.4.0
• Move DNS out of BOSH
• Add support for Kubernetes Service type LoadBalancer on NSX-T
• Separate pipeline for upgrade VMs
• Improved SSH configuration on Jumpboxes
• AWS fixes required by kubespray

Required steps

• New parameters in vSphere terraform:

# Name for DNS virtual machine template
# Default: 13
dns_template_name = "..."

# Variable to specify hardware version
vm_hardware_version = 18

# Filenames for key-pairs in sensitive-data bucket
sensitive_data_k8s_public_key_filename   = "k8s.pub"
sensitive_data_k8s_private_key_filename  = "k8s.key"
sensitive_data_dns_public_key_filename   = "dns.pub"
sensitive_data_dns_pprivate_key_filename = "dns.key"

# Flag that indicates whether OpsControl should use external cloud provider. 
# Default: true
use_external_cloud_provider = ((use_external_cloud_provider))

• New parameters in AWS terraform:

# The name of the AMI
ami_name = "..."

# VM instance type that should be used for DNS
dns_instance_type = "..."

# The ID of the hosted Route53 zone to contain DNS records
hosted_zone_id = "..."

# Private IP address of DNS instance
dns_instance_private_ip = "..."

# Filenames for key-pairs in sensitive-data bucket
sensitive_data_k8s_public_key_filename   = "k8s.pub"
sensitive_data_k8s_private_key_filename  = "k8s.key"
sensitive_data_dns_public_key_filename   = "dns.pub"
sensitive_data_dns_pprivate_key_filename = "dns.key"

# URI for docker image repository
docker_image_repo = "..."

• DNS deployment should be removed before upgrade as it may block some IP addresses, so run below from OpsControl jumpbox:

bosh delete-deployment -d dns

• During the upgrade new Concourse version will be installed. But there are some manual steps required before that:

kubectl delete -n concourse sts concourse-postgresql
kubectl delete -n concourse service concourse-web
kubectl delete -n concourse service concourse-web-worker-gateway

• New parameters in vSphere cb-config

# common.json:
    {"name": "k8s_private_key", "opscontrol_var": "k8s_private_key"},
    {"name": "k8s_public_key", "opscontrol_var": "k8s_public_key"},
    {"name": "dns_private_key", "opscontrol_var": "dns_private_key"},
    {"name": "dns_public_key", "opscontrol_var": "dns_public_key"},
    {"name": "vm_hardware_version", "opscontrol_var": "vm_hardware_version"},

# env.json:
    {"name": "k8s_lb_enabled", "value": "false"},
    {"name": "k8s_lb_cidr", "value": "10.92.1.128/26"},
    {"name": "k8s_lb_gateway", "value": "10.92.1.129"},
    {"name": "k8s_lb_allocation_start", "value": "10.92.1.170"},
    {"name": "k8s_lb_allocation_end", "value": "10.92.1.180"},

    {"name": "k8s_node_ports_enabled", "value": "false"},
    {"name": "k8s_node_ports_tcp", "value": "[\"30000-32767\"]"},
    {"name": "k8s_node_ports_udp", "value": "[\"30000-32767\"]"},
    {"name": "k8s_node_ports_whitelist", "value": "[\"100.64.112.0/24\"]"},

# k8s-deployment.json:
    {"name": "k8s_vm_hardware_version", "opscontrol_var": "vm_hardware_version"},
    {"name": "enable_lb_service", "value": "false"},

• New parameters in AWS cb-config

# common.json:
    {"name": "dns_private_key", "opscontrol_var": "dns_private_key"},
    {"name": "dns_public_key", "opscontrol_var": "dns_public_key"},
    {"name": "k8s_private_key", "opscontrol_var": "k8s_private_key"},
    {"name": "k8s_public_key", "opscontrol_var": "k8s_public_key"},

    {"name": "hosted_zone_id", "opscontrol_var": "hosted_zone_id"},
    {"name": "ami_name", "opscontrol_var": "ami_name"},

# env.json:
-    {"name": "dns_private_master_ip", "value": "10.90.2.141"},
-    {"name": "dns_private_slave_ip", "value": "10.90.2.142"},
+    {"name": "dns_instance_private_ip", "value": "10.90.2.141"},

• DNS deployment should be removed before upgrade as it may block some IP addresses, so run below from each Environment jumpbox:

bosh delete-deployment -d dns

Important notes

• This upgrade contains new kubespray version with changed inventory hosts names. Please make sure to upgrade you extensions playbooks to match those from sample inventory: https://github.com/kubernetes-sigs/kubespray/blob/master/inventory/sample/inventory.ini
• After changing kube-apiserver (if there is no Kubernetes version change) you have to manually reinit kubeadm from one of the master nodes:
  
  sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml phase control-plane all
  
• Please make sure you have a backup of your data before running an update.
• If you are going to change from in-tree to external cloud provider in OpsControl you have to manaully migrate the volumes to new CSI or remove them completly and recreate. In future Cloudboostr releases this will be handled automatically.

Release notes to version 1.6.0

• Kubernetes upgrade to version 1.20.7
• Concourse upgrade to version 6.7.2 and move from BOSH to Kubernetes
• Add configurable ingress controller. New variable ingress_type added. Possible values are traefik, nginx or none
• Add support for individual workers sizes (CPU, RAM and disk)
• Add docker_image_repo parameter in OpsControl to overwrite default dockerhub.
• Fix ELK issues with scheduling
• Moved common terraform modules to external repository. This may require some manual changes in terraform state.

Required steps

• Some parameters were added and removed in k8s-deployment.json config file:

-        {"name": "traefik_certificate_bucket", "value": ""},
-        {"name": "traefik_certificate_files", "value": ""},
+        {"name": "ingress_additional_files_bucket", "value": ""},
+        {"name": "ingress_additional_files", "value": ""},
+        {"name": "ingress_type", "value": "traefik"},
(...)
-        {"name": "windows_worker_ips", "value": ""},
+        {"name": "k8s_worker_is_windows", "value": "false"},

• Terraform modules – in order to not recreate VMs you should manually update your terraform state:

terraform state mv module.nsxt_policy[0].module.instances.vsphere_virtual_machine.jumpbox module.nsxt_policy[0].module.instances.vsphere_virtual_machine.vm

• Traefik ingress controller should be removed before upgrade as it blocks some HTTP ports on workers:

kubectl delete ds traefik-ingress-controller -n traefik-ingress

• After new version is installed you have to manually remove Concourse deployed with BOSH:

bosh delete-deployment -d concourse

Important notes

• This upgrade containse new Concourse distribution. So all custom pipelines have to be recreated.
• New docker_image_repo property affects only OpsControl and can be overwriten for control-plane with extensions.
• Please make sure you have a backup of your data before running an update.

Release notes to version 1.5.1

• Better metrics handling in Kubernetes clusters
• New variable docker_image_repo added to support custom docker repository
• Use BOSH vSphere CPI version 62
• Fix issues found in logging systems (ELK and Filebeat)
• Fix issues with unattended-upgrades on vSphere VMs

Release notes to version 1.5.0

• Upgrade Kubernetes to version 1.19.7
• Upgrade Helm to version 3.5.2
• Upgrade Velero to version 1.5.3 and use it for backup/restore cluster
• Add support for Kubespray (version 2.15) as the main method to deploy Kubernetes. This approach replaces the BOSH kubo-release.
• Experimental support for Windows workloads
• Fix memory issues on OpenDistro and upgrade OpenDistro to version 1.13.1
• Add support for different Network Plugins (e.g. Calico, Cilium)
• Add support for NTP servers on Kubernetes nodes and Jumpboxes
• Add support for external_cloud_provider in vSphere

Required steps:

• Changes in opscontrol tfvars:

    # renamed: 
    jumpbox_public_ip -> jumpbox_ip
    # added 
    jumpbox_network_cidr" (dmz will be used by default)
    kube_version
    vsphere_k8s_username
    vsphere_k8s_password
    control_plane_template_name
    control_plane_master_ips
    control_plane_master_cpu
    control_plane_master_ram
    control_plane_master_network_name
    control_plane_master_gateway_ip
    control_plane_master_network_cidr
    control_plane_worker_ips
    control_plane_worker_cpu
    control_plane_worker_ram
    control_plane_worker_disk
    # removed:
    jumpbox_private_ip

• Additional configuration is needed in common.json file (mostly moved from env.json):

    # new parameters
    {"name": "ntp_servers", "value": "[ntp.ubuntu.com, ntp.ubuntu.local]"},
    # moved form env.json
    {"name": "vcenter_datastore", "opscontrol_var": "vcenter_ds"},
    {"name": "vcenter_cluster", "opscontrol_var": "vcenter_cluster"},
    {"name": "vcenter_allow_unverified_ssl", "opscontrol_var": "vcenter_allow_unverified_ssl"},
    {"name": "vcenter_resource_pool", "value": "#####"},
    {"name": "nsx_password", "opscontrol_var": "nsx_password"},
    {"name": "nsx_user", "opscontrol_var": "nsx_user"},
    {"name": "nsx_host", "opscontrol_var": "nsx_address"},
    {"name": "nsx_allow_unverified_ssl", "opscontrol_var": "nsx_allow_unverified_ssl"},
    {"name": "nsx_ca", "opscontrol_var": "nsx_ca"},
    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "tier0_router_name", "opscontrol_var": "tier0_router_name"},
    {"name": "translated_snat_ip", "opscontrol_var": "translated_snat_ip"},
    {"name": "overlay_tz_name", "opscontrol_var": "overlay_tz_name"},
    {"name": "edge_cluster_name", "value": "#####"},
    {"name": "public_dns_ip", "value": "#####"}, 
    {"name": "dns_instance_private_ip", "value": "#####"},
    {"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},

• Additional configuration is needed in env.json file, e.g:

    # moved to common.json
    {"name": "vcenter_datastore", "opscontrol_var": "vcenter_ds"},
    {"name": "vcenter_cluster", "opscontrol_var": "vcenter_cluster"},
    {"name": "vcenter_allow_unverified_ssl", "opscontrol_var": "vcenter_allow_unverified_ssl"},
    {"name": "vcenter_resource_pool", "value": "#####"},
    {"name": "nsx_password", "opscontrol_var": "nsx_password"},
    {"name": "nsx_user", "opscontrol_var": "nsx_user"},
    {"name": "nsx_host", "opscontrol_var": "nsx_address"},
    {"name": "nsx_allow_unverified_ssl", "opscontrol_var": "nsx_allow_unverified_ssl"},
    {"name": "nsx_ca", "opscontrol_var": "nsx_ca"},
    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "tier0_router_name", "opscontrol_var": "tier0_router_name"},
    {"name": "translated_snat_ip", "opscontrol_var": "translated_snat_ip"},
    {"name": "overlay_tz_name", "opscontrol_var": "overlay_tz_name"},
    {"name": "edge_cluster_name", "value": "#####"},
    {"name": "public_dns_ip", "value": "#####"}, 
    {"name": "dns_instance_private_ip", "value": "#####"},
    {"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},

• Additional configuration is needed in k8s-deployment.json file, e.g:

    # added:
    {"name": "k8s_version", "value":"v1.19.7"},
    {"name": "vcenter_k8s_user", "opscontrol_var": "vcenter_k8s_user"},
    {"name": "vcenter_k8s_password", "opscontrol_var": "vcenter_k8s_password"},
    {"name": "use_external_cloud_provider", "value":"true"},
    {"name": "k8s_template_name", "value": "####"},
    {"name": "k8s_master_ips", "value": "####"},
    {"name": "k8s_master_cpu", "value": "8"},
    {"name": "k8s_master_ram", "value": "8096"},
    {"name": "k8s_master_network", "value": "k8s"},
    {"name": "k8s_master_network_cidr", "value": "####"},
    {"name": "k8s_master_gateway_ip", "value": "####"},
    {"name": "k8s_worker_ips", "value": "####"},
    {"name": "k8s_worker_cpu", "value": "8"},
    {"name": "k8s_worker_ram", "value": "8096"},
    {"name": "k8s_worker_disk", "value": "200"},
    {"name": "k8s_worker_network", "value": "k8s"},
    {"name": "k8s_worker_network_cidr", "value": "####"},
    {"name": "k8s_worker_gateway_ip", "value": "####"},
    // Configuration requires Windows parameters even if we put empty values
    {"name": "windows_worker_ips", "value": ""},
    {"name": "windows_template_name", "value": ""},
    {"name": "windows_admin_password", "value": ""},
    {"name": "windows_netmask", "value": ""},
    # removed:
    {"name": "k8s_masters", "value": "1"},
    {"name": "k8s_workers", "value": "2"},
    {"name": "k8s_masters_type", "value": "general_small"},
    {"name": "k8s_workers_type", "value": "storage_large"},
    {"name": "k8s_network_name", "value": "k8s"},
    {"name": "k8s_network_sg", "value": "k8s-sg"},
    # renamed:
    - {"name": "extensions_provider_directory", "value": "vsphere/env/cb-k8s-provider-deployment"},
    - {"name": "extensions_provider_properties", "value": "k8s-provider.properties"}
    + {"name": "extensions_terraform_directory", "value": "vsphere/env/cb-k8s-provider-deployment"},
    + {"name": "extensions_terraform_properties", "value": "k8s.tfvars"}

Important notes:

• This release supports the new method of Kubernetes deployment. Please note it does not remove the existing cluster deployed with kubo-release. This has to be done manually.
• Please make sure you have a backup of your data before running an update.

Release notes to version 1.4.2

• Add support for NSX-T policy API
• Upgrade to vSphere CPI v55
• Use terraform to handle SecurityGroups and LoadBalancers instead of bosh vm-extensions when Policy API is used
• Add custom Kubernetes ingress LoadBalancer ports to the configuration
• Remove https LB for Kubernetes and use a single one for both http and https

Required steps:

• Changes in opscontrol tfvars:

    # renamed:
    bosh_ip                -> bosh_private_ip 
    concourse_cert         -> concourse_ui_cert
    vsphere_bosh_datastore -> vsphere_bosh_datastore_name

    # added:
    nsxt_remote_auth - bool, indicates whether Terraform should use remote auth with NSX-T
    nsxt_policy_api - bool, indicates whether Terraform should use PolicyAPI or ManagerAPI with NSX-T

    # removed:
    control_plane_certificate_name

• Additional configuration is needed in env.json file, e.g:

    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "cf_router_lb_app_profile_name", "value": "((cf_router_lb_app_profile_name))"},
    {"name": "cf_router_lb_client_ssl_profile_name", "value": "((cf_router_lb_client_ssl_profile_name))"},
    {"name": "cf_ssh_lb_app_profile_name", "value": "((cf_ssh_lb_app_profile_name))"},
    {"name": "k8s_lb_app_profile_name", "value": "((k8s_lb_app_profile_name))"},

Important notes:

• This release supports NSX-T PolicyAPI which works differently than Manager API and may not support all resources that were created with Manager API.
• Usage of extensions ops is required to use custom ports in LoadBalancer.
• Please make sure you have a backup of your data before running an update.

Release notes to version 1.4.1

• Upgrade Kubernetes to version 1.17.9
• Upgrade Ubuntu Jumpbox and base image to version 20.04
• Upgrade Terraform to version 0.13
• Upgrade Concourse deployment process to handle docker pull request limit
• Add support for optional CloudFoundry
• Add support for extensions to Terraform
• Minor fixes

Required steps:

• Additional configuration is needed in env.json file, e.g.:

    {"name": "enable_cf", "value": "((enable_cf))"},
    {"name": "extensions_terraform_directory",  "value": "vsphere/env/cb-env-deployment"},
    {"name": "extensions_terraform_properties", "value": "terraform.tfvars"},

• There is a need to update terraform state before the upgrade, e.g for vSphere:

    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/vsphere registry.terraform.io/hashicorp/vsphere
    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/nsxt registry.terraform.io/vmware/nsxt
    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/template registry.terraform.io/hashicorp/template

Important notes:

• This release contains a major terraform version upgrade so there is a need for a manual update of terraform state file (see more details: https://www.terraform.io/upgrade-guides/0-13.html)
• This release contains a major Ubuntu version upgrade so you may experience some breakages in case you have used some deprecated functions.
• This release supports optional CloudFoundry. You can disable the CloudFoundry using the disable_cf flag in env.json. Please note that to remove unused CF LoadBalancers you have to do it manually.
• Please make sure you have a backup of your data before running an update.

Release notes to version 1.4.0

• Upgrade Kubernetes to version 1.16.8
• Upgrade Traefik to version 2.2.0
• Update vSphere terraform scripts
• Upgrade Elasticsearch to version 7.8.0 and Filebeat to version 7.2.1
• Add support for extensions to cloud config
• Minor fixes

Required steps:

• Additional configuration is needed in env.json file, e.g:

{"name": "extensions_cloud_config_directory", "value": "vsphere/env/cb-cloud-config"},
{"name": "extensions_cloud_config_properties","value": "cloud-config.properties"},

Important notes:

• This upgrade contains changes in vSphere terraform so there is no need for manual creation of LB Pools and NSGroups. Corresponding parts in configuration files can be removed.
• This upgrade contains major version update in Kubernetes. This version removes some old api versions. Please make sure you have switched to supported versions before running an upgrade. More details can be found here: https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/
• This upgrade contains major version update in Traefik which brings few architectural changes. Please note some of Ingress resources may need updates in order to work properly.
• Please make sure you have backup of your data before running an update.

Release notes to version 1.3.1

• Update Kubernetes to 1.15.7 version
• Fix running errand smoke-tests in parallel in Kubernetes cluster
• Add BBR backup for Concourse web VM
• Update Concourse stemcell to the one used by Bosh (621.51)
• Fix small issues

Important note:

• This upgrade contains major version upgrade in Kubernetes. In case of custom changes in the platform it may break an update.
• Please make sure you have backup of your data before running an update.

Release notes to version 1.3.0

• Update Bosh to 270.11.0 version and bosh-cli to 6.0.2
• Update Credhub to 2.5.9 version and credhub-cli to 2.6.2
• Update Concourse and fly to version 5.5.7
• Update AWS CPI to version 81
• Update vSphere CPI to version 53.0.5
• Add BBR to OpsControl Bosh Director
• Add storage_2xlarge vm_type to cloud config
• Fix short expiration-date on OpenDistro certificate

Important note:

• This upgrade contains major version upgrade in Bosh (which includes a Credhub update as well) and Concourse. In case of custom changes in the platform it may break an update.
• Please make sure you have backup of your data before running an update. Please note that in order to backup OpsControl Bosh Director data you have first add extension-ops which enables bbr on Director.

Release note to version 1.2.5

• Update OpenDistro to version 1.1.0
• Fix issue with static ip address for UAA LoadBalancer target in AWS.
• Insert license into .sh files

Required steps:

• Removal of OpenDistro masters Deployment in ControlPlane is required as newest version uses StatefulSet and update fails if deployment is in place.
• After upgrade is finished update opendistro_security configuration is required:

# On the OpenDistro master pod
cd plugins/opendistro_security/tools/
chmod +x securityadmin.sh

./securityadmin.sh -icl -nhnv \
   -cacert ../../../config/admin-root-ca.pem \
   -cert ../../../config/admin-crt.pem \
   -key ../../../config/admin-key.pem \
   -cd ../securityconfig/