Release notes

2021/11/12

Release 1.7.0

Release notes to version 1.7.0

  • Upgrade Kubernetes to version 1.21.5 (kubespray 2.17.0)
  • Upgrade Concourse to version 7.5.0
  • Upgrade Ansible to version 3.4.0
  • Move DNS out of BOSH
  • Add support for Kubernetes Service type LoadBalancer on NSX-T
  • Separate pipeline for upgrade VMs
  • Improved SSH configuration on Jumpboxes
  • AWS fixes required by kubespray
Required steps
  • New parameters in vSphere terraform:
# Name for DNS virtual machine template
# Default: 13
dns_template_name = "..."

# Variable to specify hardware version
vm_hardware_version = 18

# Filenames for key-pairs in sensitive-data bucket
sensitive_data_k8s_public_key_filename   = "k8s.pub"
sensitive_data_k8s_private_key_filename  = "k8s.key"
sensitive_data_dns_public_key_filename   = "dns.pub"
sensitive_data_dns_pprivate_key_filename = "dns.key"

# Flag that indicates whether OpsControl should use external cloud provider. 
# Default: true
use_external_cloud_provider = ((use_external_cloud_provider))
  • New parameters in AWS terraform:
# The name of the AMI
ami_name = "..."

# VM instance type that should be used for DNS
dns_instance_type = "..."

# The ID of the hosted Route53 zone to contain DNS records
hosted_zone_id = "..."

# Private IP address of DNS instance
dns_instance_private_ip = "..."

# Filenames for key-pairs in sensitive-data bucket
sensitive_data_k8s_public_key_filename   = "k8s.pub"
sensitive_data_k8s_private_key_filename  = "k8s.key"
sensitive_data_dns_public_key_filename   = "dns.pub"
sensitive_data_dns_pprivate_key_filename = "dns.key"

# URI for docker image repository
docker_image_repo = "..."
  • DNS deployment should be removed before upgrade as it may block some IP addresses, so run below from OpsControl jumpbox:
bosh delete-deployment -d dns
  • During the upgrade new Concourse version will be installed. But there are some manual steps required before that:
kubectl delete -n concourse sts concourse-postgresql
kubectl delete -n concourse service concourse-web
kubectl delete -n concourse service concourse-web-worker-gateway
  • New parameters in vSphere cb-config
# common.json:
    {"name": "k8s_private_key", "opscontrol_var": "k8s_private_key"},
    {"name": "k8s_public_key", "opscontrol_var": "k8s_public_key"},
    {"name": "dns_private_key", "opscontrol_var": "dns_private_key"},
    {"name": "dns_public_key", "opscontrol_var": "dns_public_key"},
    {"name": "vm_hardware_version", "opscontrol_var": "vm_hardware_version"},

# env.json:
    {"name": "k8s_lb_enabled", "value": "false"},
    {"name": "k8s_lb_cidr", "value": "10.92.1.128/26"},
    {"name": "k8s_lb_gateway", "value": "10.92.1.129"},
    {"name": "k8s_lb_allocation_start", "value": "10.92.1.170"},
    {"name": "k8s_lb_allocation_end", "value": "10.92.1.180"},

    {"name": "k8s_node_ports_enabled", "value": "false"},
    {"name": "k8s_node_ports_tcp", "value": "[\"30000-32767\"]"},
    {"name": "k8s_node_ports_udp", "value": "[\"30000-32767\"]"},
    {"name": "k8s_node_ports_whitelist", "value": "[\"100.64.112.0/24\"]"},

# k8s-deployment.json:
    {"name": "k8s_vm_hardware_version", "opscontrol_var": "vm_hardware_version"},
    {"name": "enable_lb_service", "value": "false"},
  • New parameters in AWS cb-config
# common.json:
    {"name": "dns_private_key", "opscontrol_var": "dns_private_key"},
    {"name": "dns_public_key", "opscontrol_var": "dns_public_key"},
    {"name": "k8s_private_key", "opscontrol_var": "k8s_private_key"},
    {"name": "k8s_public_key", "opscontrol_var": "k8s_public_key"},

    {"name": "hosted_zone_id", "opscontrol_var": "hosted_zone_id"},
    {"name": "ami_name", "opscontrol_var": "ami_name"},

# env.json:
-    {"name": "dns_private_master_ip", "value": "10.90.2.141"},
-    {"name": "dns_private_slave_ip", "value": "10.90.2.142"},
+    {"name": "dns_instance_private_ip", "value": "10.90.2.141"},
  • DNS deployment should be removed before upgrade as it may block some IP addresses, so run below from each Environment jumpbox:
bosh delete-deployment -d dns
Important notes
  • This upgrade contains new kubespray version with changed inventory hosts names. Please make sure to upgrade you extensions playbooks to match those from sample inventory: https://github.com/kubernetes-sigs/kubespray/blob/master/inventory/sample/inventory.ini
  • After changing kube-apiserver (if there is no Kubernetes version change) you have to manually reinit kubeadm from one of the master nodes:

    sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml phase control-plane all
  • Please make sure you have a backup of your data before running an update.
  • If you are going to change from in-tree to external cloud provider in OpsControl you have to manaully migrate the volumes to new CSI or remove them completly and recreate. In future Cloudboostr releases this will be handled automatically.
2021/09/17

Release 1.6.0

Release notes to version 1.6.0

  • Kubernetes upgrade to version 1.20.7
  • Concourse upgrade to version 6.7.2 and move from BOSH to Kubernetes
  • Add configurable ingress controller. New variable ingress_type added. Possible values are traefik, nginx or none
  • Add support for individual workers sizes (CPU, RAM and disk)
  • Add docker_image_repo parameter in OpsControl to overwrite default dockerhub.
  • Fix ELK issues with scheduling
  • Moved common terraform modules to external repository. This may require some manual changes in terraform state.
Required steps
  • Some parameters were added and removed in k8s-deployment.json config file:
-        {"name": "traefik_certificate_bucket", "value": ""},
-        {"name": "traefik_certificate_files", "value": ""},
+        {"name": "ingress_additional_files_bucket", "value": ""},
+        {"name": "ingress_additional_files", "value": ""},
+        {"name": "ingress_type", "value": "traefik"},
(...)
-        {"name": "windows_worker_ips", "value": ""},
+        {"name": "k8s_worker_is_windows", "value": "false"},
  • Terraform modules – in order to not recreate VMs you should manually update your terraform state:
terraform state mv module.nsxt_policy[0].module.instances.vsphere_virtual_machine.jumpbox module.nsxt_policy[0].module.instances.vsphere_virtual_machine.vm
  • Traefik ingress controller should be removed before upgrade as it blocks some HTTP ports on workers:
kubectl delete ds traefik-ingress-controller -n traefik-ingress
  • After new version is installed you have to manually remove Concourse deployed with BOSH:
bosh delete-deployment -d concourse
Important notes
  • This upgrade containse new Concourse distribution. So all custom pipelines have to be recreated.
  • New docker_image_repo property affects only OpsControl and can be overwriten for control-plane with extensions.
  • Please make sure you have a backup of your data before running an update.
2021/07/23

Release 1.5.1

Release notes to version 1.5.1

  • Better metrics handling in Kubernetes clusters
  • New variable docker_image_repo added to support custom docker repository
  • Use BOSH vSphere CPI version 62
  • Fix issues found in logging systems (ELK and Filebeat)
  • Fix issues with unattended-upgrades on vSphere VMs

2021/04/16

Release 1.5.0

Release notes to version 1.5.0

  • Upgrade Kubernetes to version 1.19.7
  • Upgrade Helm to version 3.5.2
  • Upgrade Velero to version 1.5.3 and use it for backup/restore cluster
  • Add support for Kubespray (version 2.15) as the main method to deploy Kubernetes. This approach replaces the BOSH kubo-release.
  • Experimental support for Windows workloads
  • Fix memory issues on OpenDistro and upgrade OpenDistro to version 1.13.1
  • Add support for different Network Plugins (e.g. Calico, Cilium)
  • Add support for NTP servers on Kubernetes nodes and Jumpboxes
  • Add support for external_cloud_provider in vSphere
Required steps:
  • Changes in opscontrol tfvars:
    # renamed: 
    jumpbox_public_ip -> jumpbox_ip
    # added 
    jumpbox_network_cidr" (dmz will be used by default)
    kube_version
    vsphere_k8s_username
    vsphere_k8s_password
    control_plane_template_name
    control_plane_master_ips
    control_plane_master_cpu
    control_plane_master_ram
    control_plane_master_network_name
    control_plane_master_gateway_ip
    control_plane_master_network_cidr
    control_plane_worker_ips
    control_plane_worker_cpu
    control_plane_worker_ram
    control_plane_worker_disk
    # removed:
    jumpbox_private_ip
  • Additional configuration is needed in common.json file (mostly moved from env.json):
    # new parameters
    {"name": "ntp_servers", "value": "[ntp.ubuntu.com, ntp.ubuntu.local]"},
    # moved form env.json
    {"name": "vcenter_datastore", "opscontrol_var": "vcenter_ds"},
    {"name": "vcenter_cluster", "opscontrol_var": "vcenter_cluster"},
    {"name": "vcenter_allow_unverified_ssl", "opscontrol_var": "vcenter_allow_unverified_ssl"},
    {"name": "vcenter_resource_pool", "value": "#####"},
    {"name": "nsx_password", "opscontrol_var": "nsx_password"},
    {"name": "nsx_user", "opscontrol_var": "nsx_user"},
    {"name": "nsx_host", "opscontrol_var": "nsx_address"},
    {"name": "nsx_allow_unverified_ssl", "opscontrol_var": "nsx_allow_unverified_ssl"},
    {"name": "nsx_ca", "opscontrol_var": "nsx_ca"},
    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "tier0_router_name", "opscontrol_var": "tier0_router_name"},
    {"name": "translated_snat_ip", "opscontrol_var": "translated_snat_ip"},
    {"name": "overlay_tz_name", "opscontrol_var": "overlay_tz_name"},
    {"name": "edge_cluster_name", "value": "#####"},
    {"name": "public_dns_ip", "value": "#####"}, 
    {"name": "dns_instance_private_ip", "value": "#####"},
    {"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},
  • Additional configuration is needed in env.json file, e.g:
    # moved to common.json
    {"name": "vcenter_datastore", "opscontrol_var": "vcenter_ds"},
    {"name": "vcenter_cluster", "opscontrol_var": "vcenter_cluster"},
    {"name": "vcenter_allow_unverified_ssl", "opscontrol_var": "vcenter_allow_unverified_ssl"},
    {"name": "vcenter_resource_pool", "value": "#####"},
    {"name": "nsx_password", "opscontrol_var": "nsx_password"},
    {"name": "nsx_user", "opscontrol_var": "nsx_user"},
    {"name": "nsx_host", "opscontrol_var": "nsx_address"},
    {"name": "nsx_allow_unverified_ssl", "opscontrol_var": "nsx_allow_unverified_ssl"},
    {"name": "nsx_ca", "opscontrol_var": "nsx_ca"},
    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "tier0_router_name", "opscontrol_var": "tier0_router_name"},
    {"name": "translated_snat_ip", "opscontrol_var": "translated_snat_ip"},
    {"name": "overlay_tz_name", "opscontrol_var": "overlay_tz_name"},
    {"name": "edge_cluster_name", "value": "#####"},
    {"name": "public_dns_ip", "value": "#####"}, 
    {"name": "dns_instance_private_ip", "value": "#####"},
    {"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},
  • Additional configuration is needed in k8s-deployment.json file, e.g:
    # added:
    {"name": "k8s_version", "value":"v1.19.7"},
    {"name": "vcenter_k8s_user", "opscontrol_var": "vcenter_k8s_user"},
    {"name": "vcenter_k8s_password", "opscontrol_var": "vcenter_k8s_password"},
    {"name": "use_external_cloud_provider", "value":"true"},
    {"name": "k8s_template_name", "value": "####"},
    {"name": "k8s_master_ips", "value": "####"},
    {"name": "k8s_master_cpu", "value": "8"},
    {"name": "k8s_master_ram", "value": "8096"},
    {"name": "k8s_master_network", "value": "k8s"},
    {"name": "k8s_master_network_cidr", "value": "####"},
    {"name": "k8s_master_gateway_ip", "value": "####"},
    {"name": "k8s_worker_ips", "value": "####"},
    {"name": "k8s_worker_cpu", "value": "8"},
    {"name": "k8s_worker_ram", "value": "8096"},
    {"name": "k8s_worker_disk", "value": "200"},
    {"name": "k8s_worker_network", "value": "k8s"},
    {"name": "k8s_worker_network_cidr", "value": "####"},
    {"name": "k8s_worker_gateway_ip", "value": "####"},
    // Configuration requires Windows parameters even if we put empty values
    {"name": "windows_worker_ips", "value": ""},
    {"name": "windows_template_name", "value": ""},
    {"name": "windows_admin_password", "value": ""},
    {"name": "windows_netmask", "value": ""},
    # removed:
    {"name": "k8s_masters", "value": "1"},
    {"name": "k8s_workers", "value": "2"},
    {"name": "k8s_masters_type", "value": "general_small"},
    {"name": "k8s_workers_type", "value": "storage_large"},
    {"name": "k8s_network_name", "value": "k8s"},
    {"name": "k8s_network_sg", "value": "k8s-sg"},
    # renamed:
    - {"name": "extensions_provider_directory", "value": "vsphere/env/cb-k8s-provider-deployment"},
    - {"name": "extensions_provider_properties", "value": "k8s-provider.properties"}
    + {"name": "extensions_terraform_directory", "value": "vsphere/env/cb-k8s-provider-deployment"},
    + {"name": "extensions_terraform_properties", "value": "k8s.tfvars"}
Important notes:
  • This release supports the new method of Kubernetes deployment. Please note it does not remove the existing cluster deployed with kubo-release. This has to be done manually.
  • Please make sure you have a backup of your data before running an update.
2021/03/22

Release 1.4.2

Release notes to version 1.4.2

  • Add support for NSX-T policy API
  • Upgrade to vSphere CPI v55
  • Use terraform to handle SecurityGroups and LoadBalancers instead of bosh vm-extensions when Policy API is used
  • Add custom Kubernetes ingress LoadBalancer ports to the configuration
  • Remove https LB for Kubernetes and use a single one for both http and https
Required steps:
  • Changes in opscontrol tfvars:
    # renamed:
    bosh_ip                -> bosh_private_ip 
    concourse_cert         -> concourse_ui_cert
    vsphere_bosh_datastore -> vsphere_bosh_datastore_name

    # added:
    nsxt_remote_auth - bool, indicates whether Terraform should use remote auth with NSX-T
    nsxt_policy_api - bool, indicates whether Terraform should use PolicyAPI or ManagerAPI with NSX-T

    # removed:
    control_plane_certificate_name
  • Additional configuration is needed in env.json file, e.g:
    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "cf_router_lb_app_profile_name", "value": "((cf_router_lb_app_profile_name))"},
    {"name": "cf_router_lb_client_ssl_profile_name", "value": "((cf_router_lb_client_ssl_profile_name))"},
    {"name": "cf_ssh_lb_app_profile_name", "value": "((cf_ssh_lb_app_profile_name))"},
    {"name": "k8s_lb_app_profile_name", "value": "((k8s_lb_app_profile_name))"},
Important notes:
  • This release supports NSX-T PolicyAPI which works differently than Manager API and may not support all resources that were created with Manager API.
  • Usage of extensions ops is required to use custom ports in LoadBalancer.
  • Please make sure you have a backup of your data before running an update.
2021/03/08

Release 1.4.1

Release notes to version 1.4.1

  • Upgrade Kubernetes to version 1.17.9
  • Upgrade Ubuntu Jumpbox and base image to version 20.04
  • Upgrade Terraform to version 0.13
  • Upgrade Concourse deployment process to handle docker pull request limit
  • Add support for optional CloudFoundry
  • Add support for extensions to Terraform
  • Minor fixes
Required steps:
  • Additional configuration is needed in env.json file, e.g.:
    {"name": "enable_cf", "value": "((enable_cf))"},
    {"name": "extensions_terraform_directory",  "value": "vsphere/env/cb-env-deployment"},
    {"name": "extensions_terraform_properties", "value": "terraform.tfvars"},
  • There is a need to update terraform state before the upgrade, e.g for vSphere:
    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/vsphere registry.terraform.io/hashicorp/vsphere
    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/nsxt registry.terraform.io/vmware/nsxt
    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/template registry.terraform.io/hashicorp/template
Important notes:
  • This release contains a major terraform version upgrade so there is a need for a manual update of terraform state file (see more details: https://www.terraform.io/upgrade-guides/0-13.html)
  • This release contains a major Ubuntu version upgrade so you may experience some breakages in case you have used some deprecated functions.
  • This release supports optional CloudFoundry. You can disable the CloudFoundry using the disable_cf flag in env.json. Please note that to remove unused CF LoadBalancers you have to do it manually.
  • Please make sure you have a backup of your data before running an update.
2020/11/04

Release 1.4.0

Release notes to version 1.4.0

  • Upgrade Kubernetes to version 1.16.8
  • Upgrade Traefik to version 2.2.0
  • Update vSphere terraform scripts
  • Upgrade Elasticsearch to version 7.8.0 and Filebeat to version 7.2.1
  • Add support for extensions to cloud config
  • Minor fixes
Required steps:
  • Additional configuration is needed in env.json file, e.g:
{"name": "extensions_cloud_config_directory", "value": "vsphere/env/cb-cloud-config"},
{"name": "extensions_cloud_config_properties","value": "cloud-config.properties"},
Important notes:
  • This upgrade contains changes in vSphere terraform so there is no need for manual creation of LB Pools and NSGroups. Corresponding parts in configuration files can be removed.
  • This upgrade contains major version update in Kubernetes. This version removes some old api versions. Please make sure you have switched to supported versions before running an upgrade. More details can be found here: https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/
  • This upgrade contains major version update in Traefik which brings few architectural changes. Please note some of Ingress resources may need updates in order to work properly.
  • Please make sure you have backup of your data before running an update.

2020/04/17

Release 1.3.1

Release notes to version 1.3.1

  • Update Kubernetes to 1.15.7 version
  • Fix running errand smoke-tests in parallel in Kubernetes cluster
  • Add BBR backup for Concourse web VM
  • Update Concourse stemcell to the one used by Bosh (621.51)
  • Fix small issues
Important note:
  • This upgrade contains major version upgrade in Kubernetes. In case of custom changes in the platform it may break an update.
  • Please make sure you have backup of your data before running an update.
2020/03/06

Release 1.3.0

Release notes to version 1.3.0

  • Update Bosh to 270.11.0 version and bosh-cli to 6.0.2
  • Update Credhub to 2.5.9 version and credhub-cli to 2.6.2
  • Update Concourse and fly to version 5.5.7
  • Update AWS CPI to version 81
  • Update vSphere CPI to version 53.0.5
  • Add BBR to OpsControl Bosh Director
  • Add storage_2xlarge vm_type to cloud config
  • Fix short expiration-date on OpenDistro certificate
Important note:
  • This upgrade contains major version upgrade in Bosh (which includes a Credhub update as well) and Concourse. In case of custom changes in the platform it may break an update.
  • Please make sure you have backup of your data before running an update. Please note that in order to backup OpsControl Bosh Director data you have first add extension-ops which enables bbr on Director.
2019/12/10

Release 1.2.5

Release note to version 1.2.5
  • Update OpenDistro to version 1.1.0
  • Fix issue with static ip address for UAA LoadBalancer target in AWS.
  • Insert license into .sh files
Required steps:
  • Removal of OpenDistro masters Deployment in ControlPlane is required as newest version uses StatefulSet and update fails if deployment is in place.
  • After upgrade is finished update opendistro_security configuration is required:
# On the OpenDistro master pod
cd plugins/opendistro_security/tools/
chmod +x securityadmin.sh

./securityadmin.sh -icl -nhnv \
   -cacert ../../../config/admin-root-ca.pem \
   -cert ../../../config/admin-crt.pem \
   -key ../../../config/admin-key.pem \
   -cd ../securityconfig/