Release notes

Release notes to version 1.5.0

• Upgrade Kubernetes to version 1.19.7
• Upgrade Helm to version 3.5.2
• Upgrade Velero to version 1.5.3 and use it for backup/restore cluster
• Add support for Kubespray (version 2.15) as the main method to deploy Kubernetes. This approach replaces the BOSH kubo-release.
• Experimental support for Windows workloads
• Fix memory issues on OpenDistro and upgrade OpenDistro to version 1.13.1
• Add support for different Network Plugins (e.g. Calico, Cilium)
• Add support for NTP servers on Kubernetes nodes and Jumpboxes
• Add support for external_cloud_provider in vSphere

Required steps:

• Changes in opscontrol tfvars:

    ```
    # renamed: 
    jumpbox_public_ip -> jumpbox_ip
    # added 
    jumpbox_network_cidr" (dmz will be used by default)
    kube_version
    vsphere_k8s_username
    vsphere_k8s_password
    control_plane_template_name
    control_plane_master_ips
    control_plane_master_cpu
    control_plane_master_ram
    control_plane_master_network_name
    control_plane_master_gateway_ip
    control_plane_master_network_cidr
    control_plane_worker_ips
    control_plane_worker_cpu
    control_plane_worker_ram
    control_plane_worker_disk
    # removed:
    jumpbox_private_ip
    ```

• Additional configuration is needed in common.json file (mostly moved from env.json):

    ```
    # new parameters
    {"name": "ntp_servers", "value": "[ntp.ubuntu.com, ntp.ubuntu.local]"},
    # moved form env.json
    {"name": "vcenter_datastore", "opscontrol_var": "vcenter_ds"},
    {"name": "vcenter_cluster", "opscontrol_var": "vcenter_cluster"},
    {"name": "vcenter_allow_unverified_ssl", "opscontrol_var": "vcenter_allow_unverified_ssl"},
    {"name": "vcenter_resource_pool", "value": "#####"},
    {"name": "nsx_password", "opscontrol_var": "nsx_password"},
    {"name": "nsx_user", "opscontrol_var": "nsx_user"},
    {"name": "nsx_host", "opscontrol_var": "nsx_address"},
    {"name": "nsx_allow_unverified_ssl", "opscontrol_var": "nsx_allow_unverified_ssl"},
    {"name": "nsx_ca", "opscontrol_var": "nsx_ca"},
    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "tier0_router_name", "opscontrol_var": "tier0_router_name"},
    {"name": "translated_snat_ip", "opscontrol_var": "translated_snat_ip"},
    {"name": "overlay_tz_name", "opscontrol_var": "overlay_tz_name"},
    {"name": "edge_cluster_name", "value": "#####"},
    {"name": "public_dns_ip", "value": "#####"}, 
    {"name": "dns_instance_private_ip", "value": "#####"},
    {"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},
    ```

• Additional configuration is needed in env.json file, e.g:

    ```
    # moved to common.json
    {"name": "vcenter_datastore", "opscontrol_var": "vcenter_ds"},
    {"name": "vcenter_cluster", "opscontrol_var": "vcenter_cluster"},
    {"name": "vcenter_allow_unverified_ssl", "opscontrol_var": "vcenter_allow_unverified_ssl"},
    {"name": "vcenter_resource_pool", "value": "#####"},
    {"name": "nsx_password", "opscontrol_var": "nsx_password"},
    {"name": "nsx_user", "opscontrol_var": "nsx_user"},
    {"name": "nsx_host", "opscontrol_var": "nsx_address"},
    {"name": "nsx_allow_unverified_ssl", "opscontrol_var": "nsx_allow_unverified_ssl"},
    {"name": "nsx_ca", "opscontrol_var": "nsx_ca"},
    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "tier0_router_name", "opscontrol_var": "tier0_router_name"},
    {"name": "translated_snat_ip", "opscontrol_var": "translated_snat_ip"},
    {"name": "overlay_tz_name", "opscontrol_var": "overlay_tz_name"},
    {"name": "edge_cluster_name", "value": "#####"},
    {"name": "public_dns_ip", "value": "#####"}, 
    {"name": "dns_instance_private_ip", "value": "#####"},
    {"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},
    ```

• Additional configuration is needed in k8s-deployment.json file, e.g:

    ```
    # added:
    {"name": "k8s_version", "value":"v1.19.7"},
    {"name": "vcenter_k8s_user", "opscontrol_var": "vcenter_k8s_user"},
    {"name": "vcenter_k8s_password", "opscontrol_var": "vcenter_k8s_password"},
    {"name": "use_external_cloud_provider", "value":"true"},
    {"name": "k8s_template_name", "value": "####"},
    {"name": "k8s_master_ips", "value": "####"},
    {"name": "k8s_master_cpu", "value": "8"},
    {"name": "k8s_master_ram", "value": "8096"},
    {"name": "k8s_master_network", "value": "k8s"},
    {"name": "k8s_master_network_cidr", "value": "####"},
    {"name": "k8s_master_gateway_ip", "value": "####"},
    {"name": "k8s_worker_ips", "value": "####"},
    {"name": "k8s_worker_cpu", "value": "8"},
    {"name": "k8s_worker_ram", "value": "8096"},
    {"name": "k8s_worker_disk", "value": "200"},
    {"name": "k8s_worker_network", "value": "k8s"},
    {"name": "k8s_worker_network_cidr", "value": "####"},
    {"name": "k8s_worker_gateway_ip", "value": "####"},
    // Configuration requires Windows parameters even if we put empty values
    {"name": "windows_worker_ips", "value": ""},
    {"name": "windows_template_name", "value": ""},
    {"name": "windows_admin_password", "value": ""},
    {"name": "windows_netmask", "value": ""},
    # removed:
    {"name": "k8s_masters", "value": "1"},
    {"name": "k8s_workers", "value": "2"},
    {"name": "k8s_masters_type", "value": "general_small"},
    {"name": "k8s_workers_type", "value": "storage_large"},
    {"name": "k8s_network_name", "value": "k8s"},
    {"name": "k8s_network_sg", "value": "k8s-sg"},
    # renamed:
    - {"name": "extensions_provider_directory", "value": "vsphere/env/cb-k8s-provider-deployment"},
    - {"name": "extensions_provider_properties", "value": "k8s-provider.properties"}
    + {"name": "extensions_terraform_directory", "value": "vsphere/env/cb-k8s-provider-deployment"},
    + {"name": "extensions_terraform_properties", "value": "k8s.tfvars"}
    ```

Important notes:

• This release supports the new method of Kubernetes deployment. Please note it does not remove the existing cluster deployed with kubo-release. This has to be done manually.
• Please make sure you have a backup of your data before running an update.

Release notes to version 1.4.2

• Add support for NSX-T policy API
• Upgrade to vSphere CPI v55
• Use terraform to handle SecurityGroups and LoadBalancers instead of bosh vm-extensions when Policy API is used
• Add custom Kubernetes ingress LoadBalancer ports to the configuration
• Remove https LB for Kubernetes and use a single one for both http and https

Required steps:

• Changes in opscontrol tfvars:

    ```
    # renamed:
    bosh_ip                -> bosh_private_ip 
    concourse_cert         -> concourse_ui_cert
    vsphere_bosh_datastore -> vsphere_bosh_datastore_name

    # added:
    nsxt_remote_auth - bool, indicates whether Terraform should use remote auth with NSX-T
    nsxt_policy_api - bool, indicates whether Terraform should use PolicyAPI or ManagerAPI with NSX-T

    # removed:
    control_plane_certificate_name
    ```

• Additional configuration is needed in env.json file, e.g:

    ```
    {"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
    {"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
    {"name": "cf_router_lb_app_profile_name", "value": "((cf_router_lb_app_profile_name))"},
    {"name": "cf_router_lb_client_ssl_profile_name", "value": "((cf_router_lb_client_ssl_profile_name))"},
    {"name": "cf_ssh_lb_app_profile_name", "value": "((cf_ssh_lb_app_profile_name))"},
    {"name": "k8s_lb_app_profile_name", "value": "((k8s_lb_app_profile_name))"},
    ```

Important notes:

• This release supports NSX-T PolicyAPI which works differently than Manager API and may not support all resources that were created with Manager API.
• Usage of extensions ops is required to use custom ports in LoadBalancer.
• Please make sure you have a backup of your data before running an update.

Release notes to version 1.4.1

• Upgrade Kubernetes to version 1.17.9
• Upgrade Ubuntu Jumpbox and base image to version 20.04
• Upgrade Terraform to version 0.13
• Upgrade Concourse deployment process to handle docker pull request limit
• Add support for optional CloudFoundry
• Add support for extensions to Terraform
• Minor fixes

Required steps:

• Additional configuration is needed in env.json file, e.g.:

    ```
    {"name": "enable_cf", "value": "((enable_cf))"},
    {"name": "extensions_terraform_directory",  "value": "vsphere/env/cb-env-deployment"},
    {"name": "extensions_terraform_properties", "value": "terraform.tfvars"},
    ```

• There is a need to update terraform state before the upgrade, e.g for vSphere:

    ```
    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/vsphere registry.terraform.io/hashicorp/vsphere
    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/nsxt registry.terraform.io/vmware/nsxt
    terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/template registry.terraform.io/hashicorp/template
    ```

Important notes:

• This release contains a major terraform version upgrade so there is a need for a manual update of terraform state file (see more details: https://www.terraform.io/upgrade-guides/0-13.html)
• This release contains a major Ubuntu version upgrade so you may experience some breakages in case you have used some deprecated functions.
• This release supports optional CloudFoundry. You can disable the CloudFoundry using the disable_cf flag in env.json. Please note that to remove unused CF LoadBalancers you have to do it manually.
• Please make sure you have a backup of your data before running an update.

Release notes to version 1.4.0

• Upgrade Kubernetes to version 1.16.8
• Upgrade Traefik to version 2.2.0
• Update vSphere terraform scripts
• Upgrade Elasticsearch to version 7.8.0 and Filebeat to version 7.2.1
• Add support for extensions to cloud config
• Minor fixes

Required steps:

• Additional configuration is needed in env.json file, e.g:

{"name": "extensions_cloud_config_directory", "value": "vsphere/env/cb-cloud-config"},
{"name": "extensions_cloud_config_properties","value": "cloud-config.properties"},

Important notes:

• This upgrade contains changes in vSphere terraform so there is no need for manual creation of LB Pools and NSGroups. Corresponding parts in configuration files can be removed.
• This upgrade contains major version update in Kubernetes. This version removes some old api versions. Please make sure you have switched to supported versions before running an upgrade. More details can be found here: https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/
• This upgrade contains major version update in Traefik which brings few architectural changes. Please note some of Ingress resources may need updates in order to work properly.
• Please make sure you have backup of your data before running an update.

Release notes to version 1.3.1

• Update Kubernetes to 1.15.7 version
• Fix running errand smoke-tests in parallel in Kubernetes cluster
• Add BBR backup for Concourse web VM
• Update Concourse stemcell to the one used by Bosh (621.51)
• Fix small issues

Important note:

• This upgrade contains major version upgrade in Kubernetes. In case of custom changes in the platform it may break an update.
• Please make sure you have backup of your data before running an update.

Release notes to version 1.3.0

• Update Bosh to 270.11.0 version and bosh-cli to 6.0.2
• Update Credhub to 2.5.9 version and credhub-cli to 2.6.2
• Update Concourse and fly to version 5.5.7
• Update AWS CPI to version 81
• Update vSphere CPI to version 53.0.5
• Add BBR to OpsControl Bosh Director
• Add storage_2xlarge vm_type to cloud config
• Fix short expiration-date on OpenDistro certificate

Important note:

• This upgrade contains major version upgrade in Bosh (which includes a Credhub update as well) and Concourse. In case of custom changes in the platform it may break an update.
• Please make sure you have backup of your data before running an update. Please note that in order to backup OpsControl Bosh Director data you have first add extension-ops which enables bbr on Director.

Release note to version 1.2.5

• Update OpenDistro to version 1.1.0
• Fix issue with static ip address for UAA LoadBalancer target in AWS.
• Insert license into .sh files

Required steps:

• Removal of OpenDistro masters Deployment in ControlPlane is required as newest version uses StatefulSet and update fails if deployment is in place.
• After upgrade is finished update opendistro_security configuration is required:

# On the OpenDistro master pod
cd plugins/opendistro_security/tools/
chmod +x securityadmin.sh

./securityadmin.sh -icl -nhnv \
   -cacert ../../../config/admin-root-ca.pem \
   -cert ../../../config/admin-crt.pem \
   -key ../../../config/admin-key.pem \
   -cd ../securityconfig/