This website uses cookies to improve its user experience and provide personalized content for you. We use cookies for web analytics and advertising.
You can accept these cookies by clicking "OK" or go to Details in order to manage your cookies preferences more precisely. To learn more, check out our Privacy and Cookies Policy
Essential website cookies are necessary to provide you with services available through the website, autosave your settings and preferences, and to enhance the performance and security of the website - you have the right not to accept them through you web browser's settings, but your access to some functionality and areas of our website may be restricted.
Analytics cookies: (our own and third-party : Google, HotJar) – you can accept these cookies below:
Marketing cookies (third-party cookies: Hubspot, Facebook, LinkedIn) – you can accept these cookies below:
sensitive-data
bucket and set proper Opscontrol terraform variables sensitive_data_offline_root_ca_key_filename
, sensitive_data_offline_root_ca_crt_filename
and/or sensitive_data_offline_root_ca_chain_filename
. If there is a password set for the key you can pass it via sensitive_data_offline_root_ca_key_password
.extensions
directory and then set extensions_bucket_name
to empty string. Then e.g. extensions_terraform_directory
will be used to locate proper directory under extensions/
dir.terraform.tfvars
:
telemetry_subnet_cidr -> control_plane_subnet_cidr
telemetry_router_ip -> control_plane_router_ip
telemetry_dhcp_server_ip -> control_plane_dhcp_server_ip
telemetry_dhcp_server_range_start -> control_plane_dhcp_server_range_start
telemetry_dhcp_server_range_end -> control_plane_dhcp_server_range_end
efk_deployment_enabled -> elk_deployment_enabled
terraform.tfvars
:
dmz_reserved_ips
dmz_static_ips
mgmt_reserved_ips
telemetry_reserved_ips
telemetry_static_ips
common.json
config file:
{"name": "elk_deployment_enabled", "opscontrol_var": "elk_deployment_enabled"}
k8s-deployment.json
config file:
{"name": "delete_k8s_pv_on_destroy", "value": "false"}
{"name": "docker_image_repo", "opscontrol_var": "docker_image_repo"}
{"name": "k8s_packages_ansible_playbook_additional_arguments", "value": ""}
{"name": "filebeat_release_state", "value": "present"}
{"name": "nginx_ingress_release_state", "value": "absent"}
{"name": "traefik_ingress_release_state", "value": "present"}
{"name": "prometheus_release_state", "value": "present"}
{"name": "thanos_release_state", "value": "present"}
{"name": "velero_release_state", "value": "present"}
k8s-deployment.json
config file, that can be removed:
{"name": "ingress_additional_files_bucket", "value": "..."}
{"name": "ingress_additional_files", "value": "..."}
{"name": "ingress_type", "value": "..."}
${ingress_additional_files_bucket}/${ENV_NAME}.k8s.key -> ${VAULT_KV_PATH_EXTENSIONS}/${ENV_NAME}/k8s_key
${ingress_additional_files_bucket}/${ENV_NAME}.k8s.crt -> ${VAULT_KV_PATH_EXTENSIONS}/${ENV_NAME}/k8s_crt
extensions_directory
:
${ingress_additional_files_bucket}/nginx-override.yaml -> ${EXTENSIONS_BUCKET}/${EXTENSIONS_DIR}/packages/nginx-ingress/values.yml
${ingress_additional_files_bucket}/traefik-override.yaml -> ${EXTENSIONS_BUCKET}/${EXTENSIONS_DIR}/packages/traefik-ingress/values.yml
efk_deployment_enabled
parameter added to Opscontrol terraform.tfvars
file. It allows to disable the ELK (OpenSearch) installation in Opscontrol. Default value is true
.velero_snapshot_volumes
parameter in k8s-deployment.json
which can be used to enable PersistentVolume snapshot feautre in Velero backup service.skip_upgrade
– to skip update and run just a migration [true/false]disable_containerd_migration
– to skip migration and run just an upgrade [true/false]ansible_strategy
parameter added to Opscontrol terraform.tfvars
file. You can check possible values here: https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.htmlconcourse_ui_certificate_name
and grafana_certificate_name
from terraform.tfvars
users
parameter in config.json
that can be used to add custom users to all jumpboxes (Opscontrol and all Environments). This is an array of object with two parameters:name
– string with usernamessh_key
– base64 encoded public key. Note: public key should include in a comment valid user email address
# Example of config.json file with users
{
"envs": [
{
"name": "test",
"backend_type": "aws",
"config_repo_url": "...",
"config_repo_branch": "..."
}
],
"users": [
{
"name": "test",
"ssh_key": "<base64_encoded_public_key>"
}
]
}
env.json
:
(...)
{
"name": "update_users",
"file": "ci/pipelines/update-users.yml",
"vars": [
{"name": "timer_interval", "value": "24h"}
]
},
(...)
bosh_
variables from env.json
bosh delete-env -n \
--state /etc/bosh-state/state.json \
--vars-store /etc/bosh-state/creds.yml \
~/configure_jumpbox_bosh_workspace/manifest.yml
ansible_strategy
parameter in k8s-deployment.json
which can be used to modify strategy for kubespray deployment. See above example from terraform.tfvars to see possible values. Note: you can use "opscontrol_var": "ansible_strategy"
to reuse value from Opscontrol.sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml phase control-plane all
# Name for DNS virtual machine template
# Default: 13
dns_template_name = "..."
# Variable to specify hardware version
vm_hardware_version = 18
# Filenames for key-pairs in sensitive-data bucket
sensitive_data_k8s_public_key_filename = "k8s.pub"
sensitive_data_k8s_private_key_filename = "k8s.key"
sensitive_data_dns_public_key_filename = "dns.pub"
sensitive_data_dns_pprivate_key_filename = "dns.key"
# Flag that indicates whether OpsControl should use external cloud provider.
# Default: true
use_external_cloud_provider = ((use_external_cloud_provider))
# The name of the AMI
ami_name = "..."
# VM instance type that should be used for DNS
dns_instance_type = "..."
# The ID of the hosted Route53 zone to contain DNS records
hosted_zone_id = "..."
# Private IP address of DNS instance
dns_instance_private_ip = "..."
# Filenames for key-pairs in sensitive-data bucket
sensitive_data_k8s_public_key_filename = "k8s.pub"
sensitive_data_k8s_private_key_filename = "k8s.key"
sensitive_data_dns_public_key_filename = "dns.pub"
sensitive_data_dns_pprivate_key_filename = "dns.key"
# URI for docker image repository
docker_image_repo = "..."
bosh delete-deployment -d dns
kubectl delete -n concourse sts concourse-postgresql
kubectl delete -n concourse service concourse-web
kubectl delete -n concourse service concourse-web-worker-gateway
# common.json:
{"name": "k8s_private_key", "opscontrol_var": "k8s_private_key"},
{"name": "k8s_public_key", "opscontrol_var": "k8s_public_key"},
{"name": "dns_private_key", "opscontrol_var": "dns_private_key"},
{"name": "dns_public_key", "opscontrol_var": "dns_public_key"},
{"name": "vm_hardware_version", "opscontrol_var": "vm_hardware_version"},
# env.json:
{"name": "k8s_lb_enabled", "value": "false"},
{"name": "k8s_lb_cidr", "value": "10.92.1.128/26"},
{"name": "k8s_lb_gateway", "value": "10.92.1.129"},
{"name": "k8s_lb_allocation_start", "value": "10.92.1.170"},
{"name": "k8s_lb_allocation_end", "value": "10.92.1.180"},
{"name": "k8s_node_ports_enabled", "value": "false"},
{"name": "k8s_node_ports_tcp", "value": "[\"30000-32767\"]"},
{"name": "k8s_node_ports_udp", "value": "[\"30000-32767\"]"},
{"name": "k8s_node_ports_whitelist", "value": "[\"100.64.112.0/24\"]"},
# k8s-deployment.json:
{"name": "k8s_vm_hardware_version", "opscontrol_var": "vm_hardware_version"},
{"name": "enable_lb_service", "value": "false"},
# common.json:
{"name": "dns_private_key", "opscontrol_var": "dns_private_key"},
{"name": "dns_public_key", "opscontrol_var": "dns_public_key"},
{"name": "k8s_private_key", "opscontrol_var": "k8s_private_key"},
{"name": "k8s_public_key", "opscontrol_var": "k8s_public_key"},
{"name": "hosted_zone_id", "opscontrol_var": "hosted_zone_id"},
{"name": "ami_name", "opscontrol_var": "ami_name"},
# env.json:
- {"name": "dns_private_master_ip", "value": "10.90.2.141"},
- {"name": "dns_private_slave_ip", "value": "10.90.2.142"},
+ {"name": "dns_instance_private_ip", "value": "10.90.2.141"},
bosh delete-deployment -d dns
sudo kubeadm init --config /etc/kubernetes/kubeadm-config.yaml phase control-plane all
ingress_type
added. Possible values are traefik
, nginx
or none
docker_image_repo
parameter in OpsControl to overwrite default dockerhub.
- {"name": "traefik_certificate_bucket", "value": ""},
- {"name": "traefik_certificate_files", "value": ""},
+ {"name": "ingress_additional_files_bucket", "value": ""},
+ {"name": "ingress_additional_files", "value": ""},
+ {"name": "ingress_type", "value": "traefik"},
(...)
- {"name": "windows_worker_ips", "value": ""},
+ {"name": "k8s_worker_is_windows", "value": "false"},
terraform state mv module.nsxt_policy[0].module.instances.vsphere_virtual_machine.jumpbox module.nsxt_policy[0].module.instances.vsphere_virtual_machine.vm
kubectl delete ds traefik-ingress-controller -n traefik-ingress
bosh delete-deployment -d concourse
docker_image_repo
property affects only OpsControl and can be overwriten for control-plane with extensions.docker_image_repo
added to support custom docker repository
# renamed:
jumpbox_public_ip -> jumpbox_ip
# added
jumpbox_network_cidr" (dmz will be used by default)
kube_version
vsphere_k8s_username
vsphere_k8s_password
control_plane_template_name
control_plane_master_ips
control_plane_master_cpu
control_plane_master_ram
control_plane_master_network_name
control_plane_master_gateway_ip
control_plane_master_network_cidr
control_plane_worker_ips
control_plane_worker_cpu
control_plane_worker_ram
control_plane_worker_disk
# removed:
jumpbox_private_ip
# new parameters
{"name": "ntp_servers", "value": "[ntp.ubuntu.com, ntp.ubuntu.local]"},
# moved form env.json
{"name": "vcenter_datastore", "opscontrol_var": "vcenter_ds"},
{"name": "vcenter_cluster", "opscontrol_var": "vcenter_cluster"},
{"name": "vcenter_allow_unverified_ssl", "opscontrol_var": "vcenter_allow_unverified_ssl"},
{"name": "vcenter_resource_pool", "value": "#####"},
{"name": "nsx_password", "opscontrol_var": "nsx_password"},
{"name": "nsx_user", "opscontrol_var": "nsx_user"},
{"name": "nsx_host", "opscontrol_var": "nsx_address"},
{"name": "nsx_allow_unverified_ssl", "opscontrol_var": "nsx_allow_unverified_ssl"},
{"name": "nsx_ca", "opscontrol_var": "nsx_ca"},
{"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
{"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
{"name": "tier0_router_name", "opscontrol_var": "tier0_router_name"},
{"name": "translated_snat_ip", "opscontrol_var": "translated_snat_ip"},
{"name": "overlay_tz_name", "opscontrol_var": "overlay_tz_name"},
{"name": "edge_cluster_name", "value": "#####"},
{"name": "public_dns_ip", "value": "#####"},
{"name": "dns_instance_private_ip", "value": "#####"},
{"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},
# moved to common.json
{"name": "vcenter_datastore", "opscontrol_var": "vcenter_ds"},
{"name": "vcenter_cluster", "opscontrol_var": "vcenter_cluster"},
{"name": "vcenter_allow_unverified_ssl", "opscontrol_var": "vcenter_allow_unverified_ssl"},
{"name": "vcenter_resource_pool", "value": "#####"},
{"name": "nsx_password", "opscontrol_var": "nsx_password"},
{"name": "nsx_user", "opscontrol_var": "nsx_user"},
{"name": "nsx_host", "opscontrol_var": "nsx_address"},
{"name": "nsx_allow_unverified_ssl", "opscontrol_var": "nsx_allow_unverified_ssl"},
{"name": "nsx_ca", "opscontrol_var": "nsx_ca"},
{"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
{"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
{"name": "tier0_router_name", "opscontrol_var": "tier0_router_name"},
{"name": "translated_snat_ip", "opscontrol_var": "translated_snat_ip"},
{"name": "overlay_tz_name", "opscontrol_var": "overlay_tz_name"},
{"name": "edge_cluster_name", "value": "#####"},
{"name": "public_dns_ip", "value": "#####"},
{"name": "dns_instance_private_ip", "value": "#####"},
{"name": "jumpbox_public_key", "opscontrol_var": "jumpbox_public_key"},
# added:
{"name": "k8s_version", "value":"v1.19.7"},
{"name": "vcenter_k8s_user", "opscontrol_var": "vcenter_k8s_user"},
{"name": "vcenter_k8s_password", "opscontrol_var": "vcenter_k8s_password"},
{"name": "use_external_cloud_provider", "value":"true"},
{"name": "k8s_template_name", "value": "####"},
{"name": "k8s_master_ips", "value": "####"},
{"name": "k8s_master_cpu", "value": "8"},
{"name": "k8s_master_ram", "value": "8096"},
{"name": "k8s_master_network", "value": "k8s"},
{"name": "k8s_master_network_cidr", "value": "####"},
{"name": "k8s_master_gateway_ip", "value": "####"},
{"name": "k8s_worker_ips", "value": "####"},
{"name": "k8s_worker_cpu", "value": "8"},
{"name": "k8s_worker_ram", "value": "8096"},
{"name": "k8s_worker_disk", "value": "200"},
{"name": "k8s_worker_network", "value": "k8s"},
{"name": "k8s_worker_network_cidr", "value": "####"},
{"name": "k8s_worker_gateway_ip", "value": "####"},
// Configuration requires Windows parameters even if we put empty values
{"name": "windows_worker_ips", "value": ""},
{"name": "windows_template_name", "value": ""},
{"name": "windows_admin_password", "value": ""},
{"name": "windows_netmask", "value": ""},
# removed:
{"name": "k8s_masters", "value": "1"},
{"name": "k8s_workers", "value": "2"},
{"name": "k8s_masters_type", "value": "general_small"},
{"name": "k8s_workers_type", "value": "storage_large"},
{"name": "k8s_network_name", "value": "k8s"},
{"name": "k8s_network_sg", "value": "k8s-sg"},
# renamed:
- {"name": "extensions_provider_directory", "value": "vsphere/env/cb-k8s-provider-deployment"},
- {"name": "extensions_provider_properties", "value": "k8s-provider.properties"}
+ {"name": "extensions_terraform_directory", "value": "vsphere/env/cb-k8s-provider-deployment"},
+ {"name": "extensions_terraform_properties", "value": "k8s.tfvars"}
# renamed:
bosh_ip -> bosh_private_ip
concourse_cert -> concourse_ui_cert
vsphere_bosh_datastore -> vsphere_bosh_datastore_name
# added:
nsxt_remote_auth - bool, indicates whether Terraform should use remote auth with NSX-T
nsxt_policy_api - bool, indicates whether Terraform should use PolicyAPI or ManagerAPI with NSX-T
# removed:
control_plane_certificate_name
{"name": "nsx_remote_auth", "opscontrol_var": "nsx_remote_auth"},
{"name": "nsx_policy_api", "opscontrol_var": "nsx_policy_api"},
{"name": "cf_router_lb_app_profile_name", "value": "((cf_router_lb_app_profile_name))"},
{"name": "cf_router_lb_client_ssl_profile_name", "value": "((cf_router_lb_client_ssl_profile_name))"},
{"name": "cf_ssh_lb_app_profile_name", "value": "((cf_ssh_lb_app_profile_name))"},
{"name": "k8s_lb_app_profile_name", "value": "((k8s_lb_app_profile_name))"},
{"name": "enable_cf", "value": "((enable_cf))"},
{"name": "extensions_terraform_directory", "value": "vsphere/env/cb-env-deployment"},
{"name": "extensions_terraform_properties", "value": "terraform.tfvars"},
terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/vsphere registry.terraform.io/hashicorp/vsphere
terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/nsxt registry.terraform.io/vmware/nsxt
terraform state replace-provider -state terraform.tfstate registry.terraform.io/-/template registry.terraform.io/hashicorp/template
{"name": "extensions_cloud_config_directory", "value": "vsphere/env/cb-cloud-config"},
{"name": "extensions_cloud_config_properties","value": "cloud-config.properties"},
# On the OpenDistro master pod
cd plugins/opendistro_security/tools/
chmod +x securityadmin.sh
./securityadmin.sh -icl -nhnv \
-cacert ../../../config/admin-root-ca.pem \
-cert ../../../config/admin-crt.pem \
-key ../../../config/admin-key.pem \
-cd ../securityconfig/