Pre-upgrade Check

The system automatically checks a cluster before its upgrade. If the cluster does not meet the pre-upgrade check conditions, the upgrade cannot continue. To avoid risks, you can perform pre-upgrade check according to the check items and solutions described in this section.

Table 1 Check items

No.	Check Item	Description
1	Node Restrictions	Check whether the node is available. Check whether the node OS supports the upgrade. Check whether the node is marked with unexpected node pool labels. Check whether the Kubernetes node name is the same as the ECS name.
2	Upgrade Management	Check whether the target cluster is under upgrade management.
3	Add-ons	Check whether the add-on status is normal. Check whether the add-on support the target version.
4	Helm Charts	Check whether the current HelmRelease record contains discarded Kubernetes APIs that are not supported by the target cluster version. If yes, the Helm chart may be unavailable after the upgrade.
5	SSH Connectivity of Master Nodes	Check whether CCE can connect to your master nodes.
6	Node Pools	Check the node pool status.
7	Security Groups	Check whether the Protocol & Port of the worker node security groups is set to ICMP: All and whether the security group with the source IP address set to the master node security group is deleted.
8	Residual Nodes	Check whether nodes need to be migrated.
9	Discarded Kubernetes Resources	Check whether there are discarded resources in the clusters.
10	Compatibility Risks	Read the version compatibility differences and ensure that they are not affected. The patch upgrade does not involve version compatibility differences.
11	CCE Agent Versions	Check whether cce-agent on the current node is of the latest version.
12	Node CPU Usage	Check whether the CPU usage of the node exceeds 90%.
13	CRDs	Check whether the key CRD packageversions.version.cce.io of the cluster is deleted. Check whether the cluster key CRD network-attachment-definitions.k8s.cni.cncf.io is deleted.
14	Node Disks	Check whether the key data disks on the node meet the upgrade requirements. Check whether the /tmp directory has 500 MB available space.
15	Node DNS	Check whether the DNS configuration of the current node can resolve the OBS address. Check whether the current node can access the OBS address of the storage upgrade component package.
16	Node Key Directory File Permissions	Check whether the owner and owner group of the files in the /var/paas directory used by the CCE are both paas.
17	kubelet	Check whether the kubelet on the node is running properly.
18	Node Memory	Check whether the memory usage of the node exceeds 90%.
19	Node Clock Synchronization Server	Check whether the clock synchronization server ntpd or chronyd of the node is running properly.
20	Node OS	Check whether the OS kernel version of the node is supported by CCE.
21	Node CPUs	Check whether the number of CPUs on the master node is greater than 2.
22	Node Python Commands	Check whether the Python commands are available on a node.
23	Node Readiness	Check whether the nodes in the cluster are ready.
24	Node journald	Check whether journald of a node is normal.
25	containerd.sock	Check whether the containerd.sock file exists on the node. This file affects the startup of container runtime in the Euler OS.
26	Internal Errors	Before the upgrade, check whether an internal error occurs.
27	Node Mount Points	Check whether inaccessible mount points exist on the node.
28	Kubernetes Node Taints	Check whether the taint needed for cluster upgrade exists on the node.
29	Everest Restrictions	Check whether there are any compatibility restrictions on the current Everest add-on.
30	cce-hpa-controller Restrictions	Check whether the current cce-controller-hpa add-on has compatibility restrictions.
31	Enhanced CPU Policies	Check whether the current cluster version and the target version support enhanced CPU policy.
32	Health of Worker Node Components	Check whether the container runtime and network components on the worker nodes are healthy.
33	Health of Master Node Components	Check whether the Kubernetes, container runtime, and network components of the master nodes are healthy.
34	Memory Resource Limit of Kubernetes Components	Check whether the resources of Kubernetes components, such as etcd and kube-controller-manager, exceed the upper limit.
35	Discarded Kubernetes APIs	The system scans the audit logs of the past day to check whether the user calls the deprecated APIs of the target Kubernetes version. Note Due to the limited time range of audit logs, this check item is only an auxiliary method. APIs to be deprecated may have been used in the cluster, but their usage is not included in the audit logs of the past day. Check the API usage carefully.
36	NetworkManager	Check whether NetworkManager of a node is normal.
37	Node ID File	Check the ID file format.
38	Node Configuration Consistency	When you upgrade a cluster to v1.19 or later, the system checks whether the following configuration files have been modified on the backend:
39	Node Configuration File	Check whether the configuration files of key components exist on the node.
40	CoreDNS Configuration Consistency	Check whether the current CoreDNS key configuration Corefile is different from the Helm release record. The difference may be overwritten during the add-on upgrade, affecting domain name resolution in the cluster.
41	sudo	Check whether the sudo commands and sudo-related files of the node are working.
42	Key Node Commands	Whether some key commands that the node upgrade depends on are working
43	Mounting of a Sock File on a Node	Check whether the docker/containerd.sock file is directly mounted to the pods on a node. During an upgrade, Docker or containerd restarts and the sock file on the host changes, but the sock file mounted to pods does not change accordingly. As a result, your services cannot access Docker or containerd due to sock file inconsistency. After the pods are rebuilt, the sock file is mounted to the pods again, and the issue is resolved accordingly.
44	HTTPS Load Balancer Certificate Consistency	Check whether the certificate used by an HTTPS load balancer has been modified on ELB.
45	Node Mounting	Check whether the default mount directory and soft link on the node have been manually mounted or modified.
46	Login Permissions of User paas on a Node	Check whether user paas is allowed to log in to a node.
47	Private IPv4 Addresses of Load Balancers	Check whether the load balancer associated with a Service is allocated with a private IPv4 address.
48	Historical Upgrade Records	Check whether the source version of the cluster is earlier than v1.11 and the target version is later than v1.23.
49	CIDR Block of the Cluster Management Plane	Check whether the CIDR block of the cluster management plane is the same as that configured on the backbone network.
50	GPU Add-on	The GPU add-on is involved in the upgrade, which may affect the GPU driver installation during the creation of a GPU node.
51	Nodes' System Parameters	Check whether the default system parameter settings on your nodes are modified.
52	Residual Package Version Data	Check whether there are residual package version data in the current cluster.
53	Node Commands	Check whether the commands required for the upgrade are available on the node.
54	Node Swap	Check whether swap has been enabled on cluster nodes.
55	containerd Pod Restart Risks	Check whether the service pods running on a containerd node are restarted when containerd is upgraded.
56	Key GPU Add-on Parameters	Check whether the configuration of the CCE AI Suite add-on in a cluster has been intrusively modified. If so, upgrading the cluster may fail.
57	GPU Pod Rebuild Risks	Check whether GPU service pods are rebuilt in a cluster when kubelet is restarted during the upgrade of the cluster.
58	ELB Listener Access Control	Check whether the access control of the ELB listener has been configured for the Service in the current cluster using annotations and whether the configurations are correct.
59	Master Node Flavor	Check whether the flavor of the master nodes in the cluster is the same as the actual flavor of these nodes.
60	Subnet Quota of Master Nodes	Check whether the number of available IP addresses in the cluster subnet supports rolling upgrade.
61	Node Runtime	Check whether an alarm is generated when a cluster is upgraded to v1.27 or later. Do not use Docker in clusters of versions later than 1.27.
62	Node Pool Runtime	Check whether an alarm is generated when a cluster is upgraded to v1.27 or later. Do not use Docker in clusters of versions later than 1.27.
63	Number of Node Images	Check the number of images on your node. If there are more than 1000 images, it takes a long time for Docker to start, affecting the standard Docker output and functions such as Nginx.

last updated: 2024-11-08 14:16 UTC - commit: 7c478af3470092ad91013d02b08e29c6a61c275d