This is a process no well documented in the official docs and I messed up in my CKA exam:<\/p>\n\n\n\n
1- check config of etcd process. Maybe you will need some details for the restore process<\/p>\n\n\n\n
$ kubectl describe pod -n kube-system etcd-master\n...\n--name=master\n--initial-cluster=master=https:\/\/127.0.0.1:2380\n--initial-advertise-peer-urls=https:\/\/127.0.0.1:2380\n...<\/pre>\n\n\n\n2- Stop api-server if not running kubeadm<\/p>\n\n\n\n
$ service kube-apiserver stop<\/pre>\n\n\n\n3- Check help for all restore options. Keep in mind you will need (very likely) to provide certs for auth.<\/p>\n\n\n\n
$ ETCDTL_API=3 etcdctl snapshot restore -h<\/pre>\n\n\n\n4- Restore ETCD using a previous backup:<\/p>\n\n\n\n
$ ETCDTL_API=3 etcdctl --endpoints 127.0.0.1:2379 snapshot restore FILE \\\n--cacert xxx --cert xx --key xxx\n\n--data-dir \/NEW\/DIR \\<\/code>\n--initial-cluster-toker TOKEN \\ (token is any word) <\/code>\n\n--name master \\ <\/code>\n--initial-cluster=master=https:\/\/127.0.0.1:2380 \\ <\/code>\n--initial-advertise-peer-urls=https:\/\/127.0.0.1:2380 <\/code>\n\nUSE HTTPS!!!!<\/code><\/pre>\n\n\n\n5- Add new lines and update volume paths in ETCD config. If it is a static pod, check in \/etc\/kubernetes\/manifests in master node.<\/p>\n\n\n\n
--data-dir=\/NEW\/DIR\n--initial-cluster-token TOKEN\n\n++ volumeMounts\/volumes to new path \/NEW\/DIR !!!!<\/pre>\n\n\n\n6- Restart services if not running kubeadm<\/p>\n\n\n\n
$ systemctl daemon-reload\n$ service etcd restart\n$ service etcd kube-apiserver start<\/pre>\n\n\n\n7- Checks<\/p>\n\n\n\n
\/\/\/ if using kubeadm, docker instance for etcd should restart\n$ docker ps -a | grep -i etcd\n\n\/\/\/ check etcd is running showing members:\n$ ETCDCTL_API=3 etcdctl member list --cacert xxx --cert xx --key xxx<\/pre>\n\n\n\nSidecar -logging<\/h2>\n\n\n\n
Based on this doc<\/a>. You want to send some logs to stderr so you create a new container that takes those.<\/p>\n\n\n\n
Container with a sidecar:<\/p>\n\n\n\n
apiVersion: v1\nkind: Pod\nmetadata:\n name: counter\nspec:\n containers:\n - name: count\n image: busybox\nargs: <\/code>\n- \/bin\/sh <\/code>\n- -c <\/code>\n- > i=0; <\/code>\nwhile true; <\/code>\ndo <\/code>\necho \"$i: $(date)\" >> \/var\/log\/1.log; <\/code>\necho \"$(date) INFO $i\" >> \/var\/log\/2.log; i=$((i+1)); sleep 1; <\/code>\ndone <\/code>\nvolumeMounts: <\/code>\n- name: varlog <\/code>\nmountPath: \/var\/log<\/code>\n - name: sidecar-1 \n image: busybox \n args: [\/bin\/sh, -c, 'tail -n+1 -f \/var\/log\/1.log'] \n volumeMounts: \n name: varlog\n mountPath: \/var\/log\n volumes:\n name: varlog\n emptyDir: {}<\/pre>\n\n\n\nNow you can see the logs of “\/var\/log\/1.log” going via “sidecar-1”<\/p>\n\n\n\n
$ kubectl logs counter sidecar-1<\/pre>\n\n\n\nCPU\/Memory of a POD<\/h2>\n\n\n\n
Based on these links: link1<\/a> , link2<\/a>, link3<\/a><\/p>\n\n\n\n
If you want to use “kubectl top” you need to install “metrics-server”<\/p>\n\n\n\n
$ kubectl top pod --all-namespaces<\/pre>\n\n\n\nKeep in mind that “kubectl top” shows metrics for a given pod. That information is based on reports from cAdvisor, which collects real pods resource usage.<\/p>\n\n\n\n
And as per link3, “kubectl top” is not the same as running “top” inside the container.<\/p>\n\n\n\n
Node NotReady<\/h2>\n\n\n\n
Based on this link<\/a>:<\/p>\n\n\n\n
$ kubectl get nodes\n$ kubectl describe nodes XXX\n\n$ ssh node \n -> check for kubelet logs \n cat \/var\/log\/kubelet.log\n $ journalctl -u kubelet \/\/ systemctl status kubelet --> if a service<\/pre>\n\n\n\n<\/p>\n","protected":false},"excerpt":{"rendered":"
Restore ETCD This is a process no well documented in the official docs and I messed up in my CKA exam: 1- check config of etcd process. Maybe you will need some details for the restore process $ kubectl describe pod -n kube-system etcd-master … –name=master –initial-cluster=master=https:\/\/127.0.0.1:2380 –initial-advertise-peer-urls=https:\/\/127.0.0.1:2380 … 2- Stop api-server if not running … Continue reading “Kubernetes Troubleshooting I”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27],"tags":[],"class_list":["post-425","post","type-post","status-publish","format-standard","hentry","category-kubernetes"],"_links":{"self":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/425","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/comments?post=425"}],"version-history":[{"count":1,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/425\/revisions"}],"predecessor-version":[{"id":426,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/posts\/425\/revisions\/426"}],"wp:attachment":[{"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/media?parent=425"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/categories?post=425"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.thomarite.uk\/index.php\/wp-json\/wp\/v2\/tags?post=425"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}