How to clean up disk space for Azure Kubernetes Node?
I followed this tutorial to create a Kubernetes cluster on Azure to run build agents: http://www.chrisjohnson.io/2018/07/07/using-azure-kubernetes-service-aks-for-your-vsts-build-agents/
To recap what is there: a helm chart to do a deployment with a secret and a config map. For this deployment, I created a kubernetes cluster on Azure with all default settings and it is pulling an image from the docker hub with vsts build agent installed.
All was working fine, but recently pods started to be evicted pretty regularly, the message on them is:
Message: Pod The node was low on resource: [DiskPressure].
How can I fix this issue?
You upgrade the size of your main node disks with something like this.
Check what pods are taking up space. Is it logs? Is it cached data? is it swap? Every application is different so you will have to go case by case.
Set local ephemeral storage at the pod level for your workloads so that they don't go over. Pods using a lot will get evicted.
Use Persistent Volumes for your workloads, especially some that are not local and just reserved for your applications.