Kubernetes Best Practices - Part 3 - Setting Resource Quotas in Kubernetes

Namespace quotas

In an ideal world, the container settings would be good enough to take care of everything. But the world is dark and terrible place sometimes. People can easily forget to set the resources, or a rogue team can set their requests and limits very high and take up more than their fair share of the cluster.

To prevent these scenarios, you can set up resource quotas and limit ranges.

Resource quotas

Kubernetes allows administrators to set quotas, in namespaces, as hard limits for resource usage. This has an additional effect, if you set a CPU request quota in a namespace, then all pods need to set a CPU request in their definition, otherwise they will not be scheduled.

After creating a namespace, you can lock them down using quotas. For example, if you have a production and development namespace, a common pattern is to put no quota on production and then put very strict quotas on the development namespace. This allows production to take all the resource that it needs in case of a spike in traffic and development is lockdown.

A quota for resources might look something like this.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: demo
spec:
  hard:
    requests.cpu: 500m
    requests.memory: 100Mib
    limits.cpu: 700m
    limits.memory: 500Mib

In this example, you can see that there are four sections. Let's go into each one.

Requested CPU is the maximum combined CPU requests that all containers in the namespace can have. So in this example, you can have 50 containers with 10m requests, five containers with 100m requests, or just one container with 500m's of requests.

As long as the total requested CPU in the namespace is less than 500m, we're good to go. request.memory is the maximum combined memory requests that all containers in the namespace can have.

So again, in the above example, you can have 50 containers with two MB requests, five containers or 20 MB requests, or just one container with 100 as long as the total requested memory in the namespace is less than 100 mebibytes.

Limits.cpu is a maximum combined speed limits that all the containers and namespace can have. It's just like requests.cpu, but for the limits.

And then finally, limits.memory is the maximum combined memory limits that all containers in the namespace can have. Again, just like the request,memory but for a limit instead.

Limit range

You can also create a limit range in your namespace. Unlike a quota, which looks at the whole namespace, a limit range enforces itself on individual containers. So this can help prevent people from creating super tiny or super large containers inside the namespace.

A limit range might look something like this.

apiVersion: v1
kind: LimitRange
metadata:
  name: demo
spec:
  limits:
- default:
    cpu: 600m
    memory: 100Mib
  defaultRequest:
    cpu: 100m
    memory: 50Mib
max: 
    cpu: 1000m
    memory: 200Mib
  min:
    cpu: 10m
    memory: 10Mib
  type: Container

So looking at the example, you can see again there's four sections. Let's go into each one.

The default section will set up the default limits for a container in the pod. If you set these values in the limit range, any containers that don't explicitly set these values themselves will get assigned the default values.

The default request section will set up the default requests for a container in a pod. Again, if you set these values in the limit range, any containers that don't explicitly set these themselves will get assigned these default values.

The max section will set up the maximum limits that a container in a pod can set. The default section cannot be set higher than this value, and the limits in a container cannot be higher as well. It's important to note that if this value is set and the default section is not set, the max value becomes the default value as well.

The min section will set up the minimum requests of a container and a pod can set. The default requests section cannot be lower than this, and requests that are on a container cannot be lower as well.

Again, it's important to note that if this value is set and the default requests section is not set, the minimum value becomes the default request as well.