Kubernetes: Authentication

Providing access to Kubernetes with multiple teams of developers, handling new joiners or people leaving means that the rotation of user access can be a daunting task. Kubernetes provides a few methods from X509 Client Certs, Static Token Files, Static Password File, Service Account Tokens, OpenID Connect Tokens and Anonymous Access.

But choosing the right authentication method is a difficult task, and it may have severe security implications if implemented incorrectly. This article will demystify the available options and provide an implementation that is Open-Source, in use by us at YLD and our clients.

Lets start with what should be avoided as much as possible, Static Token and Static Password Files, why? No method of token revocation exists, as well as any other changes require the kube-apiserver to be restarted. So the question is, do you trust the team handling user creation and deletion to have access to restart and modify the kube-apiserver? Even the docs mention “Note that basic authentication is currently supported for convenience while we finish making the more secure modes described above easier to use”. So if you must use these, please, proceed with caution!

Service Account Tokens should be a familiar concept to Kubernetes administrators, as these tokens are normally mounted into Pods for in-cluster access to the kube-apiserver, but did you know you can use these tokens outside of the cluster! The documentations suggest that this should be used for long running jobs that connect to the kube-apiserver. We use Service Account Tokens to allow our CD pipeline to communicate with our Kubernetes Clusters on a per-namespace basis. But be warned, these Service Account Tokens are stored as secrets, so be careful when granting permissions to other Service Accounts or Users. This could also be an easy method of granting a small amount of users access to your Cluster and provides the ability to revoke access by just deleting a Service Account Token from Kubernetes’ Secrets.

If you are having to manage a large number of users (say a whole engineering organisation) you might find yourself buried in administration work, and that if you are being told people are joining and leaving, if not then the possibility of having dormant active tokens lying around for team members that have left is an ever increasing problem.

X509 certificates mostly share the same pros and cons as Service Account Tokens. However revocation of certificates require to throw away the whole certificate chain and start a fresh — this is handled by the — client-ca-file=somefile parameter on the kube-api-server. However X509 certs have a feature which has the added benefit of being able to specify multiple groups that a certificate belongs to. This has the excellent side effect of providing a mechanism for building more specific RBAC roles that our bound to different groups (be ready for another post going into detail around RBAC)!

The only other alternative at the moment in time, which we have researched and experimented with is OpenID Connect Tokens. This is a flavour of Oauth2 but extended with a ID Token, which is basically a JWT Token that is signed by the server. This allows us to have Group memberships, ability to have short lived sessions or expiry on the JWT (so that user access can be rotated with 0 code changed) this is because the Identity management can be handled by an external service such as LDAP, Github or any other providers. Sadly this doesn’t come with a friendly user interface by default, so we implemented something.

What did we implement?

kubernetes-auth. An authentication front-end to Kubernetes clusters, enabling users to log into a Kubernetes cluster through the configuration and use of Dex, OIDC and Kubernetes OIDC

This has been developed for developers in large teams, with lots of new joiners to provide an easy way to switch between environments / regions in non-federated deployments.

It also provides an easy method to switch out Dex Connectors, so when your team ends up moving from Github to Okta, you have a minimal set of changes to implement.

‍
Dex? Aka Kubernetes Authentication

Dex acts as an intermediary between Github authentication and Kubernetes acting as an identity provider. This gives us the flexibility to move to another backed (LDAP, SAML, etc.) at some point in the future.

At the moment user logins are federated by github teams. Each team then belongs to a namespace with view on everything in that namespace. As time progresses we might want to restrict / expand on this.

To Login a user will use the following flow, with sandbox being replaced by their environment of choice (levels of access will be handled):

• Navigate to http://k8s-auth.sandbox.yld.io
• Login to Github and authorise the YLD github application
• Follow the instructions and Copy the commands, which will modify your ~/.kube/config
• check access with kubectl get pods

How does kubernetes-auth work with Dex?

We recommend reading the Dex Documentation before continuing as it is required to be working correctly before kubernetes-auth can start.

In our example helm chart for kubernetes-auth and dex, we specifically use only the Github Connector and the PostgresSQL backend. This was the working combination at time of implementation, but we plan to extend the chart to make it configurable.

.
└── yldio
    └── team
        ├── platform
        └── software-engineering

It becomes possible to map the team platform with the Kubernetes RBAC ClusterRole cluster-admin to give anyone in the team platform cluster-wide access to the kubernetes cluster. As such if we gave the team software-engineering a Role to a specific namespace, any new members will have access to the Kubernetes cluster in that specific namespace.