GKE Authentication and Authorization Between Cloud IAM and RBAC

By Ala Raddaoui | DevOps Architect

If you have been using Kubernetes for a while, you probably know that Kubernetes does not offer any built-in mechanism for defining and managing users. This means that Kubernetes does not store any reference to users or groups they belong to in its object store ETCD. This allows admins to integrate their organization identity service provider, and enables cloud providers to integrate their cloud identity service offering with the likes of Google Cloud Identity and Microsoft AD, thus permitting it to work with Kubernetes without having to recreate those users and manage them twice.

In this blog, we will go into the details of how users are created with Google Kubernetes Engine  (GKE), and we’ll discuss how Google Cloud IAM and RBAC play together to achieve a better authentication and authorization strategy for your cluster.

Some Background

Every request made to Kubernetes is an API call that goes through the RESTful interface provided by the Kubernetes API Server (kube-apiserver). This works the same internally when different Kubernetes components, such as the scheduler, kubelets, or kube-proxy, wish to change or retrieve cluster state from ETCD to communicate with each other. Their requests must go through the API server, which will then know how to handle it on their behalf.

The kube-apiserver performs three main consecutive tasks for each incoming request before modifying the cluster state. These tasks are: authentication, authorization and admission control, as shown below in the graph. In the next section, I will explain the notion of identity in Kubernetes, and how the API server tackles authentication and authorization, both from a general perspective and GKE perspective.

GKE Authentication image1
Kubernetes API Server Tasks

Identity in Kubernetes

There are two concepts of an identity in Kubernetes:

User Accounts: These accounts represent user personas and are global and unique across all namespaces. User accounts are not managed by Kubernetes, and they can only be referred to inside the cluster, meaning that there’s no possibility to create or delete them, and you can only grant or revoke their access to cluster resources. User accounts are typically created by a company’s identity provider.

Service Accounts: These are special types of accounts that are meant to represent the applications running inside the pods. Service accounts are namespaced, meaning each account must necessarily belong to a certain namespace, and they can have the same name as long as they are created in distinct namespaces. Service accounts are actual Kubernetes objects that are managed by the cluster, and they can be created and used as an identity for the pods running your application if it ever needs to interact with the API server. Every namespace comes with a “default” service account generated automatically when you create the new namespace. Later, when you deploy a pod, the credentials of the default service account of the namespace where you choose to run your pod will be mounted as a volume in the pod file system (unless you create a new service account and explicitly indicate that you want to use it as the identity for your pod). 

To create a service account: 

$ kubectl create serviceaccount mysa
serviceaccount/mysa created

Notice how a secret was created for the new service account:

$ kubectl get secrets
NAME                  TYPE                                     DATA   AGE
default-token-lqdkc  kubernetes.io/service-account-token 3      16h
mysa-token-r78xj     kubernetes.io/service-account-token 3      9m
$ kubectl get secrets mysa-token-r78xj   -o yaml
apiVersion: v1
data:
 ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURE.....
 namespace: ZGVmYXVsdA==
 token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklpSjkuD.....
kind: Secret

Since secrets are base64 encoded by default in Kubernetes, if you decode the secret token field, you can use that token to assume the service account identity and authenticate to the cluster:

$ TOKEN=$(kubectl get secret mysa-token-r78xj -o jsonpath='{.data.token}'| base64 --decode)
$ kubectl get pods --token $TOKEN
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:mysa" cannot list resource "pods" in API group "" in the namespace "default"

We get an error since we didn’t assign any permission to the service account yet, but you can see the request is authenticated as the service account “system:serviceaccount:default:mysa”.

Identity in GKE

GKE clusters are Kubernetes clusters that are managed by Google, and thus they have the same type of identities: user accounts or service accounts. 

Now, since user accounts are created and managed outside of Kubernetes, Google has done a great job integrating their Google Cloud IAM with GKE so that you can employ the same user to access Google Cloud resources as well as GKE cluster resources.

One thing that can be confusing to some is the distinction between Google Cloud service accounts and Kubernetes service accounts. With Google Cloud, you can use both Google accounts and Google Cloud service accounts to access GCP resources. Both of these accounts are considered external to Kubernetes, so they are regarded as Kubernetes user accounts when referred to inside the cluster.

In the same pattern as Kubernetes, Google accounts represent actual end users of the Google ecosystem. New users can be created with G Suite if you’re a G Suite customer, or Cloud Identity if you’re only using the identity service of Google and don’t need the full suite of G Suite services. Or, you can simply sign up for a Gmail account if you’re not an organization entity.

On the other hand, a Google Cloud service account is similar to a Kubernetes service account in the sense that it’s used to represent an application or a virtual machine (VM) running, in this case, in Google Cloud and not in Kubernetes. This gives your application the ability to talk to the Google Cloud API and access other Google Cloud resources such as buckets, databases and so on. To create a Google Cloud service account, refer to the Google documentation here

A common use case for employing Google Cloud service accounts with GKE is using it with your CI/CD system when it needs to deploy a part of your application in GKE and the rest in Google Compute Engine (GCE) or one of Google Cloud’s database offerings in an automated pipeline fashion.

Authentication (authn)

Authentication is simply the process of verifying the identity of who or what is issuing the request.

Now that I’ve covered the two types of identities in Kubernetes/GKE and shown how a user can be created, let’s delve into the authentication mechanism and what options we have available to authenticate in a vanilla Kubernetes cluster versus GKE.

Kubernetes Authentication

In the process of creating a Kubernetes cluster, you can choose to enable one or more Authenticator Modules from the supported methods of authentication. These modules range from using a static passwords or tokens file to Client Certificates to Bootstrap Tokens to JSON Web Tokens (JWT). It does not have to be restricted to one method versus the other— you can choose to enable many auth modules when you deploy the kube-apiserver and, as a result, the API server will verify the identity of the request against each enabled authenticator until one succeeds.

After, when a user runs a kubectl command, the Kubernetes CLI (kubectl) will formulate it into an HTTP request, and it will then decorate it with the client configuration that will be used by the API server to authenticate the request. Those client configs can be things like username and password if the cluster is configured to use HTTP basic authentication; a user x509 certificate if the cluster is configured to trust certificates issued from a certain certificate authority (CA); a bearer token that belongs to a user from a preconfigured file holding a set of records (each record follows this format: “tokens,users,uid,Groups”) representing a user, its token, its user ID and the groups they belong to (if any); or perhaps, in another case, a bearer token dynamically generated when you create a new service account; or finally, a JWT token generated by an OAuth2 provider configured as a trusted issuer by the API using the OpenID Connect protocol. Google and Microsoft Active Directory are two examples of OAuth2 providers.

GKE Authentication

In GKE, static passwords, x509 client certificates, service account bearer tokens and OAuth2 tokens are all supported. However, after GKE’s integration with Oauth, basic authentication and client certificate issuance are disabled by default for clusters created with GKE 1.12 and higher (based on the Center for Internet Security (CIS) benchmark recommendations). Basic auth is vulnerable to brute force attacks. In addition, password changes and new users require the restarting of the API server, which is not convenient and might induce downtime. On the other hand, certificates are also long-lived credentials and, if compromised, cluster credential rotation and the reissuance of new client certificates might take some time which can put your cluster in danger. That’s why we recommend leaving it disabled.

GKE Authentication image2
Basic Auth and Client Certificate Disabled by Default by GKE

GKE manages authentication with gcloud by using the OpenID connect method, which is a flavor of OAuth2 described by the Kubernetes documentation as “an extension of OAuth2 to return an additional field along with the access token called an ID Token. This token is a JSON Web Token (JWT) with well known fields, such as a user’s email, signed by the server.

Let’s look at an example of a kubeconfig file generated when you create a GKE cluster to understand how authentication is configured:

# create a GKE cluster
gcloud container clusters create authn-authz-cluster
# generate the kubeconfig file
gcloud container clusters get-credentials authn-authz-cluster

The second command will generate a kubeconfig containing the configuration to access your newly created cluster. This file will be stored under the path $HOME/.kube/config. Let’s give it a look:

$ kubectl config view
apiVersion: v1
clusters:
- cluster:
   certificate-authority-data: DATA+OMITTED
   server: https://35.225.0.29
 name: gke_mytests_us-central1-a_authn-authz-cluster
contexts:
- context:
   cluster: gke_mytests_us-central1-a_authn-authz-cluster
   user: gke_mytests_us-central1-a_authn-authz-cluster
 name: gke_mytests_us-central1-a_authn-authz-cluster
current-context: gke_mytests_us-central1-a_authn-authz-cluster
kind: Config
preferences: {}
users:
- name: gke_mytests_us-central1-a_authn-authz-cluster
 user:
   auth-provider:
     config:
       cmd-args: config config-helper --format=json
       cmd-path: /Users/arad/Desktop/google-cloud-sdk/bin/gcloud
       expiry-key: '{.credential.token_expiry}'
       token-key: '{.credential.access_token}'
   name: gcp

The config includes things like: how to reach out to the API server of your cluster (https://35.225.0.29), which contexts you can use to access your clusters, the current active context, and finally, a section for users with the config that allows you to retrieve the access token and it’s expiry date. Let’s take the cmd-path and cmd-args and execute it to get the value of the different tokens we receive from Google’s OAuth2 provider:

# cmd-path + cmd-args
/Users/arad/Desktop/google-cloud-sdk/bin/gcloud config config-helper --format=json
You will get a JSON object with a section for credentials containing three fields: access_token, id_token and token_expiry as shown below:
"credential": {
 "access_token": "ya29.a0Af...............2VzQ1w",
 "id_token": "ya29.a0AfH6SMDiEgEJ7iadS9xeOXFQq7iIUW4gKYUhGHb3g...
  ...
  ...",
 "token_expiry": "2020-06-12T06:51:49Z"
},

Since the id_token is used by the authenticator to identify the identity of the user (as described here), let’s decode that token and see what information it holds. “id_tokens” are JWT (JSON Web Token) tokens. To decode it, we will use an online JWT decoder. The decoded output is listed in the graph below:

GKE Authentication image4
Decoded Google Oauth2 JWT Token 

We can see that the decoded JWT token holds information about the issuer of the token (iss field), the email I am using for my identity (and that it has been verified), along with a signature at the end that the authenticator module in the API server can use to verify the validity of the token (basically making sure the issuer of the JWT token is who it says it is, and to ensure that the message wasn’t tampered with). To get a better understanding of the various fields in an id_token, look here.

If all the authenticators fail, the request is rejected with HTTP status code 401. Otherwise, the request is authenticated as the user specified in the email field, the authorization header containing the id_token is removed from the request, and user information is added to the request context.

Authorization (authz)

Now that we verified the identity of the request, the next logical step is to evaluate whether the requester has the necessary permission to perform the action. This is referred to as the authorization mechanism. 

Authorization in Kubernetes

When deploying a Kubernetes cluster, the API server can be configured to enable one or more authorization modules. The list includes:

  • ABAC (Attribute-Based Access Control): Policies are defined in a static file. 
  • RBAC (Role-Based Access Control): Policies can be added or removed dynamically using Kubernetes objects. Kubernetes RBAC is a built-in RBAC mechanism that lets you create and grant roles (sets of permissions) for any object or type of object within the cluster. 
  • Webhook: Policy decisions are delegated authorizations to an external HTTP(S) service. 
  • Node: Special-purpose to authorize API requests made by kubelets.

You can even choose to allow all or deny all requests. If you want to learn more about how you can configure the kube-apiserver for both authentication and authorization, check out the list of the flags you can pass in when you deploy the API server.

In addition, Kubernetes authorization can also work with existing organization-wide or cloud-provider-wide access control systems which, in the case of GKE, Kubernetes authorization is integrated with Cloud IAM to ensure a request issuer has permissions at either level to access Kubernetes cluster resources.

Authorization in GKE

In GKE, both ABAC and RBAC are authorization mode options, but starting from GKE 1.8+, ABAC (also referred to as Legacy Authorization) is disabled by default as recommended from the CIS GKE Benchmark, and RBAC is used to grant permissions to resources at the cluster and namespace level. 

GKE Authentication image3
Legacy Authorization Disabled by Default by GKE

In GKE, Cloud IAM and Kubernetes RBAC are integrated to authorize users to perform actions if they have sufficient permissions at either level. This means that overall permissions for a user are additive between Cloud IAM and RBAC.

Cloud IAM policies set on a GCP project aren’t granular to the level where you can grant users access only to a specific GKE cluster within that project. Also, Cloud IAM policies are similar to ClusterRolesBindings in the sense that you can’t restrict user access to a certain namespace, but the scope of the policy is cluster-wide, meaning it spans all the namespaces in the cluster (or, even more, all the namespaces in all the clusters in that project).

As a consequence, to configure granular access to Kubernetes resources at the cluster or namespace level, you have to assign the lowest permission for the user with Cloud IAM, and then, granularly manage which permission to grant on each cluster level using RBAC. You can further set RBAC and Cloud IAM policies on Google groups, instead of individual users, for those users who share the same responsibilities. Now that Google groups for GKE is a thing, this enables you to avoid having to change the cluster state and the project access permissions with every new user requiring access to the cluster. 

If you wish to use only Kubernetes RBAC as the single source of truth for granting and managing access to users on the cluster, GKE users need to know how to authenticate to the cluster. To make that happen, users or groups should be granted the “container.clusters.get” Cloud IAM permission in the project where the cluster is running, or the “container.clusterViewer” role, which includes that permission as well. The “container.clusters.get” permission will allow users to generate the kubeconfig file containing the configuration access to the GKE cluster, which you can use to talk to the API server employing the kubectl CLI. Finally, you can then create RBAC policies to allow them the needed access.

Wrap Up

In this blog, we covered the notion of identity in Kubernetes and GKE, and we discussed how Cloud IAM and RBAC can help you achieve your authentication and authorization strategy for your cluster.

In summary, since Cloud IAM for GKE only works at the project level, we recommend using separate projects for your operating environments: one cluster per environment, and a distinct project for each environment. Then, use the Principle of Least Privilege and the granularity that RBAC offer in your design consideration to make sure every user or application can only access the information and resources that are necessary for their legitimate purpose.

On Tuesday, August 4th, Next OnAir will be focused on Security. Tune in to learn more about detecting, investigating, and responding to online threats to help protect your business. Also, be sure to check out SADA’s Next OnAir post-session recaps on Thursdays. Here’s what’s coming up on August 6th:

Benny Louie

#SADASays With Benny Louie and Bryan Nicholson – Post-Keynote Commentary 

Join SADA’s Benny Louie, Business Development Manager, and Bryan Nicholson, Technical Account Manager, for their take on Next OnAir’s Week 4 Keynote: A Better, Safer Normal.

OA SethMoffitt

#SADASays With Seth Moffitt, Chris Hendrich and Adam Acevedo – Post-Session Commentary

Listen in as SADA’s Seth Moffitt, Solution Architect, Chris Hendrich, Sr. Consultant and Adam Acevedo, Cloud Engineer, discuss the Next OnAir session, Authentication for Anthos Clusters.

Bryan Nicholson

#SADASays With Bryan Nicholson – Post-Session Commentary 

Join Bryan Nicholson, Technical Account Manager at SADA, for his take on the Next OnAir session, Getting Started With BeyondCorp: A Deeper Look into IAP.

Google Cloud Logo Icon

THE GCP VS AWS DEBATE

We spoke to dozens of customers who shared their experiences with both cloud providers. The overwhelming trends tell a big story. Download the eBook to learn more.

Solve not just for today but for what's next.

We'll help you harness the immense power of Google Cloud to solve your business challenge and transform the way you work.

Scroll to Top