What if you could provision a Kubernetes cluster across major cloud providers like Google Cloud Platform (GCP), Microsoft Azure or Amazon Web Services (AWS) through a resource like Deployment or a PersistentVolumeClaim (PVC) and manage it like you manage any other Kubernetes resource? That’s what you can do through Crossplane (among many other things).
Okay, what’s with the PersistentVolumeClaim (PVC) analogy? PersistentVolumeClaim requests a PersistentVolume (PV), which under the hood provisions a storage volume according to whatever kind of storage you specify in the StorageClass. More about PV and PVCs and the difference between PV & PVC’s. Let’s look at how you provision a kubernetes cluster using Crossplane.
Imagine a Kubernetes resource pair like PVC-PV but for provisioning a Kubernetes cluster instead of storage. That is, if I create a claim (like PVC) which says I want a cluster, it would provision a cluster resource (like PV) in Kubernetes which in turn would create a cluster (actual cluster, in as in a GKE cluster) in whichever cloud provider I want based on the storage class, except in this case this storage class would hold the information about the cloud provider where I want to provision the cluster (and it’s called a Provider instead of StorageClass). This is roughly what Crossplane does.
Crossplane Architecture
Crossplane is basically a bunch of core controllers with core
CustomResources (claims) which can be extended using Stacks
. Stack is
another bunch of controllers with CustomResourceDefinitions (CRDs) which
plug into the core library to satisfy the claim. Of course I am
oversimplifying here. It offers lot more than just a bunch of
controllers with CRDs but for the time being, let’s assume that’s all it
is. So, you have Crossplane core which is a bunch of controllers
containing generic resource claims like (e.g., KubernetesCluster
,
analogous to PersistentVolumeClaim) while stacks on the other hand
implement the specific resources (e.g., GKECluster
in GCP stack,
analogous to PersistentVolumes) in Crossplane. Let’s take the example of
GCP stack to understand this better.
Let’s try provisioning a cluster in GKE (Google Kubernetes Engine) through Crossplane.
We will be using Minikube to install Crossplane but you can install it
in Kind or whichever cluster
you want to install it in (as long as you can use kubectl
and you
have the permissions to install CRDs aka Custom Resource Definitions).
Let’s fire up Minikube first:
$ minikube start
😄 minikube v1.2.0 on linux (amd64)
💥 Kubernetes downgrade is not supported, will continue to use v1.15.2
🏃 Re-using the currently running virtualbox VM for "minikube" ...
⌛ Waiting for SSH access ...
🐳 Configuring environment for Kubernetes v1.15.2 on Docker 18.09.6
🔄 Relaunching Kubernetes v1.15.2 using kubeadm ...
⌛ Verifying: apiserver proxy etcd scheduler controller dns
🏄 Done! kubectl is now configured to use "minikube"
We will be using Helm 3 to
install Crossplane but you should be fine using helm 2
as well
(installation
using helm 2) with slight modifications in the commands. Note that we
will be installing Crossplane in the default
namespace.
$ helm version
version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}
Let’s first add the helm repo for Crossplane:
$ helm repo add crossplane-alpha https://charts.crossplane.io/alpha
"crossplane-alpha" has been added to your repositories
Now we can check the Crossplane versions available for installation:
$ helm search repo -l crossplane
NAME CHART VERSION APP VERSION DESCRIPTION
crossplane-alpha/crossplane 0.6.0 0.6.0 Crossplane - Managed Cloud Resources Operator
crossplane-alpha/crossplane 0.5.1 0.5.1 Crossplane - Managed Cloud Resources Operator
crossplane-alpha/crossplane 0.5.0 0.5.0 Crossplane - Managed Cloud Resources Operator
crossplane-alpha/crossplane 0.4.1 0.4.1 Crossplane - Managed Cloud Resources Operator
crossplane-alpha/crossplane 0.4.0 0.4.0 Crossplane - Managed Cloud Resources Operator
crossplane-alpha/crossplane 0.3.0 0.3.0 Crossplane - Managed Cloud Resources Operator
crossplane-alpha/crossplane 0.2.0 0.2.0 Crossplane - Managed Cloud Resources Operator
crossplane-alpha/crossplane 0.1.0 0.1.0 Crossplane - Managed Cloud Resources Operator
We will be installing the latest release of Crossplane when writing this
post, which is 0.6
.
$ helm install crossplane crossplane-alpha/crossplane --version 0.6.0
manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
NAME: crossplane
LAST DEPLOYED: Tue Jan 21 14:56:28 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Release: crossplane
Chart Name: crossplane
Chart Description: Crossplane - Managed Cloud Resources Operator
Chart Version: 0.6.0
Chart Application Version: 0.6.0
Kube Version: v1.15.2
You should see two running pods:
kubectl get po
NAME READY STATUS RESTARTS AGE
crossplane-65bdd6599c-9j8tj 1/1 Running 0 37s
crossplane-stack-manager-5556749f76-r7cvh 1/1 Running 0 37s
GCP Stack
(which gives us the Custom Resources to provision
resources in GCP) does not come in-built in our Crossplane installation.
Let’s install it first. You can install the Stack
either through
Crossplane’s own cli
tool (kubectl plugin) or
if you don’t want to do that, just kubectl apply -f stack.yaml
.
Here’s our stack.yaml
(yes, you install stack using a custom resource.
Crossplane’s cli tool does the same thing under the hood):
apiVersion: stacks.crossplane.io/v1alpha1
kind: ClusterStackInstall
metadata:
name: "stack-gcp"
spec:
package: "crossplane/stack-gcp:v0.4.1"
Version v0.4.1
of GCP stack, was the latest version at the time of
writing this post. You can check all the available versions at the
Crossplane’s GCP Stack repo here. If all goes well, you
can check if the stack-gcp
pods are running using kubectl get
po
. You should see something similar to this:
NAME READY STATUS RESTARTS AGE
crossplane-65bdd6599c-9j8tj 1/1 Running 3 26h
crossplane-stack-manager-5556749f76-r7cvh 1/1 Running 2 26h
stack-gcp-5954848cfd-hsbgn 1/1 Running 1 90m
stack-gcp-7blq5 0/1 Completed 0 91m
You can also check if the GCP custom resources are available:
kubectl get crd | grep "gcp.crossplane"
Custom Resources which are of interest to us here are
gkeclusters.container.gcp.crossplane.io
(aka GKECluster
resources) , gkeclusterclasses.container.gcp.crossplane.io
(aka
GKEClusterClass
resource) and providers.gcp.crossplane.io
(aka
Provider
resource).
Sidenote: <resource>.compute.gcp.crossplane.io
is the older
version of <resource>.container.gcp.crossplane.io
so you will see same
resources in both the APIs. We will be using resources from
*.container.gcp.crossplane.io
wherever available.
Alright, now that Crossplane is up and running, let’s go a little deeper
Here’s a TL:DR; image of what we are going to do:
Relation between resources used to provision a cluster in Crossplane
A KubernetesCluster
uses a GKEClass
using cloud provider
information from Provider
to provision a GKECluster
(this is the
resource which represents an actual cluster. Deleting this deletes the
actual cluster).
Here’s
another image to give better context. Here, KubernetesCluster
is our
Resource Claim
, GKEClass
is our Resource Class
and the
resource which is provisioned by KubernetesCluster
using
GKEClass
is GKECluster Resource
(more commonly, Managed
Resource
) in the above image. External Resource
is the actual
cluster which sits in our GKE and is accessible through GCP Console.
If you don’t want to copy and paste things, here’s the intro-to-crossplane repo for all the templates used.
Provider
resourceProvider is the custom resource which holds information about the GCP Project we are connecting to (including reference to the project credentials aka GCP service account). Crossplane uses Provider to provision our cluster and communicate with it. Please make a note that you need to have a GCP project with owner or editor access.
We will setup theProvider
by creating a file named provider.yaml
.
Now, there are two ways we can go about configuring this. If you have
gcloud
cli installed (or if you want to
install it), you can just use
this:
# fill in your project ID
PROJECT_ID=
gcloud --project $PROJECT_ID services enable container.googleapis.com # enables GKE service
gcloud --project $PROJECT_ID services enable compute.googleapis.com # enables VM service
# create service account for Crossplane
gcloud iam service-accounts --project $PROJECT_ID create crossplane \
--description "The service account that Crossplane uses to provision resources" \
--display-name "crossplane-sa"
# grant Kubernetes Engine Admin role and service account user (service account user is required to use the VMs as GKE nodes)
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:crossplane@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/container.admin
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:crossplane@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/iam.serviceAccountUser
# create key (this is our service account json)
gcloud --project $PROJECT_ID iam service-accounts keys create --iam-account crossplane@$PROJECT_ID.iam.gserviceaccount.com crossplane-gcp-provider-key.json
# base64 encode it (tr -d "\n" removes the trailing newline)
BASE64ENCODED_GCP_PROVIDER_CREDS=$(base64 crossplane-gcp-provider-key.json | tr -d "\n")
OR
If you don’t have gcloud
installed or you don’t want to install it,
you can do the same thing as above manually. Crossplane has this excellent
documentation on setting up Provider
.
Following is a shortened and a hopefully simplified copypasta from that
link:
Set PROJECT_ID
env variable to your GCP Project ID and
BASE64ENCODED_GCP_PROVIDER_CREDS
to the contents of your service
account json encoded in base64.
$ export PROJECT_ID=my-project # replace this with your project id
$ export BASE64ENCODED_GCP_PROVIDER_CREDS=$(base64 crossplane-gcp-provider-key.json | tr -d "\n")
For service account, let’s go with Kubernetes Engine Admin
role in
GCP
(here’s
the doc for how to grant a role and
here’s
what Kubernetes Engine Admin can do).
Aand with that painful thing of setting up service account and all those
environment variables done, let’s create our Provider
resource
(environment variables we set before replace the ${}
thingies).
cat > provider.yaml <<EOF
---
apiVersion: v1
kind: Secret
metadata:
name: gcp-account-creds
namespace: default
type: Opaque
data:
credentials: ${BASE64ENCODED_GCP_PROVIDER_CREDS}
---
apiVersion: gcp.crossplane.io/v1alpha3
kind: Provider
metadata:
name: gcp-provider
spec:
projectID: ${PROJECT_ID}
credentialsSecretRef:
namespace: default
name: gcp-account-creds
key: credentials
EOF
Notice that the Provider
does not hold the credentials to our GCP
account. Instead, it delegates that to a Secret
which in turn is
referenced in the Provider
.
Once you’re all set, apply the the provider.yaml
with
$ kubectl apply -f "provider.yaml"
$ kubectl get providers.gcp.crossplane.io
NAME PROJECT-ID AGE
gcp-provider my-project 3h6m
Note that we are using providers.gcp.crossplane.io
instead of simply
providers
because there are multiple resources with the name
providers
in Crossplane (e.g., Azure Kubernetes Service aka AKS has
its own
Provider
resource which points to providers.azure.crossplane.io
).
Now that we’re all set with the credentials (if you face any problems, feel free to comment below or just skip to the end to see where you can get support if you get stuck).
GKEClusterClass
resourceWith that out of the way, let’s get to the heart of what this post is
all about. Configuring a cluster! This is taken care of through
GKEClusterClass
Kubernetes resource. Here’s what I am talking about:
apiVersion: container.gcp.crossplane.io/v1beta1
kind: GKEClusterClass
metadata:
labels:
className: "gke-class"
name: gke-class
namespace: default
specTemplate:
forProvider:
location: us-central1-b
providerRef:
name: gcp-provider
reclaimPolicy: Delete
writeConnectionSecretsToNamespace: default
This is where we specify the details for our cluster. Think of this is a template that you can re-use to create many clusters with the same settings. Notice the label here. This is going to be important later.
Let’s kubectl apply -f
the above GKEClusterClass
template first:
$ kubectl apply -f gkeclusterclass.yaml
Here’s what it should look like:
$ kubectl get gkeclusterclasses
NAME PROVIDER-REF RECLAIM-POLICY AGE
gke-class gcp-provider Delete 6s
If you take a closer look at the yaml template above, you might ask, hey
you said I can configure my cluster in GKEClusterClass
but where do
I specify how many nodes I want machine type, disk size and all the
important things? Well, up until recently, specifying what kind of
nodepools you want was a part of the yaml template above but it was
recently moved into a separate NodePool
resource which has to be
created separately (what a pain right? well, it’s kinda complicated. You
can read more about it in CrossPlane’s Slack community.
You might have to login to the Crossplane
slack channel to view
this. Sorry 🙁 but to summarize, this pain is a part of a grand plan to
something good. It’s a secret, well not
really).
Here’s what our NodePool
resource looks like (except you will have
to replace my-project
with your GCP project ID):
apiVersion: container.gcp.crossplane.io/v1alpha1
kind: NodePool
metadata:
name: gke-nodepool
namespace: default
spec:
providerRef:
name: gcp-provider
writeConnectionSecretToRef:
name: gke-nodepool
namespace: default
forProvider:
initialNodeCount: 2
# substitute my-project with your gcp project id here
cluster: "projects/my-project/locations/us-central1-b/clusters/example-cluster"
config:
machineType: n1-standard-2 # default is n1-standard-1 if not specified
diskSizeGb: 20 # default is 10GB if not specified
Let’s save this to a file nodepool.yaml
and apply it:
$ kubectl apply -f nodepool.yaml
If we kubectl get nodepools
, you should see something like this:
$ kubectl get nodepools
NAME STATUS STATE CLUSTER-NAME NODE-POOL-CLASS RECLAIM-POLICY AGE
gke-nodepool projects/my-project/locations/us-central1-b/clusters/example-cluster 2s
Note that the Nodepool
has no State
right now because our
cluster has not been provisioned yet. State
field will be updated
once our NodePool
resource starts provisioning a node pool (GKE
nodepool) for us which happens once the cluster has provisioned.
KubernetesCluster
resourceThere’s only one more thing left to do now (phew!) and that is our
KubernetesCluster
resource. KubernetesCluster
is the claim
resource I was talking about at the beginning of this post. This is the
analogue of PersistentVolumeClaim
in our PVC-PV analogy and the very
last piece in the puzzle to create our very own cluster using
Crossplane.
apiVersion: compute.crossplane.io/v1alpha1
kind: KubernetesCluster
metadata:
name: example-cluster
namespace: default
annotations:
crossplane.io/external-name: example-cluster
spec:
classSelector:
matchLabels:
className: "gke-class"
writeConnectionSecretToRef:
name: example-cluster
Aaand, once this is done, all you have to do is sit and watch
. Ahem,
I meant:
$ kubectl get kubernetescluster --watch
Also, remember the label we talked about when adding GKEClusterClass
,
well that is used by our KubernetesCluster
to match the class for
itself. You might ask, why not just allow adding the class name instead?
Well, label makes things more flexibile. You don’t have to depend on the
name of the class anymore but a label which can be present on any class
(if that’s not very convincing, you can use the classRef
instead to
specify your class name). Note that Crossplane does not use the name of
KubernetesCluster
resource to name the provisioned cluster in GCP
Console. You have to explicitly specify what you want to name your
provisioned cluster. Notice the crossplane.io/external-name:
my-cluster
here. This is used to specify the name of the actual
cluster being provisioned in GCP console. If you don’t specify this,
Crossplane will just pick an arbitrary name for the cluster being
provisioned.
Let’s check how our KubernetesCluster
resource is doing first.
$ kubectl get kubernetescluster
NAME STATUS CLASS-KIND CLASS-NAME RESOURCE-KIND RESOURCE-NAME AGE
example-cluster GKEClusterClass gke-class GKECluster default-example-cluster-mq75g 7s
You should see something like this (if you don’t see CLASS-NAME
name
populated with RESOURCE-NAME
name, something is wrong. Go to the end
of this post to see how you can debug this).
default-example-cluster-mq75g
is the GKECluster
resource that
Crossplane provisioned for us. These kind of resources which are managed
by another resource are called Managed Resources
in the Crossplane
lingo. If you notice, NodePool
represents an actual NodePool
resource which seems similar to GKECluster
as in, it is the resource
which actually represents an external resource in the cloud provider,
you might wonder if NodePool
might be a Managed Resource
and you
would be right but the thing is, the resource which would actually
manage this NodePool
is not there. You read that right. It’s
something that the Crossplane community is working on right now.
You can go and check the status of provisioning by doing kubectl get
gkecluster default-example-cluster-mq75g -o yaml
:
$ kubectl get gkeclusters
NAME STATUS STATE ENDPOINT CLUSTER-CLASS LOCATION RECLAIM-POLICY AGE
default-example-cluster-mq75g PROVISIONING gke-class us-central1-b Delete 19s
So the cluster is being provisioned but the STATUS
is empty. This
will be populated once the cluster is fully provisioned.
Provisioning a cluster might take some time, so if you are bored or want to stretch a bit or grab a cup of coffee maybe, now’s the time.
If everything goes well, you should see Bound
status in your
KubernetesCluster
resource.
$ kubectl get kubernetescluster
NAME STATUS CLASS-KIND CLASS-NAME RESOURCE-KIND RESOURCE-NAME AGE
example-cluster Bound GKEClusterClass gke-class GKECluster default-example-cluster-mq75g 3h9m
Well, that’s all! But oh wait, what do you do if you run into any problems?
The first place to look at is your GKECluster
. Just do kubectl get
container.gcp.crossplane.io <resource-name> -o yaml
. For example, if
we take the above case:
apiVersion: container.gcp.crossplane.io/v1beta1
kind: GKECluster
metadata:
annotations:
crossplane.io/external-name: example-cluster
creationTimestamp: "2020-01-23T09:15:39Z"
finalizers:
- finalizer.managedresource.crossplane.io
generateName: default-example-cluster-
generation: 2
name: default-example-cluster-mq75g
resourceVersion: "39804"
selfLink: /apis/container.gcp.crossplane.io/v1beta1/gkeclusters/default-example-cluster-mq75g
uid: 325ad1a4-71f9-4e38-bd55-98f9e2bfa980
spec:
claimRef:
apiVersion: compute.crossplane.io/v1alpha1
kind: KubernetesCluster
name: example-cluster
namespace: default
uid: b08afb38-f58e-422d-add4-4560c1ddcb0b
...
status:
atProvider:
...
selfLink: https://container.googleapis.com/v1beta1/projects/my-project/zones/us-central1-b/clusters/example-cluster
status: PROVISIONING
zone: us-central1-b
conditions:
- lastTransitionTime: "2020-01-23T09:15:39Z"
reason: Successfully resolved managed resource references to other resources
status: "True"
type: ReferencesResolved
- lastTransitionTime: "2020-01-23T09:15:40Z"
reason: Managed resource is being created
status: "False"
type: Ready
- lastTransitionTime: "2020-01-23T09:15:46Z"
reason: Successfully reconciled managed resource
status: "True"
type: Synced
You can find what went wrong in the conditions
section.
If you still can’t fix the issue,
Hope you enjoyed this detailed introduction and walkthrough of how to do cluster provisioning using Crossplane. For more informartive blogs, you can follow us on Twitter and LinkedIn.
Looking for help with Kubernetes adoption or Day 2 operations? learn more about our capabilities and why startups & enterprises consider as one of the best Kubernetes consulting services companies.