Kubernetes is a fascinating software for the abstraction of infrastructure. This will make deploying and configuring services more portable. It will make them independent of running on-premises or on the cloud provider of your choice. It relies on internal automation extensively. It’s clear that this should be extensible to users for their own automation purposes.
It is clear how Kubernetes works here. Many platforms use direct commands like apt install mypackage for deployment and configuration. It’s a clear directive: do this, and do that. I explicitly state the desired state of the Kubernetes cluster. It’s done by applying files describing that state. These files, typically in YAML format, provide clear descriptions of Pods, ReplicaSets, Deployments, Services, and more. A creation, update, or deletion via kubectl or directly by external applications is first reflected in the etcd database via kube-api. Its data is used by components like the kube-scheduler to do their work. The technical target state is configured for things like the assignment of pods to nodes. Kubelets run on these nodes, which are physical or virtual machines. Their task is straightforward: establish the desired state and report this back via the API. This control loop is consistent throughout the entire system. The controller pattern is used in components such as the node controller, the replication controller, the endpoints controller, and the service account & token controllers.
This is not limited to Kubernetes in the core. External clients of the Kubernetes API can use this pattern and serve as controllers for CustomResources. These components are user-defined and based on their corresponding CustomResourceDefinitions (CRDs), just like the core documents for the application of the desired state. These are applied to a cluster using the standard command describing the CRD, e.g. kubectl apply -f mycrd.yaml. Integrate automation this way into a cluster will be relative easy by following the instructions outlined later, along with the designated controller code. These controllers do not require custom resources. They can react to changes in existing resources and create, update, or delete them. Using controllers is the best way to avoid individual errors in executing recurring activities and make their way of working more natural in the world of Kubernetes. It’s a way to avoid hard-to-maintain mixed landscapes in Kubernetes cluster system management.
Controllers can be used in different scenarios. They provide a full access to the API during runtime within the scope of your authorization. Examples are an automatical roll out of dependencies for applications that are currently deployed, or also to automatical update database schemas during upgrades. Controllers must ensure that services that have just been started are registered with external services if those do not have direct access to the Kubernetes API. In cloud environments, they can monitor current resources and activate additional nodes and scale the corresponding replica sets if necessary.
The development of a controller follows a standard process. A CustomResource must be defined by its corresponding CustomResourceDefinition. These resources represent the API for a controller, so a version scheme is used for them, which will be part of the resulting URL. The alpha level is applied if the API is still changing in the early stages of development. The versions here are named v1alpha1 or v2alpha3. They move to the beta level when stability and testing increase. Numbers like v1beta1 or v2beta1 are used. They are already very robust and well designed, but they must still be entered with great care. Stable versions of an API do not carry a level. They are referred to as v1, v2, etc.
The implemented controller can now be operated outside the cluster or as a pod within it. It connects the cluster via the Kubernetes API and listens to whether its own CustomResources or other resources it is interested in are added, updated, or deleted. It can react to these events and match the reality with the desired state.
In Go, this event model can be implemented in two ways using the packages of the Kubernetes Client-Go project. The first one is using Watches, which can be set up for the respective resource type. They deliver individual events via channels. In addition to the type of event, for example Added, Modified, Deleted, Bookmark, or Error, these events contain the respective object concerned. As typical in Go, the Watches are operated in an endless loop contained in a select.
// Request a watcher for services in all namespaces
svcWatcher, err := clientset.
CoreV1().
Services("").
Watch(metav1.ListOptions{})
if err != nil {
panic(err.Error())
}
// Loop to receive events
for {
select {
case evt := <-svcWatcher.ResultChan():
...
}
}In the case branches the controller can now process the individual events. The second and more common variant is using an Informer, which is given individual callbacks for the various events.
// Retrieve an informer for services
factory := informers.NewSharedInformerFactory(client, time.Second*30)
svcInformer := factory.Core().V1().Services().Informer()
// Add individual handlers and run the informer in the background
svcInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: addServiceHandler,
UpdateFunc: updateServiceHandler,
DeleteFunc: deleteServiceHandler,
})
go cd.svcInformer.Run(wait.NeverStop)For the work of an Informer, its method Run() is sent into the background via a go. In this way, several Informers for the respective resources can be operated simultaneously in one controller. The passed handler callbacks are functions with the very simple signature func(obj interface{}). This generic approach without any type security is due to Go as a language. However, the necessary type cast in the handler is quite simple if a handler callback is only assigned to one Informer.
Regardless of the model, the controller has the task of processing the occurring events. My example here pursues the idea of a controller in its own namespace. It distributes labeled ConfigMaps or Secrets to other namespaces depending on the configuration of the controller when it will be deployed. It thus reacts to its own resource so that it can initialize or update itself. It also reacts to added or changed ConfigMaps or Secrets in its own namespace so that it can distribute them to the configured namespaces if their label is appropriate. Finally, it reacts when namespaces are added. If these are contained in the own list of namespaces, it also copies the ConfigMaps and Secrets here.
Custom Resources
The following CustomResourceDefinition of the example contains the name of the type, its group and version, the scope and name variants. It also contains rules for validating a new applied resource. For the fields, these include their types and patterns or value ranges, but they can also contain complex and alternative substructures. The type of the example contains the ConfigurationDistributionRule in the group labs.themue.dev. This version is the first and therefore still very unstable v1alpha1. The three fields mode, namespaces and selector are included as properties.
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: configurationdistributionrules.labs.themue.dev
spec:
group: labs.themue.dev
versions:
- name: v1alpha1
served: true
storage: true
scope: Namespaced
names:
plural: configurationdistributionrules
singular: configurationdistributionrule
kind: ConfigurationDistributionRule
shortNames:
- cdr
- codisrule
- rule
validation:
openAPIV3Schema:
type: object
properties:
mode:
type: string
pattern: "^(configmap)|(secret)|(both)$"
namespaces:
type: array
items:
type: string
selector:
type: string
pattern: '^\w$'The according structure of a CustomResource is now part of the program code. There are good generators for this purpose, for example the Kubebuilder. These are very helpful, especially for more extensive and complex types. For this article, however, the development should be done manually in the package v1alpha1 as a demonstration. The fields of the resource can be found in a structure, this as the ConfigurationDistributionRule together with embedded metadata. The structures for the metadata are taken from the package k8s.io/apimachinery/pkg/apis/meta/v1.
type ConfigurationDistributionRuleSpec struct {
Mode string `json:"mode"`
Selector string `json:"selector"`
Namespaces []string `json:"namespaces"`
}
type ConfigurationDistributionRule struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec ConfigurationDistributionRuleSpec `json:"spec"`
}The structures must have appropriate tags for a JSON marshalling, because the communication with the API is done via REST and JSON. However, before implementing the communication the deep copy has to be implemented. The Kubernetes libraries require this internally. Normally, this is also generated, but for smaller objects like this one, a manual implementation is not difficult.
In addition to a structure for individual resources, another structure is needed for lists. It must also allow a JSON marshalling and a deep copy. Since these types are unknown to the known schema of the library, they have to be made known to it. For this purpose there are also helpers which have to be added for the own types.
var (
// SchemeGroupVersion describes the CRD.
SchemeGroupVersion = schema.GroupVersion{
Group: "labs.themue.dev"
Version: "v1alpha1",
}
// SchemeBuilder creates a scheme builder for the known types.
SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
// AddToScheme points to a function to create the known types.
AddToScheme = SchemeBuilder.AddToScheme
)
// addKnownTypes adds our new types.
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(
SchemeGroupVersion,
&ConfigurationDistributionRule{},
&ConfigurationDistributionRuleList{},
)
metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
}It now has to be called in the main program with v1alpha1.AddToScheme(scheme.Scheme). Kubebuilder thankfully cares also for this stuff.
If the resource is defined and implemented, an interface for accessing it follows as a further component. It is a typed client that encapsulates the generic REST client, making it more convenient for the controller. A Go interface is defined, and it’s instantiated via a factory type specifying the namespace. The REST client itself is straightforward.
func (ri *ruleInterface) Create(
rule *ConfigurationDistributionRule,
opts metav1.CreateOptions,
) (*ConfigurationDistributionRule, error) {
result := ConfigurationDistributionRule{}
err := ri.restClient.
Post().
Namespace(ri.namespace).
Resource("configurationdistributionrules").
Body(rule).
VersionedParams(&opts, scheme.ParameterCodec).
Do().
Into(&result)
return &result, err
}A glance at the documentation of the core interfaces shows that they implement not all methodologies. This depends on the individual requirements of the components. However, the method names of your own interfaces should always follow the standard to make it easier for other developers to work with the component.
The last remaining part of the CustomResource is the Informer. It helps the controller to react to the events regarding added, changed or deleted resources. The first type activates a controller with the configuration it needs, the second one updates it, and the third deactivates the controller without uninstalling it. The Informer uses the already implemented RuleInterface for its communication with the API. The Informer interface itself defines two methods. One of them is used to update a cache and to integrate the callbacks later, the second one is used to access the monitored resources.
type RuleInformer interface {
Informer() cache.SharedIndexInformer
Lister() RuleLister
}
func (ri *ruleInformer) Informer() cache.SharedIndexInformer {
return cache.NewSharedIndexInformer(
&cache.ListWatch{
ListFunc: func(opts metav1.ListOptions) (result runtime.Object, err error) {
return ri.rif.List(opts)
},
WatchFunc: func(opts metav1.ListOptions) (watch.Interface, error) {
return ri.rif.Watch(opts)
},
},
&ConfigurationDistributionRule{},
30*time.Second,
cache.Indexers{},
)
}The latter is again defined by its own, fortunately quite clear type.
type RuleLister interface {
List(selector labels.Selector) ([]*ConfigurationDistributionRule, error)
Get(name string) (*ConfigurationDistributionRule, error)
}
func (rl *ruleLister) List(selector labels.Selector) ([]*ConfigurationDistributionRule, error) {
cdrl, err := rl.rif.List(metav1.ListOptions{
LabelSelector: selector.String(),
})
if err != nil {
return nil, err
}
list := make([]*ConfigurationDistributionRule, len(cdrl.Items))
for i, rule := range cdrl.Items {
list[i] = &rule
}
return list, nil
}
func (rl *ruleLister) Get(name string) (*ConfigurationDistributionRule, error) {
return rl.rif.Get(name, metav1.GetOptions{})
}This multitude of own types may seem a bit confusing and opulent at first. But especially when the tasks of your own controller go beyond simple automation, modularization with clear responsibilities and types shows its advantages.
Controller
So far, the example only includes the code for the CustomResourceDefinition and the handling of the corresponding CustomResources. The actual controller with its logic is still missing. In the example it is the type ConfigurationDistributor in the package codis. It contains the entire logic of the controller. So in principle it also could be accommodated in the main program. But for reasons of maintainability, however, modularization is also useful here.
The only exported parts of the package are the type and its constructor function. For the type itself, only its Run() method is public. It provides the internally used Informers for the ConfigurationDistributionRules, ConfigMaps, Secrets and Namespaces with their own methods as event handlers. The Informers are started in the background.
// Run executes the configuration distributor.
func (cd *ConfigurationDistributor) Run(ctx context.Context) {
cd.ruleInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: cd.addRuleHandler,
UpdateFunc: cd.updateRuleHandler,
DeleteFunc: cd.deleteRuleHandler,
})
cd.cmInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: cd.addConfigMapHandler,
UpdateFunc: cd.updateConfigMapHandler,
})
...
go cd.ruleInformer.Run(wait.NeverStop)
go cd.cmInformer.Run(wait.NeverStop)
go cd.scrtInformer.Run(wait.NeverStop)
go cd.nsInformer.Run(wait.NeverStop)
select {
case <-ctx.Done():
// Work is done.
}
}The individual handler methods are usually very compact, since they can focus their attention on a single activity for only one resource type. They filter certain criteria for a check whether they should be active at all. Then they start.
func (cd *ConfigurationDistributor) addConfigMapHandler(
obj interface{},
) {
if cd.rule == nil {
return
}
if cd.rule.Spec.Mode != "configmap" && cd.rule.spec.mode != "both" {
return
}
cm := obj.(*corev1.ConfigMap)
if cm.GetNamespace() != cd.namespace {
return
}
if cd.rule.Spec.Selector != "" {
if cm.GetLabels()["rule"] != cd.rule.Spec.Selector {
return
}
}
cd.applyConfigMap(cm, true)
}In the end, all that remains is to implement the main program. It only takes over passed flags, instantiates the actual controller and lets it run. This is already everything. For the example, the main package for this is in .../cmd/codis. It has four flags for the master URL of the API server or the kubeconfig to be used, both for access to the cluster, and the namespace and rule name for which the controller should be responsible. Loading the configuration, which is also passed on to the ConfigurationDistributor, is done using one of the corresponding help functions of the standard library.
config, err := clientcmd.BuildConfigFromFlags(masterURL, kubeconfig)Then initialize the controller, register the new schemas and let the controller do its work.
cd, err := codis.New(config, namespace, rulename)
v1alpha1.AddToScheme(scheme.Scheme)
cd.Run(context.Background())This Run() could also be sent into the background again, so that the controller may take over further tasks. However, our example is satisfied at this point. It now only wants to be started with the right arguments to start its work.
$ codis --kubeconfig $HOME/.kube/config \
--namespace ns-codis-test --rulename rule-codis-testThe last thing missing is the rule in this case. It must be created in the namespace and with the appropriate name.
apiVersion: labs.themue.dev.com/v1alpha1
kind: ConfigurationDistributionRule
metadata:
name: rule-codis-test
namespace: ns-codis-test
spec:
mode: both
selector: test
namespaces:
- default
- another-testThis rule copies both ConfigMaps and Secrets to the default and another-test namespaces if they have a rule: test label.
Conclusion
As mentioned above, this is not necessarily the most useful controller. But it demonstrates the components for its implementation. In the case of one or more custom CustomResourceDefinitions, it shows the definition of the schema, its announcement, how access to it is packaged into an API, and how the actual controller is provided with a connection to the event model. It is a bit overhead, indeed. But for more complex models this is done by Kubebuilder or similar.
The second part, the logic of the controller, is more straightforward. This depends on its purpose, of course. If the logic is more complex with more extensive access to the Kubernetes API or to external systems such as the APIs of the different cloud providers, it will be correspondingly more extensive.
Either way it turns out that the principle of controllers has its origin in the internal and complex world of Kubernetes. It has only gradually opened up to the user-side extension of the system, the v1 version of the API has been reached with Kubernetes v1.17. And with this focus, it is not surprising that the list of Awesome Operators contains only technical controllers. They demonstrate how much it pays to enter the world of controllers when managing and expanding your own clusters.
