Kubernetes-compatible API for EVE Controller

This document reviews the standard functionality of an EVE controller, and how those might be implemented using Kubernetes. It is not tied to a particular implementation of EVE controller, and as such may not cover all use cases. However, it should be extensible enough for any particular controller to adopt this and then add the specific needs.

As of this writing, there are two known implementations of controllers: Open Source Adam, and commercial Zedcloud by ZEDEDA.

Adam is a reference OSS EVE controller. It has a very limited, but simple, custom REST API, described here as its adminHandler. Specifically, it enables:

• Adding, deleting, updating and reading onboard certificates

• Adding, deleting, updating and reading an edge device

• Getting and setting global options

• Getting and setting device specific options

• Reading a device’s logs, information, metrics and certificates

• Reading and writing a device’s configuration as a single json blob, per the EVE API.

It is important to note that Adam provides no granular control over a device; it only knows how to return the entire device config as json, or receive an entirely new device config as json.

Zedcloud is an enterprise-class commercial solution, offered as a fully-managed SaaS solution, along with device support and professional services, from ZEDEDA. In addition to management services, and all of the abilities that Adam provides, ZEDEDA offers:

• Multiple forms of authentication/authorization as well as enterprise SSO

• An advanced UI

• Reading and writing individual components of a device’s configuration, as well as validations, which, in turn, are composed by Zedcloud into a device’s configuration.

• Libraries of components, applications and images

• Predefined templates

• Many more features

The goal of this document is to provide an API design that:

• Is compatible with Kubernetes.

• Supports all of current Adam functionality.

• Enables getting and setting of individual elements within the EVE API, thus enabling Adam to combine them into a full json device configuration.

• Is extensible, so that commercial controllers, such as Zedcloud, could use the API as a basis for their own API which is 100% compatible with this API, yet supports their rich additional features.

On the basis of the above:

1. Anything that is in Adam or an element of the EVE API will be covered here.

2. Anything that is not in the above will not be covered here.

3. The method of extensions will be covered here.

Note that elements of the EVE API that are intended entirely for control between the controller and device, will not have a Kubernetes-compatible component, as those are not intended to be exposed to the end-user.

Why would we want to standardize on a Kubernetes-compatible API? The Kubernetes API comes with several advantages:

• It provides a standardized interface.

• It enables controller designers to leverage existing reliable CLI and SDK tooling with which large numbers of technologists and IT departments are familiar.

• It allows controller designers to replace entire parts of their code with an off-the-shelf component that is mature, reliable, production-tested, and capable.

• It enables controller providers to plug into almost any Kubernetes ecosystem component.

• It eases locating skilled staff and shortens their learning curve.

• Globally available CLI and clients in multiple languages, supported by a large distributed team of people, with a very large existing installed base

• Ability of customers to define all EVE resources as code, in human-readable template files, and stored optionally in version control

• Easy integration with any system that works with Kubernetes, e.g. CI systems or monitoring systems

• Freely available server-side tooling that supports REST endpoints, validation, authentication, distributed highly-available processing, eventual reconciliation, role-based access control based on API paths, including individual resource-level control; all standardized and used in production, supported by a large number of engineers globally.

Custom Resources vs Native Kubernetes

Kubernetes is an API and a system for declaring workloads and scheduling them on nodes.

EVE includes an API and a system for declaring workloads and scheduling them on nodes.

The basic concepts of EVE and Kubernetes are very similar. In addition to those concepts, EVE provides node management, and Kubernetes provides higher-order scheduling options.

From that perspective, it is possible to use native Kubernetes resources:

• Pod for AppInstance workload

• Node for Edge Device

However, doing so brings minimal advantages and some real disadvantages.

1. Kubernetes itself will try to schedule the AppInstance workloads onto nodes, as it thinks they are normal Kubernetes pods. We can prevent that using Pod node affinity, but it requires additional steps.

2. Kubernetes itself will try to schedule other workloads onto the Edge Devices, as it thinks they are normal Kubernetes nodes. We can prevent that using node taints and pod tolerations, but it requires additional steps.

3. Kubernetes expects nodes to join in the normal fashion, using either a node certificate signed by an appropriate CA, or a one-time registration token. These are similar, but not identical, to the process used by EVE. We can get around that by registering the nodes separately, but it is going against the Kubernetes grain and can lead to issues.

4. Kubernetes expects nodes to be accessible and in regular communication with the control plane kube-apiserver, while EVE expects Edge Devices to be out of touch, at times for extended periods. This will cause Kubernetes to mark these nodes as unavailable for scheduling, and eventually remove them. The apiserver-node API is built with cloud assumptions of regular and reliable connectivity.

5. Kubernetes expects nodes to speak the undocumented kubelet API. This includes the internal elements, but also receiving pod specifications. Using this would require implementing the kubelet API on the Edge Device or somehow convincing Kubernetes that it was.

6. Overloading the Node and Pod resources forces the cluster to be in a confusing state for users. This would make it difficult to deploy an EVE controller onto a regular Kubernetes cluster, creating barriers to adoption.

For the above reasons, we implement the entire EVE API in custom resources (CRDs) and do not use the native APIs.

For the purposes of edification, we include a table comparing the two approaches here, and include what a native option looks like in an appendix.

The following table summarizes all resources. Original resources - either from the higher-level Adam abstraction or from the native EVE config - are in black, reused native Kubernetes resources are in blue, custom resources are in green. Where a native resource inherently works well, even the custom resource column will use the native resource, marked in blue.

Items that require special treatment:

• Device Config - the entire json that is composed by a controller and sent to a device. The core controller reads the other elements and creates this resource (which can be modified by the end-user). A separate controller reads it and applies it to the device via the API.

• Read-only elements from the Edge Device - log, info, metrics - do not receive their own resources. Kubernetes resources are designed to be created/modified, remain in relatively stable state and are of small size. They are stored in the Kubernetes backing store, by default etcd, and do not fit large streaming elements like logs or metrics. When Kubernetes itself enables reading log information from a pod, it does so by leveraging an element of the control-plane-to-kubelet API and open real-time streaming of logs.

The streaming elements - log, info, metrics - will stream to the controller according to the current EVE API design, are stored by the controller in its own datastore, and controller clients open a streaming channel to the controller. There currently is no native construct within the Kubernetes API that knows how to work with this flow, nor do CRDs have such elements.

Since Kubernetes has no native construct for this, we create a special “streaming” pod and service. This has a Web server and a Service in front of it, exposing endpoints. We use RBAC to enable access to this pod.

This pod exposes endpoints:

• /edgenode/<uuid>/metrics

• /edgenode/<uuid>/info

• /edgenode/<uuid>/log

The edgenode uuid must match one that is provided by the CRDs. Headers determine if the data should be streaming continuously or only show existing data, while query parameters determine start and end times and filter parameters.

Design

Although we are creating entirely new custom resources, we follow these rules:

• Follow native Kubernetes semantics as much as possible.

• Where concepts closely parallel those that exist in native Kubernetes resources, we adopt the nomenclature and semantics as much as possible into our custom resources.

• Where standard annotations already exist to describe a concept, use those annotations.

• Where a native resource is 100% reusable, without mismatches, reuse it.

Extensibility

Various controllers may choose to implement resources in different ways, including adding features to those resources that are not part of the primary open source specification here.

The mechanism for extensibility in all cases is annotations and additional CRDs.

Annotations

If a controller needs to add functionality to a resource defined in this specification, it should add an annotation that matches its requirement. The annotation’s prefix must match the controller provider’s unique domain. The prefix eve.lfedge.org is reserved for standard annotations that are used as part of this specification.

As each controller implementation must implement the control loops for the various resources, including the standard ones, it is up to the specific controller’s implementation of the control loop to manage the additional annotations.

Other controllers must ignore such annotations.

For example, if a fictitious commercial controller provider named edgesmart, with domain edgesmart.tv, wishes to add features to the DeviceNetwork resource, it adds annotations using that domain.

Custom Resources

Controllers are not limited to the CRDs defined in this specification. Any controller may implement additional CRDs.

In doing so they must not use the apiVersion prefix defined in this specification, but rather must use one that matches a domain owned by them.

For example, if a fictitious commercial controller provider named edgesmart, with domain edgesmart.tv, wishes to have a new type of device security configuration, they can create a CRD for it. Notice the apiVersion field.

Resource Versioning

Every resource type in Kubernetes has a unique identifier of group/version/kind. See the following references:

• https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-uris

• https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups-and-versioning

• https://kubernetes.io/docs/reference/using-api/#api-groups.

Using the above example:

The resource is:

• group: ctrl.edgesmart.tv

• version: v1beta1

• kind: DeviceSecurityConfig

For all cases where we use native Kubernetes resources, we follow the native resource’s group/version/kind scheme.

For all CRDs created here:

• Versions should follow the normal Kubernetes naming scheme of v1alpha1, v1alpha2, … v1beta1, v1beta2, … v1, v2alpha1, … v2, etc.

• Group must be eve.lfedge.org

• Group for resources defined by third-party extensions, such as commercial controllers, must use their own domain and not reuse eve.lfedge.org

Unique Resource Identification

Every instance of a resource type is uniquely identified either by name or, for namespace-scoped resources, by namespace/name. 

The EVE API, on the other hand, uses UUIDs, normally generated by the controller, to uniquely identify and link resources.

The API defined here follows the Kubernetes convention in using group/version/kind for identifying resource types, and name, with optional namespace, for uniquely identifying instances of resources.

It is up to the controller to create UUIDs, link them to specific name/namespace of resources internally, and map them to UUIDs sent to edge nodes via the EVE API.

Authentication

Authentication leverages normal Kubernetes authentication and authorization paths, whether via static users, client certificates or SSO via OIDC.

Edge Device

The Device is very similar to the native Node, except that specification items that would be loaded into annotations are made part of the core spec.

A Device can be created in one of two ways:

• User: A user can create a Device resource by adding the resource. It can include a serial, onboard certificate and/or device certificate. When the device attempts to connect to the controller, it is the controller’s onboarding policy that determines how it authenticates, using the device certificate, onboard certificate, serial, CA on certificates, or any combination of the above.

• Device: A device can self-register, automatically causing a Device resource to be created, if and only if the controller policy allows it. In this case, either it must be one of:

•  

  • Open to all self-registration.

  • Allows a device certificate signed by a known CA to self-register.

  • Allows an onboard certificate signed by a known CA to self-register.

  • Allows a known onboard certificate to self-register.

The resources that enable the following are:

• OnboardCertificate

• OnboardCA

• DeviceCA

These are all namespaced.

Note that if a user desires to allow a particular onboard certificate and serial combination, and the controller supports it, they simply create the Device resource, providing the onboard certificate and serial as part of the spec.

Onboard Certificate

Any device presenting this onboard certificate can self-register.

Onboard CA

Any device presenting a certificate signed by this CA can self-register.

Device CA

Any device presenting a device certificate signed by this CA can self-register.

Networks

Node Network

Creating an EVE-style device network requires the usage of two CRDs, one for configuration information, which can be reused, and one for the on-device network itself.

Note that the CRD NetworkConfig (below) is very similar in principle to the Kubernetes NetworkAttachmentDefinition. 

Network configuration:

Network instantiation:

Workload Network

We leverage the cncf standard annotations on the workload to indicate desired networks on the actual workload.

Storage

The EVE semantics for storage are as follows.

Every AppInstanceConfig needs one or more Volume, which in turn are based either on a blank volume or a ContentTree. When creating an Edge App Instance, these are converted either into disk images which are attached to VMs or mount points attached to containers. The first provided Volume is the bootable one for VMs or container image for containers. Subsequent Volumes may be read-only or read-write.

• AppInstanceConfig

•  

  • Volume1

  •  

    • ContentTree

  • Volume2

  •  

    • ContentTree

  • Volume3

  •  

    • Blank

  • …

Kubernetes already supports the basic structures - multiple volumes, custom storage volumes and custom storage drivers - with the StorageClass resource. We create StorageClass for blank and each source:

• eve-blank: for a blank disk or mountpoint

• eve-quay: from container image on quay.io

• eve-docker: from container image on docker hub

• etc.

for blank:

Credentials secrets, if needed, are affiliated with the StorageClass as credentialsRef.

We define Custom Resources for Image, and then use admissions controllers to validate that the requested resources exist when deploying a Pod that references them.

The Image name is then used in a PersistentVolumeClaim. See below.

Workloads

We define a workload called Application for an application to run on an Edge Device. It references all of the necessary elements. Note that it is called an Application, and not an ApplicationInstance. An instance is something that occurs when deployed on an Edge Device. The Application is the Kubernetes resource describing the intent to deploy.

We reuse standard constructs from native Kubernetes workloads, i.e. pods, as much as possible:

• To determine the node(s) to which an Application is to deploy, we use the standard Kubernetes node affiliation constructs for pods, specifically nodeAffinity and nodeSelector.

• To determine the DeviceNetwork(s) to which the Application should attach, we use the standard networks annotation k8s.v1.cni.cncf.io/networks.

• For storage volumes:

•  

  • For the boot image, we use the image field, which must match a named Image.

  • For other volumes, we use Kubernetes PersistentVolumeClaim.

• Support for multiple containers enables future packaging of multiple workloads together.

Boot image

The image field refers to the name of a defined Image.

Additional Volumes

For all additional storage volumes, we use Kubernetes Volume resources, specifically PersistentVolumeClaim.

For example:

Golden filesystem image stored on FTP site, mounted as a filesystem. Defined using the StorageClass eve-ftp.

Golden VM image stored on FTP site, mounted as a block device. Defined using the StorageClass eve-ftp.

Blank disk volume.

Status

The state of an Application, as reported by the controller, is set on the ApplicationStatus. For example:

The ApplicationStatus field is similar to the Kubernetes PodStatus, albeit not identical. The fields are as follows.

• state: current state of the Application.

• statuses: array of historical states of the application. Each state includes two fields:

•  

  • state: the state when this status occurred.

  • timestamp: the timestamp when this status occurred.

The states of the application are the ones currently supported by the EVE API. E.g. BOOTING, RUNNING, STARTED.

Complete Example

Scheduling

We define higher-level scheduling constraints, specifically ApplicationDeployment, ApplicationDaemonSet, ApplicationStatefulSet. These are optional; a controller MAY implement them, but is not required to do so.

These follow the same semantics as normal Pod-scheduling Deployment, DaemonSet and ApplicationSet, except that the spec.template.spec references an Application rather than a Pod.

BaseOS

Setting the baseOS version on a device is performed by changing the spec field eve-os-version. Note that the status field eve-os-version reports what version actually is installed and running on the device, while the spec field eve-os-version is the desired version.

If the spec field eve-os-version is blank, then there is no specific request; it simply is whatever was installed on the device. The field should be changed only when requesting a change of the eve-os version on the device. The status field is to be updated when the update is complete.

Controllers may have policies that require an eve-os image be registered on the controller, and possibly in the specific namespace, before allowing it to be applied to a device. This is a policy question for the controller, and is not covered by this specification.

Appendix A - Native Resources

Edge Device

The concepts of a Kubernetes node and an EVE edge device are very similar, with support for various properties. The flows of adding and managing them differ.

EVE

EVE’s API for joining a controller uses a combination of three distinct elements from the device:

• Onboard certificate

• Serial

• Device certificate

EVE’s API does not define how the controller will use those, and in what combination. Nor does it define if a certificate will be validated by advance knowledge of the certificate itself, or by acceptance of the CA that signed the cert, if any.

The controller can use any combination of the above it desires to authenticate and then register the device.

In practice, most controllers provide one or more of the following:

• Device pre-allocated and device certificate itself known in advance.

• Device pre-allocated and onboard certificate itself known in advance.

• Device pre-allocated and combination of onboard certificate and serial known in advance.

• Device not pre-allocated; onboard certificate known and accepted for device registration.