eKuiper
Status
Current Project Stage: Stage 1 - At Large
Website: Coming Soon
Wiki: Coming Soon
TAC Sponsors: @Leding Leding (Baidu), @jianguo zeng (Huawei)
Project Overview: TAC Presentation (April 21)
Originally applied under the name Kuiper. Final Open Source Project Name: eKuiper.
Presented during the Wednesday, April 21 TAC call: Meeting Recording (https://zoom.us/rec/share/JKTpoOax0sntLIujoRdoaP-qcfFkcRwlvvoDGo-PYB7JKRn9txqTq-wJEJfpLw_7.MkQNsH_6bYEzJosY?startTime=1619013013000)
TAC two-thirds vote approval reached on May 18, 2021.
Governing Board Strategic Planning Committee approval reached on May 24, 2021.
1. Project Description
What it does
EMQ X Kuiper is an edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices. One goal of Kuiper is to migrate the cloud streaming software frameworks (such as Apache Spark, Apache Flink) to edge side. Kuiper references these cloud streaming frameworks, and also considered a special requirement of edge analytics, and introduced rule engine, which is based on Source, SQL (business logic) and Sinks, the rule engine is used for developing streaming applications at edge side.
Why it is valuable
Lightweight
The core server package is only about 4.5M, the initial memory footprint is about 10MB
Cross-platform
CPU Arch:X86 AMD * 32, X86 AMD * 64; ARM * 32, ARM * 64; PPC
The popular Linux distributions, OpenWrt Linux, MacOS and Docker
Industrial PC, Raspberry Pi, industrial gateway, home gateway, MEC edge cloud server
Data analysis support
Support data extract, transform and filter through SQL
Data order, group, aggregation, and join
60+ functions include mathematical, string, aggregate and hash, etc
4-time windows & count window
Highly extensible
Plugin system is provided, and it supports to extend at Source, SQL functions and Sink.
Source: embedded support for MQTT, and provide extension points for sources
Sink: embedded support for MQTT and HTTP, and provide extension points for sinks
UDF functions: embedded support for 60+ functions, and provide extension points for SQL functions
Management
A web-based management dashboard (opens new window)for nodes, plugins, streams & rules management
Plugins, streams and rules management through CLI & REST API
Easily be integrate with KubeEdge (opens new window), K3s (opens new window) and Baetyl (opens new window), which bases Kubernetes
Integration with EMQ X Edge
Seamless integration with EMQ X Neuron & EMQ X Edge, and provided an end-to-end solution from messaging to analytics.
Origin and history
The project is originated from a fully-functional streaming software created and donated by EMQ. It is developed with feedback and contributions from partners, customers and community users. Below is the milestones of the project.
22, Oct 2020: 1.0.0 - The 1st stable version.
21, Feb 2020: 0.3.2 - The 1st version integrated with the LF EdgeX Foundry project.
23, Oct 2019: 0.2.0 - The 1st released version.
2. Statement of mission
The mission of Kuiper is aligned with LF Edge:
Build Kuiper as a lightweight IoT data analytics / streaming software in edge side
Enable organizations to accelerate the adoption of edge computing by providing handy tools and frameworks
Integrate with relevant edge projects by providing built-in support
Promote edge computing ecosystem by providing mechanisms for extension and integration
3. Project synergy
EdgeX Foundry rules engine, it's reference implementation edge analytics service that performs if-then conditional actuation at the edge based on sensor data collected by the EdgeX instance. Kuiper has partnered with EdgeX Foundry to provide the default rule engine service.
Baetyl project integration. Baetyl project also integrated Kuiper, which can help users to perform data analytics within Baetyle framework.
Fledge: Kuiper has a flexible and powerful data analytic feature that compliments Fledge. One possible way to harmonize both products is that Fledge collects data from all kinds of sensors and then deliver to Kuiper for analysis.
4. Leadership team and decision making process
5. Architecture
The Kuiper core runtime architecture consist a pipeline of sources (built-in or extended), SQL layer and sinks (built-in or extended) as shown below.
The SQL layer components include:
SQL/Rule Parser: Parse commands to orchestrate the pipeline of processors
SQL Processors: The working operators to do the SQL interpreted data transformation
Streaming/SQL runtime: The runtime to deal with streaming like window, event time/watermark and effective once semantic.
Storage: Components to persist rule state and system data.
There are other supportive components include:
Plugin management: Manage plugin creation, loading and deletion for source, function and sink.
Configuration management: Manage system configuration.
Monitoring: provide metrics for rules.
REST API and CLI: expose management API
6. Project Template
Name of Project | EMQ X Kuiper |
---|---|
Project Description (what it does, why it is valuable, origin and history) | EMQ X Kuiper is an edge lightweight IoT data analytics / streaming software implemented by Golang, and it can be run at all kinds of resource constrained edge devices. Considered special requirement of edge analytics, and introduced rule engine, which is based on Source, SQL (business logic) and Sink, rule engine is used for developing streaming applications at edge side. Kuiper is created by EMQ X and open sourced since July 2019. It is then developed with feedback and contributions from partners, customers and community users. |
Statement on alignment with Foundation Mission Statement | The mission of Kuiper is aligned with LF Edge:
|
High level assessment of project synergy with existing projects under LF Edge, including how the project compliments/overlaps with existing projects, and potential ways to harmonize over time. Responses may be included both here and/or in accompanying documentation. | Kuiper is a neutral edge streaming analytics framework, and it can be easily integrated with IoT edge frameworks that require streaming analytics. 1. EdgeX Foundry rules engine, it's reference implementatio |