eKuiper
Welcome to the eKuiper Wiki
Overview
LF Edge eKuiper is a lightweight IoT data analytics and stream processing engine running on resource-constraint edge devices. The major goal for eKuiper is to provide a streaming software framework (similar to Apache Flink) in edge side. eKuiper's rule engine allows user to provide either SQL based or graph based (similar to Node-RED) rules to create IoT edge analytics applications within few minutes.
User scenarios
It can be run at various IoT edge user scenarios, such as,
Real-time processing of production line data in the IIoT
Gateway of connected vehicle analyze the data from CAN in IoV
Real-time analysis of wind turbines and smart bulk energy storage data in smart energy
eKuiper processing at the edge can greatly reduce system response latency, save network bandwidth and storage costs and improve system security.
Features
Lightweight
Core server package is only about 4.5M, memory footprint is about 10MB
Cross-platform
CPU Arch:X86 AMD * 32/64; ARM * 32/64; PPC
Popular Linux distributions, OpenWrt Linux, MacOS and Docker
Industrial PC, Raspberry Pi, industrial gateway, home gateway, MEC edge cloud server
Data analysis support
Support data ETL
Data order, group, aggregation and join with different data sources (the data from databases and files)
60+ functions, includes mathematical, string, aggregate and hash etc
4 time windows & count window
Highly extensible
It supports to extend atÂ
Source
,ÂFunctions
 andÂSink
 with Golang or Python.Source: allows users to add more data source for analytics.
Sink: allows users to send analysis result to different customized systems.
UDF functions: allow users to add customized functions for data analysis (for example, AI/ML function invocation)
Management
A free web based management dashboard for visualized management
Plugins, streams and rules management through CLI, REST API and config maps(Kubernetes)
Easily be integrated with Kubernetes framworks KubeEdge, OpenYurt, K3s Baetyl
Integration with EMQX products
Seamless integration with EMQX, Neuron & NanoMQ, and provided an end-to-end solution from IIoT, IoV
Quick Start
Performance Test Result
MQTT throughput test
Using JMeter MQTT plugin to send simulation data to EMQ X Broker, such as:Â
{"temperature": 10, "humidity" : 90}
, the value of temperature and humidity are random integer between 0 - 100.eKuiper subscribe from EMQ X Broker, and analyze data with SQL:Â
SELECT * FROM demo WHERE temperature > 50
The analysis result are wrote to local file by using file sink plugin.
Devices | Message # per second | CPU usage | Memory usage |
---|---|---|---|
Raspberry Pi 3B+ | 12k | sys+user: 70% | 20M |
AWS t2.micro( 1 Core * 1 GB) | 10k | sys+user: 25% | 20M |
EdgeX throughput test
A Go application is wrote to send data to ZeroMQ message bus, the data is as following.
{ "Device": "demo", "Created": 000, … "readings": [ {"Name": "Temperature", value: "30", "Created":123 …}, {"Name": "Humidity", value: "20", "Created":456 …} ] }
eKuiper subscribe from EdgeX ZeroMQ message bus, and analyze data with SQL:Â
SELECT * FROM demo WHERE temperature > 50
. 90% of data will be filtered by the rule.The analysis result are sent to nop sink, all of the result data will be ignored.
Message # per second | CPU usage | Memory usage | |
---|---|---|---|
AWS t2.micro( 1 Core * 1 GB) | 11.4 k | sys+user: 75% | 32M |
Max number of rules support
8000 rules with 800 message/second
Configurations
2 core * 4GB memory in AWS
Ubuntu
Resource usage
Memory: 89% ~ 72%
CPU: 25%
400KB - 500KB / rule
Rule
Source: MQTT
SQL: SELECT temperature FROM source WHERE temperature > 20 (90% data are filtered)
Sink: Log
Help Us Improve the Wiki
This Wiki is owned by the Secure Device Onboard Community. Contributions are always welcomed to help make it better!
In upper right, select Log In. You will need a Linux Foundation Account (can be created at http://myprofile.linuxfoundation.org/) to log-in. For a Wiki tutorial, please see Confluence Overview. Thank you!