Work stream 2: Edge Database & Algorithms

• Leader: @Rick Cao @Qi Tang

• Objective: The Edge Database work stream is dedicated to advancing database solutions tailored for edge computing environments. It targets improvements in data handling and storage capabilities on edge devices, enhancing local data processing and decision-making.

• Approach: Efforts will include the development of lightweight, scalable database systems that support real-time data processing and analytics, pivotal for Edge AI Virtual Agents.

Background

Our initial proposal covers some use cases (Slide #4) of AI virtual agents on the edge. 

We are open for more real-life use cases and edge business service ideas. 

Current direction is to implement AI agent LLM service (e.g. customer service on the edge).

Objectives

Two long-term goals of this work stream: 

1) (Interoperability) Define clean, standardized and lightweight data and data processing interface for AI LLM services on the edge

2) (non-expert) Deliver non-expert, on-premise and low-cost AI virtual agent solution for local and small businesses (e.g. FAQs / customer LLM services in a local store)

Features

Data content / context

• Real-time information

• Location-specific information

• Event information

• Function calling (_functions, python scripts)

Input Data types

• Unstructured: PDF, HTML, Audio, Image, Video, etc.

• Structured: SQL, vector stores, knowledge graphs

• Files: Json, CSV

• APIs

Off-the-shelf vector database and embedding models

1. Vector database

2. LangChain

3. LlamaIndex

4. Sentence transformers 

Data Access Authentication

1. Public access

2. Private data (Enterprise use cases) 

3. Protocols

Algorithms

1. semantic search

2. data chunking

3. ranking

4. recommendation

5. Lora Adaptor 

6. data routing

Evaluation and analytics

1. Click through rate (from the reference links)

2. User feedback

3. Experiment

An Example Scenario

1. Company A wants to leverage AI to build a powerful agent-like services with its current data (SQL, CSV, PDF, …)

   1. Build a lightweight database for efficient retrieval

2. The service is successfully deployed and starts to receive customer’s feedback (e.g. analytics, LLM inputs/outputs stored)

   1. Is there a way to refine the current solution to improve the services?

   2. Store the LLM inputs/outputs data efficiently and labeled properly

3. Train / Tune a model to gain extra capability of the current given the new data

   1. Data serialization / de-serialization for scalability

Timeline

Phase 1: (Prototyping) 1) adopt off-the-shelf solutions; 2) provide benchmark results; 3) develop the evaluation matrix for edge database/algorithm solution

Phase 2: (Standardization) Based on the lessons learnt, consider the scalability and interoperability

Version history

(08/20/2024) initial proposal, outlines of work scope, objective, and approaches

(11/05/2024) First design and implementation: https://docs.google.com/presentation/d/19I07kyHKRLvaatBtbagcpdoaicc_UWMcqMzsGc6wDic/edit#slide=id.g312466b3fbd_0_0

(11/19/2024) First demo

Before: User selects the defined-product that meets the need closely

Now:  User defines how they want to use the data, and the code / product generated for this specific need automatically

(02/25/2025) Initial tech stacks:

1. Huggingface/smolagents is used for the underlying agent framework

2. Div99/agent-protocol (to be investigated) for the API framework

3. E2B (to be investigated) used for sandbox code execution

4. llama_index is used for the data retrieval framework

5. SQLalchemy is used for structured data retrieval framework

Note: What is a data agent? Shall we choose to hardcode a retriever with a few lines of codes, or ask the LLM to generate the code? In analogy to autonomous driving, the L5 of self-driving can transport the user from Place A to B without any intervention (no need to learn to drive) even with the road environment has not been seen before. So the ultimate goal of this agent implementation can be no codes shall be code implementation? Any codes becomes a posteriori or by-product that if the codes are useful then we might not need to re-code it again. Anyway, some thoughts I realize when I start the coding of this project:

1. minimalist → any codes will be replaced by the future AI code agent in anyway

2. AI agent is bounded by resource and environments (dependencies, low-level kernels, computer resources, data size, …) so the agent should be aware of the resources (in analogy to the self-driving car should be aware of the environments)

3. LLM is really good at creating ideas, plans, and implementations, but still lack of intentions, or how to fit to user’s intention

Some use cases to be done

1. User manual (50 pages of pdf) turned into a QA agent

2. Synthetic hospital patients documents for metadata retrieval

3. Sensory data csv file into some pandas code

4. Regulatory / Compliance document (pdf) into compliance check agent

(03/18/2025) MVP release GitHub - lfedgeai/eda: Data on-Prem, Code on-the-Fly

1. eda cli commands:

   1. eda rag “please retrieve the data and build a rag system for ../examples/data/pdf”

   2. eda run “you can ask any request here about the data base”

   3. eda learn “learn my database thoroughly ../examples/data/3gpp/" #for large and complicated database

   4. eda mcp “please serve the data under ../examples/data/finance externally 0.0.0.0"

      Open

      

       

2. There are 3 levels of intelligence for the data retrieval agents:

   1. Level 1: eda rag ⇒ rag.py python code ⇒ most efficient (compute only), no LLM tokens

      Level 2: eda run ⇒ multi-step reflections ⇒ query LLM for better summarization / search

      Level 3: eda learn ⇒ knowledge graph and memGPT ⇒ costly, but full understanding of the doc