Work stream 2: Edge Database & Algorithms

• Leader: @Rick Cao @Qi Tang

• Objective: The Edge Database work stream is dedicated to advancing database solutions tailored for edge computing environments. It targets improvements in data handling and storage capabilities on edge devices, enhancing local data processing and decision-making.

• Approach: Efforts will include the development of lightweight, scalable database systems that support real-time data processing and analytics, pivotal for Edge AI Virtual Agents.

Background

Our initial proposal covers some use cases (Slide #4) of AI virtual agents on the edge. 

We are open for more real-life use cases and edge business service ideas. 

Current direction is to implement AI agent LLM service (e.g. customer service on the edge).

Objectives

Two long-term goals of this work stream: 

1) (Interoperability) Define clean, standardized and lightweight data and data processing interface for AI LLM services on the edge

2) (non-expert) Deliver non-expert, on-premise and low-cost AI virtual agent solution for local and small businesses (e.g. FAQs / customer LLM services in a local store)

Features

Data content / context

• Real-time information

• Location-specific information

• Event information

• Function calling (_functions, python scripts)

Input Data types

• Unstructured: PDF, HTML, Audio, Image, Video, etc.

• Structured: SQL, vector stores, knowledge graphs

• Files: Json, CSV

• APIs

Off-the-shelf vector database and embedding models

1. Vector database

2. LangChain

3. LlamaIndex

4. Sentence transformers 

Data Access Authentication

1. Public access

2. Private data (Enterprise use cases) 

3. Protocols

Algorithms

1. semantic search

2. data chunking

3. ranking

4. recommendation

5. Lora Adaptor 

6. data routing

Evaluation and analytics

1. Click through rate (from the reference links)

2. User feedback

3. Experiment

An Example Scenario

1. Company A wants to leverage AI to build a powerful agent-like services with its current data (SQL, CSV, PDF, …)

   1. Build a lightweight database for efficient retrieval

2. The service is successfully deployed and starts to receive customer’s feedback (e.g. analytics, LLM inputs/outputs stored)

   1. Is there a way to refine the current solution to improve the services?

   2. Store the LLM inputs/outputs data efficiently and labeled properly

3. Train / Tune a model to gain extra capability of the current given the new data

   1. Data serialization / de-serialization for scalability

Timeline

Phase 1: (Prototyping) 1) adopt off-the-shelf solutions; 2) provide benchmark results; 3) develop the evaluation matrix for edge database/algorithm solution

Phase 2: (Standardization) Based on the lessons learnt, consider the scalability and interoperability

Version history

(08/20/2024) initial proposal, outlines of work scope, objective, and approaches

(11/05/2024) First design and implementation: https://docs.google.com/presentation/d/19I07kyHKRLvaatBtbagcpdoaicc_UWMcqMzsGc6wDic/edit#slide=id.g312466b3fbd_0_0

(11/19/2024) First demo

Before: User selects the defined-product that meets the need closely

Now:  User defines how they want to use the data, and the code / product generated for this specific need automatically