2025-01-07

Owned by Wilson Wang
Last updated: Jan 08, 2025
5 min read

Agenda

  1. CES Recap @Tom Qin -->Postponed to 01/17

  2. Workstream 3 Update. @Wilson Wang @Borui Li (李博睿)

  3. Workstream 6 Robotics, inspired by CES @Haruhisa Fukano @Yonghua Lin

 

Participants

@Wilson Wang @Chun Liu @Gary Qi @Mengyu Hang @Ning Jiang @Jianwen Pi @Jiayi Song @Joseph Sun @Lance Lim @Qingkun Li @Ning Bao @Ziang Song @Sujata Tibrewala @Tiger Li @Tina Tsou @Wenhui Zhang @Hao Xie @Xun Chen @Ye Xu @Ying Wang @Yizheng Jiao

Recording

https://bytedance.us.larkoffice.com/minutes/obuszg64xws7umfx2t2l3g86

Summary

  • Workstream 3 Update. @Wilson Wang @Borui Li (李博睿)
    Release 1 is entire of SPEAR. But some have not been developed.
    Discussed next milestone in last week meeting.
    There are two directions.
    1.Define MVP, and what is priority component/function
    2.Advertise SPEAR to industry and academic

  • Workstream 6 Robotics, inspired by CES
    Need to study about physical AI model e.g. NVIDIA cosmos

The meeting discussed the progress and plans of various work streams, as well as some technical topics and potential collaborations, the main contents included:

  • Work Stream 3 Update: Borry recruited new developers for the next release and is working on two directions for the next milestone, including system workflow and research paper.

  • Work Stream 6 Brainstorming: Focused on how to be inspired by CS and physical AI in robotics, discussing Nvidia's physical AI, training challenges, edge computing, etc.

  • Lidar in Robotics: Lidar is considered essential for physical AI in robotics for object detection and safety, and the need for related open source processing.

  • Ray Technology: Brainstormed about using Ray in multi - model language models for audio - video processing.

  • OKR and Infinite AI Collaboration: Discussed the combination of CD environments between OKR and Infinite AI for the next release.

  • Edge Gateway Update: Work Stream 5 is currently focusing on supporting Ross 2 for edge gateway in robotics.

  • Security Support: Victor may collaborate on security aspects as several work streams need security support.

Chapters

00:07 Meeting discussion on project progress after Release 1

This section begins with Tina Tsou starting the meeting a few minutes late. They discuss the meeting agenda, which includes inviting Tom to recap events in CS at the next meeting. They then move to the first item and Tina asks Borry for an update on progress since Release 1, with Borry sharing that he brought more students to the project and they're working on work stream 3's next milestone. Borry then prepares to share his screen.

05:53 Discussion on Spear System's Next Release and Future Plan, and Brief on Robotics Work Stream

This section focused on discussions about the next release and future plan of Release 3 of the spear system. In Release 1, not all planned features were implemented. Plans include bringing more students, preparing a complete workflow and MVP framework, advertising the system in industry and academia by preparing a research paper. There was also talk about using low or no code frameworks and potential collaborations. Then the discussion moved to work stream 6 Robotics.

14:08 Discussion on Nvidia's Physical AI, Focusing on Robotics and Automated PCs

This section focuses on Nvidia's emphasis on physical AI at CES, which consists of three main areas including robotics. Challenges in simulating and learning robotics in the physical world are discussed, like limited spaces. Digital twins are considered important. There's uncertainty about the tool chain and training. Automated PCs are seen as an area where products might come first. Edge computing suitability for physical AI cases is also raised.

20:03 Discussion on Real-time Interaction and Challenges in Physical AI with Focus on Lidar Potential

This section discusses the need for real-time interaction in robotics. As home vacuuming robots become more complex, there are risks like potential harm to babies. Incorporating open source physical AI models like whisper is considered. Feature recognition is crucial for various robotic applications. Speech recognition is important for giving instructions. Lidar is suggested as essential for physical AI to accurately measure and avoid crashes, beneficial for both industrial and consumer robots.

26:09 Discussion on Using Ray for Audio-Video Processing and Lidar Data in Multi-Model Language Models

This section discusses audio video processing for multi-modal language models. The use of Ray, an open source component with a large community, is considered, including a planned meetup next month. There's talk of using it for audio video data processing. Additionally, the limitations of certain technology under $20 US are mentioned, along with the role of Lidar in creating 3D perspectives and the need for open source to process its images for object recognition and depth measurement.

30:24 Discussion on Robotics, Livestreaming and Open Data Contribution

This section discussed topics related to robotics and livestreaming. For robotics, considerations included safety and efficiency, with mention of reinforcement learning. Open source models from Qing Kun University were noted. A volunteer planned to talk to the founder of Ray. Regarding livestreaming, discussion involved processing and features like muting. Yong Hua wanted to contribute to open data but had a meeting conflict. The date of OPHC was also briefly discussed.

35:47 Discussion on Integrating OKR and Infinite AI CD Environments

This section discusses Malini's return to the Bay Area and an offline sync up with OKR. It focuses on the thought process for the Infinite AI release 1.1, particularly combining CICD with the OKR CD environment. Ruoyu Ying mentions the current status of finding a proper environment. Wilson Wang says finding metrics is an issue and plans to work with Johan to push it forward. There's also discussion about potential changes to the CD environment.

41:16 Discussion on CD Lark Release 1 and Edge Gateway Plans

This section discussed the release 1 of CD Lark, noting that with reuse and the OPH's compatibility, there may be no need for major changes to the city system. It also touched on the plan for work stream 5 for edge gateway, including focus on framework support and work on robotic-based systems like ROS 2. There was discussion about the differences between ROS 2 and other systems, with some aspects still under study.

46:22 Discussion on Robotic Support, Operating System, Security and Architecture Planning

This section discusses robotic support for Ross 2 and Blake, and whether it's an open source or proprietary operating system platform. It mentions reasons for considering support like integrating robotics and IoT devices. There's also talk about security contributions, with a baseline from open SSF, and uncertainties about how to contribute. The main architect of the whole architecture and work stream leaders are discussed, and a planning for version 1.1 is set for next week.

51:42 Discussion on Security Requirements and Next Steps in 2025 Meeting

This section discusses security as a potential feature for version 1.1, along with general security requirements. There's uncertainty about whether security scans are sufficient to meet requirements. Victor is encouraged to review and suggest changes. An unknown speaker will touch base with Lucy and follow up with Wilson. The first successful meeting of 2025 is concluded, and the speaker will follow up with Sunny Chai about publishing a release announcement.