miVirtualSeat: Semantics-aware Content Distribution for Immersive Meeting Environments

People

  • Klara Nahrstedt, University of Illinois, Urbana-Champaign (UIUC): Principle Investigator
  • Ramesh K Sitaraman, University of Massachusetts, Amherst (UMass); Principal Investigator
  • Michael H Zink, University of Massachusetts, Amherst (UMass): Principle Investigator
  • Jacob Chakareski, New Jersey Institute of Technology (NJIT): Principle Investigator

Current Research Assistants

  • Bo Chen (Postdoctoral Fellow) – UIUC
  • Ryan Zhang (Postdoctoral Fellow) – UIUC
  • Mingyuan Wu (PhD  Student) – UIUC
  • Eric Lee (PhD Student) – UIUC
  • John Murray (PhD Student) – UMass
  • Siamak Beikzadeh (PhD Student) – UMass
  • Tianyu (Kevin) Chen (PhD Student) – UMass
  • Lingdong Wang (PhD Student) – UMass
  • Tung-I Chen (PhD Student) – UMass
  • Cheng-Feng Liu (Postdoc) – NJIT
  • Xiaoyan Zhou (Postdoc) – NJIT
  • Zhiheng Xu (Graduate Student) – NJIT
  • Phuoc Dang (Graduate Student) – NJIT
  • Alireza Hossein (Graduate Student) – NJIT

Past Research Assistants

  • Qian Zhou (Postdoctoral Fellow) – UIUC
  • Shiv Tripedi (Undergraduate Student) – UIUC
  • Yuhan Lu (Undergraduate Student) – UIUC
  • Siamak Beikzadeh (PhD Student) – UMass
  • Simran Singh (Postdoctoral Fellow) – NJIT
  • Amirhoseing Aghaei (PhD Student) – NJIT
  • Mohammad Reza Elahpour (PhD Student) – NJIT

Project Overview

The pandemic has greatly reinforced the need for virtual meetings in both work-related and social settings. It has also highlighted the dire lack of video conferencing tools that can simulate the rich immersive experience of in-person meetings, the current tools often leading to “zoom fatigue’’ caused by having to interact over an unnatural communication medium.  This collaborative project brings together investigators from University of Illinois, Urbana-Champaign, University of Massachusetts, Amherst, and New Jersey Institute of Technology to research, build, and evaluate a distributed system, called miVirtualSeat, that more closely simulates the immersive experience of in-person meetings, including physical and virtual participants in a physical meeting space.

The project is focused on key research challenges in providing an immersive meeting experience where physical and virtual participants interact with each other in a physical meeting room. Some participants are virtually present in the physical meeting room, but physically located in remote sites with only limited compute and network resources. The challenges are (a) detecting, tracking, and localizing distributed physical and virtual 360-degree avatars and objects in the joint immersive scene in real-time, (b) reducing the bandwidth and latency of delivering integrated and synchronized 360-degree, volumetric, and 2D/3D video, and ambisonics audio, and (c) ensuring good quality-of-experience in the form of natural interaction between physical and virtual participants.

The project addresses an immediate and important need for a post-pandemic society to enable immersive hybrid meetings that arise in the context of classrooms, conferences, office meetings, and social gatherings. miVirtualSeat will enable these meetings with a physical meeting room and remote sites situated at each of the investigators’ institutions. The outcome of the project will be new undergraduate and graduate courses in the emerging field of advanced mixed reality immersive environments. Through outreach activities, the project members will showcase miVirtualSeat and expose the broader public to the capabilities of distributed AR/VR systems.

Augmented Scene Graph Generation for content analysis. One of the design goals of the semantics-aware teleconferencing system is to extract and analyze valuable contextual information in the physical meeting environment. Given mutiple sensor stream in the physical room as inputs, our system can perform ML-based scene analysis algorithms, real-timely detect and decompose in-room events of pre-defined categories into human, meeting-related objects and relationship between them. The compositional information can be formulated as a structured graph representation, where human and meeting-related objects are detected and localized as nodes and the relationship between human and objects are recorded and classified as edges between nodes. The graph representation contributes to a foveated rendering pipeline in our teleconferencing system, preserving video quality of important sub-regions and saving bandwidth. Moreover, semantics-aware system can potentially utilize the event information extracted from the joint immersive environment to benefit the virtual participant’s experience by switching camera views and setting automatic interactive sessions on selected events.

WebRTC/RTMP streaming investigation for content delivery.  A cross-campus real-time video sreaming pipeline serves as a crucial submodule in this project. Real-Time Messaging Protocol (RTMP) and Web Real-Time Communication (WebRTC) solutions are investiagted and implemented. RTMP solution with an open-source media streaming software NGNIX.  Upstreaming is done by FFmpeg or OBS, while done streaming is done by FFplay. The solution achieves an approximate latency around 700 ms when tested on servers with public IP. In the WebRTC streaming protocol solution, peers contact with Session Traversal Utilities for NAT (STUN) for public IP first. Afterwards, peers exchange Session Description Protocol (SDP) information within a connection to a signaling server before peer to peer connection is established. Peers would connect to a Traversal Using Relays around NAT (TRUN) server if the connection fails. The solution achieves an approximate latency around 400ms. More experiments on the latency in different network condition and different NAT types are expected in the future. More group meeting streaming designs are under exploration. 

HoloLens2 investigations for content receiving (in progress). HoloLens2 is a pair of mixed reality smartglasses developed and manufactured by Microsoft. It is adopted as the major mixed reality product for providing a rich immersive meeting experience in this project.

The compatibility of webRTC on HoloLens 2 is under exploration and could potentially be achieved by the Unity integration of an open-source library, MixedReality-WebRTC. Besides the mixreality video streaming interface, real-time avatar rendering applications on HoloLens 2 are expected to be developed in this teleconferencing system. 

Publications

  1. M. Wu*, R. Ji* (* indicates equal contribution), H. Zheng, J. Li, B. Tian, B. Chen, R. Zhang, M. Zink, RK.  Sitaraman, J. Chakareski, K. Nahrstedt, “Scene Graph Driven Hybrid Interactive VR Teleconferencing”, ACM Multimedia, Oct 28-Nov 1, 2024, Melbourne, Australia (Video demonstration)
  2. A. Zeynali, M. Hajiesmaili, RK. Sitaraman, “Bola360: Near-Optimal View and Bitrate Adaptation for 360 degree Video Streaming”, ACM Multimedia Systems 2024
  3. T. Chen, Y. Lin, N. Christianson, Z. Akhtar, S. Dharmaji, M. Hajiesmaili, A. Wierman, RK. Sitaraman, “SODA: An Adaptive Bitrate Controller for Consistent High-Quality Video Streaming”, ACM SIGCOMM 2024. 
  4. J. Chakareski and M. Khan, “Live 360 video streaming to heterogeneous clients in 5G networks,” IEEE Trans. Multimedia, vol. 26, 2024.
  5. B. Badnava, J. Chakareski, and M. Hashemi, “Multi-task decision-making for multi-user 360 video processing over wireless networks,” in Proc. Int’l Conf. Multimedia Information Processing and Retrieval. San Jose, CA, USA: IEEE, Aug. 2024, invited paper.
  6. S. Shippey, S. Srinivasan, H. P. Dang, E. Aryafar, and J. Chakareski, “An experimental evaluation of 360-degree ABR video streaming over mmWave wireless links,” in Proc. Int’l Conf. Metaverse Computing, Networking, and Applications. Hong Kong, China: IEEE, Aug. 2024.

  7. L. Wang, S. Singh, J. Chakareski, M. Hajiesmaili, and R. K. Sitaraman, “BONES: Near-optimal neural-enhanced video streaming,” Proceedings of the ACM on Measurement and Analysis of Computing Systems (SIGMETRICS), vol. 8, no. 2, p. 28, June 2024

  8. B. Chen, M. Wu, H. Guo, Z. Yan, Klara Nahrstedt, “Vesper: Learning to Manage Uncertainty in Video Streaming”, ACM Multimedia Systems (MMSys 2024), April 15-18, 2024, Bary, Italy, https://doi.org/10.1145/3625468.3647
  9. M. Wu, Y. Lu, S. Trivedi, B. Chen, Q. Zhou, L. Wang, S. Singh, M. Zink, RK. Sitaraman, J. Chakareski, K. Nahrstedt, “Interactive Scene Analysis for Teleconferencing”, IEEE International Symposium on Multimedia (ISM 2023), Los Angeles, USA, December 11-13, 2023 (short paper) , DOI: 10.1109/ISM59092.2023.00048
    1.  

  10. J. Chakareski, X. Corbillon, G. Simon, and V. Swaminathan, “User navigation modeling, rate-distortion analysis, and end-to-end optimization for viewport-driven 360  video streaming,” IEEE Trans. Multimedia, vol. 25, Nov. 2023.

 

Talks

 

K. Nahrstedt, “End-To-End Perspective of 3600 Video: From Generation and Navigation to Streaming and Viewing”, Invited Lecture, 2023, MIT, Boston, MA.

K. Nahrstedt, “Is MetaVerse Dead?” Panelist at IEEE MetaCom 2023, Kyoto, Japan.

K. Nahrstedt, “Evolution of Adaptive Behaviors in Mobile and Wireless Multimedia”, Keynote at Center MAKI Retreat, May 2024, Darmstadt, Germany

R. Sitaraman, Keynote Speaker, “Video Content Delivery: The First Quarter Century and Beyond,” ACM SIG- COMM Workshop on Emerging Multimedia Systems (EMS), September 2023.

R. Sitaraman, Invited Speaker, “Living on the Edge for a Quarter Century: A Personal Retrospective,” Systems- Theory Seminar, School of Engineering and Applied Sciences, Harvard University, Oct 2023.

J. Chakareski, Invited Talk, 6th Buffalo Day for 5G and Wireless Internet of Things, October, 2024 https://urldefense.com/v3/__https://www.eng.buffalo.edu/*nmastron/wirelessday/2024.html__;fg!!DZ3fjg!7xnHWsahHPugJEigJ6pah4X3ANVV6i-eo8vPU7zMTrbgAgCCDsS8utQb0cKKXXpliLXs39tsz1E3EOsTuLs$

Software

Software is being uploaded to https://github.com/ECE-ZINK/miVirtualSeat

Funding

This work was funded by the National Science Foundation NSF 2106592.