The pandemic has greatly reinforced the need for virtual meetings in both work-related and social settings. It has also highlighted the dire lack of video conferencing tools that can simulate the rich immersive experience of in-person meetings, the current tools often leading to “zoom fatigue’’ caused by having to interact over an unnatural communication medium. This collaborative project brings together investigators from University of Illinois, Urbana-Champaign, University of Massachusetts, Amherst, and New Jersey Institute of Technology to research, build, and evaluate a distributed system, called miVirtualSeat, that more closely simulates the immersive experience of in-person meetings, including physical and virtual participants in a physical meeting space.
The project is focused on key research challenges in providing an immersive meeting experience where physical and virtual participants interact with each other in a physical meeting room. Some participants are virtually present in the physical meeting room, but physically located in remote sites with only limited compute and network resources. The challenges are (a) detecting, tracking, and localizing distributed physical and virtual 360-degree avatars and objects in the joint immersive scene in real-time, (b) reducing the bandwidth and latency of delivering integrated and synchronized 360-degree, volumetric, and 2D/3D video, and ambisonics audio, and (c) ensuring good quality-of-experience in the form of natural interaction between physical and virtual participants.
The project addresses an immediate and important need for a post-pandemic society to enable immersive hybrid meetings that arise in the context of classrooms, conferences, office meetings, and social gatherings. miVirtualSeat will enable these meetings with a physical meeting room and remote sites situated at each of the investigators’ institutions. The outcome of the project will be new undergraduate and graduate courses in the emerging field of advanced mixed reality immersive environments. Through outreach activities, the project members will showcase miVirtualSeat and expose the broader public to the capabilities of distributed AR/VR systems.