Publications

Look2React: Making VR NPCs Come Alive with Dynamic Vision-Guided Reactions

Published in The 33rd IEEE Conference on Virtual Reality and 3D User Interfaces (VR ’26), 2025

We present Look2React, an interaction system that captures what NPCs see, using a vision-based reasoning model to select pose and text responses. As such, Look2React endows NPCs with the ability to react dynamically and appropriately to player interactions.

Recommended citation: R. Vatsal, X. Huang and R. Xiao, "Look2React: Making VR NPCs Come Alive with Dynamic Vision-Guided Reactions," in IEEE Transactions on Visualization and Computer Graphics. https://doi.org/10.1109/TVCG.2026.3679121

MultiSphere: Latency Optimized Multi-User 360° VR Telepresence with Edge-Assisted Viewport Adaptive IPv6 Multicast

Published in The 31st ACM Symposium on Virtual Reality Software and Technology (VRST ’25), 2025

We present MultiSphere, a multi-user edge-assisted 360° VR telepresence system, that combines viewport-adaptive IPv6 multicast tiling with a novel dual keyframe interval (KeyInt) streaming technique.

Recommended citation: Dieter Frehlich, Xincheng Huang, and Robert Xiao. 2025. MultiSphere: Latency Optimized Multi-User 360° VR Telepresence with Edge-Assisted Viewport Adaptive IPv6 Multicast. In 31st ACM Symposium on Virtual Reality Software and Technology (VRST ’25), November 12–14, 2025, Montreal, QC, Canada. ACM, New York, NY, USA, 2 pages. https://doi.org/10.1145/3756884.3770539

TangiAR: Markerless Tangible Input for Immersive Augmented Reality with Everyday Objects

Published in The 31st ACM Symposium on Virtual Reality Software and Technology (VRST ’25), 2025

We propose TangiAR, a tangible control system which tracks everyday objects without the need for fiducial trackers, enabling them as passive controllers and virtual proxies in AR applications.

Recommended citation: Xu Fan, Xincheng Huang, and Robert Xiao. 2025. TangiAR: Markerless Tangible Input for Immersive Augmented Reality with Everyday Objects. In 31st ACM Symposium on Virtual Reality Software and Technology (VRST ’25), November 12–14, 2025, Montreal, QC, Canada. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/3756884.3766028

GaussianNexus: Room-Scale Real-Time AR/VR Telepresence with Gaussian Splatting

Published in The 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25), 2025

A Room-scale real-time AR/VR telepresence system with Gaussian Splatting. Our system provides 2D live updates with rectified video stream captured by the 360° camera. It then offers 3D live updates by tracking the movements of individual physical objects pre-selected by the users.

Recommended citation: Xincheng Huang, Dieter Frehlich, Ziyi Xia, Peyman Gholami, and Robert Xiao. 2025. GaussianNexus: Room-Scale Real-Time AR/VR Telepresence with Gaussian Splatting. In The 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25), September 28-October 1, 2025, Busan, Republic of Korea. ACM, New York, NY, USA, 18 pages. https://doi.org/10.1145/3746059.3747693

VibRing: A Wearable Vibroacoustic Sensor for Single-Handed Gesture Recognition

Published in Proceeding of the ACM on Human-Computer Interaction. Volume 9, Issue 4, Article EICS006, June, 2025

A lightweight wireless hand-gesture sensing platform, leveraging a single finger-worn vibroacoustic sensor.

Recommended citation: Bu Li, Xincheng Huang, and Robert Xiao. 2025. VibRing: A Wearable Vibroacoustic Sensor for Single-Handed Gesture Recognition. Proc. ACM Hum.-Comput. Interact. 9, 4, Article EICS006 (June 2025), 25 pages. https://doi.org/10.1145/3733052

HaloTouch: Using IR Multi-Path Interference to Support Touch Interactions with General Surfaces

Published in Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25), 2025

A vision-based system which exploits a multipath interference effect from an off-the-shelf time-of-flight depth camera to enable fast, accurate touch interactions on general surfaces.

Recommended citation: Ziyi Xia, Xincheng Huang, Sidney S Fels, and Robert Xiao. 2025. HaloTouch: Using IR Multi-Path Interference to Support Touch Interactions with General Surfaces. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25). Association for Computing Machinery, New York, NY, USA, Article 548, 1–17. https://doi.org/10.1145/3706598.3714179

VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual Replicas

Published in The 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24), 2024

A 360° Video AR/VR Remote Collaborative System with Enhanced Interactivity. The system enables better access to the remote environment with Environment Cutouts that can be pulled closer, and Virtual Replcas created with Instant-ngp.

Recommended citation: Xincheng Huang, Michael Yin, Ziyi Xia, and Robert Xiao. 2024. VirtualNexus: Enhancing 360-Degree Video AR/VR Collaboration with Environment Cutouts and Virtual Replicas. In The 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24), October 13–16, 2024, Pittsburgh, PA, USA. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3654777.3676377

SurfShare: Lightweight Spatially Consistent Physical Surface and Virtual Replica Sharing with Head-mounted Mixed-Reality

Published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Volume 7, Issue 4., 2023

A lightweight symmetric and synchronous remote mixed-reality collaboration system with ad-hoc sketch-based virtual content creation.

Recommended citation: Xincheng Huang and Robert Xiao. 2023. SurfShare: Lightweight Spatially Consistent Physical Surface and Virtual Replica Sharing with Head-mounted Mixed-Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 4, Article 162 (December 2023), 24 pages. https://doi.org/10.1145/3631418

Virtual Reality Telepresence: 360-Degree Video Streaming with Edge-Compute Assisted Static Foveated Compression

Published in IEEE Transactions on Visualization and Computer Graphics, 2023

A 360° virtual reality telepresence system with edge-compute assisted static foveated compression

Recommended citation: Xincheng Huang, James Riddell, and Robert Xiao, "Virtual Reality Telepresence: 360-Degree Video Streaming with Edge-Compute Assisted Static Foveated Compression," in IEEE Transactions on Visualization and Computer Graphics. https://doi.org/10.1109/TVCG.2023.3320255

StructureSense: Inferring Constructive Assembly Structures from User Behaviors

Published in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. Volume 6, Issue 4., 2022

A tracking system based on passive UHF-RFID sensing that infers constructive assembly structures from object motion.

Recommended citation: Xincheng Huang, Keylonnie L Miller, Alanson P. Sample, and Nikola Banovic. 2022. StructureSense: Inferring Constructive Assembly Structures from User Behaviors. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 4, Article 204 (December 2022), 25 pages. https://doi.org/10.1145/3570343