Multilingual LLM AI VR/AR/MR Chatbot Avatar
A multilingual LLM AI chatbot avatar NPC for Meta Quest 3, built as a mixed reality experience for the Askwith Kenner Immersive Technology Room for Language and Culture. Features an elf wizard character ("Faelyn Cardithas") who speaks English, French, German, and Japanese — powered by Wit.ai speech recognition, ConvAI/ChatGPT, and expressive blend shape animations in a fantasy VR/MR environment.
Overview
Built as a mixed reality experience for the Askwith Kenner Immersive Technology Room for Language and Culture, this project demonstrates the potential for cross-cultural, multilingual AI experiences in immersive mixed reality. Rather than building a basic starter template, I pushed the scope to create something genuinely novel and expandable.
The result: Faelyn Cardithas, a young elf wizard who is an “interdimensional cosmopolitan traveler” — an LLM-powered AI NPC who speaks English, French, German, and Japanese, set in a hand-crafted fantasy VR/MR world built entirely from free assets.
Tech Stack
- Meta XR All-in-One SDK — Quest 3 MR passthrough and VR modes
- ConvAI (ChatGPT-powered) — NPC LLM AI, multilingual TTS, and lip sync
- Wit.ai — built-in Meta speech recognition (hold A on right controller to speak)
- Unity 2022.3.54f1 — URP pipeline, 64-bit Android build target
- ElevenLabs — evaluated but excluded (no free Unity+ConvAI integration tier)
Key Features
- Real-time multilingual speech recognition via Wit.ai
- LLM-driven conversation using ConvAI / ChatGPT with custom character backstory
- Expressive facial animation and lip sync synced to TTS output
- Meta Quest 3 MR passthrough — character appears in your real space
- Fully enclosed VR mode also supported
- Open-source template so the painful SDK setup never has to be repeated
Demo
- Download MR APK (Meta Quest 3) — Mixed Reality passthrough mode
- Download VR APK (Meta Quest 3) — Fully enclosed Virtual Reality mode
- Demo video — MR mode
- Demo video — VR mode
Open-source template: Unity URP XR NPC AI
Gallery
