Conversational AI in VRChat 2

What's this?

• Introduction of a voicebot that implemented conversational AI in VRChat
  • https://youtu.be/j9L51pASeiQ

System

• ASR: googletrans
  • https://pypi.org/project/googletrans/
• Text Generation: Rinna (1.3B-parameter Japanese GPT model)
  • https://huggingface.co/rinna/japanese-gpt-1b

• TTS: tyc-talksoft

  • https://shirowanisan.com/tyc-talksoft

• Sound Pipeline

  • application -> speaker -> dialog.py -> mic -> application
  • Direct loop also exists.. (application -> speaker -> mic -> application)

• (TBD) Release the scripts in GitHub

  • Need to clean the codes and write documents..

Prerequisites

• Ubuntu 18.04.5 LTS
• Voice chat application (VRChat for example)

Setup


$ sudo apt install portaudio19-dev pavucontrol python-xlib scrot
$ git clone https://github.com/Lunran/sts.git
$ cd sts
$ python3 -m venv venv
$ . venv/bin/activate
$ pip install --upgrade pip
$ pip install -r requirements.txt


Usage

• Run VRChat
  • Open "Settings", "Audio Volume"
  • Mute "UI/Menu", "World" and "Avatars"
• Run the script
  • refer to "$ python dialog.py -h" for details