Conversational AI in VRChat 1

What's this?

• Introduction of a voicebot that implemented conversational AI in VRChat
  • https://youtu.be/8u22YDqhY6Y

System

• ASR: googletrans
  • https://pypi.org/project/googletrans/
• Text Generation: ILYS-aoba-chatbot (seq2seq model pretrained with Japanese dialog data)
  • https://github.com/cl-tohoku/ILYS-aoba-chatbot

• TTS: tyc-talksoft

  • https://shirowanisan.com/tyc-talksoft

• Sound Pipeline

  • application -> speaker -> dialog.py -> mic -> application
  • Direct loop also exists.. (application -> speaker -> mic -> application)

• (TBD) Release the scripts in GitHub

  • Need to clean the codes and write documents..

Prerequisites

• Ubuntu 18.04.5 LTS
• Voice chat application (VRChat for example)

Setup


$ sudo apt install portaudio19-dev pavucontrol python-xlib scrot
$ git clone https://github.com/Lunran/sts.git
$ cd sts
$ python3 -m venv venv
$ . venv/bin/activate
$ pip install --upgrade pip
$ pip install -r requirements.txt


Usage

• Run VRChat
  • Open "Settings", "Audio Volume"
  • Mute "UI/Menu", "World" and "Avatars"
• Run the script
  • refer to "$ python dialog.py -h" for details