New year, new virtual you: a conversation with Kalidoface creator Rich Yee

_The coming of a new year is a great time to imagine what cutting edge, interactive technology we’ll see trending in web app creation. One trend that I hope to see grow ever more popular on Glitch is the use of body and face motion capture. So, it was a real delight to see Rich Yee launch Kalidoface on Glitch. Rich, a full-time visual designer and free-time creative technologist, has been prolific in the Glitch community, collaborating on Whirly Birdie and creating one of my personal favorites, THEREMIX_. I reached out to him to talk more about virtual character technology and how he used Glitch to bring it into the browser.

Jenn: For those who are not familiar, can you explain what virtual characters or VTubers are? And why do you think they are such a compelling technology these days?


Rich: Vtubers, also known as Virtual Youtubers, are part of a rapidly growing genre on platforms like Youtube or Twitch where creators will base their identities around a digital avatar. While the early days of "Vtubing" (mid 2010s) involved a more expensive full body motion capture setup, the genre has slowly shifted towards face tracking connected to Live2D model, a setup better suited towards streaming. Live2D is a popular character animation format found in mobile games. Kalidoface currently uses official Live2D demo models and also allows users to upload their own Live2D models.

This technology has been a super fun way of merging fiction with reality. While there is backstory and lore behind each VTuber, the creator's voices and subtly animated expressions allow for their personalities to shine through, becoming a mix of fantasy and a genuinely human connection. I definitely couldn't have experienced this much character development by just watching seasonal anime.

Kalidoface is cute, but it’s also really powerful - not only letting users create a virtual character, but also communicate with others. What inspired you to create this app?


Kalidoface aims to bring native-app-like face capture to the browser. Without needing to install anything, I wanted to create an easy way to animate your own avatars and introduce the concept of “Vtubers” to more people.

In the early days of development, I surprised some coworkers in Zoom calls using Kalidoface + OBS. However, it wasn't until I added one-on-one voice calling that I felt truly immersed in the world I was trying to make. There was just something special about seeing my long time friend as a talking dog while I was a talking cat that made the moment magical. In fact, I've been able to reconnect with old friends who were more willing to chat using Kalidoface over a regular video call.

How long was the process from getting the idea to sharing it with the world?


I started playing around with the idea around early October, trying to get a sense of how feasibility and if the web platform was ready. I think I tried over 4 different face detection methods before finally discovering an official Tensorflow solution. From there, it was a grueling process of figuring out the vector math needed to transform a cloud of points into usable values for character rigging.

The design and UI of the app took up the other half. I wanted a jellymorphic visual design as I was already thinking that I wanted the mascot for the app to be something cute and squishy. I also wanted to be conscious of safe zone for use in streaming. My talented type designer friend helped make the cute logo, icons and custom illustrations while I was in the weeds trying to assemble it all together. It definitely was a tight deadline, but I wanted to be one of the first creators to release a "Vtuber" app for browsers.

What made Glitch the place you chose to build and share this project?



Glitch has always been my preferred platform to prototype web experiments. Being able to work from any computer (and sometimes tablet) is super convenient and powerful. Combined with the fact that I'm running a dev server that doesn't take 1-2 minutes to recompile code has made this my go-to setup. I'm currently using Svelte + Snowpack for that sweet Hot Module Refresh life.

Glitch as a community has been super welcoming and creators on it are always releasing inspiring web experiments. My last 2 projects have also seen successes on Glitch and I saw no reason not to release Kalidoface on it as well. What can I say, I am a Glitch fan!

I imagine building something like this for (and in) the browser wasn’t possible several years ago - what libraries and APIs available today did you use to build Kalidoface? Were there any pain points you hit in the process?


Definitely! At the core of this project is Tensorflow's Face Landmarks Detection (formerly Facemesh by Google Mediapipe). Their impressive machine learning model tracks 486 facial points + iris detection. Unlike other models, it tracks well even with glasses on.

Getting Live2D models to render in the browser was also a challenge. The official SDK is super hard to use and mostly in untranslated Japanese. Luckily, a single developer in China is currently working on a plugin that lets developers use the popular graphics library, PIXI.js with the Live2D format. Him and I went through alot of bug reports and feature requests throughout the development of Kalidoface, and even now, the app uses a beta version of Pixi-Live2D-Display.

The app is currently in beta, so what can we expect next for Kalidoface?


1. A mascot! I’d love to commission or learn to draw an original character for Kalidoface. I’m thinking something squishy and bouncy. A time-traveling inter-dimensional jellyfish? Maybe that’s too crazy.

2. Stickers and layers! While you could just open up Kalidoface in OBS and customize your streaming scene to have png images/text layered on top, I do want to offer a simple way to add floating stickers (maybe even animated gifs!) to your canvas. There might even be some Kalidoface exclusive stickers!

And of course, bug fixes. If anyone has any more suggestions, feel free to tweet me @yeemachine. I definitely have a long list and exciting list ahead of me.

Shouts out to Rich for teaching us more about vTubers and the tech behind it, and make sure to try out Kalidoface for yourself! If you build or see an app on Glitch that you want us to feature, tell us about it by emailing [email protected].