Automated Lip Syncing in Mini Micro

Mini Micro version 1.2, released this summer, contains a number of new features. These include a new Sound.amp property, which tells you the amplitude of a playing sound. This can be used for a lot of things — but in this post, I'll show how it can be used to animate the mouth of a character on screen as they talk.

The video above was created with the following program using this technique.

clear
mainImage = file.loadImage("mcp.png")
gfx.drawImage mainImage, 480-mainImage.width/2, 320-mainImage.height/2

jawSprite = new Sprite
jawSprite.image = file.loadImage("mcp-jaw.png")
jawSprite.x = 541; jawSprite.y = 151
jawSprite2 = new Sprite
jawSprite2.image = jawSprite.image
jawSprite2.x = 411; jawSprite2.y = 151
display(4).sprites = [jawSprite, jawSprite2]

list.swap = function(i,j)
    temp = self[i]
    self[i] = self[j]
    self[j] = temp
end function
c = jawSprite2.corners
c.swap(0,1); c.swap(2,3)  // (flips the image horizontally)
jawSprite2.setCorners c

openMouth = function(amount)
    c = jawSprite.corners
    dy = 32 * amount
    c[0][1] = c[1][1] - dy  // bottom inner corner
    c[3][1] = c[2][1] - dy  // top inner corner
    jawSprite.setCorners c
    c = jawSprite2.corners
    c[0][1] = c[1][1] - dy  // bottom inner corner
    c[3][1] = c[2][1] - dy  // top inner corner
    jawSprite2.setCorners c
end function

speech = file.loadSound("happyNewYear.ogg")

speech.play
amount = 0
while speech.isPlaying
    a = speech.amp
    target = a * 5  // (found experimentally)
    amount = (amount + target*4) / 5
    openMouth amount
    yield
end while
openMouth 0

The first half of this code is just setup: loading and drawing the background image (mcp.png), and then loading two sprites for the jaw. It's two sprites because I was lazy and only prepared an image for the right half of the jaw, which gets loaded as jawSprite; for the left half of the jaw, I just make another sprite with the same image, but swap the left and right corners, inverting it (this is jawSprite2).

To animate the mouth, I first created the openMouth function, which again monkeys with the corners of the sprites. This time it just moves the "inner" corners of the sprites (representing the center of the chin) down a few pixels, based on the given amount parameter.

Finally, the main loop at the bottom of the code does all the real work. It loads and plays an audio file (happyNewYear.ogg), and then goes into a loop for as long as that speech sound is playing. On each iteration of the loop, it reads the current amplitude of the sound using speech.amp. Because this value changes very rapidly, I do a bit of smoothing (amount = (amount + target*4)/5), and then pass the result to the openMouth function we defined above.

And that's it! With this simple technique, you can make a compelling illusion of a character talking.

All the code and resources are available over at GitHub. Why not download it and give it a try yourself? In either case, please let me know what you think in the comments below!