TOASTcapture!

So; Motion Capture!

A few weeks ago I got the bright idea that we could use a Kinect to record animation data to use for the game. That sounded like fun so I bought one and started working on building a recording program. It sorta started working, so we decided that, while a friend was in town, we'd invite a pile of friends over, stick them in front of the camera, and have them record some silliness.

Now, it took a few weeks of difficult math that's still being worked on, but the translation to our skeletal system is good enough to get a decent preview of now, hence this post. First, some numbers, a video link, then some discussion on challenges.

30 mocap sessions recorded. Each of these is about 2.5MB of raw floating point data; 21 points in 3d space per frame, a timestamp per frame, somewhere around 30 frames per second. Each session consisted of a sequence of typical animations we use for the game. Most of the glorious participants recorded a set for a specific class or enemy type, so we have a wide variety of styles here.

6.5 hours to run the skeletal solver for all the recorded mocap data. I'll be the first to admit this could be improved, but it sounds somewhat impressive or something.

60 MB of additional animation data required to be downloaded by the client (for now). Given that the entire rest of the content is about a quarter of that, it's a noticable hit. This data is horribly horribly de-compressed so it can be fixed, and the keyframe data should be smoothed anyway, but again; DATA GRAR.

27:38 of recorded animation data. Each person recorded just under a minute of live-time; even though each session took about 10 minutes to perform. The script was set up to give some time to think of what to do, as well as some lead-in and lead-out time per animation.

The Video is here. It's... WAY WAY longer than I was expecting honestly. I don't expect people to watch 30 minutes of random animating; but it could be fun to skip through and see how much we did :) OR, if you were HERE, find your own set to see what it looked like!

There are a pile of problems with the data. The most obvious is that some of the recordings just flat out come out wrong. Anything that isn't mostly standing up, moves to fast, rotates a lot, hides arms behind (my scout pose), hides legs behind the body (kneeling), etc, all totally fail to be recorded correctly. This is completely expectable given the nature of the camera, but it does mean we need to adjust what we can do with it. In some cases we could rotate the actor (and invert it when processing).

Others are problems with the translation; if you look at the data even briefly you'll see the magical swimming torso problem... MATH IS HARD. The hands and feet aren't tuned yet, the rest poses weren't all recorded correctly, etc.

But its fun! And it was educational. Once the keyframe compression is managed (either manually or automatedly) I'll be able to realistically reference the animations in the game. There's also a matter of time syncing; most of the actions recorded don't conform to the timing standards of the rest of the game, so they'd appear really late or never actually appear to swing.

Always more to do :)