Hacking animAItion with reinforcement learning
Beyond heat and vertex cluster rigging and static keyframe animation, adding a reinforcement technique is getting us closer to some decent animation!
This past August, I got to partner up with engineers from Raspberry Pi and Amazon at the Code.ai hackathon. Educators as well as software engineers discussed the problems and promises of integrating AI into secondary education.
Realtime visualizations of machine learning algos piqued some interest. I pitched the idea to use the visualizations afforded by generative animations to instruct and engage younger learners. As such, a group of interested folks sat down and we started to imagine how we could *show* vs tell how aspects of machine learning, such as reinforcement learning.
After demo-ing my animation pipeline that I was tooling for most of the spring(we will call this the CBP), and discussing the trials and tribulations, we settled on testing reinforcement learning to aid better animation as well as to illustrate how RLHF works.
I had worked on two pathways for my pipeline, one for humanoid objects (dog statue, walking broccoli), and the other for vehicular objects (bike, lego car, robot truck). For the purpose of the hackathon, we decided to focus on humanoid forms but also because papers like “Rig My Ride” had already solved the vehicular problem (and I of course found that *after* trying all sorts of things like Hough circles, RANSAC, etc).
The hackathon team also proposed a few other changes. First, to allow the user to draw the thing (like a bipedal dog or minecraft character, see above) rather having to take a picture (and then use ai to augment the image to add legs). Second, to surface the action to a dropdown, elimating extra work for a quick prototype.
The hackathon pipeline (HP) (available here), does the following: take the line drawing from the javascript canvas, clean it up, add a prompt and turn the sketch into a full-color, full-body character image. It then calls the CBP /classify, /mesh, /infer_joints and /rig endpoints. After that, we show the 3d object to the user. They select that action (version 1 limits choices), such as “waves arms in the air” and hit “Make it move.”
The desired behavior is then compared with the ideal “Model Pose,” which in this case would be a humanoid with its arms and hands in the air. This pose is in the form of one or more keyframes, each a full-body pose as per-bone quaternions. There is only one full-body pose in use at this moment, more can be added.
Using the slider weights the user sets in the UI , we calculate total weights for each run and the speed at which the trials run.
Get Cindy Sherman Bishop’s stories in your inbox
Join Medium for free to get updates from this writer.
With every episode, we update the latest value and render two skeletons side by side — the static Target pose and the live Learner pose — plus RewardChart plotting the raw reward against the running best. JointHeat indicates which joint is still off.
End Result? A lot of laughter, mostly because our animations were humorous and odd, not necessarily accurate. Yet! But what a great way to see how reinforcement learning can begin to reward the better matches the more iterations that occur. Furthermore, if the end goal can be to animate something in a humorous and fun way, we then have great content to share with our friends.
I encourage you to download the codeai-hackathon and the generative-ai-server repos, boot up both servers and check it out yourself! There is plenty of room to grow this project- we only have ONE animation pose working at the moment!
So end to end: draw or capture image → make model → rig → pose target (LLM-ish, currently mocked) → train → export GLB or animated movie.
TL/DR;
LLM joint placement is pretty good. Meshy’s humanoid joint placement is really good, and you can move the joints around. However, it doesn’t really allow for a different body types needed with custom animation — for example the dancing flower above has really bendy legs. Walking broccoli would be very top heavy….Ideally we’d provide a visual editor to allow joint correction and tweak keyframing. A few cool programs exist that do this to a differing degree, such as Meshy’s humanoid and four-legged animations, Michal Finegold‘s PuppetMaster and the site Animate Anything. Check them out as well.
CBP particulars: Claude’s joint placement on an alpha-cropped 2D image returns joint_hints, each with a position_normalized dict of {x, y, z}. Obviously the LLM can only give good X (left/right) and Y (up/down) estimates from the 2D picture because Z (depth/front-back) isn’t really observable from a single image, so it defaults to a placeholder like 0.5 (center). I then take the 3d model returned from Meshy and tweak the positioning by evaluating where the joints should be, such as shoulders and hips, and then tweaking those positions such as moving the shoulders and hips to the edge of the mesh at the supposed x and y values.
If for some reason the LLM doesn’t return good joint positions, we fallback to geometrically pinning the joints by evaluating the dimensions of spatial vertex clustering and using k-means to find what is most likely a joint (narrow band of vertices).
Notes:
the “Keep My Lines” choice on the drawing screen could be better. Takes about 4 minutes to wake up the art model via Fal.
As mentioned earlier, there is only one currect embedded pose. It would be really fun to string a bunch of the training steps together for a funny “dance.”
You will need a few API keys (Claude or Gemini, Fal, Meshy) to get this going, see the READMEs on github.
