Interactive 3d Animation
Interactive Gestures
Hand gestures become particle creatures, and the model’s uncertainty becomes something you can see.
Watch the demos
Category
Creative Technology
& Interaction
Context
University Project
Role
Unity VFX & Particle Animation
Tools
Unity, Visual Effect Graph, MediaPipe, TensorFlow
The concept
Shadow puppets, retold in particles
Shadow puppetry lets people make animal silhouettes with their hands. This project takes the idea further: in front of a webcam, a gesture is recognized and turned into a rabbit, a bird or a fox made of particles.
The three gestures as shown in the interface. The recognized one gets a green frame.
01
Interactive learning: users experiment with gestures and see how the system responds.
My contribution
Unity VFX & particle animation
Interactive Gestures was a group project, and the recognition model was a shared team effort. My part was the visual system: I built the particle animation from scratch in Unity’s Visual Effect Graph, from the animal forms to how they gather, scatter, drift and move.
From animal meshes to particle forms
Static meshes of a rabbit, a bird and a fox provide the shapes. Unity’s SDF Bake Tool turns each mesh into a Signed Distance Field: a 3D volume that stores how far every point is from the surface.
Particles read that volume to find the shape, without mesh rendering, mesh collisions or rigid constraints.
01
The animal mesh provides the original shape
02
The SDF Bake Tool converts it into a distance volume
03
Particles conform to the volume and form the silhouette
04
The form moves freely between dispersed and complete
Rabbit mesh in the SDF Bake Tool
Fox mesh in the SDF Bake Tool
Bird mesh in the SDF Bake Tool
Visual Effect Graph
Building the particle animation
I built the effect from scratch in the Visual Effect Graph. These are the mechanisms behind it, as documented in the project report.
Update Particle: a Form parameter, scaled by two Multiply nodes, drives Conform to Signed Distance Field, with Turbulence layered on top.
Spawn: a constant spawn rate feeds the system.
Initialize Particle Strip: strips inherit source position and color, with a set lifetime.
Conform to Signed Distance Field
Pulls particles toward the baked animal shape. This is what forms a recognizable silhouette.
Attraction strength
Turbulence
Uncertainty-driven dispersion
Gesture-dependent movement
Uncertainty
Making uncertainty visible
SetUncertainty receives a normalized value between 0 and 1 and passes it to the Visual Effect Graph through the Uncertainty parameter. The animation becomes feedback on the model’s current state, without asking anyone to read a confidence score.
Low uncertainty
Particles gather around the silhouette. The form is stable and legible.
High uncertainty
Particles disperse and the shape breaks apart, as when no gesture is detected or the model is unsure.
This shows how sure the model is right now. It is not a full explanation of how the model reasons.
Interaction
Connecting the animation to interaction
A central script, VFXFormController, coordinates rendering, uncertainty, the contextual UI, sound and movement around the particle system.
Green highlight
The currently recognized gesture is framed in green among the three thumbnails.
Guidance panel
Appears only when no gesture is detected or uncertainty persists, and hides again once recognition is clear.
Confident sound
Each gesture has its own sound, which plays only when uncertainty is below a threshold and fades in and out.
Smooth rotation
Rotation speed follows the movement state, fast, slow or still, and is interpolated between states.
The result
Each creature in motion
A webcam to capture the hand gesture, MediaPipe tracks its keypoints, the model recognizes the gesture, and Unity forms the matching creature in particles.
The recordings combine real-time gesture recognition with Unity’s particle simulation. Some lag may be visible because machine-learning processing and Unity’s animation run simultaneously.
Potential applications
Where the approach could go next
01
Interactive learning
The system turns abstract machine-learning behavior into visible feedback. People can experiment with their gestures and watch how the model responds.
02
Gesture practice and sign language
A similar approach could inspire tools for practicing hand gestures or learning sign language, showing when a gesture is recognized and when to adjust it. The prototype only knows three animal gestures: it is not a sign-language system and has not been validated for that use.
03
Alternative interaction methods
Gesture-driven interaction could be explored for installations, classrooms and input methods that do not rely only on traditional controls.
04
More understandable interactive AI
A system can express its uncertainty through visible behavior, instead of presenting a confidence number alone.
These are possible directions, not tested outcomes. The report proposes human-centered evaluation questions, but no evaluation has been carried out yet.