Interactive Art Installation · University of Tartu

Turning the body
into a controller
for living
digital art.

Reflexions is an art exhibition with real-time hand gesture recognition across three artworks — letting visitors shape generative visuals using only their hands and physical proximity to a depth camera.

Elina Meier  ·  Supervisor: Mathias Plans, MSc
Master's Thesis · Institute of Computer Science · 2026
See the Artworks Technical System
Rock gesture Paper gesture Scissors gesture
gesture: Closed_Fist  ·  conf: 0.97
depth: 0.42 m  ·  wrist: (0.274, 0.273)
3Artworks
21Hand Landmarks
96%Gesture Accuracy
65kFluid Particles
3Gestures
About the project

Reflexions is an interactive digital art installation at the University of Tartu. This thesis extends the original Reflexions installation with hand gesture recognition and playful elements across three artworks — Bloom, Lavalamp, and a new Rock–Paper–Scissors piece.

Visitors interact through mid-air gestures toward an Intel RealSense depth camera. Physical distance from the camera adds a spatial dimension — closer means larger elements, further means smaller. No wearables, no touchscreen, no prior instruction needed.

Three gestures
Fist
Fist Rock  /  Pull
Palm
Palm Paper  /  Push
Victory
Victory Scissors

Three artworks.

Same gesture vocabulary, three distinct aesthetic worlds.

RPS demo
Rock–Paper–Scissors

Gestures spawn animated 3D objects that flock and battle using Boids logic and rock-paper-scissors collision rules. A competitive dynamic emerges naturally when multiple players play at the same time.

Bloom demo
Bloom

Each detected hand spawns a bee that keeps following the user's hand as long as its detected. Holding still blooms a lily — a persistent trace of the visitor's presence. Extended from the original Lily artwork.

Lavalamp demo
Lavalamp

Colourful blobs simulating a lava lamp. A fist pulls fluid particles inward; an open palm pushes them away. Multiple users create competing fluid forces. Each interaction leaves a permanent trace on the simulation.

Gesture recognition

Three gestures, no instruction.

Rock, paper, and scissors are culturally universal hand signs — no explanation needed. The same three gestures take on different meanings in each artwork.

Recognition accuracy (MediaPipe Gesture Recogniser, CPU only):

Single user 96.3%
Multi-user (3 players) 89.0%

Wrist landmark is used as the depth reference — stable across all gesture types. Only detections above 0.5 confidence are emitted.

Fist — Rock
Fist
Rock / Pull
Palm — Paper
Palm
Paper / Push
Victory — Scissors
Victory
Scissors

How it works.

Two independent processes — Python for vision, Godot for rendering — communicate over a local UDP socket at 30 fps.

📷01

RealSense SR305

Short-range camera (0.2–1.5 m). RGB feeds gesture recognition; depth scales visual elements in all three artworks.

640×480 @ 30 fps
🖐02

MediaPipe

Gesture Recogniser in VIDEO mode. Outputs gesture label, confidence, and wrist coordinates per hand per frame.

~25 fps · CPU only
📡03

UDP · port 9000

JSON arrays are sent from Python to Godot over UDP. Minimal latency; dropped packets are superseded by the next frame.

500 ms timeout → clear
🎮04

Godot 4

RPSReceiver parses packets and emits a signal. Each artwork handles gesture data independently via its own interaction model.

60 fps rendering
🖥05

Horizontal Display

Horizontal layout for side-by-side play. Scales to most display sizes or projected surfaces without modification.

Any display size

Evaluation findings.

Ten participants, in solo, pair, and trio sessions. Rated statements about the experience on a five-point Likert scale. Statements were derived from the design goals of the system.

3.8

Discoverability

Most participants explored their way in. The gesture overlay acted as a recovery cue rather than an initial attractor. Participants noted it took them some time to understand how exactly the interaction works.

4.1

Gesture naturalness

When tracking was reliable, participants became physically expressive. Orientation sensitivity caused some failures at unusual angles.

4.5

Curiosity and exploration

Highest consistency across all participants. No participant rated below 4. Playful elements in the artworks and the social layer made the experience engaging.

4.7

Perceived agency

Strongest result. All ten participants felt their gestures had a direct effect on the artwork — rated 4 or 5 by everyone. This means the gesture recognition system and immediate feedback were effective in providing a sense of control and over the art.

3.8

Emotional response

Deeper resonance requires longer sessions playing sessions or more interesting content. This shows a gap between successful interaction and fully realised aesthetic experience.

4.3

Social co-presence

Most participants enjoyed the social aspects of the experience. Competition emerged in Rock-Paper-Scissors; cooperation in Lavalamp — without instruction. All group participants rated this 4 or 5.