live dance floor — wrist beacons painting projected gradients, IMU sensing crowd rhythm
INTELLIGENT DANCE ENVIRONMENTS
a bidirectional multimodal interactive system for real-time interactions between DJs and dancers
role: end-to-end design and engineering
technology: ESP32, MPU-6050 IMU, Arduino/C++,
OpenCV, Processing (Java), Python/SciPy, TCP/JSON, 3D printing
date: 2022 — MIT 6.835 (Intelligent Multimodal User Interfaces)
links: GitHub · demo video
DeepDance is an intelligent dance environment built for nightclubs — loud, dark, and
chaotic. To overcome this, the system combines wearable sensing and computer vision to
analyze movement and rhythm, generating visual and haptic feedback that lets participants
both influence and be influenced by the environment.
The system has three components: wearable devices that sense each dancer's
acceleration and orientation, computer vision that tracks their position across
the floor, and a feedback layer of projected visuals and haptics.
The project is an exploration of what a closed-loop feedback system for a
crowd could be, how music and lights could be modulated in response to crowd behavior,
and data would be useful for a DJ to read.
hardware
The wearable device was built around an ESP32 with an onboard MPU-6050 IMU, battery, haptic motors,
LEDs, and Wi-Fi, running Arduino firmware in a custom 3D-printed enclosure. The device streams
acceleration and orientation as JSON over TCP sockets — data the system uses to analyze rhythm and
energy, whether for an individual dancer or an entire crowd.
Its compact form factor allows it to be worn as a wristband, ankle band, or belt. Visual
feedback supports individual expression and gives DJs insight into the crowd’s energy,
while haptic feedback encourages playful interactions between dancers who share similar rhythms
or energy.
UNDER THE HOOD
- OpenCV beacon tracking: HSV masking → contour detection → centroid, streamed as (x, y) coordinates.
- Processing (Java) visualization: a reactive grid of gradient cells with physics-driven particle trails along the beacon path.
- SciPy peak detection on waist-IMU acceleration to classify on-beat vs off-beat engagement as a DJ-facing signal.
- Client–server architecture over TCP sockets with JSON messaging between firmware, Python analysis, and the projection server.
direct interaction — painting
A wrist-worn LED beacon is tracked in real time; the dancer’s hand trajectory paints light onto a projected gradient grid — high-agency control over the environment through movement alone.
indirect interaction — engagement sensing
Waist-worn IMU data is analyzed for rhythmic energy; the system independently judges a dancer’s engagement and produces a visual signal the DJ can read as crowd feedback.