The Body

The Body, Everywhere and Here / Week Six & Seven / Final Assignment /  by Pippa Kelmenson

Assignment:

Create something that reflects on ideas we've discussed (new/old, alone/partner) - make/document a prototype that you can use for class discussion & user testing.

Response:

For my final project, I hope to combine elements of sound, machine learning, and body tracking to create a gesture-controlled harmonic sequencer. With Tone.js, ml5, PoseNet, and Three.js, I would like to track users’ gestures to create 3D sine waveforms that are multiplied with each hand shown. This way, each hand adds a layer of harmony onto the original frequency, creating a chord of tones from the 4 hands of 2 people. My goal is that, after drawing a sine wave, the ml5 machine learning model Meyda can assume the frequency of the wave based on its amplitude and cycle, and then play it as a pure tone accordingly. Inspiration below.

Inspiration:

Testing:

Since working on my original prototypes, I was able to draw up a PoseNet sketch that included ellipses falling from users’ hands with live and recorded data. Unfortunately, I wasn’t able to access a Kinect, so my goal was to use elements of Three.Meshline ribbons and Tone.js oscillators in my PoseNet sketch. After trying to import the recorded data from a Kinect to PoseNet (code here) and combine with a Three.js environment and Three.Meshline ribbons (code here), I ran into problems making Three.js and P5.js work together as libraries. Eventually, I was able to incorporate the harmonic sequencer I made in P5.js into my PoseNet sketch, but am still debugging my Three.js tests.

I plan on continuing this project in the future for my Listening Machines class, so that Tone.js can interface with PoseNet and Meyda to reflect a harmonic sequencer drawn and controlled by two users’ hands gestures.

You can find my updated (but still broken) code here.

The Body, Everywhere and Here / Week Two / Presence in Virtual Spaces /  by Pippa Kelmenson

Assignment:

  • Create a two-player sketch using the mouse over webRTC that encourages the users to be co-present in some way. User test with your partner. Focus on what it feels like, not (or in addition to) what it looks like.

Response:

You can find my code here.

The Body, Everywhere and Here / Week One / The Body & The Computer /  by Pippa Kelmenson

Assignment:

  • Create a sketch using the pixel array that encourages the user to use their body in some way. User test with your partner. Focus on what it feels like, not (or in addition to) what it looks like.

  • Create at least two variations on the sketch that encourage the user to interact with your sketch in a specific way.

    • Consider: Moving away from your computer to test with the full body or using an external webcam placed away from you to capture your full body.

Response:

You can find my code here.