B2J Project - Cybernetic Being MOONSHOT Takeshiba, Japan

I co-directed the development of an innovative robotic choreography system, designed to empower a performer with ALS to orchestrate a captivating concert performance at a public venue. Collaborating closely with a leading company specializing in Brain-Computer Interface (BCI) technology, I engineered a sophisticated interface capable of transforming basic 1D data into a dynamic 2D matrix of animation states.
‍
This achievement enables individuals with severely limited physical mobility, who may only be able to engage with the world through their eyes and thumbs, to take creative control designing their own custom animation keyframes. Synchronizing intricate movements to the rhythm of the music, seamlessly loading DMX lighting profiles, and engaging with fellow performers and audience members through heartwarming gestures like handshakes. This project, primarily rooted in cutting-edge research, received generous funding from the Japanese government. It stands as a shining beacon of hope for the future of inclusive entertainment, demonstrating the transformative power of technology and creativity in breaking down barriers and providing unique opportunities for individuals with diverse abilities.

More Information Here!

Interaction Team - teamLab Kanda, Japan

During a two-week intensive period, I created an interactive concept, documented, prototyped to completion, and deployed the experience in collaboration with interactive software engineers at the world renown teamLab HQ. I presented the concept, development process, techniques, pitfalls, and avenues for further improvement, to computer vision experts, media artists, and creative catalysts in both English and Japanese. This experience provided critical insight into the production pipeline for large scale artistic immersive experiences, and allowed me to demonstrate and strengthen my skills with tools like Unity and TouchDesigner, non-planar projection surfaces, vector and quaternion-algebra, and computer shader language. ***Please be advised the output media content from this collaboration is under NDA***
‍

Research Assistant - Sony CSL Gotanda, Japan

Prior to starting my PhD., I joined Project Moonshot, funded by the Japanese Government, which is a research initiative that aims to realize a cybernetically intertwined future for Japan. This incorporates avatars and technology to allow people to be in multiple places at once, share experiences and bodies with others, and transcend physio-cultural limitations through assistive design.

Parallel Ping-Pong is a project that enables a single user to inhabit multiple robot avatars to play two games simultaneously. My role in this project was to design an automatic switching algorithm that would help users transition between their bodies, as well as integrating a realtime 3D Point Cloud view from within an HMD, and an expressive LED system for spectator understanding. The project went on to receive "Best Demonstration Award" at SIGGRAPH ASIA 2021, and "Honorable Mention" at Augmention Humans '22.
‍

R&D for Object Detection Algorithm
- Sony Interactive
Entertainment, Japan
‍

In just 3 weeks, I was tasked with ideating an AR application (featuring a proprietary object detection algorithm), creating a simple prototype, and a presentation to convey my idea to the president of the SIE R&D division. In this short span, I actually came up with 3 ideas, prototyped 2 and presented all 3 at the end of my internship. The committee was impressed, but most importantly I had a lot of fun.

My first presentation topic was to create a new type of tool for DJs to use at live performances, using the 6 degrees of freedom as parameters for augmenting their sound. This was further expanded into virtual collision detection, as a method to toggle on and off certain effects similar to a MIDI controller. Check out my videos below to see what this looks like.

My 2nd presentation topic involved the position and orientation of objects to create virtual clones, or "Kage Bunshin", which could be instantiated limitless times, and all programmed to act with their own behavior. I can't show too much of the project, but I used this technique to engage spectators in tournament scenarios to drop objects and powerups into fighting games in real-time from the background. My final presentation topic was a social app that allowed users to augment the view of their world from their smartphones, by scanning and interacting with entire buildings, in real-time as canvases to paint, and sculptures to mold.