SUMMARY
The Spell Eater is an IBM Watson enabled magic artifact for spellcasting related training. I love all spellcasting – the magic wand, the swish and flick, and that fantastic feeling of affecting nearby objects wirelessly with spoken charms.
The Spell Eater project began as a side experiment during the Smart Craft Studio 17. I built a Raspberry Pi based chatbot that was trained to understand magic spells from Harry Potter movies. The training worked out pretty well, people who tested it seems to enjoy it a lot. I decided to spend some time on it and built out this project. During testing, I realized that people are familiarized with the movie plots and spells, but many of them are hesitated to say the magic charms out loud. In order to make the uses comfortable casting spell with their voice. I developed a very light narrative around the Spell Eater and repurposed the mutant artifact in my imagination to be a training tool for spell casting. After the trainee wakes up a Spell Eater with some swish and flick, it’s time to feed this hungry devourer a delicious spell. Based on how well the trainee channels the spell, Spell Eater will digest the spell and give the trainee a score based on how close to the perfect form (100 points – most delicious). e.g. “This is a Levitation spell, 92% accurate!”
Our relationship with machines is evolving, and conversation based interface is the most effortless option compares to other ones like gestural wearables, I/O implants, biosensors, and brainwaves. Abilities like personal insight training, customization, unique cultural and (localized) context building will be an essential part of every conversation based and cognitive AI embedded machine for the sake of individuality and privacy. We will eventually talk like a machine for better communication and vise versa. Spell Eater is trying to say all these in a playful way.
PROCESS LOG
I am interesting in spellcasting – the magic wand, the swish and flick, the incantation, and that feeling of affecting nearby objects wirelessly with spoken words is magical and weirdly satisfying. With the recent advancement of A.I. technology, services like IBM Watson are making my dream of spellcasting closer to the reality by adding the incantational part into the casting action.
I started experimenting with the Unlocking Charm, “Alohomora”, which can be found in The Standard Book of Spells, Chapter 7, Grade 1.
“There are many ways to pass through locked doors in the magical world (see the Fire-Making Spell, Gouging Spell, and the Reductor Curse) where you wish to enter or depart discreetly, however, the Unlocking Charm is your best friend.” —The Book of Spells
Other charms that I want to include, all of them are from Grade 1.
- Wand-Lighting Charm: Lumos
- The Softening Charm: Spongify
- The Severing Charm: Diffindo
- Fire-Making Spell: Incendio
- Levitation Charm: Wingardium Leviosa
- Locking Spell: Colloportus
- Mending Charm: Reparo
Note 1: Locate a magic wand.
I thought I would use a combination of IR camera and a modified toy wand with a narrowed angled IR LED to begin my experiment. I soon found out that was the mechanism behind the The Wizarding World of Harry Potter™ attraction in Universal Studio Orlando except their wand is even lower tech with only an IR reflector. When I was researching further, I realized that Linux has an IR library (LIRC) that can be used to decode TV remote signals. I then found an interactive TV remote wand on the all-mighty Amazon.
Note 2: Understand the magic words
“Alohomora!” is not a dictionary word so most likely IBM Watson won’t be able to capture it correctly. After some experiments, IBM Watson understands “Alohomora” as “Aloha Mora”, “Hello Hope more”, “Well I hope more”, “Aloha bora”, etc. I listed all these interpretations into IBM conversation editor as user examples to make sure the editor will pick it up as an intent. There are more advanced way of handling special words by creating profiles, not right now.
Note 3: Casting the spell
Now the technology is in place for this idea to come to life, finally I need to write some code. The way I imagine how this experience will play out is that every successful swish of the wand will open up an small window of time for listening the caster’s voice. Both the STT result and wand casting result (succeed or failed) will be sent to the IBM Watson conversion editor to be evaluated. The TV remote wand I bought offers 9 different gestures, I programed each one of them with a remote control command from my TV remote. I then pair the wand with FLIRC using the provided PC FLIRC software. It looks like FLIRC can only store up to 6 commands, I can work with that.
Note 4: Wand turns out to be a great interface
When I had the idea to build a version of Spell Easter using the Optitrack system, the wand naturally came to mind as the controller/pointer for it. I had a good first impression with wand, a gestural controller, when I tried out the Fantastic Beast demo that came with the Google Pixel VR. However, in reality I am still holding the oval-shape remote controller so I wasn’t fully immersed. When I prototyped Spell Eater with a wand replica, I realized further that wand is such an intuitive controller for VR or AR. When it comes to interaction, it naturally makes the player focus on the tip of the wand but the 10 individual fingers.
Note 5: Joint forces with fellow wizards
The idea of spellcasting came to mind while working with IBM Watson Conversation on Raspberry Pi. After some research online, I found out there are a few existing projects out there also brought the spellcasting experience to live in their own way, two most interesting ones are the interactive wand attraction at Harry Potter Diagon Ally in Universal Studios Florida and the “Raspberry Potter Wand” tutorial on Makezine #55. The interactive wand in Diagon Ally uses a reflective tip to reflect IR light for tracking, and the same want is used in the Makezing’s tutorial. It makes it really low-tech on the wand side. This is a great next step for moving away from hacking commercial products for build-in gesture tracking (nothing wrong with hacking, but would like to DIY the whole thing). However, none of them has the incantational component in the casting experience, fellow wizard technologists, this is my contribution to the magical experience.
Proof of Concept Prototype
I had built examples and tutorials with IBM Watsons in Unity 3D and on Raspberry Pi. I started playing with IBM Watson last year as a part of my research in designing UX/UI for VR/AR. I personally don’t believe that we will be wearing all those tracking contraptions on our body in order to interact with our AR/VR apps in daily life. Verbal conversation and wear-to-forgot biosensors are probably the two most convenient inputs right now while brainwave sensors getting its acts together. I was fortunate enough to co-taught a course called “Interface Cognitive A.I.” with a senior UX architect from Bell Labs (Wei-Wei) and an ex-UX associate director from IBM (Hanshen Chen) at Parsons School of Design. The goal is to look at what would be a user-centered UX/UI approach to future A.I., spellcasting and wand swishing is a natural next step for me. The form of the Spell Eater is based on one of the Raspberry Pi prototypes I made. The design of the case is based on TRIPNUTZ’s GR8 PI2 on thingiverse with some minor modifications.
Original Design
It is said that the artifact was originally made of an ancient magical orange tree that has the ability to neutralize the power of spoken words. I used a 3D scanned tree model as the base and merged it with an urn. I also need to build a speaker so it has everything it needs to operate.
After a couple iteration, here is the final setup for the Spell Eater. A mutant tree trunk with a blossomed speaker and some other electronic parts plugged into its core. A connoisseur of spellcasting and a bottomless devours for the power of spoken words.
The Spell Eater Crest
To resurface this spell eating research laboratory from Hogwarts, one must rediscover the crest of this secret organization, so I did.
The Spell Eater VR
While preparing the brand new Still Photography and Motion Capture room for student check-out at Parsons, I was able to build a VR version of the Spell Eater as one of the demos. In Spell Eater VR, players are put into a deserted warehouse with many wood crates and barrels inside, they can then use their wand to make things float or set things on fire – “Wingardium Leviosa” and “Incendio.” The magical moment happens when a player’s spoken words turning into an action in the VR space. The simulation made many participants’ dream come true, including me and my family’s. Spoken words are powerful inputs.
MayThe Chatbot Be With You Workshop
[NYC MEDIA LAB 17 WORKSHOP] Following my workshop last year, I’ve continued to explore and to experiment different chatbot scenarios by introducing conversation based interaction into different user interfaces. A.I. will continue to evolve and conversation based interface will become one of the most convenient ways to interact with future machines. In the first half of the workshop, my collaborator and I will share various scenarios we had experimented such as anxiety and panic attack, Japanese woodworking, “wingardium leviOsa!”, and virtual reality. We will spend the second half demoing and discussing more chatbot. Welcome to PARSONS, and may the chatbot be with you!
Finally done with Maker Faire New York and NYCML 17 (09/23/17 – 09/29/17). I was showing, demoing, and workshopping Spell Eater that I’ve been experimenting since this summer. Gonna chill this weekend, there are still many things to be improved. Thanks to everyone who supported this project along the way. Yipin, Tim, folks @ FabCafe Taipei, Fi, Kous, Minori, Miki, Chiehping, Kuo-Jui, Wei Wei, Han-Shen, Barsha, Santangelo, Aiden, and Iva. A specially thank you to the surprise visit from Ray!! That made my day.
Funny Stories:
In one of our demos with Spell Eater, we had two participants reenacted the scene where Hermione taught Ron to cast “Wingardium LeviOsa” correctly. The Spell Eater gave Ron’s version a score of 42 and Hermione’s a score of 96, which was exactly like what’s in the movie. That was an awesome moment!
The speaker cone of the final Spell Easter was made of a ice cream cup from Mr. Donut (see the flyer below). This particular one was from when me and my friends, Tim Wong and Yiping Huang, visited Kobayashi sensei at IAMAS in Ogaki. We went early so we waited in Mr. Donut next to the Ogaki station, and there I fall in love with the cup in the first sight.
Future steps:
1. more spells
2. more visible and haptic feedback (on the physical wand) – vibration and light (LED Fan added)
3. add sound effects
4. Shorten the recognition time (try direct NLP)5. connects to household appliances or a game6. wireless, portable, and content