The Ultimate DT Workshop

This workshop is meant to cover all the popular software subjects and tools in the Design and Technology program at Parsons: Machine Learning & A.I., Javascript, node.js, socket.io, OSC, and Unity 3D.

Since Flash was around, the web browser has become a forbidden ground for local access due to security reasons. Javascript libraries like Socket.io allowed technologists to connect the two – web browser (Javascript) and local application (Other languages) – through a local server. Today this pipeline is very useful for bridging the power of cloud computing to local software and hardware experiment and development. Here are definitions of subjects coved in this workshop.

Machine Learning – Machine learning is the scientific study of algorithms and statistical models that computer systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. It is seen as a subset of artificial intelligence. This workshop will be using Google’s Teachable Machine which allows you to train a computer to recognize your own images, sounds, & poses. A fast, easy way to create machine learning models for your sites, apps, and more.

Javascript – Often abbreviated as JS, is a high-level, just-in-time (JIT) compiled, multi-paradigm programming language that conforms to the ECMAScript specification.

Node.js – Node.js is an open-source, cross-platform, JavaScript runtime environment that executes JavaScript code outside of a browser. Node.js represents a “JavaScript everywhere” paradigm, unifying web application development around a single programming language, rather than different languages for server- and client-side scripts.

npm – npm is a package manager for the JavaScript programming language. It is the default package manager for the JavaScript runtime environment Node.js. It consists of a command-line client, also called npm, and an online database of public and paid-for private packages called the npm registry.  Example

Socket.IO – Socket.IO is a JavaScript library for realtime web applications. It enables realtime, bi-directional communication between web clients and servers. It has two parts: a client-side library that runs in the browser, and a server-side library for Node.js.

OSC – Open Sound Control is a non-handshake protocol for networking sound synthesizers, computers, and other multimedia devices for purposes such as musical performance or show control. The objective of osc-web is to make possible to send and receive OSC messages on the web browser. With this browser capability, you could do interesting things and connect more things together.

Unity 3D – Unity is a cross-platform game engine developed by Unity Technologies, first announced and released in June 2005 at Apple Inc.’s Worldwide Developers Conference as a Mac OS X-exclusive game engine. As of 2018, the engine had been extended to support more than 25 platforms. You can replace Unity 3D with openframeworks, processing, or other similar visual engines, most of them will support the OSC protocol.

Download the workshop zip file here.

I. teachable machine

1. Go to https://teachablemachine.withgoogle.com/ and Get Started.
2. I am using Image Project for this experiment.
3. Train the 2 classes with different gestures using the Webcam function.
4. Keep the name (class 1, class 2, …etc.) as they are, do not change them.
5. In the Training box, click Train Model. (This could take a minute or 2)
6. Once the training is done, in the Preview box, click Export Model.
7. In the pop-up window, click Upload My Model.
8. Save the Your Sharable Link for later use.

II. node.js/socket.io/osc-web

1. Download and install node.js.
2. Download and install Github.
3. Create an empty folder on your computer. I put mine in the document folder. This will be the project folder.
4. Open the terminal (or Node.js command prompt [Run as administrator] on Windows) and cd to your project folder and run:
   npm init
   This step will create a package.json file in the folder.
5. Install socket.io with the help of the node.js New Packet Manager (npm):
   npm install socket.io --save
   The --save ensures your module is saved as a dependency in your package.json file in the project folder.
6. Then install the osc-web module by Vilson Vieira, which will create the folder “osc-web” in the project folder (root) and enable us to send/receive OSC messages via socket.io:
   
   git clone git://github.com/automata/osc-web.git
cd osc-web/
npm install
   
   *if you run into “fatal: unable to connect to github.com:” you are probably behind a firewall. Try cloning via https – that has a higher chance of not being blocked:
   
   git clone https://github.com/angular/angular-phonecat.git
   
7. Run the bridge app (this file is inside of osc-web folder):
   
   node bridge.js
   
8. Ctrl + C to stop the app

III. Unity 3D:

1. Start a new 3D Unity project
2. Unzip the workshop zip file
3. Import Week3-All-for-one.unitypackage from the unzipped
4. Open scene KyleInterview in the Scenes folder.

IV. Put Them All Together:

1. Open app-TeachableMachine-Image-v2.html that came with the download, locate the line const URL and replace the Teachable Machine link with your own.
   
2. Close app.html in the browser if it is still open from the Quick Test.
3. Drag app-TeachableMachine-Image-v2.html into the browser.
4. After it’s open, press the Start button.
5. If the browser asks for permission to access the webcam, click Allow.
6. At this time, you should see webcam video show up on the web page and also lines of data flowing continuously in the terminal window.
7. In Unity, make sure the KyleInterview scene is open and press the Play button.
8. Do what you have to do in front of the webcam, by default, class 1 = wave, class 2 = clap, and class 3 = idle.