Contents
- Background
- Quick start for non-TUIO systems. <--native touch support (no translator)
- What is TUIO?
- Bridging The Gap
- Running The Translator
- A Basic Application
- Go To Games Page
- FAQ
Background
UPDATE: Chrome natively supports multitouch events and the games should work without any middleware. The TUIO->WebSocket translater is now obsolete and may not work correctly. I am no longer maintaining it.
In 2013 built a TUIO gaming table with a 46 inch playable surface. The table can seat six quite comfortably, handles over thirty simultaneous touches, and has a crisp, LCD display running at 1920x1080.
The problem is, there aren't a whole lot of games available for such a setup so I decided to create my own. Frustratingly, there is no easy way to write browser-based, JavaScript software for TUIO surface systems. So, I decided to write an application to bridge the gap between TUIO and JavaScript.
*NEW* The software now supports native multitouch for non-TUIO systems as well. Refer to this guide (you don't have to run the translator in that case.)
What is TUIO?
TUIO is an open framework and protocol for multitouch surfaces. You can read all about it at the tuio.org web site. TUIO defines a standard for multitouch hardware to send data to applications running on your system. The multitouch device detects touches and passes them to its driver software, and then the driver software encodes and publishes those touches using the TUIO protocol. This is typically done over a UDP socket on the host machine. Applications can connect to this socket and utilize the touch data coming from the driver.
Bridging the Gap
For security reasons, browsers cannot connect directly to the UDP socket where the TUIO data is available. In fact, before HTML5 browsers didn't natively support any kind of persistent socket connection at all. Fortunately HTML5 includes something called WebSockets which let the browser connect, and stay conneted, to a TCP/IP port.
The natural solution is to create a native application to gather data from the TUIO UDP port and re-publish it on a TCP/IP WebSocket. The browser then can open up WebSocket connection and read the TUIO touch events. I call this small piece of middleware the "translator." Right now the translator only forwards touch add, remove, and update events with the event ID and the X,Y coordinates of the event.
When I started working on this, Google Chrome was the only browser with proper WebSocket support, so this whole project is based on Chrome and has not been verified in other browsers.
Running the Translator
TUIO/translator is obsolete/deprecated. Use native support
The default configuration of the translator is to read TUIO data at UDP port 3333 and re-publish it on a WebSocket at port 3334. Both of these settings are configurable. The translator can be minimized to the system tray where it will run in the background. The translator will only connect to your TUIO stream when a browser is connected to the translator via WebSocket. This leaves your TUIO stream available for other applications to use when browser-based apps are not active. The translator is called "TouchEventServer.exe" and can be downloaded and installed via the link below.
The translator is only available for Microsoft Windows XP and newer operating systems. If the translator won't launch you may need to install the Microsoft .NET4 Redistributable and the Microsoft Visual C++ 2010 Redistributable. Those are free downloads from Microsoft, available here:
- Microsoft .NET4 Redistributable (free download here)
- Microsoft Visual C++ 2010 Redistributable (free download here )
Your firewall may prompt you!! The translator needs to talk both TCP and UDP on your network, so watch for those firewall popups and grant the appropriate permissions.
If you need to change the port number settings in the translator, or are interested in other advanced features of the program, you may be interested in reading the TouchEventServer Manual.
A Basic Application (watch video)
TUIO/translator is obsolete/deprecated. Use native support
Now you're ready to connect a JavaScript-based app running in a browser to the translator and start processing multitouch events. This basic program will serve as a good test to make sure your system is configured properly and the translator is working. Click the button below to launch the test application, and make sure to full screen it with the F11 key on your keyboard! Hit backspace when done.
This test app displays your screen resolution and the connection status of the WebSocket, as well as a list of the ID and x,y location of all currently detected TUIO touch events. This small app also serves as an example of how to write your own JavaScript code that interfaces with TouchEventServer via WebSocket.
All of the code that executes in this app is found in Test.js which is commented, and written in 100% pure JavaScript with no external libraries. It should serve as an adequate tutorial for developing your own multitouch programs using TouchEventServer.
The program shows how to use an HTML5 WebSocket to:
- Create a connection on port 3334 when the app has focus.
- Close the WebSocket when the app loses focus.
- Parse messages that arrive on the WebSocket:
- Add touch events
- Update existing touch events
- Remove touch events
- Respond appropriately to socket closures or errors.
The program also shows how to use the HTML5 Canvas element to:
- Check for fullscreen status.
- Set up a render loop at 60 frames per second.
- Display socket connection status to the user.
- Display a list of all active touch events.
- Draw a small box around all active touch events.
My Games
I have written several games for use with the TUIO to WebSocket translator. Please see the games page to read about my games.
FAQ
Troubleshooting
- Is there a manual for the translator (TouchEventServer.exe)?
- The translator doesn't launch. It just displays an error message.
- I see a "please wait" or a "Connecting to TUIO" flashing message on my game.
- Everything is running and I still see that flashing message.
- My touches do strange things like zooming the screen or navigating the browser.
- My touches aren't matched up at all with what I see on the screen.
- Do I have to run the games fullscreen? Can I make fullscreen the default?
- The Chrome button on the translator interface doesn't do anything.
- On which ports does my firewall need to allow traffic for this to work?
- My TUIO driver is not at port 3333. How can I make the translator see my port?
General Q&A
- Do I have to use the translator? Can't I just run the games in Chrome?
- I have Chrome game, but it's not from DougX.net. Will it work?
- Will these games run in other browsers?
- Can the Translator run in the background?
- Can I develop my own games using this system?
- Contact information
THE WEBSOCKET TRANSLATOR IS NO LONGER SUPPORTED AND SHOULD NOT BE USED - use native touch support only
Is there a manual for the translator (TouchEventServer.exe)?
Yes. The translator is fully documented here.
The translator doesn't launch. It just displays an error message.
Most systems already have the runtime environments from Microsoft that are required by the translator. Your system needs either Microsoft .NET4 or the Microsoft Visual C++ 2010 redistributable, or both. They are free downloads from Microsoft. Install them one at a time and try running the translator again after each installation.
- Microsoft .NET4 Redistributable (free download here)
- Microsoft Visual C++ 2010 Redistributable (free download here )
I see a "please wait" or a "Connecting to TUIO" flashing message on my game.
Your touch table's hardware will not send TUIO events directly to Chrome. TouchEventServer.exe must be running in order for this to happen. The most likely cause of the flashing message is that you aren't running TouchEventServer.exe, or it experienced a problem of some kind. Try restarting TouchEventServer.exe as well as your TUIO driver, as often times these problems trace back to an actual driver crash. Then make sure your port numbers match. Your TUIO driver settings must match the port settings in TouchEventServer.exe, and the settings there must match those in the Chrome games themselves.
Everything is running and I still see that flashing message.
Is another application using your TUIO connection? This includes other tabs of Chrome that may be up! Make sure you don't already have a game open in another tab or instance of Chrome. Other multitouch games and apps cannot share the connection with the translator. Make sure no other multitouch games or apps are running. Make sure your driver is in TUIO mode, restarting it if necessary. Lastly press F5 in the game to reload it and see if this fixes the problem, or better yet, restart Chrome. Some Chrome plugins cause zombie Chrome processes to linger. Use the process monitor (CTRL+SHIFT+ESC) to kill off any zombie Chromes. A plugin-free installation of Chrome is highly recommended.
My touches do strange things like zooming the screen or navigating the browser.
The zooming is caused by your system recognizing gestures and forwarding special events to the operating system and then the browser. If your driver is sending mouse or Windows Touch events you will have these problems. Your driver should be set up to be sending only TUIO events. Windows gestures or other HID events must be disabled.
My touches aren't matched up at all with what I see on the screen.
The games must be run fullscreen for touches to line up correctly. Press F11 to make the game run fullscreen. If this is not the cause of the problem, your surface may be out of calibration. Run the calibration tool on your TUIO driver. Also, make sure your frame is in TUIO mode. If it's sending mouse or Windows Touch events you may experience problems. Your driver should be set up to be sending only TUIO events. Windows gestures or other HID events must be disabled.
Do I have to run the games fullscreen? Can I make fullscreen the default?
Yes. The touch events from your system will be misinterpreted unless the games are run fullscreen. You can press F11 to put an app into fullscreen mode, or you can make it always run fullscreen. In Chrome open a new tab and view your apps. Right click on any app and select "Open Full Screen" from the popup menu. You can also use this menu to create desktop shortcuts to the apps, as well as start-menu shortcuts.
The Chrome button on the translator interface doesn't do anything, or is disabled.
TouchEventServer.exe provides a button to launch Chrome purely as a convenience. If it isn't working then it probably simply can't find Chrome on your system. Just launch Chrome the way you normally would.
On which ports does my firewall need to allow traffic for this to work?
Your system is probably already sending TUIO data over port 3333. This is the default, and will work nicely. The WebSocket Translator (TouchEventServer.exe) needs port 3334. Naturally, if you changed either of these defaults your firewall will need to allow traffic on the new ports.
My TUIO driver is not at port 3333. How can I make the translator see my port?
Detailed information about advanced settings in the Translator can be found in the manual. But the short answer is to bring up the Translator and click the "advanced configuration" button. Then (important!) uncheck the automatic mode checkbox. Now that the Translator is not in automatic mode, click the Close Server button in the Websocket settings and click the Disconnect button under the TUIO settings. This kills any existing connections. Then change your port numbers to their new settings. If you are adjusting the location of your TUIO driver change only the TUIO Client Connection port number, do not change the Websocket port. Lastly re-check "Automatic mode" and re-launch your game. If you changed the WebSocket port (unusual) then don't forget to change the settings inside the game as well.
Do I have to use the translator? Can't I just run the games in Chrome?
As of version 1.1 native multitouch is supported. You only need to run the translator if you have a TUIO system. If you have a system that makes touch events available to the browser you can install the app and start playing immediately. Just touch the launch screen to make the app detect your native multitouch system. If the TUIO indicator continues to flash then your multitouch system is nonstandard, or you have to enable multitouch in Chrome. This is done on the via the chrome://flags/ (Chrome configuration page) under "Enable touch events."
I have a Chrome game, but it's not from your site. Will it work?
No. In order for the Translator to supply multitouch data to the Chrome-based games they have to be specially written. The games connect to the translator via WebSocket to receive TUIO touch events from your system. If a game is not listed in the "games" section of this web site, then it is not supported.
Will these games run in other browsers?
As of this writing, Chrome is the only browser that both fully supports HTML5 WebSockets and has the performance to run graphics-intensive applications. Additionally, the games are packaged as Chrome Web Store apps which are exclusive to Chrome. As more browsers begin to support WebSockets this will be re-evaluated.
Can the Translator run in the background?
The Translator will minimize to the system tray, and will display notifications when it establishes or loses connections. It automatically connects to your TUIO driver when a WebSocket connection from a game is detected. This means that it won't use your TUIO port if you aren't running Chrome (keeping it free for other applications). Feel free to set up your system to launch the Translator at startup. You may want to enter the system tray customization menu and set the translator to "show icon and notifications." Double click it in the tray to open the interface.
Can I develop my own games using this system?
Sure! The only provision is that (if you intend to distribute it) you must offer it for free, preferably through the Chrome Web Store. The test application described above is a good starting point for learning how to write a WebSocket/Canvas application that is capable of getting multitouch events.
Contact Information
Please read this entire FAQ to make sure your question hasn't been answered here! Also, please read the entire translator manual if you are having issues getting your system working. Replies may be slow, but you can email contact@dougx.net with your questions.