Controller: Lab Week Workshop 2023

The integration of sensor technology with artificial intelligence opens up a world of intriguing new possibilities for interaction. The goal of this recent workshop was to explore creative and unconventional ways of engaging with devices and simple games, and bring these ideas to life in the form of functional prototypes.

Participants delved into the essentials of machine learning, examining the foundations of neural networks and their operational principles. The workshop addressed key questions about training a model with custom data, the steps necessary for data collection and preparation, crucial parameters in the training process, and strategies for model improvement. A crucial element of the workshop was to run the trained models on a microcontroller, enabling the creation of custom controllers.

In this workshop, we worked with Edge Impulse, a platform that enables code-free training of machine learning models and their deployment on edge devices.

Initially, students concentrated on understanding data, exploring the types of data that could be captured using the given sensors and how these could translate into distinct interaction patterns. The objective was to develop interesting and unconventional sensor-based interactions that could be consistently recognized by a machine learning model. After a phase of experimentation with sensor interactions, the students transformed their ideas into specific concepts for interacting with technology and implemented them in hands-on prototypes.

The concepts were brought to life using the Arduino Nano BLE Sense microcontroller. For those interested in the technical details and code from the workshop, information can be found here.

The week-long workshop concluded with a showcase of diverse interaction prototypes, demonstrating the creative and technical achievements of the participants. Highlights included Olman Arias Valverde’s pressure-gesture-controlled lamp using barometer data (Fig.2), and Tamara Solís Matamoros’s cooking-themed gesture game, where actions such as flipping a pancake were translated into game controls (Fig.3). Ron Eros Mandic and Lukas Speidel created a sound-based snake game controller (Fig.4).

The workshop aimed to encourage students to develop engaging and novel ways of interacting with the world around us. It demonstrated the potential and practical applications of machine learning and sensor technology to create innovative interfaces and user experiences.