Introduction

Theodore is an interactive installation capable to write sentimental letters based only on image content. The main idea underlying this project is to find out if machines are able to understand feelings, context, and generate intimate and confidential letters that express human-like sentiments using only pictures as input. For this purpose, the machine learning frameworks for the web ml5.js and face-api.js are used in the field of image recognition, in particular facial expression detection, landscape and landmarks recognition, as well as text generation.

Motivation

Machine learning libraries and frameworks have come a long way in these last years, offering a new class of applications, democratizing this tool by making it accessible not only to python and C++ developers but also to a wider group of coders like the JavaScript community. For this purpose, the Tensorflow.js library was developed. TensorFlow.js is a library for building and executing machine learning algorithms in JavaScript. The library is part of the TensorFlow ecosystem, providing a set of APIs that are compatible with those in Python, allowing models to be ported between the Python and JavaScript ecosystems. The main technical goal of this project is to use and to experiment with a well- established programming language like JavaScript, along with state-of-the-art Machine Learning frameworks for the web, in order to develop an interactive installation capable of generating sentimental letters that express human-like sentiments using only images as input. The aim is to identify the potential of such libraries in the development of interactive installations.

Interaction Concept

The interaction between the user and the installation happens through a mobile web application, the system invites the user to upload a picture with the subject of the letter. In a second step, the user inserts his name, as well as the name of their loved ones and the type of relationship. After this process, the system starts to analyze the picture, extracting the emotional traits of the subject, along with contextual attributes from the location. In conjunction with the basic information inserted by the user, these attributes are passed to the text generation algorithm. Once the text has been generated, the system sends the result to the physical installation. The generated letter is now ready to be printed by an analog typewriter automated by an Arduino MEGA and 33 solenoid type motors powered by four 8-channel relay modules and a one channel relay.

Physical Installation

The installation consists of a steel structure measuring 110 cm high, 30 cm wide and 35 cm in depth. At the top of the structure is positioned a typewriter Olivetti Lettera 32, german edition. 32 12 volt solenoids are mounted on the front and back wall of the structure. These motors are connected to the typewriter by a metal wire rope that controls 32 keys on the typewriter, an extra solenoid is used to manage the carriage return level of the typewriter. In order to get the solenoids to work, four 8-channel relays module and one channel relay module are used. The relay modules are controlled by the Arduino MEGA, which is connected to a laptop via serial communication.

Exhibitions

Theodore was exhibited at JSNation in Amsterdam (2019, https://medium.com/@amsterdamjs/these-were-the-artists-showcased-at-jsnation-2019-52fbfa4d1743) and at the media art festival LAB30 in Augsburg (2019, http://www.lab30.de/programm-2019/theodore). It also had to be exhibited at InTake in The Hague and Meta Marathon (https://www.metamarathon.net/) in Düsseldorf this year, both events were canceled due to coronavirus.

Weblinks

• homepage