Program Highlights »
Fri Dec 8th 08:00 AM -- 06:30 PM @ Hyatt Hotel, Seaview Ballroom
Machine Learning for Creativity and Design
Douglas Eck · David Ha · S. M. Ali Eslami · Sander Dieleman · Rebecca Fiebrink · Luba Elliott

Workshop Home Page

In the last year, generative machine learning and machine creativity have gotten a lot of attention in the non-research world. At the same time there have been significant advances in generative models for media creation and for design. This one-day workshop explores several issues in the domain of generative models for creativity and design. First, we will look at algorithms for generation and creation of new media and new designs, engaging researchers building the next generation of generative models (GANs, RL, etc) and also from a more information-theoretic view of creativity (compression, entropy, etc). Second, we will investigate the social and cultural impact of these new models, engaging researchers from HCI/UX communities. Finally, we’ll hear from some of the artists and musicians who are adopting machine learning approaches like deep learning and reinforcement learning as part of their artistic process. We’ll leave ample time for discussing both the important technical challenges of generative models for creativity and design, as well as the philosophical and cultural issues that surround this area of research.

In 2016, DeepMind’s AlphaGo made two moves against Lee Sedol that were described by the Go community as “brilliant,” “surprising,” “beautiful,” and so forth. Moreover, there was little discussion surrounding the fact that these very creative moves were actually made by a machine (Wired); it was enough that they were great examples of go playing. At the same time, the general public showed more concern for other applications of generative models. Algorithms that allow for convincing voice style transfer (Lyrebird) or puppet-like video face control (Face2Face) have raised concerns that generative ML will be used to make convincing forms of fake news (FastCompany).

Balancing this, the arts and music worlds have positively embraced generative models. Starting with DeepDream and expanding with image and video generation advances (e.g. GANs) we’ve seen lots of new and interesting art and music [citations] technologies provided by the machine learning community. We’ve seen research projects like Google Brain’s Magenta, Sony CSL’s FlowMachines and IBM’s Watson undertake collaborations and attempt to build tools and ML models for use by these communities.

Recent advances in generative models enable new possibilities in art and music production. Language models can be used to write science fiction film scripts (Sunspring) and even replicate the style of individual authors (Deep Tingle). Generative models for image and video allow us to create visions of people, places and things that resemble the distribution of actual images (GANs etc). Sequence modelling techniques have opened up the possibility of generating realistic musical scores (MIDI generation etc) and even raw audio that resembles human speech and physical instruments (DeepMind’s WaveNet, MILA’s Char2Wav and Google’s NSynth). In addition, sequence modelling allows us to model vector images to construct stroke-based drawings of common objects according to human doodles (sketch-rnn).

In addition to field-specific research, a number of papers have come out that are directly applicable to the challenges of generation and evaluation such as learning from human preferences (Christiano et al., 2017) and CycleGAN. The application of Novelty Search (Stanley), evolutionary complexification (Stanley - CPPN, NEAT, Nguyen et al - Plug&Play GANs, Innovation Engine) and intrinsic motivation (Oudeyer et al 2007, Schmidhuber on Fun and Creativity) techniques, where objective functions are constantly evolving, is still not common practice in art and music generation using machine learning.

Another focus of the workshop is how to better enable human influence over generative models. This could include learning from human preferences, exposing model parameters in ways that are understandable and relevant to users in a given application domain (e.g., similar to Morris et al. 2008), enabling users to manipulate models through changes to training data (Fiebrink et al. 2011), allowing users to dynamically mix between multiple generative models (Akten & Grierson 2016), or other techniques. Although questions of how to make learning algorithms controllable and understandable to users are relatively nacesent in the modern context of deep learning and reinforcement learning, such questions have been a growing focus of work within the human-computer interaction community (e.g., examined in a CHI 2016 workshop on Human-Centred Machine Learning), and the AI Safety community (e.g. Christiano et al. 2017, using human preferences to train deep reinforcement learning systems). Such considerations also underpin the new Google “People + AI Research” (PAIR) initiative.

Artists and Musicians
All the above techniques improve our capabilities of producing text, sound and images. Art and music that stands the test of time however requires more than that. Recent research includes a focus on novelty in creative adversarial networks (Elgammal et al., 2017) and considers how generative algorithms can integrate into human creative processes, supporting exploration of new ideas as well as human influence over generated content (Atken & Grierson 2016a, 2016b). Artists including Mario Klingemann, Gene Kogan, Mike Tyka, and Memo Akten have further contributed to this space of work by creating artwork that compellingly demonstrates capabilities of generative algorithms, and by publicly reflecting on the artistic affordances of these new tools.

The goal of this workshop is to bring together researchers interested in advancing art and music generation to present new work, foster collaborations and build networks.

In this workshop, we are particularly interested in how the following can be used in art and music generation: reinforcement learning, generative adversarial networks, novelty search and evaluation as well as learning from user preferences. We welcome submissions of short papers, demos and extended abstracts related to the above.

There will also be an open call for a display of artworks incorporating machine learning techniques.

08:30 AM Welcome and Introduction (Introduction)
08:45 AM Invited Talk (Talk)
Jürgen Schmidhuber
09:15 AM Invited Talk (Talk)
Emily Denton
09:45 AM Invited Talk (Talk)
Rebecca Fiebrink
10:15 AM GANosaic - Mosaic Creation with Generative Texture Manifolds (Spotlight talk)
Nikolay Jetchev, Urs Bergmann, Calvin Seward
10:20 AM TopoSketch: Drawing in Latent Space (Spotlight talk)
10:25 AM Input parameterization for DeepDream (Spotlight talk)
11:00 AM Invited Talk (Talk)
Ian Goodfellow
11:30 AM Improvised Comedy as a Turing Test (Contributed Talk)
12:00 PM Lunch
01:00 PM Invited Talk (Talk)
Ahmed Elgammal
01:30 PM Hierarchical Variational Autoencoders for Music (Contributed Talk)
02:00 PM Lexical preferences in an automated story writing system (Contributed Talk)
02:30 PM ObamaNet: Photo-realistic lip-sync from text (Contributed Talk)
Rithesh Kumar, Jose Sotelo, kundan Kumar, Alexandre de Brébisson
03:00 PM Art / Coffee Break (Break)
03:30 PM Towards the High-quality Anime Characters Generation with Generative Adversarial Networks (Spotlight talk)
03:35 PM Crowd Sourcing Clothes Design Directed by Adversarial Neural Networks (Spotlight talk)
Hiroyuki Osone, Natsumi Kato, Daitetsu Sato, Naoya Muramatsu
03:40 PM Paper Cubes: Evolving 3D characters in Augmented Reality using Recurrent Neural Networks (Spotlight talk)
Anna Fuste, Jonas Jongejan
03:45 PM Open discussion (Discussion)
04:15 PM Generating Black Metal and Math Rock: Beyond Bach, Beethoven, and Beatles (Poster)
04:15 PM Algorithmic composition of polyphonic music with the WaveCRF (Poster)
04:15 PM SOMNIA: Self-Organizing Maps as Neural Interactive Art (Poster)
04:15 PM The Emotional GAN: Priming Adversarial Generation of Art with Emotion (Poster)
Judith Amores Fernandez
04:15 PM Compositional Pattern Producing GAN (Poster)
04:15 PM Generative Embedded Mapping Systems for Design (Poster)
04:15 PM Exploring Audio Style Transfer (Poster)
04:15 PM Imaginary Soundscape : Cross-Modal Approach to Generate Pseudo Sound Environments (Poster)
Yuma Kajihara, Nao Tokui
04:15 PM Repeating and Remembering: GANs in an art context (Poster)
Anna Ridler
04:15 PM Improvisational Storytelling Agents (Poster)
04:15 PM Learning to Create Piano Performances (Poster)
04:15 PM Neural Style Transfer for Audio Spectograms (Poster)
Prateek Verma, Julius O Smith
04:15 PM SocialML: machine learning for social media video creators (Poster)
04:15 PM Artwork
Daniel Ambrosi, Michel Erler, Jason Salavon, Theresa Reimann-Dubbers, Robbie Barrat
04:15 PM Deep Interactive Evolutionary Computation (Poster)
Philip Bontrager
04:15 PM ASCII Art Synthesis with Convolutional Networks (Poster)
04:15 PM Disentangled representations of style and content for visual art with generative adversarial networks (Poster)
04:15 PM Sequential Line Search for Generative Adversarial Networks (Poster)
04:15 PM AI for Fragrance Design (Poster)
Richard Segal
04:15 PM Consistent Comic Colorization with Pixel-wise Background Classification (Poster)
Jaegul Choo, Sungmin Kang
04:15 PM Deep Learning for Identifying Potential Conceptual Shifts for Co-creative Drawing (Poster)
Pegah Karimi
04:15 PM Combinatorial Meta Search (Poster)