Timezone: »
Machine co-creativity continues to grow and attract a wider audience to machine learning. Generative models, for example, have enabled new types of media creation across language, images, and music--including recent advances such as CLIP, VQGAN, and DALL·E. This one-day workshop will broadly explore topics in the applications of machine learning to creativity and design, which includes:
State-of-the-art algorithms for the creation of new media. Machine learning models achieving state-of-the-art in traditional media creation tasks (e.g., image, audio, or video synthesis) that are also being used by the artist community will be showcased.
Artist accessibility of machine learning models. Researchers building the next generation of machine learning models for media creation will be challenged in understanding the accessibility needs of artists. Artists and Human Computer interaction / User Experience community members will be encouraged to engage in the conversation.
The sociocultural and political impact of these new models. With the increased popularity of generative machine learning models, we are witnessing these models start to impact our everyday surroundings, ranging from racial and gender bias in algorithms and datasets used for media creation to how new media manipulation tools may erode our collective trust in media content.
Artistic applications. We will hear directly from some of the artists who are adopting machine learning--including deep learning and reinforcement learning--as part of their own artistic process as well as showcasing their work.
The goal of this workshop is to bring together researchers and artists interested in exploring the intersection of human creativity and machine learning and foster collaboration between them, as well as promote the sharing of ideas, perspectives, new research, artwork, and artistic needs.
Discord invite --> https://bit.ly/3puzVuM
Mon 8:15 a.m. - 8:30 a.m.
|
Welcome
(
Short Live Talk
)
SlidesLive Video » Welcome and introduction presented by Mattie Tesfaldet. |
🔗 |
Mon 8:30 a.m. - 9:30 a.m.
|
Poster Session 1
(
Poster Session
)
link »
Will be held in Discord. |
🔗 |
Mon 9:30 a.m. - 10:00 a.m.
|
Computers, Creativity, and Lovelace
(
Speaker Presentation
)
SlidesLive Video » |
Mark Riedl 🔗 |
Mon 10:00 a.m. - 10:30 a.m.
|
AI for Augmenting Human Creativity
(
Speaker Presentation
)
SlidesLive Video » |
Devi Parikh 🔗 |
Mon 10:30 a.m. - 11:00 a.m.
|
Interspecies Intelligence in Pharmako-AI
(
Speaker Presentation
)
SlidesLive Video » |
Kenric McDowell 🔗 |
Mon 11:00 a.m. - 11:30 a.m.
|
Art Show
(
Video
)
SlidesLive Video » Pre-recorded slide show of the art submissions |
🔗 |
Mon 11:30 a.m. - 12:00 p.m.
|
Panel Discussion: Mark Riedl, Devi Parikh, Kenric Allado-McDowell
(
Q&A Panel
)
SlidesLive Video » Mark, Devi, and Kenric on live panel answering questions, moderated by Mattie Tesfaldet. Please ask questions on Rocket.chat, not Discord. |
🔗 |
Mon 12:00 p.m. - 12:30 p.m.
|
Social 1
(
Informal Discussion
)
link »
Will be held in Discord. |
🔗 |
Mon 12:30 p.m. - 12:40 p.m.
|
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis
(
Paper Oral
)
SlidesLive Video » Generating images that fit a given text description using machine learning has improved greatly with the release of technologies such as the CLIP image-text encoder model; however, current methods lack artistic control of the style of image to be generated. We introduce StyleCLIPDraw which adds a style loss to the CLIPDraw text-to-drawing synthesis model to allow artistic control of the synthesized drawings in addition to control of the content via text. Whereas performing decoupled style transfer on a generated image only affects the texture, our proposed coupled approach is able to capture a style in both texture and shape, suggesting that the style of the drawing is coupled with the drawing process itself. More results and our code are available at https://github.com/pschaldenbrand/StyleCLIPDraw |
Peter Schaldenbrand · Jean Oh · Zhixuan Liu 🔗 |
Mon 12:40 p.m. - 12:50 p.m.
|
Soundify: Matching Sound Effects to Video
(
Paper Oral
)
SlidesLive Video » In the art of video editing, sound is really half the story. A skilled video editor overlays sounds, such as effects and ambients, over footage to add character to an object or immerse the viewer within a space. However, through formative interviews with professional video editors, we found that this process can be extremely tedious and time-consuming. We introduce Soundify, a system that matches sound effects to video. By leveraging labeled, studio-quality sound effects libraries and extending CLIP, a neural network with impressive zero-shot image classification capabilities, into a "zero-shot detector", we are able to produce high-quality results without resource-intensive correspondence learning or audio generation. We encourage you to have a look at, or better yet, have a listen to the results at https://chuanenlin.com/soundify. |
David Chuan-En Lin 🔗 |
Mon 12:50 p.m. - 1:00 p.m.
|
Controllable and Interpretable Singing Voice Decomposition via Assem-VC
(
Paper Oral
)
SlidesLive Video » We propose a singing decomposition system that encodes time-aligned linguistic content, pitch, and source speaker identity via Assem-VC. With decomposed speaker-independent information and the target speaker's embedding, we could synthesize the singing voice of the target speaker. In conclusion, we made a perfectly synced duet with the user's singing voice and the target singer's converted singing voice. |
Kangwook Kim · Junhyeok Lee 🔗 |
Mon 1:00 p.m. - 1:10 p.m.
|
Extending the Vocabulary of Fictional Languages using Neural Networks
(
Paper Oral
)
SlidesLive Video » Fictional languages have become increasingly popular over the recent years appearing in novels, movies, TV shows, comics, and video games. While some of these fictional languages have a complete vocabulary, most do not. We propose a deep learning solution to the problem. Using style transfer and machine translation tools, we generate new words for a given target fictional language, while maintaining the style of its creator, hence extending this language vocabulary. |
Thomas Zacharias · Raja Giryes 🔗 |
Mon 1:10 p.m. - 1:15 p.m.
|
Jabberwocky
(
Artwork Spotlight
)
SlidesLive Video » Famous poem, illustrated with the automated text-to-video synthesis tool. Rendered with CLIP-FFT method, optimizing frequencies of Fourier inverse transformation. voiceover: Benedict Cumberbatch pretrained ML model: github.com/openai/CLIP |
Vadim Epstein 🔗 |
Mon 1:15 p.m. - 1:20 p.m.
|
Iterative Iterative
(
Artwork Spotlight
)
SlidesLive Video » I began by feeding thousands of images of my past artwork into a GAN. As a ceramic artist, the source imagery were photographs of sculptures and pottery I made between 2015-2019. The GAN required me to augment the data in multiple ways, including photographing my work from all sides. My vision was that the GAN would "imagine" my future work based on past work. However, the GAN was limited in ways that made this unfeasible. It was trapped within my prior work and could not imagine the new. I developed a collaboration with the machine where the shortcomings of the GAN become generative of new form. I prompted the GAN with my past work, and I interpreted its output as a prompt to either create the unknown half or interpret the output in 3 dimensions. It was within this unknown space that a void was created which offered the opportunity for novelty. As the GAN navigated this void, attempting to create new form from old, I also navigated this void. I embrace and utilize these digital shortcomings as a marking of time within technology and my own art practice. |
Erin Smith 🔗 |
Mon 1:20 p.m. - 1:25 p.m.
|
Artificial Intelligence for High Heel Shoe Design
(
Artwork Spotlight
)
SlidesLive Video » The images generated in this project use the VQGAN+CLIP Script created by Katherine Crowson (2021), licensed under the MIT license. The model has been modified to create new designs of high heel shoes using a text prompt. The text prompt can be as simple as one describing word to redesign the entire shoe. The text prompt is manipulating an image of a high heel shoe , which is applied as an ‘init-image’. This tool can be used to create hundreds of original designs, whether you are a designer or not. Find more of Sophia's work at; https://www.sophianeilldesign.com |
Sophia Neill 🔗 |
Mon 1:25 p.m. - 1:30 p.m.
|
text2pixelart
(
Artwork Spotlight
)
SlidesLive Video » This project explores opportunities for an unsupervised generation of animated pixel-art based on the given text prompt. As the base, I used PixelDraw by @dribnet, based on the ClipDraw library, which in turn is based on the diffvg library. First, I've implemented some handy options, such as smoothness enforcement, palette enforcement, and saturation enforcement. Next, I've exploited the CLIP's "healing" ability and a couple of "demoscene" tricks to keep the coherence between adjacent frames to make several different animations: the infinite panorama, the parallax effect, the 3d swirl around the object, the moving background, the endless looping, and so on. I've tried to give a tribute to old-school classics (using prompts like [fallout], [r-type], and [Guybrush Threepwood]) and to explore the fractal and space thematics, trying to find the best match between the prompt and the effect in each case. The code of some tricks is published already; other hacks are on their way to be publicly available. Web link: https://altsoph.github.io/text2pixelart/ Full video can be downloaded here: https://www.dropbox.com/s/r88x7ze1wy8wjo5/text2pixelartclipv2.mp4?dl=0 |
Alexey Tikhonov 🔗 |
Mon 1:30 p.m. - 2:30 p.m.
|
Poster Session 2
(
Poster Session
)
link »
Will be held in Discord. |
🔗 |
Mon 2:30 p.m. - 3:00 p.m.
|
From Technologies to Tools
(
Speaker Presentation
)
SlidesLive Video » |
Joel Simon 🔗 |
Mon 3:00 p.m. - 3:30 p.m.
|
Okachihuali: Strategies for New Future Building with AI Art
(
Speaker Presentation
)
SlidesLive Video » |
Moisés Horta Valenzuela 🔗 |
Mon 3:30 p.m. - 4:00 p.m.
|
Imaginatory Processes with VQGANxCLIP
(
Speaker Presentation
)
SlidesLive Video » |
Sebastián Rojas · María Constanza Lobos 🔗 |
Mon 4:00 p.m. - 4:30 p.m.
|
Art Show (repeated)
(
Video
)
SlidesLive Video » |
🔗 |
Mon 4:30 p.m. - 5:00 p.m.
|
Panel Discussion: Joel Simon, Hypereikon
(
Q&A Panel
)
SlidesLive Video » Joel and Hypereikon on live panel answering questions, moderated by David Ha. Please ask questions on Rocket.chat, not Discord. |
🔗 |
Mon 5:00 p.m. - 5:15 p.m.
|
Closing Remarks
(
Short Live Talk
)
SlidesLive Video » Closing remarks presented by Tom White. |
🔗 |
Mon 5:15 p.m. - 6:00 p.m.
|
Social 2
(
Informal Discussion
)
link »
Will be held in Discord. |
🔗 |
Author Information
Tom White (Victoria University of Wellington School of Design)
Tom is a New Zealand based artist investigating machine perception. His current work focuses on creating physical artworks that highlight how machines “see” and thus how they think, suggesting that these systems are capable of abstraction and conceptual thinking. He has exhibited computer based artwork internationally over the past 25 years with themes of artificial intelligence, interactivity, and computational creativity. He is currently a lecturer and researcher at University of Wellington School of Design where he teaches students the creative potential of computer programming and artificial intelligence.
Mattie Tesfaldet (McGill University & MILA)
Mattie Tesfaldet (they/them) is a computer vision and machine learning researcher, artist, and DJ based in Montréal, Canada. They are pursuing their PhD at McGill University and Mila researching generative models for visual content creation, specifically, looking for novel and interesting ways images and videos can be represented with neural networks. Outside of academia, they like to apply their research with the aim of exploring the intersection of human creativity and artificial intelligence. Particularly, developing new AI-based mediums for communication, expression, and sharing of visual imagery.
Samaneh Azadi (UC Berkeley)
Daphne Ippolito (University of Pennsylvania)
Lia Coleman (RISD)
Lia Coleman is an AI researcher and artist whose work revolves around the interplay of AI technology, art & design, coding, and ethics. Currently, Lia conducts creative AI research at Carnegie Mellon University as a masters student in the Robotics Institute. They also organize the NeurIPS Conference Workshop on Machine Learning in Creativity and Design, and work with RunwayML on an explainable ML video series. They were an adjunct professor at the Rhode Island School of Design, and created a guide with the Partnership on AI for artists to use ML responsibly. Their work has been exhibited at / featured in Vox, Wired, Tribeca Film Festival, Mozilla Festival, Science Gallery Detroit, New York University, the NeurIPS Conference, and Gray Area. Their writing on AI art has been published by Princeton Architectural Press, DISEÑA, and Neocha Magazine. Lia is an alum of Massachusetts Institute of Technology (BSc, Computer Science) and the School For Poetic Computation in NYC. https://www.liacoleman.com/
David Ha (Google Brain)
More from the Same Authors
-
2021 : Benchmark for Compositional Text-to-Image Synthesis »
Dong Huk Park · Samaneh Azadi · Xihui Liu · Trevor Darrell · Anna Rohrbach -
2021 Spotlight: The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning »
Yujin Tang · David Ha -
2023 Workshop: NeurIPS 2023 Workshop on Machine Learning for Creativity and Design »
Yingtao Tian · Tom White · Lia Coleman · Hannah Johnston -
2022 : PORTAGING (live AI Performance) »
Kory Mathewson · Piotr Mirowski · Hannah Johnston · Tom White · Jason Baldridge -
2022 Workshop: Workshop on Machine Learning for Creativity and Design »
Tom White · Yingtao Tian · Lia Coleman · Samaneh Azadi -
2022 : Tangible Abstractions »
Tom White -
2022 Poster: Attention-based Neural Cellular Automata »
Mattie Tesfaldet · Derek Nowrouzezahrai · Chris Pal -
2021 Poster: The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning »
Yujin Tang · David Ha -
2020 : Panel Discussion 2 »
Tom White · Jesse Engel · Aaron Hertzmann · Stephanie Dinkins · Holly Grimm -
2020 Workshop: Machine Learning for Creativity and Design 4.0 »
Luba Elliott · Sander Dieleman · Adam Roberts · Tom White · Daphne Ippolito · Holly Grimm · Mattie Tesfaldet · Samaneh Azadi -
2020 Workshop: Resistance AI Workshop »
Suzanne Kite · Mattie Tesfaldet · J Khadijah Abdurahman · William Agnew · Elliot Creager · Agata Foryciarz · Raphael Gontijo Lopes · Pratyusha Kalluri · Marie-Therese Png · Manuel Sabin · Maria Skoularidou · Ramon Vilarino · Rose Wang · Sayash Kapoor · Micah Carroll -
2020 Session: Orals & Spotlights Track 04: Reinforcement Learning »
David Ha · Aleksandra Faust -
2019 : Poster Session 2 »
Mayur Saxena · Nicholas Frosst · Vivien Cabannes · Gene Kogan · Austin Dill · Anurag Sarkar · Joel Ruben Antony Moniz · Vibert Thio · Scott Sievert · Lia Coleman · Frederik De Bleser · Brian Quanz · Jonathon Kereliuk · Panos Achlioptas · Mohamed Elhoseiny · Songwei Ge · Aidan Gomez · Jamie Brew -
2019 Workshop: NeurIPS Workshop on Machine Learning for Creativity and Design 3.0 »
Luba Elliott · Sander Dieleman · Adam Roberts · Jesse Engel · Tom White · Rebecca Fiebrink · Parag Mital · Christine McLeavey · Nao Tokui -
2019 : Poster Session »
Ethan Harris · Tom White · Oh Hyeon Choung · Takashi Shinozaki · Dipan Pal · Katherine L. Hermann · Judy Borowski · Camilo Fosco · Chaz Firestone · Vijay Veerabadran · Benjamin Lahner · Chaitanya Ryali · Fenil Doshi · Pulkit Singh · Sharon Zhou · Michel Besserve · Michael Chang · Anelise Newman · Mahesan Niranjan · Jonathon Hare · Daniela Mihai · Marios Savvides · Simon Kornblith · Christina M Funke · Aude Oliva · Virginia de Sa · Dmitry Krotov · Colin Conwell · George Alvarez · Alex Kolchinski · Shengjia Zhao · Mitchell Gordon · Michael Bernstein · Stefano Ermon · Arash Mehrjou · Bernhard Schölkopf · John Co-Reyes · Michael Janner · Jiajun Wu · Josh Tenenbaum · Sergey Levine · Yalda Mohsenzadeh · Zhenglong Zhou -
2018 : SpaceSheets: Interactive Latent Space Exploration through a Spreadsheet Interface »
Tom White -
2018 Workshop: Second Workshop on Machine Learning for Creativity and Design »
Luba Elliott · Sander Dieleman · Rebecca Fiebrink · Jesse Engel · Adam Roberts · Tom White -
2016 Demonstration: Neural Puppet »
Tom White