Timezone: »
Counterpoint is an important concept in Western music theory. In the past century, there have been significant interests in incorporating counterpoint into Chinese folk music composition. In this paper, we propose a reinforcement learning-based system, named FolkDuet, towards the online countermelody generation for Chinese folk melodies. With no existing data of Chinese folk duets, FolkDuet employs two reward models based on out-of-domain data, i.e. Bach chorales, and monophonic Chinese folk melodies. An interaction reward model is trained on the duets formed from outer parts of Bach chorales to model counterpoint interaction, while a style reward model is trained on monophonic melodies of Chinese folk songs to model melodic patterns. With both rewards, the generator of FolkDuet is trained to generate countermelodies while maintaining the Chinese folk style. The entire generation process is performed in an online fashion, allowing real-time interactive human-machine duet improvisation. Experiments show that the proposed algorithm achieves better subjective and objective results than the baselines.
Author Information
Nan Jiang (Tsinghua University)
Sheng Jin (Tsinghua University)
Zhiyao Duan (Unversity of Rochester)
Zhiyao Duan is an associate professor in Electrical and Computer Engineering, Computer Science and Data Science at the University of Rochester. He received his B.S. in Automation and M.S. in Control Science and Engineering from Tsinghua University, China, in 2004 and 2008, respectively, and received his Ph.D. in Computer Science from Northwestern University in 2013. His research interest is in the broad area of computer audition, i.e., designing computational systems that are capable of understanding sounds, including music, speech, and environmental sounds. He is also interested in the connections between computer audition and computer vision, natural language processing, and augmented and virtual reality. He received a best paper award at the 2017 Sound and Music Computing (SMC) conference, a best paper nomination at the 2017 International Society for Music Information Retrieval (ISMIR) conference, a BIGDATA award and a CAREER award from the National Science Foundation (NSF). His research is funded by NSF, NIH, New York State Center of Excellence on Data Science, and University of Rochester internal awards on AR/VR, data science and health analytics.
Changshui Zhang (Tsinghua University)
More from the Same Authors
-
2022 Poster: Synergy-of-Experts: Collaborate to Improve Adversarial Robustness »
Sen Cui · Jingfeng ZHANG · Jian Liang · Bo Han · Masashi Sugiyama · Changshui Zhang -
2021 Poster: Addressing Algorithmic Disparity and Performance Inconsistency in Federated Learning »
Sen Cui · Weishen Pan · Jian Liang · Changshui Zhang · Fei Wang -
2021 Poster: Skipping the Frame-Level: Event-Based Piano Transcription With Neural Semi-CRFs »
Yujia Yan · Frank Cwitkowitz · Zhiyao Duan -
2021 Poster: ReSSL: Relational Self-Supervised Learning with Weak Augmentation »
Mingkai Zheng · Shan You · Fei Wang · Chen Qian · Changshui Zhang · Xiaogang Wang · Chang Xu -
2020 Poster: Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space »
Shangchen Du · Shan You · Xiaojie Li · Jianlong Wu · Fei Wang · Chen Qian · Changshui Zhang -
2019 Poster: Subspace Attack: Exploiting Promising Subspaces for Query-Efficient Black-box Attacks »
Yiwen Guo · Ziang Yan · Changshui Zhang -
2018 Poster: Sparse DNNs with Improved Adversarial Robustness »
Yiwen Guo · Chao Zhang · Changshui Zhang · Yurong Chen -
2018 Poster: Connectionist Temporal Classification with Maximum Entropy Regularization »
Hu Liu · Sheng Jin · Changshui Zhang -
2018 Spotlight: Connectionist Temporal Classification with Maximum Entropy Regularization »
Hu Liu · Sheng Jin · Changshui Zhang -
2018 Poster: Deep Defense: Training DNNs with Improved Adversarial Robustness »
Ziang Yan · Yiwen Guo · Changshui Zhang -
2012 Poster: Multi-Stage Multi-Task Feature Learning »
Pinghua Gong · Jieping Ye · Changshui Zhang -
2012 Spotlight: Multi-Stage Multi-Task Feature Learning »
Pinghua Gong · Jieping Ye · Changshui Zhang -
2010 Poster: Learning Kernels with Radiuses of Minimum Enclosing Balls »
Kun Gai · Guangyun Chen · Changshui Zhang