Nian Liu

Nian Liu

Auto-generated profile, based on publicly available information

Nian Liu

Beijing University of Posts and Telecommunications, Beijing, China and Beijing Institute for General Artificial Intelligence, Beijing, China

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 October 2024 in IEEE Transactions on Computational Social Systems
Verified
Journal Article
Closed access
XTXinyi Tong
Xinyi Tong
SCSitong Chen
Sitong Chen
PYPeiyang Yu
Peiyang Yu
NLNian Liu
Nian Liu
HQHui Qv
Hui Qv
SZSong-Chun Zhu
Song-Chun Zhu
Automatically generating video background music mitigates the inefficiency and time-consuming drawbacks of current manual video editing. Two key challenges hinder the expansion of the inception of video-to-music tasks. 1) Limited availability of high-quality video–music datasets and annotations. 2) Absence of music generation methods that consider actual musicality, which are controlled by interpretable factors based on music theory. In the article, we propose video echoed in harmony (VEH), a method for learning and sampling video-integrated chord progression sequences. Our approach adopts harmony, represented by chord progressions that are aligned with various music formats [musical instrument digital interface (MIDI), audio, and score], imitating chord precedence in human music composition. Visual-language models link visual features to chord progressions through genre labels and descriptive words in generated textualized videos. The two aforementioned features collectively obviate the necessity of extensive video–music paired data. Besides, an energy-based chord progression learning and sampling algorithm quantifies abstract harmony impressions to statistical features, serving as interpretable factors for the controllable music generation based on music theory. Experimental results demonstrate that the proposed method outperforms the state-of-the-art, producing a superior music alignment for the given video.