Songkuk Kim

Dr. Songkuk Kim


Research Focus & Metrics

Academic Career Length

11Years (2016-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
3 July 2023 in IEEE Access
Verified
Journal Article
Open access
HYHanju Yoo
Hanju Yoo
LDLinglong Dai
Linglong Dai
SKSongkuk Kim
Songkuk Kim
CCChan-Byoung Chae
Chan-Byoung Chae
Semantic communications have shown promising advancements by optimizing source and channel coding jointly. However, the dynamics of these systems remain understudied, limiting research and performance gains. Inspired by the robustness of Vision Transformers (ViTs) in handling image nuisances, we propose a ViT-based model for semantic communications. Our approach achieves a peak signal-to-noise ratio (PSNR) gain of +0.5 dB over convolutional neural network variants. We introduce novel measures, average cosine similarity and Fourier analysis, to analyze the inner workings of semantic communications and optimize the system’s performance. We also validate our approach through a real wireless channel prototype using software-defined radio (SDR). To the best of our knowledge, this is the first investigation of the fundamental workings of a semantic communications system, accompanied by the pioneering hardware implementation. To facilitate reproducibility and encourage further research, we provide open-source code, including neural network implementations and LabVIEW codes for SDR-based wireless transmission systems.

24 January 2018 in Advanced Energy Materials
Verified
Journal Article
Closed access
PDF file
DBDivij Bhatia
Divij Bhatia
JLJongseo Lee
Jongseo Lee
HHHee Jae Hwang
Hee Jae Hwang
JBJeong Min Baik
Jeong Min Baik
SKSongkuk Kim
Songkuk Kim
Dukhyun ChoiDukhyun Choi
Dukhyun Choi

8 June 2022 in arXiv
Verified
Preprint
Open access
PDF file
NPNamuk Park
Namuk Park
SKSongkuk Kim
Songkuk Kim
The success of multi-head self-attentions (MSAs) for computer vision is now indisputable. However, little is known about how MSAs work. We present fundamental explanations to help better understand the nature of MSAs. In particular, we demonstrate the following properties of MSAs and Vision Transformers (ViTs): (1) MSAs improve not only accuracy but also generalization by flattening the loss landscapes. Such improvement is primarily attributable to their data specificity, not long-range dependency. On the other hand, ViTs suffer from non-convex losses. Large datasets and loss landscape smoothing methods alleviate this problem; (2) MSAs and Convs exhibit opposite behaviors. For example, MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are complementary; (3) Multi-stage neural networks behave like a series connection of small individual models. In addition, MSAs at the end of a stage play a key role in prediction. Based on these insights, we propose AlterNet, a model in which Conv blocks at the end of a stage are replaced with MSA blocks. AlterNet outperforms CNNs not only in large data regimes but also in small data regimes. The code is available at https://github.com/xxxnell/how-do-vits-work.

24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
JKJuyeop Kim
Juyeop Kim
JPJun-Ha Park
Jun-Ha Park
SKSongkuk Kim
Songkuk Kim
JLJong-Seok Lee
Jong-Seok Lee
Neural networks with self-attention (a.k.a. Transformers) like ViT and Swin have emerged as a better alternative to traditional convolutional neural networks (CNNs). However, our understanding of how the new architecture works is still limited. In this paper, we focus on the phenomenon that Transformers show higher robustness against corruptions than CNNs, while not being overconfident. This is contrary to the intuition that robustness increases with confidence. We resolve this contradiction by empirically investigating how the output of the penultimate layer moves in the representation space as the input data moves linearly within a small area. In particular, we show the following. (1) While CNNs exhibit fairly linear relationship between the input and output movements, Transformers show nonlinear relationship for some data. For those data, the output of Transformers moves in a curved trajectory as the input moves linearly. (2) When a data is located in a curved region, it is hard to move it out of the decision region since the output moves along a curved trajectory instead of a straight line to the decision boundary, resulting in high robustness of Transformers. (3) If a data is slightly modified to jump out of the curved region, the movements afterwards become linear and the output goes to the decision boundary directly. In other words, there does exist a decision boundary near the data, which is hard to find only because of the curved representation space. This explains the underconfident prediction of Transformers. Also, we examine mathematical properties of the attention operation that induce nonlinear response to linear perturbation. Finally, we share our additional findings, regarding what contributes to the curved representation space of Transformers, and how the curvedness evolves during training.

3 October 2019 in IEEE Access
Verified
Journal Article
Open access
TLTaekyu Lee
Taekyu Lee
SMSeong-Eun Moon
Seong-Eun Moon
JBJongsoo Baek
Jongsoo Baek
JLJong-Seok Lee
Jong-Seok Lee
SKSongkuk Kim
Songkuk Kim
In this paper, we present our study on investigating the effect of music for sleep induction and wake-up. Unlike existing studies that mostly focused on understanding how specific types of music affect sleep for insomnia patients or older people in laboratory conditions, we aim to study the effect of pop music with respect to the musical features on both sleep and wake-up of young adults without serious sleep disorder in the real-world environment. We conduct two extensive experiments, for which we carefully design a system consisting of hardware and software components for reliable data collection. Our analysis focuses on what types of musical features are effective for fast sleep induction and wake-up. Furthermore, we present two application scenarios of sleep music recommendation based on our findings.

4 February 2021 in Sensors
Verified
Journal Article
Open access
PDF file
NPNamuk Park
Namuk Park
SKSongkuk Kim
Songkuk Kim
Efficient and accurate estimation of the probability distribution of a data stream is an important problem in many sensor systems. It is especially challenging when the data stream is non-stationary, i.e., its probability distribution changes over time. Statistical models for non-stationary data streams demand agile adaptation for concept drift while tolerating temporal fluctuations. To this end, a statistical model needs to forget old data samples and to detect concept drift swiftly. In this paper, we propose FlexSketch, an online probability density estimation algorithm for data streams. Our algorithm uses an ensemble of histograms, each of which represents a different length of data history. FlexSketch updates each histogram for a new data sample and generates probability distribution by combining the ensemble of histograms while monitoring discrepancy between recent data and existing models periodically. When it detects concept drift, a new histogram is added to the ensemble and the oldest histogram is removed. This allows us to estimate the probability density function with high update speed and high accuracy using only limited memory. Experimental results demonstrate that our algorithm shows improved speed and accuracy compared to existing methods for both stationary and non-stationary data streams.