Jia-Hong Huang

Jia-Hong Huang

Auto-generated profile, based on publicly available information

Jia-Hong Huang

University of Amsterdam, Amsterdam, Netherlands

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
30 May 2024
Verified
Conference Paper
Open access
HZHongyi Zhu
Hongyi Zhu
JHJia-Hong Huang
Jia-Hong Huang
SRStevan Rudinac
Stevan Rudinac
EKEvangelos Kanoulas
Evangelos Kanoulas
Image search stands as a pivotal task in multimedia and computer vision, finding applications across diverse domains, ranging from internet search to medical diagnostics. Conventional image search systems operate by accepting textual or visual queries, retrieving the top-relevant candidate results from the database. However, prevalent methods often rely on single-turn procedures, introducing potential inaccuracies and limited recall. These methods also face the challenges, such as vocabulary mismatch and the semantic gap, constraining their overall effectiveness. To address these issues, we propose an interactive image retrieval system capable of refining queries based on user relevance feedback in a multi-turn setting. This system incorporates a vision language model (VLM) based image captioner to enhance the quality of text-based queries, resulting in more informative queries with each iteration. Moreover, we introduce a large language model (LLM) based denoiser to refine text-based query expansions, mitigating inaccuracies in image descriptions generated by captioning models. To evaluate our system, we curate a new dataset by adapting the MSR-VTT video retrieval dataset to the image retrieval task, offering multiple relevant ground truth images for each query. Through comprehensive experiments, we validate the effectiveness of our proposed system against baseline methods, achieving state-of-the-art performance with a notable 10% improvement in terms of recall. Our contributions encompass the development of an innovative interactive image retrieval system, the integration of an LLM-based denoiser, the curation of a meticulously designed evaluation dataset, and thorough experimental validation.

17 July 2019 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
JHJia-Hong Huang
Jia-Hong Huang
CDCuong Duc Dao
Cuong Duc Dao
MAModar Alfadly
Modar Alfadly
BGBernard Ghanem
Bernard Ghanem
Deep neural networks have been playing an essential role in many computer vision tasks including Visual Question Answering (VQA). Until recently, the study of their accuracy was the main focus of research but now there is a trend toward assessing the robustness of these models against adversarial attacks by evaluating their tolerance to varying noise levels. In VQA, adversarial attacks can target the image and/or the proposed main question and yet there is a lack of proper analysis of the later. In this work, we propose a flexible framework that focuses on the language part of VQA that uses semantically relevant questions, dubbed basic questions, acting as controllable noise to evaluate the robustness of VQA models. We hypothesize that the level of noise is negatively correlated to the similarity of a basic question to the main question. Hence, to apply noise on any given main question, we rank a pool of basic questions based on their similarity by casting this ranking task as a LASSO optimization problem. Then, we propose a novel robustness measure Rscore and two largescale basic question datasets (BQDs) in order to standardize robustness analysis for VQA models.

19 June 2019
Verified
Book Chapter
Open access
PDF file
YLYi-Chieh Liu
Yi-Chieh Liu
HYHao-Hsiang Yang
Hao-Hsiang Yang
CYC.-H. Huck Yang
C.-H. Huck Yang
JHJia-Hong Huang
Jia-Hong Huang
MTMeng Tian
Meng Tian
JTJesper Tegner
Jesper Tegner
Age-Related Macular Degeneration (AMD) is an asymptomatic retinal disease which may result in loss of vision. There is limited access to high-quality relevant retinal images and poor understanding of the features defining sub-classes of this disease. Motivated by recent advances in machine learning we specifically explore the potential of generative modeling, using Generative Adversarial Networks (GANs) and style transferring, to facilitate clinical diagnosis and disease understanding by feature extraction. We design an analytic pipeline which first generates synthetic retinal images from clinical images; a subsequent verification step is applied. In the synthesizing step we merge GANs (DCGANs and WGANs architectures) and style transferring for the image generation, whereas the verified step controls the accuracy of the generated images. We find that the generated images contain sufficient pathological details to facilitate ophthalmologists’ task of disease classification and in discovery of disease relevant features. In particular, our system predicts the drusen and geographic atrophy sub-classes of AMD. Furthermore, the performance using CFP images for GANs outperforms the classification based on using only the original clinical dataset. Our results are evaluated using existing classifier of retinal diseases and class activated maps, supporting the predictive power of the synthetic images and their utility for feature extraction. Our code examples are available online. ( https://github.com/huckiyang/EyeNet-GANs ).

27 February 2026 in ACM Transactions on Multimedia Computing, Communications, and Applications
Verified
Journal Article
Open access
JHJia-Hong Huang
Jia-Hong Huang
CYChao-Han Huck Yang
Chao-Han Huck Yang
PCPin-Yu Chen
Pin-Yu Chen
MCMin-Hung Chen
Min-Hung Chen
MWMarcel Worring
Marcel Worring
The aim of video summarization is to shorten videos automatically while retaining the key information necessary to convey the overall story. Video summarization methods mainly rely on visual factors, such as visual consecutiveness and diversity, which may not be sufficient to fully understand the content of the video. There are other non-visual factors, such as interestingness, representativeness, and storyline consistency that should also be considered for generating high-quality video summaries. Current methods do not adequately take into account these non-visual factors, resulting in suboptimal performance. In this work, a new approach to video summarization is proposed based on insights gained from how humans create ground truth video summaries. The method utilizes a conditional modeling perspective and introduces multiple meaningful random variables and joint distributions to characterize the key components of video summarization. Helper distributions are employed to improve the training of the model. A conditional attention module is designed to mitigate potential performance degradation in the presence of multi-modal input. The proposed video summarization method incorporates the above innovative design choices that aim to narrow the gap between human-generated and machine-generated video summaries. Extensive experiments show that the proposed approach outperforms existing methods and achieves state-of-the-art performance on commonly used video summarization datasets.

19 June 2019
Verified
Book Chapter
Open access
PDF file
CYC.-H. Huck Yang
C.-H. Huck Yang
FLFangyu Liu
Fangyu Liu
JHJia-Hong Huang
Jia-Hong Huang
MTMeng Tian
Meng Tian
MLM. D. I-Hung Lin
M. D. I-Hung Lin
JTJesper Tegner
Jesper Tegner
Automatic clinical diagnosis of retinal diseases has emerged as a promising approach to facilitate discovery in areas with limited access to specialists. Based on the fact that fundus structure and vascular disorders are the main characteristics of retinal diseases, we propose a novel visual-assisted diagnosis hybrid model mixing the support vector machine (SVM) and deep neural networks (DNNs). Furthermore, we present a new clinical retina labels collection sorted by the professional ophthalmologist from the educational project Retina Image Bank, called EyeNet, for ophthalmology incorporating 52 retina diseases classes. Using EyeNet, our model achieves 90.40% diagnosis accuracy, and the model performance is comparable to the professional ophthalmologists ( https://github.com/huckiyang/EyeNet2 ).

30 May 2024
Verified
Conference Paper
Open access
JHJia-Hong Huang
Jia-Hong Huang
The increasing prevalence of video content on platforms like YouTube and Vimeo has led to a growing demand for effective video summarization techniques. These methods aim to extract relevant content from videos and present it in a condensed form, addressing the challenge of information overload faced by users. However, the quality of the generated summaries is crucial, especially in fields like forensics, journalism, and sports analysis, where accurate and concise summaries are essential for decision-making and analysis. Traditional video summarization methods often rely solely on visual information, limiting their ability to capture textual cues and emotional content present in videos. To overcome this limitation, this thesis proposes leveraging text-based queries as context to enhance the effectiveness of video summarization. By incorporating textual information alongside visual data, the proposed approach aims to generate query-dependent video summaries that better align with users' requirements. Furthermore, the thesis leverages a novel conditional modeling perspective to impart a more human-like quality to the video summarization process. These methods hold promise for various applications, including documentary filmmaking and educational content creation. Additionally, the thesis addresses the challenge of data scarcity in video summarization by proposing a self-supervised learning approach that leverages pretext tasks to generate pseudo-labels for model training. In 2020, we proposed a query-controllable video summarization technology, which is elaborated upon in the second chapter of this thesis. Notably, Google has adopted a similar feature, integrating it into their large language model (LLM) Gemini Pro-1.5 version, in 2024.