With the emergence of large digitized fine art collections and the successful performance of deep learning techniques, new research prospects unfold in the intersection of artificial intelligence and art. In order to explore the applicability of deep learning techniques in understanding art images beyond object recognition and classification, we employ convolutional neural networks (CNN) to predict scores related to three subjective aspects of human perception: aesthetic evaluation of the image, sentiment evoked by the image and memorability of the image. For each concept, we evaluate several different CNN models trained on various natural image datasets and select the best performing model based on the qualitative results and the comparison with existing subjective ratings of artworks. Furthermore, we employ different decision tree based machine learning models to analyze the relative importance of various image features related to content, composition and color in determining image aesthetics, visual sentiment and memorability scores. Our findings suggest that content and image lighting have significant influence on aesthetics, that color vividness and harmony strongly influence sentiment prediction, while object emphasis has a high impact on memorability. Additionally, we explore the predicted aesthetic, sentiment and memorability scores in the context of art history by analyzing their distribution in regard to different artistic styles, genres, artists and centuries. The presented approach enables new ways of exploring fine art collections based on highly subjective aspects of art, as well as represents one step forward towards bridging the gap between traditional formal analysis and computational analysis of fine art.

To automatically generate accurate and meaningful textual descriptions of images is an ongoing research challenge. Recently, a lot of progress has been made by adopting multimodal deep learning approaches for integrating vision and language. However, the task of developing image captioning models is most commonly addressed using datasets of natural images, while not many contributions have been made in the domain of artwork images. One of the main reasons for that is the lack of large-scale art datasets of adequate image-text pairs. Another reason is the fact that generating accurate descriptions of artwork images is particularly challenging because descriptions of artworks are more complex and can include multiple levels of interpretation. It is therefore also especially difficult to effectively evaluate generated captions of artwork images. The aim of this work is to address some of those challenges by utilizing a large-scale dataset of artwork images annotated with concepts from the Iconclass classification system. Using this dataset, a captioning model is developed by fine-tuning a transformer-based vision-language pretrained model. Due to the complex relations between image and text pairs in the domain of artwork images, the generated captions are evaluated using several quantitative and qualitative approaches. The performance is assessed using standard image captioning metrics and a recently introduced reference-free metric. The quality of the generated captions and the model’s capacity to generalize to new data is explored by employing the model to another art dataset to compare the relation between commonly generated captions and the genre of artworks. The overall results suggest that the model can generate meaningful captions that indicate a stronger relevance to the art historical context, particularly in comparison to captions obtained from models trained only on natural image datasets.

Extensive digitization efforts in the recent years have led to a large increase of digitized and online available fine-art collections. With digitization of artworks, we aim to preserve all those valuable evidences of various human creative expressions, as well as make them available to a broader audience. The digitalization process of artworks should not constrain only to fulfilling the purpose of preservation, but also serve as a starting point for exploring of this type of data in a novel way, which is made possible with the rise of new achievements in computer vision. In the domain of computer analysis of visual art there are various ongoing research challenges. In this paper, we explore different image feature extraction methods and their applicability in the task of classifying painting by genre. Our dataset includes paintings of various styles grouped in seven genre categories. We achieved an accuracy of 77.57% for the task of genre classification. We concluded that the best performance is achieved when using features derived from a pretrained deep convolutional neural network.

Hands represent an important aspect of pictorial narration but have rarely been addressed as an object of study in art history and digital humanities. Although hand gestures play a significant role in conveying emotions, narratives, and cultural symbolism in the context of visual art, a comprehensive terminology for the classification of depicted hand poses is still lacking. In this article, we present the process of creating a new annotated dataset of pictorial hand poses. The dataset is based on a collection of European early modern paintings, from which hands are extracted using human pose estimation (HPE) methods. The hand images are then manually annotated based on art historical categorization schemes. From this categorization, we introduce a new classification task and perform a series of experiments using different types of features, including our newly introduced 2D hand keypoint features, as well as existing neural network-based features. This classification task represents a new and complex challenge due to the subtle and contextually dependent differences between depicted hands. The presented computational approach to hand pose recognition in paintings represents an initial attempt to tackle this challenge, which could potentially advance the use of HPE methods on paintings, as well as foster new research on the understanding of hand gestures in art.