Increasingly, we use mobile applications and services in our daily life activities, to support our needs for information, communication or leisure. However, user acceptance of a mobile application depends on at least two conditions: the application's perceived experience, and the appropriateness of the application to the user's context and needs. However, we have a weak understanding of a mobile user's quality of experience (QoE) and the factors influencing it. This article presents a 4-week-long 29-Androidphone- user study, where we collected both QoE and the underlying network's quality of service measurements through a combination of user, application, and network data on the user's phones. We aimed to derive and improve the understanding of users' QoE for a set of widely used mobile applications in users' natural environments and different daily contexts. We present data acquired in the study and discuss implications for mobile applications design.

Traditionally, audio quality and video quality are evaluated separately in subjective tests. Best practices within the quality assessment community were developed before many modern mobile audiovisual devices and services came into use, such as internet video, smart phones, tablets and connected televisions. These devices and services raise unique questions that require jointly evaluating both the audio and the video within a subjective test. However, audiovisual subjective testing is a relatively under-explored field. In this paper, we address the question of determining the most suitable way to conduct audiovisual subjective testing on a wide range of audiovisual quality. Six laboratories from four countries conducted a systematic study of audiovisual subjective testing. The stimuli and scale were held constant across experiments and labs; only the environment of the subjective test was varied. Some subjective tests were conducted in controlled environments and some in public environments (a cafeteria, patio or hallway). The audiovisual stimuli spanned a wide range of quality. Results show that these audiovisual subjective tests were highly repeatable from one laboratory and environment to the next. The number of subjects was the most important factor. Based on this experiment, 24 or more subjects are recommended for Absolute Category Rating (ACR) tests. In public environments, 35 subjects were required to obtain the same Student's t-test sensitivity. The second most important variable was individual differences between subjects. Other environmental factors had minimal impact, such as language, country, lighting, background noise, wall color, and monitor calibration. Analyses indicate that Mean Opinion Scores (MOS) are relative rather than absolute. Our analyses show that the results of experiments done in pristine, laboratory environments are highly representative of those devices in actual use, in a typical user environment.

Recently an impressive development in immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR) and 360 video, has been witnessed. However, methods for quality assessment have not been keeping up. This paper studies quality assessment of 360 video from the cross-lab tests (involving ten laboratories and more than 300 participants) carried out by the Immersive Media Group (IMG) of the Video Quality Experts Group (VQEG). These tests were addressed to assess and validate subjective evaluation methodologies for 360 video. Audiovisual quality, simulator sickness symptoms, and exploration behavior were evaluated with short (from 10 seconds to 30 seconds) 360 sequences. The following factors' influences were also analyzed: assessment methodology, sequence duration, Head-Mounted Display (HMD) device, uniform and non-uniform coding degradations, and simulator sickness assessment methods. The obtained results have demonstrated the validity of Absolute Category Rating (ACR) and Degradation Category Rating (DCR) for subjective tests with 360 videos, the possibility of using 10-second videos (with or without audio) when addressing quality evaluation of coding artifacts, as well as any commercial HMD (satisfying minimum requirements). Also, more efficient methods than the long Simulator Sickness Questionnaire (SSQ) have been proposed to evaluate related symptoms with 360 videos. These results have been instrumental for the development of the ITU-T Recommendation P.919. Finally, the annotated dataset from the tests is made publicly available for the research community.

The concept of conducting ecologically valid user studies is gaining traction in the field of Quality of Experience (QoE). However, despite previous research exploring this concept, the increasing volume of studies has made it challenging to obtain a comprehensive overview of existing guidelines and the key aspects to consider when designing ecologically valid studies. Therefor this paper aims to provide a systematic review of research articles published between 2011 and 2021 that offer insight into conducting ecologically valid user studies. From an initial count of 782 retrieved studies, a final count of 12 studies met the predefined criteria and were included in the final review. The systematic review resulted in the extraction of 55 guidelines that provide guidance towards conducting ecologically valid user studies. These guidelines have been grouped within 8 categories (Environment, Technology, Content, Participant Recruitment, User Behavior, Study Design, Task and data collection) overarching the three main dimensions (Setting, Users and Research Methodology). Furthermore, the review discusses: the flip side of ecological validity, the implications for QoE research, as well as provides a basic visualisation model for assessing the ecological validity of a study. In conclusion, the current review indicates that future research should address more in detail how and when research approaches characterized by high ecological validity (and correspondingly, low internal validity) and those characterized by low ecological validity (and normally high internal validity) can best complement each other in order to better understand the key factors influencing QoE for various types of applications, user segments, settings. Further, we argue that more transparency around the (sub)dimensions of ecological validity with respect to a particular study or set of studies is necessary.

Subjective tests of telecommunication services engaging a panel of subjects aim to analyze how service users experience their quality (QoE) with varying parameters of both a network and/or application itself. Subjects express their opinion on a few-grade categorical scale during tests performed in a controlled environment. The proposed Generalized Linear Model (GLZ)1 relates probabilities of particular QoE levels to independent network and application explanatory variables in a statistically credible way. The GLZ model is executed in several steps including maximum likelihood estimation of model parameters, selection of the most effective model with the Bayesian information criterion, model validation with a chi-squared test, and last but not least confidence interval estimation. Monitoring a QoE level with a GLZ model provides to operators an effective tool towards a mid- and long-term adaptation of both network and/or application parameters in order to follow users' experience and expectations.

The paper proposes a No-Reference (NR) metric to objectively assess the H.264/AVC video quality. The proposed model takes into account the typical artefacts introduced by hybrid block-based motion compensated predictive video codecs as the one related to the H.264/AVC standard. More specifically, these artefacts are the blockiness introduced at the boundaries of each coded block and the temporal flickering due to different coding modes used for the same macroblock along the video sequence. Furthermore, a flickering metric for intra coded frames is also derived. The quality prediction accuracy of the proposed NR quality metric is validated over subjective data collected during a video subjective evaluation experiments. Moreover, the quality prediction accuracy is also compared with the one provided by the well known state-of-the-art Structural SIMilarity (SSIM) metric which works in a full-reference mode. The proposed metric achieves a higher Pearson's correlation coefficient with subjective scores than the one achieved by the SSIM metric.