Automatic quality prediction of authentically distorted pictures

Social media and rapid advances in camera and mobile device technology have led to the creation and consumption of a seemingly limitless supply of visual content. However, the vast majority of these digital images are captured by casual amateur photographers whose unsure hands and eyes often introduce annoying artifacts during acquisition. In addition, subsequent storage and transmission of visual media can further degrade their visual quality.

Recent developments in visual modeling have elucidated the impact of visual distortions on perception of such pictures and videos. They have laid the foundation for automatic and accurate metrics that can identify and predict the quality of visual media as perceived by human observers.¹To address this problem, several objective blind or no-reference (NR) image quality assessment (IQA) algorithms have been developed to predict the perceptual quality of a given (possibly distorted) image without additional information.^2–7 Such quality metrics could be used to monitor and control multimedia services on networks and devices or to prioritize quality of transmission over speed, for example.

Real-world images are usually afflicted by mixtures of distortions that differ significantly from the single, unmixed distortions contained in restrictive and unrepresentative legacy databases.^9–12 We recently designed a unique and challenging image data set with associated human opinion scores called the Laboratory for Image and Video Engineering (LIVE) authentic image quality challenge database⁸ (see Figure 1). Using this LIVE challenge database, we have been developing a robust blind IQA model for images suffering from real-world, authentic distortions. We call our model the ‘feature maps driven referenceless image quality evaluation engine’ (FRIQUEE) index. FRIQUEE outperforms other state-of-the-art blind IQA algorithms on both the LIVE legacy IQA⁹ and the LIVE challenge database⁸ (see Table 1).

Figure 1. Sample images from the Laboratory for Image and Video Engineering (LIVE) authentic image quality challenge database.⁸ This collection comprises 1163 images afflicted with complex mixtures of unknown distortions, of different types and severities, from diverse camera devices, and under varied illumination conditions. The content includes pictures of faces, people, animals, close-up shots, wide-angle shots, nature scenes, man-made objects, images with distinct foreground/background configurations, and images without any notable object of interest.

We have empirically observed that the presence of different mixtures of authentic distortions affects the scene statistics¹ of an image differently from when these distortions occur in isolation.¹³ FRIQUEE follows a feature-maps-driven statistical approach, avoiding any latent assumption about the type of distortion(s) contained in an image, focusing instead on the remarkable regularities of the scene statistics of real-world images in the absence of distortions. We capture a large and comprehensive collection of perceptually relevant and reliable ‘quality-sensitive’ statistical image features from these feature maps that supply us with greater discriminative power on authentic image distortion ensembles than the most successful NR IQA models. We also leverage the capability of a powerful deep belief network¹⁴ (DBN) to learn a model that maps the image features to complex feature representations that in turn predicts the subjective quality scores more accurately than the state-of-the-art models.

We process every image by first transforming it into color spaces known as RGB, LMS, and CIE LAB, then deriving a set of feature maps by applying operations (including steerable pyramid decomposition in the complex domain,⁷ difference of Gaussians,¹⁶ and Laplacian decomposition¹⁷) on the luminance component and on the four chroma components, i.e., A and B from LAB space, and M and S from the LMS color space (see Figure 2). The definition of each feature map is driven by established perceptual models of the transformations that occur at various stages of visual processing.

Figure 2. Given (a) any input image, our feature maps driven referenceless image quality evaluation engine (FRIQUEE) first constructs several feature maps in multiple transform domains—some are shown here (b–i)—then extracts scene statistics from these maps after performing perceptually significant divisive normalization¹⁵ on them.

We then perform perceptually significant debiasing and divisive normalization operations¹⁵ on each feature map and model the statistical regularities and irregularities exhibited by their histograms using a generalized Gaussian distribution (in real or complex domains)¹⁹ or an asymmetric generalized Gaussian distribution.²⁰We compute model parameters, such as shape and variance of these fits, and sample statistics, such as kurtosis and skewness, and use them as image features. A DBN, when trained with these image features as input, generates deep feature representations. These deep features are later used to train a support vector regressor (SVR) such that, given a test image's deep features, its quality is predicted (see Figure 3).

Figure 3. Configuration of our deep belief network (DBN) model. It has four hidden layers formed by stacking multiple restricted Boltzmann machines (RBMs). Each RBM is trained in a greedy layer-by-layer manner, with the hidden activities of one RBM as the visible input data for training a higher-level RBM. The number of visible units equals the number of features computed on each image. The features from the top three layers serve as ‘deep features’ to train a support vector regressor (SVR). The unit in the topmost layer of the DBN that is activated determines the quality class.

We evaluate the performance of several NR IQA algorithms by computing the correlation between the scores predicted by the algorithm and the ground truth subjective scores (see Table 1, which shows that our proposed model combining robust features and a DBN outperforms several other IQA models on unseen test data).

By identifying and addressing the challenges in perceptual quality prediction of images containing mixtures of authentic distortions using FRIQUEE, we have shown that novel techniques push the boundaries of quality prediction powers of state-of-the-art IQA models. To further improve the prediction power of blind IQA models on real-world distortions, we plan to combine techniques to consider high-level cognitive factors such as semantic content, attention, and aesthetic quality, in addition to accommodating skewness in the real-world distribution of distortions by machine learning.^21–23