After reading through several similar computer vision projects [1] and exploring different image features of my test cloud pictures, I settled on a list of six different thresholding techniques to evaluate. Each relied on extracting a particular feature from the images and then thresholding the pixels into two groups, cloud or sky pixels. Two of the six methods I tested rely on adaptive thresholding; the others identify an optimal fixed threshold for each particular feature.

Fixed Thresholding Techniques

Saturation: This feature was extracted from the HSV color space, where a color pixel is described by percentages of Hue, Saturation, and Value. Saturation specifically describes the “whiteness” of a color, with pure white having a saturation value of zero and pure black a value of one [2]. Within the context of this problem, we can hypothesize that a cloud pixel will have a lower saturation value than a sky pixel. I used OpenCV’s Imgproc.cvtColor(imageMatrix, hsvMatrix, Imgproc.COLOR_BGR2HSV) function to convert the cloud image to the HSV color space. The saturation channel from this color space is shown in the second image in Figure 1 below. Areas with lower saturation values are shown as darker because in the standard 0-255 gray scale, black corresponds to a value of 0. This range is non-intuitive because we would like to end up with a binary segmentation mask where cloud pixels are white, not black, so an inversion is performed using Core.bitwise_not(result, result). This operation produces the third image in Figure 1, where clouds are of a lighter shade than sky. The same reasoning for performing this inversion holds for other algorithms involving an inversion operation. Finally, Imgproc.THRESH_BINARY is used to perform the actual thresholding with a threshold value in the range 0-255.

Figure 1: Visualization of Process for Saturation Thresholding

Original Saturation Feature
from HSV Space
Inverted Saturation Binary Segmentation
Original Image Saturation Feature from HSV Space Inverted Saturation Binary Segmentation

Red-Blue Difference: This feature consists of the absolute difference between the red and blue saturation levels of a given pixel. It is calculated in the standard RGB (Red, Green, Blue) color space, where each saturation is specified by a value in the range 0-255. A pixel with RGB values of (0, 0, 0) corresponds to a fully black pixel, while (255, 255, 255) indicates pure white. To extract this feature, I used OpenCV functions Core.absdiff(redMatrix, blueMatrix, diffMatrix)Core.bitwise_not(result, result) to invert the result, and Imgproc.THRESH_BINARY. A higher difference between the red and blue saturation should correspond to sky pixels, where blue saturation will be quite high.

Red-Blue Ratio: For this feature, the red and blue values of a pixel were used to calculate a ratio of red saturation to blue saturation. This value was found using Core.divide(redMatrix, blueMatrix, ratioMatrix, 255), which scales the red-blue ratio to be in the range of 0-255. Sky pixels are expected to have a lower ratio because of their higher blue saturation.

Red-Blue Ratio Normalized: The final fixed thresholding feature computes (B-R)/(R+B) where R and B are the saturation values for the red and blue channels of the image respectively [1]. This ratio will be in the range of zero to one, but was scaled to 0-255 to better compare with the other features. I used OpenCV’s Core add, subtract, and divide functions, as well as a bitwise not inversion to prepare the image for thresholding using Imgproc.THRESH_BINARY.

Each of these techniques was used to prepare a feature matrix, which can be thought of as a gray-scale image where each pixel represents an intensity value for the features listed below in Figure 2. A visualization of the feature matrix and final segmented image for the Red-Blue Ratio feature is shown in Figure 3 below.

In order to find an optimal threshold value for each method, a range of 50 thresholds was tested on 100 randomly selected sample pictures for each technique. Ideally, all 50 thresholds would have been tested on all 1013 training images; however, this was computationally too intensive for the machine I was working on. Additionally, for the fixed thresholding techniques, since each pixel is thresholded independently and then compared to the corresponding pixel in the ground truth image, each individual pixel can be viewed as a training sample. Since the sky images in the SWIMSEG data set are 600 x 600 pixels,  each threshold was tested on 36,000,000 sample pixels to evaluate its effectiveness. Once an image was segmented using a test threshold, that image was compared to the ground truth from the SWIMSEG data set to calculate percentages of true positives, false positives, true negatives, and false negatives. These metrics were then used to compare the performance of each of the 50 thresholds, as explained in Comparing Thresholding Techniques. The optimal thresholds found for each technique are listed below in Figure 2. 

Adaptive Thresholding Techniques

Otsu’s Method for Adaptive Thresholding: This thresholding technique assumes a bimodal image with clearly defined foreground and background, as we have in the sky and cloud images. The algorithm iteratively tests thresholds to find which one produces the minimum variance in the foreground and background saturation values [3]. The feature extraction process is the same as described above for the fixed thresholds; the only difference is that an optimal threshold is found for every image instead of for the whole set of training images.

Figure 2: Thresholding Methods 

Features Calculation Optimal Threshold
Saturation ¬ S  178.5
Red – Blue Difference ¬ (R – B) 204.5
Red – Blue Ratio Normalized ¬ ((B – R)/(B+R)) 196.4
Red – Blue Ratio R/B 178.5
Adaptive Red – Blue Ratio R/B Calculated per image
Adaptive Red – Blue Difference ¬ (R – B) Calculated per image

Figure 3: Red-Blue Ratio Thresholding Visualization

Red-Blue Ratio Thresholding Visualization

References

  1. Dev, Soumyabrata, et al. “Color-Based Segmentation of Sky/Cloud Images From Ground-Based Cameras.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 10, no. 1, 12 June 2016, pp. 231–242.
  2. “4.3. The Hue-Saturation-Value (HSV) Color Model.” Introduction to Color Theory, New Mexico Tech, 16 Oct. 2012, infohost.nmt.edu/tcc/help/pubs/colortheory/web/hsv.html.
  3. Greensted, Andrew. “Otsu Thresholding.” The Lab Book Pages, 17 June 2010, www.labbookpages.co.uk/software/imgProc/otsuThreshold.html.