A Printer Indexing System for Color Calibration with Applications in Dietary Assessment

Abstract

In image based dietary assessment, color is a very important feature in food identification. One issue with using color in image analysis in the calibration of the color imaging capture system. In this paper we propose an indexing system for color camera calibration using printed color checkerboards also known as fiducial markers (FMs). To use the FM for color calibration one must know which printer was used to print the FM so that the correct color calibration matrix can be used for calibration. We have designed a printer indexing scheme that allows one to determine which printer was used to print the FM based on a unique arrangement of color squares and binarized marks (used for error control) printed on the FM. Using normalized cross correlation and pattern detection, the index corresponding to the printer for a particular FM can be determined. Our experimental results show this scheme is robust against most types of lighting conditions.

1 Introduction

In this paper we describe a printer indexing system for color image calibration used in a mobile telephone dietary assessment system, the Technology Assisted Dietary Assessment (TADA) system, that we have been developing for the past seven years [1,3,15]. The TADA system, and the associated mobile Food Record (mFR), allows users to acquire food images using a mobile telephone [1]. Image processing and analysis methods are then used to determine the food type and nutrient value of the food [1,14,15]. The current system uses color as an important feature for identifying food types, therefore it is crucial to maintain the consistency of color for accurate food classification [14]. A color calibration process based on reference information is required prior to food classification to eliminate the influences of varying lighting conditions and mitigate variations in camera sensor response. To provide reference information, we have designed a color checkerboard pattern or a fiducial marker (FM) as illustrated in Figure 1(a). This color checkerboard consists of M colors where M = 11 including background “white” for our current version of the FM. The fiducial marker is included in the scene by the user to serve as a reference for the estimation of scale and pose of the objects in the scene and to provide reference information for color calibration [12]. Our research group has generated and distributed all the FMs used in our previous studies by printing the FMs on the same printer (a Canon i9900). As the number of users in our studies increases, we need to develop a method for the users to generate the FM themselves.

Fig. 1.

Food images from our TADA system with the fiducial marker (FM) present.

The issue of reproducing colors is a fundamentally difficult problem [10]. We have tested printing the FMs using various printers and significant color mismatch can be observed based on both the perception of a human observer (see Figure 3) and our estimates of the sRGB values [7]. Therefore we must design a system that allows us to know which printer was used to print the FM so that we can properly color calibrate the images. Our goal is to design the FM so that we can determine the printer by extracting the printer index from an image of the FM. Note that by “printer index” we mean a number that we can use to associate to a particular printer used to print the FM. We assume for this paper that we know the color calibration matrix for the printer. We are in the process of planning a large study whereby the number of simultaneous users of the mFR will be in the 100s. For this study we are designing a process where a user will be sent an FM as a digital file (e.g. a pdf file) with the indexing described in this paper that assigns that FM to the particular user. They will be asked to print the FM and send the printed FM back to us for extraction of the color calibration matrix. This process is beyond the scope of this paper. In this paper we are describing the FM indexing scheme and it relative robustness. One approach for printer indexing is to add a QR code containing the printer index in addition to the FM. However, this would require an additional step for the user to scan the QR code, thus increasing user burden [3]. Another approach we have used in the past for printer identification is based on texture features [5]. Unfortunately a texture-based printer identification technique will have issues with insufficient texture details in the FMs we use. We are interested in developing a method that will embed the index information in the fiducial marker without interfering with the color calibration and image analysis processes.

Fig. 3.

FM color differences using two printers.

2 Color Correction and Printer Indexing

2.1 Color Correction

Color is one of the key variables in imaging [7,10]. It is difficult to maintain color consistency due to illumination of the scene and camera settings such as auto exposure and auto white balance. Existing approaches attempt to increase the robustness of color descriptors based on features such as the RGB histogram, color moments and C-SIFT [6,9]. Our approach for color correction is a linear RGB mapping based on the von-Kries model [4]. Before we can calibrate the colors in an image acquired with our mFR one needs to calibrate the printer and determine its calibration matrix. This is done for a specific printer by printing the FM assigned to that printer and using a spectral radiometer to determine the sRGB estimates under CIE standard illuminant D65 [8]. A M × 3 color reference matrix, where M is the number of checkerboard colors, is constructed and denoted as C_ref. C_ref is constructed by assigning color sRGB estimates to the rows of C_ref based on the appearances of colors using a raster scan order, with the background color “white” last. The location associated with “red” square on the fiducial marker as indicated in Figure 3 will always be used as the starting point for raster scan order.

For color calibration, both the presence of the FM and the pixel coordinates of vertices for each color square in the FM need to be detected in the image we want to color correct. In our system, the detection is done on a gray scale version of the image [11]. Once the pixel coordinates of the vertices of the color squares are obtained, the sRGB values for each of the M colors in FMs are then estimated by examining the pixels in each color square. A color matrix for the image to be color corrected, denoted as C_test, is then constructed similarly to C_ref. Note that C_test is constructed using the lighting conditions in the scene and not the CIE standard illuminant D65. Color calibration is conducted using C_ref based on linear least squares [12]. The color correction matrix $D\in {\mathbb{R}}^{3\times 3}$ is:

$$\widehat{D}=\underset{D\in {\mathbb{R}}^{3\times 3}}{\arg \min }\sum _{j=1}^M{\Vert {\left({\mathit{C}}_{re{f}_j}\right)}^t-D{\left({\mathit{C}}_{tes{t}_j}\right)}^t\Vert }^2$$ (1)

where D̂ is the estimated color correction matrix, and C_refj, ${\mathit{C}}_{tes{t}_j}\in {\mathbb{R}}^{1\times 3}$ are the j^th color, j ∈ {1, · · · , M}. The image can be color corrected pixel by pixel as:

$${\mathit{C}}_{corrected}^t=\widehat{D}{\mathit{C}}_{original}^t$$ (2)

where C_original $\in {\mathbb{R}}^{1\times 3}$ is the original uncorrected sRGB values at any pixel location and C_corrected $\in {\mathbb{R}}^{1\times 3}$ is the color corrected result.

2.2 Printer Indexing System

We are interested in constructing an indexing system that allows us to identify from an image containing an FM which printer is used to print the FM. We designed the indexing system by associating each printer with a unique FM with different color squares arrangement. Rearranging the color squares on the checkerboard with no constraint yields a theoretical maximum of N_max permutations (or printers) where N_max = 10! = 3, 628, 000 and it is sufficient to address our needs. Denote i as the index for the i^th FM (or the i^th printer) and its corresponding color reference matrix is $C_{ref}^{\left(i\right)}\in {\mathbb{R}}^{M\times 3}$.

C_test is the color matrix estimated from an image to be color corrected, we shall refer to this image as the test image. Denote the lighting condition as I, the conditional probability that the FM with assigned index i is in the test image is defined as:

$$p(C_{test}\mid C_{ref}^{\left(i\right)},I)$$ (3)

We want to estimate the index $\widehat{i}$ based on (3) such that:

$$\widehat{i}=\underset{i\in \{1,\dots ,N\}}{\arg \max }\left\{p(C_{test}\mid C_{ref}^{\left(i\right)},I)\right\}$$ (4)

We will find $\widehat{i}$ using normalized cross correlation (NCC).

Normalized cross correlation is a method for template or image matching [2,13]. Our experimental results indicate that NCC can minimize the impact of the external lighting condition when estimating the printer index. The printer index is estimated using the NCC score between C_test and $C_{ref}^{\left(i\right)}$ where the NCC score, $f(C_{test},C_{ref}^{\left(i\right)})$, is defined as:

$$f(C_{test},C_{ref}^{\left(i\right)})=\frac{1}{3\cdot M-1}\frac{{\sum }_{k=1}^{3\cdot M}\left\{({\mathit{C}}_{test}\left(k\right)-{\mu }_{{\mathit{C}}_{test}})({\mathit{C}}_{ref}^{\left(i\right)}\left(k\right)-{\mu }_{{\mathit{C}}_{ref}^{\left(i\right)}})\right\}}{{\sigma }_{{\mathit{C}}_{test}}{\sigma }_{{\mathit{C}}_{ref}^{\left(i\right)}}}$$ (5)

where C_test and $C_{ref}^{\left(i\right)}$ are vectorized as C_test, ${\mathit{C}}_{ref}^{\left(i\right)}\in {\mathbb{R}}^{1\times 3\cdot M}$, M = 11, μ and σ² are mean and sample standard deviation, respectively. Based on the above definition, we have the estimated index as:

$$\widehat{i}=\underset{i\in \{1,\dots ,N\}}{\arg \max }\left\{f(C_{test},C_{ref}^{\left(i\right)})\right\}$$ (6)

After we obtain the estimate $\widehat{i}$, we use the reference information associated with this specific FM for color calibration as described in Section 2.1.

2.3 Error Control Using Binarized Marks

From our experiments, we observed that similar FM colors (such as red, orange and brown) may be very difficult to differentiate under certain lighting conditions (e.g. a dim restaurant) or due to poor printing quality. For these similar colors that are likely to cause incorrect index decisions using NCC, we define “similar colors” sets. For example, red, orange and brown can form a “similar colors” set. If similar colors can not be differentiated, the assumption that each FM has a unique arrangement of color squares can no longer hold true. To address this issue, we propose the use of a “binarized mark” that we can add to the FM to serve as an error control method in addition to NCC. Binarized marks are combinations of small black squares placed at the center of one or more white squares as illustrated in Figure 2.

Fig. 2.

Magnified FM in Figure 1(c).

A numeric value can be generated from the binarized mark based on the detection of the black square. Following raster scan order begins at the second white square in the FM, each subsequent white square represents a “bit” in the binary sequence starting from the least significant bit. A bit is assigned a “1” when a black square is present and “0” otherwise. The corresponding numeric value in decimal can be obtained by converting the binary sequence. For the FM shown in Figure 2, the binary sequence is “000000011” and accordingly the numeric value in decimal is “3”. Since the length of the binary sequence is 9, only 2⁹ = 512 binarized marks can be generated. However, the theoretical maximum number of printers we can index is N_max = 3, 628, 800. Since we “assign” a binarized mark to every FM, we will quickly run out of binarized marks without having identical binarized marks assigned to each FM. Since we cannot assign a unique binarized mark to each FM, we need to define a criteria for assigning binarized marks to the FMs.

We define a threshold T to activate the error control. For a given printer index $\widehat{i}$ obtained from (6), if there is no other i ∈ {1, ···, N} and $i\ne \widehat{i}$ such that the NCC score defined in (5) satisfies the following:

$$f(C_{test},C_{ref}^{\left(\widehat{i}\right)})-f(C_{test},C_{ref}^{\left(i\right)})<T$$ (7)

we can safely assume that using the NCC score is sufficient for indexing the printer, hence no binarized marks are needed in this case. We set T = 0.01 based on our experimental results. Otherwise, error control method is activated and printer index can be corrected based on the detection of the binarized mark. As a result, we only need to assign a unique binarized mark to each FM that meets the criteria of (7). Based on our experimental results, we have observed that the number of FMs with “similar colors” swapped generally do not exceed the maximum number of binarized marks that can be generated, which is 512. For example, in the case where similar FM colors such as red, orange and brown are swapped, we only need 3! = 6 binarized marks to guarantee the correct printer indexing.

3 Experimental Results

The initial evaluation of our printer indexing system is based on FMs we printed using several printers. Our test images contain various foods images with different FMs taken under several lighting conditions. There is no other information in the test images that can indicate which FM is used in a specific test image. After extracting the color information from the FMs and the binarized marks, we estimated the index i using the methods described in Section 2.2. The indexing is estimated primarily using the NCC score. Binarzied marks will be used only when error control is activated as described in (7). The ground truth is obtained by a human observer examining the arrangement of color squares. The accuracy of indexing decisions can then be obtained by comparing to the ground truth.

We are interested in testing the NCC-based method for a variety of FMs. To conduct such a test we generated 9 FMs, where 1 had no color swapped (original FM with the following colors in raster scan order: red, green, blue, black, brown, cyan, magenta, yellow, dark green and orange), 7 had two colors swapped compared to the original one (red and brown, green and dark green, green and yellow, red and magenta, yellow and orange, blue and cyan, red and orange), and 1 had three colors swapped compared to the original one (green, yellow and dark green). Figure 1(b) and (c) show examples of images with FMs that have color swapped and binarized marks.

We have obtained 579 test images with different lighting conditions using 9 models of laser and inkjet printers. These lighting conditions include incandescent and fluorescent lightings with various luminance, sunlight, shadows and more complex lighting conditions in the restaurants. The index decisions can be made accurately as reflected by the average NCC scores illustrated in Figure 4. The average NCC scores for each FM are obtained from test images containing the same FM. Note that printers with “*” in Figure 4 are laser printers.

Fig. 4.

Accuracy of estimated printer index based on average NCC scores from 9 printers, with each printer associated with a unique FM as listed below: (A) Canon i9900 (original FM), (B) Canon PIXMA Pro-100 (red and brown swapped), (C*) HP LaserJet M551 (green and dark green swapped), (D) Canon PIXMA Pro-10 (green and yellow swapped), (E*) HP Color LaserJet 4700 (red and magenta swapped), (F*) Ricoh Aficio MP C6501 (yellow and orange swapped), (G*) TOSHIBA e-STUDIO 3530c (blue and cyan swapped), (H) Epson WF3540 (green, yellow and dark green swapped), (I) HP D110a (red and orange swapped)

We also test the accuracy of printer indexing based on NCC scores for each test image from a subset of the 579 test images used above. This subset of test images contains 40 images with the same FM printed from a HP LaserJet M551 printer. For this particular example, color squares green and dark green are swapped compared to the original FM (no color swapped). A wrong indexing decision is made based on NCC scores alone as shown in the zoomed in red rectangular area of Figure 5. The printer is indexed to HP Color LaserJet 4700 instead of HP LaserJet M551 without activating error control for one of the test images. The test image that generates the wrong indexing decision is shown in Figure 1(c), with a zoomed in image of the fiducial marker shown in Figure 2. However, the criteria for activating the error control method as defined in (7) is satisfied for this test image. Therefore, by detecting the binarized marks, the correct indexing decision can be made.

Fig. 5.

A test set of 40 images from 9 printers. The two lines inside the red rectangular area show an example where a wrong printer indexing decision is made based on NCC scores alone.

4 Conclusion

In this paper we have described a printer indexing system for use in color image correction. We have shown that the printer index can be accurately estimated. We are currently investigating how to use this approach in user studies that may have hundreds of printers by constructing a database of color calibration information.