Table 4. Profile fitting of 1 × 105 simulated spots.
The stated precisions are given by the standard errors of the estimators.
| Zone A | Zone B | |||||
|---|---|---|---|---|---|---|
| Statistics over 1 × 105 samples | Refer to equation | Raw | Corrected | Raw | Corrected | |
| Profile-fitted intensity distribution | mean(I P) (ADU) | Equation (15) | 8181.8 (4) | 8240.3 (4) | 8217.7 (4) | 8209.8 (4) |
| var(I P) (ADU2) | Equation (15) | 13640 (60) | 13770 (60) | 13520 (60) | 13420 (60) | |
| Variance estimates | mean( ) (ADU2) |
Equation (39) | 8137 (2) | 3598 (1) | 8170 (2) | 3898 (1) |
|
with M
f
† (ADU2) |
Equation (18) | 13669 | 13305 | 13530 | 12954 | |
†
In this case, the variance–covariance matrix of observations, M f, was pre-calculated from the set of 1 × 105 samples, rather than estimated individually for each.