Why use a Binning Analysis?
As described in Markus Wallerberger (2019), Vinay Ambegaokar, Matthias Troyer (2010), Tony F. Chan, Gene H. Golub, Randall J. LeVeque (1983), Carsten Bauer (2020), and James Gubernatis, Naoki Kawashima, Philipp Werner (2016), the presence of correlations in a data stream generated in Markov Chain Monte Carlo simulations renders any measures of error severely underestimated.
Theoretical background
The naive approach is to calculate the mean and the variance of the mean, var_of_mean, for a given data stream $X$. For uncorrelated data, this becomes
\[\begin{aligned} \mathtt{mean}(X) &= \frac{1}{M} \sum_{i = 1}^M x_i \\ \mathtt{var\_of\_mean}(X) &= \frac{1}{M(M-1)} \sum_{i = 1}^M (x_i - \mathtt{mean}(X))^2. \end{aligned}\]
Fortunately, in the presence of correlations, the mean doesn't change. Unfortunately, the var_of_mean does. Indeed, one can show that it's given by
\[\mathtt{var\_of\_mean}(X) = \frac{1 + 2\tau_X}{M(M-1)} \sum_{i = 1}^M (x_i - \mathtt{mean}(X))^2,\]
where the as shown in the Figure within the Accumulator type hierarchy. Clearly, the uncorrelated var_of_mean is just increased by a factor of $R_X \equiv 1 + 2\tau_X$, where $\tau_X$ is defined as the integrated autocorrelation_time of the data stream. The binning analysis provides a fast as an O(N) method for calculating the autocorrelation_time. It's also cheap, only requiring O(log N) in RAM. For those who are unfamiliar with the term, normally one would calculate $\tau_X$ by
\[\tau_X = \sum_{i = 1}^M \sum_{i < j} \left[ x_ix_j - \mathtt{mean}(x)^2 \right],\]
which is an O(N^2) summation and can suffer from numerical instabilities.
As one performs pairwise means in each binning level to construct the next highest, and then calculates the var_of_mean for each level, one can see that it rises and then eventually saturates around a particular plateau value. Normalizing by the original var_of_mean calculation assuming the data stream is uncorrelated, one can then calculate $R_X$ in the $\ell^{\rm th}$ binning level as
\[R_X(\ell) = \frac{\mathtt{var\_of\_mean}(X^{(\ell)})}{\mathtt{var\_of\_mean}(X^{(0)})} = \frac{ m^{(\ell)} \mathtt{var}(X^{(\ell)}) }{ \mathtt{var}(X^{(0)}) },\]
where var is the normal variance and $m^{(\ell)}$ is the bin size, or the number of original data values accumulated into a single data point at the $\ell^{\rm th}$ level.
Visualizing a Binning Analysis
The final result of the binning analysis, for sufficiently long data streams, will be to see a sigmoid-like curve. We've wrapped up a fitting and plotting workflow into the plot_binning_analysis function below to demonstrate how it works. Feel free to skip over to the Visualization Examples if you have your own workflow established.
using OnlineLogBinning, Plots, LaTeXStrings
pyplot()
# Return the fitted inflection point
sigmoid_inflection(fit) = fit.param[2] / fit.param[3]
# Define a plotting function with Plots.jl
# This function plots the RxValues, the Sigmoid fit,
# the maximum trustworthy level, and the fitted
# inflection point.
function plot_binning_analysis(bacc)
# Plot the RxValues irrespective of the trusting cutoff
all_levels, all_rxvals = levels_RxValues(bacc, false)
plt = plot( all_levels, all_rxvals;
label = "Binning Analysis",
xlabel = L"Bin Level $\ell$",
ylabel = L"$R_X(\ell)$",
markershape = :circle,
legend = :topleft )
# Fit only the trustworthy levels
levels, rxvals = levels_RxValues(bacc)
fit = fit_RxValues(levels, rxvals)
# Plot the sigmoid fit
plot!(plt, all_levels, sigmoid(all_levels, fit.param);
label = "Sigmoid Fit")
# Plot the maximum trustworthy level
vline!(plt, [ max_trustworthy_level(bacc[level = 0].num_bins) ];
ls = :dash, color = "gray",
label = "Maximum Trustworthy Level")
# Plot the fitted inflection point if it's positive
if sigmoid_inflection(fit) > zero(fit.param[2])
vline!(plt, [ sigmoid_inflection(fit) ];
ls = :dot, color = "red",
label = "Sigmoid Inflection Point")
end
return plt
end[ Info: Installing matplotlib via the Conda matplotlib package...
[ Info: Running `conda install -q -y matplotlib` in root environment
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: /home/runner/.julia/conda/3
added / updated specs:
- matplotlib
The following packages will be downloaded:
package | build
---------------------------|-----------------
alsa-lib-1.2.6.1 | h7f98852_0 578 KB conda-forge
attr-2.5.1 | h166bdaf_1 69 KB conda-forge
brotli-1.0.9 | h166bdaf_7 18 KB conda-forge
brotli-bin-1.0.9 | h166bdaf_7 19 KB conda-forge
cycler-0.11.0 | pyhd8ed1ab_0 10 KB conda-forge
dbus-1.13.6 | h5008d03_3 604 KB conda-forge
expat-2.4.8 | h27087fc_0 187 KB conda-forge
fftw-3.3.10 |nompi_ha7695d1_103 6.5 MB conda-forge
font-ttf-dejavu-sans-mono-2.37| hab24e00_0 388 KB conda-forge
font-ttf-inconsolata-3.000 | h77eed37_0 94 KB conda-forge
font-ttf-source-code-pro-2.038| h77eed37_0 684 KB conda-forge
font-ttf-ubuntu-0.83 | hab24e00_0 1.9 MB conda-forge
fontconfig-2.14.0 | h8e229c2_0 305 KB conda-forge
fonts-conda-ecosystem-1 | 0 4 KB conda-forge
fonts-conda-forge-1 | 0 4 KB conda-forge
fonttools-4.35.0 | py39hb9d737c_0 1.8 MB conda-forge
freetype-2.12.1 | hca18f0e_0 884 KB conda-forge
gettext-0.19.8.1 | h73d1719_1008 3.6 MB conda-forge
glib-2.72.1 | h6239696_0 443 KB conda-forge
glib-tools-2.72.1 | h6239696_0 107 KB conda-forge
gst-plugins-base-1.20.3 | hf6a322e_0 2.8 MB conda-forge
gstreamer-1.20.3 | hd4edc92_0 2.0 MB conda-forge
icu-70.1 | h27087fc_0 13.5 MB conda-forge
jack-1.9.18 | h8c3723f_1002 643 KB conda-forge
jpeg-9e | h166bdaf_2 269 KB conda-forge
keyutils-1.6.1 | h166bdaf_0 115 KB conda-forge
kiwisolver-1.4.4 | py39hf939315_0 76 KB conda-forge
krb5-1.19.3 | h3790be6_0 1.4 MB conda-forge
lcms2-2.12 | hddcbb42_0 443 KB conda-forge
lerc-4.0.0 | h27087fc_0 275 KB conda-forge
libbrotlicommon-1.0.9 | h166bdaf_7 65 KB conda-forge
libbrotlidec-1.0.9 | h166bdaf_7 33 KB conda-forge
libbrotlienc-1.0.9 | h166bdaf_7 287 KB conda-forge
libcap-2.64 | ha37c62d_0 96 KB conda-forge
libclang-14.0.6 |default_h2e3cab8_0 127 KB conda-forge
libclang13-14.0.6 |default_h3a83d3e_0 10.6 MB conda-forge
libcups-2.3.3 | h3e49a29_2 4.5 MB conda-forge
libdb-6.2.32 | h9c3ff4c_0 23.3 MB conda-forge
libdeflate-1.13 | h166bdaf_0 79 KB conda-forge
libedit-3.1.20191231 | he28a2e2_2 121 KB conda-forge
libevent-2.1.10 | h9b69904_4 1.1 MB conda-forge
libflac-1.3.4 | h27087fc_0 474 KB conda-forge
libglib-2.72.1 | h2d90d5f_0 3.1 MB conda-forge
libiconv-1.16 | h516909a_0 1.4 MB conda-forge
libllvm14-14.0.6 | he0ac6c6_0 35.2 MB conda-forge
libogg-1.3.4 | h7f98852_1 206 KB conda-forge
libopus-1.3.1 | h7f98852_1 255 KB conda-forge
libpng-1.6.37 | h753d276_4 371 KB conda-forge
libpq-14.5 | hd77ab85_0 3.0 MB conda-forge
libsndfile-1.0.31 | h9c3ff4c_1 602 KB conda-forge
libtiff-4.4.0 | h0e0dad5_3 642 KB conda-forge
libtool-2.4.6 | h9c3ff4c_1008 511 KB conda-forge
libudev1-249 | h166bdaf_4 109 KB conda-forge
libvorbis-1.3.7 | h9c3ff4c_0 280 KB conda-forge
libwebp-base-1.2.4 | h166bdaf_0 404 KB conda-forge
libxcb-1.13 | h7f98852_1004 391 KB conda-forge
libxkbcommon-1.0.3 | he3ba5ed_0 581 KB conda-forge
libxml2-2.9.14 | h22db469_4 771 KB conda-forge
matplotlib-3.5.3 | py39hf3d152e_1 7 KB conda-forge
matplotlib-base-3.5.3 | py39h700656a_1 7.4 MB conda-forge
munkres-1.1.4 | pyh9f0ad1d_0 12 KB conda-forge
mysql-common-8.0.30 | haf5c9bc_0 1.9 MB conda-forge
mysql-libs-8.0.30 | h28c427c_0 1.9 MB conda-forge
nspr-4.32 | h9c3ff4c_1 233 KB conda-forge
nss-3.78 | h2350873_0 2.1 MB conda-forge
openjpeg-2.5.0 | h7d73246_1 533 KB conda-forge
packaging-21.3 | pyhd8ed1ab_0 36 KB conda-forge
pcre-8.45 | h9c3ff4c_0 253 KB conda-forge
pillow-9.2.0 | py39hd5dbb17_2 44.9 MB conda-forge
ply-3.11 | py_1 44 KB conda-forge
portaudio-19.6.0 | h57a0ea0_5 131 KB conda-forge
pthread-stubs-0.4 | h36c2ea0_1001 5 KB conda-forge
pulseaudio-14.0 | h7f54b18_8 1.7 MB conda-forge
pyparsing-3.0.9 | pyhd8ed1ab_0 79 KB conda-forge
pyqt-5.15.7 | py39h18e9c17_0 6.2 MB conda-forge
pyqt5-sip-12.11.0 | py39h5a03fae_0 86 KB conda-forge
python-dateutil-2.8.2 | pyhd8ed1ab_0 240 KB conda-forge
qt-main-5.15.4 | ha5833f6_2 61.5 MB conda-forge
sip-6.6.2 | py39h5a03fae_0 495 KB conda-forge
toml-0.10.2 | pyhd8ed1ab_0 18 KB conda-forge
tornado-6.2 | py39hb9d737c_0 658 KB conda-forge
unicodedata2-14.0.0 | py39hb9d737c_1 498 KB conda-forge
xcb-util-0.4.0 | h166bdaf_0 20 KB conda-forge
xcb-util-image-0.4.0 | h166bdaf_0 24 KB conda-forge
xcb-util-keysyms-0.4.0 | h166bdaf_0 12 KB conda-forge
xcb-util-renderutil-0.3.9 | h166bdaf_0 15 KB conda-forge
xcb-util-wm-0.4.1 | h166bdaf_0 55 KB conda-forge
xorg-libxau-1.0.9 | h7f98852_0 13 KB conda-forge
xorg-libxdmcp-1.1.3 | h7f98852_0 19 KB conda-forge
zstd-1.5.2 | h6239696_4 448 KB conda-forge
------------------------------------------------------------
Total: 259.5 MB
The following NEW packages will be INSTALLED:
alsa-lib conda-forge/linux-64::alsa-lib-1.2.6.1-h7f98852_0
attr conda-forge/linux-64::attr-2.5.1-h166bdaf_1
brotli conda-forge/linux-64::brotli-1.0.9-h166bdaf_7
brotli-bin conda-forge/linux-64::brotli-bin-1.0.9-h166bdaf_7
cycler conda-forge/noarch::cycler-0.11.0-pyhd8ed1ab_0
dbus conda-forge/linux-64::dbus-1.13.6-h5008d03_3
expat conda-forge/linux-64::expat-2.4.8-h27087fc_0
fftw conda-forge/linux-64::fftw-3.3.10-nompi_ha7695d1_103
font-ttf-dejavu-s~ conda-forge/noarch::font-ttf-dejavu-sans-mono-2.37-hab24e00_0
font-ttf-inconsol~ conda-forge/noarch::font-ttf-inconsolata-3.000-h77eed37_0
font-ttf-source-c~ conda-forge/noarch::font-ttf-source-code-pro-2.038-h77eed37_0
font-ttf-ubuntu conda-forge/noarch::font-ttf-ubuntu-0.83-hab24e00_0
fontconfig conda-forge/linux-64::fontconfig-2.14.0-h8e229c2_0
fonts-conda-ecosy~ conda-forge/noarch::fonts-conda-ecosystem-1-0
fonts-conda-forge conda-forge/noarch::fonts-conda-forge-1-0
fonttools conda-forge/linux-64::fonttools-4.35.0-py39hb9d737c_0
freetype conda-forge/linux-64::freetype-2.12.1-hca18f0e_0
gettext conda-forge/linux-64::gettext-0.19.8.1-h73d1719_1008
glib conda-forge/linux-64::glib-2.72.1-h6239696_0
glib-tools conda-forge/linux-64::glib-tools-2.72.1-h6239696_0
gst-plugins-base conda-forge/linux-64::gst-plugins-base-1.20.3-hf6a322e_0
gstreamer conda-forge/linux-64::gstreamer-1.20.3-hd4edc92_0
icu conda-forge/linux-64::icu-70.1-h27087fc_0
jack conda-forge/linux-64::jack-1.9.18-h8c3723f_1002
jpeg conda-forge/linux-64::jpeg-9e-h166bdaf_2
keyutils conda-forge/linux-64::keyutils-1.6.1-h166bdaf_0
kiwisolver conda-forge/linux-64::kiwisolver-1.4.4-py39hf939315_0
krb5 conda-forge/linux-64::krb5-1.19.3-h3790be6_0
lcms2 conda-forge/linux-64::lcms2-2.12-hddcbb42_0
lerc conda-forge/linux-64::lerc-4.0.0-h27087fc_0
libbrotlicommon conda-forge/linux-64::libbrotlicommon-1.0.9-h166bdaf_7
libbrotlidec conda-forge/linux-64::libbrotlidec-1.0.9-h166bdaf_7
libbrotlienc conda-forge/linux-64::libbrotlienc-1.0.9-h166bdaf_7
libcap conda-forge/linux-64::libcap-2.64-ha37c62d_0
libclang conda-forge/linux-64::libclang-14.0.6-default_h2e3cab8_0
libclang13 conda-forge/linux-64::libclang13-14.0.6-default_h3a83d3e_0
libcups conda-forge/linux-64::libcups-2.3.3-h3e49a29_2
libdb conda-forge/linux-64::libdb-6.2.32-h9c3ff4c_0
libdeflate conda-forge/linux-64::libdeflate-1.13-h166bdaf_0
libedit conda-forge/linux-64::libedit-3.1.20191231-he28a2e2_2
libevent conda-forge/linux-64::libevent-2.1.10-h9b69904_4
libflac conda-forge/linux-64::libflac-1.3.4-h27087fc_0
libglib conda-forge/linux-64::libglib-2.72.1-h2d90d5f_0
libiconv conda-forge/linux-64::libiconv-1.16-h516909a_0
libllvm14 conda-forge/linux-64::libllvm14-14.0.6-he0ac6c6_0
libogg conda-forge/linux-64::libogg-1.3.4-h7f98852_1
libopus conda-forge/linux-64::libopus-1.3.1-h7f98852_1
libpng conda-forge/linux-64::libpng-1.6.37-h753d276_4
libpq conda-forge/linux-64::libpq-14.5-hd77ab85_0
libsndfile conda-forge/linux-64::libsndfile-1.0.31-h9c3ff4c_1
libtiff conda-forge/linux-64::libtiff-4.4.0-h0e0dad5_3
libtool conda-forge/linux-64::libtool-2.4.6-h9c3ff4c_1008
libudev1 conda-forge/linux-64::libudev1-249-h166bdaf_4
libvorbis conda-forge/linux-64::libvorbis-1.3.7-h9c3ff4c_0
libwebp-base conda-forge/linux-64::libwebp-base-1.2.4-h166bdaf_0
libxcb conda-forge/linux-64::libxcb-1.13-h7f98852_1004
libxkbcommon conda-forge/linux-64::libxkbcommon-1.0.3-he3ba5ed_0
libxml2 conda-forge/linux-64::libxml2-2.9.14-h22db469_4
matplotlib conda-forge/linux-64::matplotlib-3.5.3-py39hf3d152e_1
matplotlib-base conda-forge/linux-64::matplotlib-base-3.5.3-py39h700656a_1
munkres conda-forge/noarch::munkres-1.1.4-pyh9f0ad1d_0
mysql-common conda-forge/linux-64::mysql-common-8.0.30-haf5c9bc_0
mysql-libs conda-forge/linux-64::mysql-libs-8.0.30-h28c427c_0
nspr conda-forge/linux-64::nspr-4.32-h9c3ff4c_1
nss conda-forge/linux-64::nss-3.78-h2350873_0
openjpeg conda-forge/linux-64::openjpeg-2.5.0-h7d73246_1
packaging conda-forge/noarch::packaging-21.3-pyhd8ed1ab_0
pcre conda-forge/linux-64::pcre-8.45-h9c3ff4c_0
pillow conda-forge/linux-64::pillow-9.2.0-py39hd5dbb17_2
ply conda-forge/noarch::ply-3.11-py_1
portaudio conda-forge/linux-64::portaudio-19.6.0-h57a0ea0_5
pthread-stubs conda-forge/linux-64::pthread-stubs-0.4-h36c2ea0_1001
pulseaudio conda-forge/linux-64::pulseaudio-14.0-h7f54b18_8
pyparsing conda-forge/noarch::pyparsing-3.0.9-pyhd8ed1ab_0
pyqt conda-forge/linux-64::pyqt-5.15.7-py39h18e9c17_0
pyqt5-sip conda-forge/linux-64::pyqt5-sip-12.11.0-py39h5a03fae_0
python-dateutil conda-forge/noarch::python-dateutil-2.8.2-pyhd8ed1ab_0
qt-main conda-forge/linux-64::qt-main-5.15.4-ha5833f6_2
sip conda-forge/linux-64::sip-6.6.2-py39h5a03fae_0
toml conda-forge/noarch::toml-0.10.2-pyhd8ed1ab_0
tornado conda-forge/linux-64::tornado-6.2-py39hb9d737c_0
unicodedata2 conda-forge/linux-64::unicodedata2-14.0.0-py39hb9d737c_1
xcb-util conda-forge/linux-64::xcb-util-0.4.0-h166bdaf_0
xcb-util-image conda-forge/linux-64::xcb-util-image-0.4.0-h166bdaf_0
xcb-util-keysyms conda-forge/linux-64::xcb-util-keysyms-0.4.0-h166bdaf_0
xcb-util-renderut~ conda-forge/linux-64::xcb-util-renderutil-0.3.9-h166bdaf_0
xcb-util-wm conda-forge/linux-64::xcb-util-wm-0.4.1-h166bdaf_0
xorg-libxau conda-forge/linux-64::xorg-libxau-1.0.9-h7f98852_0
xorg-libxdmcp conda-forge/linux-64::xorg-libxdmcp-1.1.3-h7f98852_0
zstd conda-forge/linux-64::zstd-1.5.2-h6239696_4
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... doneVisualization Examples
Now we demonstrate four cases that tend to appear with binning analyses:
- The binning analysis converges to a plateau for a correlated data stream $R_X(\ell \rightarrow \infty) > 0$.
- The binning analysis reveals that the data stream is truly uncorrelated $R_X(\ell \rightarrow \infty) = 0$.
- The binning analysis has not converged because too few data were taken in the data stream to isolate the uncorrelated data.
- The binning analyis has not convered because all of the data in the data stream are more-or-less equally correlated.
The first two cases are ideal. They represent situations where the data stream has enough data to distinguish correlated blocks from one another. In these first two cases, the BinningAnalysisResult will yield plateau_found == true.
The second two cases are not so ideal. Either way they show that the correlations in the data stream are so strong that individual uncorrelated blocks can not be robustly created and more sampling is required. Therefore, their BinningAnalysisResult yields plateau_found == false.
The data shown are pre-simulated signals generated by the TelegraphNoise.jl package (v0.1.0). Random telegraph signals have an analytically-defined autocorrelation time $\tau_X$ related to their average dwell time $T_D$ by $\tau_X = T_D / 2$. Since the signals are random, they are saved previously, but I'll provide each chosen $T_D$ for reproducibility purposes.
Example 1: Clear $R_X$ plateau at a finite value
This first case shows the textbook (James Gubernatis, Naoki Kawashima, Philipp Werner (2016)) example of an $R_X$ plateau found in a binning analysis. The dwell time for this signal was chosen to be $T_D = 16$, meaning $\tau_X = 8$, and the signal generated was $2^{18} = 262144$ in length.
# Read in pre-simulated TelegraphNoise.jl data with a plateau
signal = zeros(Float64, Int(2^18))
read!( joinpath("assets", "telegraph_plateau.bin"), signal )
bacc = BinningAccumulator()
push!(bacc, signal)
plot_binning_analysis(bacc)
result = fit_RxValues(bacc)
result.plateau_found
# output
trueImportantly, note that the fitted inflection point is greater than zero, but less than the maximum trustworthy level. The variations in the calculated $R_X$ values for $\ell$ greater than the maximum trustworthy level are due to strong fluctuations because of small number statistics.
Example 2: Uncorrelated data ~ $R_X(\ell \rightarrow \infty) = 0$
The dwell time for this signal was chosen to be $T_D = 1/2$, meaning $\tau_X = 1/4$, and the signal generated was $2^{14} = 16384$ in length.
# Read in pre-simulated TelegraphNoise.jl data with a plateau
signal = zeros(Float64, Int(2^14))
read!( joinpath("assets", "telegraph_uncorrelated.bin"), signal )
bacc = BinningAccumulator()
push!(bacc, signal)
plot_binning_analysis(bacc)
result = fit_RxValues(bacc)
result.plateau_found
# output
trueNotice that the $R_X$ values decay for increasing $\ell$. This is because the binning analysis cannot detect any smaller blocks of uncorrelated data since the original data stream is indeed correlated.
Example 3: No plateau due to data insufficiency
For insufficiently long data streams, we do not expect a plateau, as shown in the following case:
# Read in pre-simulated TelegraphNoise.jl data without a plateau
signal = zeros(Float64, Int(2^10))
read!( joinpath("assets", "telegraph_no_plateau.bin"), signal )
bacc = BinningAccumulator()
push!(bacc, signal)
plot_binning_analysis(bacc)
result = fit_RxValues(bacc)
result.plateau_found
# output
falseHere, the dwell time was chosen to be $T_D = 256$, meaning $\tau_X = 128$, and the signal generated was $2^{10} = 1024$ in length. Notice, that the binning analysis began to pick up on a set of uncorrelated blocks, but there was not enough simulated data to reveal them before the maximum trustworthy level was reached. In a case like this, one would simply need to run their simulation about 64 times longer reveal a fitted plateau.
Example 4: No plateau due to totally-correlated data
In this final example, we demonstrate what happens for a data stream that has "frozen-in" correlations. By this, we mean one where the autocorrelation time $\tau_X$ far exceeds the data stream size. For this case, $T_D = 2^{16} = 65536$, so $\tau_X = 2^{15} = 32768$, while the signal length is only $2^{12} = 4096$.
# Read in pre-simulated TelegraphNoise.jl data with a plateau
signal = zeros(Float64, Int(2^12))
read!( joinpath("assets", "telegraph_totally_correlated.bin"), signal )
bacc = BinningAccumulator()
push!(bacc, signal)
plot_binning_analysis(bacc)
result = fit_RxValues(bacc)
result.plateau_found
# output
falseIn this case, eventually the binning analysis returns $R_X = 0$, since the blocks start comparing literally identical elements. Notice that this case would return a plateau_found if it were not for the check if any of the $R_X$ values were too small. (In principle this case also would result from a data stream with a defined periodicity, despite having correlations, but this is unavoidable.)
This is the most dangerous case to automate because the variance of such a data stream is naturally very small relative to the mean. As is the case with all Monte Carlo simulations, or statistical data streams, one should be very wary when the error is a mathematical zero, or vanishingly small compared to the calculated mean.