OnlineLogBinning API Reference

const MINIMUM_RX_64 = eps(Float64)
const MINIMUM_RX_32 = eps(Float32) 
const MINIMUM_RX_16 = eps(Float16)

Minimum allowable variance values based on the least-squares fit type. Any data stream variances smaller than these are suspiciously small, and one should not trust an automated binning analysis in these instances.

OLB_tested_numbers

Defines the list of tested numerical types for OnlineLogBinning.jl.

TRUSTING_CUTOFF = 128

The minimum number of bins required for any given binning level to be considered statistically trustworthy so as to minimally suffer from small-number fluctuations.

BinningAccumulator{T}() where {T <: Number} (default T = Float64)
BinningAccumulator{T}(::Vector{LevelAccumulator{T}})
BinningAccumulator{T}(::Int) where {T <: Number} (default T = Float64)

Main data structure for the binning analysis. T == Float64 by default in the empty constructor. There are three constructors, an empty one, one that copy-constructs with a Vector of LevelAccumulators, and one that pre-allocates that Vector based on an anticipated datastream size.

Contents

• LvlAccums::Vector{LevelAccumulator{T}}
  • A wrapper around the LevelAccumulators from each binning level

Example

julia> # Create a BinningAccumulator with the default type T == Float64

julia> bacc = BinningAccumulator()  
BinningAccumulator{Float64} with 0 binning levels.
0th Binning Level (unbinned data):
LevelAccumulator{Float64} with online fields:
    level    = 0
    num_bins = 0
    Taccum   = 0.0
    Saccum   = 0.0
    Paccum   = PairAccumulator{Float64}(true, [0.0, 0.0])

    Calculated Level Statistics:
    Current Mean             = NaN
    Current Variance         = -0.0
    Current Std. Deviation   = -0.0
    Current Var. of the Mean = NaN
    Current Std. Error       = NaN

julia> # Add a data stream using the push! function

julia> # (The data stream does not have to have a length == power of 2.)

julia> push!(bacc, [1, 2, 3, 4])
BinningAccumulator{Float64} with 2 binning levels.
0th Binning Level (unbinned data):
LevelAccumulator{Float64} with online fields:
    level    = 0
    num_bins = 4
    Taccum   = 10.0
    Saccum   = 5.0
    Paccum   = PairAccumulator{Float64}(true, [0.0, 0.0])

    Calculated Level Statistics:
    Current Mean             = 2.5
    Current Variance         = 1.6666666666666667
    Current Std. Deviation   = 1.2909944487358056
    Current Var. of the Mean = 0.4166666666666667
    Current Std. Error       = 0.6454972243679028

1th Binning Level:
LevelAccumulator{Float64} with online fields:
    level    = 1
    num_bins = 2
    Taccum   = 5.0
    Saccum   = 2.0
    Paccum   = PairAccumulator{Float64}(true, [0.0, 0.0])

    Calculated Level Statistics:
    Current Mean             = 2.5
    Current Variance         = 2.0
    Current Std. Deviation   = 1.4142135623730951
    Current Var. of the Mean = 1.0
    Current Std. Error       = 1.0

2th Binning Level:
LevelAccumulator{Float64} with online fields:
    level    = 2
    num_bins = 0
    Taccum   = 0.0
    Saccum   = 0.0
    Paccum   = PairAccumulator{Float64}(false, [0.0, 2.5])

    Calculated Level Statistics:
    Current Mean             = NaN
    Current Variance         = -0.0
    Current Std. Deviation   = -0.0
    Current Var. of the Mean = NaN
    Current Std. Error       = NaN

BinningAnalysisResult{T <: AbstractFloat}

Small struct to determine if there is a _plateau_found from a BinningAccumulator, and what its value is.

Contents

• plateau_found::Bool: whether the fit_RxValues found a plateau from the binned data.
• RxAmplitude::T: the value for the plateau as calculated by fit_RxValues.
  • If plateau_found == false, then RxAmplitude = length(X) for a datastream X, so as to maximize the error estimation.
• effective_length::Int: the effective number of uncorrelated data points in the datastream X as calculated by

\[m_{\rm eff} = \mathtt{floor} \left( \frac{\mathtt{length}(X)}{R_X} \right).\]

• binning_mean::T: the value of the mean as calculated by

\[\mathtt{mean}(X) = \frac{ T^{(0)} }{ m^{(0)} }.\]

• binning_error::T: the value of the error as calculated by

\[\begin{aligned} \mathtt{error}(X) &= \sqrt{ \frac{ S^{(0)} }{ m_{\rm eff} \left( m^{(0)} - 1 \right) } } \\ &= \sqrt{ \left[ \mathtt{floor}\left( \frac{m^{(0)}}{R_X} \right) \right]^{-1} \, \frac{ S^{(0)} }{ m^{(0)} - 1 } }. \end{aligned}\]

LevelAccumulator{T <: Number}

Accumulator structure for a given binning level.

Contents

• level::Int
  • Registers the binning level this accumulator is assigned
• num_bins::Int
  • How many elements (i.e. bins) have been added to this accumulator
• Taccum::T
  • Stands for Total Accumulator.
  • This represents the T accumulator for the mean: mean ≡ T / num_bins.
• Saccum::T
  • Stands for Square Accumulator.
  • This represents the S accumulator for the variance: var ≡ S/(num_bins - 1).
• Paccum::PairAccumulator{T}
  • An outward facing PairAccumulator to meet incoming data streams.
  • This accumulator processes the incoming data and then exports the Tvalue and Svalue into updates for Taccum and Saccum, respectively.

PairAccumulator{T <: Number}

Accumulator that directly faces an incoming data stream. Two values from that stream enter and are processed into the exported values of Tvalue and Svalue.

Contents

• fullpair::Bool
  • A Boolean to keep track of which element of the pair is being accessed. Additionally, when fullpair == true then the contents are exported.
• values::MVector{2, T}
  • The individual values taken from the data stream to be processed. Both Tvalue and Svalue rely on them being accessible.

eltype(::BinningAccumulator{T}) → T

Returns the type parameter for the BinningAccumulator.

getindex(bacc::BinningAccumulator; level)

Overload the [] notation by accessing the BinningAccumulator's LvlAccums at a specific binning level keyword.

Example

julia> bacc = BinningAccumulator();

julia> bacc[level = 0]
LevelAccumulator{Float64} with online fields:
    level    = 0
    num_bins = 0
    Taccum   = 0.0
    Saccum   = 0.0
    Paccum   = PairAccumulator{Float64}(true, [0.0, 0.0])

    Calculated Level Statistics:
    Current Mean             = NaN
    Current Variance         = -0.0
    Current Std. Deviation   = -0.0
    Current Var. of the Mean = NaN
    Current Std. Error       = NaN

length(bacc::BinningAccumulator)

Return the number of LevelAccumulators there are.

Example

julia> bacc = BinningAccumulator();

julia> push!(bacc, [1, 2, 3, 4, 3, 2, 1]); # Data stream with 7 elements

julia> length(bacc) # Only 2 binning levels (1 for unbinned data)
3

push!(bacc::BinningAccumulator, itr)

push! each value of the data stream itr through the BinningAccumulator.

Example

julia> bacc = BinningAccumulator()
BinningAccumulator{Float64} with 0 binning levels.
0th Binning Level (unbinned data):
LevelAccumulator{Float64} with online fields:
    level    = 0
    num_bins = 0
    Taccum   = 0.0
    Saccum   = 0.0
    Paccum   = PairAccumulator{Float64}(true, [0.0, 0.0])

    Calculated Level Statistics:
    Current Mean             = NaN
    Current Variance         = -0.0
    Current Std. Deviation   = -0.0
    Current Var. of the Mean = NaN
    Current Std. Error       = NaN

julia> push!(bacc, [42, -26])
BinningAccumulator{Float64} with 1 binning levels.
0th Binning Level (unbinned data):
LevelAccumulator{Float64} with online fields:
    level    = 0
    num_bins = 2
    Taccum   = 16.0
    Saccum   = 2312.0
    Paccum   = PairAccumulator{Float64}(true, [0.0, 0.0])

    Calculated Level Statistics:
    Current Mean             = 8.0
    Current Variance         = 2312.0
    Current Std. Deviation   = 48.08326112068523
    Current Var. of the Mean = 1156.0
    Current Std. Error       = 34.0

1th Binning Level:
LevelAccumulator{Float64} with online fields:
    level    = 1
    num_bins = 0
    Taccum   = 0.0
    Saccum   = 0.0
    Paccum   = PairAccumulator{Float64}(false, [0.0, 8.0])

    Calculated Level Statistics:
    Current Mean             = NaN
    Current Variance         = -0.0
    Current Std. Deviation   = -0.0
    Current Var. of the Mean = NaN
    Current Std. Error       = NaN

push!(bacc::BinningAccumulator, value::Number)

Add a single value from the data stream into the online binning analysis. The single value enters at the bin at the lowest level.

Example

julia> bacc = BinningAccumulator()
BinningAccumulator{Float64} with 0 binning levels.
0th Binning Level (unbinned data):
LevelAccumulator{Float64} with online fields:
    level    = 0
    num_bins = 0
    Taccum   = 0.0
    Saccum   = 0.0
    Paccum   = PairAccumulator{Float64}(true, [0.0, 0.0])

    Calculated Level Statistics:
    Current Mean             = NaN
    Current Variance         = -0.0
    Current Std. Deviation   = -0.0
    Current Var. of the Mean = NaN
    Current Std. Error       = NaN

julia> push!(bacc, 42)
BinningAccumulator{Float64} with 0 binning levels.
0th Binning Level (unbinned data):
LevelAccumulator{Float64} with online fields:
    level    = 0
    num_bins = 0
    Taccum   = 0.0
    Saccum   = 0.0
    Paccum   = PairAccumulator{Float64}(false, [0.0, 42.0])        

    Calculated Level Statistics:
    Current Mean             = NaN
    Current Variance         = -0.0
    Current Std. Deviation   = -0.0
    Current Var. of the Mean = NaN
    Current Std. Error       = NaN

push!(pacc::PairAccumulator, value::Number)

Overload Base.push! for a PairAccumulator. One can only push! a single value <: Number at a time into this type of accumulator.

show([io::IO = stdout], bacc::BinningAccumulator)

Overload the Base.show function for human-readable displays.

show([io = stdout], lacc::LevelAccumulator)

Overload Base.show for human-readable displays.

measurement(::BinningAnalysisResult)

Convert a BinningAnalysisResult into a Measurements.jl measurement.

RxValue(bacc::BinningAccumulator, [trustworthy_only = true]; [trusting_cutoff])

Calculate the RxValues from the statistically trustworthy binning levels by default, or from all of them if trustworthy_only == false.

RxValue(bacc::BinningAccumulator, level)

Compute the $R_X$ quantity from the binning analysis. This quantity starts at $1$ for low binning levels, then gradually rises, until the bins become statistically uncorrelated at which point $R_X$ should saturate. Once saturated, the effective number of uncorrelated elements in a correlated data stream of size $M$ is given in terms of $R_X$ by $M / R_X$.

Svalue(lacc::LevelAccumulator)

Function to calculate the online $S_{1,m+2}$ summation as:

\[S_{1,m+2} = S_{1,m} + S_{m+1,m+2} + \frac{m}{2(m+2)}\left( \frac{2}{m} T_{1,m} - T_{m+1,m+2} \right)^2.\]

where $T_{m+1,m+2}$ is the pairwise Tvalue for the PairAccumulator.

Svalue(pacc::PairAccumulator)

The $S$ function for a single pair following the accumulation of $m$ data points follows as

\[\begin{aligned} S_{m+1, m+2} &\equiv \sum_{k = m+1}^{m+2} \left( x_k - \frac{1}{2} T_{m+1,m+2} \right)^2 \\ &= \frac{1}{2}\left( x_{m+2} - x_{m+1} \right)^2. \end{aligned}\]

Thus, $S_{m+1,m+2}$ does not need to take $T_{m+1,m+2}$ as an argument.

Tvalue(lacc::LevelAccumulator)

Function to calculate the online $T_{1,m+2}$ summation as:

\[T_{1,m+2} = T_{1,m} + T_{m+1,m+2},\]

where $T_{m+1,m+2}$ is the pairwise Tvalue for the PairAccumulator.

Tvalue(pacc::PairAccumulator)

The $T$ function for a single pair following the accumulation of $m$ data points follows as

\[T_{m+1, m+2} \equiv \sum_{k = m+1}^{m+2} x_k = x_{m+1} + x_{m+2},\]

as expected.

_binning_index_to_findex(level)

Convert the binning_index ∈ {0, 1, ... } to a (Fortran) findex ∈ {1, 2, ... }.

_plateau_found(bacc, fit) → Bool

Test whether a plateau has been found from the fit using the LsqFit.jl package. This includes finding reasonable values for the sigmoid parameters.

autocorrelation_time(RxVal)

Calculation of the autocorrelation time $\tau_X = \frac{1}{2}\left( R_X - 1 \right)$.

bin_depth(bacc::BinningAccumulator)

Number of binned levels present. length of the [BinningAccumulator] minus 1.

Example

julia> bacc = BinningAccumulator();

julia> push!(bacc, [1, 2, 3, 4, 3, 2, 1]); # Data stream with 7 elements

julia> bin_depth(bacc) # Only 2 binning levels (1 for unbinned data)
2

binning_level(index::Int)

Conversion from LvlAccums index to binning_level.

effective_uncorrelated_values(mvals, RxVal)

Calculation of the effective number of uncorrelated values in a correlated datastream:

\[m_{\rm eff} = \mathtt{floor} \left( \frac{ m^{(0)} }{R_X} \right).\]

fit_RxValues(levels, rxvalues, [p0])

Use LsqFit.jl to fit a sigmoid to a set of RxValues generated by a BinningAccumulator.

Additional information

The default arguments passed take on the following values:

• Initial guess for sigmoid parameters: p0 = [1, 0, 1].

fit_RxValues(bacc::BinningAccumulator, [p0])

Use LsqFit.jl to fit a sigmoid to a BinningAccumulator. Note, only statistically trustworthy binning levels are used. This function returns a BinningAnalysisResult struct.

Additional information

The default arguments passed take on the following values:

• Initial guess for sigmoid parameters: p0 = [1, 0, 1].
• See BinningAnalysisResult for a description of how each fieldname is computed.

levels_RxValues(bacc::BinningAccumulator, [trustworthy_only = true]; [trusting_cutoff = TRUSTING_CUTOFF])

Return a Tuple of identically-sized Vectors. The first element of the Tuple are the binning levels and the second are the corresonding RxValues. If trustworthy_only == true, then only the trustworthy levels and values are returned. If trustworthy_only == false, then all levels and values are returned (except for the last level which is typically not full).

This function is meant to make visualization more convenient and does not offer any different functionality than what was available before.

max_trustworthy_level(nelements; [trusting_cutoff])

Calculates the highest binning level that remains statistically trustworthy according to the TRUSTING_CUTOFF, $t_c$.

Given a number of elements in a data stream, $N$, this quantity is

\[\ell_{\rm max} = {\rm floor} \left[ \log_2 \left( \frac{N}{t_c} \right) \right].\]

reset!(bacc::BinningAccumulator{T})

Reset the BinningAccumulator by reconstruction.

Additional information

While this is not a literal reset per se, with a large enough BinningAccumulator it will be certainly faster just to blow up the old one (in memory) and start over.

reset!(pacc::PairAccumulator)

Return the PairAccumulator to its initial state. Presumably one just exported the Tvalue and Svalue from it before the reset!.

sigmoid(x, [amp = 1], [θ₁ = 0], [θ₂ = 1])

Calculate a Sigmoid at a given argument x. The Sigmoid function $S(x; A, \theta_1, \theta_2)$ is of the form

\[S(x; A, \theta_1, \theta_2) = \frac{A}{1 + \exp\left( \theta_1 - \theta_2 x \right)}.\]

sigmoid(x, pvals) = sigmoid(x, pvals...)

Vectorized sigmoid function.

sigmoid_jacobian(x, pvals)

Calculate the "Jacobian" of first derivatives for a sigmoid to speed the LsqFit fitting. The derivatives are given by

\[\begin{aligned} \frac{\partial S}{\partial A} &= \frac{1}{1 + \exp\left( \theta_1 - \theta_2 x \right)}, \\ & \\ \frac{\partial S}{\partial \theta_1} &= -\frac{A \, \exp\left( \theta_1 - \theta_2 x \right) }{\left[ 1 + \exp\left( \theta_1 - \theta_2 x \right) \right]^2}, \\ & \\ \frac{\partial S}{\partial \theta_2} &= \frac{A \, x \, \exp\left( \theta_1 - \theta_2 x \right) }{\left[ 1 + \exp\left( \theta_1 - \theta_2 x \right) \right]^2}. \end{aligned}\]

std_error( bacc::BinningAccumulator )

Online measurement of the [BinningAccumulator] standard error.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

std_error( lacc::LevelAccumulator ) = sqrt(var_of_mean(lacc))

Online measurement of the [LevelAccumulator] standard error.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

trustworthy_indices(bacc::BinningAccumulator; [trusting_cutoff])

Return the (Fortran) indices 1, 2, 3, ... corresponding to trustworthy_levels.

trustworthy_level(level; [trustworthy_cutoff = 64])

A binning level is said to be a trustworthy_level if the number of bins it contains is greater than or equal to the trustworthy_cutoff.

The number of bins $N_{\rm bin}$ in any binning level is related to the number of elements $N$ and its binning level $\ell \in \{0, 1, \dots \}$ by

\[N_{\rm bin} = \frac{N}{2^{\ell}}.\]

This means that, for a given trustworthy_cutoff of $t_c$, then the maximum number of trustworthy_levels present are

\[{\rm Total}(\ell) = 1 + {\rm floor} \left[ \log_2 \left( \frac{N}{t_c} \right) \right],\]

where the extra 1 comes from assuming the original data stream has more than $t_c$ elements in it, making the $\ell = 0$ level a trustworthy_level.

update_SandT!(lacc::LevelAccumulator)

Apply the Svalue and Tvalue formula to update lacc.Taccum and return the PairAccumulator Tvalue increment.

Additional information

• $S$ must be updated before $T$ since the former depends on the latter's history.

update_num_bins!(lacc::LevelAccumulator, [incr = 2])

Increment the number of bins accumulated by incr.

var_of_mean( bacc::BinningAccumulator; [level = 0] )

Online measurement of the [BinningAccumulator] variance of the mean.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

var_of_mean( lacc::LevelAccumulator ) = var(lacc) / lacc.num_bins

Online measurement of the [LevelAccumulator] variance of the mean.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

mean( bacc::BinningAccumulator; [level = 0] )

Online measurement of the [BinningAccumulator] mean.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

mean( lacc::LevelAccumulator )

Online measurement of the [LevelAccumulator] mean.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

std( bacc::BinningAccumulator )

Online measurement of the [BinningAccumulator] standard deviation.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

std( lacc::LevelAccumulator ) = sqrt(var(lacc))

Online measurement of the [LevelAccumulator] standard deviation.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

var( bacc::LevelAccumulator; [level = 0] )

Online measurement of the [BinningAccumulator] variance.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.

var( lacc::LevelAccumulator )

Online measurement of the [LevelAccumulator] variance.

Additional information

• This quantity is considered online despite that it is not regularly updated when data is push!ed from the stream.