Batches

Batch

A container for running and analyzing multiple simulations. Stores simulation configurations that are executed sequentially by process!.

Base.append!(batch1::Batch, batch2::Batch)

Append all simulation configs and setups from batch2 to batch1.

merge(batches::Batch...)
merge(batches::Vector{Batch})

Generates a new Batch that contains all simulation configs of the input Batches.

process!(batch::Batch; keep_rundata=true, rd_style="LightRD", median_by=nothing, group_by=nothing, seed=nothing, customlogger=nothing, on_run=nothing)

Processes all simulation configurations in batch sequentially, accumulating results into a BatchProcessor.

Keyword Arguments

• keep_rundata: if true, every run's ResultData is stored in bp.rundata. Default: true.
• rd_style: the ResultData style for individual and median runs. Default: "LightRD".
• median_by: a function pp::PostProcessor -> scalar for selecting the median run. The simulation with criterion closest to the median across all runs is re-run and stored. For multi-group batches one median run per group is computed. Default: nothing
• group_by: a Symbol naming a field in each simulation config NamedTuple to use as the grouping key. When nothing (default), no per-group tracking is performed. Pass e.g. group_by = :label to group by the :label field.
• seed: seed for reproducibility. Passing the same value produces the same results. If nothing, a random seed is generated. Retrieve the used seed via seed(bp).
• customlogger: a CustomLogger to attach to each simulation. An independent copy is created per run so data is not mixed across runs. Default: nothing.
• on_run: a callback (rd::ResultData, i::Integer) -> ... invoked as each run finishes, receiving that run's ResultData. Lets callers consume runs in a streaming fashion without retaining them all. Default: nothing.

simconfigs(batch::Batch)

Returns the vector of simulation configuration NamedTuples in this Batch.

BatchData <: AbstractResultData

A struct that stores all processed data of a batch of simulation runs. It holds four internal dictionaries with meta-, simulation-, system- and data as well as multiple DataFrames that are the BatchProcessor outcomes. Note that some information (i.e., execution date or GEMS version) is only read out upon generation of this BatchData object. Thus, there can be inconsistencies if the BatchData object is not generated right after simulation execution.

allocations(bd::BatchData)

Returns allocation data of the simulation runs in this batch. (Note: This data is only available if the simulation runs were done via the main() function)

attack_rate(bd::BatchData)

Returns aggregated values for the attack_rate accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

cpu_data(bd::BatchData)

Returns the processor information (not available for ARM Macs)

cumulative_cases(bd::BatchData)

Returns aggregated cumulative case counts per tick across simulation runs. Returns a Dict{String, DataFrame} keyed by column name (e.g. exposed_cum, recovered_cum, deaths_cum).

cumulative_disease_progressions(bd::BatchData)

Returns aggregated values for cumulativediseaseprogressions per tick accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

cumulative_quarantines(bd::BatchData)

Returns aggregated values cumulative_quarantines per tick accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

dark_figure(bd::BatchData)

Returns aggregated dark figure fractions per tick across simulation runs as a DataFrame. Columns: tick, minimum, maximum, mean, std, lower_95, upper_95.

dataframes(batchData)

Returns the dataframes dict of the batch data object.

effectiveR(bd::BatchData)

Returns aggregated values for the effective R value for each tick accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

execution_date(bd::BatchData)

Returns the timestamp of batch data generation.

exportJLD(batchData, directory)

Exports the BatchData object as a JLD2 file, storing it in the specified directory.

free_mem_size(bd::BatchData)

Returns the available system memory

generation_times(bd::BatchData)

Returns aggregated mean generation times per tick across simulation runs as a DataFrame. Columns: tick, minimum, maximum, mean, std, lower_95, upper_95.

GEMS_version(bd::BatchData)

Returns the GEMS version this BatchData object was generated with.

git_branch(bd::BatchData)

Returns the current git branch.

git_commit(bd::BatchData)

Returns the current git commit.

git_repo(bd::BatchData)

Returns the current git repository.

household_secondary_attack_rate(bd::BatchData)

Returns aggregated values for the pooled household secondary attack rates (household_sar, household_sar_home) accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

id(bd::BatchData)

Returns the stringified SHA1 hash that serves as a unique identifier.

import_batchdata(filepath::AbstractString)

Import the BatchData object from a jld2 file. Returns the BatchData object.

info(bd::BatchData)

Prints info about available fields in the BatchData object.

julia_version(bd::BatchData)

Returns the Julia version that was used to generate this result data object.

kernel(bd::BatchData)

Returns the system kernel information

meta_data(bd::BatchData)

Returns the metadata dict of the batch data object.

number_of_runs(bd::BatchData)

Returns the number of runs in this batch.

median_run(bd::BatchData)

Returns the ResultData of the simulation whose criterion is the median across all runs, or nothing if median_by was not set.

runs(bd::BatchData)

Returns the ResultData objects of each of the runs in the the batch data object.

runtime(bd::BatchData)

Returns runtime data of the simulation runs in this batch. (Note: This data is only available if the simulation runs were done via the main() function)

seed(bd::BatchData)

Returns the master seed used to derive per-simulation seeds for this batch, or 0 if the batch was processed without explicit seeding.

sim_data(bd::BatchData)

Returns the simdata dict of the batch data object.

system_data(bd::BatchData)

Returns the systemdata dict of the batch data object.

tests(bd::BatchData)

Returns aggregated values for tests per tick accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

tick_cases(bd::BatchData)

Returns aggregated values for newly exposed inviduals per tick accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

threads(bd::BatchData)

Returns the number of threads this Julia instance was started with

total_infections(bd::BatchData)

Returns aggregated values for total_infections accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

total_mem_size(bd::BatchData)

Returns the total system memory

total_quarantines(batchData)

Returns aggregated values for total_quarantines accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

total_tests(bd::BatchData)

Returns aggregated values for total_tests accross the simulation runs in this batch. It returns mean, standard deviation, range, and confidence intervals.

word_size(bd::BatchData)

Returns the system word size

DefaultBatchData <: BatchDataStyle

The default style for BatchData objects. It contains all that can currently be calculated from a BatchProcessor.

Fields

• data::Dict{String, Any}

  • meta_data::Dict{String, Any}

    • execution_date::String: Time this BatchData object was generated
    • GEMS_version::VersionNumber: GEMS version this BatchData object was generated with

  • system_data::Dict{String, Any}

    • kernel::String: System kernel
    • julia_version::String: Julia version that was used to generate this data object
    • word_size::Int64: System word size
    • threads::Int64: Number of threads this Julia instance was started with
    • cpu_data::Markdown.MD: Information on the processor (not available for ARM Macs)
    • total_mem_size::Float64: Total system memory
    • free_mem_size::Float64: Available system memory
    • git_repo::SubString{String}: Current Git repository
    • git_branch::SubString{String}: Current Git branch
    • git_commit::SubString{String}: Current Git commit ID

  • sim_data::Dict{String, Any}

    • runs::Union{Nothing, Vector{ResultData}}: Individual ResultData objects, or nothing if keep_rundata was false during the batch run
    • median_run::Union{Nothing, ResultData}: The run whose criterion is closest to the median across all runs, or nothing if median_by was not set
    • number_of_runs::Int64: Number of simulation runs
    • total_infections::Dict{Int8, Dict{String, Real}}: Per-pathogen summary statistics on total infections across simulation runs
    • attack_rate::Dict{Int8, Dict{String, Real}}: Per-pathogen summary statistics on attack rates across simulation runs
    • r0::Dict{Int8, Dict{String, Real}}: Per-pathogen summary statistics on the basic reproduction number (R0)
    • household_secondary_attack_rate::Dict{Int8, Dict{String, Dict{String, Real}}}: Per-pathogen summary statistics on the pooled household secondary attack rates (household_sar, household_sar_home)
    • total_quarantines::Dict{String, Real}: Summary statistics on total quarantines across simulation runs
    • total_tests::Dict{String, Dict{String, Real}}: Summary statistics on total tests per TestType

  • dataframes::Dict{String, Any}

    • tick_cases::DataFrame: Aggregated data on infections per tick across simulation runs
    • effectiveR::DataFrame: Aggregated data on the effective reproduction number per tick across simulation runs
    • tests::Dict{String, Dict{String, DataFrame}}: Aggregated data on tests per tick across simulation runs
    • cumulative_quarantines::DataFrame: Aggregated data on cumulative quarantines per tick across simulation runs
    • cumulative_disease_progressions::Dict{String, DataFrame}: Aggregated data on cumulative disease progressions per tick across simulation runs