The Hubverse: Streamlining Collaborative Infectious Disease Modeling

RSECon26 · Sheffield

Anna Krystalli

info@r-rse.eu

R-RSE SMPC

Consortium of Infectious Disease Modeling Hubs

9 September 2026

Background

Hi everyone,

My name is Anna Krystalli.

I used to be an RSE at the University of Sheffield, so it’s lovely to be back, but a few years ago I returned home to Greece and set up my own RSE company.

Today I’m going to talk to you about the hubverse, a project I’ve been involved in since 2023.

And I’ll start with some motivating background on the project.

❌ Where we started

Infectious disease modeling scaled rapidly through COVID-19…

• …but early on the landscape was fragmented:
  • Each group used its own data structures
  • Results were hard to integrate or compare
  • Modelers and decision-makers not always aligned

“Comparing the accuracy of forecasting applications is difficult because forecasting methods, forecast outcomes, and reported validation metrics varied widely.”

• Chretien et al., PLOS ONE, 2014

Over the past two decades, and especially through the COVID-19 pandemic, infectious disease modeling became a central tool in public health decision-making.

With this, we’ve seen an explosion of forecasting efforts.

The growth has been encouraging: it reflects increased computational capacity, improved data availability, and global interest in predictive analytics for public health.

But it initially also came with some growing pains.

• Modeling efforts developed independently: each group used its own data structure for model outputs, metadata and evaluation, making integration and comparison difficult.

• Modelers and decision-makers didn’t always communicate clearly, so model outputs didn’t always match what public health actually needed.

This initially limited the collective utility of early efforts.

✨ The promise of modeling hubs

Modeling hubs coordinate collaborative forecasting:

• Provide centralised location for effort coordination

• Define data standards and modeling targets

• Improve transparency and comparability

• Aggregate forecasts enabling ensembles

• Facilitate timely public health decision-making

  
  

“Collaborative Hubs: Making the Most of Predictive Epidemic Modeling”, American Journal of Public Health Reich, et al. 2022

This is where the idea of modeling hubs came in, to help coordinate collaborative forecasting.

• First, hubs bring structure and a central location for collecting model outputs.

• They establish a shared modeling protocol and data standard.

• By enforcing standards and submission rules, hubs ensure that forecasts can be compared.

• and also integrated into ensemble models.

• Finally, hubs make it easier for:

  • decision-makers to set the forecasting task and
  • stakeholders to access timely, relevant forecasts.

🕰️ Project origins

• Pre-COVID: Forecasting code base existed for CDC influenza hubs
• During COVID: That code was reused for new COVID-19 hubs + demand internationally (e.g. Europe) for similar setups
• ❗ Problem: Each hub required manual editing of source code

Figure credits: Alex Vespignani and Nicole Samay

Before COVID, the US CDC had already established a foundation for collaborative infectious disease forecasting through its FluSight initiative, focused on influenza.

This resulted in an initial working codebase.

When the pandemic hit, interest in modeling hubs exploded with many groups wanting to repurpose this codebase:

• First the US CDC COVID-19 Forecast Hub

• Then interest grew to scenario modeling hubs, as well as internationally.

‼ The problem was, each new hub required forking and manual changes to the source code, obviously not ideal.

When the dust settled, effort was initiated to develop a more configurable, generalized and modular system to support new hubs.

🌐 Enter the hubverse

An open-source software ecosystem to power modeling hubs:

• GitHub repositories for centralising hub activity
• Data standards for infectious disease modeling data
• Schema-driven configuration for modeling tasks + hub setup
• Modular tools for hub administration, data validation, access, evaluation, ensembling and communication

📄 A software platform for collaborative infectious disease modelling (2026), Nature Health, Consortium of Infectious Disease Modeling Hubs et al. https://doi.org/10.1038/s44360-026-00145-7

… and this is how the hubverse began,

It’s an open-source software ecosystem

• (leaning on free tiers of a couple of commercial platforms)
• designed to support the full lifecycle of infectious disease modeling hubs.

It brings together:

• A centralised version-controlled location for hub activities through GitHub repositories

• Standards for infectious disease modeling data.

• Schema-driven configuration, for defining a hub’s modeling tasks, submission schedule, and other expectations.

• A suite of tools for hub administration, data validation, access, evaluation, ensembling, visualisation and communication.

Anatomy of a hub

So let’s start by looking at what a hub actually is.

🏗️ What a hub actually is

1. Modeling teams submit model output
2. Submissions validated against tasks.json
3. Accepted output merged into the hub

Everything downstream, ensembles, visualisations, evaluations, comes for free once the data are standardised.

Figure 1 from the hubverse paper (preprint, CC BY 4.0)

Here’s a general overview of a hub

It starts with the hub administrator. They configure the hub,

and they also maintain target-data: observed data, used to evaluate model performance.

With that in place, different modeling teams

• submit model outputs
• of any type of model they want
• as long as they match hub requirements.

To confirm this, each submission is validated against the config. If it passes, it’s merged.

… and downstream activities like building ensembles, visualising, and evaluating model outputs follow.

☑️ A shared data standard

Modeling hubs are built around a shared data standard:

• Structured hub layout: consistent file system for organizing hub materials
• Standard model output format: for file content and naming
• Modeling task definition: what’s predicted, by when, for where, in what form

✅ Enable comparability, validation and integration

FluSight-forecast-hub/
├── hub-config/
│   ├── admin.json
│   ├── tasks.json
│   ├── model-metadata-schema.json
│   └── target-data.json
├── model-output/
│   └── <model_id>/
│       └── <round_id>-<model_id>.csv
├── model-metadata/
│   └── <model_id>.yml
└── target-data/

All this is enabled by the core of the hubverse which is our shared data standard.

The standard covers three things.

1. the hub file structure itself with standard directories.

   • hub-config for config files,
   • model-output for submissions, one directory per model,
   • model-metadata for metadata files describing each model, and
   • target-data for storing observed data.

2. the model output files themselves follow a consistent format and naming convention.

3. how a modeling task is defined: what’s predicted, by when, for where, in what form.

Together, these enable comparability, validation and integration.

🦠 Meet the hub: CDC FluSight

To make the things I’m going to talk about concrete, let me introduce a real hub: the CDC FluSight hub

The hub we’ll follow: CDC FluSight

https://github.com/cdcepi/FluSight-forecast-hub

• Used by US CDC to monitor influenza severity
• In season, ~60 models from ~30 teams, every single week
• Managed with the full hubverse stack since 2023/24
• 95 models from 47 teams across 88 rounds since 2023
• Hosted on GitHub + S3 cloud mirror

This is a flagship example of the hubverse stack in action.

It’s used by the US CDC to monitor influenza severity.

In season, they can expect submissions from:

• around 60 models
• from about 30 teams, every week.

The hub is fully managed using the hubverse since the 2023/24 season. Cumulatively, 95 models from 47 teams have submitted across 88 rounds since then.

📁 Where model output lives

model-output/<model_id>/<round_id>-<model_id>.csv · one directory per model, one file per round

While we’re here, let’s dig into the model-output directory. We see:

1. one directory per model.

2. one file per round inside that directory.

3. the file name is constructed from the round and model identifiers so is self-describing.

In FluSight the round ID is a date, so the folder’s contents read as that model’s submission history.

Model output: expected patterns

Configuring FluSight

Now let’s talk about configuring a hub.

⚙️ Config-driven hub setup

Hub administrators configure hubs with structured JSON config files:

• admin.json: hub-level metadata
• tasks.json: the scientific question: what to predict, when, and how
• model-metadata-schema.json: what modelers must tell you about their model
• target-data.json: what observed data should look like

All validated against a versioned JSON schema. The config is the contract between administrators, modelers and tooling.

Hub setup is configuration-based. No more forking and editing source code.

Administrators configure the hub through structured JSON files in hub-config.

• admin.json for hub-level metadata.
• tasks.json for the scientific question: what to predict, when, and how
• model-metadata-schema.json to validate model level metadata the teams submit.
• target-data.json which defines expectations for the observed data.

Config files themselves are validated against a versioned JSON schema that we maintain.

The config is the contract. It’s:

• how the admin communicates with the teams,
• what the validation enforces,
• and what downstream tools use to understand the hub.

Hub configuration files

🗂️ admin.json: the hub’s identity card

{
  "schema_version": ".../schemas/main/v6.0.0/admin-schema.json",
  "name": "US CDC FluSight",
  "maintainer": "US CDC",
  "contact": {
    "name": "Rebecca Borchering",
    "email": "xhq2@cdc.gov"
  },
  "repository": {
    "host": "github",
    "owner": "cdcepi",
    "name": "FluSight-forecast-hub"
  },
  "file_format": ["csv", "parquet"],
  "timezone": "US/Eastern",
  "cloud": {
    "enabled": true,
    "host": {
      "name": "aws",
      "storage_service": "s3",
      "storage_location": "cdcepi-flusight-forecast-hub"
    }
  }
}

Let’s look at the FluSight admin.json.

First, we have schema_version which pins the config to a specific schema version, v6 here.

Then typical administrative metadata like hub name, maintainer, etc.

Then where the hub lives.

Followed by operational settings like what file_formats are allowed and the timezone submission deadlines are gated relative to.

Finally cloud allows admins to define whether and where a cloud mirror of hub data exists. More on that shortly.

🎯 Inside tasks.json

tasks.json
└── rounds[]                                   # a submission cycle
    ├── round_id_from_variable: true           # round IDs live in the data...
    ├── round_id: "reference_date"             # ...in this task ID
    ├── submissions_due                        # window, relative to the round
    └── model_tasks[]                          # groups of related predictions
        ├── task_ids{}                         # the DIMENSIONS of a prediction
        ├── output_type{}                      # the FORM the prediction takes
        └── target_metadata[]                  # what the target means

A hub can have many rounds, each with many model tasks, so a single hub can ask for several different kinds of prediction at once.

Now the more complex but also more consequential config, tasks.json. Let’s take a closer look.

At the top level we have rounds. A round is one submission cycle.

Each round has a round_id.

1. It can be a literal string if a hub uses separate config blocks for each round.
2. But if the rounds are driven by a variable in the data,
   • admins supply the name of a task ID,
   • that task ID’s values become the round IDs while all other configuration in the block is shared between rounds.
   • That’s what round_id_from_variable: true is flagging.

submissions_due sets the submission window.

Then model tasks. This is where a hub defines what it actually wants predicted.

Each task is configured with three blocks.

• task_ids define the dimensions of what’s predicted.
• output_type is the form the prediction takes.
• And target_metadata holds additional information about modeling targets.

Because a round can hold several model tasks, a hub can ask for quite different predictions at the same time.

FluSight asks for both:

• a count of hospitalisations
• a categorical rate-change outlook, in the same round.

🏷️ Task IDs: the dimensions of a prediction

"task_ids": {
  "reference_date": {
    "required": null,
    "optional": ["2023-10-07", "2023-10-14", "..."]
  },
  "target": {
    "required": null,
    "optional": ["wk inc flu hosp"]
  },
  "horizon": {
    "required": null,
    "optional": [-1, 0, 1, 2, 3]
  },
  "location": {
    "required": null,
    "optional": ["US", "01", "02", "..."]
  },
  "target_end_date": {
    "required": null,
    "optional": ["2023-09-23", "2023-09-30", "..."]
  }
}

reference_date + horizon × 1 week = target_end_date

Here’s the task ID block from FluSight, abridged.

Every task ID gets two arrays. required and optional.

• Required means you must submit a prediction for every one of those values.
• Optional means you may. For FluSight, every combination is optional.

These are all standard forecasting task IDs, so they’re covered by the hubverse schema and validated strongly out of the box.

Hubs aren’t restricted to them, though. You can define whatever task ID your modelling needs, you’ll just have to add custom validation for it.

Also target_end_date is what we consider a derived task ID.

• Its values have a one to one relationship with other task IDs:
• here the reference date plus the horizon in weeks gives you the target end date.

Finally target is a special task ID, so every value in the target column gets its own entry in target_metadata.

Task ID variables

🎯 Targets carry more than a name

target is a special task ID: every value in the column gets its own metadata entry

"target_metadata": [{
  "target_id": "wk inc flu hosp",
  "target_name": "incident influenza
     hospitalizations",
  "description": "Count of new hospital
     admissions in the week ending...",
  "target_units": "count",
  "is_step_ahead": true,
  "time_unit": "week",
  "target_type": "continuous"
}]

Descriptive

• target_name, description: for humans, and for axis labels

Quantitative

• target_units: what is being counted
• is_step_ahead, time_unit: one step in a sequence
• target_type: its statistical type (continuous, ordinal, date, …)

target_type constrains how a prediction of this target can be expressed. Which is exactly what an output type is…

Let’s have a look at metadata collected at the target level:

Some are descriptive. A human readable name, a fuller description. Useful for people, and for labelling an axis on a visualisation.

It also carries quantitative properties. The units. Whether it’s a step-ahead forecast, in what time unit.

Also the target’s statistical type.

And this is intimately linked to the last important tasks config block. Output type.

So let’s take a look.

Target metadata

📐 Output types: how a prediction is expressed

output_type	output_type_id	value
mean	(none)	mean of the predictive distribution
median	(none)	median of the predictive distribution
quantile	a probability level, e.g. 0.75	the value at that quantile
cdf	a possible value, e.g. 500	P(outcome ≤ 500)
pmf	a possible category, e.g. "increase"	P(outcome = "increase")
sample	a sample index, e.g. "s3"	one draw from the distribution

An output type is the form a prediction takes. The hubverse currently supports six.

• We accept standard point estimates, mean and median. So one number per prediction.

• The most common output type hubs collect, though, is quantile. The hub defines the probability levels it wants, and teams submit the predicted value at each one.

• cdf is the reverse. For each value the hub defines, teams submit the probability of landing at or below it. Useful when the question is about a threshold.

• pmf is for categorical targets. A probability on each category, summing to one.

• sample is a collection of draws from the predictive distribution.

Hubs can also ask for samples drawn from a joint distribution. Trajectories are a common example: a single draw across all horizons at once.

Formats of model output

🎛️ …and how a hub configures them

"output_type": {
  "quantile": {
    "output_type_id": {
      "required": [0.01, 0.025, "...", 0.975, 0.99]
    },
    "is_required": true,
    "value": { "type": "double", "minimum": 0 }
  },
  "sample": {
    "output_type_id_params": {
      "compound_taskid_set": ["reference_date", "location", "target"],
      "min_samples_per_task": 100,
      "max_samples_per_task": 100
    },
    "is_required": false,
    "value": { "type": "integer", "minimum": 0 }
  }
}

So that’s what output types are. Here’s how a hub actually configures them.

Within an output type, output_type_id.required is where a hub configures which output type IDs it wants submitted. For quantiles, that’s the probability levels.

Notice there’s no optional property here. Output type IDs are all required because a partial set breaks things downstream.

is_required defines whether the output type itself is required within the model task. Quantiles are required for FluSight.

Finally value constrains the predicted value. For FluSight type must be double with zero as minimum, so no negative counts allowed.

Samples have a slightly different config block:

• compound_taskid_set is what declares the joint distribution. It lists the task IDs held constant within a group, so everything else, here horizon, is sampled jointly. This is how a trajectory is configured.
• The hub also configures the min and max number of samples required, here both are 100 so the hub expects exactly 100 samples per compound task.

Output types

🔍 Configured model output in the wild

Real FluSight rows from round 2026-01-10, national. task IDs · output type · value

---

reference_date  horizon  target                   target_end_date  location  output_type  output_type_id  value
2026-01-10      1        wk inc flu hosp          2026-01-17       US        quantile     0.01            18429
2026-01-10      1        wk inc flu hosp          2026-01-17       US        quantile     0.025           20177
2026-01-10      1        wk inc flu hosp          2026-01-17       US        quantile     0.05            21981
                                                                                          …                    
2026-01-10      1        wk inc flu hosp          2026-01-17       US        quantile     0.5             38936
                                                                                          …                    
2026-01-10      1        wk inc flu hosp          2026-01-17       US        quantile     0.975           61292
2026-01-10      1        wk inc flu hosp          2026-01-17       US        quantile     0.99            68125
2026-01-10      0        wk inc flu hosp          2026-01-10       US        sample       us_s1           36268
2026-01-10      1        wk inc flu hosp          2026-01-17       US        sample       us_s1           36397
2026-01-10      2        wk inc flu hosp          2026-01-24       US        sample       us_s1           33669
2026-01-10      3        wk inc flu hosp          2026-01-31       US        sample       us_s1           33727
2026-01-10      1        wk flu hosp rate change  2026-01-17       US        pmf          large_decrease  0.114
2026-01-10      1        wk flu hosp rate change  2026-01-17       US        pmf          decrease        0.244
2026-01-10      1        wk flu hosp rate change  2026-01-17       US        pmf          stable          0.158
2026-01-10      1        wk flu hosp rate change  2026-01-17       US        pmf          increase        0.231
2026-01-10      1        wk flu hosp rate change  2026-01-17       US        pmf          large_increase  0.253

So let’s look at how all this configuration gets translated into valid model output.

• In blue are the task IDs
• Next we have output_type and output_type_id
• And finally, the predicted value.

Let’s see some FluSight output types:

1. First, the required quantile output type for the hospitalisation count target. Here you can see the output type IDs represent quantile levels.

2. Next a single sample representing a single draw of a trajectory, for that same target. Here output_type_id represents the sample index and the values of all task IDs stay constant apart from horizon.

3. Finally the pmf output type for a categorical rate-change outlook for a completely different target. Here output type IDs are categories and values are probabilities that sum to one across individual predictions.

Three very different statistical objects in the same eight columns.

🏷️ Hubs collect metadata about the models too

model-metadata-schema.json sets out what each team must document about their model

• One file per model in model-metadata/, validated on every PR
• The hubverse ships a schema template, hubs adapt it

Provenance for every prediction, and a natural basis for model cards.

team_name: "UMass-Amherst"
model_abbr: "flusion"
model_version: "1.1"
model_contributors:
  - name: "Evan Ray"
    affiliation: "UMass Amherst"
license: "CC-BY-4.0"
data_inputs: "NHSN, FluSurv-NET and ILINet."
methods: "Ensemble of statistical and machine
  learning time series models."
designated_model: true

I just wanted to quickly mention that hubs also collect information from submitting teams about their models.

Admins set out what teams have to document as a schema and that’s used to validate the metadata files submitted by teams.

We ship a template of this schema and hubs are free to adapt it.

Model metadata

Beyond the data standard

That’s the standard, and how you configure it. Everything from here on builds on top of it.

🔁 Automating everything we can

GitHub Actions do the operational work:

• ✅ PR-level model output validation
• ✅ Hub configuration validation
• ☁️ Cloud hub data syncing
• 📊 Dashboard builds

All actions live in hubverse-actions and install with hubCI::use_hub_github_action()

Automation is a cornerstone of hub operational work and we provide GitHub Action templates and maintain composite actions for common workflows.

This includes workflows for:

• Validating model output on every pull request.
• Validating hub config whenever hub-config changes.
• Syncing validated output to the cloud.
• Regenerating dashboard data and evaluation metrics on every update and building the dashboard.

They all live in our hubverse-actions repo, and admins install them using the hubCI package.

☁️ Cloud storage and access

Validated hub data is mirrored to a public S3 bucket

• Opened directly as an Arrow dataset: no cloning, no full download
• Query-able via 📦 hubData (R) and 📦 hubdata (Python)
• Turns a hub into an open data resource, not just a repository

Enable it with a few lines in admin.json. The hubverse provisions the bucket.

Alongside the GitHub repo, validated hub data can also be mirrored to a public S3 bucket which we currently provision and maintain.

The mirror is synced as a parquet Arrow dataset which can be queried and filtered remotely before downloading data, in both R and Python.

No cloning, or downloading a full hub’s worth of data required.

📊 Dashboards & communication

• Built with Quarto so easily customisable via Quarto configuration
• Deployed as a fully static site, no backend required
• Powered by JSON data prepared via GitHub workflows
• Interactive UI built with client-side JavaScript (fast!)
• New instances set up by copying/configuring the hub-dashboard-template

Finally because dashboards matter for communicating with a wider audience, we developed and maintain a plug-and-play template that admins can just copy and configure.

It’s based on Quarto, so it’s easy to customise.

Importantly it’s fully static with no backend. GitHub workflows turn hub data into JSON, and a JavaScript frontend makes it interactive.

So it’s fast but also free and easy to host on GitHub Pages.

👥 Hubverse roles and their tooling

Now let’s recap, this time from the perspective of each hubverse role and the tools designed to support them.

1. Starting with administrators who set the hub up and maintain the target data.

   • They can use hubAdmin to build and validate the hub’s config,
   • consult the schemas in addition to our documentation to plan hub configuration,
   • hubCI to install operational workflows,
   • and hubValidations to help them customise validation.

2. Then modelers who submit their model output can use

   • hubValidations to check a submission before opening a pull request,
   • and hubData to access hub data (including target data).

3. Analysts who want to perform analyses can use:

   • hubData to query and access data from the hub,
   • hubVis to plot model outputs,
   • hubEnsembles to build ensembles and
   • hubEvals to evaluate model performance against the target data.

4. Stakeholders, who might be less technical and just want answers,

   • can access insights via the dashboard (supported by the dashboard template).
   • But if they need data, they can also download raw data directly from the cloud mirror.

5. Finally underneath it all, the hubverse developers and community. We maintain a bunch of additional tools and templates to support our activities.

🦠 FluSight in operation

Now let’s look at a hub in operation

✅ Model output validation with hubValidations

Submitted by pull request, validated by GitHub Actions, reported back on the PR

• Passed
• Failed

github-actionsbot commented

Submission validation

✅ All validation checks passed.

Check results

── hub-config ────
✔ [valid_config]: All hub config files are valid.

── team1-goodmodel/2022-10-29-team1-goodmodel.csv ────
✔ [file_name]: File name is valid.
✔ [file_location]: File directory name matches
  `model_id` metadata in file name.
✔ [file_format]: File is accepted hub format.
✔ [metadata_exists]: Metadata file exists.
✔ [colnames]: Column names are consistent with
  expected round task IDs and std column names.
✔ [col_types]: Column data types match hub schema.
✔ [req_vals]: Required task ID/output type/output
  type ID combinations all present.
… 18 further checks

Results for commit a1b2c3d

github-actionsbot commented

Submission validation

❌ Validation failed. The checks below did not pass. Push a new commit to the pull request to re-run them.

── hub-config ────
✔ [valid_config]: All hub config files are valid.

── team1-goodmodel/2022-10-22-team1-goodmodel.csv ────
✔ [file_name]: File name is valid.
✔ [metadata_exists]: Metadata file exists.
✖ [submission_time]: Submission time must be within
  accepted submission window for round. Current time
  "2026-08-12 18:55:10 UTC" is outside window
  2022-10-16 EDT--2022-10-23 23:59:59 EDT.
✔ [colnames]: Column names are consistent with
  expected round task IDs and std column names.
✔ [req_vals]: Required task ID/output type/output
  type ID combinations all present.
✔ [value_col_non_desc]: Quantile or cdf values
  increase when ordered by `output_type_id`.
… 21 further checks

Results for commit a1b2c3d

And specifically at a hub’s core operation which is model submission.

As mentioned, every model output comes in as a pull request, and gets validated by CI which effectively deploys hubValidations functions.

• On the left, we see submissions from different teams.

• On the right, we see validation results, posted to the PR.

(Passed tab) When everything passes, you get a green tick, and the detail stays collapsed but still accessible.

(Failed tab) When it fails you get a red cross, check results are expanded with failed checks highlighted. Here the only problem is that the submission arrived outside the submission window.

🔎 What actually gets checked

Standard checks, generated from tasks.json

• 📄 File: name, location, format, submission window
• 🧱 Structure: columns, types, duplicates, round ID
• 🎯 Content: valid task ID combinations, required tasks present, values in bounds
• 📐 Output-type specific: quantiles don’t cross, pmf sums to 1, sample counts
• 🏷️ Model metadata and 🎯 target data: each with their own checks

Hub-specific checks in hub-config/validations.yml

default:
  validate_model_data:
    horizon_timediff:
      fn: "opt_check_tbl_horizon_timediff"
      pkg: "hubValidations"
      args:
        t0_colname: "reference_date"
        t1_colname: "target_end_date"

• Opt-in checks shipped with hubValidations
• Or the hub’s own R functions, in src/validations/R

create_custom_check() scaffolds a custom check: right structure, return classes and conventions, ready to fill in.

So what does validation actually cover?

On the left, the list of standard checks:

• at the file level: we check whether a file is:
  • named correctly and stored in the right place and format,
  • whether it can be parsed and
  • whether it was submitted inside the window.
• Structural properties we check include:
  • expected columns and data types,
  • no duplicate rows, and that
  • file round IDs match IDs in the data.
• further Content checks include:
  • whether task ID and output type ID values are valid,
  • whether every required task has been submitted and
  • that predicted values fall within the configured bounds.
• Finally we also include output-type-specific ones. For example:
  • Quantile predicted values mustn’t cross.
  • pmf probabilities must sum to one per task.
  • The correct number of samples must be submitted, with a consistent compound task ID set.

Plus we validate model metadata against their schema and target data have their own validation suite.

But hubs aren’t all the same, and they might want to include additional checks.

Admins can configure extra checks using the validations.yml config file.

They can use opt-in checks we ship, like this one, which checks target_end_date really is the correct number of horizon weeks after reference_date.

Beyond that, hubs can write their own custom check functions in R and register them in the same way.

Admins can use the create_custom_check() function to create a function template, with the right structure and return classes, and they only write the logic.

📂 Accessing model output via hubData

Connect to Arrow dataset of forecast submissions

library(hubData)

hub_path <- s3_bucket(
  "cdcepi-flusight-forecast-hub"
)
hub_con <- connect_hub(hub_path)
hub_con

hub_connection
9 columns
reference_date: date32[day]
target: string
horizon: int32
target_end_date: date32[day]
location: string
output_type: string
output_type_id: string
value: double
model_id: string

Query and collect data

# Filter for one model and forecast date using dplyr
library(dplyr)
hub_con |>
  filter(
    model_id == "CADPH-FluCAT_Ensemble",
    target_end_date == "2023-10-28"
  ) |>
  collect_hub()

# A tibble: 92 × 9
  model_id    reference_date target horizon target_end_date location output_type
* <chr>       <date>         <chr>    <int> <date>          <chr>    <chr>      
1 CADPH-FluC… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
2 CADPH-FluC… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
3 CADPH-FluC… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
4 CADPH-FluC… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
5 CADPH-FluC… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
# ℹ 87 more rows
# ℹ 2 more variables: output_type_id <chr>, value <dbl>

See more in Accessing data vignette.

Python analogue hub-data also available.

Let’s talk about accessing data now.

We’ve designed the format so that, while teams look after their own submission files, model outputs can be accessed as a whole as a queryable Arrow dataset.

Here we connect to the FluSight hub’s S3 mirror with hubData. At this point we can interrogate the hub’s schema.

We can then filter and collect data using standard dplyr verbs.

In this example we filter for model output from a single model for a specific target end date before collecting data into our local R session.

Note that the same functionality is available in Python via hubdata and both can also be used to access data from a local clone.

🌐 Ensembling with hubEnsembles

Combine models using simple or weighted rules

forecast_df <- hub_con |>
  filter(
    model_id %in%
      c(
        "CADPH-FluCAT_Ensemble",
        "CEPH-Rtrend_fluH",
        "CFA_Pyrenew-Pyrenew_HE_Flu"
      ),
    output_type == "quantile"
  ) |>
  collect_hub()


hubEnsembles::simple_ensemble(
  forecast_df,
  agg_fun = median,
  model_id = "simple-ensemble-median"
)

# A tibble: 492,476 × 9
  model_id    reference_date target horizon target_end_date location output_type
* <chr>       <date>         <chr>    <int> <date>          <chr>    <chr>      
1 simple-ens… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
2 simple-ens… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
3 simple-ens… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
4 simple-ens… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
5 simple-ens… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
# ℹ 492,471 more rows
# ℹ 2 more variables: output_type_id <chr>, value <dbl>

Next ensembling.

Ensembling is central to the hub idea: combining multiple model predictions usually gives more robust forecasts than any single model.

The package hubEnsembles provides a simple interface for building ensembles from hubverse-formatted data.

In this example we’ve:

• downloaded quantile model output data from 3 models.
• And from that we can create a simple ensemble, here using median as the aggregation function.

📈 Dashboard - forecasts

The last feature I’d like to demo is our dashboards:

This is the FluSight dashboard, forecast page.

It lets users explore predictions interactively: - different outcomes, - forecast dates, - models can be compared side by side, - and against observed data.

You can also get more info on individual models by clicking a model name which opens its hub model metadata file.

🩺 Dashboard - model evaluations

Evaluates forecasts against target (observed) data.

• Table
• Plot

Dashboards also support an evaluation tab, which enables the comparison of model performance using standard metrics.

It’s powered by the hubEvals package, which scores model output against the target data stored in the hub.

It supports scoring rules such as: - WIS (Weighted Interval Score), - MAE, and - coverage of the 50% and 95% prediction intervals.

Each is also available relative to the baseline model, and on transformed scales.

Every metric has a definition in the sidebar, so it’s interpretable by someone who isn’t a forecasting specialist.

(switch tab) The same scores can be viewed as a plot rather than a table.

This is a heatmap of relative WIS, disaggregated by location, so you can see at a glance not just which models did well overall, but where they did better.

Evaluations are computed via CI during the dashboard build, straight from the hub’s model output and target data.

Closing remarks

Let me finish by stepping back: who is actually using this, what have we learned building it.

🪩 Who’s using it

34 hubs across 15 organizations, on five continents

736 models · 4.6 billion rows of model output (Sep 2026)

Five kinds of hub:

• 🌍 Community: open, public repos
• 🔒 Collaborative, private: e.g. Paraguay, Australia–Aotearoa
• 🧪 Local / model development: a hub on your own laptop
• 🎓 Training: teaching hubs
• 🗄️ Archival: retired hubs, kept queryable

hubverse.io/community/hubs.html

We’ve been really happy to see the breadth of adoption of the framework:

Firstly you can find a list of registered hubs on our website along with a bunch of useful metadata for each.

Overall we know of 34 hubs across 15 organisations, 736 models, with about 4.6 billion rows of public model output submitted! Some examples include:

• US CDC hubs which include FluSight, COVID-19 and RSV.
• The European CDC runs RespiCast and RespiCompass hubs.
• There are hubs in Canada, Australia and New Zealand, Paraguay and China.

Some hubs might be collaborative but private

A hub doesn’t have to be collaborative.

• Developing a model, you already need somewhere to keep many versions of many forecasts in one format so you can score them against each other.
• That’s a hub, on a laptop, and it leaves you ready to join a community one.

Finally there are training hubs, as well as archival hubs.

The team has invested time and effort into retro-fitting historical hubs into the hubverse standard, including:

• FluSight all the way back to 2015 and
• COVID 19 era ad hoc hubs.

💡 Lessons & wider relevance

• 📐 A good standard is leverage: define the interface once, reuse everything downstream
• ✅ Standards + automation reduce friction
• ⚙️ Configuration over code: versioned schemas let the standard evolve without breaking hubs
• 🧰 Open source keeps it free & accessible
• 🌍 Standardised, open data fuels downstream use: teaching, reproducible research, and teams’ own model development

Built on hubverse data

SISMID

Short course teaching real-world outbreak forecast evaluation, worked directly on hub data

RespiLens

A responsive web app to visualise US respiratory disease forecasts, built for state health departments and the public. ACCIDDA, UNC Chapel Hill

MicroHub workshop

Hands-on workshop: run forecast models locally in a Docker app, then stand up your own hubverse hub on GitHub. Northern Arizona University

To wrap up, I’ll end with some lessons I think generalise well beyond epidemiology:

• A good standard is leverage. We spent a lot of effort on the model output format and the config schema which set the foundation for all the downstream functionality.

• Standards plus automation reduce friction and lower the barrier to adoption.

• Configuration over code. Every hub was once a fork. Now it’s a JSON file validated against a versioned schema, which is what lets us evolve the ecosystem without breaking anyone.

Open-source tooling makes this freely accessible, which matters in fast-moving or resource-constrained public health contexts.

And standardised, open data opens doors. Three examples, all built by other people on top of hub data.

• SISMID, a short course teaching real-world forecast evaluation, using hub data.

• RespiLens, out of ACCIDDA at UNC, a visualisation web app for exploring US respiratory forecasts, aimed at state health departments and the public. They rely on the hubverse to standardise and consolidate the forecast data.

And the MicroHub workshop, from Northern Arizona University, gets public health teams forecasting on their own surveillance data without touching model code, and the outputs land in hub format, in a hub they build on GitHub.

None of these are our projects. They exist because the data and the format are open.

🙏 Thank you!

• https://hubverse.io
• https://docs.hubverse.io
• hubverse-org
• Install the packages: https://hubverse-org.r-universe.dev
• info@r-rse.eu

Members of hubverse team at the 2024 work retreat, UMass Amherst

Tip

Interested in getting involved? Check out our Getting Involved page!

📄 Read more

Consortium of Infectious Disease Modeling Hubs et al. (2026)
A software platform for collaborative infectious disease modelling
Nature Health

https://doi.org/10.1038/s44360-026-00145-7

Open-access preprint:
https://doi.org/10.1101/2025.10.03.25337284

Thanks so much for listening!

I leave you with some useful links, and the paper about the hubverse, which all have more detail. There’s an open-access preprint if you hit a paywall.

On installing the packages: everything is on our R-universe, which you can use as a repository directly. We’re in the process of getting the packages onto CRAN.

If this is of interest and you’d like to get involved, we have a newsletter, monthly community meetings and bi-weekly dev meetings.

I’m happy to chat about anything hubverse!