The Hubverse: Streamlining Collaborative Infectious Disease Modeling

US-RSE Conference 2025

Anna Krystalli

info@r-rse.eu

R-RSE SMPC

Consortium of Infectious Disease Modeling Hubs

7 October 2025

Background

Hi everyone,

My name is Anna Krystalli.

I used to be an RSE at the University of Sheffield but a few years ago, I returned back home to Greece and set up my own RSE company.

Today I’m going to talk to you about the hubverse, a project I’ve been involved in since 2023.

And I’ll start with some motivating background on the project

❌ The problem

Infectious disease modeling has scaled rapidly…

• But the landscape is fragmented:
  • Inconsistent formats
  • Redundant or conflicting forecasts
  • Lack of coordination between modelers and stakeholders

“Comparing the accuracy of forecasting applications is difficult because forecasting methods, forecast outcomes, and reported validation metrics varied widely.”

• Chretien et al., PLOS ONE, 2014

Over the past two decades, and especially since the COVID-19 pandemic, infectious disease modeling has become a central tool in public health decision-making.

We’ve seen an explosion of forecasting efforts.

This growth is encouraging and reflects increased computational capacity, improved data availability, and global interest in predictive analytics for public health.

However, it has also come with growing pains.

• Modeling efforts often develop independently: each group uses its own data structure for model outputs, metadata, and evaluation, making integration and comparison difficult.

• Modelers and decision-makers don’t always communicate clearly. model outputs may not match what public health needs.

  This results in significant fragmentation which limits the collective utility of modeling efforts.

✨ The promise of modeling hubs

Modeling hubs coordinate collaborative forecasting:

• Provide centralised location for effort coordination

• Define data standards and modeling targets

• Improve transparency and comparability

• Aggregate forecasts enabling ensembles

• Facilitate timely public health decision-making

  
  

“Collaborative Hubs: Making the Most of Predictive Epidemic Modeling”, American Journal of Public Health Reich, et al. 2022

Modeling hubs which provide centralized infrastructure for coordinating the modeling efforts of multiple groups have been proposed as a solution:

• Firstly Hubs bring structure and a standardised central location for collecting model outputs.

• They establish a shared modeling protocol: what outcomes to forecast, for which locations and time horizons, in what format.

• By enforcing standards and submission rules, hubs ensure that forecasts can be examined and compared.

• Standardization also allows for ensemble models that combine many forecasts, often outperforming individual models, a major asset during uncertain conditions.

• Hubs make it easier for stakeholders, from local agencies to national institutions, to access timely, interpretable forecasts.

Modeling hubs transform isolated efforts into coordinated forecasting ecosystems.

They aren’t just data repositories, they’re collaborative infrastructures.

🕰️ Project origins

• Pre-COVID: Forecasting code base existed for CDC influenza hubs
• During COVID: That code was reused for new COVID-19 hubs + demand internationally (e.g. Europe) for similar setups
• ❗ Problem: Each hub required manual editing of source code

➡️ Need for generalisation, modularity, and configurability

Figure credits: Alex Vespignani and Nicole Samay

Before COVID, the US CDC had already established a foundation for collaborative infectious disease forecasting through its FluSight initiative, focused on influenza.

This resulted in an initial working codebase to support such efforts.

During the pandemic, interest in modeling hubs exploded and this infrastructure was called on for repurposing:

• First by the US CDC COVID-19 Forecast Hub

• Then interest grew to scenario modeling hubs, as well as internationally.

‼ BUT to set up each new hub, manual changes to the source code was required, obviously not ideal.

This is what motivated the development of a more configurable, more generalized and more modular system to support new infectious disease modeling hubs.

🌐 Enter the hubverse

An open-source software ecosystem to power modeling hubs:

• GitHub repositories for centralising hub activity
• Data standards for infectious disease modeling data
• Schema-driven configuration for modeling tasks + hub setup
• Modular tools for validation, access, evaluation, ensembling, communication and hub administration
• Supports full lifecycle: from hub set up, data submission to decision-making

… an what lead to the creation of the hubverse

> which is an open-source software ecosystem designed to support the full lifecycle of infectious disease modeling hubs.

It brings together:

• Centralised version controlled location for hub activities through GitHub repositories

• Standardised data formats for infectious disease modeling.

• Schema-driven configuration, for defining a hub’s modeling tasks, submission schedule, and other expectations.

• A suite of tools for hub administration, data validation, access, evaluation, ensembling, visualisation and communication.

Hubverse overview

☑️ Standardised Data

Modeling hubs are built around a shared data standard:

• Modeling task definition: targets (response variables), standard predictors, output types (e.g. mean, quantiles)
• Structured hub layout: consistent file system for organizing submissions
• Standard model output format: for file content and naming

✅ Enables comparability, validation, and streamlined data access

At its core, the hubverse represents a shared data standard.
Everything else builds on that foundation.

This covers how modeling task are defined:

• what’s being predicted,

• which predictors are to be used,

• and what output types are accepted.

Model outputs are submitted within a structured hub layout, so files are always stored in predictable standard locations.

Finally, the model output files themselves follow a consistent format and naming convention.

Together, these standards make model outputs directly comparable and easy to combine in downstream analysis.

⚙️ Config-driven hub setup

Hub administrators configure hubs using structured JSON config files:

• admin.json: hub-level metadata.
• task.json: modeling task specification:
  • Task IDs: Targets (response), horizons, locations (predictors) etc.
  • Output types: accepted model outputs e.g. mean, median, quantiles, cdf, pmf, samples.
• Configs are validated against a shared JSON schema

Another key feature of the hubverse is that hub setup is configuration-based.

Administrators define the hub’s goals and expectations through structured JSON config files:

• admin.json describes hub-level properties like hub name, maintainers, timezone, allowed data formats etc.
  
• task.json defines the modeling tasks:
  • what targets (response variable),
  • what predictor variables (other task IDs),
  • and which output types are accepted for a given modeling task.

Config files are validated against a JSON schema and are used for validating submissions.

The result is that new hubs can be spun up quickly and modeling tasks are clearly and unambiguously defined from the start.

This enables clear communication of tasks by administrators ensuring modeling efforts are policy relevant.

📦 The R (and friends) package stack

The hubverse package ecosystem is organized by role. Each tool is designed to support a particular group of users in the hub workflow.

Hub roles

• 🛠️ Hub administrators
  
• 🔬 Modelers
  
• 📊 Analysts
  
• 🏛️ Policy makers
  

Tools & packages

• hubAdmin : config creation + validation 🛠️

• hubValidations : submission checks (structure, schema, content) 🔬 🛠️

• hubData (R) / hubdata (Python) : access multi-file model output via Arrow 🛠️ 🔬 📊 🏛️

• hubEvals : compute evaluation metrics 🛠️ 📊 🏛️

• hubEnsembles : build weighted/unweighted ensembles 🛠 📊 🏛️

• hubVis : visualise model outputs 🛠️ 📊 🏛️

Let me now introduce the core hubverse packages that support our modeling hub infrastructure.

We’ve intentionally organized the ecosystem by user role. That’s because a modeling hub isn’t a single-user platform, it involves multiple roles, each with distinct responsibilities, technical skills, and needs.

On the left, we show the primary roles:

• Hub administrators are responsible for setting up and maintaining the hub’s infrastructure and configuration.
• Modelers focus on submitting model outputs via pull requests.
• Analysts need to explore, combine, and evaluate forecast data, often across many models and time points.
• Policy makers or Public health officials typically need high-level summaries, dashboards, and access to curated insights, not raw files.

On the right, we map each of these user groups to the packages that support their work:

• hubAdmin helps administrators create and validate the structure of their configuration files.
• hubValidations is the gatekeeper, enforcing the standards set by the admins by validating every PR, ensuring files are named, structured, and formatted correctly. Can also be used by modelers prior to submitting.
• hubData (available in both R and Python) gives analysts and modelers an easy way to directly access multi-file model outputs directly as Arrow datasets.
• hubEvals supports model evaluation.
• hubEnsembles provides functions for combining model outputs into ensembles.
• hubVis supports model output and ensemble visualisations.

📊 Dashboards & communication

• Built with Quarto so easily customisable via Quarto configuration
• Deployed as a fully static site , no backend required
• Powered by JSON data prepared via GitHub workflows
• Interactive UI built with client-side JavaScript (fast!)
• New instances can be set up by copying/configuring the hub-dashboard-template

Dashboards are crucial for communicating model outputs to a broader audience, including policy makers, public health officials, and even the general public so we put effort into developing a plug and play dashboard template.

• Our template is built in Quarto, so it’s easy to customise and extend.

• What makes it powerful is that it’s fully static, no backend needed.
  JSON data are generated from hub data by GitHub workflows, and the JavaScript frontend fetches this data to create a dynamic, interactive experience for the user.

This design makes it fast, easy to host (e.g. on GitHub Pages), and simple to replicate.

Hub admins can launch new dashboards quickly by copying and configuring the hub-dashboard-template.

☁️ Cloud hub storage and access

• Hubs mirrored to public AWS S3 buckets
• Multi-file data can be opened as Arrow datasets
• Enables query-able data access via R 📦 hubData and Python 📦 hubdata.

As a bonus, we’ve developed infrastructure to mirror validated hub data to AWS S3 buckets, making it publicly accessible as Arrow datasets.

This enables downstream users, especially analysts, to query the data without needing to clone the full repository.

They can access it via the R or python equivalent package hubdata.

This cloud mirroring service is something the hubverse currently provides.
To enable it for a new hub:

• The hub administration needs to contact us so we can provision a bucket.

• They need to add a bit of lightweight configuration to their admin.json file to activate it and install the relevant action workflow.

🔁 GitHub workflows

We automate everything we can:

• ✅ PR-level model output validation
• ✅ Hub configuration validation
• ☁️ AWS Cloud hub data synching
• 📊 dashboard data regeneration and model evaludation with each update

All hubverse actions stored in the hubverse-actions repo and can be installed with hubCI::use_hub_github_action()

Automation is a cornerstone of the hubverse.
We use GitHub Actions to handle hub operations.

This includes:

• Validating model output submissions and target data at the pull request level,

• Validating hub configuration files,

• Synchronising validated model output to cloud storage,

• Generating dashboard data + computing model evaluation metrics.

All our workflows are stored in the hubverse-actions repo.
They’re easy to adopt, hub administrators can install them using the hubCI::use_hub_github_action() helper.

This approach ensures consistency, and relatively low overhead for administrators.

🪩 List of adopting hubs

https://hubverse.io/community/hubs.html

Now that we’ve seen the infrastructure, let’s look at adoption.

This page on our site showcases hubs using the hubverse stack, from long-standing hubs like the Flusight Forecast Hub to newer efforts like the COVID-19 Variant Nowcast or West Nile Virus hubs.

Each hub entry includes structured metadata: like a hub description, whether data are publicly available, and links to the relevant GitHub repos, dashboards and cloud storage.

This kind of shared metadata and visibility wouldn’t be possible without the consistency enforced by the hubverse framework.

It also serves as a community resource, a growing catalog of open modeling hubs that are easy to discover and learn from.

🦠 Real-world example: CDC FluSight Hub

Now let’s take a look at some this infrastrcuture in action by focusing on the CDC Flusight Hub

Real-world example: CDC FluSight Hub

https://github.com/cdcepi/FluSight-forecast-hub

• Used by US CDC to monitor influenza severity
• Weekly forecasts from 40 teams across 70 different models.
• Hosted on GitHub + S3 cloud mirror.
• Managed using full hubverse stack since 2023/2024 season.

The CDC FluSight Hub is a flagship example of the hubverse stack in action.

It’s used by the US CDC to monitor influenza severity through weekly model outputs, submitted by over 40 teams, spanning 70 different models.

The hub is fully managed using the hubverse software, workflows and S3 cloud mirroring since the 2023/24 season.

If we peek into the repository structure, you’ll see the key directories that define a hub:

• hub-config: holds the JSON config files.
• model-output: where teams submit their model output files
• target-data: stores the observed data used to evaluate forecasts, such as reported influenza-like illness rates.

📁 File structure: model output (CDC FluSight)

Model outputs committed by teams to versioned directories > one directory per model > one file per modeling round.

This slide breaks down a single model output file and it’s location in the CDC FluSight hub.

On the top, we see the file lives under the model-output/<model_id> directory.

Files are named using a round ID and model ID, e.g., 2023-10-14-CADPH-FluCAT_Ensemble.csv. Each file contains all predictions for that round by that model.

Inside the file:

- The Task ID columns, like target, horizon, and location store information about individual predictions.

- The Output Type columns describe the prediction format (e.g., quantiles).

- And the value column holds the actual predicted values.

This file structure (task ids, output type, values) is fully standardised across all hubs.

✅ Model output validation with hubValidations

Model outputs submitted through PRs and validated through GitHub Actions

Every model output is submitted via GitHub pull requests, and each submission is validated using the hubValidations package.

On the left, we see the latest submissions from different teams for a specific forecast round.

On the right is an example validation log from GitHub Actions. Here, hubValidations checks that the submission:

• Passes file level checks e.g.:

  • Follows the required file structure,

  • Is named and placed correctly,

• And passes content-level checks:

  • valid task ID and output type value combinations,

  • required tasks submitted,

  • valid output types - output type specific checks (e.g quantile crossing)

  • Custom checks also supported

These automated checks catch errors early and keep hub data clean and trustworthy.

📂 Accessing model output via hubData

Connect to Arrow dataset of forecast submissions

library(hubData)

hub_path <- s3_bucket(
  "cdcepi-flusight-forecast-hub"
)
hub_con <- connect_hub(
  hub_path,
  skip_checks = TRUE
)
hub_con

hub_connection
9 columns
reference_date: date32[day]
target: string
horizon: int32
target_end_date: date32[day]
location: string
output_type: string
output_type_id: string
value: double
model_id: string

Query and collect data

# Filter for one model and forecast date using dplyr
library(dplyr)
hub_con |>
  filter(
    model_id == "CADPH-FluCAT_Ensemble",
    target_end_date == "2023-10-28"
  ) |>
  collect_hub()

# A tibble: 92 × 9
   model_id   reference_date target horizon target_end_date location output_type
 * <chr>      <date>         <chr>    <int> <date>          <chr>    <chr>      
 1 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
 2 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
 3 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
 4 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
 5 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
 6 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
 7 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
 8 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
 9 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
10 CADPH-Flu… 2023-10-14     wk in…       2 2023-10-28      06       quantile   
# ℹ 82 more rows
# ℹ 2 more variables: output_type_id <chr>, value <dbl>

See more in Accessing data vignette.

Python analogue hub-data also available.

We’ve designed the hubverse format so that, while teams look after their own submission files, model outputs can be accessed as a whole as a queryable Arrow dataset.

Here we show how to connect to the FluSight hub’s S3 mirror using hubData.

We can then filter and collect data using standard dplyr verbs.

This makes it easy for analysts, modelers, or public health teams to extract relevant subsets of data they need.

The same functionality is available in Python via the hub-data package.

🌐 Ensembling with hubEnsembles

Combine models using simple or weighted rules

forecast_df <- hub_con |>
  filter(
    model_id %in%
      c(
        "CADPH-FluCAT_Ensemble",
        "CEPH-Rtrend_fluH",
        "CFA_Pyrenew-Pyrenew_HE_Flu"
      ),
    output_type == "quantile"
  ) |>
  collect_hub()


hubEnsembles::simple_ensemble(
  forecast_df, 
  agg_fun = median,
  model_id = "simple-ensemble-median"
)

# A tibble: 282,716 × 9
   model_id   reference_date target horizon target_end_date location output_type
 * <chr>      <date>         <chr>    <int> <date>          <chr>    <chr>      
 1 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
 2 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
 3 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
 4 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
 5 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
 6 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
 7 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
 8 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
 9 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
10 simple-en… 2023-10-14     wk in…      -1 2023-10-07      01       quantile   
# ℹ 282,706 more rows
# ℹ 2 more variables: output_type_id <chr>, value <dbl>

Ensembling is central to the hub idea as it helps improve prediction robustness by combining multiple model predictions into a single output.

hubEnsembles provides a simple interface to build various types of ensembles from hubverse formatted data.

This example shows how to pull data from a few different models and apply a simple ensemble method.

📈 Dashboard - forecasts

This is the FluSight dashboard and specifically the forecast page.

It enables users to explore predictions interactively: you can choose outcomes, forecast dates and compare multiple models visually.

Crucially, the dashboard also shows the target (observed) data alongside forecasts, which helps contextualise model behaviour and monitor real-world performance.

Clicking on any model name in the left-hand panel opens that model’s metadata page , which includes details about the model, contributors, assumptions etc.

🩺 Dashboard - model evaluations

Evaluates forecasts against target (observed) data.

This is the evaluation tab of the FluSight dashboard, which helps users compare model performance using standard metrics.

It’s powered by the hubEvals package, which computes evaluation scores by comparing model outputs to the target data, also stored in the hub.

The evaluation supports scoring rules such as:

• WIS (Weighted Interval Score)

• MAE (Mean Absolute Error)

• Coverage of 50% and 95% prediction intervals

Evaluations are performed during the dashboard build and are summarised into an interactive table, allowing for quick comparison across models.

💡 Lessons & wider relevance

• ✅ Standards + automation reduce friction
  
• 🧰 Open source keeps it free & accessible
  
• 🏥 Collaborative infrastructure empowers public health
  
• 🌍 Standardised, open data fuels downstream use cases like training, education, and reproducible research

https://nfidd.github.io/sismid/sessions/real-world-forecasts.html

To wrap up:

• Standards and automation reduce friction at every level -submitting, validating, accessing and comparing model outputs becomes much easier and more reliable.
• Open-source tooling makes all this infrastructure freely accessible, which is especially valuable in fast-moving or resource-constrained public health contexts.
• Collaborative infrastructure like this fosters shared understanding and coordinated and efficient efforts.
• But beyond that, making the data openly available in a standardised format opens doors to wider impact. For example, hubverse data are already being used in teaching, like this SISMID course on real-world outbreak forecast evaluation, which builds directly on hubverse data and tools.

We hope this inspires other communities to adopt or adapt similar workflows.

🙏 Thank you!

• https://hubverse.io
• https://docs.hubverse.io
• hubverse-org
• info@r-rse.eu

Tip

Interested in getting involved in the community? Check out our Getting Involved page!

Thanks so much! I’ll leave you with some useful links, including information on how to get involved in the community like out newsletter, monthly community meetings or bi-weekly dev meetings.

I’m happy to chat about anything hubverse!