Skip to contents

Changelog

Source: NEWS.md

hubPredEvalsData 1.3.0

Bug Fixes

  • The number of predictions scored (n) in scores.csv now counts only forecasts that had a corresponding observation to score against, rather than every submitted prediction, correcting an overcount wherever a model submitted predictions for units with no observation. In the rare case where this count differs across a target’s output types, scores.csv reports a separate n_<output_type> column per output type in place of the single n (#19).
  • generate_eval_data() now works on hubs whose target is identified by target_id alone, with no target key task id variable (target_keys: null), as used by nowcast hubs. Previously these failed partway through the run with an error about converting NULL to a symbol (#87).

Performance

  • Evaluation data generation is faster. The hub connection is opened once and reused across targets and evaluation sets rather than reopened for every combination, and oracle output is subset to the target before scoring (#82).

hubPredEvalsData 1.2.0

New Features

  • generate_predevals_options() now pulls target_name and target_units from the hub’s tasks.json target_metadata into each targets entry, just after target_id. The dashboard uses target_name to label target menu items; target_units is passed through for future labelling of unit-valued scores (#21).
  • scores.csv rows are now sorted on the (model_id, by) key immediately before writing, so the file is byte-stable across runs regardless of arrow’s scan order. Scores are unchanged; the deterministic order means outputs can be diffed or md5-compared to confirm a no-change situation (#25).

Bug Fixes

  • get_and_save_scores() now merges the per-output-type score frames with a full_join instead of a left_join anchored on the first output type. Previously any model, or any (model_id, by) cell, scored only on a later output type was silently dropped, and which models survived depended on the arbitrary order metrics were listed in the config (#75).

hubPredEvalsData 1.1.1

Bug Fixes

  • generate_eval_data() no longer fails when a hub’s oracle output includes an as_of provenance column (versioned target-data). Oracle output holds a single value per observational unit, so the column is dropped before scoring rather than being passed through to hubEvals::score_model_out(), which rejects columns outside the task-id keys plus oracle_value (#70).

hubPredEvalsData 1.1.0

New Features

  • Added optional scale-transformation support to predevals-config.yml via schema version v1.1.0 (#39):
    • transform_defaults — top-level default transform applied to all transformable targets.
    • targets[*].transform — per-target transform that overrides transform_defaults, or the literal false to opt a target out.
    • Supported transform functions: log_shift, sqrt, log1p, log, log10, log2.
  • Configured transforms are now applied during scoring (#40).
  • scores.csv is now emitted in wide format, with transformed-scale metrics as <metric>__<label>-suffixed columns (e.g. wis__log). Setting append: false emits only the suffixed columns.
  • Transform-invariant metrics (interval_coverage_<n> and bias) are now always reported on a single, un-suffixed scale in scores.csv and predevals-options.json, even when a transform is configured. Both are unchanged by any strictly monotonic transform (interval containment and the sign of forecast-vs-observation error are rank-based), so the transformed-scale column was a duplicate of the natural-scale column. Matches the invariance set documented in hubverse-org/hubDocs#464 (#63).
  • Existing v1.0.1 configs continue to validate against v1.1.0 without changes.
  • Added generate_predevals_options(), which assembles the contents of the predevals-options.json file used to initialise the predevals dashboard. It returns the validated config with each target’s metrics expanded to the columns present in scores.csv (relative-skill metrics, plus transformed-scale <metric>__<label> metrics when a transform applies) and a resolved transform block attached (#41, closes #4).
  • generate_eval_data() now discovers oracle output from hub_path via hubData::connect_target_oracle_output() when oracle_output is not supplied; the argument remains supported for back-compat (#51).

Bug Fixes

  • generate_eval_data() no longer fails on ordinal pmf targets that request rps. The ordinal level order is now read from the hub’s tasks.json and forwarded to hubEvals::score_model_out() so scoringutils dispatches the data as ordinal (#48).
  • read_predevals_config() now warns when an ordinal-only pmf metric (e.g. rps) is requested against a pre-v4 tasks-schema (where output_type_id is split across required/optional), and errors if the hub’s pmf output_type_id$optional is non-empty (#48).

hubPredEvalsData 1.0.0

This is a breaking change release that adds support for hubs with multiple rounds.

Breaking Changes

  • Configuration files now require schema version 1.0.1 and a new rounds_idx property.

How to Update

  1. Update your config file’s schema_version to:

    schema_version: https://raw.githubusercontent.com/hubverse-org/hubPredEvalsData/main/inst/schema/v1.0.1/config_schema.json

  2. Add the rounds_idx property after schema_version, specifying which round to use (0-based index):

    rounds_idx: 0

New Features

  • Added rounds_idx configuration property to specify which round entry to use from the hub’s tasks.json file (#26).
  • Hubs with multiple rounds are now supported.

hubPredEvalsData 0.0.0.9000

  • Initial development version.