Skip to contents

Utilities for submitting teams

Functions to help teams create valid submission model output files.

Higher level validation functions

Functions combining appropriate check unit test used for higher level validations. All return hub_validations or target_validations S3 class objects.

validate_model_data()
Validate the contents of a submitted model data file
validate_model_file()
Valid file level properties of a submitted model output file.
validate_model_metadata()
Valid properties of a metadata file.
validate_pr()
Validate Pull Request
validate_submission()
Validate a submitted model data file.
validate_submission_time()
Validate a submitted model data file submission time.
validate_target_data()
Validate the contents of a submitted target data file.
validate_target_dataset()
Validate dataset level properties of a given target type
validate_target_file()
Validate file level properties of a target data file.
validate_target_pr()
Validate Target Data Pull Request
validate_target_submission()
Validate a submitted target data file.
check_for_errors()
Raise conditions stored in validation objects

Unit validation checks

Unit validation checks. All return hub_check S3 class objects.

check_config_hub_valid()
Check hub correctly configured
check_file_exists()
Check file exists at the file path specified
check_file_format()
Check file format is accepted by hub.
check_file_location()
Check file is being submitted to the correct folder
check_file_n()
Check number of files submitted per round does not exceed the allowed number of submissions per team.
check_file_name()
Check a model output file name can be correctly parsed.
check_file_read()
Check file can be read successfully
check_for_errors()
Raise conditions stored in validation objects
check_metadata_file_exists()
Check whether a metadata schema file exists
check_metadata_file_ext()
Check file is being submitted to the correct folder
check_metadata_file_location()
Check that the metadata file is being submitted to the correct folder
check_metadata_file_name()
Check whether the file name of a metadata file matches the model_id or combination of team_abbr and model_abbr specified within the metadata file
check_metadata_matches_schema()
Check whether a metadata file matches the schema provided by the hub
check_metadata_schema_exists()
Check whether a metadata schema file exists
check_submission_metadata_file_exists()
Check whether a metadata file for the given model exists
check_submission_time()
Checks submission is within the valid submission window for a given round.
check_target_dataset_exists()
Check target dataset can be detected for a given target type
check_target_dataset_file_ext_unique()
Check that all files of a given target type share a single unique file format
check_target_dataset_rows_unique()
Check target dataset rows are all unique
check_target_dataset_unique()
Check that a single unique target dataset exists for a given target type.
check_target_file_ext_valid()
Check that a target data file has a valid extension.
check_target_file_name()
Check that a hive-partitioned target data file name can be correctly parsed.
check_target_file_read()
Check target file can be read successfully
check_target_tbl_colnames()
Check that a target data file has the correct column names according to target type
check_target_tbl_coltypes()
Check that a target data file has the correct column types according to target type
check_target_tbl_oracle_value()
Check that oracle values in an oracle output target data file are valid
check_target_tbl_output_type_ids()
Check that output type ID values in a target data file are valid and complete
check_target_tbl_rows_unique()
Check target data rows are all unique
check_target_tbl_ts_targets()
Check that targets in a time-series target data file are valid
check_target_tbl_values()
Check that task ID columns in a target data file have valid task ID values
check_tbl_col_types()
Check model data column data types
check_tbl_colnames()
Check column names of model output data
check_tbl_derived_task_id_vals() superseded
Check derived task ID columns contain valid values
check_tbl_match_round_id()
Check model output data tbl round ID matches submission round ID.
check_tbl_rows_unique()
Check model data rows are all unique
check_tbl_spl_compound_taskid_set()
Check model output data tbl sample compound task id sets for each modeling task match or are coarser than the expected set defined in the config.
check_tbl_spl_compound_tid() superseded
Check model output data tbl samples contain single unique values for each compound task ID within individual samples
check_tbl_spl_mt_unique()
Check that individual sample output_type_ids do not span multiple model tasks
check_tbl_spl_n()
Check model output data tbl samples contain the appropriate number of samples for a given compound idx.
check_tbl_spl_non_compound_tid()
Check model output data tbl samples contain single unique combination of non-compound task ID values across all samples
check_tbl_spl_overlap()
Check model output data tbl samples that share a compound modeling task cover the same compound modeling tasks
check_tbl_unique_round_id()
Check model output data tbl contains a single unique round ID.
check_tbl_value_col()
Check output type values of model output data against config
check_tbl_value_col_ascending()
Check that quantile and cdf output type values of model output data are non-descending
check_tbl_value_col_sum1()
Check that pmf output type values of model output data sum to 1.
check_tbl_values()
Check model output data tbl contains valid value combinations
check_tbl_values_required()
Check all required task ID/output type/output type ID value combinations present in model data.
check_valid_round_id()
Check whether the round_id determined for the submission is valid
check_valid_round_id_col()
Check that any round_id_col name provided or extracted from the hub config is valid.

Optional unit validation checks

Optional unit validation checks. All return hub_check S3 class objects and need to be deployed via validations.yml file.

opt_check_metadata_team_max_model_n()
Check that submitting team does not exceed maximum number of allowed models per team
opt_check_tbl_col_timediff()
Check time difference between values in two date columns equal a defined period.
opt_check_tbl_counts_lt_popn()
Check that predicted values per location are less than total location population.
opt_check_tbl_horizon_timediff()
Check time difference between values in two date columns equals a defined time period defined by values in a horizon column

Custom unit validation checks

Functions to support creation and deployment of custom unit validation checks

create_custom_check()
Create a custom validation check function template file.

Lower level functions

Lower level functions for assisting users in developing custom unit checks.

capture_check_cnd()
Capture a condition of the result of validation check.
capture_check_info()
Capture a simple info message condition
capture_exec_error()
Capture an execution error condition
capture_exec_warning()
Capture an execution warning condition
capture_validation_warning()
Capture a validation warning condition
is_success() is_failure() is_error() is_info() not_pass() is_exec_error() is_exec_warn() is_any_error()
Get status of a hub check
parse_file_name()
Parse model output file metadata from file name
read_model_out_file()
Read a model output file
read_target_file()
Read a single target data file
try_check()
Wrap check expression in try to capture check execution errors
expand_model_out_grid()
Create expanded grid of valid task ID and output type value combinations
get_tbl_compound_taskid_set()
Detect the compound_taskid_set for a tbl for each modeling task in a given round.
match_tbl_to_model_task()
Match model output data to their model tasks in config_tasks.
get_target_task_id()
Get Unique Target Task ID

<hub_validations> methods

Methods for <hub_validations> S3 objects and subclasses.

print(<hub_validations>)
Print results of validate_...() function as a bullet list
print(<hub_validations_collection>)
Print results of multi-file validation as a hierarchical bullet list
combine()
Concatenate validation objects
new_hub_validations() as_hub_validations()
Create new or convert list to hub_validations S3 class object
new_hub_validations_collection() as_hub_validations_collection()
Create new or convert list to hub_validations_collection S3 class object
new_target_validations() as_target_validations()
Create new or convert list to target_validations S3 class object
new_target_validations_collection() as_target_validations_collection()
Create new or convert list to target_validations_collection S3 class object