LbAPCommon#

LbAPCommon.lint_all(data)[source]#

Placeholder for linting.

Parameters:

data –

…

Raises:

NotImplementedError – This function is not implemented.

LbAPCommon.parse_yaml(rendered_yaml)[source]#

Parse rendered YAML text.

Parameters:

rendered_yaml (str) – The rendered YAML jinja template.

Raises:

ValueError – raised if errors occurred during parsing.

Returns:

a tuple of the parsed configuration data2 (dict) and checks data (dict).

Return type:

tuple

LbAPCommon.render_yaml(raw_yaml)[source]#

Render a “raw” YAML jinja template.

Accepts LbAP yaml configuration jinja template and renders it into a full YAML configuration.

Parameters:

raw_yaml (str) – YAML jinja-template string

Raises:

ValueError – raised if jinja2 couldn’t render the raw_yaml string.

Returns:

a jinja-rendered YAML string.

Return type:

str

LbAPCommon.validate_yaml(jobs_data, checks_data, repo_root, prod_name)[source]#

Validate YAML configuration for anything that would definitely break a job or the production.

Other validations that aren’t essential for at least testing a job are done via the checks framework.

Parameters:

  • jobs_data (dict) – Parsed job configuration.

  • checks_data (dict) – Parsed checks configuration.

  • repo_root (str) – Repository location.

  • prod_name (str) – Production name.

Raises:

ValueError – Raised if there are showstopper issues in the parsed job configuration.

LbAPCommon.write_jsroot_compression_options(dynamic_dir)[source]#

Write options file to configure JSROOT-compatible compression on job output files.

Parameters:

dynamic_dir – Location to write the use-jsroot-compression.py options file.

Linters#

LbAPCommon.linting.lint_all(data)[source]#

Placeholder for linting.

Parameters:

data –

…

Raises:

NotImplementedError – This function is not implemented.

Options Parsing#

LbAPCommon.options_parsing.validate_options(json_file: str, ntuple_fn: str, job_name: str, prod_data: dict)[source]#

Validate YAML options.

Check existence of expected TTrees and ensure at least one DecayTree and MCDecayTree (for MC samples) exist(s) in output files.

Parameters:

  • json_file (str) – json_file listing the expected TTrees

  • ntuple_fn (str) – Local test output tuple to validate against

  • job_name (str) – Name of job to validate

  • prod_data (dict) – Entire production information from yaml parsing

Returns:

Errors if any expected TTrees are not found in job output,

warnings if at least one (MC)DecayTree isn’t found in job output

Return type:

tuple[list[str]]

Validators#

LbAPCommon.validators.count_log_messages(log_text)[source]#

Parse the log text for errors and advice.

Parameters:

log_text (str) – Contents of the log file.

Returns:

Tuple containing the number of WARNING/ERROR/FATAL level messages in the job log

Return type:

tuple[int]

LbAPCommon.validators.explain_log(log_text)[source]#

Splits a Gaudi job log by message level.

Parameters:

log_text (str) – Contents of the log file.

Raises:

NotImplementedError – when a check is not implemented

Returns:

A tuple of three dictionaries corresponding to (explanations, suggestion, error) where each is a dictionary of messages to a list of log file line numbers.

Return type:

tuple[dict[list[int]]]

LbAPCommon.validators.match_output_filenames(filenames, output_filetypes, application_name, step_index)[source]#

File the XML, log and output filenames from a test job.

Parameters:

  • filenames (list of str) – The filenames found in the test output

  • output_filetypes (list of str) – The output filetype(s) of the job

  • application_name (str) – Name of the application being ran (e.g. DaVinci)

  • step_index (int) – The index of this test within the job

Returns:

  • xml_summary_fn (str or None) – The filename of the XML summary

  • xml_bk_fn (str or None) – The filename of the Bookkeeping summary

  • log_fn (str or None) – The filename of the application log

  • output_fns (dict) – A mapping of output filetype to a filename or None

LbAPCommon.validators.parse_bookkeeping_xml(fp, expected_input_files)[source]#

Extract information from the bookkeeping XML.

Parameters:

  • fp (file-like) – File pointer of the XML summary

  • expected_input_files (list) – List of object describing the expected input files

Returns:

passed – True to the XML summary reports the job as passed

Return type:

bool

LbAPCommon.validators.status_from_xml_summary(fp)[source]#

Check if the job succeeded using the XML summary file.

Parameters:

fp (file-like) – _description_

Raises:

NotImplementedError – _description_

Returns:

whether the XML summary reports the job as passed

Return type:

bool

Checks#

LbAPCommon.checks.run_job_checks(jobs_data, job_name: str, checks_list: list[str], check_data, test_ntuple_path_list: list[str])[source]#

Run checks and return CheckResult objects.

Parameters:

  • jobs_data – Job configuration, usually loaded from a config file.

  • job_name (str) – Job name to run checks or validations against.

  • checks_data – Checks configuration, usually loaded from a config file.

  • checks_list (list[str]) – List of checks to run specified by their names.

  • test_ntuple_path_list (list[str]) – A list of paths to ntuple files to be used by each check.

Returns:

A dictionary of CheckResults detailing the result of each check.

Return type:

dict[CheckResult]