We borrow the validate.py script from the SystemReady scripts
repository[1] to validate our YAML files against schemas.

We deprecate the `--validate-config' and the `--validate-seq-db'
command line options of the parser.

We adapt the `make check' commands.

[1]: https://gitlab.arm.com/systemready/systemready-scripts



Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a shell script to test functionalities of the parser and run it from
`make check'.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a configuration to process SCT results from the Security Interface
Extension ACS.

The rules in this new configuration can override tests results with a new
`KNOWN RPMB LIMITATION' result.

Add the necessary enums to our schema and adjust the sequence files
database to use the new configuration automatically when the SIE ACS is
detected.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

For the moment we have only the EBBR.yaml configuration. Command line does
still take precedence. No functional change.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a schema and a `--validate-seq-db' command line option to validate
our sequence files database, now that it is in YAML format.
We call the validation from `make check'.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Let's use YAML as much as we can as we can better validate it and extend
it.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

To prepare for adding more validation schemas, create a dedicated folder
for our schemas.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

IR 1.1 certifications should use the ir1 branch so that development can
continue on the main branch.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add the BBSR sequence file from the Security interface extension ACS
v21.10_SIE_REL1.0.

While at it, fix sequence files permissions.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

- Add a jsonschema describing requirements on the configuration file.
- Add a --validate-config and a --schema parser options to make use of the
  schema.
- Add a check for the EBBR.yaml configuration in `make check'.
- Document in the README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Re-classify warnings issued by the SCT regarding SetState() returning
unsupported as known ACS limitations and drop the known SCT limitation
category. Also, add a reference to the corresponding edk2-test bug entry.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Deal with plural in a better way, in a couple of additional non-debug
places. While at it, update the README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A specific case where SetState() return "unsupported" makes the SCT emit a
warning. As this is a valid return value, add a rule to force this warning
as "known sct limitation".

While at it, document this new "known sct limitation" category in the
README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Now that ACS-IR v1.0 is out, add its sequence files to the `contrib' folder
as well as to the `seq.db' database for identification.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Make our dependencies a bit clearer in the documentation and in the warning
messages we print from the parser.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Give an example command, to quickly see the help comments added by some
rules to help debugging.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Triaging results when testing for EBBR compliance is the main use case of
the parser; let's make it the default.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a database of the sequence files we know about with their sha256,
and we try to identify the input sequence file among those.

A `--seq-db' option is added to allow specifying the database.

Update documentation accordingly. While at it, fix spacing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

To avoid issues with the sequence file getting modified in the test report,
recommend using the copy of EBBR.seq that is stored in this repo.

Signed-off-by: Grant Likely <grant.likely@arm.com>

Let's classify the ACS-IR BETA 0 MemoryMap failure automatically.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Update the documentation accordingly. While at it, remove obsolete mention
of default behaviour.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Refine the documentation of the result.md and explain how to generate HTML
with pandoc.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to perform further processing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a checker for python scripts and a checker for shell scripts to the
`make check' target. Fix all issues reported by the checker in parser.py.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

It is convenient to have those sequence files at hand to analyse results
obtained with the different ACS-IR published versions.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Adds some simple instructions to the top of the README file so that
a users doesn't need to go searching for the instructions.

Signed-off-by: Grant Likely <grant.likely@arm.com>

Start to document the EBBR.yaml configuration file specificities.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We put the links definitions in the same section as the first use, to
facilitate reading the README.md directly.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This runs yamllint for the moment, but can be extended to do more in the
future.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to print a quick summary, after filtering. We document some
usage in the README.md.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is similar in its principle to the UNIX `uniq -c' and is handy for
triaging results, in conjunction with the `--fields' option.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Keeping only some fields is handy to generate summaries for triaging.

This does not apply to the markdown and config template formats, which
depend on certain fields being present.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Document how the `--filter' option can be somewhat (ab)used to apply
transformations to tests data.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

cat.sh and dropped entries have been verified to be fine on runs with more
than 10k tests.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Allow to filter tests data using a python expression. This complements the
existing key/value filtering mechanism.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Try to semi-automate the writing of yaml configuration files by generating
a template yaml configuration from tests data, which can be further edited
manually.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --config option to Parser.py, to use a configuration file.

This configuration file can contain a list of transformations to apply to
the test results. This will allow to mark tests as false positives or to
waive failures.

We describe the YAML configuration file format in details in the README.md.
We also add a sample configuration, to demonstrate usage of configuration
files with the already provided sample log and sequence files.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A skip test set is a test set, which was considered but had zero of its
tests run. This is not the same as a dropped test set, which we do not find
in the log at all.
Generate artificial test entries with the result set to SKIPPED to keep
track of the former.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Currently we read db1 and db2 from the log and sequence files respectively.
Then we compute a cross_check database with all the tests, which have run,
as well as a would_not_run database with all the tests sets, which have not
run.

Combine those databases into a unified cross_check database and get rid of
would_not_run. We create artificial test entries for the dropped tests
sets.
While at it, we keep the spurious run tests, if any, and mark them as such.

The benefits are that we do not need special handling for dropped tests
sets down the pipe anymore, and that our future configuration file
filtering rules will also apply to spurious and dropped tests.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>