We add a schema and a `--validate-seq-db' command line option to validate
our sequence files database, now that it is in YAML format.
We call the validation from `make check'.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Let's use YAML as much as we can as we can better validate it and extend
it.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

To prepare for adding more validation schemas, create a dedicated folder
for our schemas.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

IR 1.1 certifications should use the ir1 branch so that development can
continue on the main branch.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add the BBSR sequence file from the Security interface extension ACS
v21.10_SIE_REL1.0.

While at it, fix sequence files permissions.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

- Add a jsonschema describing requirements on the configuration file.
- Add a --validate-config and a --schema parser options to make use of the
  schema.
- Add a check for the EBBR.yaml configuration in `make check'.
- Document in the README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Re-classify warnings issued by the SCT regarding SetState() returning
unsupported as known ACS limitations and drop the known SCT limitation
category. Also, add a reference to the corresponding edk2-test bug entry.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Deal with plural in a better way, in a couple of additional non-debug
places. While at it, update the README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A specific case where SetState() return "unsupported" makes the SCT emit a
warning. As this is a valid return value, add a rule to force this warning
as "known sct limitation".

While at it, document this new "known sct limitation" category in the
README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Now that ACS-IR v1.0 is out, add its sequence files to the `contrib' folder
as well as to the `seq.db' database for identification.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Make our dependencies a bit clearer in the documentation and in the warning
messages we print from the parser.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Give an example command, to quickly see the help comments added by some
rules to help debugging.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Triaging results when testing for EBBR compliance is the main use case of
the parser; let's make it the default.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a database of the sequence files we know about with their sha256,
and we try to identify the input sequence file among those.

A `--seq-db' option is added to allow specifying the database.

Update documentation accordingly. While at it, fix spacing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

To avoid issues with the sequence file getting modified in the test report,
recommend using the copy of EBBR.seq that is stored in this repo.

Signed-off-by: Grant Likely <grant.likely@arm.com>

Let's classify the ACS-IR BETA 0 MemoryMap failure automatically.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Update the documentation accordingly. While at it, remove obsolete mention
of default behaviour.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Refine the documentation of the result.md and explain how to generate HTML
with pandoc.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to perform further processing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a checker for python scripts and a checker for shell scripts to the
`make check' target. Fix all issues reported by the checker in parser.py.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

It is convenient to have those sequence files at hand to analyse results
obtained with the different ACS-IR published versions.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Adds some simple instructions to the top of the README file so that
a users doesn't need to go searching for the instructions.

Signed-off-by: Grant Likely <grant.likely@arm.com>

Start to document the EBBR.yaml configuration file specificities.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We put the links definitions in the same section as the first use, to
facilitate reading the README.md directly.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This runs yamllint for the moment, but can be extended to do more in the
future.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to print a quick summary, after filtering. We document some
usage in the README.md.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is similar in its principle to the UNIX `uniq -c' and is handy for
triaging results, in conjunction with the `--fields' option.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Keeping only some fields is handy to generate summaries for triaging.

This does not apply to the markdown and config template formats, which
depend on certain fields being present.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Document how the `--filter' option can be somewhat (ab)used to apply
transformations to tests data.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

cat.sh and dropped entries have been verified to be fine on runs with more
than 10k tests.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Allow to filter tests data using a python expression. This complements the
existing key/value filtering mechanism.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Try to semi-automate the writing of yaml configuration files by generating
a template yaml configuration from tests data, which can be further edited
manually.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --config option to Parser.py, to use a configuration file.

This configuration file can contain a list of transformations to apply to
the test results. This will allow to mark tests as false positives or to
waive failures.

We describe the YAML configuration file format in details in the README.md.
We also add a sample configuration, to demonstrate usage of configuration
files with the already provided sample log and sequence files.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A skip test set is a test set, which was considered but had zero of its
tests run. This is not the same as a dropped test set, which we do not find
in the log at all.
Generate artificial test entries with the result set to SKIPPED to keep
track of the former.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Currently we read db1 and db2 from the log and sequence files respectively.
Then we compute a cross_check database with all the tests, which have run,
as well as a would_not_run database with all the tests sets, which have not
run.

Combine those databases into a unified cross_check database and get rid of
would_not_run. We create artificial test entries for the dropped tests
sets.
While at it, we keep the spurious run tests, if any, and mark them as such.

The benefits are that we do not need special handling for dropped tests
sets down the pipe anymore, and that our future configuration file
filtering rules will also apply to spurious and dropped tests.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a --sort command line option to Parser.py, to allow sorting the tests
data in a configurable manner before producing the outputs.

Also, document usage in the README.md and fix parser.py case.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Use argparse for command line arguments. This gives us the online help and
will give use more flexibility to add command line options.

We preserve the current behaviour with up to two positional arguments for
the ekl and seq input files, and up to two arguments for the key:value
search.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Specify language of the python bloc. This enables syntax coloring with
pandoc.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Remove spurious whitespaces at end of line in Parser.py and README.md.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a Makefile and a configuration file to automate the generation of
README.pdf with pandoc.

Also, update the .gitignore accordingly.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>