This is a first attempt at static checking our python scripts, to prepare
adding more type annotations in the future.

- Rework bits to make mypy happy.
- Add a mypy config file to deal with the junit.xml module.
- Check all scripts with mypy in `make check'.
- Mention mypy in the README and in `make help'.
- Add mypy install to the CI files.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Mention that gitlab.arm.com is where the latest version is.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a .gitlab-ci.yml file to run `make check' as CI.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The SIE errors, which we originally classified as "known rpmb limitations"
can happen not only with an eMMC but in the more general case with a small
storage for UEFI variables. This is typically the case with OP-TEE.
Let's re-classify those errors as "known size limitations". This is just
renaming at this stage.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We borrow the validate.py script from the SystemReady scripts
repository[1] to validate our YAML files against schemas.

We deprecate the `--validate-config' and the `--validate-seq-db'
command line options of the parser.

We adapt the `make check' commands.

[1]: https://gitlab.arm.com/systemready/systemready-scripts



Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a shell script to test functionalities of the parser and run it from
`make check'.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a configuration to process SCT results from the Security Interface
Extension ACS.

The rules in this new configuration can override tests results with a new
`KNOWN RPMB LIMITATION' result.

Add the necessary enums to our schema and adjust the sequence files
database to use the new configuration automatically when the SIE ACS is
detected.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

For the moment we have only the EBBR.yaml configuration. Command line does
still take precedence. No functional change.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a schema and a `--validate-seq-db' command line option to validate
our sequence files database, now that it is in YAML format.
We call the validation from `make check'.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Let's use YAML as much as we can as we can better validate it and extend
it.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

To prepare for adding more validation schemas, create a dedicated folder
for our schemas.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

IR 1.1 certifications should use the ir1 branch so that development can
continue on the main branch.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add the BBSR sequence file from the Security interface extension ACS
v21.10_SIE_REL1.0.

While at it, fix sequence files permissions.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

- Add a jsonschema describing requirements on the configuration file.
- Add a --validate-config and a --schema parser options to make use of the
  schema.
- Add a check for the EBBR.yaml configuration in `make check'.
- Document in the README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Re-classify warnings issued by the SCT regarding SetState() returning
unsupported as known ACS limitations and drop the known SCT limitation
category. Also, add a reference to the corresponding edk2-test bug entry.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Deal with plural in a better way, in a couple of additional non-debug
places. While at it, update the README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

A specific case where SetState() return "unsupported" makes the SCT emit a
warning. As this is a valid return value, add a rule to force this warning
as "known sct limitation".

While at it, document this new "known sct limitation" category in the
README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Now that ACS-IR v1.0 is out, add its sequence files to the `contrib' folder
as well as to the `seq.db' database for identification.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Make our dependencies a bit clearer in the documentation and in the warning
messages we print from the parser.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Give an example command, to quickly see the help comments added by some
rules to help debugging.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Triaging results when testing for EBBR compliance is the main use case of
the parser; let's make it the default.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a database of the sequence files we know about with their sha256,
and we try to identify the input sequence file among those.

A `--seq-db' option is added to allow specifying the database.

Update documentation accordingly. While at it, fix spacing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

To avoid issues with the sequence file getting modified in the test report,
recommend using the copy of EBBR.seq that is stored in this repo.

Signed-off-by: Grant Likely <grant.likely@arm.com>

Let's classify the ACS-IR BETA 0 MemoryMap failure automatically.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Update the documentation accordingly. While at it, remove obsolete mention
of default behaviour.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Refine the documentation of the result.md and explain how to generate HTML
with pandoc.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to perform further processing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add a checker for python scripts and a checker for shell scripts to the
`make check' target. Fix all issues reported by the checker in parser.py.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

It is convenient to have those sequence files at hand to analyse results
obtained with the different ACS-IR published versions.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Adds some simple instructions to the top of the README file so that
a users doesn't need to go searching for the instructions.

Signed-off-by: Grant Likely <grant.likely@arm.com>

Start to document the EBBR.yaml configuration file specificities.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We put the links definitions in the same section as the first use, to
facilitate reading the README.md directly.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This runs yamllint for the moment, but can be extended to do more in the
future.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is useful to print a quick summary, after filtering. We document some
usage in the README.md.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This is similar in its principle to the UNIX `uniq -c' and is handy for
triaging results, in conjunction with the `--fields' option.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Keeping only some fields is handy to generate summaries for triaging.

This does not apply to the markdown and config template formats, which
depend on certain fields being present.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Document how the `--filter' option can be somewhat (ab)used to apply
transformations to tests data.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

cat.sh and dropped entries have been verified to be fine on runs with more
than 10k tests.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>