Added support for outputing results in junit format

See merge request !13

Drew Reed authored Jul 23, 2021 and Drew Reed committed Jul 14, 2022

Added new --junit command line option to instruct the script to output
the results in junit format for easy inclusion in CI systems

Signed-off-by: Drew Reed <Drew.Reed@arm.com>

No functional change.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Let's make sure our sample is correct.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

For the moment we have only the EBBR.yaml configuration. Command line does
still take precedence. No functional change.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a schema and a `--validate-seq-db' command line option to validate
our sequence files database, now that it is in YAML format.
We call the validation from `make check'.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Let's use YAML as much as we can as we can better validate it and extend
it.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

To prepare for adding more validation schemas, create a dedicated folder
for our schemas.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

The SCT in the ACS-IR 2.0 Alpha has been updated; add some rules to triage
the results accordingly in preparation of IR 2.0.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

IR 1.1 certifications should use the ir1 branch so that development can
continue on the main branch.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Deal with empty config gracefully. This allows to pass e.g. /dev/null as
config without crashing the parser.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

schema + BBSR seq file

See merge request !11

Add the BBSR sequence file from the Security interface extension ACS
v21.10_SIE_REL1.0.

While at it, fix sequence files permissions.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We know all the valid values for many fields of the configuration rules.
Refine the schema to constrain the EBBR.yaml config file a bit more.

This means we are giving up the ability to mention only a sub-string for
those fields in the config file, which we were not doing anyway.

Also, forbid additional properties where we can.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Schema + minors

See merge request !10

- Add a jsonschema describing requirements on the configuration file.
- Add a --validate-config and a --schema parser options to make use of the
  schema.
- Add a check for the EBBR.yaml configuration in `make check'.
- Document in the README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Even tparm() can fail on some platforms; make the whole colors probing part
more robust by catching exceptions.

Reported-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

setafb must be bytes, as we will decode() it later on.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Colors and SetState

See merge request !9

Add red/yellow/green colors to more messages.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Re-classify warnings issued by the SCT regarding SetState() returning
unsupported as known ACS limitations and drop the known SCT limitation
category. Also, add a reference to the corresponding edk2-test bug entry.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

This fixes some crashes when using e.g. `--uniq --print --fields <list
without count in it>'.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Colors and plurals

See merge request !8

Deal with plural in a better way, in a couple of additional non-debug
places. While at it, update the README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Now that we have colors, use them in our summary line. While at it, deal
more nicely with plurals.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Add some colors to our logging messages. We convert a couple of remaining
cases not using the logging facility for error messages.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

New "known sct limitation"

See merge request !7

A specific case where SetState() return "unsupported" makes the SCT emit a
warning. As this is a valid return value, add a rule to force this warning
as "known sct limitation".

While at it, document this new "known sct limitation" category in the
README.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

New "known u-boot limitation" revealed by ACS-IR v1.0

See merge request !6

The QueryVariableInfo() implementation in U-Boot does not check properly
for all the invalid inputs conditions and we already mark some failures as
"known u-boot limitations".

The new ACS-IR v1.0 uses a new SCT version with commit 814b2af535d9
("uefi-sct/SctPkg: uefi-sct:QueryVariableInfo(EFI_VARIABLE_NON_VOLATILE)"),
which causes a new failure.

Mark this new failure as "known u-boot limitation".

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Changes to parser, config and doc

See merge request !5

Now that ACS-IR v1.0 is out, add its sequence files to the `contrib' folder
as well as to the `seq.db' database for identification.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Make our dependencies a bit clearer in the documentation and in the warning
messages we print from the parser.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Give an example command, to quickly see the help comments added by some
rules to help debugging.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Ignore a couple of network failures, which happen with edk2. Network
functions are allowed to return "unsupported" as EBBR does not require a
full implementation of the UEFI general purpose networking ABIs.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Some simple network failures happen when the ethernet cable is unplugged.
Add comments in that case, to help debugging.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Triaging results when testing for EBBR compliance is the main use case of
the parser; let's make it the default.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

We add a database of the sequence files we know about with their sha256,
and we try to identify the input sequence file among those.

A `--seq-db' option is added to allow specifying the database.

Update documentation accordingly. While at it, fix spacing.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>

Make printing a bit closer to markdown, with header line, wider separations
between columns and capitalized header fields.

Signed-off-by: Vincent Stehlé <vincent.stehle@arm.com>