README.md was previously not very clear for newcommers, so make it clear
that the documentation contains a quickstart guide, and also add an even
quicker guide in-place for the lazy.

We also remove the instructions for building the documentation locally
and move it to the readthedocs documenation.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add a diagram of the shrinkwrap workflow along with some textual
descriotion to aid understanding of Shrinkwrap's core concepts.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

docker: Build FVP into aarch64 docker image

See merge request !6

Now that Arm have released a public build of the FVP_Base_RevC-2xAEMvA
for aarch64 linux systems, add that to the aarch64 docker image, just
like we do with x86_64. This brings aarch64 to feature parity with
x86_64.

Note that while we are using FVP 11.20 for aarch64, we are still using
18.16 for x86_64. 11.20 is the first available release for aarch64, but
a performance regression was seen with it on x86_64 so we are holding
off upgrading for x86_64 until the regression is fixed.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

runtime: Add support for podman containers

See merge request !2

Build: Fix "ValueError: I/O operation on closed file"

See merge request !5

Very occasionally, a build would fail with the above error. The root
cause was that neither of the output streams for a process would be
unregistered from the selector until both had reported EOF. This means
that one of them could be requeued after it reported EOF, while the
second was handled and (and the first unregistered/closed as a result).
Then we would get around to handling the first one from the queue and
bang.

Solve this by unregistering the stream from the selector as soon as it
reports EOF. Acomplish by refactoring _proc_deactivate() into
_proc_stream_deactivate() + _proc_deactivate(). And also simplify some
process shutdown code since popen is doing all the stream close() calls
for us (proven with trace).

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Various UI Improvements

See merge request !4

We were previously converting UART output to strings using Python's
universal newline technique, where '\r', '\n' and '\r\n' were all
replaced with '\n'. This caused a few issues; First, telnet would
occasionally output '\r\r\n' which would get converted to 2 newlines,
whereas only one was intended. Secondly, we were swallowing legitimate
'\r's when working at the interactive shell and the cursor would
occasionally get messed up.

Solve this by never converting line endings when converting the binary
data to string data. As a result we must be more carefull with '\r'
since they may now exist in the stream.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Use labels of the same length for both normal and verbose output, and
reserve the default tezt colour for normal output to distingush it from
verbose prints.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Git is chatty by default and so would previously output informtive
messages even when shrinkwrap's --verbose was not specified. This would
make the output less legible. So pass --quiet to git to shut if up when
the user has not specified --verbose.

Similarly, when injecting a CHOSEN node into a device tree, dtc would
re-emit warnings that are already output from the original compile.
Suppress these duplicates by passing -q to dtc when doing the recompile.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Red implies an error has occured. So using red as a colour for an
informational output stream could be confusing for the user. Remove it
from the set of colours we use.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

config: Update to latest tagged component versions

See merge request !3

We prefer to default to a tagged component version rather than tracking
the tip of a branch, since this minimizes chances of hitting stability
issues (users that want to follow a branch can override it themselves).
However this means we need to periodically up-rev all components and
test that they continue to work. All tests have passed with these new
versions.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

--smoke-test will cause only a single arch variant to be tested.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

As an alternative to Docker, users can now choose to use
--runtime=podman. It uses the same container images (from Docker hub)
but some people prefer podman because it doesn't require the use of a
damemon with root privilege.

Tuxmake provides most of the infrastructure so we just wire it up to the
--runtime cli option and fix a few issues where podman is more picky
(mount points and fully qualified container image names). Documentation
is also updated to describe how to set up for podman.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

core: Add support for Python 3.6.9

See merge request !1

Previously we were restricting support to Python 3.7.0 and newer, due to
that version being the first to support ordered dicts, which are relied
upon to pretify generated config files. However it turns out that Python
3.6 implements ordered dicts despite that version of the language not
requiring it.

Ubuntu 18.04 ships with Python 3.6.9 by default and a number of people
are still using that release, so let's support that as the minimum
version.

In order for it to work, there are a few mods we have to make to API
calls etc.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Switch from using Arm's internal Artifactory repo to using the public
docker hub to store images. We also take the opportunity to modify the
naming scheme so that the arch is part of the tag instead of part of the
repository so that all images live in the same repository.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Some users are reporting issues setting up shrinkwrap, so add this
option to help debug. Commonly, we don't want to be too chatty, but if
something goes wrong during early startup or if the user has specified
--verbose, then let's get all the info we can.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

A higher layer config can now optionally provide INITRD_START and
INITRD_END addresses to encode these parameters in the generated device
tree. If used, then the higher level config is responsible for loading
the initrd into memory.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

A higher layer config can now optionally override the DTS that gets
compiled. While we are at it, we also change the method of injecting the
kernel command line to be more robust (the previous method broke if the
chosen{} node was not already present or was in an included file).

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

When in overdraw mode, we must always output the whole block of labels
on first draw (or first draw after a warning is output). Otherwise the
second draw could overdraw the warning. Also fix the case where stdout
is not a terminal.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We can run shrinkwrap on MacOS as long as docker desktop is installed.
There are a couple of issues to work around though:

- readlink on MacOS does not have the -f (follow) option. Its not really
  important, so just remove use of it entirely.
- We were getting an error when trying to add the GID to the container
  due to Linux already having allocated the GID. So just run as root
  within the container on MacOS. The MacOS version of docker appears to
  ensure that files on the host file system have the correct permissions
  anyway.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

It turns out that when setting a file descriptor to nonblocking, if that
fd is stdin and connected to a tty, it also makes stdout nonblocking.
Which means that prints() would occasionally fail if the output buffer
was full.

Solve this by only making file descriptors temporarily nonblocking when
reading them. Since the application is single threaded, this means that
the bit is never set when subsequently doing a print.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Modify the behavior so that even when --verbose is not specified, stderr
is still output to the console. This means that compiler warnings and
the like will be seen more clearly. In the absense of warnings, no
stderr is expected to be output so the console still looks "pretty",
with overdrawn status labels etc. If a warning occurs it is output
consicely and labels are reset so they don't overdraw the warning.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

- Drop support for podman since it is untested. It's simple to add it
  back in future if requried.
- Make `docker` the default runtime; this is what we want to push users
  towards.
- Improve runtimes documentation to include discussion of the image
  variants and dependency requirements.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Requiring the user to explicitly set the SHRINKWRAP_CONFIG environment
variable is onerous and error-prone. So pre-populate the path list with
common locations and prepend with any (now optionally) supplied paths
from the user. And filter so that we only keep paths that exist, which
also resolves a pre-existing bug where shrinkwrap errored out if
SHRINKWRAP_CONFIG contained a non-existing path.

While we are at it, print the final workspace paths when --verbose is
specified.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

To build the docker images do the following:

  cd docker
  ./build.sh latest

This will build and push the images for your native architecture (works
for both x86_64 and aarch64). Images are currently pushed to Arm's
internal Artifactory repo, and you will need to have docker logged in
for this to be successful. (We plan to change this in future to publish
to docker.io).

You will need to build on both x86_64 and aarch64, then do this on one
of the systems:

  ./publish.sh latest

This will tie the images together to form multiarch images, such that
docker can resolve the correct image based on the arch on which you are
running.

The following images are built:

  - base-slim-nofvp
  - base-slim
  - base-full-nofvp
  - base-full

'slim' variants don't contain aarch32 toolchains. These are massive and
bloat the image and most people don't need them. 'full' variants are
based on 'slim' but add in the aarch32 toolchains. 'nofvp' variants
don't contain an fvp. These images are intended for use as base images
if the need arises to add a custom fvp.

Currently there is no public fvp support for aarch64, so there is no fvp
included with this arch. We hope to solve this in future.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

ns-edk2-* previously relied upon assembling all the assets required for
semihosting in the package directory. Some of these assets were
dynamically generated at runtime. This was problematic for running
multiple instances in parallel.

So instead, create a per-instance temporary directory for semihosting
that is cleaned up at exit.

This approach means that we no longer need to expose the
{param:packagedir} macro, so remove that while we are at it.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

When running the FVP inside a container it can be useful to know the IP
address for use cases such as attaching the debugger or ssh'ing into the
hosted linux system.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Building and packaging the Linux modules is expensive, even for
incremental builds. Often this is not required, so make it optional with
a shell variable. A derived config can add the following to the linux
prebuild command list:

export BUILD_KMODULES=false

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

While this will be deployed with the python package, it is not strictly
part of it, and we want to reduce the coupling between the tool and the
configs. The SHRINKWRAP_CONFIG environment variable must be set to tell
Shrinkwrap where the config store location(s) are.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add a 'toolchain' key to the component dictionary, which specifies the
toolchain to use. This string is set in the CROSS_COMPILE environment
variable before invoking any prebuild, build, postbuild or clean
commands.

When using the standard docker runtime image, the following options are
supported. Users are free to configure their own thing when using the
null runtime.

* aarch64-none-elf-    (arm64 bare-metal, from Arm)
* arm-none-eabi-       (arm32 bare-metal, from Arm)
* aarch64-linux-gnu-   (arm64 linux, from Debian)
* arm-linux-gnueabihf- (arm32 linux, from Debian)

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We want to make it possible for multiple config stores to coexist. This
enables the user to use the standard store that ships with shrinkwrap
and add in extra stores for configs that can't be shared publicly (for
example). References to yaml files should resolve across stores.

We solve this by extending the SHRINKWRAP_CONFIG environment variable to
behave like $PATH. It is now a colon-separated list of config store
paths, in preference order, with the most preferred appearing first. We
have also removed the old behaviour where shrinkwrap would default to
the config store at its root when the variable is not set. Instead we
now raise an error. This change is so that we can decouple the config
store from shrinkwrap entirely.

Everything works out fairly well, except for one aspect; Resolving the
${param:configdir} macro now requires knowing what's being looked up in
the config store and finding the store that contains it. This is
implemented and appears to work reliably. But means we can only allow
the use of this macro for resolving files that already exist and are
inputs to the process. This is not a problem for existing uses.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>