Update benchmarks library with yaml file fragment for repro-collection
mysql workload. It defines the suite, name, & type of the benchmark.
Benchmark container image is already saved in gitlab registry.
This image path is specified in the yaml file.

Other params for repro-collection benchmark include:
- workload which is currently `mysql`
- sut_nr_cpus fixed at 16 to test workload correctly

Update Ubuntu license info along with Debian.
Docker image for repro-collection benchmark uses Ubuntu:22.04.

Signed-off-by: Aishwarya Rambhadran <aishwarya.rambhadran@arm.com>

Create an Ubuntu-based Docker container to run the aws/repro-collection
MySQL workload.
The repro-collection repository is cloned into the Docker container
at build time. SUT and Load Generator (LDG) installations for the
workload are also performed during the build process.

Apply patch to the repro-collection's main.sh.
This is to fix the number of SUT CPUs to an optimum value provided
via benchmark parameters. This helps to tune the benchmark to
simulate a bottleneck in the SUT.

Add wrapper script repro_collection.sh.
`service mysql stop` fails during repro_collection's SUT cleanup step.
Root cause was traced to the use of `debian-sys-maint` user credentials
from `/etc/mysql/debian.cnf` which were not available in the custom
MySQL setup within repro_collection.
New wrapper script adds a `mysql:post:configure:sut` hook to
repro_collection's main.sh script. It defines the commands to
fetch credentials from /etc/mysql/debian.cnf, create this user in
MySQL DB, set password & grant shutdown permission. This resolves
service shutdown issue cleanly.
The script also sources repro_collection's run.sh which starts
the workload execution.

Implement exec.py as the entry point of the docker container to
run repro_collection.
This invokes wrapper script repro_collection.sh and provides the
required input parameters as per the benchmark plan.
Once the workload completes, the results are parsed and populated
as Fastpath results.

Signed-off-by: Aishwarya Rambhadran <aishwarya.rambhadran@arm.com>

By default, merge request pipelines run with detached heads. This is
problematic for building the docs using ReadTheDocs because it requires
a branch name to map to a version in RTD. But with a detached head,
there is no branch.

So let's disable MR pipelines. When pushing changes to a branch that has
an MR, the branch pipeline will still run and the result shows up in the
MR.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The "human readable id" format, used to identify benchmarks, swprofiles
and suts in the command line interface, is usually just the name that
the user provided in the plan (in the case of swprofile and sut) or
"suite/name" in the case of benchmark. But these names are not
guaranteed to be unique. If there are multiple objects of the same type
with the same name in the resultstore, the we suffix ":<id>", where <id>
is the unique integer id in the resultstore. That ensures we always have
a unique and friendly way to refer to these objects.

But it turns out that "result merge" was buggy and did not correctly
filter for the "<name>:<id>" case. This was previously implemented with
extra, pre-calculated columns in the tables, which was intended to
simplify the filtering. But the columns were precalculated prior to
inserting the object so the id was None at that point, meaning the
column ended up with "<suite>/<name>:None" in the case of benchmark. For
sut and swprofile, the problem was even worse as the column was misnamed
due to a copy/paste error causing the column to always be NULL.

So let's get rid of those pre-computed columns and just complicate the
filter so that it can concatenate and compare on the fly.

We deliberately allow <name> to be provided as the filter even when
there are multiple objects with <name>. In this case the filter will
match all of them.

Fixes: a869c532 ("cli: Implement resultstore abstraction around database")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Some systems lock down grub.cfg so that it is only readable by root.
Therefore, on these systems, ctx.get() would fail because it runs as the
regular user. Work around this with "sudo cat" then capture the stdout.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

To comply with security requirements, we must rebuild the benchmark
containers regularly. We will do this using a gitlab scheduled pipeline.
To prepare for that, let's add the concept of a pipeline type via the
PIPELINE_TYPE variable. The existing type is now called "Continuous
Integration (CI)" and we introduce "Build Container (BC)", which when
invoked, will build a specified container tagged with the specified
version.

The scheduled pipelines will be setup to invoke this new type.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Let's parameterize the project name, just like we do for the domain and
token.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add a README.md landing page for gitlab which contains the key links for
the documentation, repo and container registry. Additionally add those
same links to the introduction page within the main docs.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Introduce basic documentation to describe what fastpath is, how to
install it and how to create and execute plans.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

To date, the term "config" has been used to refer to the SW
configuration of the SUT. This includes the kernel that's installed, the
kernel command line, sysctls, etc. But "config" is a very overloaded
term, and people were getting confused between the fastpath config and
the kernel (K) config.

So let's rename this key object to "swprofile". A SW profile is a lot
more self-descriptive and distinct from the myriad of other configs we
commonly talk about.

This is mostly just renaming within the code, but there are 3 places
were backwards compatibility is severely broken:

  - The CLI used to accept --config all over the place. It now expects
    --swprofile. So saved command invocations need to be updated.

  - The plan.yaml schema used to have a "configs" list. The list is now
    called "swprofiles". So existing plans need to be updated.

  - The resultstore schema used to have a CONFIG table. This is now
    renamed to SWPROFILE. Additionally the RESULT and ERROR tables used
    to have a column called config_id. That's renamed to swprofile_id.
    Existing resultstores (in all formats; CSV, MySQL, SQLite) will need
    to be modified by hand to work with the modified tooling.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Commit 754d6d6 ("cli: Generalize FASTPATH_LIBRARY into a key-value
store") and commit ef6cf61 ("cli: Add "config set" and "config get"
commands") introduced a general purpose "user config" key-value store
together with a command line interface ("config get"/"config set") to
access and modify the values.

But the term "config" is massively overloaded and confusing to users. So
let's rename to "preferences" instead. The cli noun is now called
"preference" and the peristent store file is renamed from
~/.fastpathconfig to ~/.fastpathpreferences.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The number of configs is becoming rather large in our internal database,
so let's add an option to filter the list of objects that are output
from "result list" by id.

For example:

$ fastpath result list --object config --id ryan-*

Will list all configs whose ID begins with "ryan-".

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We have had some issues with dodgy kernels falling over during benchmark
runs. But because we only have an SSH connection and don't have access
to the serial port, we can't see exactly what caused the panic.

Let's fix that by using netconsole to stream the kernel log over UDP to
the system running the fastpath driver. The log is then saved to a
per-session log file. If the machine panics we will see the panic log in
this file.

The file is named "session-<uuid>.kmsg" within the config directory.
Logging is best effort; it is fails to initialize we carry on.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We have a problem; the same config, when installed on 2 different suts,
may have a slightly different fingerprint, so there will be 2 separate
entries in the database. For instance, if the rootfs location is
different, the kernel_cmdline_full_hash will be different between the 2
machines. And it's possible the userspace_name may differ if the Ubuntu
point release is different for example.

So let's disregard these small differences by default; we will still
keep the separate entries in the resultstore, but unless told otherwise,
we will merge similar configs and fix up all the references from the
other tables when we read the data in. This means we can easily compare
benchmarks across different suts. If the user knows better, they can opt
out of this merging behavior by specifying --no-merge-similar.

Additionally, let's also merge similar benchmarks so that the container
image can change without considering it a new benchmark by default.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

In the happy path we aim to avoid tearing down and restarting the
container between repeats of the same benchmark to speed up execution
time. But if there was an error on a previous run, let's force restart
the container to clean up any bad state that was left behind.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

In some very niche corner cases (devs wanting to run fastpath on a
system that initially has a very custom kernel installed), the initial
running kernel does not have sufficient Kconfigs enabled to run docker
(this is true for defconfig for example). Therefore "docker image pull"
would fail. Let's workaround this by installing the first config first
then we can do the "docker image pull" in the context of that kernel
which must support docker.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Aishwarya TCV authored Mar 31, 2025 and Ryan Roberts committed Jun 09, 2025

Introduces a new function `render_deviation_comparison_chart` to
visualize relative performance changes across configs or SUTs.

Users can select a baseline config or SUT via dropdown to compare
against others. The chart shows deviation bars, color-coded by
change type: green for improvements, red for regressions, and grey
for neutral changes within a configurable threshold (default 0.1%).

The logic aligns with `compute_change` from show.py and supports
relative/absolute views and filtering by change type via sidebar.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

The logic that determines if there is a statistically significant change
between two results was previously embedded in pretty_results_multi().
We want to reuse this code in the dashboard, so let's factor it out.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The benchmarks must be extracted from results prior to doing the pivot,
because "benchmark" is no longer a column after that.

Fixes: a832759 ("cli: Clarify SUT or config for "result show"")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

I've seen issues where put() fails with "OSError: Socket is closed"
after a previous timeout while running a command. I haven't been able to
reliably reproduce, but I suspect this could be due to a bug in fabric
(or paramiko). Attempt to work around this by explicitly disconnecting
and reconnecting to the SUT upon timeout (or any other exception).

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

I've had reports of a particular server system taking longer than 10
mins to reboot and the reboot timeout kicking in. As a quick fix, let's
increase the timeout to 15 mins. A better solution would be to make this
programmable via the plan. We can implement that if 15 mins proves to be
too short for some.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Commit d853eeb ("cli: Keep SSH connection alive") enabled an ssh
keepalive ping every 30 seconds to stop eager network applicances for
tearing down the connection. This solved the issues we were previously
seeing but I think I saw something similar recently. So let's increase
it to a 15 second ping to be safe.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

When showing results, add a title describing that SUT and/or config that
the results are for to make things easier for the user to understand.

Additionally when --verbose is provided, let's output a list of all the
SUTs, configs, and benchmarks that are considered in the results.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Aishwarya Rambhadran authored May 16, 2025 and Ryan Roberts committed Jun 09, 2025

Fastpath MySQL transactions failed on macOS as mysqlclient tried to
load native '.so' plugin files. These files may not be available in
MySQL installations via Homebrew, which caused
MySQLdb.OperationalError during `fastpath result list/show` commands.

To resolve this, the PyMySQL module is now used as the MySQL driver.
PyMySQL is compatible with SQLAlchemy when used via monkey-patching
with `pymysql.install_as_MySQLdb()`.

Changes:
- Add pymysql to Python requirements.txt
- Patch resultstore in fastpath to use PyMySQL
- Remove default-libmysqlclient-dev from Dockerfiles

Signed-off-by: Aishwarya Rambhadran <aishwarya.rambhadran@arm.com>

The Gigabyte AmpereOne-based R263-P30-AAH1 server has a BCM57414
NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller, which is not in the
defconfig, so let's add it to fastpath's standard kernel config.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

It's sometimes useful to have XFS support when we need a file system
that supports large folios. Since we are about to modify the standard
config to support AmpereOne and this will change the Kconfig hash, we
might as well add XFS at the same time.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add simple commands to get and set user configuration data stored in
~/.fastpathconfig.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Let's allow users to store their default resultstore URL in
.fastpathconfig to avoid having to provide it on the command line all
the time. This is useful when using a central DB a lot.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add a general purpose key-value store to fastpath to contain user
configuration items. The store is a file in INI format, stored at
~/.fastpathconfig. For now it is only possible to set/modify the values
by directly editing the file. But we may add a get/set interface to
simplify this in future.

Values have a category and a name. The combined key is referred to as
"<category>.<name>" (always lower case). It is possible to override a
given value using an environment variable of the form
"FASTPATH_<CATEGORY>_<NAME>" (always upper case). Note that values in
the "global" category can also use the form "FASTPATH_<NAME>", which
conveniently allows compatibility with the existing FASTPATH_LIBRARY
environment variable.

The getter API, getvar() takes strings for category and name and returns
the string value, if it is defined, or None if it isn't. There is
deliberately no global list of valid keys; they are free to expand (and
contract) as needed.

Our first user is global.library.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

There is a race between the reboot command exiting and the remote system
tearing down the ssh connection. -1 (255) means the ssh connection was
lost. 0 means reboot exited cleanly. Either way the system will reboot.

Until recently we only ever saw -1. 0 was seen recently so let's make
sure we treat that as success too.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Previously, if there were insufficient iteration results to calculate
the confidence interval, fastpath would not indicate regression or
improvement. Let's change that to fall back to the noise_threshold
(defaults to 1%) in such cases so we can still indicate
regression/improvement, albeit with less confidence.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Aishwarya TCV authored Mar 24, 2025 and Ryan Roberts committed Jun 09, 2025

Adds a new Streamlit-based dashboard for visualizing benchmark data
from the resultstore. The dashboard script enables interactive views
of kernel performance results filtered by SUT, config, and benchmark.
It uses Altair charts for statistical plots and includes data tables.
The __init__.py file is added to support dashboard module structure.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Adds a new 'serve' verb under the result noun to launch a dashboard
server using Streamlit. The dashboard is launched via a CLI command
and is initialized with the provided resultstore URL. This enables
users to visually explore  benchmark results stored in CSV, SQLite,
or MySQL resultstores.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Aishwarya TCV authored Mar 24, 2025 and Ryan Roberts committed Jun 09, 2025

Adds Streamlit to the Python requirements to support the new
dashboard feature in Fastpath. The dashboard enables interactive
visualization of benchmark results using a web-based UI. This
dependency is required to run the dashboard.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Currently the pipeline simply automates the building of the CI image and
additionally runs "black" over all the python code to check it for
correct formatting.

In future this could be expanded to run unit tests too.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

When building kernels on the fastpath CI, include the vmalloc test
module. This module is required by the micromm benchmark suite.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

fastpath.frag contains all the Kconfig additions on top of defconfig
that fastpath requires for an arm64 kernel. The additions are primarily
to enable docker which is core to running benchmarks with fastpath.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

_get_grub_defaults() reads /etc/default/grub and parses the contents
into key/value pairs. It previously did this by splitting on "=". But
this breaks if the value contains an "=". So instead let's find the
first "=" and split on that marker.

Fixes: 27aae23a ("cli: Implement "fastpath sut configure" command")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>