To date, the term "config" has been used to refer to the SW
configuration of the SUT. This includes the kernel that's installed, the
kernel command line, sysctls, etc. But "config" is a very overloaded
term, and people were getting confused between the fastpath config and
the kernel (K) config.

So let's rename this key object to "swprofile". A SW profile is a lot
more self-descriptive and distinct from the myriad of other configs we
commonly talk about.

This is mostly just renaming within the code, but there are 3 places
were backwards compatibility is severely broken:

  - The CLI used to accept --config all over the place. It now expects
    --swprofile. So saved command invocations need to be updated.

  - The plan.yaml schema used to have a "configs" list. The list is now
    called "swprofiles". So existing plans need to be updated.

  - The resultstore schema used to have a CONFIG table. This is now
    renamed to SWPROFILE. Additionally the RESULT and ERROR tables used
    to have a column called config_id. That's renamed to swprofile_id.
    Existing resultstores (in all formats; CSV, MySQL, SQLite) will need
    to be modified by hand to work with the modified tooling.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Commit 754d6d6 ("cli: Generalize FASTPATH_LIBRARY into a key-value
store") and commit ef6cf61 ("cli: Add "config set" and "config get"
commands") introduced a general purpose "user config" key-value store
together with a command line interface ("config get"/"config set") to
access and modify the values.

But the term "config" is massively overloaded and confusing to users. So
let's rename to "preferences" instead. The cli noun is now called
"preference" and the peristent store file is renamed from
~/.fastpathconfig to ~/.fastpathpreferences.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The number of configs is becoming rather large in our internal database,
so let's add an option to filter the list of objects that are output
from "result list" by id.

For example:

$ fastpath result list --object config --id ryan-*

Will list all configs whose ID begins with "ryan-".

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We have had some issues with dodgy kernels falling over during benchmark
runs. But because we only have an SSH connection and don't have access
to the serial port, we can't see exactly what caused the panic.

Let's fix that by using netconsole to stream the kernel log over UDP to
the system running the fastpath driver. The log is then saved to a
per-session log file. If the machine panics we will see the panic log in
this file.

The file is named "session-<uuid>.kmsg" within the config directory.
Logging is best effort; it is fails to initialize we carry on.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We have a problem; the same config, when installed on 2 different suts,
may have a slightly different fingerprint, so there will be 2 separate
entries in the database. For instance, if the rootfs location is
different, the kernel_cmdline_full_hash will be different between the 2
machines. And it's possible the userspace_name may differ if the Ubuntu
point release is different for example.

So let's disregard these small differences by default; we will still
keep the separate entries in the resultstore, but unless told otherwise,
we will merge similar configs and fix up all the references from the
other tables when we read the data in. This means we can easily compare
benchmarks across different suts. If the user knows better, they can opt
out of this merging behavior by specifying --no-merge-similar.

Additionally, let's also merge similar benchmarks so that the container
image can change without considering it a new benchmark by default.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

In the happy path we aim to avoid tearing down and restarting the
container between repeats of the same benchmark to speed up execution
time. But if there was an error on a previous run, let's force restart
the container to clean up any bad state that was left behind.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

In some very niche corner cases (devs wanting to run fastpath on a
system that initially has a very custom kernel installed), the initial
running kernel does not have sufficient Kconfigs enabled to run docker
(this is true for defconfig for example). Therefore "docker image pull"
would fail. Let's workaround this by installing the first config first
then we can do the "docker image pull" in the context of that kernel
which must support docker.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Aishwarya TCV authored Mar 31, 2025 and Ryan Roberts committed Jun 09, 2025

Introduces a new function `render_deviation_comparison_chart` to
visualize relative performance changes across configs or SUTs.

Users can select a baseline config or SUT via dropdown to compare
against others. The chart shows deviation bars, color-coded by
change type: green for improvements, red for regressions, and grey
for neutral changes within a configurable threshold (default 0.1%).

The logic aligns with `compute_change` from show.py and supports
relative/absolute views and filtering by change type via sidebar.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

The logic that determines if there is a statistically significant change
between two results was previously embedded in pretty_results_multi().
We want to reuse this code in the dashboard, so let's factor it out.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The benchmarks must be extracted from results prior to doing the pivot,
because "benchmark" is no longer a column after that.

Fixes: a832759 ("cli: Clarify SUT or config for "result show"")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

I've seen issues where put() fails with "OSError: Socket is closed"
after a previous timeout while running a command. I haven't been able to
reliably reproduce, but I suspect this could be due to a bug in fabric
(or paramiko). Attempt to work around this by explicitly disconnecting
and reconnecting to the SUT upon timeout (or any other exception).

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

I've had reports of a particular server system taking longer than 10
mins to reboot and the reboot timeout kicking in. As a quick fix, let's
increase the timeout to 15 mins. A better solution would be to make this
programmable via the plan. We can implement that if 15 mins proves to be
too short for some.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Commit d853eeb ("cli: Keep SSH connection alive") enabled an ssh
keepalive ping every 30 seconds to stop eager network applicances for
tearing down the connection. This solved the issues we were previously
seeing but I think I saw something similar recently. So let's increase
it to a 15 second ping to be safe.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

When showing results, add a title describing that SUT and/or config that
the results are for to make things easier for the user to understand.

Additionally when --verbose is provided, let's output a list of all the
SUTs, configs, and benchmarks that are considered in the results.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Aishwarya Rambhadran authored May 16, 2025 and Ryan Roberts committed Jun 09, 2025

Fastpath MySQL transactions failed on macOS as mysqlclient tried to
load native '.so' plugin files. These files may not be available in
MySQL installations via Homebrew, which caused
MySQLdb.OperationalError during `fastpath result list/show` commands.

To resolve this, the PyMySQL module is now used as the MySQL driver.
PyMySQL is compatible with SQLAlchemy when used via monkey-patching
with `pymysql.install_as_MySQLdb()`.

Changes:
- Add pymysql to Python requirements.txt
- Patch resultstore in fastpath to use PyMySQL
- Remove default-libmysqlclient-dev from Dockerfiles

Signed-off-by: Aishwarya Rambhadran <aishwarya.rambhadran@arm.com>

The Gigabyte AmpereOne-based R263-P30-AAH1 server has a BCM57414
NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller, which is not in the
defconfig, so let's add it to fastpath's standard kernel config.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

It's sometimes useful to have XFS support when we need a file system
that supports large folios. Since we are about to modify the standard
config to support AmpereOne and this will change the Kconfig hash, we
might as well add XFS at the same time.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add simple commands to get and set user configuration data stored in
~/.fastpathconfig.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Let's allow users to store their default resultstore URL in
.fastpathconfig to avoid having to provide it on the command line all
the time. This is useful when using a central DB a lot.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add a general purpose key-value store to fastpath to contain user
configuration items. The store is a file in INI format, stored at
~/.fastpathconfig. For now it is only possible to set/modify the values
by directly editing the file. But we may add a get/set interface to
simplify this in future.

Values have a category and a name. The combined key is referred to as
"<category>.<name>" (always lower case). It is possible to override a
given value using an environment variable of the form
"FASTPATH_<CATEGORY>_<NAME>" (always upper case). Note that values in
the "global" category can also use the form "FASTPATH_<NAME>", which
conveniently allows compatibility with the existing FASTPATH_LIBRARY
environment variable.

The getter API, getvar() takes strings for category and name and returns
the string value, if it is defined, or None if it isn't. There is
deliberately no global list of valid keys; they are free to expand (and
contract) as needed.

Our first user is global.library.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

There is a race between the reboot command exiting and the remote system
tearing down the ssh connection. -1 (255) means the ssh connection was
lost. 0 means reboot exited cleanly. Either way the system will reboot.

Until recently we only ever saw -1. 0 was seen recently so let's make
sure we treat that as success too.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Previously, if there were insufficient iteration results to calculate
the confidence interval, fastpath would not indicate regression or
improvement. Let's change that to fall back to the noise_threshold
(defaults to 1%) in such cases so we can still indicate
regression/improvement, albeit with less confidence.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Aishwarya TCV authored Mar 24, 2025 and Ryan Roberts committed Jun 09, 2025

Adds a new Streamlit-based dashboard for visualizing benchmark data
from the resultstore. The dashboard script enables interactive views
of kernel performance results filtered by SUT, config, and benchmark.
It uses Altair charts for statistical plots and includes data tables.
The __init__.py file is added to support dashboard module structure.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Adds a new 'serve' verb under the result noun to launch a dashboard
server using Streamlit. The dashboard is launched via a CLI command
and is initialized with the provided resultstore URL. This enables
users to visually explore  benchmark results stored in CSV, SQLite,
or MySQL resultstores.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Aishwarya TCV authored Mar 24, 2025 and Ryan Roberts committed Jun 09, 2025

Adds Streamlit to the Python requirements to support the new
dashboard feature in Fastpath. The dashboard enables interactive
visualization of benchmark results using a web-based UI. This
dependency is required to run the dashboard.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Currently the pipeline simply automates the building of the CI image and
additionally runs "black" over all the python code to check it for
correct formatting.

In future this could be expanded to run unit tests too.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

When building kernels on the fastpath CI, include the vmalloc test
module. This module is required by the micromm benchmark suite.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

fastpath.frag contains all the Kconfig additions on top of defconfig
that fastpath requires for an arm64 kernel. The additions are primarily
to enable docker which is core to running benchmarks with fastpath.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

_get_grub_defaults() reads /etc/default/grub and parses the contents
into key/value pairs. It previously did this by splitting on "=". But
this breaks if the value contains an "=". So instead let's find the
first "=" and split on that marker.

Fixes: 27aae23a ("cli: Implement "fastpath sut configure" command")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Wrap the micromm benchmarks from David Hildenbrand into the micromm
container and expose the fork and munmap benchmarks.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The micromm benchmark needs to load the test_vmalloc module, so let's
map the modules into the container. This obviously assumes that the
kernel under test has been built with the test_vmalloc module.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

This micro benchmark tests the performance of the kernel's vmalloc APIs.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Initially this container can only run the vmalloc performance tests that
are part of the test_vmalloc.ko kernel module. If no parameters are
provided it will run a full set of tests with default parameters and
provide results for each. Alternatively, parameters may be provided to
specify a single specific test case with a set of custom parameters:

suite: micromm
name: vmalloc
params:
  test_case: fix_size_alloc_test
  nr_pages: 1
  huge: false
  nr_loops: 500000

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The help text for --noise-threshold previously contained 2 percent
symbols (%) which were interpretted by argparse as format specifiers,
causing an exception.

Fixes: f17a1fc ("cli: noise-threshold argument for "result show"")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

When a single config and sut is in the filtered result set, we show a
full set of summary statistics. Previously we failed to round ci95min
and ci95max columns to 2 decimal places. Let's fix that.

Additionally, when --relative is provided for the single config/sut
case, let's still show the absolute mean, then show the various other
summary stats as percentages relative to it. Previously we were showing
0% for the mean which isn't very helpful.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

This version is no longer available in the apt repos, so let's just use
the latest whenever we rebuild. We will need to ensure that any new
container that gets built does not recycle an existing version tag.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Commit ac88ccb ("cli: Remove RESULTCLASS summary field") removed the
RESULTCLASS.summary field, and as a result this was removed from the
results.csv file passed back from the container to the test runner. This
was a breaking change and was intentional; The plan was to rebuild the
containers.

But it turns out we can't easily rebuild the speedometer container
because the version of Chromium that the old version was using is no
longer available. Changing the browser version will change the benchmark
performance so a new container needs to be properly versioned.

So let's just allow the old container to work with the new code. We
allow "summary" but ignore it.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

pts will automatically detect when the system changes (CPU, compiler,
etc) and force a reinstall of the test in that case as an opportunity to
tune for the platform. But we don't want this behaviour when testing the
kernel; we want the user space to be constant. Additionally we don't
want to waste the extra time doing unneccessary recompiles. So let's
patch pts to avoid the forced reinstall. See the patch in the commit for
details.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Phoronix Test Suite is an open source test suite that incorporates a
number of system-level workloads. Let's wrap it into fastpath in a
similar way to how we wrapped mmtests. There is a single container image
which can figure out what test to run from the meta data that fastpath
passes in, then it parses the pts output to assemble the results in the
fastpath format.

Add an initial set of benchmark yaml fragments for various database and
language runtimes. More benchmark integrations will follow.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The database schema has a field in the CONFIG table for the git sha
corresponding to the kernel source that was built. It is not possible to
get this info from a running kernel in any reliable way though, so to
date, the field has remained empty.

But when we start running developer branches through the system, they
may be subject to rebasing so we need a way to tell one version from
another when the kernel name may not be unique. So let's allow the user
to (optionally) provide the git sha as a config field in the plan. And,
if present, connect that up to the CONFIG.kernel_git_sha field. So now
configs will be treated as separate items if their git sha differs.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>