Add some simple benchmarks to test performance of syscall entry and exit
logic. For now, we have 3 benchmarks, getpid, getppid and invalid. The
first 2 call the respective syscalls, the 3rd one makes a syscall with
an invalid syscall number.

Benchmark yamls are added to the benchmark library:

  syscall/getpid.yaml
  syscall/getppid.yaml
  syscall/invalid.yaml

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We have a number of small/drive-by benchmarks that are starting to
collect, so let's rename the micromm container image to microbench, then
we can add future non-mm benchmarks to it.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

"plan exec" always pulls all the docker images to the SUT at the start
of the process to ensure it has the most up-to-date versions.

But for benchmark integration development, this can be problematic as
the container only exists locally and is not in the remote registry. So
let's allow the operation to fail and we will just move on in that case.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Update execute_bisection.sh to align with the user-specified
bisection_context.yaml file by sourcing the context path from an
environment file (.env).

This simplifies the script interface and ensures that all key input
values (context, kernel, modules, gitsha) can be consistently passed
via a single environment file, improving clarity and making the
invocation from GitLab CI cleaner and more robust.

Fixes: 2f7460b0 ("scripts: add execute_bisection.sh for git bisect-based regression detection")

Fixes: 2f7460b0 ("scripts: add execute_bisection.sh for git bisect-based regression detection")
Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Aishwarya TCV authored Jun 16, 2025 and Ryan Roberts committed Jul 08, 2025

Adds a new helper script to support automated kernel regression
bisection using 'git bisect run'.

The script builds the kernel using build_local_kernel.sh, exports
kernel image and modules path, then runs benchmarks using
'fastpath bisect run'. It captures and propagates the exit status
from fastpath, enabling correct signaling to git bisect:

  0 => improvement or no change (good)
  1 => regression (bad)
  125 => skip (no data or filter mismatch)

This supports fast identification of performance regressions
across kernel versions.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Aishwarya TCV authored Jun 13, 2025 and Ryan Roberts committed Jul 08, 2025

This script provides an example of how to build a Linux kernel using
tuxmake, targeting both CI and local developer workflows. It assumes
a locally available Linux source tree, the location of which can be
configured using the LINUX_REPO_PATH environment variable.

It sets default values for architecture, toolchain, config, and output
paths, while allowing overrides for custom configurations. It selects
the tuxmake runtime based on context: 'null' for CI systems, or
'docker' if available locally.

The script ensures a clean build environment by removing untracked
files, then invokes tuxmake to build the kernel, modules, and config.
If an Image.gz is generated, it is uncompressed to produce an Image
file suitable for booting.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Aishwarya TCV authored Jun 13, 2025 and Ryan Roberts committed Jul 08, 2025

This patch includes the implementation of the `bisect run` subcommand
in the CLI. It resolves and updates the bisection template by injecting
kernel image and module paths, then executes the benchmark plan using
plan_exec to generate new result data.

After execution, the results are loaded and joined with metadata tables
to build a comprehensive dataset. The data is filtered by benchmark and
resultclass, with support for partial string matches and normalized
profiles. The latest run is assumed as the test profile.

It uses show.pivot_results() to compute statistical summaries, and
reduces the pivoted output using slicing to match the expectations of
compute_change(). The change status is extracted and reported as
regression, improvement, or no significant change.

This enables automated result analysis and verdict detection as part of
kernel regression bisection.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Aishwarya TCV authored Jun 12, 2025 and Ryan Roberts committed Jul 08, 2025

Update dispatch() of 'start' verb for preparing bisection inputs.
This routine loads resultstore tables and extracts kernel SHAs
for the given good and bad SWPROFILEs. If both SHAs match, the
process aborts early to avoid a meaningless bisection.

The SUT name is resolved by tracing the good SWPROFILE through the
RESULT and SUT tables. A check ensures the bad SWPROFILE was also
run on the same SUT. If not, bisection is aborted to maintain
consistency of execution environment.

A bisection template is then generated using the resolved SUT,
benchmark reference, and commit metadata.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Aishwarya TCV authored Jun 05, 2025 and Ryan Roberts committed Jul 08, 2025

Introduce initial structure for the `bisect` noun with `start` and `run`
verbs in the FastPath CLI. These will support automated kernel regression
bisection workflows.

- Adds CLI argument parsing for `fastpath bisect start`
- Adds CLI argument parsing for `fastpath bisect run`
- Registers both verbs under a new `bisect` noun
- Includes placeholder dispatch() methods for future logic

This sets up the interface and CLI plumbing in preparation for the
bisection plan generation and benchmark evaluation implementation.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

Introduce new utility functions `join_results()` and `filter_results()`
in fastpath/utils/table.py to encapsulate this shared logic. The former
takes the dictionary of resultstore tables and returns a single results
table with much of the common metadata joined in. The latter filters the
results table by sut/swprofile/benchmark/resultclass.

Update all previous call sites to use this new utility, reducing
redundancy and improving maintainability.

As a beneficial side effect, we clean up display of the resultclass in
the dashboard so remove the units. Units are still displayed on the
Y-axis of the graph.

Co-developed-by: Aishwarya TCV <aishwarya.tcv@arm.com>
Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>
Co-developed-by: Ryan Roberts <ryan.roberts@arm.com>
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Update benchmarks library with yaml file fragment for repro-collection
mysql workload. It defines the suite, name, & type of the benchmark.
Benchmark container image is already saved in gitlab registry.
This image path is specified in the yaml file.

Other params for repro-collection benchmark include:
- workload which is currently `mysql`
- sut_nr_cpus fixed at 16 to test workload correctly

Update Ubuntu license info along with Debian.
Docker image for repro-collection benchmark uses Ubuntu:22.04.

Signed-off-by: Aishwarya Rambhadran <aishwarya.rambhadran@arm.com>

Create an Ubuntu-based Docker container to run the aws/repro-collection
MySQL workload.
The repro-collection repository is cloned into the Docker container
at build time. SUT and Load Generator (LDG) installations for the
workload are also performed during the build process.

Apply patch to the repro-collection's main.sh.
This is to fix the number of SUT CPUs to an optimum value provided
via benchmark parameters. This helps to tune the benchmark to
simulate a bottleneck in the SUT.

Add wrapper script repro_collection.sh.
`service mysql stop` fails during repro_collection's SUT cleanup step.
Root cause was traced to the use of `debian-sys-maint` user credentials
from `/etc/mysql/debian.cnf` which were not available in the custom
MySQL setup within repro_collection.
New wrapper script adds a `mysql:post:configure:sut` hook to
repro_collection's main.sh script. It defines the commands to
fetch credentials from /etc/mysql/debian.cnf, create this user in
MySQL DB, set password & grant shutdown permission. This resolves
service shutdown issue cleanly.
The script also sources repro_collection's run.sh which starts
the workload execution.

Implement exec.py as the entry point of the docker container to
run repro_collection.
This invokes wrapper script repro_collection.sh and provides the
required input parameters as per the benchmark plan.
Once the workload completes, the results are parsed and populated
as Fastpath results.

Signed-off-by: Aishwarya Rambhadran <aishwarya.rambhadran@arm.com>

By default, merge request pipelines run with detached heads. This is
problematic for building the docs using ReadTheDocs because it requires
a branch name to map to a version in RTD. But with a detached head,
there is no branch.

So let's disable MR pipelines. When pushing changes to a branch that has
an MR, the branch pipeline will still run and the result shows up in the
MR.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The "human readable id" format, used to identify benchmarks, swprofiles
and suts in the command line interface, is usually just the name that
the user provided in the plan (in the case of swprofile and sut) or
"suite/name" in the case of benchmark. But these names are not
guaranteed to be unique. If there are multiple objects of the same type
with the same name in the resultstore, the we suffix ":<id>", where <id>
is the unique integer id in the resultstore. That ensures we always have
a unique and friendly way to refer to these objects.

But it turns out that "result merge" was buggy and did not correctly
filter for the "<name>:<id>" case. This was previously implemented with
extra, pre-calculated columns in the tables, which was intended to
simplify the filtering. But the columns were precalculated prior to
inserting the object so the id was None at that point, meaning the
column ended up with "<suite>/<name>:None" in the case of benchmark. For
sut and swprofile, the problem was even worse as the column was misnamed
due to a copy/paste error causing the column to always be NULL.

So let's get rid of those pre-computed columns and just complicate the
filter so that it can concatenate and compare on the fly.

We deliberately allow <name> to be provided as the filter even when
there are multiple objects with <name>. In this case the filter will
match all of them.

Fixes: a869c532 ("cli: Implement resultstore abstraction around database")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Some systems lock down grub.cfg so that it is only readable by root.
Therefore, on these systems, ctx.get() would fail because it runs as the
regular user. Work around this with "sudo cat" then capture the stdout.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

To comply with security requirements, we must rebuild the benchmark
containers regularly. We will do this using a gitlab scheduled pipeline.
To prepare for that, let's add the concept of a pipeline type via the
PIPELINE_TYPE variable. The existing type is now called "Continuous
Integration (CI)" and we introduce "Build Container (BC)", which when
invoked, will build a specified container tagged with the specified
version.

The scheduled pipelines will be setup to invoke this new type.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Let's parameterize the project name, just like we do for the domain and
token.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add a README.md landing page for gitlab which contains the key links for
the documentation, repo and container registry. Additionally add those
same links to the introduction page within the main docs.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Introduce basic documentation to describe what fastpath is, how to
install it and how to create and execute plans.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

To date, the term "config" has been used to refer to the SW
configuration of the SUT. This includes the kernel that's installed, the
kernel command line, sysctls, etc. But "config" is a very overloaded
term, and people were getting confused between the fastpath config and
the kernel (K) config.

So let's rename this key object to "swprofile". A SW profile is a lot
more self-descriptive and distinct from the myriad of other configs we
commonly talk about.

This is mostly just renaming within the code, but there are 3 places
were backwards compatibility is severely broken:

  - The CLI used to accept --config all over the place. It now expects
    --swprofile. So saved command invocations need to be updated.

  - The plan.yaml schema used to have a "configs" list. The list is now
    called "swprofiles". So existing plans need to be updated.

  - The resultstore schema used to have a CONFIG table. This is now
    renamed to SWPROFILE. Additionally the RESULT and ERROR tables used
    to have a column called config_id. That's renamed to swprofile_id.
    Existing resultstores (in all formats; CSV, MySQL, SQLite) will need
    to be modified by hand to work with the modified tooling.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Commit 754d6d6 ("cli: Generalize FASTPATH_LIBRARY into a key-value
store") and commit ef6cf61 ("cli: Add "config set" and "config get"
commands") introduced a general purpose "user config" key-value store
together with a command line interface ("config get"/"config set") to
access and modify the values.

But the term "config" is massively overloaded and confusing to users. So
let's rename to "preferences" instead. The cli noun is now called
"preference" and the peristent store file is renamed from
~/.fastpathconfig to ~/.fastpathpreferences.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The number of configs is becoming rather large in our internal database,
so let's add an option to filter the list of objects that are output
from "result list" by id.

For example:

$ fastpath result list --object config --id ryan-*

Will list all configs whose ID begins with "ryan-".

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We have had some issues with dodgy kernels falling over during benchmark
runs. But because we only have an SSH connection and don't have access
to the serial port, we can't see exactly what caused the panic.

Let's fix that by using netconsole to stream the kernel log over UDP to
the system running the fastpath driver. The log is then saved to a
per-session log file. If the machine panics we will see the panic log in
this file.

The file is named "session-<uuid>.kmsg" within the config directory.
Logging is best effort; it is fails to initialize we carry on.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

We have a problem; the same config, when installed on 2 different suts,
may have a slightly different fingerprint, so there will be 2 separate
entries in the database. For instance, if the rootfs location is
different, the kernel_cmdline_full_hash will be different between the 2
machines. And it's possible the userspace_name may differ if the Ubuntu
point release is different for example.

So let's disregard these small differences by default; we will still
keep the separate entries in the resultstore, but unless told otherwise,
we will merge similar configs and fix up all the references from the
other tables when we read the data in. This means we can easily compare
benchmarks across different suts. If the user knows better, they can opt
out of this merging behavior by specifying --no-merge-similar.

Additionally, let's also merge similar benchmarks so that the container
image can change without considering it a new benchmark by default.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

In the happy path we aim to avoid tearing down and restarting the
container between repeats of the same benchmark to speed up execution
time. But if there was an error on a previous run, let's force restart
the container to clean up any bad state that was left behind.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

In some very niche corner cases (devs wanting to run fastpath on a
system that initially has a very custom kernel installed), the initial
running kernel does not have sufficient Kconfigs enabled to run docker
(this is true for defconfig for example). Therefore "docker image pull"
would fail. Let's workaround this by installing the first config first
then we can do the "docker image pull" in the context of that kernel
which must support docker.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Aishwarya TCV authored Mar 31, 2025 and Ryan Roberts committed Jun 09, 2025

Introduces a new function `render_deviation_comparison_chart` to
visualize relative performance changes across configs or SUTs.

Users can select a baseline config or SUT via dropdown to compare
against others. The chart shows deviation bars, color-coded by
change type: green for improvements, red for regressions, and grey
for neutral changes within a configurable threshold (default 0.1%).

The logic aligns with `compute_change` from show.py and supports
relative/absolute views and filtering by change type via sidebar.

Signed-off-by: Aishwarya TCV <aishwarya.tcv@arm.com>

The logic that determines if there is a statistically significant change
between two results was previously embedded in pretty_results_multi().
We want to reuse this code in the dashboard, so let's factor it out.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

The benchmarks must be extracted from results prior to doing the pivot,
because "benchmark" is no longer a column after that.

Fixes: a832759 ("cli: Clarify SUT or config for "result show"")
Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

I've seen issues where put() fails with "OSError: Socket is closed"
after a previous timeout while running a command. I haven't been able to
reliably reproduce, but I suspect this could be due to a bug in fabric
(or paramiko). Attempt to work around this by explicitly disconnecting
and reconnecting to the SUT upon timeout (or any other exception).

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

I've had reports of a particular server system taking longer than 10
mins to reboot and the reboot timeout kicking in. As a quick fix, let's
increase the timeout to 15 mins. A better solution would be to make this
programmable via the plan. We can implement that if 15 mins proves to be
too short for some.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Commit d853eeb ("cli: Keep SSH connection alive") enabled an ssh
keepalive ping every 30 seconds to stop eager network applicances for
tearing down the connection. This solved the issues we were previously
seeing but I think I saw something similar recently. So let's increase
it to a 15 second ping to be safe.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

When showing results, add a title describing that SUT and/or config that
the results are for to make things easier for the user to understand.

Additionally when --verbose is provided, let's output a list of all the
SUTs, configs, and benchmarks that are considered in the results.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Aishwarya Rambhadran authored May 16, 2025 and Ryan Roberts committed Jun 09, 2025

Fastpath MySQL transactions failed on macOS as mysqlclient tried to
load native '.so' plugin files. These files may not be available in
MySQL installations via Homebrew, which caused
MySQLdb.OperationalError during `fastpath result list/show` commands.

To resolve this, the PyMySQL module is now used as the MySQL driver.
PyMySQL is compatible with SQLAlchemy when used via monkey-patching
with `pymysql.install_as_MySQLdb()`.

Changes:
- Add pymysql to Python requirements.txt
- Patch resultstore in fastpath to use PyMySQL
- Remove default-libmysqlclient-dev from Dockerfiles

Signed-off-by: Aishwarya Rambhadran <aishwarya.rambhadran@arm.com>

The Gigabyte AmpereOne-based R263-P30-AAH1 server has a BCM57414
NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller, which is not in the
defconfig, so let's add it to fastpath's standard kernel config.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

It's sometimes useful to have XFS support when we need a file system
that supports large folios. Since we are about to modify the standard
config to support AmpereOne and this will change the Kconfig hash, we
might as well add XFS at the same time.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Add simple commands to get and set user configuration data stored in
~/.fastpathconfig.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>

Let's allow users to store their default resultstore URL in
.fastpathconfig to avoid having to provide it on the command line all
the time. This is useful when using a central DB a lot.

Signed-off-by: Ryan Roberts <ryan.roberts@arm.com>