And so it begins ...

Merge-in the "next" branch, that contains the new version of LISA. The
old master can be found under the "legacy" branch that has been only
taking minor tweaks for quite a while now, while hot new stuff were
merged into next.

A transition guide listing major API changes and how to port code from
LISA legacy is available here:
https://lisa-linux-integrated-system-analysis.readthedocs.io/en/master/transition_guide.html

note: this is not a merge commit proper, in order to speed up git
subtree split. The commit of "next" branch merged is:
c543fb77 Merge pull request #871 from douglas-raillard-arm/_pr133

Merging to make sure we get clean test data. If systemd-based systems become used again, it will be time to fix the speed issue.

utils/energy_model: Fix loading EM from recent android-4.14 targets

Since commit 9001b8f1a,
"ANDROID: sched: Add support for frequency/power energy model"
android-4.14 has been able to load a frequency-based energy model.

Adding this extended the capacity_state structure in
sched_group_energy with a new frequency member. This is automatically
reflected in the energy model tuples printed in procfs and so Lisa can
no longer load sensible Energy models from any target running
android-4.14 later than the above commit, although the error is not
detected so you do get an EnergyModel object which contains junk
capacity and power values.

We can figure out which kind of kernel we have by reading the number
of capacity states for a cpu and check how many items are in the
cap_state tuples so that we can handle 2 or 3 values per state
transparently.

Signed-off-by: Chris Redpath <chris.redpath@arm.com>

Devlib has bug fixes we want to have in Lisa, so let's progress it to
the latest master.

Tested with:

  lisa-test -n 1 tests/eas/generic.py:EnergyModelWakeMigration

on Android and Linux targets. No regressions observed.

Signed-off-by: Quentin Perret <quentin.perret@arm.com>

ifconfig returns
 inet adr:172.17.0.1
instead
 inet addr:172.17.0.1
when language is set to fr-FR

update the sed command to support it

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>

If no kernel path has been defined, kernel_refs is empty and
df['kernel_sha1'].replace(kernel_refs) fails

Add sha1 as kernel ref in case no kernel path is provied

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>

Update all submodules to latest origin/master

1. pcmark needs internet access, so disable airplane mode
2. use v2.4 of exoplayer inline with sched-evaluation-full.yaml
3. s/test/test_ids/ in jankbench attributes

Signed-off-by: Qais Yousef <qais.yousef@arm.com>

We need some fixes on devlib and workload-automation. Namely:

1. ssh leak fix in devlib
2. logcat crashes on workload-automation
3. pcmark support for android Q on workload automation

And generally we should try to regularly stay up-to-date to avoid
diverging too much.

Signed-off-by: Qais Yousef <qais.yousef@arm.com>

Since all commands part of the systemd package are named
systemd-<something>, we need a more refined pattern in order to properly
freeze systemd-journald process for example.

Add a heads-up for the changes that are coming soon™.

tests/eas/generic.py: Tweak RampDown energy threshold

The RampDown test happens to fail on TC2 with a 16% or 17% energy delta,
while the behaviour is clearly correct. This happens because of the
overutilized band. The exact location of that band depends on the
capacity ratio of CPUs of the platform, but the RampDown test
configuration is hardcoded, without taking the platform's topology into
account. As a result, the test is hitting more sensible part of the
power-perf curve on some targets than others, and that is the cause of
the problems observed on TC2.

Fix this by tweaking the threshold at a level that should suit TC2
better. A proper fix would be to compute the threshold depending on the
EM of the target, but this is difficult to do in the current state of
the test framework. The new test framework should handle these scenarios
a lot better, so implementing 'the right thing' should be easier then.
In the meantime, the threshold tweak should workaround the issue just
fine.

Signed-off-by: Quentin Perret <quentin.perret@arm.com>

This test suite targets the behaviour of misfit tasks.

For now it only has the "staggered finishes" test case, which consists
of NR_CPUS 100% tasks, briefly pinned to one CPU to ensure there is one
task per CPU.
The tasks running on the big CPUs should complete first, and at this
point we expect them to pull the tasks on the LITTLEs via idle balance.

The tests associated with this workload are:

- test_preempt_time:
    Asserts that tasks are not preempted for more than 1% of the
    test duration, since each task should get its own CPU.

- test_migration_delay:
    Asserts that big CPUs pull tasks ASAP after their initial task
    is completed.

- test_throughput
    Asserts that big CPUs are never idle for more than 1ms when there
    are tasks running the LITTLEs.

======
Known issues:

Because there can be unwanted processes (e.g. sshd) popping up at the
wrong time, causing the imbalance to be of type group_overloaded
instead of group_misfit_task, the misfit logic can be skipped in the
load balance. This is something that should eventually be fixed.

Also, test_migration_delay is just a subset of test_throughput. However,
because of the group_overloaded issues, tasks can still move from big
to LITTLE CPUs, and it takes >1ms to move them back, causing
test_throughput to fail.

test_migration_delay is here to at least ensure the newidle pull
happened, since test_throughput is going to fail a lot in the
foreseeable future.

This method takes in a dataframe with a time delta column (as created by
addEventsDeltas), and slices it to fit within a given interval.

Instead of writing a wall of text, here's what it's doing:

df is:
Time len state
15    1   1
16    1   0
17    1   1
18    1   0
-------------

squash_df(df, 16.5, 17.5) =>

Time len state
16.5  .5   0
17    .5   1

squash_df(df, 16.2, 16.8) =>

Time len state
16.2  .6   0

This computes the time delta between rows of a dataframe and adds it
to a new column. This can be helpful for events that track an on/off
state, e.g. overutilized. This commit replaces the current delta
computation for sanitize_SchedOverutilized() by this helper.

Use the frequency behaviour when running rt and cfs tasks
to determine if the schedtune hold functionality is working
properly.

Signed-off-by: Chris Redpath <chris.redpath@arm.com>

Ideally, one would rewrite the monsoon devlib instrument to provide the
data in a form which allows the DerivedEnergyMeasurement class to give
summed energy & power values. However, the monsoon python driver does
not appear to be written in such a way that this could be done easily.

For the time being, lets make our own combosite measurement here if it
not supplied by Devlib and we have the known monsoon EM channels.

Signed-off-by: Chris Redpath <chris.redpath@arm.com>

The v5 of the simplified EM doesn't expose the 'capacity' file in sysfs.
Fix this by checking if the file exists, and re-compute those values
form within LISA if not.

Signed-off-by: Quentin Perret <quentin.perret@arm.com>

Recent EAS versions (with the simplified EM) do not rely on the
SD_SHARE_CAP_STATES flag to be set anywhere. As a result, the
preliminary test checking its presence fails, although the kernel
configuration is fine.

Fix this by making preliminary.py check the kernel version to skip
the test if it doesn't make sense.

Signed-off-by: Quentin Perret <quentin.perret@arm.com>

Fix cpuhp test

The CPU hotplug test is full of useless dependencies which silently talk
to the target (rt-app, freezing userspace, ftrace, ...). When the target
crashes during the hotplug stress, requests sent to the target because
of the above modules can hang indefinitely, or fail miserably. That is
an issue since, in the specific case of the hotplug test, we would like
to classify these crashes as test failures.

Remove all the dependencies by overwritting the runExperiments method of
LisaTest in order to avoid using an Executor. The test spawns a bash
script on the target that plugs CPUs in and out for a specific duration,
and check on the host side if the target is still alive after a while.

Signed-off-by: Quentin Perret <quentin.perret@arm.com>

The slack test case allows a task slack error percentage of 15%. In case
a test case passes it is impossible to know if it did so with 0% or
almost 15% error rate.

Log the slack percentage value per task to make even in case the task
stays under the slack error percentage to make it easier to post process
a test run.

Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>

The new Energy Model/EAS mainline code does not use the ENERGY_AWARE
sched feature any more.

Disable it temporarily. We don't have feedback from lkml about the new
Energy Model/EAS mainline code yet so I would like to avoid to change
the test code permanently.

Once we can be more certain that this sched feature is not needed any
more it would have to be removed.

This will allow that the generic tests run against a target with the new
Energy Model/EAS mainline code.

Signed-off-by: Dietmar Eggemann <dietmar.eggemann@arm.com>

With the Simplified Energy Model, the location of the EM in sysfs has
changed.

Fix the test in preliminary.py to support both flavors by checking the
two possible paths.

Signed-off-by: Quentin Perret <quentin.perret@arm.com>

When the ENERGY_AWARE sched_feature is present in the kernel, it should
be set before starting EAS tests.

Check if the configuration is correct in the 'preliminary.py' test.

Signed-off-by: Quentin Perret <quentin.perret@arm.com>

with sched-full-evaluation.ipnb, plot_comparisons returns an error in my
setup (vagrant vm image provided by the tools)

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>

Add a new parameter to WaResultsCollector to skip all kind of charts
rendering. This helps to easily disable any kind of graphic rendering when
the class is used to only get raw data results in a csv file as an example
and doesn't want or support graphic rendering

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>

Handle correctly when an agenda hasn't been fully executed or when some
iterations have empty file because something wrong happened during the
test.
Instead of stopping and returning error, WaResultsCollector continues and
post processes available files and results

Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>

Accidentally, the original patch proposed in #608 has been merged on
top of the series #602 but that version was missing the fixups suggested
by @valschneider in the original PR.

Let's fix this by adding the proposed changes now.

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>

Devlib has a timeout parameter for the execute() API. Expose this
parameter in the run() method of target_script in order to let user get
a TimeoutException whenever a script runs for longer than expected.

Signed-off-by: Quentin Perret <quentin.perret@arm.com>

Misc fixes

This will fix some trace errors we've been getting.

Right now, the plotLatency method returns a DF with statistics for the
latencies of a specified task. However, sometimes we could be
interested in getting these stats without necessarily plotting all the
data.

Let's move the statistics DF generation into an internal _get_latency_df
method, which can then be used by the plotLatency but also from an
additional DF gatter: _dfg_latency_stats_df.
Thus, this will make available a new:

   trace.data_frame.latency_stats_df()

which just return the overall latency stats for the specified task.

Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>