From f650a230c0af813aa54233730a15aec800b57da0 Mon Sep 17 00:00:00 2001 From: Ryan Roberts Date: Tue, 31 Jan 2023 12:10:51 +0000 Subject: [PATCH 1/6] config: Change linux-base.yaml to not build modules by default Previously linux-base.yaml would build modules by default, and require a higher layer to specify `export BUILD_KMODULES=false` to disable this behaviour. Since building modules is time-consuming and rarely actually needed, switch the default to disabled, and allow higher layers to opt-in. Signed-off-by: Ryan Roberts --- config/linux-base.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/config/linux-base.yaml b/config/linux-base.yaml index bb31dad..94fb4b1 100644 --- a/config/linux-base.yaml +++ b/config/linux-base.yaml @@ -6,7 +6,8 @@ description: >- Linux kernel build config. Builds the kernel image and set of modules. Image is exported as the KERNEL artifact, while the modules are exported in a tgz - archive as the KMODULES artifact. + archive as the KMODULES artifact. Modules are only built if a higher level + layer specifies `export BUILD_KMODULES=true` in the prebuild section. By default the defconfig is built. Users can optionally add commands to the prebuild list to modify the config (e.g. `./scripts/config ...`) or even @@ -21,7 +22,7 @@ build: toolchain: aarch64-none-elf- prebuild: - - export BUILD_KMODULES=true + - export BUILD_KMODULES=false - export ARCH=arm64 - make -j${param:jobs} O=${param:builddir} defconfig -- GitLab From 464933ab4780c27c71cd205af2a2e1e1c8dda46f Mon Sep 17 00:00:00 2001 From: Ryan Roberts Date: Tue, 31 Jan 2023 13:21:48 +0000 Subject: [PATCH 2/6] config: Add Arm CCA demonstrator config Brings together a software stack to demonstrate Arm CCA running on FVP in a four-world configuration. Includes TF-A in root world, Hafnium and some demo secure partitions in secure world, RMM in realm world, and Linux in Normal world. In order to launch realm VMs, the user must bring their own rootfs that contains a realm-aware kvmtool and a payload suitable for execution in a realm. This could be kvm-unit-tests or a copy of the kernel running as host (since it also supports the RSI guest additions). Signed-off-by: Ryan Roberts --- config/cca.yaml | 123 +++++++++++++++++++++++++++++++++++++++++ config/rmm-base.yaml | 24 ++++++++ docker/Dockerfile.slim | 12 ++++ 3 files changed, 159 insertions(+) create mode 100644 config/cca.yaml create mode 100644 config/rmm-base.yaml diff --git a/config/cca.yaml b/config/cca.yaml new file mode 100644 index 0000000..95a1ef0 --- /dev/null +++ b/config/cca.yaml @@ -0,0 +1,123 @@ +# Copyright (c) 2022, Arm Limited. +# SPDX-License-Identifier: MIT + +%YAML 1.2 +--- +description: >- + Brings together a software stack to demonstrate Arm CCA running on FVP in a + four-world configuration. Includes TF-A in root world, Hafnium and some demo + secure partitions in secure world, RMM in realm world, and Linux in Normal + world. + + In order to launch realm VMs, the user must bring their own rootfs that + contains a realm-aware kvmtool and an RSI-aware guest kernel image. + +concrete: true + +layers: + - tfa-base.yaml + - rmm-base.yaml + - hafnium-base.yaml + - linux-base.yaml + - FVP_Base_RevC-2xAEMvA-base.yaml + - arch/v9.2.yaml + +build: + tfa: + params: + ENABLE_RME: 1 + RMM: ${artifact:RMM} + CTX_INCLUDE_EL2_REGS: 1 + SPD: spmd + SPMD_SPM_AT_SEL2: 1 + SP_LAYOUT_FILE: ${artifact:SP_LAYOUT} + BL32: ${artifact:HAFNIUM} + ARM_LINUX_KERNEL_AS_BL33: 1 + PRELOADED_BL33_BASE: 0x84000000 + + # SME is not supported concurrently with CCA yet. + ENABLE_SME_FOR_NS: 0 + ENABLE_SME_FOR_SWD: 0 + ENABLE_FEAT_HCX: 0 + + linux: + repo: + remote: https://git.gitlab.arm.com/linux-arm/linux-cca.git + revision: cca-full/rfc-v1 + + prebuild: + # Disable CPU_IDLE as a workaround to speed up the FVP. Since we are using + # the TF-A DT, which provides CPU idle state parameters, it otherwise + # causes Linux to constantly enter cpu idle, slowing the FVP down. We + # can't easily use the upstream DT right now, due to some RAM having been + # carved out for the RMM and this is not reflected in that DT. CPU_IDLE is + # selected by ACPI, so we have to disable that too to maintain a legal + # config. That's OK for now since we are using the DT. + - ./scripts/config --file ${param:builddir}/.config --disable CONFIG_ACPI + - ./scripts/config --file ${param:builddir}/.config --disable CONFIG_CPU_IDLE + + # Provides secure partitions that run under Hafnium for demonstration. + tfa-tests: + repo: + remote: https://git.trustedfirmware.org/TF-A/tf-a-tests.git + revision: v2.8 + + toolchain: aarch64-none-elf- + + params: + PLAT: fvp + DEBUG: 1 + + build: + # TODO: tfa-tests has makefile dependency bug that prevents parallel make (-j > 1). + - make BUILD_BASE=${param:builddir} ${param:join_equal} all pack_realm + + clean: + - make BUILD_BASE=${param:builddir} realclean + + artifacts: + SP_LAYOUT: ${param:builddir}/fvp/debug/sp_layout.json + +run: + rtvars: + BL1: + type: path + value: ${artifact:BL1} + + FIP: + type: path + value: ${artifact:FIP} + + KERNEL: + type: path + value: ${artifact:KERNEL} + + ROOTFS: + type: path + value: '' + + params: + -C bp.secureflashloader.fname: ${rtvar:BL1} + -C bp.flashloader0.fname: ${rtvar:FIP} + -C bp.virtioblockdevice.image_path: ${rtvar:ROOTFS} + --data cluster0.cpu0: ${rtvar:KERNEL}@0x84000000 + -C cluster0.rme_support_level: 2 + -C cluster1.rme_support_level: 2 + + # SME is not supported concurrently with CCA yet. + -C SVE.ScalableVectorExtension.has_sme: 0 + + + terminals: + bp.terminal_0: + friendly: tfa+linux + type: stdinout + + bp.terminal_1: + friendly: tfa-rt + + bp.terminal_2: + friendly: hafnium + + bp.terminal_3: + friendly: rmm diff --git a/config/rmm-base.yaml b/config/rmm-base.yaml new file mode 100644 index 0000000..3b4a1d8 --- /dev/null +++ b/config/rmm-base.yaml @@ -0,0 +1,24 @@ +# Copyright (c) 2022, Arm Limited. +# SPDX-License-Identifier: MIT + +%YAML 1.2 +--- +build: + rmm: + repo: + remote: https://git.trustedfirmware.org/TF-RMM/tf-rmm.git + revision: tf-rmm-v0.2.0 + + toolchain: aarch64-none-elf- + + params: + -DRMM_CONFIG: fvp_defcfg + -DCMAKE_BUILD_TYPE: Release + -DLOG_LEVEL: 40 + + build: + - cmake ${param:join_equal} -S . -B ${param:builddir} + - cmake --build ${param:builddir} -j ${param:jobs} + + artifacts: + RMM: ${param:builddir}/rmm.img diff --git a/docker/Dockerfile.slim b/docker/Dockerfile.slim index db2a93c..f55ea03 100644 --- a/docker/Dockerfile.slim +++ b/docker/Dockerfile.slim @@ -70,6 +70,18 @@ RUN apt-get install --assume-yes --no-install-recommends --option=debug::pkgProb python3-distutils \ uuid-dev +# Install packages requried by RMM. +# From https://tf-rmm.readthedocs.io/en/latest/getting_started/getting-started.html. +RUN apt-get install --assume-yes --no-install-recommends --option=debug::pkgProblemResolver=yes \ + build-essential \ + git \ + make \ + ninja-build \ + python3 \ + python3-pip +RUN pip3 install \ + cmake + # TODO: Install any packages required by U-Boot, OP-TEE, Trusty, etc. # Install the aarch64-linux-gnu- toolchain. We use the stock Debian packages for -- GitLab From 8d25a74053dc2368ed324c13c1fee366dbe66960 Mon Sep 17 00:00:00 2001 From: Ryan Roberts Date: Fri, 3 Feb 2023 10:24:06 +0000 Subject: [PATCH 3/6] config: Extend cca.yaml to also build kvmtool VMM The binary is built and added to the package, but it is the user's responsibility to add it to the rootfs. Signed-off-by: Ryan Roberts --- config/cca.yaml | 9 +++++++++ config/kvmtool-base.yaml | 37 +++++++++++++++++++++++++++++++++++++ docker/Dockerfile.slim | 5 +++++ 3 files changed, 51 insertions(+) create mode 100644 config/kvmtool-base.yaml diff --git a/config/cca.yaml b/config/cca.yaml index 95a1ef0..1f55868 100644 --- a/config/cca.yaml +++ b/config/cca.yaml @@ -19,6 +19,7 @@ layers: - rmm-base.yaml - hafnium-base.yaml - linux-base.yaml + - kvmtool-base.yaml - FVP_Base_RevC-2xAEMvA-base.yaml - arch/v9.2.yaml @@ -78,6 +79,14 @@ build: artifacts: SP_LAYOUT: ${param:builddir}/fvp/debug/sp_layout.json + kvmtool: + repo: + dtc: + revision: v1.6.1 + kvmtool: + remote: https://gitlab.arm.com/linux-arm/kvmtool-cca + revision: cca/rfc-v1 + run: rtvars: BL1: diff --git a/config/kvmtool-base.yaml b/config/kvmtool-base.yaml new file mode 100644 index 0000000..4ad3412 --- /dev/null +++ b/config/kvmtool-base.yaml @@ -0,0 +1,37 @@ +# Copyright (c) 2022, Arm Limited. +# SPDX-License-Identifier: MIT + +%YAML 1.2 +--- +description: >- + kvmtool is a lightweight Virtual Machine Monitor that can be used, in concert + with KVM, to create and manage virtual machines. This config fragment can be + used to build the kvmtool binary, which is exported as an artifact called + LKVM. Fancy optional extras which depend on 3rd party libraries are not + enabled. + +build: + kvmtool: + repo: + dtc: + remote: git://git.kernel.org/pub/scm/utils/dtc/dtc.git + revision: master + kvmtool: + remote: git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git + revision: master + + toolchain: aarch64-linux-gnu- + + build: + - export CC=$${CROSS_COMPILE}gcc + - export ARCH=arm64 + - export LIBFDT_DIR=${param:sourcedir}/dtc/libfdt + - make -j${param:jobs} -C dtc libfdt + - make -j${param:jobs} -C kvmtool + + clean: + - make -j${param:jobs} -C dtc clean + - make -j${param:jobs} -C kvmtool clean + + artifacts: + LKVM: ${param:sourcedir}/kvmtool/lkvm \ No newline at end of file diff --git a/docker/Dockerfile.slim b/docker/Dockerfile.slim index f55ea03..7898a7d 100644 --- a/docker/Dockerfile.slim +++ b/docker/Dockerfile.slim @@ -82,6 +82,11 @@ RUN apt-get install --assume-yes --no-install-recommends --option=debug::pkgProb RUN pip3 install \ cmake +# Install packages requried by kvmtool. +RUN apt-get install --assume-yes --no-install-recommends --option=debug::pkgProblemResolver=yes \ + build-essential \ + pkg-config + # TODO: Install any packages required by U-Boot, OP-TEE, Trusty, etc. # Install the aarch64-linux-gnu- toolchain. We use the stock Debian packages for -- GitLab From c184409059cfcc0da1f6c6a5635fef279c8e05ee Mon Sep 17 00:00:00 2001 From: Ryan Roberts Date: Thu, 9 Feb 2023 16:28:27 +0000 Subject: [PATCH 4/6] config: Extend cca.yaml to also build kvm-unit-tests VMM The binary fw images are built, packaged into a tar and added to the package, but it is the user's responsibility to add it to the rootfs. Signed-off-by: Ryan Roberts --- config/cca.yaml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/config/cca.yaml b/config/cca.yaml index 1f55868..3b1b3c8 100644 --- a/config/cca.yaml +++ b/config/cca.yaml @@ -87,6 +87,29 @@ build: remote: https://gitlab.arm.com/linux-arm/kvmtool-cca revision: cca/rfc-v1 + kvm-unit-tests: + repo: + remote: https://gitlab.arm.com/linux-arm/kvm-unit-tests-cca + revision: cca/rfc-v1 + + toolchain: aarch64-linux-gnu- + + params: + --arch: arm64 + --cross-prefix: $${CROSS_COMPILE} + --target: kvmtool + + build: + - ./configure ${param:join_equal} + - make -j${param:jobs} + - tar -caf ${param:builddir}/kvm-unit-tests.tgz -C ${param:sourcedir} . + + clean: + - make -j${param:jobs} clean + + artifacts: + KVM_UNIT_TESTS: ${param:builddir}/kvm-unit-tests.tgz + run: rtvars: BL1: -- GitLab From bcb1b78df1886022a3a4555aa14a4787a2707adc Mon Sep 17 00:00:00 2001 From: Ryan Roberts Date: Fri, 10 Feb 2023 13:10:57 +0000 Subject: [PATCH 5/6] test: Build and run cca config Signed-off-by: Ryan Roberts --- test/test.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/test/test.py b/test/test.py index ec9c25f..1938aad 100755 --- a/test/test.py +++ b/test/test.py @@ -171,6 +171,9 @@ def do_main(smoke_test): build_configs(['bootwrapper.yaml'], arch) run_config_bootwrap('bootwrapper.yaml', BOOTWRAPPER, ROOTFS, arch) + build_configs(['cca.yaml']) + run_config_kern('cca.yaml', KERNEL, ROOTFS) + print_results() -- GitLab From ec1a6465a64bea565ae937a90ddc50feab062c87 Mon Sep 17 00:00:00 2001 From: Ryan Roberts Date: Fri, 10 Feb 2023 13:11:43 +0000 Subject: [PATCH 6/6] test: Fix failing bootwrapper tests Bootwrapper runtime tests were previously failing due to bootwrapper's PSCI implementation not shutting down the FVP on request. So the FVP would just hang at shutdown and the test would timeout. It turns out that the FVP has a mechanism where it can search for a specific piece of text from a uart and will shut itself down when it sees it. So let's use this mechanism to look for the "reboot: System halted" message that Linux outputs on shutdown. This fixes the test timeouts. Signed-off-by: Ryan Roberts --- config/bootwrapper.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/config/bootwrapper.yaml b/config/bootwrapper.yaml index 8d7ecde..f1f7e30 100644 --- a/config/bootwrapper.yaml +++ b/config/bootwrapper.yaml @@ -32,6 +32,7 @@ run: -C bp.secure_memory: 0 -a cluster*.cpu*: ${rtvar:BOOTWRAPPER} -C bp.virtioblockdevice.image_path: ${rtvar:ROOTFS} + -C bp.pl011_uart0.shutdown_tag: '"reboot: System halted"' terminals: bp.terminal_0: -- GitLab