diff --git a/README.md b/README.md index 3288a5c7902003cb08c3d095abbe20afdc058607..3bb621627a67bd4e60f0dbdfa82dc997c9f610a9 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,30 @@ # Template Structure for SystemReady ES or SR Compliance Reports -This repo structure is the template for collecting compliance evidence for a SystemReady ES or SR certification. +This repository structure contains the template for collecting compliance testing data that is required for Arm SystemReady ES or SR certifications. [systemready-es-sr-template]: https://gitlab.arm.com/systemready/systemready-es-sr-template ## Prerequisites: - Check the documents below for basic information: - Arm SystemReady Requirements Specification - https://developer.arm.com/documentation/den0109/latest - SystemReady ES Test and Certification Guide - https://developer.arm.com/documentation/102529/0100/Test-SystemReady-ES-compliance -- Flash/update the firmware again to clear UEFI variable store/NVRAM right before running the tests, which will avoid fake failures that are caused by using up the UEFI variable store/NVRAM space. -- It is strongly recommended to flash ACS image to a USB disk enclosure with a fast SSD or SATA drive or a directly attached SATA/NVMe drive instead of USB stick, which would significantly shorten the testing time and will be more stable than using USB stick/flash drive. - - Running SCT with a USB stick may take more than 10 hours, and may run into some unexpected problem caused by USB HW issue. In the worst case, the USB stick may be bricked becasue of so many writes to the USB stick. -- Go to UEFI/BIOS setup menu to change your system’s "OS hardware description" option to "ACPI mode". You may find the option in Device Manage > O/S Hardware Description Selection. -- Go to UEFI/BIOS setup menu to move the ACS media device (USB drive) to the top of the boot order. -- Install network cable to all network ports and connect to a DHCP server for network related testing. - +- Make sure to update the firmware on the system and reset any UEFI variables/NVRAM settings to default before running the tests. This is needed to avoid false positives when the UEFI nopn-volatile variable store is full. +- It is strongly recommended to use a fast storage media (NVMe, SSD or SATA drive) when running the SystemReady ACS test suite. + - Using USB media will significantly increase the testing time and reduce the stability of the test run. + - Running the UEFI SCT part of the ACS test suite from USB media may take more than 10 hours, and may result into unexpected problems. In some extreme cases, the USB media may be worn out from very frequent writes. +- Enable the UEFI firmware ACPI mode, and disable DeviceTree mode if it exists. + - The setting may be found in the UEFI/BIOS setup mene, under Device Manage > O/S Hardware Description Selection, or similar names. +- Change the UEFI Boot Order in the Setup menu to make sure the media that contains the SystemReady ACS test suite is at the top of the boot order. +- Install a network cable to all network ports in the system, and connect to a DHCP server for network related testing. +- When running UEFI SCT manually, always run "connect -r" from the UEFI shell first. This ensures that all UEFI drivers are loaded, and necessary UEFI protocols present before testing starts. ## General Instructions -General instructions for collecting SystemReady ES or SR compliance logs: +Please follow the following general instructions for collecting SystemReady ES or SR compliance logs ### `./report.txt` -Fill in with information about the system being certified +- Fill in with information about the system being certified ### `./acs-auto-results/` -- Place the automation test result (Copy the folders below in acs_results in `RESULT` partitions in ACS USB drive/stick to this acs-auto-results folder): +- Place the ACS automation test results in this folder. This can be a copy the folders from the `RESULT` partitions in ACS media: ``` /acs_results @@ -37,60 +39,55 @@ Fill in with information about the system being certified ### `./manual-results/` -- Place any results from manually performed tests. +- Optionally place any results from manually performed tests in this folder. +- This is not needed if the automated test completes without issues and the acs_results are copied above. ### `./manual-results/bsa-linux/` -- If you're running the SR ACS, please rename this folder from 'bsa-linux' to 'sbsa-linux' -- Place any BSA or SBSA results (e.g. serial console log) collected manually from Linux (Linux BusyBox). -- Note that if automation test works fine, you won't need to run manul test and put any file in this folder. +- Place any BSA or SBSA results collected manually from Linux (BusyBox) in this folder. This can be for instance the serial console output from running the commands below. -- Examples for SR ACS: +- Examples of SBSA Linux command usage: ``` - Complete SBSA Linux results can be collected by running the following commands after booting Linux BusyBox insmod /lib/modules/sbsa_acs.ko sbsa - You can include verbose output (-v): - sbsa -l 3 -v 2 + You can include verbose output (-v) and run the test in different SBSA levels (-l): + sbsa -l 3 -v 1 To skip a specific problematic test, use -skip - sbsa -l 3 -v 2 -skip 413 + sbsa -l 3 -v 1 -skip 413 ``` -- Examples for ES ACS +- Examples of BSA Linux command usage: ``` - Complete BSA UEFI results can be collected by running the following command after booting Linux BusyBox: + Complete BSA Linux results can be collected by running the following command after booting Linux BusyBox: insmod /lib/modules/bsa_acs.ko bsa You can include verbose output (-v): bsa -v 1 - To skip a specific problematic tests, use -skip + To skip a specific problematic test, use -skip bsa -skip 104 ``` ### `./manual-results/bsa-uefi/` +- Place any BSA or SBSA results (e.g. serial console log) collected manually from UEFI Shell in this folder. -- If you're running the SR ACS, please rename this folder from 'bsa-uefi' to 'sbsa-uefi' -- Place any BSA or SBSA results (e.g. serial console log) collected manually from UEFI Shell. -- Note that if automation test works fine, you won't need to run manul test and put any file in this folder. - -- Examples for SR ACS: +- Examples of SBSA UEFI command usage: ``` Complete SBSA UEFI results can be collected by running the following commands: FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi - You can include verbose output (-v): - FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -v 1 + You can include verbose output (-v) and run the test in different SBSA levels (-l): + FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -l 3 -v 1 - To skip a specific problematic tests, use -skip , + To skip multiple problematic tests, use -skip , FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -skip 448,449 ``` -- Examples for ES ACS: +- Examples of BSA UEFI command usage: ``` Complete BSA UEFI results can be collected by running the following command: FS1:\EFI\BOOT\bsa> bsa.efi @@ -98,56 +95,54 @@ Fill in with information about the system being certified You can include verbose output (-v): FS1:\EFI\BOOT\bsa> bsa.efi -v 1 - To skip a specific problematic tests, use -skip + To skip a specific problematic test, use -skip FS1:\EFI\BOOT\bsa> bsa.efi -skip 606 ``` ### `./manual-results/fwts/` - -- Place any additional FWTS results collected manually from Linux (Linux BusyBox): -- Note that if automation test works fine, you won't need to run manul test and put any file in this folder. +- Place any additional FWTS results collected manually from Linux (BusyBox) in this folder. - Examples: ``` - Complete SBSA UEFI results can be collected by running the following commands: - fwts -r stdout -q --uefi-set-var-multiple=1 --uefi-get-mn-count-multiple=1 --sbbr esrt uefibootpath + Complete FWTS results can be collected by running the following command: + fwts -r stdout -q --uefi-set-var-multiple=1 --uefi-get-mn-count-multiple=1 --sbbr esrt uefibootpath - To skip a specific problematic tests, use --skip-test= + To skip a specific problematic tests use --skip-test= fwts -r stdout -q --sbbr --skip-test=dmicheck fwts -r stdout -q --sbbr --skip-test=uefirtmisc,uefirtvariable,uefirttime ``` ### `./manual-results/sct/` - -- Place any additional SCT results collected manually from UEFI shell: -- Note: If automation test works fine, you won't need to run manul test and put any file in this folder. -- Note: Before manually running SCT, please run "connect -r" in UEFI shell first. - - +- Place any additional UEFI SCT results collected manually from the UEFI shell in this folder. - Examples: ``` - Complete SCT results can be collected by running the following commands + Complete SCT results can be collected by running the following command: FS1:\EFI\BOOT\bbr\SCT>SCT.efi -s SBBR.seq - Specific SCT test's result can be collected by running the following commands to use the Menu-Driven Interface - For details, please check the section 2.2 in https://github.com/tianocore/edk2-test/blob/master/uefi-sct/Doc/UserGuide/UEFI_SCT_II_UserGuide_2_6_A.pdf + The result of the specific SCT test case can be collected by running the following commands to use the Menu-Driven Interface + For details, please check section 2.2 at https://github.com/tianocore/edk2-test/blob/master/uefi-sct/Doc/UserGuide/UEFI_SCT_II_UserGuide_2_6_A.pdf FS1:\EFI\BOOT\bbr\SCT>SCT.efi -u ``` ### `./docs/` -- Place any firmware or device documentation, manuals, user guides, build instructions, etc.. +- Place any firmware or device documentation, manuals, user guides, build instructions, etc... -### `./fw/` +### `./fw/fw-image-files/` +- Place FW binary and/or source trees. Include information on the additional firmware (BMC, SCP, MCP, etc...) if applicable. #### `./fw/screenshots/` -- Place some screenshots showing the FW menus (such as UEFI Setup, BMC web console, uboot shell) +- Place some screenshots showing the FW menus (such as the UEFI Setup, BMC web console, uboot shell, etc...) if applicable. -#### `./fw/uefi-sniff.log` -- Run the following commands in UEFI Shell and attach the logs (if not already done by ACS). The UEFI Shell can be run from the ACS image by pressing a key at the UEFI timeout prompt before tests begin to execute. +#### `./fw/uefi-shell-cmds-logs/` +- Run the UEFI Shell commands below and attach the logs. +- The UEFI Shell can be run from the ACS test suite by interrupting the automated run, and pressing a key at the UEFI timeout prompt before the tests begin to execute. ``` Shell> connect -r + Shell> map -r + Shell> ver + Shell> dmem Shell> pci Shell> drivers Shell> devices @@ -163,33 +158,37 @@ Fill in with information about the system being certified Shell> acpiview -s SSDT -d (if SSDT table is present) Shell> bcfg boot dump -v Shell> ifconfig -l - Shell> ifconfig -s eth0 dhcp (If eth0 still hasn’t get IP, run "connect -r" again) + Shell> ifconfig -s eth0 dhcp (If eth0 doesn't display an updated IP address, run "connect -r" again) Shell> ifconfig -l eth0 Shell> ping xxx.xxx.xxx.xxx (If don’t know any other system’s IP address, you can get DNS Server's IP address from ifconfig output and ping it) ``` ### `./os-logs/` -- Please check ./os_logs/OS_image_download_links.txt to get the latest release for testing +- Please check ./os_logs/OS_image_download_links.txt to get the latest publicly available AARCH64 OS images for testing. #### `./os-logs/[distroname] [distroversion]/` -- Please rename the subfolders. For example, rename 'linux_distro_1' to 'Fedora Server 35'. -- It is recommended to choose one OS per group below (totally 4 OSes) for certification testing to widely cover all the kernel versions and kernel bases. - - Groups of the heritage: RHEL/Fedora/CentOS/AlmaLinux, or SLES/openSUSE, or Ubuntu/Debian, or NetBSD/OpenBSD/FreeBSD. - - For example, you can choose Fedora, openSUSE, Ubuntu, and FreeBSD for your testing. As for more information about OS reference for specific band or condition, please check ./os_logs/OS_image_download_links.txt. - - Moreover, it would be good to test OS as many as possible for showing a better result on Arm SCL web even if the OSes are in the same group. -- Install the OS to a disk, and boot it. (Note that a network controller (PCIe or USB) is needed.) -- The install log must begin when the platform is released from reset and must include: - - All firmware output - - Output from Linux installer - - Output of reboot into installed OS. -- Run the following commands and attach the logs: (For example, save dmesg messages to dmesg.txt) - - Note that some commands may not exist on some distros, and you may need to install the command or run an alternative command. +- Please rename the subfolders to indicate the OS name and version. For example, rename 'linux-distro1-version' to 'Fedora Server 37'. +- It is recommended to choose one OS per group below, an OS based on an old kernel, and an OS based on an kernel version close to the latest kernel for certification testing to cover as many kernel versions as possible. + - Groups based on heritage: RHEL/Fedora/CentOS/AlmaLinux, SLES/openSUSE, Ubuntu/Debian, or NetBSD/OpenBSD/FreeBSD. + - For example: + - For SystemReady SR, You can choose RHEL, SLES, Ubuntu, CentOS Linux 7, and Fedora for your testing. + - For SystemReady ES, you can choose Fedora, openSUSE, Ubuntu, and CentOS 7 for your testing. +- For each OS being tested + - Install the OS from some media (eg. USB drive) to a disk (e.g. SATA, NVMe, USB), and boot it. + - Note that a network controller (PCIe or USB) may be needed during installation. + - The install log must begin when the platform is released from reset and must include: + - All firmware output + - Output from the Linux installer + - Output of reboot to the installed OS. + - Run the following commands and attach the logs: (For example, save dmesg messages to dmesg.txt). Note that some commands may not exist on some OS distros, and you may need to install additional packages or run alternative commands. ``` sudo -s (Prevent from running into "Permission denied" error) dmesg lspci lspci -vvv cat /proc/interrupts + cat /proc/cpuinfo + cat /proc/meminfo lscpu lsblk lsusb @@ -210,11 +209,11 @@ Fill in with information about the system being certified - Note that running "tar -cvzf //sys_firmware.tgz /sys/firmware/" may result in corruption. In this case, you can run "cp -r sys/firmware " and then separately compress the files. -#### `./os-logs/VMware ESXi [version]/` -- Please rename the subfolders. For example, rename 'esxi' to 'VMware ESXi 1.9' -- Install VMWare ESXi to a disk, and boot it. (Note that a network controller (PCIe or USB) is needed.) -- Collect the installation and OS boot logs and save in a text file. If no serial output log, attach screenshots or video instead. -- Connect to vSphere web console over network and capture screenshots +#### `./os-logs/vmware [version]/` +- Please rename the subfolders. For example, rename 'vmware-version' to 'VMware ESXi 1.11' +- Install VMWare ESXi to a disk, and boot it. Note that a network controller (PCIe or USB) is required to complete the installation. +- Collect the installation and OS boot logs and save in a text file. If there is no serial output log, attach screenshots or a video instead. +- Connect to vSphere web console over network (from a remote client) and capture some screenshots. - Follow the instruction in https://kb.vmware.com/s/article/2004746 to enable ESXi Shell and SSH. - Connect ESXi shell through the SSH and Run the following commands and attach the logs: ``` @@ -225,25 +224,40 @@ Fill in with information about the system being certified localcli hardware pci list esxcli hardware cpu list esxcli hardware usb passthrough device list + esxcli storage filesystem list + esxcli storage core device list + cd /dev/disks + ls vim-cmd hostsvc/hosthardware vmware -v reboot -f ``` -#### `./os-logs/winpe/` +#### `./os-logs/winpe [version]/` - Boot WinPE from USB key - Include screenshots and video of WinPE booting (from FW menu to OS startup, and keyboard working in command prompt) -- Run SAC commands and/or WinPE commands below: +- Run SAC commands and WinPE commands below: ``` id d - cmd - ch + i + s + cmd (run this command after the message "EVENT: The CMD command is now available.") + ch (run this command after the message "EVENT: A new channel has been created".) Esc + Tab (Switch channel to command channel (like Cmd0001) for running WinPE commands) ver Systeminfo (This may not be supported) pnputil/enum-devices pnputil/enum-drivers + diskpart + list disk + list partition + exit + c: + dir/w + ipconfig + date + time Esc + Tab (Switch channel to SAC> for running SAC commands) shutdown ``` diff --git a/SystemReady-[band]-[SystemName]-[DateYYYYMMDD].zip b/SystemReady-[band]-[SystemName]-[DateYYYYMMDD].zip index 9c6602c31e0ce705037ac42de9d723304268a44d..2cfbab5fe0e151f8db660a8e4d96fb4bdd1d2575 100644 Binary files a/SystemReady-[band]-[SystemName]-[DateYYYYMMDD].zip and b/SystemReady-[band]-[SystemName]-[DateYYYYMMDD].zip differ diff --git a/fw/readme.txt b/fw/fw-image-files/readme.txt similarity index 100% rename from fw/readme.txt rename to fw/fw-image-files/readme.txt diff --git a/fw/screenshots/readme.txt b/fw/screenshots/readme.txt new file mode 100644 index 0000000000000000000000000000000000000000..455ad5e4b830d954cb5c640a54aee6de8a0364f4 --- /dev/null +++ b/fw/screenshots/readme.txt @@ -0,0 +1 @@ +Place some screenshots showing the FW menus (such as UEFI Setup, BMC web console, uboot shell) \ No newline at end of file diff --git a/fw/uefi-shell-cmds-logs/readme.txt b/fw/uefi-shell-cmds-logs/readme.txt new file mode 100644 index 0000000000000000000000000000000000000000..fa9530f6a8d700655746898250f1e3589816e384 --- /dev/null +++ b/fw/uefi-shell-cmds-logs/readme.txt @@ -0,0 +1 @@ +Place the UEFI Shell commands logs. \ No newline at end of file diff --git a/os-logs/OS-image-download-links.txt b/os-logs/OS-image-download-links.txt index ffd8f15e9b4a3c2f0bc525e00eab2982b6b0a411..4b0a76c2a7389e1be5d44e3400d6c7ec7b291352 100644 --- a/os-logs/OS-image-download-links.txt +++ b/os-logs/OS-image-download-links.txt @@ -7,56 +7,48 @@ For SystemReady SR/ES - VMware ESXi-Arm Fling - https://flings.vmware.com/esxi-arm-edition - - RHEL/Fedora/CentOS/AlmaLinux - - RHEL - https://developers.redhat.com/products/rhel/download - - Require Red Hat account - - - Fedora - - Fedora Server - https://getfedora.org/en/server/download/ - - Fedora Workstation live - https://getfedora.org/en/workstation/download/ - - Fedora IoT: https://getfedora.org/en/iot/download/ - - If formal releases do not work, Fedora Rawhide - - Server Rawhide: https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Server/aarch64/ - - Workstation Rawhide: https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/aarch64/ - - IoT Rawhide: https://dl.fedoraproject.org/pub/alt/iot/rawhide/IoT/aarch64/ - - - CentOS Stream - https://www.centos.org/download/ - - If you already tested latest version of Fedora/AlmaLinux, it is recommended to test CentOS Stream 8 instead 9 for covering old linux kernel. - - - AlmaLinux - https://mirrors.almalinux.org/isos.html - - If you already tested latest version of Fedora/AlmaLinux, it is recommended to test AlmaLinux 8.x instead 9 for covering old linux kernel. - - - - SLES/openSUSE - - SLES - https://www.suse.com/download/sles/ - - If certifying with SystemReady SR band, we prefer SLES to OpenSUSE. - - - OpenSUSE Leap - https://get.opensuse.org/leap/ - - If certifying with SystemReady ES band, we prefer OpenSUSE to SLES. - - If OpenSUSE Leap does not work, OpenSUSE Tumbleweed - https://get.opensuse.org/tumbleweed/ - - - - Ubuntu/Debian - - Ubuntu Server LTS - https://ubuntu.com/download/server/arm - - If LTS does npt work, Ubuntu Server interim release - https://ubuntu.com/download/server/arm - - If interim release does not work, Ubuntu daily build - https://cdimage.ubuntu.com/daily-live/current/ + - RHEL - https://developers.redhat.com/products/rhel/download + - Require Red Hat account + + - Fedora + - Fedora Server - https://getfedora.org/en/server/download/ + - Fedora Workstation live - https://getfedora.org/en/workstation/download/ + - Fedora IoT: https://getfedora.org/en/iot/download/ + - If formal releases do not work, Fedora Rawhide + - Server Rawhide: https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Server/aarch64/ + - Workstation Rawhide: https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/aarch64/ + - IoT Rawhide: https://dl.fedoraproject.org/pub/alt/iot/rawhide/IoT/aarch64/ + + - CentOS Stream - https://www.centos.org/download/ + - If you already tested latest version of Fedora/AlmaLinux, it is recommended to test CentOS Stream 8 instead 9 for covering old linux kernel. + + - AlmaLinux - https://mirrors.almalinux.org/isos.html + - If you already tested latest version of Fedora/AlmaLinux, it is recommended to test AlmaLinux 8.x instead 9 for covering old linux kernel. + + - SLES - https://www.suse.com/download/sles/ + - If certifying with SystemReady SR band, we prefer SLES to OpenSUSE. + + - OpenSUSE Leap - https://get.opensuse.org/leap/ + - If certifying with SystemReady ES band, we prefer OpenSUSE to SLES. + - If OpenSUSE Leap does not work, OpenSUSE Tumbleweed - https://get.opensuse.org/tumbleweed/ + + - Ubuntu Server LTS - https://ubuntu.com/download/server/arm + - If LTS does npt work, Ubuntu Server interim release - https://ubuntu.com/download/server/arm + - If interim release does not work, Ubuntu daily build - https://cdimage.ubuntu.com/daily-live/current/ - - Debian - https://www.debian.org/releases/ -> Stable -> installation information - + - Debian - https://www.debian.org/releases/ -> Stable -> installation information - CBL-Mariner - Source - https://github.com/microsoft/CBL-Mariner - Arm64 ISO in preview - https://packages.microsoft.com/cbl-mariner/2.0/preview/Microsoft/aarch64/ + + - FreeBSD (FreeBSD xx.x-RELEASE) - https://www.freebsd.org/where/ + - If xx.x-REELEASE not work, FreeBSD xx-x-CURRENT image - https://www.freebsd.org/where/ + - NetBSD (evbarm-aarch64) + - Releases - http://wiki.netbsd.org/ports/aarch64/ - - FreeBSD/NetBSD/OpenBSD - - FreeBSD (FreeBSD xx.x-RELEASE) - https://www.freebsd.org/where/ - - If xx.x-REELEASE not work, FreeBSD xx-x-CURRENT image - https://www.freebsd.org/where/ - - - NetBSD (evbarm-aarch64) - - Releases - http://wiki.netbsd.org/ports/aarch64/ - - - OpenBSD (latest release) - - Releases - https://cdn.openbsd.org/pub/OpenBSD/ - - Instructions - https://www.openbsd.org/arm64.html and https://ftp.openbsd.org/pub/OpenBSD/snapshots/arm64/INSTALL.arm64 + - OpenBSD (latest release) + - Releases - https://cdn.openbsd.org/pub/OpenBSD/ + - Instructions - https://www.openbsd.org/arm64.html and https://ftp.openbsd.org/pub/OpenBSD/snapshots/arm64/INSTALL.arm64 diff --git a/os-logs/esxi/.keep b/os-logs/vmware-version/.keep similarity index 100% rename from os-logs/esxi/.keep rename to os-logs/vmware-version/.keep diff --git a/os-logs/winpe/.keep b/os-logs/winpe-version/.keep similarity index 100% rename from os-logs/winpe/.keep rename to os-logs/winpe-version/.keep diff --git a/report.txt b/report.txt index 749b5cc71e25582df8b4641566e65260f4af0cc8..f766189a5d2ae8d893c6e33414de67d0f8bff5f1 100644 --- a/report.txt +++ b/report.txt @@ -3,6 +3,7 @@ General information - Arm SystemReady Band: SR|ES - System name: - Prepared by: +- Company: - E-mail: - Date: