# Template Structure for SystemReady ES or SR Compliance Reports This repository structure contains the template for collecting compliance testing data that is required for Arm SystemReady ES or SR certifications. [systemready-es-sr-template]: https://gitlab.arm.com/systemready/systemready-es-sr-template ## Prerequisites: - Check the documents below for basic information: - Arm SystemReady Requirements Specification - https://developer.arm.com/documentation/den0109/latest - SystemReady ES Test and Certification Guide - https://developer.arm.com/documentation/102529/0100/Test-SystemReady-ES-compliance - Make sure to update the firmware on the system and reset any UEFI variables/NVRAM settings to default before running the tests. This is needed to avoid false positives when the UEFI nopn-volatile variable store is full. - It is strongly recommended to use a fast storage media (NVMe, SSD or SATA drive) when running the SystemReady ACS test suite. - Using USB media will significantly increase the testing time and reduce the stability of the test run. - Running the UEFI SCT part of the ACS test suite from USB media may take more than 10 hours, and may result into unexpected problems. In some extreme cases, the USB media may be worn out from very frequent writes. - Enable the UEFI firmware ACPI mode, and disable DeviceTree mode if it exists. - The setting may be found in the UEFI/BIOS setup mene, under Device Manage > O/S Hardware Description Selection, or similar names. - Change the UEFI Boot Order in the Setup menu to make sure the media that contains the SystemReady ACS test suite is at the top of the boot order. - Install a network cable to all network ports in the system, and connect to a DHCP server for network related testing. - When running UEFI SCT manually, always run "connect -r" from the UEFI shell first. This ensures that all UEFI drivers are loaded, and necessary UEFI protocols present before testing starts. ## General Instructions Please follow the following general instructions for collecting SystemReady ES or SR compliance logs ### `./report.txt` - Fill in with information about the system being certified ### `./acs-auto-results/` - Place the ACS automation test results in this folder. This can be a copy the folders from the `RESULT` partitions in ACS media: ``` /acs_results /app_output /fwts /linux /linux_dump /sct_results /uefi /uefi_dump ``` - When certifying with the recommended Security Interface Extension (SIE), the ACS test results will include the SIE folder: ``` /acs_results /SIE ``` ### `./manual-results/` - Optionally place any results from manually performed tests in this folder. - This is not needed if the automated test completes without issues and the acs_results are copied above. ### `./manual-results/bsa-linux/` - Place any BSA or SBSA results collected manually from Linux (BusyBox) in this folder. This can be for instance the serial console output from running the commands below. - Examples of SBSA Linux command usage: ``` Complete SBSA Linux results can be collected by running the following commands after booting Linux BusyBox insmod /lib/modules/sbsa_acs.ko sbsa You can include verbose output (-v) and run the test in different SBSA levels (-l): sbsa -l 3 -v 1 To skip a specific problematic test, use -skip sbsa -l 3 -v 1 -skip 413 ``` - Examples of BSA Linux command usage: ``` Complete BSA Linux results can be collected by running the following command after booting Linux BusyBox: insmod /lib/modules/bsa_acs.ko bsa You can include verbose output (-v): bsa -v 1 To skip a specific problematic test, use -skip bsa -skip 104 ``` ### `./manual-results/bsa-uefi/` - Place any BSA or SBSA results (e.g. serial console log) collected manually from UEFI Shell in this folder. - Examples of SBSA UEFI command usage: ``` Complete SBSA UEFI results can be collected by running the following commands: FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi You can include verbose output (-v) and run the test in different SBSA levels (-l): FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -l 3 -v 1 To skip multiple problematic tests, use -skip , FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -skip 448,449 ``` - Examples of BSA UEFI command usage: ``` Complete BSA UEFI results can be collected by running the following command: FS1:\EFI\BOOT\bsa> bsa.efi You can include verbose output (-v): FS1:\EFI\BOOT\bsa> bsa.efi -v 1 To skip a specific problematic test, use -skip FS1:\EFI\BOOT\bsa> bsa.efi -skip 606 ``` ### `./manual-results/fwts/` - Place any additional FWTS results collected manually from Linux (BusyBox) in this folder. - Examples: ``` Complete FWTS results can be collected by running the following command: fwts -r stdout -q --uefi-set-var-multiple=1 --uefi-get-mn-count-multiple=1 --sbbr esrt uefibootpath To skip a specific problematic tests use --skip-test= fwts -r stdout -q --sbbr --skip-test=dmicheck fwts -r stdout -q --sbbr --skip-test=uefirtmisc,uefirtvariable,uefirttime ``` ### `./manual-results/sct/` - Place any additional UEFI SCT results collected manually from the UEFI shell in this folder. - Examples: ``` Complete SCT results can be collected by running the following command: FS1:\EFI\BOOT\bbr\SCT>SCT.efi -s SBBR.seq The result of the specific SCT test case can be collected by running the following commands to use the Menu-Driven Interface For details, please check section 2.2 at https://github.com/tianocore/edk2-test/blob/master/uefi-sct/Doc/UserGuide/UEFI_SCT_II_UserGuide_2_6_A.pdf FS1:\EFI\BOOT\bbr\SCT>SCT.efi -u ``` ### `./docs/` - Place any firmware or device documentation, manuals, user guides, build instructions, etc... ### `./fw/fw-image-files/` - Place FW binary and/or source trees. Include information on the additional firmware (BMC, SCP, MCP, etc...) if applicable. #### `./fw/screenshots/` - Place some screenshots showing the FW menus (such as the UEFI Setup, BMC web console, uboot shell, etc...) if applicable. #### `./fw/uefi-shell-cmds-logs/` - Run the UEFI Shell commands below and attach the logs. - The UEFI Shell can be run from the ACS test suite by interrupting the automated run, and pressing a key at the UEFI timeout prompt before the tests begin to execute. ``` Shell> connect -r Shell> map -r Shell> ver Shell> dmem Shell> pci Shell> drivers Shell> devices Shell> devtree Shell> dh -d -v Shell> dmpstore Shell> memmap Shell> smbiosview Shell> acpiview -l Shell> acpiview Shell> acpiview -r 2 Shell> acpiview -s DSDT -d (if DSDT table is present) Shell> acpiview -s SSDT -d (if SSDT table is present) Shell> bcfg boot dump -v Shell> ifconfig -l Shell> ifconfig -s eth0 dhcp (If eth0 doesn't display an updated IP address, run "connect -r" again) Shell> ifconfig -l eth0 Shell> ping xxx.xxx.xxx.xxx (If don’t know any other system’s IP address, you can get DNS Server's IP address from ifconfig output and ping it) ``` ### `./os-logs/` - Please check ./os_logs/OS_image_download_links.txt to get the latest publicly available AARCH64 OS images for testing. #### `./os-logs/[distroname] [distroversion]/` - Please rename the subfolders to indicate the OS name and version. For example, rename 'linux-distro1-version' to 'Fedora Server 37'. - It is recommended to choose one OS per group below, an OS based on an old kernel, and an OS based on an kernel version close to the latest kernel for certification testing to cover as many kernel versions as possible. - Groups based on heritage: RHEL/Fedora/CentOS/AlmaLinux, SLES/openSUSE, Ubuntu/Debian, or NetBSD/OpenBSD/FreeBSD. - For example: - For SystemReady SR, You can choose RHEL, SLES, Ubuntu, CentOS Linux 7, and Fedora for your testing. - For SystemReady ES, you can choose Fedora, openSUSE, Ubuntu, and CentOS 7 for your testing. - For each OS being tested - Install the OS from some media (eg. USB drive) to a disk (e.g. SATA, NVMe, USB), and boot it. - Note that a network controller (PCIe or USB) may be needed during installation. - The install log must begin when the platform is released from reset and must include: - All firmware output - Output from the Linux installer - Output of reboot to the installed OS. - Run the following commands and attach the logs: (For example, save dmesg messages to dmesg.txt). Note that some commands may not exist on some OS distros, and you may need to install additional packages or run alternative commands. ``` sudo -s (Prevent from running into "Permission denied" error) dmesg lspci lspci -vvv cat /proc/interrupts cat /proc/cpuinfo cat /proc/meminfo lscpu lsblk lsusb dmidecode uname -a cat /etc/os-release efibootmgr (Check the output message to know the ) efibootmgr -t 20 efibootmgr -t efibootmgr -c ifconfig (If ifconfig is not supported, use "ip addr show" and "ip link show" instead.) (If network port doesn't get IP address, run "dhclient" or "ip a add /255.255.255.0 dev eth" and "ip link set dev eth up") ping -c 5 www.arm.com (If the system is connected to intranet, please ping another system's IP address) Go to /sys/firmware and use "ls -lR" (Or use tree /sys/firmware/) ``` - Copy the entire content of /sys/firmware and attach zipped/archive file. - Note that running "tar -cvzf //sys_firmware.tgz /sys/firmware/" may result in corruption. In this case, you can run "cp -r sys/firmware " and then separately compress the files. #### `./os-logs/vmware [version]/` - Please rename the subfolders. For example, rename 'vmware-version' to 'VMware ESXi 1.11' - Install VMWare ESXi to a disk, and boot it. Note that a network controller (PCIe or USB) is required to complete the installation. - Collect the installation and OS boot logs and save in a text file. If there is no serial output log, attach screenshots or a video instead. - Connect to vSphere web console over network (from a remote client) and capture some screenshots. - Follow the instruction in https://kb.vmware.com/s/article/2004746 to enable ESXi Shell and SSH. - Connect ESXi shell through the SSH and Run the following commands and attach the logs: ``` dmesg lspci irqinfo localcli storage core adapter list localcli hardware pci list esxcli hardware cpu list esxcli hardware usb passthrough device list esxcli storage filesystem list esxcli storage core device list cd /dev/disks ls vim-cmd hostsvc/hosthardware vmware -v reboot -f ``` #### `./os-logs/winpe [version]/` - Boot WinPE from USB key - Include screenshots and video of WinPE booting (from FW menu to OS startup, and keyboard working in command prompt) - Run SAC commands and WinPE commands below: ``` id d i s cmd (run this command after the message "EVENT: The CMD command is now available.") ch (run this command after the message "EVENT: A new channel has been created".) Esc + Tab (Switch channel to command channel (like Cmd0001) for running WinPE commands) ver pnputil/enum-devices pnputil/enum-drivers diskpart list disk list volume exit c: dir/w ipconfig /all date time Esc + Tab (Switch channel to SAC> for running SAC commands) shutdown ```