Newer
Older
# Template Structure for SystemReady band Compliance Reports
This repository structure contains the template for collecting compliance testing data that is required for Arm SystemReady band compliance.
[systemready-band-template]: https://gitlab.arm.com/systemready/systemready-band-template
Sunny Wang
committed
- Check the documents below for basic information:
- Arm SystemReady Requirements Specification - https://developer.arm.com/documentation/den0109/latest
- Make sure to update the firmware on the system and reset any UEFI variables/NVRAM settings to default before running the tests. This is needed to avoid false positives when the UEFI nopn-volatile variable store is full.
- It is strongly recommended to use a fast storage media (NVMe, SSD or SATA drive) when running the test suites in SystemReady band ACS live image.
- Using USB media will significantly increase the testing time and reduce the stability of the test run.
- Running the UEFI SCT (BBR test suite) from a USB media may take more than 10 hours, and may result into unexpected problems. In some extreme cases, the USB media may be worn out from very frequent writes.
- Enable the UEFI firmware ACPI mode, and disable DeviceTree mode if it exists.
- The setting may be found in the UEFI/BIOS setup mene, under Device Manage > O/S Hardware Description Selection, or similar names.
- Change the UEFI Boot Order in the Setup menu to make sure the media that contains the test suites is at the top of the boot order.
- Install a network cable to all network ports in the system, and connect to a DHCP server for network related testing.
- When running UEFI SCT manually, always run "connect -r" from the UEFI shell first. This ensures that all UEFI drivers are loaded, and necessary UEFI protocols present before testing starts.
Please follow the following general instructions for collecting SystemReady band compliance logs
- Fill in with information about the system being tested
- Place the ACS automation test results in this folder. In other words, copy all the folders and files in acs_results folder below from ACS media to this folder.
/app_output
/edk2-test-parser
/fwts
/linux
/linux_dump
/sct_results
/uefi
/uefi_dump
- After running BBSR complaince test, the acs_results folder in ACS media will also include the BBSR folder.
/fwts
/sct_results
/tpm2
- Optionally place any results from manually performed tests in this folder.
- This is not needed if the automated test completes without issues and the acs_results are copied above.
- Place any BSA or SBSA results collected manually from ACS Linux in this folder. This can be for instance the serial console output from running the commands below.
- The latest bsa and sbsa commands used in automation can be found on https://github.com/ARM-software/arm-systemready/blob/main/common/ramdisk/init.sh
- Examples of SBSA Linux command usage:
Complete SBSA Linux results can be collected by running the following commands after booting ACS Linux
Sunny Wang
committed
insmod /lib/modules/sbsa_acs.ko
You can include verbose output (-v) and run the test in different SBSA levels (-l):
sbsa -l 3 -v 1
To skip a specific problematic test, use -skip <testnum>
sbsa -l 3 -v 1 -skip 413
- Examples of BSA Linux command usage:
Complete BSA Linux results can be collected by running the following command after booting ACS Linux:
To skip a specific problematic test, use -skip <testnum>
- Place any BSA or SBSA results (e.g. serial console log) collected manually from UEFI Shell in this folder.
- The latest bsa and sbsa commands used in automation can be found on https://github.com/ARM-software/arm-systemready/blob/main/common/config/sbsa.nsh.
- Examples of SBSA UEFI command usage:
```
Complete SBSA UEFI results can be collected by running the following commands:
Sunny Wang
committed
FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi
You can include verbose output (-v) and run the test in different SBSA levels (-l):
FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -l 3 -v 1
To skip multiple problematic tests, use -skip <testnum>,<testnum>
Sunny Wang
committed
FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -skip 448,449
- Examples of BSA UEFI command usage:
Complete BSA UEFI results can be collected by running the following command:
Sunny Wang
committed
FS1:\EFI\BOOT\bsa> bsa.efi
Sunny Wang
committed
FS1:\EFI\BOOT\bsa> bsa.efi -v 1
To skip a specific problematic test, use -skip <testnum>
Sunny Wang
committed
FS1:\EFI\BOOT\bsa> bsa.efi -skip 606
- Place any additional FWTS results collected manually from ACS Linux in this folder.
- The latest fwts command used in automation can be found on https://github.com/ARM-software/arm-systemready/blob/main/common/ramdisk/init.sh
Sunny Wang
committed
- Examples:
Complete FWTS results can be collected by running the following command:
fwts -r stdout -q --uefi-set-var-multiple=1 --uefi-get-mn-count-multiple=1 --sbbr esrt uefibootpath aest cedt slit srat hmat pcct pdtt bgrt bert einj erst hest sdei nfit iort mpam ibft ras2
To skip a specific problematic tests use --skip-test=<testname>
fwts -r stdout -q --sbbr --skip-test=dmicheck
fwts -r stdout -q --sbbr --skip-test=uefirtmisc,uefirtvariable,uefirttime
```
- Place any additional UEFI SCT results collected manually from the UEFI shell in this folder.
Sunny Wang
committed
- Examples:
Complete SCT results can be collected by running the following command:
The result of the specific SCT test case can be collected by running the following commands to use the Menu-Driven Interface
For details, please check section 2.2 at https://github.com/tianocore/edk2-test/blob/master/uefi-sct/Doc/UserGuide/UEFI_SCT_II_UserGuide_2_6_A.pdf
### `./manual-results/uefi_dump/`
- Place any UEFI shell commands output collected manually from the UEFI shell in this folder.
- If no log in ./acs_results/uefi_dump folder, please manually run debug_dump.nsh to collect the logs. Two ways below to find and run debug_dump.nsh.
- Go to ACS media device's boot partition's Boot\EFI\BOOT\debug folder, and directly run debug_dump.nsh. The uefi_dump folder and logs will be generated and collected in /acs-auto-results/ folder.
- The latest debug_dump.nsh is on https://github.com/ARM-software/arm-systemready/blob/main/common/config/debug_dump.nsh
- Download the debug_dump.nsh to an ESP (EFI system partition) or a media device's FAT32 partition, and boot to UEFI shell to load the script from ESP or media device. The following is the example of how to do it on SUT OS.
```
sudo -s (Prevent from running into "Permission denied" error)
cd /boot/efi
wget https://raw.githubusercontent.com/ARM-software/arm-systemready/main/common/config/debug_dump.nsh
reboot
```
```
Shell> fsx: (Replace x with the correct number of partition. For example, USB drive could be fs0:)
FSx:\> debug_dump.nsh
FSx:\uefi_dump\> ls (make sure if the logs are collected)
```
- Place any firmware or device documentation, manuals, user guides, build instructions, etc...
### `./fw/fw-image-files/`
- Place FW binary and/or source trees. Include information on the additional firmware (BMC, SCP, MCP, etc...) if applicable.
- Place some screenshots showing the FW menus (such as the UEFI Setup, BMC web console, uboot shell, etc...) if applicable.
#### `./fw/uefi-shell-cmds-logs/`
- Run additional UEFI Shell commands below that haven't been covered by https://raw.githubusercontent.com/ARM-software/arm-systemready/main/common/config/debug_dump.nsh and attach the logs.
- The UEFI Shell can be run by interrupting the automated run, and pressing a key at the UEFI timeout prompt before the tests begin to execute.
Shell> dh -d -v > dh_d_v.log
Shell> acpiview -s DSDT -d (if DSDT table is present)
Shell> acpiview -s SSDT -d (if SSDT table is present)
Shell> bcfg boot dump -v > bcfg_boot_dump_v.log
(If "ifconfig -l" output message shows a network port having IP address, skip other ifconfig commands below and directly run ping command.)
Shell> ifconfig -s eth<number> dhcp (The <number> is shown in )
Shell> connect -r
Shell> ifconfig -l eth<number>
Shell> ping xxx.xxx.xxx.xxx (If don't know any other system's IP address, you can get DNS Server's IP address from ifconfig output and ping it)
- OS testing is recommended rather than required for compliance testing
- Please check ./os_logs/OS_image_download_links.txt to get the latest publicly available AARCH64 OS images for testing.
- Place OS install, boot, commands output logs in this folder. Both OS install and boot logs can be collected by serial console and are recommended to begin when the platform is released from reset and include:
- All firmware output
- Output from the Linux installer
- Output of reboot to the installed OS.
Sunny Wang
committed
#### `./os-logs/[distroname] [distroversion]/`
- Please rename the subfolders to indicate the OS name and version. For example, rename 'linux-distro1-version' to 'RHEL 9.4'.
- It is recommended to choose one OS per group below.
- Groups based on heritage: RHEL/Fedora/CentOS/AlmaLinux, SLES/openSUSE, Ubuntu/Debian, or NetBSD/OpenBSD/FreeBSD.
- For example, you can choose RHEL, SLES, Ubuntu for your testing.
- For each OS being tested
- Install the OS from some media (eg. USB drive) to a disk (e.g. SATA, NVMe, USB), and boot it.
- A network controller (PCIe or USB) may be needed during installation.
- The OS commands output and files are required. For collecting them, run the commands below to download and run the script linux-distro-cmds.sh. Then, copy all the files that are collected in ~/ComplianceTestLogs folder to the template's ./os-logs/[distroname] [distroversion]/ folder.
- If network port doesn't get IP address, giving "dhclient" or "ip a add <assigned IP address>/255.255.255.0 dev eth<number>" and "ip link set dev eth<number> up" a try might be helpful.
- If the network doesn't work, please manually run the commands in ./os-logs/linux-distro-cmds.sh.
- For manually running the ping commnad, if the system is connected to intranet, please ping the IP address of another system within the intranet instead.
sudo -s (Prevent from running into "Permission denied" error)
wget https://gitlab.arm.com/systemready/systemready-band-template/-/raw/main/os-logs/linux-distro-cmds.sh
chmod +x linux-distro-cmds.sh
./linux-distro-cmds.sh
```
#### `./os-logs/vmware [version]/`
- Please rename the subfolders. For example, rename 'vmware-version' to 'VMware ESXi 1.15'
- Install VMware ESXi to a disk, and boot it. Note that a network controller (PCIe or USB) is required to complete the installation.
- Connect to vSphere web console over network (from a remote client) and capture some screenshots.
- Follow the instruction in https://kb.vmware.com/s/article/2004746 to enable ESXi Shell and SSH.
- Connect ESXi shell through the SSH and Run the following commands and attach the logs:
dmesg
lspci
irqinfo
localcli storage core adapter list
localcli hardware pci list
esxcli hardware cpu list
esxcli hardware usb passthrough device list
esxcli storage filesystem list
esxcli storage core device list
cd /dev/disks
ls
vim-cmd hostsvc/hosthardware
vmware -v
reboot -f
#### `./os-logs/winpe [version]/`
- Boot WinPE from USB key
- Include screenshots and video of WinPE booting (from FW menu to OS startup, and keyboard working in command prompt)
- Run SAC commands and WinPE commands below:
cmd (run this command after the message "EVENT: The CMD command is now available.")
ch (run this command after the message "EVENT: A new channel has been created".)
Esc + Tab (Switch channel to command channel (like Cmd0001) for running WinPE commands)
ver
pnputil/enum-devices
pnputil/enum-drivers
diskpart
list disk
exit
c:
dir/w
date
time
Esc + Tab (Switch channel to SAC> for running SAC commands)
shutdown