README.md

# Template Structure for SystemReady ES or SR Compliance Reports
This repo structure is the template for collecting compliance evidence for a SystemReady ES or SR certification.
[systemready-es-sr-template]: https://gitlab.arm.com/systemready/systemready-es-sr-template

## Prerequisites:
- Check the documents below for basic information:
    - Arm SystemReady Requirements Specification - https://developer.arm.com/documentation/den0109/latest
    - SystemReady ES Test and Certification Guide - https://developer.arm.com/documentation/102529/0100/Test-SystemReady-ES-compliance
- Flash/update the firmware again to clear UEFI variable store/NVRAM right before running the tests, which will avoid fake failures that are caused by using up the UEFI variable store/NVRAM space.
- It is strongly recommended to flash ACS image to a USB disk enclosure with a fast SSD or SATA drive or a directly attached SATA/NVMe drive instead of USB stick, which would significantly shorten the testing time and will be more stable than using USB stick/flash drive.
    - Running SCT with a USB stick may take more than 10 hours, and may run into some unexpected problem caused by USB HW issue. In the worst case, the USB stick may be bricked becasue of so many writes to the USB stick.
- Go to UEFI/BIOS setup menu to change your system’s "OS hardware description" option to "ACPI mode". You may find the option in Device Manage > O/S Hardware Description Selection.
- Go to UEFI/BIOS setup menu to move the ACS media device (USB drive) to the top of the boot order.
- Install network cable to all network ports and connect to a DHCP server for network related testing.
- Before manually running SCT, please run "connect -r" in UEFI shell first.

## General Instructions
General instructions for collecting SystemReady ES or SR compliance logs:

### `./report.txt`
Fill in with information about the system being certified


### `./acs-auto-results/`
- Place the automation test result (Copy the folders below in acs_results in `RESULT` partitions in ACS USB drive/stick to this acs-auto-results folder):

```
    /acs_results
        /app_output
        /fwts
        /linux
        /linux_dump
        /sct_results
        /uefi
        /uefi_dump
```


### `./manual-results/`
- Place any results from manually performed tests. 
- Note: If automation test works fine and no issue needs to be further investigated, you won't need to run manul test and put any file in this folder.

### `./manual-results/bsa-linux/`
- Place any BSA or SBSA results (e.g. serial console log) collected manually from Linux (Linux BusyBox).

- Examples for SBSA Linux:
```
        Complete SBSA Linux results can be collected by running the following commands after booting Linux BusyBox
            insmod /lib/modules/sbsa_acs.ko
            sbsa

        You can include verbose output (-v) and run the test in differnt SBSA level (-l):
            sbsa -l 3 -v 1

        To skip a specific problematic test, use -skip <testnum>
            sbsa -l 3 -v 1 -skip 413
```

- Examples for BSA Linux
```
        Complete BSA Linux results can be collected by running the following command after booting Linux BusyBox:
            insmod /lib/modules/bsa_acs.ko
            bsa

        You can include verbose output (-v):
            bsa -v 1

        To skip a specific problematic tests, use -skip <testnum>
            bsa -skip 104

```

### `./manual-results/bsa-uefi/`
- Place any BSA or SBSA results (e.g. serial console log) collected manually from UEFI Shell.

- Examples for SBSA UEFI:
```
        Complete SBSA UEFI results can be collected by running the following commands:
            FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi

        You can include verbose output (-v)  and run the test in differnt SBSA level (-l):
            FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -l 3 -v 1

        To skip multiple problematic tests, use -skip <testnum>,<testnum>
            FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -skip 448,449
```

- Examples for BSA UEFI:
```
        Complete BSA UEFI results can be collected by running the following command:
            FS1:\EFI\BOOT\bsa> bsa.efi

        You can include verbose output (-v):
            FS1:\EFI\BOOT\bsa> bsa.efi -v 1

        To skip a specific problematic tests, use -skip <testnum>
            FS1:\EFI\BOOT\bsa> bsa.efi -skip 606
```


### `./manual-results/fwts/`
- Place any additional FWTS results collected manually from Linux (Linux BusyBox):
- Examples:
```
        Complete FWTS results can be collected by running the following commands:
            fwts -r stdout -q --uefi-set-var-multiple=1 --uefi-get-mn-count-multiple=1 --sbbr esrt uefibootpath

        To skip a specific problematic tests, use --skip-test=<testname>
            fwts -r stdout -q --sbbr --skip-test=dmicheck
            fwts -r stdout -q --sbbr --skip-test=uefirtmisc,uefirtvariable,uefirttime
```


### `./manual-results/sct/`
- Place any additional SCT results collected manually from UEFI shell:
- Examples:
```
       Complete SCT results can be collected by running the following commands
       FS1:\EFI\BOOT\bbr\SCT>SCT.efi -s SBBR.seq

       Specific SCT test's result can be collected by running the following commands to use the Menu-Driven Interface
       For details, please check the section 2.2 in https://github.com/tianocore/edk2-test/blob/master/uefi-sct/Doc/UserGuide/UEFI_SCT_II_UserGuide_2_6_A.pdf
       FS1:\EFI\BOOT\bbr\SCT>SCT.efi -u

```

### `./docs/`
- Place any firmware or device documentation, manuals, user guides, build instructions, etc..

### `./fw/fw-image-files/`
- Place FW binary and/or source trees. Include information on the additional firmware (BMC, SCP, MCP, etc..) if applicable.

#### `./fw/screenshots/`
- Place some screenshots showing the FW menus (such as UEFI Setup, BMC web console, uboot shell)

#### `./fw/uefi-shell-cmds-logs/`
- Run the UEFI Shell commands below and attach the logs (if not already done by ACS). The UEFI Shell can be run from the ACS image by pressing a key at the UEFI timeout prompt before tests begin to execute.
```
    Shell> connect -r
    Shell> map -r
    Shell> ver
    Shell> dmem
    Shell> pci
    Shell> drivers
    Shell> devices
    Shell> devtree
    Shell> dh -d -v
    Shell> dmpstore
    Shell> memmap
    Shell> smbiosview
    Shell> acpiview -l
    Shell> acpiview
    Shell> acpiview -r 2
    Shell> acpiview -s DSDT -d  (if DSDT table is present)
    Shell> acpiview -s SSDT -d (if SSDT table is present)
    Shell> bcfg boot dump -v
    Shell> ifconfig -l
    Shell> ifconfig -s eth0 dhcp (If eth0 still hasn’t get IP, run "connect -r" again)
    Shell> ifconfig -l eth0
    Shell> ping xxx.xxx.xxx.xxx (If don’t know any other system’s IP address, you can get DNS Server's IP address from ifconfig output and ping it)
```

### `./os-logs/`
- Please check ./os_logs/OS_image_download_links.txt to get the latest release for testing

#### `./os-logs/[distroname] [distroversion]/`
- Please rename the subfolders. For example, rename 'linux-distro1-version' to 'Fedora Server 37'.
- It is recommended to choose one OS per group below and an OS based on old kernel for certification testing to widely cover all the kernel versions and kernel bases. 
    - Groups of the heritage: RHEL/Fedora/CentOS/AlmaLinux, or SLES/openSUSE, or Ubuntu/Debian, or NetBSD/OpenBSD/FreeBSD.
    - For example, you can choose RHEL, SLES, Ubuntu, and CentOS 7 for SystemReady SR testing. For SystemReady ES, you can choose Fedora, openSUSE, Ubuntu, and and CentOS 7 for your testing. As for more information about OS reference for specific band or condition, please check ./os_logs/OS_image_download_links.txt. 
    - Moreover, it would be good to test OS as many as possible for showing a better result on Arm SCL web even if the OSes are in the same group.
- Install the OS to a disk, and boot it. (Note that a network controller (PCIe or USB) is needed.)
- The install log must begin when the platform is released from reset and must include:
    - All firmware output
    - Output from Linux installer
    - Output of reboot into installed OS.
- Run the following commands and attach the logs: (For example, save dmesg messages to dmesg.txt)
   - Note that some commands may not exist on some distros, and you may need to install the command or run an alternative command.
```
    sudo -s                      (Prevent from running into "Permission denied" error)
    dmesg
    lspci
    lspci -vvv
    cat /proc/interrupts
    cat /proc/cpuinfo
    cat /proc/meminfo
    lscpu
    lsblk
    lsusb
    dmidecode
    uname -a
    cat /etc/os-release
    efibootmgr                  (Check the output message to know the <Timeout value> )
    efibootmgr -t 20
    efibootmgr -t <Timeout value>
    efibootmgr -c
    ifconfig                    (If ifconfig is not supported, use "ip addr show" and "ip link show" instead.)
                                (If network port doesn't get IP address, run "dhclient" or "ip a add <assigned IP address>/255.255.255.0 dev eth<number>" and "ip link set dev eth<number> up")
    ping -c 5 www.arm.com       (If the system is connected to intranet, please ping another system's IP address)
    Go to /sys/firmware and use "ls -lR" (Or use tree /sys/firmware/)

```
- Copy the entire content of /sys/firmware and attach zipped/archive file.
   - Note that running "tar -cvzf /<Target folder path>/sys_firmware.tgz /sys/firmware/" may result in corruption. In this case, you can run "cp -r sys/firmware <Target folder path>" and then separately compress the files.


#### `./os-logs/vmware [version]/`
- Please rename the subfolders. For example, rename 'vmware-version' to 'VMware 1.11'
- Install VMWare ESXi to a disk, and boot it. (Note that a network controller (PCIe or USB) is needed.)
- Collect the installation and OS boot logs and save in a text file. If no serial output log, attach screenshots or video instead.
- Connect to vSphere web console over network and capture screenshots
- Follow the instruction in https://kb.vmware.com/s/article/2004746 to enable ESXi Shell and SSH.
- Connect ESXi shell through the SSH and Run the following commands and attach the logs:
```
    dmesg
    lspci
    irqinfo
    localcli storage core adapter list
    localcli hardware pci list
    esxcli hardware cpu list
    esxcli hardware usb passthrough device list
    esxcli storage filesystem list
    esxcli storage core device list
    cd /dev/disks
    ls
    vim-cmd hostsvc/hosthardware
    vmware -v
    reboot -f
```

#### `./os-logs/winpe [version]/`
- Boot WinPE from USB key
- Include screenshots and video of WinPE booting (from FW menu to OS startup, and keyboard working in command prompt)
- Run SAC commands and WinPE commands below:
```
    id
    d
    i
    s
    cmd (running this command after the message "EVENT: The CMD command is now available.")
    ch  (running this command after the message "EVENT:   A new channel has been created".)
    Esc + Tab (Switch channel to command channel (like Cmd0001) for running WinPE commands)
    ver
    Systeminfo (This may not be supported)
    pnputil/enum-devices
    pnputil/enum-drivers
    diskpart
    list disk
    list partition
    exit
    c:
    dir/w
    ipconfig
    date 
    time
    Esc + Tab (Switch channel to SAC> for running SAC commands)
    shutdown
```