README.md

# Template Structure for SystemReady ES or SR Compliance Reports
This repository structure contains the template for collecting compliance testing data that is required for Arm SystemReady ES or SR certifications.
[systemready-es-sr-template]: https://gitlab.arm.com/systemready/systemready-es-sr-template

## Prerequisites:
- Check the documents below for basic information:
    - Arm SystemReady Requirements Specification - https://developer.arm.com/documentation/den0109/latest
    - SystemReady ES Test and Certification Guide - https://developer.arm.com/documentation/102529/0100/Test-SystemReady-ES-compliance
- Make sure to update the firmware on the system and reset any UEFI variables/NVRAM settings to default before running the tests. This is needed to avoid false positives when the UEFI nopn-volatile variable store is full.
- It is strongly recommended to use a fast storage media (NVMe, SSD or SATA drive) when running the SystemReady ACS test suite.
    - Using USB media will significantly increase the testing time and reduce the stability of the test run.
    - Running the UEFI SCT part of the ACS test suite from USB media may take more than 10 hours, and may result into unexpected problems. In some extreme cases, the USB media may be worn out from very frequent writes.
- Enable the UEFI firmware ACPI mode, and disable DeviceTree mode if it exists.
    - The setting may be found in the UEFI/BIOS setup mene, under  Device Manage > O/S Hardware Description Selection, or similar names.
- Change the UEFI Boot Order in the Setup menu to make sure the media that contains the SystemReady ACS test suite is at the top of the boot order.
- Install a network cable to all network ports in the system, and connect to a DHCP server for network related testing.
- When running UEFI SCT manually, always run "connect -r" from the UEFI shell first. This ensures that all UEFI drivers are loaded, and necessary UEFI protocols present before testing starts.

## General Instructions
Please follow the following general instructions for collecting SystemReady ES or SR compliance logs

### `./report.txt`
- Fill in with information about the system being certified


### `./acs-auto-results/`
- Place the ACS automation test results in this folder. This can be a copy the folders from the `RESULT` partitions in ACS media:

```
    /acs_results
        /app_output
        /fwts
        /linux
        /linux_dump
        /sct_results
        /uefi
        /uefi_dump
```

- When certifying with the recommended Security Interface Extension (SIE), the ACS test results will include the SIE folder:

```
    /acs_results
        /SIE
```

### `./manual-results/`
- Optionally place any results from manually performed tests in this folder. 
- This is not needed if the automated test completes without issues and the acs_results are copied above.

### `./manual-results/bsa-linux/`
- Place any BSA or SBSA results collected manually from Linux (BusyBox) in this folder. This can be for instance the serial console output from running the commands below.

- Examples of SBSA Linux command usage:
```
        Complete SBSA Linux results can be collected by running the following commands after booting Linux BusyBox
            insmod /lib/modules/sbsa_acs.ko
            sbsa

        You can include verbose output (-v) and run the test in different SBSA levels (-l):
            sbsa -l 3 -v 1

        To skip a specific problematic test, use -skip <testnum>
            sbsa -l 3 -v 1 -skip 413
```

- Examples of BSA Linux command usage:
```
        Complete BSA Linux results can be collected by running the following command after booting Linux BusyBox:
            insmod /lib/modules/bsa_acs.ko
            bsa

        You can include verbose output (-v):
            bsa -v 1

        To skip a specific problematic test, use -skip <testnum>
            bsa -skip 104

```

### `./manual-results/bsa-uefi/`
- Place any BSA or SBSA results (e.g. serial console log) collected manually from UEFI Shell in this folder.

- Examples of SBSA UEFI command usage:
```
        Complete SBSA UEFI results can be collected by running the following commands:
            FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi

        You can include verbose output (-v)  and run the test in different SBSA levels (-l):
            FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -l 3 -v 1

        To skip multiple problematic tests, use -skip <testnum>,<testnum>
            FS1:\EFI\BOOT\bsa\sbsa> sbsa.efi -skip 448,449
```

- Examples of BSA UEFI command usage:
```
        Complete BSA UEFI results can be collected by running the following command:
            FS1:\EFI\BOOT\bsa> bsa.efi

        You can include verbose output (-v):
            FS1:\EFI\BOOT\bsa> bsa.efi -v 1

        To skip a specific problematic test, use -skip <testnum>
            FS1:\EFI\BOOT\bsa> bsa.efi -skip 606
```


### `./manual-results/fwts/`
- Place any additional FWTS results collected manually from Linux (BusyBox) in this folder.
- Examples:
```
        Complete FWTS results can be collected by running the following command:
            fwts -r stdout -q --uefi-set-var-multiple=1 --uefi-get-mn-count-multiple=1 --sbbr esrt uefibootpath

        To skip a specific problematic tests use --skip-test=<testname>
            fwts -r stdout -q --sbbr --skip-test=dmicheck
            fwts -r stdout -q --sbbr --skip-test=uefirtmisc,uefirtvariable,uefirttime
```


### `./manual-results/sct/`
- Place any additional UEFI SCT results collected manually from the UEFI shell in this folder.
- Examples:
```
       Complete SCT results can be collected by running the following command:
       FS1:\EFI\BOOT\bbr\SCT>SCT.efi -s SBBR.seq

       The result of the specific SCT test case can be collected by running the following commands to use the Menu-Driven Interface
       For details, please check section 2.2 at https://github.com/tianocore/edk2-test/blob/master/uefi-sct/Doc/UserGuide/UEFI_SCT_II_UserGuide_2_6_A.pdf
       FS1:\EFI\BOOT\bbr\SCT>SCT.efi -u

```
### `./manual-results/uefi_dump/`
- Place any UEFI shell commands output collected manually from the UEFI shell in this folder.
- If no log in ./acs_results/uefi_dump folder, please manually run debug_dump.nsh to collect the logs. Two ways below to find and run debug_dump.nsh. 
    - Go to ACS media device's boot partition's Boot\EFI\BOOT\debug folder, and directly run debug_dump.nsh. The uefi_dump folder and logs will be generated and collected in /acs-auto-results/ folder.
    - Download the debug_dump.nsh to an ESP (EFI system partition) or a media device's FAT32 partition, and boot to UEFI shell to load the script from ESP or media device. The following is the example of how to do it on SUT OS.
```
    sudo -s                      (Prevent from running into "Permission denied" error)
    cd /boot/efi
    wget https://raw.githubusercontent.com/ARM-software/arm-systemready/main/common/config/debug_dump.nsh
    reboot
```
```
    Shell> fsx:                  (Replace x with the correct number of partition. For example, USB drive could be fs0:)
    FSx:\> debug_dump.nsh
    FSx:\uefi_dump\> ls          (make sure if the logs are collected)     
```

### `./docs/`
- Place any firmware or device documentation, manuals, user guides, build instructions, etc...

### `./fw/fw-image-files/`
- Place FW binary and/or source trees. Include information on the additional firmware (BMC, SCP, MCP, etc...) if applicable.

#### `./fw/screenshots/`
- Place some screenshots showing the FW menus (such as the UEFI Setup, BMC web console, uboot shell, etc...) if applicable.

#### `./fw/uefi-shell-cmds-logs/`
- Run additional UEFI Shell commands below that haven't been covered by https://raw.githubusercontent.com/ARM-software/arm-systemready/main/common/config/debug_dump.nsh and attach the logs.  
- The UEFI Shell can be run from the ACS test suite by interrupting the automated run, and pressing a key at the UEFI timeout prompt before the tests begin to execute.
```
    Shell> connect -r
    Shell> dh -d -v > dh_d_v.log
    Shell> acpiview -s DSDT -d  (if DSDT table is present)
    Shell> acpiview -s SSDT -d  (if SSDT table is present)
    Shell> bcfg boot dump -v > bcfg_boot_dump_v.log
    Shell> ifconfig -l
    (If "ifconfig -l" output message shows a network port having IP address, skip other ifconfig commands below and directly run ping command.)
    Shell> ifconfig -s eth<number> dhcp  (The <number> is shown in ) 
    Shell> connect -r                  
    Shell> ifconfig -l eth<number>
    Shell> ping xxx.xxx.xxx.xxx          (If don't know any other system's IP address, you can get DNS Server's IP address from ifconfig output and ping it)
```

### `./os-logs/`
- Please check ./os_logs/OS_image_download_links.txt to get the latest publicly available AARCH64 OS images for testing.
- The boot and install logs are required and can be collected by serial console. Both logs must begin when the platform is released from reset and must include:
        - All firmware output
        - Output from the Linux installer
        - Output of reboot to the installed OS.
  
#### `./os-logs/[distroname] [distroversion]/`
- Please rename the subfolders to indicate the OS name and version. For example, rename 'linux-distro1-version' to 'Fedora Server 37'.
- It is recommended to choose one OS per group below, an OS based on an old kernel, and an OS based on an kernel version close to the latest kernel for certification testing to cover as many kernel versions as possible. 
    - Groups based on heritage: RHEL/Fedora/CentOS/AlmaLinux, SLES/openSUSE, Ubuntu/Debian, or NetBSD/OpenBSD/FreeBSD.
    - For example:
        - For SystemReady SR, You can choose RHEL, SLES, Ubuntu, CentOS Linux 7, and Fedora for your testing.
        - For SystemReady ES, you can choose Fedora, openSUSE, Ubuntu, and CentOS 7 for your testing.
- For each OS being tested
    - Install the OS from some media (eg. USB drive) to a disk (e.g. SATA, NVMe, USB), and boot it.
    - A network controller (PCIe or USB) may be needed during installation.
    - The OS commands output and files are required. For collecting them, run the commands below to download and run the script linux-distro-cmds.sh. Then, copy all the files that are collected in ~/CertTestLogs folder to the template's ./os-logs/[distroname] [distroversion]/ folder.
        - If network port doesn't get IP address, giving "dhclient" or "ip a add <assigned IP address>/255.255.255.0 dev eth<number>" and "ip link set dev eth<number> up" a try might be helpful. 
        - If the network doesn't work, please manually run the commands in ./os-logs/linux-distro-cmds.sh. 
        - For manually running the ping commnad, if the system is connected to intranet, please ping the IP address of another system within the intranet instead.
```
    sudo -s                      (Prevent from running into "Permission denied" error)
    wget https://gitlab.arm.com/systemready/systemready-es-sr-template/-/raw/main/os-logs/linux-distro-cmds.sh
    chmod +x linux-distro-cmds.sh
    ./linux-distro-cmds.sh
```


#### `./os-logs/vmware [version]/`
- Please rename the subfolders. For example, rename 'vmware-version' to 'VMware ESXi 1.12'
- Install VMware ESXi to a disk, and boot it. Note that a network controller (PCIe or USB) is required to complete the installation.
- Connect to vSphere web console over network (from a remote client) and capture some screenshots.
- Follow the instruction in https://kb.vmware.com/s/article/2004746 to enable ESXi Shell and SSH.
- Connect ESXi shell through the SSH and Run the following commands and attach the logs:
```
    dmesg
    lspci
    irqinfo
    localcli storage core adapter list
    localcli hardware pci list
    esxcli hardware cpu list
    esxcli hardware usb passthrough device list
    esxcli storage filesystem list
    esxcli storage core device list
    cd /dev/disks
    ls
    vim-cmd hostsvc/hosthardware
    vmware -v
    reboot -f
```

#### `./os-logs/winpe [version]/`
- Boot WinPE from USB key
- Include screenshots and video of WinPE booting (from FW menu to OS startup, and keyboard working in command prompt)
- Run SAC commands and WinPE commands below:
```
    id
    d
    i
    s
    cmd (run this command after the message "EVENT: The CMD command is now available.")
    ch  (run this command after the message "EVENT:   A new channel has been created".)
    Esc + Tab (Switch channel to command channel (like Cmd0001) for running WinPE commands)
    ver
    pnputil/enum-devices
    pnputil/enum-drivers
    diskpart
    list disk
    list volume
    exit
    c:
    dir/w
    ipconfig /all
    date 
    time
    Esc + Tab (Switch channel to SAC> for running SAC commands)
    shutdown
```