Skip to content
  1. May 09, 2025
    • John Chung's avatar
      Add FaultLog DumpEntry interface to FaultLog object · de253b21
      John Chung authored
      
      
      Type property can used to identify CPER / Crashdump log.
      PrimaryLogId can used to identify the path for log file.
      
      $> busctl introspect xyz.openbmc_project.Dump.Manager \
              /xyz/openbmc_project/dump/faultlog/entry/1
      
      xyz.openbmc_project.Dump.Entry.FaultLog interface -
      .AdditionalTypeName                     property  s
      .PrimaryLogId                           property  s
      .Type                                   property  s
      
      Signed-off-by: John Chung's avatarJohn Chung <john.chung@arm.com>
      Change-Id: I9a2a089bedc120da4cde8521f032fea92a84de10
      de253b21
  2. May 02, 2025
    • Deepak Kamat's avatar
      Dreport: Add SoftIRQ Information Collection · 9a5fac9a
      Deepak Kamat authored
      
      
      Add functionality to collect softIRQ statistics as part of the debug
      data collection process. The new file will contain:
      
      - softIRQs: Software interrupt statistics including counts and
        processing details
      
      Including this information in debug reports will enhance
      troubleshooting capabilities for system performance and resource
      utilization in OpenBMC based systems. It provides a snapshot of
      software interrupt processing activity at the time of report generation.
      
      This addition will help identify potential bottlenecks and excessive
      softIRQ activity that may impact system performance or stability,
      particularly for network stack processing, timers, and tasklet
      execution analysis.
      
      Tested on the qemuarm platform.
      
      '''
      Dump size: 82k
      Untar Dump Size: 936K
      Added File Size: 282 bytes
      Time for the Dump Completion: 15sec
      Time for the command to run: 0.038s
      
      Data can be used like
      Network Analysis (NET_RX/NET_TX)
      NET_RX: 170 suggests:
      170 packets received via network interfaces during collection
      Potential issues if:
      Counts increase without traffic
      Asymmetry between NET_RX and actual received packets (ifconfig/ethtool)
      
      root@qemuarm:~# busctl call xyz.openbmc_project.Dump.Manager /xyz/openbmc_project/dump/bmc xyz.openbmc_project.Dump.Create CreateDump a{sv} 1 "i" i 1234
      
      root@qemuarm:~# journalctl -f -u xyz.openbmc_project.Dump.Manager
      ...
      Mar 26 04:09:34 qemuarm phosphor-dump-manager[220]: OriginatorId is not provided
      Mar 26 04:09:34 qemuarm phosphor-dump-manager[220]: OriginatorType is not provided. Replacing the string with the default value
      Mar 26 04:09:34 qemuarm phosphor-dump-manager[220]: Initiating new BMC dump with type: user path:
      ...
      Mar 26 04:09:49 qemuarm phosphor-dump-manager[304]: Wed Mar 26 04:09:49 UTC 2025 Report is available in /var/lib/phosphor-debug-collector/dumps/1
      Mar 26 04:09:49 qemuarm phosphor-dump-manager[304]: Wed Mar 26 04:09:49 UTC 2025 Successfully completed
      Mar 26 04:09:49 qemuarm phosphor-dump-manager[220]: User initiated dump completed, resetting flag
      
      root@qemuarm:~# time cat /proc/softirqs
      ...
      real    0m0.038s
      user    0m0.021s
      sys     0m0.002s
      root@qemuarm:~#
      
      ls -lrth obmcdump_1_1742962174.tar.xz
      -rw-r--r-- 1 dkamat domain-users 82K Mar 26 10:16 obmcdump_1_1742962174.tar.xz
      
      ls -lrth obmcdump_1_1742962174/
      total 936K
      -rw-r--r-- 1 dkamat domain-users  223 Mar 26 09:39 summary.log
      ...
      -rw-r--r-- 1 dkamat domain-users  282 Mar 26 09:39 softIRQs.log
      
      cat obmcdump_1_1742962174/softIRQs.log | head
                          CPU0
                HI:          0
             TIMER:       4457
            NET_TX:          3
            NET_RX:        170
             BLOCK:          0
          IRQ_POLL:          0
           TASKLET:          1
             SCHED:          0
           HRTIMER:          0
      '''
      
      Change-Id: I154e9a4ddfad4099ea57d31fb689ac9eefdda8e7
      Signed-off-by: default avatarDeepak Kamat <dkamat@nvidia.com>
      9a5fac9a
    • Deepak Kamat's avatar
      Dreport: Add D-Bus Statistics Collection · 73cec52d
      Deepak Kamat authored
      
      
      Add functionality to collect statistical information about
      D-Bus usage as part of the debug data collection process.
      
      Including this information in debug reports will enhance
      troubleshooting capabilities for D-Bus related issues in
      OpenBMC-based systems. It provides a snapshot of D-Bus
      activity and performance at the time of report generation.
      
      This addition will help identify potential communication
      bottlenecks, unexpected service behaviors, and performance
      issues related to inter-process communication, system
      management interfaces, and sensor data retrieval.
      
      Tested on qemuarm.
      
      '''
      Dump size: 76k
      Untar Dump Size:  916K
      Added File Size: 79K
      Time for the Dump Completion: 15 sec
      Time for the command to run: 0.260s
      
      [1] root@qemuarm:~# time dbus-send --system --dest=org.freedesktop.DBus --type=method_call --print-reply /org/freedesktop/DBus org.freedesktop.DBus.Debug.Stats.GetStats
      real    0m0.260s
      user    0m0.049s
      sys     0m0.204s
      
      High OutgoingBytes:
      While all connections show 0 in the given data, a connection
      with consistently high OutgoingBytes might suggest a blocked
      receiver or network issues.
      
      Rapidly Increasing NameObjects:
      If a connection's NameObjects count grows quickly over time,
      it may indicate a resource leak.
      
      Mismatched IncomingBytes and OutgoingBytes:
      Large discrepancies between these values across multiple
      connections could suggest communication issues.
      
      High Number of Connections:
      An unusually high number of active connections
      (the log shows 33) might indicate connection leaks
      or denial-of-service attempts.
      
      [2] root@qemuarm:~# busctl call xyz.openbmc_project.Dump.Manager /xyz/openbmc_project/dump/bmc xyz.openbmc_project.Dump.Create CreateDump a{sv} 1 "i" i 1234
      
      ...
      [3] Mar 26 11:14:45 qemuarm phosphor-dump-manager[223]: Initiating new BMC dump with type: user path:
      ...
      [4] Mar 26 11:15:00 qemuarm phosphor-dump-manager[305]: Wed Mar 26 11:15:00 UTC 2025 Successfully completed
      [5] Mar 26 11:15:00 qemuarm phosphor-dump-manager[223]: User initiated dump completed, resetting flag
      
      ls -lrth obmcdump_1_1742987685
      total 916K
      ...
      -rw-r--r-- 1 dkamat domain-users  79K Mar 26 16:44 dbusstats.log
      ...
      
      ls -lrth obmcdump_1_1742987685/dbusstats.log
      domain-users 79K Mar 26 16:44 obmcdump_1_1742987685/dbusstats.log
      cat obmcdump_1_1742987685/dbusstats.log | -n 10
      ...
                        array [
                           dict entry(
                              string "NameObjects"
                              uint32 0
                           )
                           dict entry(
                              string "MatchBytes"
                              uint32 6150
                           )
                           dict entry(
                              string "Matches"
                              uint32 8
                           )
                           dict entry(
                              string "ReplyObjects"
                              uint32 0
                           )
                           dict entry(
                              string "IncomingBytes"
                              uint32 16
                           )
                           dict entry(
                              string "IncomingFds"
                              uint32 0
                           )
                           dict entry(
                              string "OutgoingBytes"
                              uint32 0
                           )
                           dict entry(
                              string "OutgoingFds"
                              uint32 0
                           )
                           dict entry(
                              string "ActivationRequestBytes"
                              uint32
                           )
                           dict entry(
                              string "ActivationRequestFds"
                              uint32 0
                           )
                        ]
                     }
      '''
      
      Change-Id: I37d93f1a6a55c306cd3a05e7319c99413353aa98
      Signed-off-by: default avatarDeepak Kamat <dkamat@nvidia.com>
      73cec52d
  3. Apr 15, 2025
  4. Apr 08, 2025
    • Zami Seck's avatar
      dreport: Add audit.log Data to Dump · 4e0251d1
      Zami Seck authored
      
      
      Add all /var/log/audit/audit.log files to BMC dumps. This
      file contains a log of the events that occurred and who initiated
      them.
      
      Tested:
      ```
      BMCDUMP$ ls -la audit-log.log
      -rw-r--r-- 1 xxxx xxxx 3993 Sep 19 10:59 audit-log.log
      
      $ tar -tvf $DUMPFILE | grep audit
      
      drwx------ root/root 0 2023-03-22 14:07 BMCDUMP.SIMP10R.00000000.20230322190714/audit/
      
      -rw------- root/root   1729455 2023-03-22 14:07 BMCDUMP.SIMP10R.00000000.20230322190714/audit/audit.log
      
      -r-------- root/root   2097279 2023-03-22 14:07 BMCDUMP.SIMP10R.00000000.20230322190714/audit/audit.log.3
      
      -r-------- root/root   2097160 2023-03-22 14:07 BMCDUMP.SIMP10R.00000000.20230322190714/audit/audit.log.2
      
      -r-------- root/root   2097227 2023-03-22 14:07 BMCDUMP.SIMP10R.00000000.20230322190714/audit/audit.log.1
      ```
      
      Change-Id: I8e36edeca9ad40f8ddc02becf3a9246e0989d72a
      Signed-off-by: default avatarJanet Adkins <janeta@us.ibm.com>
      Signed-off-by: default avatarGopichand Paturi <gopichandpaturi@gmail.com>
      4e0251d1
  5. Apr 01, 2025
    • Deepak Kamat's avatar
      Dreport: Add Kernel Command Line Info · 07aa6b6c
      Deepak Kamat authored
      
      
      Add functionality to collect kernel command line
      information as part of the debug data collection
      process. The new plugin retrieves data from
      /proc/cmdline, which contains the kernel parameters
      passed during boot.
      
      Including this information in debug reports will
      enhance troubleshooting capabilities for system
      boot and configuration issues in OpenBMC-based
      systems. It provides a snapshot of the kernel's
      runtime parameters, which can help identify
      misconfigurations or diagnose boot-related
      problems.
      
      Tested on qemuarm platform.
      
      '''
      Kernel cmdline: console=ttyAMA0 root=/dev/vda rw
      Enables serial console debugging
      Specifies root filesystem location
      Mounts root in read-write mode
      
      [1] busctl call xyz.openbmc_project.Dump.Manager /xyz/openbmc_project/dump/bmc xyz.openbmc_project.Dump.Create CreateDump a{sv} 1 "i" i 1234
      
      tar -xvf obmcdump_1_1742903558.tar.xz
      obmcdump_1_1742903558/
      obmcdump_1_1742903558/summary.log
      ...
      obmcdump_1_1742903558/kernalcmdline.log
      ...
      
      cat obmcdump_1_1742903558/kernalcmdline.log | head
      console=ttyAMA0 root=/dev/vda rw
      '''
      
      Change-Id: I71177a2d3ff7b9b80ce11eb22ef01434c6ecd686
      Signed-off-by: default avatarDeepak Kamat <dkamat@nvidia.com>
      07aa6b6c
  6. Mar 26, 2025
    • Deepak Kamat's avatar
      Dreport: Add Mount Information File Collection · a3e6a48c
      Deepak Kamat authored
      
      
      Add functionality to collect system mount information as part of the
      debug data collection process. The mount info file contains details
      about all mounted filesystems including mount points, filesystem
      types, and mount options.
      
      Including mount information in debug reports will help with
      troubleshooting filesystem and storage related issues in OpenBMC
      based systems by providing a snapshot of the system's mount
      configuration at the time of report generation.
      
      Tested on the qemuarm platform.
      
      '''
      busctl call xyz.openbmc_project.Dump.Manager /xyz/openbmc_project/dump/bmc xyz.openbmc_project.Dump.Create CreateDump a{sv} 1 "i" i 1234
      
      root@qemuarm:~# journalctl -f -u  xyz.openbmc_project.Dump.Manager.service
      ...
      Mar 20 07:22:10 qemuarm phosphor-dump-manager[367]: Thu Mar 20 07:22:10 UTC 2025 Report is available in /var/lib/phosphor-debug-collector/dumps/1
      Mar 20 07:22:10 qemuarm phosphor-dump-manager[367]: Thu Mar 20 07:22:10 UTC 2025 Successfully completed
      Mar 20 07:22:10 qemuarm phosphor-dump-manager[223]: User initiated dump completed, resetting flag
      
      root@qemuarm:~# ls /var/lib/phosphor-debug-collector/dumps/1/obmcdump_1_1742479997.tar.xz
      /var/lib/phosphor-debug-collector/dumps/1/obmcdump_1_1742479997.tar.xz
      
      tar -xvf obmcdump_1_1742479997.tar.xz
      obmcdump_1_1742479997/
      obmcdump_1_1742479997/summary.log
      ...
      obmcdump_1_1742479997/mountinfo.log
      ...
      
      cat obmcdump_1_1742479997/mountinfo.log
      /dev/vda on / type ext4 (rw,relatime)
      devtmpfs on /dev type devtmpfs (rw,relatime,size=118004k,nr_inodes=29501,mode=755)
      proc on /proc type proc (rw,relatime)
      sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
      tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
      devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
      tmpfs on /run type tmpfs (rw,nosuid,nodev,size=47612k,nr_inodes=819200,mode=755)
      cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
      bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
      mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
      debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
      tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
      tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=119032k,nr_inodes=1048576)
      tmpfs on /run/credentials/systemd-journald.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,noswap)
      tmpfs on /run/credentials/systemd-resolved.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,noswap)
      tmpfs on /var/volatile type tmpfs (rw,relatime)
      tmpfs on /run/credentials/systemd-networkd.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,noswap)
      tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
      tmpfs on /run/credentials/getty@tty1.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,noswap)
      '''
      
      Change-Id: I4e8fc3c8ba5a8ec40ec91256e0a2e7390e915be9
      Signed-off-by: default avatarDeepak Kamat <dkamat@nvidia.com>
      a3e6a48c
  7. Feb 11, 2025
    • Jian Zhang's avatar
      watch: fix memory leak in Watch class · fa6a47bf
      Jian Zhang authored
      See systemd manunal[0],
      ```
      If the second parameter of sd_event_add_io() is NULL no reference to
      the event source object is returned. In this case the event source is
      considered "floating", and will be destroyed implicitly when the event
      loop itself is destroyed.
      ```
      
      The Watch class should release the event source object when destructing
      the Watch object.
      
      Tested:
      1. Run CreateDump Stress Test
      ```
      while true; do busctl call xyz.openbmc_project.Dump.Manager /xyz/openbmc_project/dump/bmc xyz.openbmc_project.Dump.Create CreateDump a{sv} 0; sleep 200; done
      
      ```
      2. run the command to monitor the memory heap size of dump-manager,
      and no memory leak found.
      ~# watch -n 10 "cat /proc/[PID]/smaps | grep heap -A 4 | grep ^Rss"
      
      [0]: https://www.freedesktop.org/software/systemd/man/latest/sd_event_add_io.html
      
      
      
      Change-Id: I530bda40a7a7307ee87f03a24c242dde4eae2b14
      Signed-off-by: default avatarJian Zhang <zhangjian.3032@bytedance.com>
      fa6a47bf
    • Jian Zhang's avatar
      dreport: collecting all namespace journal logs · 41167a2a
      Jian Zhang authored
      For BMCs that support multiple namespaces to manage logs, add the
      --namespace=* option to the journalctl command to collect all logs.
      
      refer to systemd-journald.service.html[0].
      
      Tested:
      1. Prepare a namespace configuration, to isolate logs (in general for
         the third-party code, like silicon vendor, etc), avoid too many
         unexpected to noise the system.
      ```
      ~# cp /etc/systemd/journald.conf /etc/systemd/journald@intel.conf
      ~# mkdir -p /etc/systemd/system/com.intel.crashdump.service.d
      ~# echo "[Service]" > /etc/systemd/system/com.intel.crashdump.service.d/override.conf
      ~# echo "LogNamespace=intel" >> /etc/systemd/system/com.intel.crashdump.service.d/override.conf
      ~# reboot # or restart systemd-journald, and daemon-reload
      ````
      
      2. using journalctl vs journalctl --namespace=*
      ```
      ~# journalctl -u com.intel.crashdump.service
      Dec 10 16:56:49 xxx systemd[1]: Starting Intel BMC CPU Crashdump...
      Dec 10 16:56:55 xxx systemd[1]: Started Intel BMC CPU Crashdump.
      ```
      
      ```
      ~# journalctl --namespace=*
      Dec 10 16:56:49 xxx systemd[1]: Starting Intel BMC CPU Crashdump...
      Dec 10 16:56:55 xxx systemd[1]: Started Intel BMC CPU Crashdump.
      Dec 10 16:56:57 xxx crashdump[258]: Crashdump version: ....
      Dec 10 16:56:57 xxx crashdump[258]: Initializing crashdump...
      ....
      ```
      
      3. After this change, check the journal logs, and the journal logs
         should be collect the logs from all namespaces.
      ```
      cat journal-pretty.log| ag "Initializing crashdump"
      	"MESSAGE" : "Initializing crashdump...",
      ```
      
      [0]: https://www.freedesktop.org/software/systemd/man/latest/systemd-journald.service.html#Journal%20Namespaces
      
      
      
      Change-Id: I70f33eb6119cbe041bb0dc2aac527e02d8f5e757
      Signed-off-by: default avatarJian Zhang <zhangjian.3032@bytedance.com>
      41167a2a
    • Jian Zhang's avatar
      dreport: fix procfd missing · d56bab55
      Jian Zhang authored
      
      
      During the execution of 'ls -Al /proc/*/fd/ 2> /dev/null', some
      processes may stop, like `while true; do echo "Hello" > /dev/null; done`
      
      Causing the ls cmd to fail, and as a result, the procfs.log will not be
      collected in the dump log.
      
      So ensure ls procfd returns success.
      
      Tested:
      1. Run `while true; do echo "Hello" > /dev/null; done` in the
      background.
      
      2. Run `set -e; while true; do ls -Al /proc/*/fd/ 2> /dev/null
      || true; done`, the command will not stop
      (before the fix, the command will stop quickly).
      
      Change-Id: I1f4e8fda000f303fd579825e0a94efdba5a097eb
      Signed-off-by: default avatarJian Zhang <zhangjian.3032@bytedance.com>
      d56bab55
    • Patrick Williams's avatar
      clang-tidy: filter out subproject headers · ffbb9d6d
      Patrick Williams authored
      
      
      Subproject headers should be filtered out so that when building
      outside of the CI docker container (or Yocto) we do not get tons
      of clang-tidy failures for things present in those subprojects.
      
      Signed-off-by: default avatarPatrick Williams <patrick@stwcx.xyz>
      Change-Id: I35e909d667ce700e6c68ff3c2f269f1ef549dca3
      ffbb9d6d
    • Patrick Williams's avatar
      openpower-dumps: eliminate special generated config file · 2c63448f
      Patrick Williams authored
      
      
      Meson calls for 'clang-tidy' changed behavior such that it will call
      for all "c-like" files in a repository rather than just the ones pulled
      in as dependencies.  When openpower extensions are disabled, this means
      that we fail clang-tidy because required header files do not exist.
      
      The generated header file has a few constants that are not openpower
      specific.  Move them to the top-level config.h.
      
      Signed-off-by: default avatarPatrick Williams <patrick@stwcx.xyz>
      Change-Id: I2ee5c9246b9c47b564130c22ff8e02fee7f752b1
      2c63448f
    • Patrick Williams's avatar
      meson: reformat with meson formatter · 57de251c
      Patrick Williams authored
      
      
      Apply the `meson format` results.
      
      Change-Id: Ic556a6e6065b9df4cf778ba91d7bf5619f2955ca
      Signed-off-by: default avatarPatrick Williams <patrick@stwcx.xyz>
      57de251c
  8. Feb 06, 2025
  9. Jan 16, 2025
    • Lakshmi Yadlapati's avatar
      Move to libpldm pldm_transport APIs · 03414ffa
      Lakshmi Yadlapati authored
      
      
      Replaced pldm transport APIs with libpldm pldm_transport APIs to
      remove the dependency on pldm.
      
      This change removes the dependency on pldm by utilizing the
      standardized libpldm APIs for transport operations, improving
      maintainability and compatibility.
      
      We don't currently have the infrastructure in place to get the correct
      TIDs, so to keep everything working as before use the EID as the TID in
      the EID-to-TID mapping.
      
      Test: Tested the resource dump download and delete using GUI
      and it works as expected.
      
      Change-Id: I3ec741024c7b99f0ee630678f3b3b49f741e1ea1
      Signed-off-by: default avatarLakshmi Yadlapati <lakshmiy@us.ibm.com>
      Signed-off-by: default avatarEddie James <eajames@linux.ibm.com>
      03414ffa
  10. Jan 15, 2025
  11. Dec 26, 2024
    • Matt Fischer's avatar
      Fix some things in dreport · 971d1a74
      Matt Fischer authored
      
      
      The version command for dreport doesn't do anything
      and so this commit removes it.
      
      Additionally I was having trouble with --quiet and
      --verbose. After enough squinting one can see that
      a different dash character was being used in the
      original code, likely these never worked.
      
      Fixes openbmc/phosphor-debug-collector#18
      
      Change-Id: I86eb58e848790122f2c23841de3c39e18acd7d0b
      Signed-off-by: default avatarMatt Fischer <matthew.fischer@hpe.com>
      971d1a74
  12. Dec 18, 2024
    • Patrick Williams's avatar
      clang-format: re-format for clang-19 · 454dee5f
      Patrick Williams authored
      
      
      clang-format-19 isn't compatible with the clang-format-18 output, so we
      need to reformat the code with the latest version.  A few parameters
      in clang-tidy have been deprecated, so adjust the style file
      accordingly.
      
      See Ie2f6eb3b043f2d655c9df806815afd7971fd0947 for updated style.
      See I88192b41ab7a95599a90915013579608af7bc56f for clang-19 enablement.
      
      Change-Id: Ice3075169401fab4568d278569e08892702411a7
      Signed-off-by: default avatarPatrick Williams <patrick@stwcx.xyz>
      454dee5f
  13. Nov 28, 2024
Loading