This change will group the commands executed for block and scsi devices
into a section with a comment.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
In addition we did get feedback for promoting some commands to the top.
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands executed for Z device subsystem
into a section with a comment.
By grouping, we try to enhance the handling in our runtime.out file and
help to understand the purpose of command groups.
In addition we did get feedback for promoting some commands to the top.
lspci -t was added on request of SMEs during the discussions
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Acked-by: Nicklas Schnelle <Niklas.Schnelle@ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands executed for overall state into a
first section with a comment.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
In addition we did get feedback for promoting some commands to the top.
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands executed for crypto, special SW or
specific for a distro into sections with a comment.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands with long output into two
sections with a comment.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
We did get feedback for moveing some independent long output files to
the end for speed up the scrolling in the upper parts.
Files which already have a separate output file are bundled in a
separate group at the end.
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will regroup & reorder the commands executed for network.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
In addition we did get feedback for promoting some commands to the top.
"ip -br a" was added on request of SMEs during the discussions
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Acked-by: Sandy Winter <WINTERA@de.ibm.com>
Acked-by: Stefan Raspl <stefan.raspl@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
'which' tool is packaged separately from other core tools and there will
likely be scenarios where this check will fail because 'which' is
unavailable.
The "type" is builtin itself and should solve this problem.
For a built in command, we will not suppress error out -> removing 2>&1.
On this replace - in call_run_command - get cmd_type as variable only once
Reported by: Peter Oberparleiter <oberpar@linux.ibm.com>
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
a missing dump2tar command did show misleading error messages
now check it and give clear messages
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
as order itself does not matter for collection of log and config files,
we decided to use alphabetic order for better maintenance
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
critical paths in dump2tar data collection have been already excluded -
this update will sync the excludes in the fall back, running in case
dump2tar is missing or failing on a system
By this we also sync the filtering to match exact the same pattern.
Reported by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Fixes: 12c84469fd ("dbginfo.sh: exclude reading page_idle/bitmap sysfs
attribute")
Fixes: b627b8d8e1 ("Initial s390-tools-2.0.0 import")
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
adopt all indents to have the same setting overall - NO logic changes !
tooling: vim used with setting "shiftwidth=8",expandtab,autoindent
commands "gg=G",":retab"
some additional line brakes to stay below 100 chars
and one line added manual before "collect_procfs"
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Extend cpictl to report Linux distribution type and version information,
and another sublevel of Linux kernel version data via the system level
field of the SCLP CPI firmware interface. This data is intended to be
decoded by the IBM Z HMC web-UI to provide users a more detailed view of
Linux software levels installed in LPARs.
For this purpose, the format of the 64-bit system level word is extended
in a compatible way, that is, the meaning of existing fields remains
unchanged while additional data is added to previously unused fields.
The new format is defined as 0xabccddeeeeffgghh, where:
- a=statistics flags
- b=distro id
- c=distro major version
- d=distro minor version(s)
- e=kernel sublevel 2
- f=kernel version
- g=kernel patchlevel
- h=kernel sublevel 1
This format will be automatically reported for supported distributions
based on data from /etc/os-release. As before, users can override the
data by specifying a custom system level word in /etc/sysconfig/cpi.
To support manual specification of new data fields, the format of
cpictl's -L command line option and the CPI_SYSTEM_LEVEL environment
variable are extended as follows:
- 0x<level>
Specifies the system level as 64 bit hex value
- [[[flags:]distro_id:distro_version:]kernel_version]
Specifies the system level as list of optional fields. Any field that
is omitted is initialized with automatically determined values.
Signed-off-by: Peter Oberparleiter <oberpar@linux.ibm.com>
Reviewed-by: Jan Höppner <hoeppner@linux.ibm.com>
Acked-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Under certain conditions, e.g. in a KVM guest, a write operation
to /sys/firmware/cpi/set can result in an error message to
stderr while the script returns with a zero exit code, indicating
success.
This can cause confusion if cpictl is called by systemd due to
the existence of a writable /sys/firmware/cpi/. Let's silence
cpictl when writing to the /sys/firmware/cpi/set. This is OK, as
nothing can be inferred from failure or success from the perspective
of the invoking Linux instance.
Fixes: https://github.com/ibm-s390-linux/s390-tools/issues/120
Signed-off-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
Reviewed-by: Jan Höppner <hoeppner@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Without the fix, readers of syslog / systemd-journal can erroneously
think that dbginfo.sh was incomplete because the last logged step is
"17 of 18".
The fix also makes the missing line appear in dbginfo.log inside the
generated DBGINFO....tgz.
s390-tools v2.17.0 commit b7807d0195 ("dbginfo.sh: Add KVM commands
and rework data collection") turned create_package into a numbered
step.
Snippet of terminal output:
...
17 of 18: Postprocessing
18 of 18: Finalizing: Creating archive with collected data
Collected data was saved to:
>> /tmp/DBGINFO-....tgz <<
Please review all collected data before sending to your service organization.
Syslog before:
...
dbginfo.sh[...]: 17 of 18: Postprocessing
dbginfo.sh[...]: Data collection completed
Syslog after:
...
dbginfo.sh[...]: 17 of 18: Postprocessing
dbginfo.sh[...]: 18 of 18: Finalizing: Creating archive with collected data
dbginfo.sh[...]: Data collection completed
Fixes: b7807d0195 ("dbginfo.sh: Add KVM commands and rework data collection")
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
When built with `D=1` chreipl-fcp-mpath automatically creates debug data
for the helpers that run during udev event processing. This is the only
way to effectively debug what is happening during event processing in
chreipl-fcp-mpath.
Collect this data under its default path in /run/udev/.
Also collect the chreipl-fcp-mpath statefile which contains WWID, Device
Bus-ID and LUN of the last path that was detected to represent a FCP
re-IPL device.
Suggested-by: Steffen Maier <maier@linux.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Benjamin Block <bblock@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
running mpio data disks with offline paths might cause dbginfo
to hang up on the command "blockdev --report" -> we remove
it to ensure a non blocking data collection.
In addition we add a second kill step with the same timeout
value to retry stopping a command
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Add new tool named 'sclpdbf'. This script displays the contents of
s390dbf sclp debug logs in human-readable format.
Signed-off-by: Peter Oberparleiter <oberpar@linux.ibm.com>
Reviewed-by: Jan Höppner <hoeppner@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Had a confusing decoded trace record:
Timestamp : ...
Area : SCSI
...
Tag : rsl_err
...
SCSI result : 0x00000002
SCSI opcode : 1a...
FCP rsp inf cod: 0x00
FCP rsp IU : 00000000 00000000 00000a02 000000ff
00000012 00000000
^^
and was wondering where that sense data would be or if the
corresponding PAY record "fcp_riu" was lost due to trace area wrapping.
So the fix here adds the following two lines to the same trace record:
FCP rsp IU len : 42
FCP rsp IU all : record not available anymore.
Similarly, the following LUN reset Task Management Function response
was missing the last two lines regarding the full FCP response IU
and its optional part with the FCP response info:
Timestamp : ...
Area : SCSI
...
Tag : lr_okay
...
FCP rsp inf cod: 0x00
FCP rsp IU : 00000000 00000000 00000100 00000000
00000000 00000008
^^
FCP rsp IU len : 32
FCP rsp IU all : record not available anymore.
As of today, depending on the kernel version we have
either fcp_sns or fcp_riu PAY trace records. See Linux kernel v4.14 commit
12c3e5754c80 ("scsi: zfcp: fix payload with full FCP_RSP IU in SCSI trace
records").
Always print non-zero payload length independent of payload type.
Actually this also fixes a regression for wrapped away fcp_sns.
If there is no fcp_sns it either wrapped away or it's a kernel with fcp_riu.
Only print fcp_riu if it contains more content than the preceding
unconditional fixed-size field "FCP rsp IU".
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Reviewed-by: Fedor Loshakov <loshakov@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
We found situations, where the debug commands from dbginfo.sh
did overwrite some debug files (e.g. OSA s390dbf) - so we change
the collection order
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
- adding new print function for simpler output customizing
- consolidate some output files
- sync structures and use of checks
- some formatting updates
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
- clenaup outdated kernel - we no longer have 2.x kenrel in support
- add timeout on command excution - ensure to not miss all data for a
single command to hang
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Zipl.conf is collected via /etc/*.conf in CONFIGFILES but BLS snippets
do not appear in /etc/zipl.conf.
Complements v2.5.0 commit d280e5b1fa ("zipl: Add BootLoaderSpec support")
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Add virsh commands for KVM debug data collection and rework the domain
data collection and step numbering. Update the man page accordingly.
[hoeppner@linux.ibm.com: Reword commit message]
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Since crash 7.2.9 commit
f1f43bf355
in support of kernel v5.9 commit 0990d836cecb
("s390/debug: debug feature version 3"),
the symbolic caller function name is left aligned instead of right
aligned, so now we alternatively have to remove whitespace between
function name and offset.
Otherwise, zfcpdbf output contains wrong warnings about ambiguous payload
records with wrong values for reqid and area, as well as incorrectly
decoded trace records with wrong content values and potential error
messages on not defined record id, and finally a potential list of
occurrences of unknown tags which should empty:
Loading trace records ...(this might take a while)
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:V^D^@fcp_
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:W^D^@fsf_
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:i^B^@san_
...
Timestamp : ...
Area : SAN
Subarea : 00
Level : 1
Exception : -
CPU ID : 0001
Caller : <zfcp_dbf_san_in_els
Record ID : 0
Tag : 6^B^Cfcie
Request ID : >>>wrong<<<
Destination ID : >>>wrong<<<
SAN record id=0 not defined.
Timestamp : ...
Area : SAN
Subarea : 00
Level : 1
Exception : -
CPU ID : 0002
Caller : <zfcp_dbf_san_in_els
Record ID : 0
Tag : 6^B^Cfcie
Request ID : >>>wrong<<<
Destination ID : >>>wrong<<<
SAN record id=0 not defined.
...
occurrences of unknown tags:
...
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Reviewed-by: Benjamin Block <bblock@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Some drivers provide per-queue IRQ coalescing information, collect it.
Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Format and print HBA FC Endpoint Security trace records with a record ID
of ZFCP_DBF_HBA_FCES (5). Align output to the one of FSF response trace
records for readability. Note that the FSF protocol status is most likely
FSF_PROT_GOOD (0x00000001).
Signed-off-by: Jens Remus <jremus@linux.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This can be helpful when diagnosing network performance issues.
Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This can be helpful in diagnosing network-related issues.
Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
The basic statistics are already collected via ifconfig. But that tool
is deprecated, and provides no way of collecting the extended stats.
So also gather data from 'ip link'.
Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This patch adds ethtool commands for network interfaces.
Especially the offload settings are important.
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This patch adds the base lspci command to list pci devices
and the smc_dbg command from the smc-tools package to show
SMC socket related information.
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Function get_payload_records() reading from disk, blindly adds payload
record (junks) to a list in a 2-stage hierarchical hash keyed with
request ID and PAY area.
On adapter offline/online, zfcp resets the request ID to re-start with 1.
Hence it's not a unique foreign key in the lifetime of the PAY area.
The last payload record (junks) read for the same request ID and PAY area
is the one printed as named PAY field in the tool output.
This is not necessarily the chronologically last record due to PAY area
wrapping.
With this workaround, we:
* warn on reading PAY records if ambiguous entries occurred,
* print additional PAY timestamps which the user can compare
with the actual trace record timestamps to determine if they are close
enough to indicate a match.
We only found this by accident because we kept "short" "payload" being a
built-in part of some actual trace record types in areas HBA, SAN, SCSI.
For above cases, the full PAYload did not match the
built-in "short" "payload".
While at it, also add a comment that PAY output can also be wrong
in another case when junks of multi-part PAY records are missing due
to PAY area wrapping.
Below is an abbreviated example output with this workaround:
Loading trace records ...(this might take a while)
Warning: Ambiguous PAYload records. reqid:0000000000000037 area:san_res
...
Timestamp : 2017-07-07-18:01:34:574827
Area : SAN
...
Tag : fssct_1
Request ID : 0x0000000000000037
Destination ID : 0x00fffffc
SAN req short : 01000000 fc020000 01720ffc 00000000
00000008
SAN req length : 20
Timestamp : 2017-07-07-18:01:34:619310 ------------------+
Area : SAN |
... |
Tag : fsscth2 |
Request ID : 0x0000000000000037 |
Destination ID : 0x00fffffc |
SAN resp short : 01000000 fc020000 80020000 00000000 |
00671a07 00000000 c05076ff d6801850 |
00671a7e 00000000 c05076ff d6801968 |
SAN resp length: 16384 |
Payload time : 2017-07-07-18:16:17:165708 <== does not match
San resp info : 01000000 fc020000 80020000 00000000 <== wrong
4a49424d 20202020 20323936 34202020 <== wrong
20202020 20202020 20303230 30303030 <== wrong
30303745 43383720 20504348 49443a20 <== wrong
30313938 204e5049 5620556c 7049643a <== wrong
20303536 30303330 30202020 00000000 <== wrong
00000000 00000000 00000000 00000000 <== wrong
... <== wrong
...
Timestamp : 2017-07-07-18:16:17:165713
Area : SAN
...
Tag : fsscth2
Request ID : 0x0000000000000037 <== same reqid & pay "area"
Destination ID : 0x00fffffc
SAN resp short : 01000000 fc020000 80020000 00000000
4a49424d 20202020 20323936 34202020
20202020 20202020 20303230 30303030
SAN resp length: 273
Payload time : 2017-07-07-18:16:17:165708 <== 2nd & last, match
close to Timestamp
San resp info : 01000000 fc020000 80020000 00000000
4a49424d 20202020 20323936 34202020
20202020 20202020 20303230 30303030
30303745 43383720 20504348 49443a20
30313938 204e5049 5620556c 7049643a
20303536 30303330 30202020 00000000
00000000 00000000 00000000 00000000
...
Signed-off-by: Steffen Maier <maier@linux.vnet.ibm.com>
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
The output of lsof command is sometimes too long and makes it difficult
to search in runtime.out. Move the output to a separate file.
Signed-off-by: Sa Liu <saliu@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
