This can now include /lib/modprobe.d/10-unsupported-modules.conf
Strictly speaking, modprobe would only consider *.conf files inside
those directories, but for consistency with the already existing
collection of /etc/modprobe.d instead of /etc/modprobe.d/*.conf,
do the same for the added missing locations.
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Recent multipath-tools can depend on this.
a1eabea75e
("multipathd.service: drop ExecStartPre for loading dm-multipath")
92f0893ac1
("multipath-tools: install modules-load.d/multipath.conf")
This can now also include /usr/lib/modules-load.d/s390-pkey.conf
Complements v2.8.0 commit
dffd41943e ("pkey: Support autoloading kernel pkey module").
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
The "detour" with NUL terminated records ensures that sorting
works as expected for environment variables with values containing
one or more lines and thus '\n' as regular record separator.
Reviewed-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Make use of multipath-tools 0.8.9 commit
("libmultipath: add %L path wildcard for 64-bit hex LUN")
260d7cb411 .
Multipath-tools versions that do not know the format wildcard simply
ignore it and expand to an empty string, so it's backwards compatible.
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Eases mapping of multipath paths and FCP by-path information
without having to use different tool output and correlate,
such as "multipathd -k'show topo'" and "lszfcp -D".
Possibly the HBA is reported as "[undef]" for zfcp
[without multipath-tools 0.8.9 commit
("libmultipath: support host adapter name lookup for s390x ccw bus")
852a1dfd94 ]
and the FCP LUN needs to be manually converted from the SCSI LUN in h:c:i:l.
While at it, append correlation information for block device name and
major:minor, as well as all information from a standard 'show paths',
which is the path state 3-tuple and next_check,
plus the number of path failures.
Example output:
$ multipathd -k'show paths format "%w|%a|%r|%p|%i|%d|%D|%t|%T|%o|%0|%C"'
uuid |host adapter|target WWPN |pri|hcil |dev|dev_t|dm_st |chk_st|dev_st |failures|next_check
3600507640081818ab00000000000025e|[undef] |0x500507680b2581fa|10 |0:0:0:606|sdb|8:16 |active|ready |running|0 |XXXXXX.... 13/20
3600507640081818ab00000000000025e|[undef] |0x500507680b2581fb|50 |0:0:1:606|sdc|8:32 |active|ready |running|0 |XX........ 4/20
3600507640081818ab00000000000025e|[undef] |0x500507680b2681fa|10 |1:0:0:606|sdd|8:48 |active|ready |running|0 |XXXX...... 8/20
3600507640081818ab00000000000025e|[undef] |0x500507680b2681fb|50 |1:0:1:606|sde|8:64 |active|ready |running|0 |XXXXXXX... 15/20
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands executed for block and scsi devices
into a section with a comment.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
In addition we did get feedback for promoting some commands to the top.
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands executed for Z device subsystem
into a section with a comment.
By grouping, we try to enhance the handling in our runtime.out file and
help to understand the purpose of command groups.
In addition we did get feedback for promoting some commands to the top.
lspci -t was added on request of SMEs during the discussions
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Acked-by: Nicklas Schnelle <Niklas.Schnelle@ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands executed for overall state into a
first section with a comment.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
In addition we did get feedback for promoting some commands to the top.
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands executed for crypto, special SW or
specific for a distro into sections with a comment.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will group the commands with long output into two
sections with a comment.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
We did get feedback for moveing some independent long output files to
the end for speed up the scrolling in the upper parts.
Files which already have a separate output file are bundled in a
separate group at the end.
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This change will regroup & reorder the commands executed for network.
By grouping, we intend to enhance the handling in our runtime.out file
and help to understand the purpose of command groups.
In addition we did get feedback for promoting some commands to the top.
"ip -br a" was added on request of SMEs during the discussions
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Acked-by: Sandy Winter <WINTERA@de.ibm.com>
Acked-by: Stefan Raspl <stefan.raspl@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
'which' tool is packaged separately from other core tools and there will
likely be scenarios where this check will fail because 'which' is
unavailable.
The "type" is builtin itself and should solve this problem.
For a built in command, we will not suppress error out -> removing 2>&1.
On this replace - in call_run_command - get cmd_type as variable only once
Reported by: Peter Oberparleiter <oberpar@linux.ibm.com>
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
a missing dump2tar command did show misleading error messages
now check it and give clear messages
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
as order itself does not matter for collection of log and config files,
we decided to use alphabetic order for better maintenance
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
critical paths in dump2tar data collection have been already excluded -
this update will sync the excludes in the fall back, running in case
dump2tar is missing or failing on a system
By this we also sync the filtering to match exact the same pattern.
Reported by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Sa Liu <saliu@de.ibm.com>
Fixes: 12c84469fd ("dbginfo.sh: exclude reading page_idle/bitmap sysfs
attribute")
Fixes: b627b8d8e1 ("Initial s390-tools-2.0.0 import")
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
adopt all indents to have the same setting overall - NO logic changes !
tooling: vim used with setting "shiftwidth=8",expandtab,autoindent
commands "gg=G",":retab"
some additional line brakes to stay below 100 chars
and one line added manual before "collect_procfs"
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Extend cpictl to report Linux distribution type and version information,
and another sublevel of Linux kernel version data via the system level
field of the SCLP CPI firmware interface. This data is intended to be
decoded by the IBM Z HMC web-UI to provide users a more detailed view of
Linux software levels installed in LPARs.
For this purpose, the format of the 64-bit system level word is extended
in a compatible way, that is, the meaning of existing fields remains
unchanged while additional data is added to previously unused fields.
The new format is defined as 0xabccddeeeeffgghh, where:
- a=statistics flags
- b=distro id
- c=distro major version
- d=distro minor version(s)
- e=kernel sublevel 2
- f=kernel version
- g=kernel patchlevel
- h=kernel sublevel 1
This format will be automatically reported for supported distributions
based on data from /etc/os-release. As before, users can override the
data by specifying a custom system level word in /etc/sysconfig/cpi.
To support manual specification of new data fields, the format of
cpictl's -L command line option and the CPI_SYSTEM_LEVEL environment
variable are extended as follows:
- 0x<level>
Specifies the system level as 64 bit hex value
- [[[flags:]distro_id:distro_version:]kernel_version]
Specifies the system level as list of optional fields. Any field that
is omitted is initialized with automatically determined values.
Signed-off-by: Peter Oberparleiter <oberpar@linux.ibm.com>
Reviewed-by: Jan Höppner <hoeppner@linux.ibm.com>
Acked-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Under certain conditions, e.g. in a KVM guest, a write operation
to /sys/firmware/cpi/set can result in an error message to
stderr while the script returns with a zero exit code, indicating
success.
This can cause confusion if cpictl is called by systemd due to
the existence of a writable /sys/firmware/cpi/. Let's silence
cpictl when writing to the /sys/firmware/cpi/set. This is OK, as
nothing can be inferred from failure or success from the perspective
of the invoking Linux instance.
Fixes: https://github.com/ibm-s390-linux/s390-tools/issues/120
Signed-off-by: Viktor Mihajlovski <mihajlov@linux.ibm.com>
Reviewed-by: Jan Höppner <hoeppner@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Without the fix, readers of syslog / systemd-journal can erroneously
think that dbginfo.sh was incomplete because the last logged step is
"17 of 18".
The fix also makes the missing line appear in dbginfo.log inside the
generated DBGINFO....tgz.
s390-tools v2.17.0 commit b7807d0195 ("dbginfo.sh: Add KVM commands
and rework data collection") turned create_package into a numbered
step.
Snippet of terminal output:
...
17 of 18: Postprocessing
18 of 18: Finalizing: Creating archive with collected data
Collected data was saved to:
>> /tmp/DBGINFO-....tgz <<
Please review all collected data before sending to your service organization.
Syslog before:
...
dbginfo.sh[...]: 17 of 18: Postprocessing
dbginfo.sh[...]: Data collection completed
Syslog after:
...
dbginfo.sh[...]: 17 of 18: Postprocessing
dbginfo.sh[...]: 18 of 18: Finalizing: Creating archive with collected data
dbginfo.sh[...]: Data collection completed
Fixes: b7807d0195 ("dbginfo.sh: Add KVM commands and rework data collection")
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
When built with `D=1` chreipl-fcp-mpath automatically creates debug data
for the helpers that run during udev event processing. This is the only
way to effectively debug what is happening during event processing in
chreipl-fcp-mpath.
Collect this data under its default path in /run/udev/.
Also collect the chreipl-fcp-mpath statefile which contains WWID, Device
Bus-ID and LUN of the last path that was detected to represent a FCP
re-IPL device.
Suggested-by: Steffen Maier <maier@linux.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Benjamin Block <bblock@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
running mpio data disks with offline paths might cause dbginfo
to hang up on the command "blockdev --report" -> we remove
it to ensure a non blocking data collection.
In addition we add a second kill step with the same timeout
value to retry stopping a command
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Reviewed-by: Mario Held <mario.held@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Add new tool named 'sclpdbf'. This script displays the contents of
s390dbf sclp debug logs in human-readable format.
Signed-off-by: Peter Oberparleiter <oberpar@linux.ibm.com>
Reviewed-by: Jan Höppner <hoeppner@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Had a confusing decoded trace record:
Timestamp : ...
Area : SCSI
...
Tag : rsl_err
...
SCSI result : 0x00000002
SCSI opcode : 1a...
FCP rsp inf cod: 0x00
FCP rsp IU : 00000000 00000000 00000a02 000000ff
00000012 00000000
^^
and was wondering where that sense data would be or if the
corresponding PAY record "fcp_riu" was lost due to trace area wrapping.
So the fix here adds the following two lines to the same trace record:
FCP rsp IU len : 42
FCP rsp IU all : record not available anymore.
Similarly, the following LUN reset Task Management Function response
was missing the last two lines regarding the full FCP response IU
and its optional part with the FCP response info:
Timestamp : ...
Area : SCSI
...
Tag : lr_okay
...
FCP rsp inf cod: 0x00
FCP rsp IU : 00000000 00000000 00000100 00000000
00000000 00000008
^^
FCP rsp IU len : 32
FCP rsp IU all : record not available anymore.
As of today, depending on the kernel version we have
either fcp_sns or fcp_riu PAY trace records. See Linux kernel v4.14 commit
12c3e5754c80 ("scsi: zfcp: fix payload with full FCP_RSP IU in SCSI trace
records").
Always print non-zero payload length independent of payload type.
Actually this also fixes a regression for wrapped away fcp_sns.
If there is no fcp_sns it either wrapped away or it's a kernel with fcp_riu.
Only print fcp_riu if it contains more content than the preceding
unconditional fixed-size field "FCP rsp IU".
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Reviewed-by: Fedor Loshakov <loshakov@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
We found situations, where the debug commands from dbginfo.sh
did overwrite some debug files (e.g. OSA s390dbf) - so we change
the collection order
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
- adding new print function for simpler output customizing
- consolidate some output files
- sync structures and use of checks
- some formatting updates
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
- clenaup outdated kernel - we no longer have 2.x kenrel in support
- add timeout on command excution - ensure to not miss all data for a
single command to hang
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Zipl.conf is collected via /etc/*.conf in CONFIGFILES but BLS snippets
do not appear in /etc/zipl.conf.
Complements v2.5.0 commit d280e5b1fa ("zipl: Add BootLoaderSpec support")
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Add virsh commands for KVM debug data collection and rework the domain
data collection and step numbering. Update the man page accordingly.
[hoeppner@linux.ibm.com: Reword commit message]
Signed-off-by: Joern Siglen <siglen@de.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Since crash 7.2.9 commit
f1f43bf355
in support of kernel v5.9 commit 0990d836cecb
("s390/debug: debug feature version 3"),
the symbolic caller function name is left aligned instead of right
aligned, so now we alternatively have to remove whitespace between
function name and offset.
Otherwise, zfcpdbf output contains wrong warnings about ambiguous payload
records with wrong values for reqid and area, as well as incorrectly
decoded trace records with wrong content values and potential error
messages on not defined record id, and finally a potential list of
occurrences of unknown tags which should empty:
Loading trace records ...(this might take a while)
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:V^D^@fcp_
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:W^D^@fsf_
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:i^B^@san_
...
Timestamp : ...
Area : SAN
Subarea : 00
Level : 1
Exception : -
CPU ID : 0001
Caller : <zfcp_dbf_san_in_els
Record ID : 0
Tag : 6^B^Cfcie
Request ID : >>>wrong<<<
Destination ID : >>>wrong<<<
SAN record id=0 not defined.
Timestamp : ...
Area : SAN
Subarea : 00
Level : 1
Exception : -
CPU ID : 0002
Caller : <zfcp_dbf_san_in_els
Record ID : 0
Tag : 6^B^Cfcie
Request ID : >>>wrong<<<
Destination ID : >>>wrong<<<
SAN record id=0 not defined.
...
occurrences of unknown tags:
...
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Reviewed-by: Benjamin Block <bblock@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Some drivers provide per-queue IRQ coalescing information, collect it.
Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Format and print HBA FC Endpoint Security trace records with a record ID
of ZFCP_DBF_HBA_FCES (5). Align output to the one of FSF response trace
records for readability. Note that the FSF protocol status is most likely
FSF_PROT_GOOD (0x00000001).
Signed-off-by: Jens Remus <jremus@linux.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This can be helpful when diagnosing network performance issues.
Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This can be helpful in diagnosing network-related issues.
Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
The basic statistics are already collected via ifconfig. But that tool
is deprecated, and provides no way of collecting the extended stats.
So also gather data from 'ip link'.
Signed-off-by: Julian Wiedmann <jwi@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This patch adds ethtool commands for network interfaces.
Especially the offload settings are important.
Signed-off-by: Ursula Braun <ubraun@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
