New trace tags "sysuas*" added in kernel to trace zfcp sysfs unit add
store events.
Print the trace events under HBA with new record ID 6. The fields under
this tag are WWPN, FCP LUN and return value of the device having issues.
Example zfcpdbf output for unit add store events:
Timestamp : 2026-02-02-10:50:02:474983
Area : HBA
Subarea : 00
Level : 4
Exception : -
CPU ID : 0003
Caller : 0x000001d89a86bd0e
Record ID : 6
Tag : sysuas2
Description : sysuas2 HBA, Sysfs unit add store failure, unit add failed
Request ID : 0x00000000ffffffff
Request status : 0xffffffff
FSF cmnd : 0xffffffff
FSF sequence no: 0xffffffff
WWPN : 0x500507680b26c449
LUN : 0x01d0000000000000
Return Value : 0xfffffff4
Timestamp : 2026-02-02-10:50:02:475037
Area : HBA
Subarea : 00
Level : 4
Exception : -
CPU ID : 0003
Caller : 0x000001d89a86bd0e
Record ID : 6
Tag : sysuas2
Description : sysuas2 HBA, Sysfs unit add store failure, unit add failed
Request ID : 0x00000000ffffffff
Request status : 0xffffffff
FSF cmnd : 0xffffffff
FSF sequence no: 0xffffffff
WWPN : 0x500507680b26c449
LUN : 0x01d1000000000000
Return Value : 0xfffffff4
Signed-off-by: Chinmaya Kajagar <chinmayk@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Had a confusing decoded trace record:
Timestamp : ...
Area : SCSI
...
Tag : rsl_err
...
SCSI result : 0x00000002
SCSI opcode : 1a...
FCP rsp inf cod: 0x00
FCP rsp IU : 00000000 00000000 00000a02 000000ff
00000012 00000000
^^
and was wondering where that sense data would be or if the
corresponding PAY record "fcp_riu" was lost due to trace area wrapping.
So the fix here adds the following two lines to the same trace record:
FCP rsp IU len : 42
FCP rsp IU all : record not available anymore.
Similarly, the following LUN reset Task Management Function response
was missing the last two lines regarding the full FCP response IU
and its optional part with the FCP response info:
Timestamp : ...
Area : SCSI
...
Tag : lr_okay
...
FCP rsp inf cod: 0x00
FCP rsp IU : 00000000 00000000 00000100 00000000
00000000 00000008
^^
FCP rsp IU len : 32
FCP rsp IU all : record not available anymore.
As of today, depending on the kernel version we have
either fcp_sns or fcp_riu PAY trace records. See Linux kernel v4.14 commit
12c3e5754c80 ("scsi: zfcp: fix payload with full FCP_RSP IU in SCSI trace
records").
Always print non-zero payload length independent of payload type.
Actually this also fixes a regression for wrapped away fcp_sns.
If there is no fcp_sns it either wrapped away or it's a kernel with fcp_riu.
Only print fcp_riu if it contains more content than the preceding
unconditional fixed-size field "FCP rsp IU".
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Reviewed-by: Fedor Loshakov <loshakov@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Since crash 7.2.9 commit
f1f43bf355
in support of kernel v5.9 commit 0990d836cecb
("s390/debug: debug feature version 3"),
the symbolic caller function name is left aligned instead of right
aligned, so now we alternatively have to remove whitespace between
function name and offset.
Otherwise, zfcpdbf output contains wrong warnings about ambiguous payload
records with wrong values for reqid and area, as well as incorrectly
decoded trace records with wrong content values and potential error
messages on not defined record id, and finally a potential list of
occurrences of unknown tags which should empty:
Loading trace records ...(this might take a while)
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:V^D^@fcp_
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:W^D^@fsf_
Warning: Ambiguous PAYload records. reqid:>>>wrong<<< area:i^B^@san_
...
Timestamp : ...
Area : SAN
Subarea : 00
Level : 1
Exception : -
CPU ID : 0001
Caller : <zfcp_dbf_san_in_els
Record ID : 0
Tag : 6^B^Cfcie
Request ID : >>>wrong<<<
Destination ID : >>>wrong<<<
SAN record id=0 not defined.
Timestamp : ...
Area : SAN
Subarea : 00
Level : 1
Exception : -
CPU ID : 0002
Caller : <zfcp_dbf_san_in_els
Record ID : 0
Tag : 6^B^Cfcie
Request ID : >>>wrong<<<
Destination ID : >>>wrong<<<
SAN record id=0 not defined.
...
occurrences of unknown tags:
...
Signed-off-by: Steffen Maier <maier@linux.ibm.com>
Reviewed-by: Benjamin Block <bblock@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Format and print HBA FC Endpoint Security trace records with a record ID
of ZFCP_DBF_HBA_FCES (5). Align output to the one of FSF response trace
records for readability. Note that the FSF protocol status is most likely
FSF_PROT_GOOD (0x00000001).
Signed-off-by: Jens Remus <jremus@linux.ibm.com>
Reviewed-by: Steffen Maier <maier@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
Function get_payload_records() reading from disk, blindly adds payload
record (junks) to a list in a 2-stage hierarchical hash keyed with
request ID and PAY area.
On adapter offline/online, zfcp resets the request ID to re-start with 1.
Hence it's not a unique foreign key in the lifetime of the PAY area.
The last payload record (junks) read for the same request ID and PAY area
is the one printed as named PAY field in the tool output.
This is not necessarily the chronologically last record due to PAY area
wrapping.
With this workaround, we:
* warn on reading PAY records if ambiguous entries occurred,
* print additional PAY timestamps which the user can compare
with the actual trace record timestamps to determine if they are close
enough to indicate a match.
We only found this by accident because we kept "short" "payload" being a
built-in part of some actual trace record types in areas HBA, SAN, SCSI.
For above cases, the full PAYload did not match the
built-in "short" "payload".
While at it, also add a comment that PAY output can also be wrong
in another case when junks of multi-part PAY records are missing due
to PAY area wrapping.
Below is an abbreviated example output with this workaround:
Loading trace records ...(this might take a while)
Warning: Ambiguous PAYload records. reqid:0000000000000037 area:san_res
...
Timestamp : 2017-07-07-18:01:34:574827
Area : SAN
...
Tag : fssct_1
Request ID : 0x0000000000000037
Destination ID : 0x00fffffc
SAN req short : 01000000 fc020000 01720ffc 00000000
00000008
SAN req length : 20
Timestamp : 2017-07-07-18:01:34:619310 ------------------+
Area : SAN |
... |
Tag : fsscth2 |
Request ID : 0x0000000000000037 |
Destination ID : 0x00fffffc |
SAN resp short : 01000000 fc020000 80020000 00000000 |
00671a07 00000000 c05076ff d6801850 |
00671a7e 00000000 c05076ff d6801968 |
SAN resp length: 16384 |
Payload time : 2017-07-07-18:16:17:165708 <== does not match
San resp info : 01000000 fc020000 80020000 00000000 <== wrong
4a49424d 20202020 20323936 34202020 <== wrong
20202020 20202020 20303230 30303030 <== wrong
30303745 43383720 20504348 49443a20 <== wrong
30313938 204e5049 5620556c 7049643a <== wrong
20303536 30303330 30202020 00000000 <== wrong
00000000 00000000 00000000 00000000 <== wrong
... <== wrong
...
Timestamp : 2017-07-07-18:16:17:165713
Area : SAN
...
Tag : fsscth2
Request ID : 0x0000000000000037 <== same reqid & pay "area"
Destination ID : 0x00fffffc
SAN resp short : 01000000 fc020000 80020000 00000000
4a49424d 20202020 20323936 34202020
20202020 20202020 20303230 30303030
SAN resp length: 273
Payload time : 2017-07-07-18:16:17:165708 <== 2nd & last, match
close to Timestamp
San resp info : 01000000 fc020000 80020000 00000000
4a49424d 20202020 20323936 34202020
20202020 20202020 20303230 30303030
30303745 43383720 20504348 49443a20
30313938 204e5049 5620556c 7049643a
20303536 30303330 30202020 00000000
00000000 00000000 00000000 00000000
...
Signed-off-by: Steffen Maier <maier@linux.vnet.ibm.com>
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Jan Höppner <hoeppner@linux.ibm.com>
This commit is based on the s390-tools-1.39.0 version.
Changes on top of s390-tools-1.39.0:
- Add MIT license to all source files
- Add LICENSE file
- Transform REAMDE to README.md (markdown)
- Add AUTHORS.md file
- Add CONTRIBUTING.md file
- Move changelog from README to CHANGELOG.md file
Reviewed-by: Stefan Haberland <sth@linux.vnet.ibm.com>
Signed-off-by: Michael Holzheu <holzheu@linux.vnet.ibm.com>
