libseckey: RSA signature: Support OSSL_PKEY_RSA_PSS_SALT_LEN_AUTO_DIGEST_MAX

Since OpenSSL 3.1 a new RSA-PSS salt length constant exists to select
the maximum possible salt length based on the RSA-PSS parameters and the
digest used: OSSL_PKEY_RSA_PSS_SALT_LEN_AUTO_DIGEST_MAX
This is the default salt length when no other salt length is set by
the caller.

In contrast to OSSL_PKEY_RSA_PSS_SALT_LEN_MAX, OSSL_PKEY_RSA_PSS_SALT_LEN_AUTO_DIGEST_MAX
also ensures that the resulting salt length is not larger than the used
digest size. The salt length calculated with OSSL_PKEY_RSA_PSS_SALT_LEN_MAX
may be larger than the digest size, dependent on the RSA-PSS parameters.

FIPS 186-4 section 5 "The RSA Digital Signature Algorithm", subsection
5.5 "PKCS #1" says: "For RSASSA-PSS […] the length (in bytes) of the
salt (sLen) shall satisfy 0 <= sLen <= hLen, where hLen is the length of
the hash function output block (in bytes)."

See OpenSSL commit 6c73ca4a2f

Signed-off-by: Ingo Franzki <ifranzki@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
This commit is contained in:
Ingo Franzki
2023-01-09 11:49:41 +01:00
committed by Steffen Eiden
parent 4f5d33988d
commit 1fb05038cd

View File

@@ -1738,6 +1738,11 @@ static int sk_prov_sign_op_get_pss_saltlen(struct sk_prov_op_ctx *ctx,
salt_len = max_saltlen;
else if (strcmp(saltlen, OSSL_PKEY_RSA_PSS_SALT_LEN_AUTO) == 0)
salt_len = max_saltlen;
#ifdef OSSL_PKEY_RSA_PSS_SALT_LEN_AUTO_DIGEST_MAX
else if (strcmp(saltlen,
OSSL_PKEY_RSA_PSS_SALT_LEN_AUTO_DIGEST_MAX) == 0)
salt_len = MIN(max_saltlen, EVP_MD_size(mgf_md));
#endif
else
salt_len = atoi(saltlen);