TSS.CPP RsaEncrypt ignores hash algorithm

[Alainx277]

The RsaEncrypt function in TSS.CPP ignores the passed hash algorithm:

TSS.MSR/TSS.CPP/Src/Crypto.cpp

Lines 334 to 343 in 52cb9f4

	size_t RsaEncrypt(const RSA_KEY *key, // IN: key to use
	TPM_ALG_ID /*scheme*/, // IN: the scheme to use
	TPM_ALG_ID /*hashAlg*/, // IN: hash algorithm
	UINT32 secretSize, // IN: size of digest to be checked
	const BYTE *secret, // IN: digest buffer
	UINT32 paddingSize, // IN: size of signature
	const BYTE *padding, // IN: signature
	UINT32 *outBufferSize, // IN: salt size for PSS
	BYTE *outBuffer
	)

This is obviously incorrect. In my project it caused the TPM ActivateCredential command to fail as encryption used the only implemented hash function SHA1 instead of the required SHA256 hash.

The following implementation fixed the problem for my case:

size_t RsaEncrypt(const RSA_KEY *key,         // IN: key to use
                  TPM_ALG_ID /*scheme*/,      // IN: the scheme to use
                  TPM_ALG_ID   hashAlg,     // IN: hash algorithm
                  UINT32       secretSize,    // IN: size of digest to be checked
                  const BYTE   *secret,       // IN: digest buffer
                  UINT32       paddingSize,   // IN: size of signature
                  const BYTE  *padding,       // IN: signature
                  UINT32      *outBufferSize, // IN: salt size for PSS
                  BYTE        *outBuffer
)
{

    BYTE encBuffer[4096];
    RSA *keyX;

    BIGNUM *bn_mod = NULL;
    BIGNUM *bn_exp = NULL;
    BYTE exponent[] {1, 0, 1};

    bn_mod = BN_bin2bn(key->publicKey->buffer, key->publicKey->size, NULL);
    bn_exp = BN_bin2bn(exponent, 3, NULL);

    keyX = RSA_new();
    keyX->n = bn_mod;
    keyX->e = bn_exp;
    keyX->d = NULL;
    keyX->p = NULL;
    keyX->q = NULL;

    int wasNumBytes = (int) * outBufferSize;
    int numBytes = 0;

    const EVP_MD *md = nullptr;
    switch (hashAlg) {
    case TPM_ALG_ID::SHA1:   md = EVP_sha1();   break;
    case TPM_ALG_ID::SHA256: md = EVP_sha256(); break;
    case TPM_ALG_ID::SHA384: md = EVP_sha384(); break;
    case TPM_ALG_ID::SHA512: md = EVP_sha512(); break;
    default:
        throw std::invalid_argument("Unsupported OAEP hash");
    }

    if (paddingSize == 0)
        numBytes = RSA_public_encrypt(secretSize, secret, outBuffer, keyX, RSA_PKCS1_OAEP_PADDING);
    else {
        int encLen = key->publicKey->size;
        RSA_padding_add_PKCS1_OAEP_mgf1(encBuffer, encLen,
                                    secret, secretSize,
                                    paddingSize ? padding : nullptr,
                                    paddingSize,
                                    md,
                                    md);
        numBytes = RSA_public_encrypt(encLen, encBuffer, outBuffer, keyX, RSA_NO_PADDING);
    }

    // Note, we will already've written the buffer if this assert fails, but perhaps it will help.
    _ASSERT(wasNumBytes >= numBytes);

    *outBufferSize = numBytes;
    RSA_free(keyX);
    return numBytes;
}

Note the use of RSA_padding_add_PKCS1_OAEP_mgf1 with the correct hash argument.