Clean up ESP and ISAKMP decryption.

At least as I read RFC 5996 section 3.14 and RFC 4303 section 2.4, if
the cipher has a block size of which the ciphertext's size must be a
multiple, the payload must be padded to make that happen, so the
ciphertext length must be a multiple of the block size.  Instead of
allocating a buffer, copying the ciphertext to it, and padding it to the
block size, fail if its size isn't a multiple of the block size.

(Note also that the old padding code added a block's worth of padding to
the end of a ciphertext block that *was* a multiple of the cipher block
size; this might have caused problems.)

Don't use the undocumented EVP_Cipher(); the lack of documentation means
a lack of information about whatever requirements it might impose.  Use
EVP_DecryptUpdate() instead.

Before calling it, use EVP_CIPHER_CTX_set_padding() to say "don't do
your own padding, this block is a multiple of the cipher block size".

Instead of using EVP_CipherInit() or EVP_CipherInit_ex(), use
EVP_DecryptInit() or EVP_DecryptInit_ex().  as we're always doing
decryption and never doing encryption - the extra parameter to
EVP_CipherInit() and EVP_CipherInit_ex() is always 0.

This may address GitHub issue #814.

It may also make it a bit easier to have the code use Common Crypto on
macOS (rather than requiring that OpenSSL be installed - macOS ships
with an OpenSSL shared library for binary compatibility with older
releases, but doesn't ship with the headers, because Apple wants you
using their crypto code) and use Cryptography API: Next Generation on
Windows (Vista/Server 2008 and later) (rather than requiring a Windows
build of OpenSSL).

(Hopefully this will all work with LibreSSL.)

1 files changed, 3 insertions, 3 deletions

diff --git a/config.h.in b/config.h.in
index 4b22e15c..365651fa 100644
--- a/config.h.in
+++ b/config.h.in
@@ -33,12 +33,12 @@
 /* Define to 1 if you have the `ether_ntohost' function. */
 #undef HAVE_ETHER_NTOHOST
 
-/* Define to 1 if you have the `EVP_CipherInit_ex' function. */
-#undef HAVE_EVP_CIPHERINIT_EX
-
 /* Define to 1 if you have the `EVP_CIPHER_CTX_new' function. */
 #undef HAVE_EVP_CIPHER_CTX_NEW
 
+/* Define to 1 if you have the `EVP_DecryptInit_ex' function. */
+#undef HAVE_EVP_DECRYPTINIT_EX
+
 /* Define to 1 if you have the <fcntl.h> header file. */
 #undef HAVE_FCNTL_H