Now that SecureD v1 is in the books I thought it would be worthwhile to explore what a second version could like. I specifically want to focus on expanding compatibility with other systems.

For example: AWS uses SHA2-256 for signing requests. As implemented today SecureD does not support this. However, this is one of the scenarios that falls under SecureD's mission of ease-to-use cryptography for non-transport layer security.

I am curious what people would like to see in SecureD moving forward and I would love to get your feedback on what works and what doesn't work in v1 of SecureD.

With this in mind here are my ideas for expanding SecureD's capabilities. I'll start with the idea that I think will be the least controversial.

1. Full support for SHA-2 and SHA-3 for Hashes and HMAC's.

This means the following modes would be supported:
SHA2: 224, 256, 384, 512, 512/224, 512/256
SHA3: 224, 256, 384, 512

The PBKDF2 implementation would also be updated to support the above hash functions.

Hash methods supporting salts will be added.

To maintain backwards compatibility with v1, methods that match the v1 signatures and forward to the correct implementation would be provided as the v1 defaults are still secure.

This requires OpenSSL 1.1.1 at a minimum. Note that SHA3 support could be delayed until SecureD 2.1 if OpenSSL 1.1.1 is not available on a majority of systems prior to shipping 2.0. In such case OpenSSL 1.1.0 would become the minimum supported version.

2. More Flexibility for Symmetric Encryption.

Currently SecureD only supports AES256-CTR-HMAC384 and the encrypted data is laid out as: HASH+IV+DATA. This is only useful in scenarios where the developer controls both the encryption and decryption of the data. Additionally, this does not support AEAD modes, only AE modes.

The first thing I would like to do is to support all AES, SHA2, and SHA3 modes. I also want to add support for the CBC cipher mode with PCKS#7 padding as that is the most common cipher mode in use.

One of the primary use cases for SecureD is long-term storage of data. To facilitate this requirement I would like to add a binary encoded header to each encrypted blob. This header would contain the necessary information to decode the blob (minus the key of course).

The header would cost 26 bytes per encrypted blob and would have the following layout:
struct cryptoHeader {
    ubyte hdrVersion;	// The version of the header
    ubyte encAlg;	// The encryption algorithm used
    ubyte hashAlg;	// The hash algorithm used
    uint  kdfIters;	// The number of PBKDF2 iterations
    ubyte authLen;	// The length of the authentication value
    ubyte saltLen;	// The length of the PBKDF2 salt
    ubyte ivLen;	// The length of the IV
    ulong encLen;	// The length of the encrypted data
    ulong adLen;	// The length of the additional data
}

The data layout would be: HEADER+AUTH+SALT+IV+ENC+AD
Any excluded elements would be marked as 0 in the header and completely excluded from the data layout.
The MAC would be computed as: HMAC(HEADER+SALT+IV+ENC+AD)

The full encryption process would be as follows:
- Create a PBKDF2 salt via RNG
- Create IV via RNG
- Create header from user inputs
- EncKey = PBKDF2 over RawKey unless hashAlg is None or kdfIters == 0
  - Use RawKey as-is if hashAlg is None or kdfIters == 0
- MacKey = PBKDF2 over EncKey once using same inputs as EncKey
- Encrypt data
- Compute HMAC(HEADER+SALT+IV+ENC+AD)
- Return RESULT(HEADER+AUTH+SALT+IV+ENC+AD)

Methods would be added to work with the following:
- Simple encrypted blobs (no header/auth/salt/iv/ad included)
- Creating and Verifying AE/AD MACs without headers
- Methods to read SecureD v1 encrypted data

Note that this plan is NOT compatible with the SecureD v1 encryption data layout and data would need to be re-encrypted.

3. Re-implement the OpenSSL Bindings

The Deimos/DUB bindings are ancient and do not support OpenSSL 1.1.x. We can re-implement the bindings taking care to only include the symbols we need.

4. Streams

I get asked about this, and, as much as I want to do this, it is not a simple topic. std.streams was removed and right now the only viable replacement is Vibe.D streams. But that means pulling in Vibe.D for a simple cryptography library. At this point that doesn't seem like the right idea. If someone is willing to step-up and do the work I'd be willing to look at it, but for now I want to wait on this, preferably for a standard/generic streams interface to be made available.

Please let know what you think! I am very interested to hear about what would make your life easier when working with SecureD an cryptography in general.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:
> Now that SecureD v1 is in the books I thought it would be worthwhile to explore what a second version could like. I specifically want to focus on expanding compatibility with other systems.
>
> [...]


No, it’s not. Look at IOpipe and no further it provides the exact abstraction that works for any form buffered I/O doing stream processing at top speeds. I’m doing IOpipe regex, it looks to be just what the doctor ordered.

>  But that means pulling in Vibe.D for a simple cryptography library. At this point that doesn't seem like the right idea. If someone is willing to step-up and do the work I'd be willing to look at it, but for now I want to wait on this, preferably for a standard/generic streams interface to be made available.
>
> [...]

On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:
> Now that SecureD v1 is in the books

This would have been a great place to insert a brief description of what SecureD is or a link to the project.

On 05/27/2018 09:54 AM, Neia Neutuladh wrote:
> On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:
>> Now that SecureD v1 is in the books
> 
> This would have been a great place to insert a brief description of what SecureD is or a link to the project.

Good point. SecureD is a cryptography library that is designed to make non transport-layer cryptography simple to use. I created it after working with OpenSSL and observing that the API design of OpenSSL actively encourages broken implementations. Think of it as a cryptography library for the rest of us.

You can read more about it here: http://code.dlang.org/packages/secured

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:
> struct cryptoHeader {
>     ubyte hdrVersion;	// The version of the header
>     ubyte encAlg;	// The encryption algorithm used
>     ubyte hashAlg;	// The hash algorithm used
>     uint  kdfIters;	// The number of PBKDF2 iterations
>     ubyte authLen;	// The length of the authentication value
>     ubyte saltLen;	// The length of the PBKDF2 salt
>     ubyte ivLen;	// The length of the IV
>     ulong encLen;	// The length of the encrypted data
>     ulong adLen;	// The length of the additional data
> }

Should there be any kind of key identifier, or do you consider that a separate issue?

> - MacKey = PBKDF2 over EncKey once using same inputs as EncKey

How about this?

Use PBKDF2 to generate a key-derivation-key (KDK), then use HKDF-Expand (https://tools.ietf.org/html/rfc5869) to derive the encryption key and MAC key separately.

I think that's more standard.  I don't know how much it matters in practice, but a lot of cryptographers don't like generating MAC/encryption keys from each other directly.

Also, it's probably safer to use HKDF-Extract instead of using raw keys directly when not using PBKDF2.

On 05/27/2018 05:11 PM, sarn wrote:
> On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:
>> struct cryptoHeader {
>>     ubyte hdrVersion;    // The version of the header
>>     ubyte encAlg;    // The encryption algorithm used
>>     ubyte hashAlg;    // The hash algorithm used
>>     uint  kdfIters;    // The number of PBKDF2 iterations
>>     ubyte authLen;    // The length of the authentication value
>>     ubyte saltLen;    // The length of the PBKDF2 salt
>>     ubyte ivLen;    // The length of the IV
>>     ulong encLen;    // The length of the encrypted data
>>     ulong adLen;    // The length of the additional data
>> }
> 
> Should there be any kind of key identifier, or do you consider that a separate issue?
> 

hashAlg is used for everything related to key derivation. Salts and IVs are random.

>> - MacKey = PBKDF2 over EncKey once using same inputs as EncKey
> 
> How about this?
> 
> Use PBKDF2 to generate a key-derivation-key (KDK), then use HKDF-Expand (https://tools.ietf.org/html/rfc5869) to derive the encryption key and MAC key separately.
> 
> I think that's more standard.  I don't know how much it matters in practice, but a lot of cryptographers don't like generating MAC/encryption keys from each other directly.
> 

I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key.

Something like this:
struct cryptoHeader {
    ubyte hdrVersion;    // The version of the header
    ubyte encAlg;        // The encryption algorithm used
    ubyte hashAlg;       // The hash algorithm used
    uint kdfIters;       // The number of PBKDF2 iterations

    ubyte kdkSLen;       // The length of the KDK Salt
    ubyte macSLen;       // The length of the MAC Salt
    ubyte keySLen;       // The length of the KEY Salt
    ubyte ivLen;         // The length of the IV
    ulong encLen;        // The length of the encrypted data
    ulong adLen;         // The length of the additional data
    ubyte authLen;       // The length of the authentication value
}

> Also, it's probably safer to use HKDF-Extract instead of using raw keys directly when not using PBKDF2.
> 

It is. And we could dot that if you stick to the default encrypt/decrypt routines. I want to expand the symmetric capabilities of SecureD so I am going to rebuild the encrypt/decrypt routines as a composition of smaller methods. That will allow users to either use the default encryption (simple) or if they have to interoperate with other systems, use the component methods to perform their operations. I am not looking to cover all scenarios though, just the common ones. If you have something truly unusual, you'll still need to drop down the base crypto libraries.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote:
> I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key.

HKDF-Expand doesn't need a salt.  You just need one salt to make the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for deriving the encryption and MAC key.

On 05/27/2018 08:52 PM, sarn wrote:
> On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote:
>> I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key.
> 
> HKDF-Expand doesn't need a salt.  You just need one salt to make the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for deriving the encryption and MAC key.

Strictly speaking, it's is Optional but Strongly Recommended per RFC5869-3.1

The use case here is that this data is going into storage and that storage is cheap. We don't have to be strict on our byte budget. :)

https://tools.ietf.org/html/rfc5869

https://www.openssl.org/docs/man1.1.0/crypto/EVP_PKEY_CTX_set_hkdf_md.html

SecureD is supposed to be "Crypto done right." So I might as well do it right and follow the RFC.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On Monday, 28 May 2018 at 06:22:02 UTC, Adam Wilson wrote:
> On 05/27/2018 08:52 PM, sarn wrote:
>> On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote:
>>> I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key.
>> 
>> HKDF-Expand doesn't need a salt.  You just need one salt to make the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for deriving the encryption and MAC key.
>
> Strictly speaking, it's is Optional but Strongly Recommended per RFC5869-3.1

There's HKDF-Expand and there's HKDF-Extract.  HKDF-Extract takes an optional salt and HKDF-Expand doesn't use a salt at all (take a closer look at that RFC).  That OpenSSL routine is the combination of the two, so that's why it takes the salt.

On 05/28/2018 12:14 AM, sarn wrote:
> On Monday, 28 May 2018 at 06:22:02 UTC, Adam Wilson wrote:
>> On 05/27/2018 08:52 PM, sarn wrote:
>>> On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote:
>>>> I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key.
>>>
>>> HKDF-Expand doesn't need a salt.  You just need one salt to make the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for deriving the encryption and MAC key.
>>
>> Strictly speaking, it's is Optional but Strongly Recommended per RFC5869-3.1
> 
> There's HKDF-Expand and there's HKDF-Extract.  HKDF-Extract takes an optional salt and HKDF-Expand doesn't use a salt at all (take a closer look at that RFC).  That OpenSSL routine is the combination of the two, so that's why it takes the salt.

I understand that. But the way I read the first paragraph of 3.1 is that yes, it can work without a Salt, but HKDF, in general, is strongly suggested to be used with a salt. If we are really concerned about bytes we could reuse the salt for all three steps (KDK/MAC/KEY), but TBH, I'm not worried about disk usage.

Let me ask the question a different way. What is the reason NOT to use 2 different salts for the MAC and KEY generation steps?

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;