On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote:
> Hello
>
> I'm trying to contact AWS API with D according to documentation:
>
> [...]

check https://github.com/yannick/vibe-aws
it has v4 implemented

On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote:
> Last but not least, how to write such function in D:
>
> def sign(key, msg):
>     return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()
>
> ?
>
> I can't find in standard libraryt hmac function, is it existing?

The next version of D will have a std.digest.hmac module:

http://dlang.org/phobos-prerelease/std_digest_hmac.html

On 04/10/15 2:31 AM, holo wrote:
> On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole wrote:
>> On 04/10/15 1:49 AM, holo wrote:
>>> On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:
>>>> On 04/10/15 1:09 AM, holo wrote:
>>>>> [...]
>>>>
>>>> By the looks of things the problem is with SHA256, I'm guessing it
>>>> doesn't have its block size defined.
>>>> https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394
>>>>
>>>>
>>>> You'll also need to update sha.d as well. Otherwise it looks good.
>>>
>>> I updated sha.d and now im getting following error:
>>>
>>> $ ./app.d
>>> /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function
>>> 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure
>>> function 'core.cpuid.ssse3'
>>> /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template
>>> instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
>>> const(ubyte)[]) error instantiating
>>> /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here:
>>> isDigest!(SHA!(512u, 160u))
>>> /usr/include/dlang/dmd/std/digest/sha.d(1179):        while looking for
>>> match for WrapperDigest!(SHA!(512u, 160u))
>>> Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]
>>
>> And one more file (ssse3.d) needs to be updated.
>> You are going to be playing whack a mole I think for a while.
>
> I downloaded whole master branch phobos and tried to use it (changed
> path in dmd.conf), but there are missing much more files (i think that
> master branch is not ready yet). So i get back to my original library
> and tried to updated file you mention. But i can't find it (ssse3.d),
> its not appearing in my stable lib and even in that master-branch from
> zip file.

Apologies, I didn't see this till after I got to bed. It's core.cpuid not ssse3.d. It will be in druntime, not Phobos.

On Saturday, 3 October 2015 at 23:58:39 UTC, Rikki Cattermole wrote:
> On 04/10/15 2:31 AM, holo wrote:
>> On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole wrote:
>>> On 04/10/15 1:49 AM, holo wrote:
>>>> On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:
>>>>> On 04/10/15 1:09 AM, holo wrote:
>>>>>> [...]
>>>>>
>>>>> By the looks of things the problem is with SHA256, I'm guessing it
>>>>> doesn't have its block size defined.
>>>>> https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394
>>>>>
>>>>>
>>>>> You'll also need to update sha.d as well. Otherwise it looks good.
>>>>
>>>> I updated sha.d and now im getting following error:
>>>>
>>>> $ ./app.d
>>>> /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function
>>>> 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure
>>>> function 'core.cpuid.ssse3'
>>>> /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template
>>>> instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
>>>> const(ubyte)[]) error instantiating
>>>> /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here:
>>>> isDigest!(SHA!(512u, 160u))
>>>> /usr/include/dlang/dmd/std/digest/sha.d(1179):        while looking for
>>>> match for WrapperDigest!(SHA!(512u, 160u))
>>>> Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]
>>>
>>> And one more file (ssse3.d) needs to be updated.
>>> You are going to be playing whack a mole I think for a while.
>>
>> I downloaded whole master branch phobos and tried to use it (changed
>> path in dmd.conf), but there are missing much more files (i think that
>> master branch is not ready yet). So i get back to my original library
>> and tried to updated file you mention. But i can't find it (ssse3.d),
>> its not appearing in my stable lib and even in that master-branch from
>> zip file.
>
> Apologies, I didn't see this till after I got to bed. It's core.cpuid not ssse3.d. It will be in druntime, not Phobos.

Sorry i was busy at weekend. Thank you all for answers. I will try to write some solution basing on yawniek example and will back if some troubles appears.

@Vladimir where can i check or do you know when next version of phobos will be available?

@Rikki i downloaded both master zip files for runtime and phobos librarys all dependencies looks are satisfied right now but still getting errors:

[holo@ultraxps workplace]$ dmd hmac.d
hmac.o: In function `_D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':
hmac.d:(.text._D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x18): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6__initZ'
hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x1b): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x31): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x3e): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h'
hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x8d): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0xa4): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x5f): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h':
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xa1): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xaa): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xc4): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xdb): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xe8): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h'
hmac.o: In function `_D3std6format61__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderThTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv':
hmac.d:(.text._D3std6format61__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderThTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv'
hmac.o: In function `_D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxhTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv':
hmac.d:(.text._D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxhTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv'
hmac.o: In function `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTlTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxlKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv':
hmac.d:(.text._D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTlTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxlKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv+0x46): undefined reference to `_D3std6format64__T14formatUnsignedTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatUnsignedFNaNbNfS3std5array17__T8AppenderTAyaZ8AppendermKxS3std6format18__T10FormatSpecTaZ10FormatSpeckbZv'
hmac.o: In function `_D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxmTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxmKS3std6format18__T10FormatSpecTaZ10FormatSpecZv':
hmac.d:(.text._D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxmTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxmKS3std6format18__T10FormatSpecTaZ10FormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv'
collect2: error: ld returned 1 exit status
--- errorlevel 1
[holo@ultraxps workplace]$ cat hmac.d
#!/usr/bin/rdmd

import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
	auto hmac = hmac!SHA1("secretkey".representation);
	hmac.put("texttohash".representation);
	auto digest = hmac.finish();

	writeln(digest);
}

[holo@ultraxps workplace]$

On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
> @Vladimir where can i check or do you know when next version of phobos will be available?

You can use Digger to get the latest version of D:

https://github.com/CyberShadow/Digger

On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev wrote:
> On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
>> @Vladimir where can i check or do you know when next version of phobos will be available?
>
> You can use Digger to get the latest version of D:
>
> https://github.com/CyberShadow/Digger

Thank you, now it is working as expected. Hope i wont miss upgrade in my system to uninstall digger version before.

[holo@ultraxps workplace]$ cat ./hmac.d
#!/usr/bin/rdmd

import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
	auto hmac = hmac!SHA256("secretkey".representation);
	hmac.put("texttohash".representation);
	auto digest = hmac.finish();

	writeln(digest);
}

[holo@ultraxps workplace]$ ./hmac.d
[28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 148, 84, 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 252, 26, 124, 215, 6]
[holo@ultraxps workplace]$ dmd --version
DMD64 D Compiler v2.069-devel-5e73c30
Copyright (c) 1999-2015 by Digital Mars written by Walter Bright
[holo@ultraxps workplace]$

Now need try to write some code which will connect me to AWS api based on examples.

On 06/10/15 11:26 AM, holo wrote:
> On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev wrote:
>> On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
>>> @Vladimir where can i check or do you know when next version of
>>> phobos will be available?
>>
>> You can use Digger to get the latest version of D:
>>
>> https://github.com/CyberShadow/Digger
>
> Thank you, now it is working as expected. Hope i wont miss upgrade in my
> system to uninstall digger version before.
>
> [holo@ultraxps workplace]$ cat ./hmac.d
> #!/usr/bin/rdmd
>
> import std.stdio;
> import std.digest.sha, std.digest.hmac;
> import std.string : representation;
>
> void main()
> {
>      auto hmac = hmac!SHA256("secretkey".representation);
>      hmac.put("texttohash".representation);
>      auto digest = hmac.finish();
>
>      writeln(digest);
> }
>
> [holo@ultraxps workplace]$ ./hmac.d
> [28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 148, 84,
> 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 252, 26, 124,
> 215, 6]
> [holo@ultraxps workplace]$ dmd --version
> DMD64 D Compiler v2.069-devel-5e73c30
> Copyright (c) 1999-2015 by Digital Mars written by Walter Bright
> [holo@ultraxps workplace]$
>
> Now need try to write some code which will connect me to AWS api based
> on examples.

Congrats on getting it working!

> Congrats on getting it working!

@Rikki Thanks :)

I was trying to write my own lib from beginning based on examples but after some time i resign from that idea (will back to it when i will have some more experience) and right now im trying to customize that one from link which yawniek paste:

https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d

I removed import from vibe.d library and copy/paste missed functions formEncode and filterURLEncode (BTW: what that "(R)" mean in it? filterURLEncode(R)(ref R dst, ..., ..) ).

Next thing what i did was to replace hmac function to use hmac module from newest library. Now whole code looks like this:

module sigv4;

import std.array;
import std.algorithm;
import std.digest.sha;
import std.range;
import std.stdio;
import std.string;
import std.format;
import std.digest.hmac;


const algorithm = "AWS4-HMAC-SHA256";


void filterURLEncode(R)(ref R dst, string str, string allowed_chars = null, bool form_encoding = false)
{
	while( str.length > 0 ) {
		switch(str[0]) {
			case ' ':
				if (form_encoding) {
					dst.put('+');
					break;
				}
				goto default;
			case 'A': .. case 'Z':
			case 'a': .. case 'z':
			case '0': .. case '9':
			case '-': case '_': case '.': case '~':
				dst.put(str[0]);
				break;
			default:
				if (allowed_chars.canFind(str[0])) dst.put(str[0]);
				else formattedWrite(dst, "%%%02X", str[0]);
		}
		str = str[1 .. $];
	}
}

string formEncode(string str, string allowed_chars = null)
@safe {
	auto dst = appender!string();
	dst.reserve(str.length);
	filterURLEncode(dst, str, allowed_chars, true);
	return dst.data;
}

struct CanonicalRequest
{
    string method;
    string uri;
    string[string] queryParameters;
    string[string] headers;
    ubyte[] payload;
}

string canonicalQueryString(string[string] queryParameters)
{
    alias encode = formEncode;

    string[string] encoded;
    foreach (p; queryParameters.keys())
    {
        encoded[encode(p)] = encode(queryParameters[p]);
    }
    string[] keys = encoded.keys();
    sort(keys);
    return keys.map!(k => k ~ "=" ~ encoded[k]).join("&");
}

string canonicalHeaders(string[string] headers)
{
    string[string] trimmed;
    foreach (h; headers.keys())
    {
        trimmed[h.toLower().strip()] = headers[h].strip();
    }
    string[] keys = trimmed.keys();
    sort(keys);
    return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join("");
}

string signedHeaders(string[string] headers)
{
    string[] keys = headers.keys().map!(k => k.toLower()).array();
    sort(keys);
    return keys.join(";");
}

string hash(T)(T payload)
{
    auto hash = sha256Of(payload);
    return hash.toHexString().toLower();
}

string makeCRSigV4(CanonicalRequest r)
{
    auto cr =
        r.method.toUpper() ~ "\n" ~
        (r.uri.empty ? "/" : r.uri) ~ "\n" ~
        canonicalQueryString(r.queryParameters) ~ "\n" ~
        canonicalHeaders(r.headers) ~ "\n" ~
        signedHeaders(r.headers) ~ "\n" ~
        hash(r.payload);

    return hash(cr);
}

unittest {
    string[string] empty;

    auto r = CanonicalRequest(
            "POST",
            "/",
            empty,
            ["content-type": "application/x-www-form-urlencoded; charset=utf-8",
             "host": "iam.amazonaws.com",
             "x-amz-date": "20110909T233600Z"],
            cast(ubyte[])"Action=ListUsers&Version=2010-05-08");

    auto sig = makeCRSigV4(r);

    assert(sig == "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2");
}

struct SignableRequest
{
    string dateString;
    string timeStringUTC;
    string region;
    string service;
    CanonicalRequest canonicalRequest;
}

string signableString(SignableRequest r) {
    return algorithm ~ "\n" ~
        r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~
        r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~ "/aws4_request\n" ~
        makeCRSigV4(r.canonicalRequest);
}

unittest {
    string[string] empty;

    SignableRequest r;
    r.dateString = "20110909";
    r.timeStringUTC = "233600";
    r.region = "us-east-1";
    r.service = "iam";
    r.canonicalRequest = CanonicalRequest(
            "POST",
            "/",
            empty,
            ["content-type": "application/x-www-form-urlencoded; charset=utf-8",
             "host": "iam.amazonaws.com",
             "x-amz-date": "20110909T233600Z"],
            cast(ubyte[])"Action=ListUsers&Version=2010-05-08");

    auto sampleString =
        algorithm ~ "\n" ~
        "20110909T233600Z\n" ~
        "20110909/us-east-1/iam/aws4_request\n" ~
        "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

    assert(sampleString == signableString(r));
}

ubyte[] array_xor(ubyte[] b1, ubyte[] b2)
{
    assert(b1.length == b2.length);
    ubyte[] ret;
    for (uint i = 0; i < b1.length; i++)
        ret ~= b1[i] ^ b2[i];
    return ret;
}

auto hmac_sha256(string key, string message)
{
    auto hmac = hmac!SHA256(key.representation);
    hmac.put(message.representation);
    return hmac.finish;
}

unittest {
    string key = "key";
    string message = "The quick brown fox jumps over the lazy dog";

    string mac = hmac_sha256(key, message).toHexString().toLower();
    assert(mac == "f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8");
}

auto signingKey(string secret, string dateString, string region, string service)
{
    auto kSecret = "AWS4" ~ secret;
    auto kDate = hmac_sha256(kSecret, dateString);
    auto kRegion = hmac_sha256(kDate, region);
    auto kService = hmac_sha256(kRegion, service);
    return hmac_sha256(kService, "aws4_request");
}

unittest {
    string secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
    auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

    ubyte[] expected = [152, 241, 216, 137, 254, 196, 244, 66, 26, 220, 82, 43, 171, 12, 225, 248, 46, 105, 41, 194, 98, 237, 21, 229, 169, 76, 144, 239, 209, 227, 176, 231 ];
    assert(expected == signKey);
}

alias sign = hmac_sha256;

unittest {
    auto sampleString =
        "AWS4-HMAC-SHA256\n" ~
        "20110909T233600Z\n" ~
        "20110909/us-east-1/iam/aws4_request\n" ~
        "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

    auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
    auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

    auto signature = sign(signKey, sampleString).toHexString().toLower();
    auto expected = "ced6826de92d2bdeed8f846f0bf508e8559e98e4b0199114b84c54174deb456c";

    assert(signature == expected);
}

/**
 * CredentialScope == date / region / service / aws4_request
 */
string createSignatureHeader(string accessKeyID, string credentialScope, string[string] reqHeaders, ubyte[] signature)
{
    return algorithm ~ " Credential=" ~ accessKeyID ~ "/" ~ credentialScope ~ "/aws4_request, SignedHeaders=" ~ signedHeaders(reqHeaders) ~ ", Signature=" ~ signature.toHexString().toLower();
}

string dateFromISOString(string iso)
{
    auto i = iso.indexOf('T');
    if (i == -1) throw new Exception("ISO time in wrong format: " ~ iso);
    return iso[0..i];
}

string timeFromISOString(string iso)
{
    auto t = iso.indexOf('T');
    auto z = iso.indexOf('Z');
    if (t == -1 || z == -1) throw new Exception("ISO time in wrong format: " ~ iso);
    return iso[t+1..z];
}

unittest {
    assert(dateFromISOString("20110909T1203Z") == "20110909");
}


void main()
{
    auto sampleString =
        "AWS4-HMAC-SHA256\n" ~
        "20110909T233600Z\n" ~
        "20110909/us-east-1/iam/aws4_request\n" ~
        "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";

    auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
    auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");

    auto signature = sign(signKey, sampleString).toHexString().toLower();

    writeln(signature);
}

When i try to compile it im getting such error:

[holo@ultraxps test]$ dmd -unittest hello.d
hello.d(196): Error: function sigv4.hmac_sha256 (string key, string message) is not callable using argument types (ubyte[32], string)
[holo@ultraxps test]$ dmd hello.d
hello.d(196): Error: function sigv4.hmac_sha256 (string key, string message) is not callable using argument types (ubyte[32], string)
[holo@ultraxps test]$


Line 196 is: "auto kRegion = hmac_sha256(kDate, region);"

I was looking for ubyte[32] variable but i cant find it anywhere. What is strange unit tests are passing (i think, maybe im wrong).

On 07/10/15 3:18 PM, holo wrote:
>> Congrats on getting it working!
>
> @Rikki Thanks :)
>
> I was trying to write my own lib from beginning based on examples but
> after some time i resign from that idea (will back to it when i will
> have some more experience) and right now im trying to customize that one
> from link which yawniek paste:
>
> https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d
>
> I removed import from vibe.d library and copy/paste missed functions
> formEncode and filterURLEncode (BTW: what that "(R)" mean in it?
> filterURLEncode(R)(ref R dst, ..., ..) ).

If you see a template argument (which R is) it is typically meant for a range. In this case an output range. Although there should be isOutputRange!R in an template if condition, to check this.

> Next thing what i did was to replace hmac function to use hmac module
> from newest library. Now whole code looks like this:
>
> module sigv4;
>
> import std.array;
> import std.algorithm;
> import std.digest.sha;
> import std.range;
> import std.stdio;
> import std.string;
> import std.format;
> import std.digest.hmac;
>
>
> const algorithm = "AWS4-HMAC-SHA256";
>
>
> void filterURLEncode(R)(ref R dst, string str, string allowed_chars =
> null, bool form_encoding = false)
> {
>      while( str.length > 0 ) {
>          switch(str[0]) {
>              case ' ':
>                  if (form_encoding) {
>                      dst.put('+');
>                      break;
>                  }
>                  goto default;
>              case 'A': .. case 'Z':
>              case 'a': .. case 'z':
>              case '0': .. case '9':
>              case '-': case '_': case '.': case '~':
>                  dst.put(str[0]);
>                  break;
>              default:
>                  if (allowed_chars.canFind(str[0])) dst.put(str[0]);
>                  else formattedWrite(dst, "%%%02X", str[0]);
>          }
>          str = str[1 .. $];
>      }
> }
>
> string formEncode(string str, string allowed_chars = null)
> @safe {
>      auto dst = appender!string();
>      dst.reserve(str.length);
>      filterURLEncode(dst, str, allowed_chars, true);
>      return dst.data;
> }
>
> struct CanonicalRequest
> {
>      string method;
>      string uri;
>      string[string] queryParameters;
>      string[string] headers;
>      ubyte[] payload;
> }
>
> string canonicalQueryString(string[string] queryParameters)
> {
>      alias encode = formEncode;
>
>      string[string] encoded;
>      foreach (p; queryParameters.keys())
>      {
>          encoded[encode(p)] = encode(queryParameters[p]);
>      }
>      string[] keys = encoded.keys();
>      sort(keys);
>      return keys.map!(k => k ~ "=" ~ encoded[k]).join("&");
> }
>
> string canonicalHeaders(string[string] headers)
> {
>      string[string] trimmed;
>      foreach (h; headers.keys())
>      {
>          trimmed[h.toLower().strip()] = headers[h].strip();
>      }
>      string[] keys = trimmed.keys();
>      sort(keys);
>      return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join("");
> }
>
> string signedHeaders(string[string] headers)
> {
>      string[] keys = headers.keys().map!(k => k.toLower()).array();
>      sort(keys);
>      return keys.join(";");
> }
>
> string hash(T)(T payload)
> {
>      auto hash = sha256Of(payload);
>      return hash.toHexString().toLower();
> }
>
> string makeCRSigV4(CanonicalRequest r)
> {
>      auto cr =
>          r.method.toUpper() ~ "\n" ~
>          (r.uri.empty ? "/" : r.uri) ~ "\n" ~
>          canonicalQueryString(r.queryParameters) ~ "\n" ~
>          canonicalHeaders(r.headers) ~ "\n" ~
>          signedHeaders(r.headers) ~ "\n" ~
>          hash(r.payload);
>
>      return hash(cr);
> }
>
> unittest {
>      string[string] empty;
>
>      auto r = CanonicalRequest(
>              "POST",
>              "/",
>              empty,
>              ["content-type": "application/x-www-form-urlencoded;
> charset=utf-8",
>               "host": "iam.amazonaws.com",
>               "x-amz-date": "20110909T233600Z"],
>              cast(ubyte[])"Action=ListUsers&Version=2010-05-08");
>
>      auto sig = makeCRSigV4(r);
>
>      assert(sig ==
> "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2");
> }
>
> struct SignableRequest
> {
>      string dateString;
>      string timeStringUTC;
>      string region;
>      string service;
>      CanonicalRequest canonicalRequest;
> }
>
> string signableString(SignableRequest r) {
>      return algorithm ~ "\n" ~
>          r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~
>          r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~
> "/aws4_request\n" ~
>          makeCRSigV4(r.canonicalRequest);
> }
>
> unittest {
>      string[string] empty;
>
>      SignableRequest r;
>      r.dateString = "20110909";
>      r.timeStringUTC = "233600";
>      r.region = "us-east-1";
>      r.service = "iam";
>      r.canonicalRequest = CanonicalRequest(
>              "POST",
>              "/",
>              empty,
>              ["content-type": "application/x-www-form-urlencoded;
> charset=utf-8",
>               "host": "iam.amazonaws.com",
>               "x-amz-date": "20110909T233600Z"],
>              cast(ubyte[])"Action=ListUsers&Version=2010-05-08");
>
>      auto sampleString =
>          algorithm ~ "\n" ~
>          "20110909T233600Z\n" ~
>          "20110909/us-east-1/iam/aws4_request\n" ~
> "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";
>
>      assert(sampleString == signableString(r));
> }
>
> ubyte[] array_xor(ubyte[] b1, ubyte[] b2)
> {
>      assert(b1.length == b2.length);
>      ubyte[] ret;
>      for (uint i = 0; i < b1.length; i++)
>          ret ~= b1[i] ^ b2[i];
>      return ret;
> }
>
> auto hmac_sha256(string key, string message)
> {
>      auto hmac = hmac!SHA256(key.representation);
>      hmac.put(message.representation);
>      return hmac.finish;
> }
>
> unittest {
>      string key = "key";
>      string message = "The quick brown fox jumps over the lazy dog";
>
>      string mac = hmac_sha256(key, message).toHexString().toLower();
>      assert(mac ==
> "f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8");
> }
>
> auto signingKey(string secret, string dateString, string region, string
> service)
> {
>      auto kSecret = "AWS4" ~ secret;
>      auto kDate = hmac_sha256(kSecret, dateString);
>      auto kRegion = hmac_sha256(kDate, region);
>      auto kService = hmac_sha256(kRegion, service);
>      return hmac_sha256(kService, "aws4_request");
> }
>
> unittest {
>      string secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
>      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");
>
>      ubyte[] expected = [152, 241, 216, 137, 254, 196, 244, 66, 26, 220,
> 82, 43, 171, 12, 225, 248, 46, 105, 41, 194, 98, 237, 21, 229, 169, 76,
> 144, 239, 209, 227, 176, 231 ];
>      assert(expected == signKey);
> }
>
> alias sign = hmac_sha256;
>
> unittest {
>      auto sampleString =
>          "AWS4-HMAC-SHA256\n" ~
>          "20110909T233600Z\n" ~
>          "20110909/us-east-1/iam/aws4_request\n" ~
> "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";
>
>      auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
>      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");
>
>      auto signature = sign(signKey, sampleString).toHexString().toLower();
>      auto expected =
> "ced6826de92d2bdeed8f846f0bf508e8559e98e4b0199114b84c54174deb456c";
>
>      assert(signature == expected);
> }
>
> /**
>   * CredentialScope == date / region / service / aws4_request
>   */
> string createSignatureHeader(string accessKeyID, string credentialScope,
> string[string] reqHeaders, ubyte[] signature)
> {
>      return algorithm ~ " Credential=" ~ accessKeyID ~ "/" ~
> credentialScope ~ "/aws4_request, SignedHeaders=" ~
> signedHeaders(reqHeaders) ~ ", Signature=" ~
> signature.toHexString().toLower();
> }
>
> string dateFromISOString(string iso)
> {
>      auto i = iso.indexOf('T');
>      if (i == -1) throw new Exception("ISO time in wrong format: " ~ iso);
>      return iso[0..i];
> }
>
> string timeFromISOString(string iso)
> {
>      auto t = iso.indexOf('T');
>      auto z = iso.indexOf('Z');
>      if (t == -1 || z == -1) throw new Exception("ISO time in wrong
> format: " ~ iso);
>      return iso[t+1..z];
> }
>
> unittest {
>      assert(dateFromISOString("20110909T1203Z") == "20110909");
> }
>
>
> void main()
> {
>      auto sampleString =
>          "AWS4-HMAC-SHA256\n" ~
>          "20110909T233600Z\n" ~
>          "20110909/us-east-1/iam/aws4_request\n" ~
> "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2";
>
>      auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
>      auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam");
>
>      auto signature = sign(signKey, sampleString).toHexString().toLower();
>
>      writeln(signature);
> }
>
> When i try to compile it im getting such error:
>
> [holo@ultraxps test]$ dmd -unittest hello.d
> hello.d(196): Error: function sigv4.hmac_sha256 (string key, string
> message) is not callable using argument types (ubyte[32], string)
> [holo@ultraxps test]$ dmd hello.d
> hello.d(196): Error: function sigv4.hmac_sha256 (string key, string
> message) is not callable using argument types (ubyte[32], string)
> [holo@ultraxps test]$
>
>
> Line 196 is: "auto kRegion = hmac_sha256(kDate, region);"
>
> I was looking for ubyte[32] variable but i cant find it anywhere. What
> is strange unit tests are passing (i think, maybe im wrong).

It's in signingKey. Remember auto just means work out type based upon assigning value's type.
I would really really recommend since you keep having issues with hmac, to go the Botan way. Since it works.

I was fighting with it a little bit and after all i just leave original function which was in code, removed dependencies from vibe.d and finally tried to contact api. Here is my present code:

#!/usr/bin/rdmd -L-lcurl

import std.stdio;
import std.string;
import std.file;
import std.datetime;
import std.process;
import std.digest.sha;
import std.net.curl;
import sigv4;


auto zone = "us-east-1";
auto service = "ec2";


void main()
{
	auto accKey = environment["AWS_ACCESS_KEY"];
	auto secKey = environment["AWS_SECRET_KEY"];

	auto currentClock = Clock.currTime;

	auto currentDate = cast(Date)currentClock;
	auto curDateStr = currentDate.toISOString;

	auto currentTime = cast(TimeOfDay)currentClock;
	auto curTimeStr = currentTime.toISOString;

	auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

	string[string] empty;

	SignableRequest r;
	r.dateString = curDateStr;
	r.timeStringUTC = curTimeStr;
	r.region = zone;
	r.service = service;
	r.canonicalRequest = CanonicalRequest(
			"GET",
			"/",
			["action" : "DescribeInstances", "version" : "2013-10-15"],
			["content-type" : "application/x-www-form-urlencoded; charset=utf-8",
			 "host" : service ~ ".amazonaws.com",
			 "x-amz-date" : xamztime],
			 cast(ubyte[])"");
	
	auto qParm = canonicalQueryString(r.canonicalRequest.queryParameters);

	auto sigHead = canonicalHeaders(r.canonicalRequest.headers);

	auto sigStr = signableString(r);

	auto sigKey = signingKey(secKey, curDateStr, zone, service);
	
	auto signature = sign(sigKey, cast(ubyte[])sigStr).toHexString().toLower();

	writeln();	
	writeln(qParm);
	writeln();
	writeln(sigHead);
	writeln();
	writeln(sigStr);
	writeln();
	writeln(signature);
	writeln();

	auto client = HTTP();
	client.clearRequestHeaders;
	client.addRequestHeader("content-type:", "application/x-www-form-urlencoded; charset=utf-8");
	client.addRequestHeader("host:", service ~ ".amazonaws.com");
	client.addRequestHeader("x-amz-date:", xamztime);
	client.addRequestHeader("Authoryzation:", "AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ ", " ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " ~ "Signature=" ~ signature);

	auto url = service ~ ".amazonaws.com?" ~ "Action=DescribeInstances&Version=2013-10-15";
	writeln(url);
	auto content = get(url, client);
	writeln(content);
}

Everything is compiling but im getting 400 (Bad Request):

  [root@ultraxps aws]# ./header.d

action=DescribeRegions&version=2013-10-15

content-type: application/x-www-form-urlencoded; charset=utf-8
host: ec2.amazonaws.com
x-amz-date: 20151009T053800Z


AWS4-HMAC-SHA256
20151009T053800Z
20151009/us-east-1/ec2/aws4_request
888595748692147ceafafcae3941ec0d83ac42c97641e4d954d7447a00c56270

69b1e4c5212cc6b485569fdfb43f7dde94413b36c50393c55d4810ced47f167b

ec2.amazonaws.com?Action=DescribeRegions&Version=2013-10-15
std.net.curl.CurlException@/usr/include/dlang/dmd/std/net/curl.d(824): HTTP request returned status code 400 (Bad Request)
----------------
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(pure @safe bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong)+0x65) [0x5004bd]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[] std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[], std.net.curl.HTTP)+0x1a2) [0x4fc0ba]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[] std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[], std.net.curl.HTTP)+0x6a) [0x4fbec2]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_Dmain+0x840) [0x4f8f98]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv+0x1f) [0x533eab]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x2b) [0x533e67]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_d_run_main+0x1d2) [0x533d86]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(main+0x12) [0x52cb62]
/usr/lib/libc.so.6(__libc_start_main+0xf0) [0x7f9c39f71610]
[root@ultraxps aws]#

What am i doing wrong? What is strange when when i do same with curl its responding normally of course is not authenticated).

[root@ultraxps aws]# curl ec2.amazonaws.com?Action=DescribeInstances&Version=2013-10-15
[1] 20390
[root@ultraxps aws]# <?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>MissingParameter</Code><Message>The request must contain the parameter AWSAccessKeyId</Message></Error></Errors><RequestID>e1352781-c2b4-4e74-ade3-80d655efd0ac</RequestID></Response>