October 03, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to holo | On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote: > Hello > > I'm trying to contact AWS API with D according to documentation: > > [...] check https://github.com/yannick/vibe-aws it has v4 implemented |
October 03, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to holo | On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote: > Last but not least, how to write such function in D: > > def sign(key, msg): > return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest() > > ? > > I can't find in standard libraryt hmac function, is it existing? The next version of D will have a std.digest.hmac module: http://dlang.org/phobos-prerelease/std_digest_hmac.html |
October 03, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to holo | On 04/10/15 2:31 AM, holo wrote:
> On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole wrote:
>> On 04/10/15 1:49 AM, holo wrote:
>>> On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:
>>>> On 04/10/15 1:09 AM, holo wrote:
>>>>> [...]
>>>>
>>>> By the looks of things the problem is with SHA256, I'm guessing it
>>>> doesn't have its block size defined.
>>>> https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394
>>>>
>>>>
>>>> You'll also need to update sha.d as well. Otherwise it looks good.
>>>
>>> I updated sha.d and now im getting following error:
>>>
>>> $ ./app.d
>>> /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function
>>> 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure
>>> function 'core.cpuid.ssse3'
>>> /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template
>>> instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
>>> const(ubyte)[]) error instantiating
>>> /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here:
>>> isDigest!(SHA!(512u, 160u))
>>> /usr/include/dlang/dmd/std/digest/sha.d(1179): while looking for
>>> match for WrapperDigest!(SHA!(512u, 160u))
>>> Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]
>>
>> And one more file (ssse3.d) needs to be updated.
>> You are going to be playing whack a mole I think for a while.
>
> I downloaded whole master branch phobos and tried to use it (changed
> path in dmd.conf), but there are missing much more files (i think that
> master branch is not ready yet). So i get back to my original library
> and tried to updated file you mention. But i can't find it (ssse3.d),
> its not appearing in my stable lib and even in that master-branch from
> zip file.
Apologies, I didn't see this till after I got to bed. It's core.cpuid not ssse3.d. It will be in druntime, not Phobos.
|
October 05, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Rikki Cattermole | On Saturday, 3 October 2015 at 23:58:39 UTC, Rikki Cattermole wrote: > On 04/10/15 2:31 AM, holo wrote: >> On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole wrote: >>> On 04/10/15 1:49 AM, holo wrote: >>>> On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote: >>>>> On 04/10/15 1:09 AM, holo wrote: >>>>>> [...] >>>>> >>>>> By the looks of things the problem is with SHA256, I'm guessing it >>>>> doesn't have its block size defined. >>>>> https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394 >>>>> >>>>> >>>>> You'll also need to update sha.d as well. Otherwise it looks good. >>>> >>>> I updated sha.d and now im getting following error: >>>> >>>> $ ./app.d >>>> /usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function >>>> 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure >>>> function 'core.cpuid.ssse3' >>>> /usr/include/dlang/dmd/std/digest/digest.d(285): Error: template >>>> instance std.range.primitives.isOutputRange!(SHA!(512u, 160u), >>>> const(ubyte)[]) error instantiating >>>> /usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here: >>>> isDigest!(SHA!(512u, 160u)) >>>> /usr/include/dlang/dmd/std/digest/sha.d(1179): while looking for >>>> match for WrapperDigest!(SHA!(512u, 160u)) >>>> Failed: ["dmd", "-v", "-o-", "./app.d", "-I."] >>> >>> And one more file (ssse3.d) needs to be updated. >>> You are going to be playing whack a mole I think for a while. >> >> I downloaded whole master branch phobos and tried to use it (changed >> path in dmd.conf), but there are missing much more files (i think that >> master branch is not ready yet). So i get back to my original library >> and tried to updated file you mention. But i can't find it (ssse3.d), >> its not appearing in my stable lib and even in that master-branch from >> zip file. > > Apologies, I didn't see this till after I got to bed. It's core.cpuid not ssse3.d. It will be in druntime, not Phobos. Sorry i was busy at weekend. Thank you all for answers. I will try to write some solution basing on yawniek example and will back if some troubles appears. @Vladimir where can i check or do you know when next version of phobos will be available? @Rikki i downloaded both master zip files for runtime and phobos librarys all dependencies looks are satisfied right now but still getting errors: [holo@ultraxps workplace]$ dmd hmac.d hmac.o: In function `_D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC': hmac.d:(.text._D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x18): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6__initZ' hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC': hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x1b): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x31): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x3e): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h' hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC': hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x8d): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0xa4): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC': hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x5f): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.o: In function `_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h': hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xa1): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xaa): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xc4): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xdb): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv' hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6finishMFNaNbNiNfZG20h+0xe8): undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h' hmac.o: In function `_D3std6format61__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderThTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv': hmac.d:(.text._D3std6format61__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderThTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv' hmac.o: In function `_D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxhTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv': hmac.d:(.text._D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxhTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxhKS3std6format18__T10FormatSpecTaZ10FormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv' hmac.o: In function `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTlTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxlKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv': hmac.d:(.text._D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTlTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxlKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv+0x46): undefined reference to `_D3std6format64__T14formatUnsignedTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatUnsignedFNaNbNfS3std5array17__T8AppenderTAyaZ8AppendermKxS3std6format18__T10FormatSpecTaZ10FormatSpeckbZv' hmac.o: In function `_D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxmTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxmKS3std6format18__T10FormatSpecTaZ10FormatSpecZv': hmac.d:(.text._D3std6format62__T11formatValueTS3std5array17__T8AppenderTAyaZ8AppenderTxmTaZ11formatValueFNaNfS3std5array17__T8AppenderTAyaZ8AppenderxmKS3std6format18__T10FormatSpecTaZ10FormatSpecZv+0x16a): undefined reference to `_D3std6format64__T14formatIntegralTS3std5array17__T8AppenderTAyaZ8AppenderTmTaZ14formatIntegralFNaNbNfS3std5array17__T8AppenderTAyaZ8AppenderxmKxS3std6format18__T10FormatSpecTaZ10FormatSpeckmZv' collect2: error: ld returned 1 exit status --- errorlevel 1 [holo@ultraxps workplace]$ cat hmac.d #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA1("secretkey".representation); hmac.put("texttohash".representation); auto digest = hmac.finish(); writeln(digest); } [holo@ultraxps workplace]$ |
October 05, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to holo | On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote: > @Vladimir where can i check or do you know when next version of phobos will be available? You can use Digger to get the latest version of D: https://github.com/CyberShadow/Digger |
October 05, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Vladimir Panteleev | On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev wrote: > On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote: >> @Vladimir where can i check or do you know when next version of phobos will be available? > > You can use Digger to get the latest version of D: > > https://github.com/CyberShadow/Digger Thank you, now it is working as expected. Hope i wont miss upgrade in my system to uninstall digger version before. [holo@ultraxps workplace]$ cat ./hmac.d #!/usr/bin/rdmd import std.stdio; import std.digest.sha, std.digest.hmac; import std.string : representation; void main() { auto hmac = hmac!SHA256("secretkey".representation); hmac.put("texttohash".representation); auto digest = hmac.finish(); writeln(digest); } [holo@ultraxps workplace]$ ./hmac.d [28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 148, 84, 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 252, 26, 124, 215, 6] [holo@ultraxps workplace]$ dmd --version DMD64 D Compiler v2.069-devel-5e73c30 Copyright (c) 1999-2015 by Digital Mars written by Walter Bright [holo@ultraxps workplace]$ Now need try to write some code which will connect me to AWS api based on examples. |
October 06, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to holo | On 06/10/15 11:26 AM, holo wrote:
> On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev wrote:
>> On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
>>> @Vladimir where can i check or do you know when next version of
>>> phobos will be available?
>>
>> You can use Digger to get the latest version of D:
>>
>> https://github.com/CyberShadow/Digger
>
> Thank you, now it is working as expected. Hope i wont miss upgrade in my
> system to uninstall digger version before.
>
> [holo@ultraxps workplace]$ cat ./hmac.d
> #!/usr/bin/rdmd
>
> import std.stdio;
> import std.digest.sha, std.digest.hmac;
> import std.string : representation;
>
> void main()
> {
> auto hmac = hmac!SHA256("secretkey".representation);
> hmac.put("texttohash".representation);
> auto digest = hmac.finish();
>
> writeln(digest);
> }
>
> [holo@ultraxps workplace]$ ./hmac.d
> [28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 148, 84,
> 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 252, 26, 124,
> 215, 6]
> [holo@ultraxps workplace]$ dmd --version
> DMD64 D Compiler v2.069-devel-5e73c30
> Copyright (c) 1999-2015 by Digital Mars written by Walter Bright
> [holo@ultraxps workplace]$
>
> Now need try to write some code which will connect me to AWS api based
> on examples.
Congrats on getting it working!
|
October 07, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Rikki Cattermole | > Congrats on getting it working! @Rikki Thanks :) I was trying to write my own lib from beginning based on examples but after some time i resign from that idea (will back to it when i will have some more experience) and right now im trying to customize that one from link which yawniek paste: https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d I removed import from vibe.d library and copy/paste missed functions formEncode and filterURLEncode (BTW: what that "(R)" mean in it? filterURLEncode(R)(ref R dst, ..., ..) ). Next thing what i did was to replace hmac function to use hmac module from newest library. Now whole code looks like this: module sigv4; import std.array; import std.algorithm; import std.digest.sha; import std.range; import std.stdio; import std.string; import std.format; import std.digest.hmac; const algorithm = "AWS4-HMAC-SHA256"; void filterURLEncode(R)(ref R dst, string str, string allowed_chars = null, bool form_encoding = false) { while( str.length > 0 ) { switch(str[0]) { case ' ': if (form_encoding) { dst.put('+'); break; } goto default; case 'A': .. case 'Z': case 'a': .. case 'z': case '0': .. case '9': case '-': case '_': case '.': case '~': dst.put(str[0]); break; default: if (allowed_chars.canFind(str[0])) dst.put(str[0]); else formattedWrite(dst, "%%%02X", str[0]); } str = str[1 .. $]; } } string formEncode(string str, string allowed_chars = null) @safe { auto dst = appender!string(); dst.reserve(str.length); filterURLEncode(dst, str, allowed_chars, true); return dst.data; } struct CanonicalRequest { string method; string uri; string[string] queryParameters; string[string] headers; ubyte[] payload; } string canonicalQueryString(string[string] queryParameters) { alias encode = formEncode; string[string] encoded; foreach (p; queryParameters.keys()) { encoded[encode(p)] = encode(queryParameters[p]); } string[] keys = encoded.keys(); sort(keys); return keys.map!(k => k ~ "=" ~ encoded[k]).join("&"); } string canonicalHeaders(string[string] headers) { string[string] trimmed; foreach (h; headers.keys()) { trimmed[h.toLower().strip()] = headers[h].strip(); } string[] keys = trimmed.keys(); sort(keys); return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join(""); } string signedHeaders(string[string] headers) { string[] keys = headers.keys().map!(k => k.toLower()).array(); sort(keys); return keys.join(";"); } string hash(T)(T payload) { auto hash = sha256Of(payload); return hash.toHexString().toLower(); } string makeCRSigV4(CanonicalRequest r) { auto cr = r.method.toUpper() ~ "\n" ~ (r.uri.empty ? "/" : r.uri) ~ "\n" ~ canonicalQueryString(r.queryParameters) ~ "\n" ~ canonicalHeaders(r.headers) ~ "\n" ~ signedHeaders(r.headers) ~ "\n" ~ hash(r.payload); return hash(cr); } unittest { string[string] empty; auto r = CanonicalRequest( "POST", "/", empty, ["content-type": "application/x-www-form-urlencoded; charset=utf-8", "host": "iam.amazonaws.com", "x-amz-date": "20110909T233600Z"], cast(ubyte[])"Action=ListUsers&Version=2010-05-08"); auto sig = makeCRSigV4(r); assert(sig == "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"); } struct SignableRequest { string dateString; string timeStringUTC; string region; string service; CanonicalRequest canonicalRequest; } string signableString(SignableRequest r) { return algorithm ~ "\n" ~ r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~ r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~ "/aws4_request\n" ~ makeCRSigV4(r.canonicalRequest); } unittest { string[string] empty; SignableRequest r; r.dateString = "20110909"; r.timeStringUTC = "233600"; r.region = "us-east-1"; r.service = "iam"; r.canonicalRequest = CanonicalRequest( "POST", "/", empty, ["content-type": "application/x-www-form-urlencoded; charset=utf-8", "host": "iam.amazonaws.com", "x-amz-date": "20110909T233600Z"], cast(ubyte[])"Action=ListUsers&Version=2010-05-08"); auto sampleString = algorithm ~ "\n" ~ "20110909T233600Z\n" ~ "20110909/us-east-1/iam/aws4_request\n" ~ "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; assert(sampleString == signableString(r)); } ubyte[] array_xor(ubyte[] b1, ubyte[] b2) { assert(b1.length == b2.length); ubyte[] ret; for (uint i = 0; i < b1.length; i++) ret ~= b1[i] ^ b2[i]; return ret; } auto hmac_sha256(string key, string message) { auto hmac = hmac!SHA256(key.representation); hmac.put(message.representation); return hmac.finish; } unittest { string key = "key"; string message = "The quick brown fox jumps over the lazy dog"; string mac = hmac_sha256(key, message).toHexString().toLower(); assert(mac == "f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8"); } auto signingKey(string secret, string dateString, string region, string service) { auto kSecret = "AWS4" ~ secret; auto kDate = hmac_sha256(kSecret, dateString); auto kRegion = hmac_sha256(kDate, region); auto kService = hmac_sha256(kRegion, service); return hmac_sha256(kService, "aws4_request"); } unittest { string secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); ubyte[] expected = [152, 241, 216, 137, 254, 196, 244, 66, 26, 220, 82, 43, 171, 12, 225, 248, 46, 105, 41, 194, 98, 237, 21, 229, 169, 76, 144, 239, 209, 227, 176, 231 ]; assert(expected == signKey); } alias sign = hmac_sha256; unittest { auto sampleString = "AWS4-HMAC-SHA256\n" ~ "20110909T233600Z\n" ~ "20110909/us-east-1/iam/aws4_request\n" ~ "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); auto signature = sign(signKey, sampleString).toHexString().toLower(); auto expected = "ced6826de92d2bdeed8f846f0bf508e8559e98e4b0199114b84c54174deb456c"; assert(signature == expected); } /** * CredentialScope == date / region / service / aws4_request */ string createSignatureHeader(string accessKeyID, string credentialScope, string[string] reqHeaders, ubyte[] signature) { return algorithm ~ " Credential=" ~ accessKeyID ~ "/" ~ credentialScope ~ "/aws4_request, SignedHeaders=" ~ signedHeaders(reqHeaders) ~ ", Signature=" ~ signature.toHexString().toLower(); } string dateFromISOString(string iso) { auto i = iso.indexOf('T'); if (i == -1) throw new Exception("ISO time in wrong format: " ~ iso); return iso[0..i]; } string timeFromISOString(string iso) { auto t = iso.indexOf('T'); auto z = iso.indexOf('Z'); if (t == -1 || z == -1) throw new Exception("ISO time in wrong format: " ~ iso); return iso[t+1..z]; } unittest { assert(dateFromISOString("20110909T1203Z") == "20110909"); } void main() { auto sampleString = "AWS4-HMAC-SHA256\n" ~ "20110909T233600Z\n" ~ "20110909/us-east-1/iam/aws4_request\n" ~ "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); auto signature = sign(signKey, sampleString).toHexString().toLower(); writeln(signature); } When i try to compile it im getting such error: [holo@ultraxps test]$ dmd -unittest hello.d hello.d(196): Error: function sigv4.hmac_sha256 (string key, string message) is not callable using argument types (ubyte[32], string) [holo@ultraxps test]$ dmd hello.d hello.d(196): Error: function sigv4.hmac_sha256 (string key, string message) is not callable using argument types (ubyte[32], string) [holo@ultraxps test]$ Line 196 is: "auto kRegion = hmac_sha256(kDate, region);" I was looking for ubyte[32] variable but i cant find it anywhere. What is strange unit tests are passing (i think, maybe im wrong). |
October 07, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to holo | On 07/10/15 3:18 PM, holo wrote: >> Congrats on getting it working! > > @Rikki Thanks :) > > I was trying to write my own lib from beginning based on examples but > after some time i resign from that idea (will back to it when i will > have some more experience) and right now im trying to customize that one > from link which yawniek paste: > > https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d > > I removed import from vibe.d library and copy/paste missed functions > formEncode and filterURLEncode (BTW: what that "(R)" mean in it? > filterURLEncode(R)(ref R dst, ..., ..) ). If you see a template argument (which R is) it is typically meant for a range. In this case an output range. Although there should be isOutputRange!R in an template if condition, to check this. > Next thing what i did was to replace hmac function to use hmac module > from newest library. Now whole code looks like this: > > module sigv4; > > import std.array; > import std.algorithm; > import std.digest.sha; > import std.range; > import std.stdio; > import std.string; > import std.format; > import std.digest.hmac; > > > const algorithm = "AWS4-HMAC-SHA256"; > > > void filterURLEncode(R)(ref R dst, string str, string allowed_chars = > null, bool form_encoding = false) > { > while( str.length > 0 ) { > switch(str[0]) { > case ' ': > if (form_encoding) { > dst.put('+'); > break; > } > goto default; > case 'A': .. case 'Z': > case 'a': .. case 'z': > case '0': .. case '9': > case '-': case '_': case '.': case '~': > dst.put(str[0]); > break; > default: > if (allowed_chars.canFind(str[0])) dst.put(str[0]); > else formattedWrite(dst, "%%%02X", str[0]); > } > str = str[1 .. $]; > } > } > > string formEncode(string str, string allowed_chars = null) > @safe { > auto dst = appender!string(); > dst.reserve(str.length); > filterURLEncode(dst, str, allowed_chars, true); > return dst.data; > } > > struct CanonicalRequest > { > string method; > string uri; > string[string] queryParameters; > string[string] headers; > ubyte[] payload; > } > > string canonicalQueryString(string[string] queryParameters) > { > alias encode = formEncode; > > string[string] encoded; > foreach (p; queryParameters.keys()) > { > encoded[encode(p)] = encode(queryParameters[p]); > } > string[] keys = encoded.keys(); > sort(keys); > return keys.map!(k => k ~ "=" ~ encoded[k]).join("&"); > } > > string canonicalHeaders(string[string] headers) > { > string[string] trimmed; > foreach (h; headers.keys()) > { > trimmed[h.toLower().strip()] = headers[h].strip(); > } > string[] keys = trimmed.keys(); > sort(keys); > return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join(""); > } > > string signedHeaders(string[string] headers) > { > string[] keys = headers.keys().map!(k => k.toLower()).array(); > sort(keys); > return keys.join(";"); > } > > string hash(T)(T payload) > { > auto hash = sha256Of(payload); > return hash.toHexString().toLower(); > } > > string makeCRSigV4(CanonicalRequest r) > { > auto cr = > r.method.toUpper() ~ "\n" ~ > (r.uri.empty ? "/" : r.uri) ~ "\n" ~ > canonicalQueryString(r.queryParameters) ~ "\n" ~ > canonicalHeaders(r.headers) ~ "\n" ~ > signedHeaders(r.headers) ~ "\n" ~ > hash(r.payload); > > return hash(cr); > } > > unittest { > string[string] empty; > > auto r = CanonicalRequest( > "POST", > "/", > empty, > ["content-type": "application/x-www-form-urlencoded; > charset=utf-8", > "host": "iam.amazonaws.com", > "x-amz-date": "20110909T233600Z"], > cast(ubyte[])"Action=ListUsers&Version=2010-05-08"); > > auto sig = makeCRSigV4(r); > > assert(sig == > "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"); > } > > struct SignableRequest > { > string dateString; > string timeStringUTC; > string region; > string service; > CanonicalRequest canonicalRequest; > } > > string signableString(SignableRequest r) { > return algorithm ~ "\n" ~ > r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~ > r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~ > "/aws4_request\n" ~ > makeCRSigV4(r.canonicalRequest); > } > > unittest { > string[string] empty; > > SignableRequest r; > r.dateString = "20110909"; > r.timeStringUTC = "233600"; > r.region = "us-east-1"; > r.service = "iam"; > r.canonicalRequest = CanonicalRequest( > "POST", > "/", > empty, > ["content-type": "application/x-www-form-urlencoded; > charset=utf-8", > "host": "iam.amazonaws.com", > "x-amz-date": "20110909T233600Z"], > cast(ubyte[])"Action=ListUsers&Version=2010-05-08"); > > auto sampleString = > algorithm ~ "\n" ~ > "20110909T233600Z\n" ~ > "20110909/us-east-1/iam/aws4_request\n" ~ > "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; > > assert(sampleString == signableString(r)); > } > > ubyte[] array_xor(ubyte[] b1, ubyte[] b2) > { > assert(b1.length == b2.length); > ubyte[] ret; > for (uint i = 0; i < b1.length; i++) > ret ~= b1[i] ^ b2[i]; > return ret; > } > > auto hmac_sha256(string key, string message) > { > auto hmac = hmac!SHA256(key.representation); > hmac.put(message.representation); > return hmac.finish; > } > > unittest { > string key = "key"; > string message = "The quick brown fox jumps over the lazy dog"; > > string mac = hmac_sha256(key, message).toHexString().toLower(); > assert(mac == > "f7bc83f430538424b13298e6aa6fb143ef4d59a14946175997479dbc2d1a3cd8"); > } > > auto signingKey(string secret, string dateString, string region, string > service) > { > auto kSecret = "AWS4" ~ secret; > auto kDate = hmac_sha256(kSecret, dateString); > auto kRegion = hmac_sha256(kDate, region); > auto kService = hmac_sha256(kRegion, service); > return hmac_sha256(kService, "aws4_request"); > } > > unittest { > string secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; > auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); > > ubyte[] expected = [152, 241, 216, 137, 254, 196, 244, 66, 26, 220, > 82, 43, 171, 12, 225, 248, 46, 105, 41, 194, 98, 237, 21, 229, 169, 76, > 144, 239, 209, 227, 176, 231 ]; > assert(expected == signKey); > } > > alias sign = hmac_sha256; > > unittest { > auto sampleString = > "AWS4-HMAC-SHA256\n" ~ > "20110909T233600Z\n" ~ > "20110909/us-east-1/iam/aws4_request\n" ~ > "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; > > auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; > auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); > > auto signature = sign(signKey, sampleString).toHexString().toLower(); > auto expected = > "ced6826de92d2bdeed8f846f0bf508e8559e98e4b0199114b84c54174deb456c"; > > assert(signature == expected); > } > > /** > * CredentialScope == date / region / service / aws4_request > */ > string createSignatureHeader(string accessKeyID, string credentialScope, > string[string] reqHeaders, ubyte[] signature) > { > return algorithm ~ " Credential=" ~ accessKeyID ~ "/" ~ > credentialScope ~ "/aws4_request, SignedHeaders=" ~ > signedHeaders(reqHeaders) ~ ", Signature=" ~ > signature.toHexString().toLower(); > } > > string dateFromISOString(string iso) > { > auto i = iso.indexOf('T'); > if (i == -1) throw new Exception("ISO time in wrong format: " ~ iso); > return iso[0..i]; > } > > string timeFromISOString(string iso) > { > auto t = iso.indexOf('T'); > auto z = iso.indexOf('Z'); > if (t == -1 || z == -1) throw new Exception("ISO time in wrong > format: " ~ iso); > return iso[t+1..z]; > } > > unittest { > assert(dateFromISOString("20110909T1203Z") == "20110909"); > } > > > void main() > { > auto sampleString = > "AWS4-HMAC-SHA256\n" ~ > "20110909T233600Z\n" ~ > "20110909/us-east-1/iam/aws4_request\n" ~ > "3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2"; > > auto secretKey = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; > auto signKey = signingKey(secretKey, "20110909", "us-east-1", "iam"); > > auto signature = sign(signKey, sampleString).toHexString().toLower(); > > writeln(signature); > } > > When i try to compile it im getting such error: > > [holo@ultraxps test]$ dmd -unittest hello.d > hello.d(196): Error: function sigv4.hmac_sha256 (string key, string > message) is not callable using argument types (ubyte[32], string) > [holo@ultraxps test]$ dmd hello.d > hello.d(196): Error: function sigv4.hmac_sha256 (string key, string > message) is not callable using argument types (ubyte[32], string) > [holo@ultraxps test]$ > > > Line 196 is: "auto kRegion = hmac_sha256(kDate, region);" > > I was looking for ubyte[32] variable but i cant find it anywhere. What > is strange unit tests are passing (i think, maybe im wrong). It's in signingKey. Remember auto just means work out type based upon assigning value's type. I would really really recommend since you keep having issues with hmac, to go the Botan way. Since it works. |
October 09, 2015 Re: AWS API Dlang, hmac sha256 function. | ||||
---|---|---|---|---|
| ||||
Posted in reply to Rikki Cattermole | I was fighting with it a little bit and after all i just leave original function which was in code, removed dependencies from vibe.d and finally tried to contact api. Here is my present code: #!/usr/bin/rdmd -L-lcurl import std.stdio; import std.string; import std.file; import std.datetime; import std.process; import std.digest.sha; import std.net.curl; import sigv4; auto zone = "us-east-1"; auto service = "ec2"; void main() { auto accKey = environment["AWS_ACCESS_KEY"]; auto secKey = environment["AWS_SECRET_KEY"]; auto currentClock = Clock.currTime; auto currentDate = cast(Date)currentClock; auto curDateStr = currentDate.toISOString; auto currentTime = cast(TimeOfDay)currentClock; auto curTimeStr = currentTime.toISOString; auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z"; string[string] empty; SignableRequest r; r.dateString = curDateStr; r.timeStringUTC = curTimeStr; r.region = zone; r.service = service; r.canonicalRequest = CanonicalRequest( "GET", "/", ["action" : "DescribeInstances", "version" : "2013-10-15"], ["content-type" : "application/x-www-form-urlencoded; charset=utf-8", "host" : service ~ ".amazonaws.com", "x-amz-date" : xamztime], cast(ubyte[])""); auto qParm = canonicalQueryString(r.canonicalRequest.queryParameters); auto sigHead = canonicalHeaders(r.canonicalRequest.headers); auto sigStr = signableString(r); auto sigKey = signingKey(secKey, curDateStr, zone, service); auto signature = sign(sigKey, cast(ubyte[])sigStr).toHexString().toLower(); writeln(); writeln(qParm); writeln(); writeln(sigHead); writeln(); writeln(sigStr); writeln(); writeln(signature); writeln(); auto client = HTTP(); client.clearRequestHeaders; client.addRequestHeader("content-type:", "application/x-www-form-urlencoded; charset=utf-8"); client.addRequestHeader("host:", service ~ ".amazonaws.com"); client.addRequestHeader("x-amz-date:", xamztime); client.addRequestHeader("Authoryzation:", "AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ ", " ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " ~ "Signature=" ~ signature); auto url = service ~ ".amazonaws.com?" ~ "Action=DescribeInstances&Version=2013-10-15"; writeln(url); auto content = get(url, client); writeln(content); } Everything is compiling but im getting 400 (Bad Request): [root@ultraxps aws]# ./header.d action=DescribeRegions&version=2013-10-15 content-type: application/x-www-form-urlencoded; charset=utf-8 host: ec2.amazonaws.com x-amz-date: 20151009T053800Z AWS4-HMAC-SHA256 20151009T053800Z 20151009/us-east-1/ec2/aws4_request 888595748692147ceafafcae3941ec0d83ac42c97641e4d954d7447a00c56270 69b1e4c5212cc6b485569fdfb43f7dde94413b36c50393c55d4810ced47f167b ec2.amazonaws.com?Action=DescribeRegions&Version=2013-10-15 std.net.curl.CurlException@/usr/include/dlang/dmd/std/net/curl.d(824): HTTP request returned status code 400 (Bad Request) ---------------- /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(pure @safe bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool, lazy const(char)[], immutable(char)[], ulong)+0x65) [0x5004bd] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[] std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[], std.net.curl.HTTP)+0x1a2) [0x4fc0ba] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[] std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[], std.net.curl.HTTP)+0x6a) [0x4fbec2] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_Dmain+0x840) [0x4f8f98] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv+0x1f) [0x533eab] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x2b) [0x533e67] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_d_run_main+0x1d2) [0x533d86] /tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(main+0x12) [0x52cb62] /usr/lib/libc.so.6(__libc_start_main+0xf0) [0x7f9c39f71610] [root@ultraxps aws]# What am i doing wrong? What is strange when when i do same with curl its responding normally of course is not authenticated). [root@ultraxps aws]# curl ec2.amazonaws.com?Action=DescribeInstances&Version=2013-10-15 [1] 20390 [root@ultraxps aws]# <?xml version="1.0" encoding="UTF-8"?> <Response><Errors><Error><Code>MissingParameter</Code><Message>The request must contain the parameter AWSAccessKeyId</Message></Error></Errors><RequestID>e1352781-c2b4-4e74-ade3-80d655efd0ac</RequestID></Response> |
Copyright © 1999-2021 by the D Language Foundation