Hi all,

I am throwing my hand at writing D ports of the OpenSSL libraries.  It looks like a mammoth task but I figure I'll just chip away at it a piece at a time.

This seems like the right time to ask if anyone else is having a go at it, or wants to give me a hand.  I've actually just got a job (yay me) but wont start for a bit (11th June) so I have some spare time.  Once I start work I wont have as much time so it would be good to find someone else who can help.

In the meantime I have reached a point where a type PQ_64BIT is used.  Looking at the headers it appears this type is defined differently for each system/compiler based on whether they have a 64 bit type, so on windows using M$ compilers etc it is __int64 for example.  In the rest of cases it is a custom BIGNUM struct.

My question is this.. what the heck do I put in my D port when I see this type used?

In D, long is a 64 bit type, but... I'm not writing new code I am writing a wrapper so I need to put the actual type the C library is using (was compiled using), otherwise any struct using PQ_64BIT might be the wrong size and disastrous things might happen.

Am I understanding this correctly?

It seems that I can probably get away with swapping PQ_64BIT for long in the short term, as I suspect my library was compiled with a 64bit type, not the BIGNUM struct...

Thoughts?

Regan Heath wrote:
> Hi all,
> 
> I am throwing my hand at writing D ports of the OpenSSL libraries.  It looks like a mammoth task but I figure I'll just chip away at it a piece at a time.  
> 
> This seems like the right time to ask if anyone else is having a go at it, or wants to give me a hand.  I've actually just got a job (yay me) but wont start for a bit (11th June) so I have some spare time.  Once I start work I wont have as much time so it would be good to find someone else who can help.
> 
> In the meantime I have reached a point where a type PQ_64BIT is used.  Looking at the headers it appears this type is defined differently for each system/compiler based on whether they have a 64 bit type, so on windows using M$ compilers etc it is __int64 for example.  In the rest of cases it is a custom BIGNUM struct.
> 
> My question is this.. what the heck do I put in my D port when I see this type used?
> 
> In D, long is a 64 bit type, but... I'm not writing new code I am writing a wrapper so I need to put the actual type the C library is using (was compiled using), otherwise any struct using PQ_64BIT might be the wrong size and disastrous things might happen.
> 
> Am I understanding this correctly?
> 
> It seems that I can probably get away with swapping PQ_64BIT for long in the short term, as I suspect my library was compiled with a 64bit type, not the BIGNUM struct...
> 
> Thoughts?
> 
> 

How's the BIGNUM struct defined?

Regan Heath Wrote:

> This seems like the right time to ask if anyone else is having a go at it, or wants to give me a hand.
...maybe you like to join the IRC channel #d on freenode.net. :-)
In my opinion it's good for simple questions, this NG is very good for the harder ones.

> In the meantime I have reached a point where a type PQ_64BIT is used.  Looking at the headers it appears this type is defined differently for each system/compiler based on whether they have a 64 bit type, so on windows using M$ compilers etc it is __int64 for example.  In the rest of cases it is a custom BIGNUM struct.
> My question is this.. what the heck do I put in my D port when I see this type used?
I think you can go with long/ulong. Since the size is fixed for D, the compiler does the job emulating a 64 integer on systems where int is smaller 32bits.
The NEWS file of the openssl source package mentions that BIGNUM
is optimized for special tasks, probably optimizations the D compiler couldn't figure out.
Imho it's a good idea to use long/long for now and to postpone this optimization for such (more rare) systems. Maybe the D compilers can do this on it's own, I did not took a look a the BIGNUM yet.

good luck!

It sounds like PQ_64BIT is intended to always be 64 bits in length, so it's probably safe to just use a long.


Sean

Regan Heath wrote:
> Hi all,
> 
> I am throwing my hand at writing D ports of the OpenSSL libraries.  It looks like a mammoth task but I figure I'll just chip away at it a piece at a time.
> 
> This seems like the right time to ask if anyone else is having a go at it, or wants to give me a hand.  I've actually just got a job (yay me) but wont start for a bit (11th June) so I have some spare time.  Once I start work I wont have as much time so it would be good to find someone else who can help.
> 
> In the meantime I have reached a point where a type PQ_64BIT is used.  Looking at the headers it appears this type is defined differently for each system/compiler based on whether they have a 64 bit type, so on windows using M$ compilers etc it is __int64 for example.  In the rest of cases it is a custom BIGNUM struct.
> 
> My question is this.. what the heck do I put in my D port when I see this type used?
> 
> In D, long is a 64 bit type, but... I'm not writing new code I am writing a wrapper so I need to put the actual type the C library is using (was compiled using), otherwise any struct using PQ_64BIT might be the wrong size and disastrous things might happen.
> 
> Am I understanding this correctly?
> 
> It seems that I can probably get away with swapping PQ_64BIT for long in the short term, as I suspect my library was compiled with a 64bit type, not the BIGNUM struct...
> 
> Thoughts?

Having a very quick poke around the OpenSSL docs, it looks like PQ_64BIT is designed to be used as an opaque type.  The problem, of course, is that you don't know how the library was compiled.

It looks like the only safe way to do it is to mimic the C headers. Something like this:

version( SIXTY_FOUR_BIT )
{
    typedef ulong PQ_64BIT; // Note that I think OpenSSL uses ulongs!
    void pq_64bit_init(ref PQ_64BIT x) {}
    void pq_64bit_add_word(ref PQ_64BIT x, ref PQ_64BIT w) {x += w}
}
else
{
    typedef BIGNUM PQ_64BIT;
    void pq_64bit_init(ref PQ_64BIT x) { BN_INIT(&x); }
    void pq_64bit_add_word(ref PQ_64BIT x, ref PQ_64BIT w) {
BN_add_word(&x, &w); }
}

And so on.  Ugly as hell, but that's how OpenSSL is doing things.

Hope that's of some help :P

	-- Daniel

-- 
int getRandomNumber()
{
    return 4; // chosen by fair dice roll.
              // guaranteed to be random.
}

http://xkcd.com/

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/

torhu wrote:
> How's the BIGNUM struct defined?

As far as I can tell, the OpenSSL docs don't specify; they state that BIGNUM should be treated as an opaque type, and that you must never attempt to directly access its members.

That practically screams of "BIGNUM's implementation is platform-dependant" :P

Mandel wrote:
> I think you can go with long/ulong. Since the size is fixed for D, the compiler does the job emulating a 64 integer on systems where int is smaller 32bits.

It doesn't matter what D is capable of; if OpenSSL has been compiled in 32-bit mode, then it's using BIGNUM internally.  If you try passing it a long, then you can very well end up with serious errors (see below).

Sean Kelly wrote:
> It sounds like PQ_64BIT is intended to always be 64 bits in length, so it's probably safe to just use a long.
>
> Sean

Actually, PQ_64BIT can be aliased to BIGNUM which can be an *arbitrary* number of bits; it's a dynamically allocated structure.

However, if his version is compiled with 64-bit integer support, and he doesn't mind not supporting BIGNUM versions, he could just stick to ulong.  Maybe just stub out the other side of the version path with some static assert(false)'s. :)

Ok; bed time.

	-- Daniel

-- 
int getRandomNumber()
{
    return 4; // chosen by fair dice roll.
              // guaranteed to be random.
}

http://xkcd.com/

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/

Daniel Keep Wrote:
> Regan Heath wrote:
> > Hi all,
> > 
> > I am throwing my hand at writing D ports of the OpenSSL libraries.  It looks like a mammoth task but I figure I'll just chip away at it a piece at a time.
> > 
> > This seems like the right time to ask if anyone else is having a go at it, or wants to give me a hand.  I've actually just got a job (yay me) but wont start for a bit (11th June) so I have some spare time.  Once I start work I wont have as much time so it would be good to find someone else who can help.
> > 
> > In the meantime I have reached a point where a type PQ_64BIT is used.  Looking at the headers it appears this type is defined differently for each system/compiler based on whether they have a 64 bit type, so on windows using M$ compilers etc it is __int64 for example.  In the rest of cases it is a custom BIGNUM struct.
> > 
> > My question is this.. what the heck do I put in my D port when I see this type used?
> > 
> > In D, long is a 64 bit type, but... I'm not writing new code I am writing a wrapper so I need to put the actual type the C library is using (was compiled using), otherwise any struct using PQ_64BIT might be the wrong size and disastrous things might happen.
> > 
> > Am I understanding this correctly?
> > 
> > It seems that I can probably get away with swapping PQ_64BIT for long in the short term, as I suspect my library was compiled with a 64bit type, not the BIGNUM struct...
> > 
> > Thoughts?
> 
> Having a very quick poke around the OpenSSL docs, it looks like PQ_64BIT is designed to be used as an opaque type.  The problem, of course, is that you don't know how the library was compiled.
> 
> It looks like the only safe way to do it is to mimic the C headers. Something like this:
> 
> version( SIXTY_FOUR_BIT )
> {
>     typedef ulong PQ_64BIT; // Note that I think OpenSSL uses ulongs!
>     void pq_64bit_init(ref PQ_64BIT x) {}
>     void pq_64bit_add_word(ref PQ_64BIT x, ref PQ_64BIT w) {x += w}
> }
> else
> {
>     typedef BIGNUM PQ_64BIT;
>     void pq_64bit_init(ref PQ_64BIT x) { BN_INIT(&x); }
>     void pq_64bit_add_word(ref PQ_64BIT x, ref PQ_64BIT w) {
> BN_add_word(&x, &w); }
> }
> 
> And so on.  Ugly as hell, but that's how OpenSSL is doing things.
> 
> Hope that's of some help :P

I think you're right.  Ick.

Regan

Hi, some time ago I already used bcd (http://www.dsource.org/projects/bcd/) to successfully convert the openssl headers to d.

It's working. The only problem is that bcd doesn't convert the macros from the .h files, so that I had to manually resolve them:
in C: SSL_CTX_set_options(ctx, SSL_OP_ALL);
in D: SSL_CTX_ctrl((ctx),SSL_CTRL_OPTIONS,(SSL_OP_ALL),null);

If you didn't know bcd then you should check it out, since
it might save you a lot of work. I know the openssl headers are very extensive.

Best regards,
Daniel

Daniel Keep Wrote:
> torhu wrote:
> > How's the BIGNUM struct defined?
> 
> As far as I can tell, the OpenSSL docs don't specify; they state that BIGNUM should be treated as an opaque type, and that you must never attempt to directly access its members.
> 
> That practically screams of "BIGNUM's implementation is platform-dependant" :P

Yep, it looks like this:

struct bignum_st
	{
	BN_ULONG *d;	/* Pointer to an array of 'BN_BITS2' bit chunks. */
	int top;	/* Index of last used d +1. */
	/* The next are internal book keeping for bn_expand. */
	int dmax;	/* Size of the d array. */
	int neg;	/* one if the number is negative */
	int flags;
	};

which answers the question "is it 64 bits", the answer is "nope".

> Actually, PQ_64BIT can be aliased to BIGNUM which can be an *arbitrary* number of bits; it's a dynamically allocated structure.

Yep, that's the problem as I see it.

> However, if his version is compiled with 64-bit integer support, and he doesn't mind not supporting BIGNUM versions, he could just stick to ulong.  Maybe just stub out the other side of the version path with some static assert(false)'s. :)

This is what I did, to progress with other parts in the meantime.

However, I think I may need to re-think my strategy.

I started with ssl.h, the top level C include file.  Created an ssl.d copying/converting everything, then compiled with a main.d which imported openssl.ssl;

This kicked up all the undefined symbols (handled by includes in the original source) I tracked the first of these and converted that file, however, each new source includes more undefined symbols and I end up in the murky depths of openssl.

The problem with my method is that of course I am going to end up converting the whole thing one file at a time.

It just strikes me that I should be able to get away with converting less, just the stuff a library end-user would actually use.  Maybe I should start by identifying that.. hmm.

Regan

Daniel919 Wrote:
> Hi, some time ago I already used bcd (http://www.dsource.org/projects/bcd/) to successfully convert the openssl headers to d.
> 
> It's working. The only problem is that bcd doesn't convert the macros
> from the .h files, so that I had to manually resolve them:
> in C: SSL_CTX_set_options(ctx, SSL_OP_ALL);
> in D: SSL_CTX_ctrl((ctx),SSL_CTRL_OPTIONS,(SSL_OP_ALL),null);
> 
> If you didn't know bcd then you should check it out, since
> it might save you a lot of work. I know the openssl headers are very
> extensive.

Good point, this might be a better starting point.  Yep, the openssl headers are chock full of macros.  How long did it take you to resolve/replace them all with function calls?

What version of openssl did you convert/wrap?  Perhaps it would be easier to bring those up to date?

Regan