Are there any good examples of using std.bitarray around?

(And, is my memiry deceiving me or did D have a Bit type once upon a time?)
-- 

Me Here wrote:

> (And, is my memiry deceiving me or did D have a Bit type once upon a time?)

The "bool" type was originally called "bit". It is still aliased even.

--anders

Me Here wrote:
> Are there any good examples of using std.bitarray around?

The garbage collector uses it.

> (And, is my memiry deceiving me or did D have a Bit type once upon a time?)

Yes, but it was dropped.

Walter Bright wrote:

> Me Here wrote:
> > Are there any good examples of using std.bitarray around?
> 
> The garbage collector uses it.
> 
> > (And, is my memiry deceiving me or did D have a Bit type once upon a time?)
> 
> Yes, but it was dropped.

Thanks. I got sidetracked away from this. Now I've got back to it I'm still having trouble working out if I can use std.bitarray for my purposes. I need to decompose an unsigned 16-bit value into 8 x 2-bit integers. I C I'd use bitfields. I know and understand why these are not a part of D, but that doesn't help me solve my problem. The data, 900+MB of it is delivered in this packed format and I need to unpack it.

Can I use std.bitarray to extract these 2-bit numbers? If so, a cluebat as to how would be useful.

Thanks, b.
-- 

On 09/05/2008, Me Here <p9e883002@sneakemail.com> wrote:
> Walter Bright wrote:
>
>  > Me Here wrote:
>  > > Are there any good examples of using std.bitarray around?
>  >
>  > The garbage collector uses it.
>  >
>  > > (And, is my memiry deceiving me or did D have a Bit type once upon a time?)
>  >
>  > Yes, but it was dropped.
>
> Thanks. I got sidetracked away from this. Now I've got back to it I'm still
>  having trouble working out if I can use std.bitarray for my purposes. I need to
>  decompose an unsigned 16-bit value into 8 x 2-bit integers. I C I'd use
>  bitfields. I know and understand why these are not a part of D, but that
>  doesn't help me solve my problem. The data, 900+MB of it is delivered in this
>  packed format and I need to unpack it.
>
>  Can I use std.bitarray to extract these 2-bit numbers? If so, a cluebat as to
>  how would be useful.

std.bitarray is the wrong module to use. It's being deprecated anyway.

What you want is std.bitmanip.

    import std.bitmanip;

    struct A
    {
        ushort n; // unsigned 16-bit number
        mixin(bitfields!(
           uint, "a", 2,
           uint, "b", 2,
           uint, "c", 2,
           uint, "d", 2,
           uint, "e", 2,
           uint, "f", 2,
           uint, "g", 2,
           uint, "h", 2));
    }

Now you've got bitfields, just like in C. Given

    A x;

assigning x.n assigns the whole 16 bit number, while reading and writing x.a to x.h will read or write the eight individual two-bit fields.

The documentation for std.bitarray consists in entirety of the sentence: "Scheduled for deprecation. Use std.bitmanip instead", so I'm surprised you didn't go there yourself.

On 10/05/2008, Janice Caron <caron800@googlemail.com> wrote:
>  assigning x.n assigns the whole 16 bit number, while reading and
>  writing x.a to x.h will read or write the eight individual two-bit
>  fields.

Actually, I got that wrong. The example I gave is equivalent to C's:

    struct A
    {
        unsigned short n;
        unsigned int a : 2;
        unsigned int b : 2;
        unsigned int c : 2;
        unsigned int d : 2;
        unsigned int e : 2;
        unsigned int f : 2;
        unsigned int g : 2;
        unsigned int h : 2;
     }

I inadvertantly gave the impression that n overlaps a to h. It doesn't - it's a separate variable. If you wanted n and a to h to overlap, you'd have to use a struct inside a union.

Apologies for confusion. But anyway, std.bitmanip is what you want.

Janice Caron wrote:


> The documentation for std.bitarray consists in entirety of the sentence: "Scheduled for deprecation. Use std.bitmanip instead", so I'm surprised you didn't go there yourself.

Really? Which documentation are you reading because the documentation on my harddrive that D1/Phobos starts ands with:

std.bitarray
struct BitArray;
An array of bits.

....

BitArray opCat(bool b); BitArray opCat_r(bool b); BitArray opCat(BitArray b); Support for binary operator ~ for bit arrays.

And there is no std.bitmanip list in the index?

Cheers, b.

-- 

On 10/05/2008, Me Here <p9e883002@sneakemail.com> wrote:
> Really? Which documentation are you reading because the documentation on my
>  harddrive that D1/Phobos

Oops. Using D2.

D2 has bitfields. You might want to consider moving up. :-)

Janice Caron wrote:

> You might want to consider moving up. :-)

Hm. And so we come full circle. I started out with D2, but the performance impact of using invariant strings (for may application) is just to costly.

See http://genome.ucsc.edu/FAQ/FAQformat.html#format7

Essentially, once the compacted (2-bit format has been expanded back to the
(huge) strings of ACGTs, then the nBlocks (count/Starts/sizes) and maskBlocks
(count/Starts/Sizes) describe ranges of those huge strings that need to be
changed to 'N's (nBlocks) or be lower-cased (A->a, C->c etc.).

Breaking these huge strings up into pieces to do these tranformations, and then sticking all the pieces back together, when they are *all* 1 to 1 substitutions, is just ludicrous. The impact of generating all those iddy biddy invariant char[]s from one huge invriant char[] and then sticking them all back together to form another huge invariant char[] and throwing away all the intermediates causes the GC to go into fits.

When the (compressed) input is 900+MB and the expanded output is > 3GB, the performance impact is considerable and inacceptable.

Even once I get around to multi-threading this, the use of invariant char[]s will still be no advantage because I want (*need*) the processing of the strings to operate /in-place/.

I (the programmer) need to control what gets copied when. And to control
concurrent access by sharing ranges of the data (without copying), for
modification /in-place/.

Attempting to isolate the programmer (me) from the concerns of multi-threading by duplicating data over and over isn't an option given the volumes of data.

(Personnally I think is a waste of time and effort anyway. Better to educate the programmer than to try and nanny his use of threads. The effort expended on this would be far better spent on other things--like fixing the GC. But that's not my call)

Cheers, b.
-- 

Janice Caron wrote:

> You might want to consider moving up. :-)

Hm. And so we come full circle. I started out with D2, but the performance impact of using invariant strings (for may application) is just to costly.

See http://genome.ucsc.edu/FAQ/FAQformat.html#format7

Essentially, once the compacted (2-bit format has been expanded back to the
(huge) strings of ACGTs, then the nBlocks (count/Starts/sizes) and maskBlocks
(count/Starts/Sizes) describe ranges of those huge strings that need to be
changed to 'N's (nBlocks) or be lower-cased (A->a, C->c etc.).

Breaking these huge strings up into pieces to do these tranformations, and then sticking all the pieces back together, when they are *all* 1 to 1 substitutions, is just ludicrous. The impact of generating all those iddy biddy invariant char[]s from one huge invriant char[] and then sticking them all back together to form another huge invariant char[] and throwing away all the intermediates causes the GC to go into fits.

When the (compressed) input is 900+MB and the expanded output is > 3GB, the performance impact is considerable and inacceptable.

Even once I get around to multi-threading this, the use of invariant char[]s will still be no advantage because I want (*need*) the processing of the strings to operate /in-place/.

I (the programmer) need to control what gets copied when. And to control
concurrent access by sharing ranges of the data (without copying), for
modification /in-place/.

Attempting to isolate the programmer (me) from the concerns of multi-threading by duplicating data over and over isn't an option given the volumes of data.

(Personnally I think is a waste of time and effort anyway. Better to educate the programmer than to try and nanny his use of threads. The effort expended on this would be far better spent on other things--like fixing the GC. But that's not my call)

Cheers, b.
--