Hi,

Has anyone used D to work with arbitrary length bitfields with multiple occurences of a sub-bitfield. I am working with DVB Sections and EIT packets are defined as bitfields with loops in them and the header is 112 bits. The loops are handleable with subfields obviously, assuming you can work out how the bigendian works on the byte sequence.

As far as I can see std.bitmanip only caters for 8, 16, 32, and 64 bit long bitfields.

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Tuesday, 21 May 2019 at 17:16:05 UTC, Russel Winder wrote:
> Hi,
>
> Has anyone used D to work with arbitrary length bitfields with multiple occurences of a sub-bitfield. I am working with DVB Sections and EIT packets are defined as bitfields with loops in them and the header is 112 bits. The loops are handleable with subfields obviously, assuming you can work out how the bigendian works on the byte sequence.
>
> As far as I can see std.bitmanip only caters for 8, 16, 32, and 64 bit long bitfields.

Never used it myself, but BitArray with careful handling of endianess might fit your task.

https://dlang.org/phobos/std_bitmanip.html#.BitArray.this.2
https://dlang.org/phobos/std_bitmanip.html#.peek

On Tuesday, 21 May 2019 at 17:16:05 UTC, Russel Winder wrote:
> As far as I can see std.bitmanip only caters for 8, 16, 32, and 64 bit long bitfields.

 I worked on/with bitfields in the past, the limit sizes is more or less for natural int types that D supports.

 However this limitation is kinda arbitrary, as for simplicity it relies on shifting bits, going larger or any byte size is possible depending on what needs to be stored, but ti's the speed that really takes a penalty when you aren't using native types or you have to do a lot of shifting to get the job done.

 What's the layout of what you need? I'll see if i can't make something that would work for you.

 Would be better if you can use a object that breaks the parts down and you can actually fully access those parts, then just re-store it into the limited space you want for storage, which then would be faster than bitfields (although not by much)

On Tue, 2019-05-21 at 18:22 +0000, Boris-Barboris via Digitalmars-d-learn
wrote:
[…]
> 
> Never used it myself, but BitArray with careful handling of endianess might fit your task.
> 
> https://dlang.org/phobos/std_bitmanip.html#.BitArray.this.2 https://dlang.org/phobos/std_bitmanip.html#.peek

I'll have to mull that one over. The incoming data is a sequence of bytes that is treated as a bitfield. Although the standard says "big-endian" it is isn't entirely clear how this relates to the bitfields, I guess I need to read the standard more. :-(

I guess the question is whether BitArray can work with bytes rather than size_t as elements of the backing array.

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Tue, 2019-05-21 at 19:14 +0000, Era Scarecrow via Digitalmars-d-learn wrote:
> 
[…]
>   I worked on/with bitfields in the past, the limit sizes is more
> or less for natural int types that D supports.

Rust bitfield crate and it's macros are the same, the underlying type for a bitfield must be a primitive integer type. Fortunately, Rust has i128 and u128 which is enough for my 112 bit EIT header.

Boris Barboris suggested using BitArray and I willinvestigate but the size_t/byte problem would need to go away.

>   However this limitation is kinda arbitrary, as for simplicity it
> relies on shifting bits, going larger or any byte size is
> possible depending on what needs to be stored, but ti's the speed
> that really takes a penalty when you aren't using native types or
> you have to do a lot of shifting to get the job done.
> 
>   What's the layout of what you need? I'll see if i can't make
> something that would work for you.
> 
>   Would be better if you can use a object that breaks the parts
> down and you can actually fully access those parts, then just
> re-store it into the limited space you want for storage, which
> then would be faster than bitfields (although not by much)

I found an interesting way forward in the source code of dvbsnoop. It basically uses the byte sequence as a backing store and then has a function to do the necessary accesses to treat it as a bit array.

If D's Bit Array can work with bytes instead of size_t then it is exactly what dvbsnoop does (in C) but adds writing as well as reading.

The Rust solution using bitfield with a u128 backing it seems to work, but it is all very clumsy even if it is efficacious.

-- 
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk

On Wednesday, 22 May 2019 at 08:54:45 UTC, Russel Winder wrote:
> On Tue, 2019-05-21 at 19:14 +0000, Era Scarecrow via Digitalmars-d-learn wrote:
>> 
> […]
>>   I worked on/with bitfields in the past, the limit sizes is more
>> or less for natural int types that D supports.
>
> Rust bitfield crate and it's macros are the same, the underlying type for a bitfield must be a primitive integer type. Fortunately, Rust has i128 and u128 which is enough for my 112 bit EIT header.
>
> Boris Barboris suggested using BitArray and I willinvestigate but the size_t/byte problem would need to go away.
>
>>   However this limitation is kinda arbitrary, as for simplicity it
>> relies on shifting bits, going larger or any byte size is
>> possible depending on what needs to be stored, but ti's the speed
>> that really takes a penalty when you aren't using native types or
>> you have to do a lot of shifting to get the job done.
>> 
>>   What's the layout of what you need? I'll see if i can't make
>> something that would work for you.
>> 
>>   Would be better if you can use a object that breaks the parts
>> down and you can actually fully access those parts, then just
>> re-store it into the limited space you want for storage, which
>> then would be faster than bitfields (although not by much)
>
> I found an interesting way forward in the source code of dvbsnoop. It basically uses the byte sequence as a backing store and then has a function to do the necessary accesses to treat it as a bit array.
>
> If D's Bit Array can work with bytes instead of size_t then it is exactly what dvbsnoop does (in C) but adds writing as well as reading.
>
> The Rust solution using bitfield with a u128 backing it seems to work, but it is all very clumsy even if it is efficacious.

If the type of operations you need on your 128 bit container is simple enough (bitwise op) you can implement them, backed by two ulong in a custom struct.
I did something similar for the tokens if my language[1]. There are probably wrong stuff in there but for my usage (include and bt) that works seamlessly.

[1]: https://github.com/Basile-z/styx/blob/master/src/styx/data_structures.d#L40