On 03/28/2013 03:24 PM, Vidar Wahlberg wrote:
> To follow up with some new woes I'm currently struggling with:
> I'm storing some various values in an ubyte array. I discovered that
> it's probably std.bitmanip I wish to use in order to "convert" i.e. an
> int to 4 bytes (although I went first to std.conv looking for this
> feature).
> So I have "ubyte[] buffer;", and my second thought is that the "append"
> method (http://dlang.org/phobos/std_bitmanip.html#.append) is what I
> want to append values to my ubyte-array (my first thought was something
> like "buffer ~= to!ubyte[](42);", although then I forgot about
> endianness). In the example in the documentation it does say "auto
> buffer = appender!(const ubyte[])();", with no explanation as of what
> "appender" is (I later learned that this is from std.array), but just
> looking a bit up I see that the "write" method explained just above use
> "ubyte[] buffer; buffer.write!ubyte(42);", so I assumed that I could use
> ubyte[] myself instead of this "appender" which I thought was some
> legacy code.
> So I write some simple test code:
> import std.bitmanip, std.stdio;
> void main() {
> ubyte[] buffer;
> buffer.append!ubyte(42);
> }
> Run it through rdmd, and get:
> "core.exception.AssertError@/usr/include/d/std/array.d(591): Attempting
> to fetch the front of an empty array of ubyte".
> Just to see what happens I set the size of the buffer ("buffer.length =
> 1;") before appending and run it again. Now it runs, but instead of
> appending it behaves like write(), which was not exactly what I wanted.
>
> At this time I google for this "appender" used in the example and learn
> that it comes from std.array, so I import std.array and try again using
> "auto buffer = appender!(ubyte[])();", and surely enough, now it does
> append correctly to the buffer. Great, I have a solution, so I go back
> to my project and implement it like I implemented it in my test code,
> but when I compile my project after this addition I get a new cryptic
> error message: "Error: __overloadset isn't a template".
> After digging a bit I realized that it's because in my project I also
> import std.file, apparently there are some collisions between
> std.bitmanip and std.file. Again it's solvable, but it's yet another
> fight with the language/standard library. I would also assume that it's
> not that uncommon for a module that use std.bitmanip to also use
> std.file, meaning that this error potentially may occur often.
>
> A bit on the side: It seems to me as importing std.bitmanip somehow adds
> new properties to my array (".read()" and ".write()", for example). Not
> necessarily a bad thing, more of "I've not seen this before, I was
> expecting that I were to concatenate the bytes from the conversion to my
> buffer using ~".


Yes, some functions do overload one another's functions...
Usually it's because they are templated, and can accept the same arguments.

using std.file.read will work, or import stdfile = std.file;
stdfile.read;

Just makin sure ya knew how to fix that.

On Thursday, 28 March 2013 at 20:24:50 UTC, Vidar Wahlberg wrote:
> To follow up with some new woes I'm currently struggling with:
> I'm storing some various values in an ubyte array. I discovered that it's probably std.bitmanip I wish to use in order to "convert" i.e. an int to 4 bytes (although I went first to std.conv looking for this feature).
> --snip--

I am completely confused as to why you're doing what you are doing ... std.conv does work (and in the case you've listed, is unnecessary anyway). Try this:

import std.stdio, std.conv;

void main() {
    ubyte[] buffer;
    buffer ~= 5; // Simple solution
    buffer ~= to!ubyte(6); // Proper usage of "to"
    writeln(buffer);
}

---

Now, for performance reasons you might want to use appender (or buffer.reserve(n), if you happen to know how many items you'll be storing), but the simplest thing works.

On Thursday, 28 March 2013 at 20:24:50 UTC, Vidar Wahlberg wrote:
> that it's probably std.bitmanip I wish to use in order to "convert" i.e. an int to 4 bytes (although I went first to std.conv looking for this feature).

Ah, I reread it a couple of times and realized what you mean now.

You want to turn an int like 5 into an array of ubytes like [0,0,0,5].

So, you were on the right track, here's one way on how it's done:

---

import std.stdio, std.bitmanip, std.array;
void main() {
    auto app = appender!(ubyte[])(); // Create an appender of type ubyte[]
    app.append!int(5);
    writeln(app.data());
}

---

Without using appender, it's a bit more complicated:

---
import std.stdio, std.bitmanip;
void main() {
    auto buf = new ubyte[](4);
    buf.append!int(5);
    writeln(buf);
}
---

So, what append is doing is writing into the buffer, which must have enough space to put the int. So, thus, it must have 4 bytes. Guess what happens if we try to store a long in it? Yeah, it'll break on that as well. You'll also notice that subsequent calls to append on that buf will overwrite what's already in there.


It's not obvious, but append needs either an array with enough space to store the element or an output range, like appender. Maybe the documentation could use a little work in this regard.

On Thursday, 28 March 2013 at 21:16:32 UTC, Chris Cain wrote:
> I am completely confused as to why you're doing what you are doing ... std.conv does work (and in the case you've listed, is unnecessary anyway). Try this:
>
> import std.stdio, std.conv;
>
> void main() {
>     ubyte[] buffer;
>     buffer ~= 5; // Simple solution
>     buffer ~= to!ubyte(6); // Proper usage of "to"
>     writeln(buffer);
> }

This is not what I'm trying to achieve.
This gives me an array with two elements, [5, 6]. What I want is to append the 4 bytes that make up one integer value, which using your values means buffer should hold a total of 8 bytes (two integers).

H. S. Teoh answered well on how this can be achieved, although my feedback was not really meant as a question of "how is this done?", more of "why is this done like this, couldn't it be done much easier?".

On Thursday, 28 March 2013 at 20:24:50 UTC, Vidar Wahlberg wrote:
> A bit on the side: It seems to me as importing std.bitmanip somehow adds new properties to my array (".read()" and ".write()", for example). Not necessarily a bad thing, more of "I've not seen this before, I was expecting that I were to concatenate the bytes from the conversion to my buffer using ~".

Sorry about the repeated postings ... I'm trying to read & answer it while also dealing with the norovirus :x

---

import std.stdio : writeln;
void main() {
    ubyte[] array;
    array ~= 5.toUbytes();
    array ~= 6.toUbytes();
    writeln(array);
}

ubyte[T.sizeof] toUbytes(T)(T val) @safe {
    ubyte[T.sizeof] buf;
    std.bitmanip.append!T(buf[], val);
    return buf;
}

---

I see your latest post and see that you don't necessary care how to do it, but, I figured I might as well provide yet another way that's @safe and simple.

On Thursday, 28 March 2013 at 20:24:50 UTC, Vidar Wahlberg wrote:
> Great, I have a solution, so I go back to my project and implement it like I implemented it in my test code, but when I compile my project after this addition I get a new cryptic error message: "Error: __overloadset isn't a template".
> After digging a bit I realized that it's because in my project I also import std.file, apparently there are some collisions between std.bitmanip and std.file. Again it's solvable, but it's yet another fight with the language/standard library. I would also assume that it's not that uncommon for a module that use std.bitmanip to also use std.file, meaning that this error potentially may occur often.

Yeah, that's a bit of an issue. Weird cryptic error. Oh well, as a general rule, try to import only the parts of the module you're actually using. This does two great things: it prevents namespace pollution causing errors like you've seen, and it documents WHERE someone has to look in order to find documentation on a function you're using in your module.

As 1100110 noted, using a fully qualified ID is also a potential solution, especially if you only intend on using it in one place and the line isn't very noisy to begin with (as I showed in my example above).

> "why is this done like this, couldn't it be done much easier?".

Maybe. What needs to be made easier and how would you suggest to fix it? The error message, certainly. Probably the documentation too. But the API itself seems sane to me in this instance, it just needs a better description.

On Thursday, 28 March 2013 at 22:22:49 UTC, Chris Cain wrote:
> ---
> ubyte[T.sizeof] toUbytes(T)(T val) @safe {
>     ubyte[T.sizeof] buf;
>     std.bitmanip.append!T(buf[], val);
>     return buf;
> }
> ---

This should be:

ubyte[T.sizeof] toUbytes(T)(T val) @safe {
    ubyte[T.sizeof] buf;
    import std.bitmanip : append;
    append!T(buf[], val);
    return buf;
}

On Thursday, 28 March 2013 at 22:22:49 UTC, Chris Cain wrote:
> Sorry about the repeated postings ... I'm trying to read & answer it while also dealing with the norovirus :x

Been there. Not amusing, I wish you well.


>> "why is this done like this, couldn't it be done much easier?".
>
> Maybe. What needs to be made easier and how would you suggest to fix it? The error message, certainly. Probably the documentation too. But the API itself seems sane to me in this instance, it just needs a better description.

Well, I'm not so proficient in the language yet that I'm going to climb to the top of Mount Stupid and say how it should be, because for what I know this may be perfectly logical with just me being blind to it, but to try to explain how it would make more sense to me:
Since you got "ubyte[] buffer = [0, 0, 0, 0]; buffer.write!int(42); buffer.read!int();", I think it would be logical that "ubyte[] buffer; buffer.append!int(42); buffer.read!int()" would do pretty much the same (except instead of writing 4 bytes at index 0, it appends 4 bytes to the end of the array, then reads back the value). The latter code does however give you "Attempting to fetch the front of an empty array of ubyte".
I don't get why you need to drag in std.array.appender() for std.bitmanip.append(), when you don't need it for read() and write().

On Friday, March 29, 2013 00:38:25 Vidar Wahlberg wrote:
> I don't get why you need to drag in std.array.appender() for
> std.bitmanip.append(), when you don't need it for read() and
> write().

Because read and write operate in place, and append doesn't. read and write operate on input ranges - read just reads the values from it as it iterates, and write overwrites what's there as it iterates. However, append uses an output range, because it's appending rather than overwriting, and for reasons that I don't understand, when treating an array as an output range, rather than appending (like an output range normally would), the put function (which is how stuff gets written to an output range) overwrites what's in the array rather than appending to it. So, using an empty array with any function operating on output ranges isn't going to work. Maybe there's a good reason for arrays working that way when they're treated as output ranges, but for me, it's a good reason to not use arrays when you need an output range.

In either case, I'd suggest reading this if you want to know more about ranges:

http://ddili.org/ders/d.en/ranges.html

And since D's standard library ranges quite heavily, you're going to need at least a basic understanding of them if you want to use much of it. We really need a good article on ranges on dlang.org, but until we do, that link is probably your best resource for learning about ranges.

- Jonathan M Davis

Definitely need to add some updates to the docs. Long story:

D provides an iterable interface called a Range. There are two base forms, inputRange and outputRange.

Dynamic Arrays have the privilege of being, a inputRange, outputRange, and a container.

An array however doesn't operate as you might expect, especially when using a function called append on it.

An output range consists of the ability call put for Range and Element (defined in std.range) for a dynamic array this means you can assign to front.

ubyte[] buffer;
buffer.append!ubyte(42);

The append function takes an outputRange, if we drill down the call that would be made (ignoring my value isn't correct)

buffer.front = 42;
buffer.popFront();

Thus when using an array as an outputRange it
1) Must have a size (hence the error: "Attempting to fetch the front of an empty array of ubyte")
2) Starts at the beginning (hence the observation: "instead of appending it behaves like write()")
3) Is consumed (You didn't run into this)

That is why arrays are awkward and the example makes use of appender (a more traditional form of an outputRange)

------------

The write function seems a little odd as it uses random access (indexing).

Instead of assigning to front like append does it assigns at index buffer[0]...

------------

The implementation of append is what you will find more in idiomatic D. In fact if the module was written today write wouldn't exist and append would probably have been named write.

-------------
-------------

"Error: __overloadset isn't a template"

That needs fixed, it usually does a better job of specifying conflicting modules across modules.

------------
------------

"It seems to me as importing std.bitmanip somehow adds new properties"

D provides UFCS (Uniform Function Call Syntax). For a given type A, foo(A a) is callable in both foo(a) and a.foo().

(Historical note: UFCS is recent addition, a bug allowed it to work with dynamic arrays like you see in these docs)

On Friday, March 29, 2013 03:11:08 Jesse Phillips wrote:
> The write function seems a little odd as it uses random access
> (indexing).
> 
> Instead of assigning to front like append does it assigns at index buffer[0]...

That's because it's operating on multiple bytes at a time (e.g. writing the 4 bytes of an int at once). Really, it's  written for arrays and was generalized because it could be rather than really having been written for ranges.

> The implementation of append is what you will find more in idiomatic D. In fact if the module was written today write wouldn't exist and append would probably have been named write.

That could be argued for, but write and append do different things and both exist for a reason. Only having append would actually be problematic, as there are cases where you really do need write and not append. And neither of them have been in Phobos for all that long (peek, read, write, and append were added in 2.060).

- Jonathan M Davis