Here's a little snippet of code that interfaces with Scintilla (it works btw.):

File file = File("test.txt", "r");
foreach (ubyte[] buf; file.byChunk(4096))
{
	sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
}

The cast looks ugly, but I *have* to send a copy. Am I doing it right, or can I make this a bit simpler? Otherwise I can use file.byLine, but I think this would be much slower (unless D reads the entire contents at once into memory, but I think it does not).

Actually, I shouldn't even complain. I can't believe how easy it is to open a file, use buffers and send it all to a C interface. This is a 4-liner, the C equivalent spans 20+ lines. Ha!

Hello Andrej,

> Here's a little snippet of code that interfaces with Scintilla (it
> works btw.):
> 
> File file = File("test.txt", "r");
> foreach (ubyte[] buf; file.byChunk(4096))
> {
> sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
> }
> The cast looks ugly, but I *have* to send a copy.

What do those have to do with each other?

> Am I doing it right,

It doesn't seem right to me either, because you are allocating memory in chunks and passing it to C without keeping pointers around. Also what does that function do with them? If it keeps them around, you might be faster to load the whole file in one go and make a single call (fewer allocations, fewer calls, no-extra memory usage, etc.)

> or can I make this a bit simpler? Otherwise I can use file.byLine, but
> I think this would be much slower (unless D reads the entire contents
> at once into memory, but I think it does not).
> 
> Actually, I shouldn't even complain. I can't believe how easy it is to
> open a file, use buffers and send it all to a C interface. This is a
> 4-liner, the C equivalent spans 20+ lines. Ha!
> 
-- 
... <IXOYE><

On Sunday 12 September 2010 18:04:14 Andrej Mitrovic wrote:
> Here's a little snippet of code that interfaces with Scintilla (it works
> btw.):
> 
> File file = File("test.txt", "r");
> foreach (ubyte[] buf; file.byChunk(4096))
> {
> 	sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
> }
> 
> The cast looks ugly, but I *have* to send a copy. Am I doing it right, or can I make this a bit simpler? Otherwise I can use file.byLine, but I think this would be much slower (unless D reads the entire contents at once into memory, but I think it does not).

I'd have to check, but I believe that byLine() reads the whole line into memory but not the whole file.

- Jonathan M Davis

Ah, silly me, disregard that code. I was working around the issue but I didn't have to, here's the correct code:

foreach (ubyte[] buf; file.byChunk(4096))
{
	sendEditor(SCI_ADDTEXT, buf.length, cast(LPARAM)buf.ptr);
}

Now, the explanation. The sendEditor function takes an opcode (uint),
and 2 arguments. It uses message passing to communicate with
scintilla, the two arguments are aliased to uint (which is usually a
length argument but is sometimes ignored) and LPARAM (a pointer's
address as ULONG).

Scintilla converts the address back to a pointer, and passes it (and the first argument) to the relevant function based on the opcode. Here's the header of that function:

SCI_ADDTEXT(int length, const char *s)

The reason my first code was working is because the wrapper has two overloaded sendEditor functions. With one you can pass an opcode, a length and pointer address, with the other you can pass an opcode, a length and a string, and the function makes a copy of the string, adds a null terminator and uses its address and the other arguments to send a message.

The ADDTEXT function creates a copy of the pointed-to contents to it's internal scintilla buffers, so I don't have to keep any pointers around after the call.

I hope I've explained this right (and didn't mess something up). :p


On Mon, Sep 13, 2010 at 3:47 AM, BCS <none@anon.com> wrote:
> Hello Andrej,
>
>> Here's a little snippet of code that interfaces with Scintilla (it
>> works btw.):
>>
>> File file = File("test.txt", "r");
>> foreach (ubyte[] buf; file.byChunk(4096))
>> {
>> sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
>> }
>> The cast looks ugly, but I *have* to send a copy.
>
> What do those have to do with each other?
>
>> Am I doing it right,
>
> It doesn't seem right to me either, because you are allocating memory in chunks and passing it to C without keeping pointers around. Also what does that function do with them? If it keeps them around, you might be faster to load the whole file in one go and make a single call (fewer allocations, fewer calls, no-extra memory usage, etc.)
>
>> or can I make this a bit simpler? Otherwise I can use file.byLine, but I think this would be much slower (unless D reads the entire contents at once into memory, but I think it does not).
>>
>> Actually, I shouldn't even complain. I can't believe how easy it is to open a file, use buffers and send it all to a C interface. This is a 4-liner, the C equivalent spans 20+ lines. Ha!
>>
> --
> ... <IXOYE><
>
>
>
>

I was refering to the open method, or when calling the constructor of Text. But it doesn't actually *read* the contents until I ask it to (which is a good thing). In other news it's late and I'm talking crazy tonight, sorry. :)

On Mon, Sep 13, 2010 at 4:01 AM, Jonathan M Davis <jmdavisprog@gmail.com> wrote:
> On Sunday 12 September 2010 18:04:14 Andrej Mitrovic wrote:
>> Here's a little snippet of code that interfaces with Scintilla (it works
>> btw.):
>>
>> File file = File("test.txt", "r");
>> foreach (ubyte[] buf; file.byChunk(4096))
>> {
>>       sendEditor(SCI_ADDTEXT, buf.length, (cast(char[])buf).idup);
>> }
>>
>> The cast looks ugly, but I *have* to send a copy. Am I doing it right, or can I make this a bit simpler? Otherwise I can use file.byLine, but I think this would be much slower (unless D reads the entire contents at once into memory, but I think it does not).
>
> I'd have to check, but I believe that byLine() reads the whole line into memory but not the whole file.
>
> - Jonathan M Davis
>

On Sunday 12 September 2010 19:31:23 Andrej Mitrovic wrote:
> I was refering to the open method, or when calling the constructor of Text. But it doesn't actually *read* the contents until I ask it to (which is a good thing). In other news it's late and I'm talking crazy tonight, sorry. :)

Well, you do appear to be 9 hours ahead of me, time zone-wise, and you do appear to be up rather late.

Entertainingly, the time difference makes for interesting timestamps in the messages as our respective clients put in the local time that a message is sent:

> On Mon, Sep 13, 2010 at 4:01 AM, Jonathan M Davis <jmdavisprog@gmail.com>
wrote:
> > On Sunday 12 September 2010 18:04:14 Andrej Mitrovic wrote:

Mine also appears to be in 24 time, while yours is in 12 hour time, and mine seems to want to put second-precision, while yours only cares about minute- precision. The Date format is a fair bit different as well. Not that it really matters, but it's interesting nonetheless.

- Jonathan M Davis

I prefer 24-hour timestamps, but I've no idea where to set this in Gmail. And I'm not using a special newsgroup reader because they're all clumsy in their own little ways.

On Mon, Sep 13, 2010 at 4:46 AM, Jonathan M Davis <jmdavisprog@gmail.com> wrote:
> On Sunday 12 September 2010 19:31:23 Andrej Mitrovic wrote:
>> I was refering to the open method, or when calling the constructor of Text. But it doesn't actually *read* the contents until I ask it to (which is a good thing). In other news it's late and I'm talking crazy tonight, sorry. :)
>
> Well, you do appear to be 9 hours ahead of me, time zone-wise, and you do appear to be up rather late.
>
> Entertainingly, the time difference makes for interesting timestamps in the messages as our respective clients put in the local time that a message is sent:
>
>> On Mon, Sep 13, 2010 at 4:01 AM, Jonathan M Davis <jmdavisprog@gmail.com>
> wrote:
>> > On Sunday 12 September 2010 18:04:14 Andrej Mitrovic wrote:
>
> Mine also appears to be in 24 time, while yours is in 12 hour time, and mine seems to want to put second-precision, while yours only cares about minute- precision. The Date format is a fair bit different as well. Not that it really matters, but it's interesting nonetheless.
>
> - Jonathan M Davis
>

On Sunday 12 September 2010 19:56:28 Andrej Mitrovic wrote:
> I prefer 24-hour timestamps, but I've no idea where to set this in Gmail. And I'm not using a special newsgroup reader because they're all clumsy in their own little ways.

I'm using a gmail account, but I'm using kmail to manage my mail. So, the timestamps and whatnot are going to depend on my system settings. I like knode (KDE's newsgroup reader) overall, but I stopped using it because I had no way of syncing between machines without copying all of knode's files over every time, which is not pleasant. Using gmail/kmail with IMAP allows for keeping things in sync. Unfortunately, since gmail never sends you messages from the list that you sent to it, it doesn't really work properly to send messages which reply to your own messages, and kmail threads replies to my messages poorly (usually associating them with the message that I responded to). So, I'd prefer to use knode, but the lack of syncing was too painful. I suppose that it would work better if I used a non-gmail account though. I wouldn't know what to use though, since I'd want free IMAP support. And another provider would use proper folders rather than the annoying labels which mail programs like kmail don't deal with quite properly. So maybe I should think of finding a better e-mail provider than gmail...

- Jonathan M Davis

Andrej Mitrovic:

> foreach (ubyte[] buf; file.byChunk(4096))
> {
> 	sendEditor(SCI_ADDTEXT, buf.length, cast(LPARAM)buf.ptr);
> }
> ...
> SCI_ADDTEXT(int length, const char *s)

Keep in mind that the length of a D array is a size_t, this means a 32 or 64 bit long unsigned word.


> The ADDTEXT function creates a copy of the pointed-to contents to it's internal scintilla buffers, so I don't have to keep any pointers around after the call.

In byChunk() the content of buffer is reused across calls, so you are not wasting allocations. I don't know if it's possible to use a fixed-size char[4096] array to remove the first memory allocation too. I think byChunk() needs a second optional argument, to give it a preallocated buffer (like a slice of a fixed-size array).

Bye,
bearophile

> I think byChunk() needs a second optional argument, to give it a preallocated buffer (like a slice of a fixed-size array).

http://d.puremagic.com/issues/show_bug.cgi?id=4859