June 30, 2010
Walter Bright Wrote:

> Adam Ruppe wrote:
> > My network thing is very simple: it opens a socket, then wraps it up in a File struct, via FILE*. Then, you can treat it the same way.
> 
> That's the traditional way to do it, but I'm not so sure it's the right way for the future. Wouldn't it be better to have an interface to it that is a range, rather than pretend it's a FILE* ?

And in either case, if the buffer is empty a get() operation will block, correct?
June 30, 2010
Sean Kelly wrote:
> Walter Bright Wrote:
> 
>> Adam Ruppe wrote:
>>> My network thing is very simple: it opens a socket, then wraps it up
>>> in a File struct, via FILE*. Then, you can treat it the same way.
>> That's the traditional way to do it, but I'm not so sure it's the right way for the future. Wouldn't it be better to have an interface to it that is a range, rather than pretend it's a FILE* ?
> 
> And in either case, if the buffer is empty a get() operation will block, correct?

I'm no expert in network programming, but yes. I suspect there should be a settable timeout which should throw, too.
June 30, 2010
Walter Bright Wrote:

> Sean Kelly wrote:
> > Walter Bright Wrote:
> > 
> >> Adam Ruppe wrote:
> >>> My network thing is very simple: it opens a socket, then wraps it up in a File struct, via FILE*. Then, you can treat it the same way.
> >> That's the traditional way to do it, but I'm not so sure it's the right way for the future. Wouldn't it be better to have an interface to it that is a range, rather than pretend it's a FILE* ?
> > 
> > And in either case, if the buffer is empty a get() operation will block, correct?
> 
> I'm no expert in network programming, but yes. I suspect there should be a settable timeout which should throw, too.

I've been thinking about this and I think this is probably what the majority of apps want anyway.  It isn't scalable, but server apps are a whole 'nother ball of wax.
June 30, 2010
Robert Jacques wrote:
> Great, but if binary are excluded from the libcurl license, then binaries don't have any license and are unusable.

That is not my understanding. The binary is not excluded from the license, it is just excluded from the attribution requirement of the license.

> Besides, legally I don't think he can change the interpretation of his license without changing its text. I'd recommend kindly asking him to grant you/D the right to use libcurl under the BOOST license, for legal reasons.

Legally, he has every right to change and clarify his license, and since the email is after his license, it supercedes it. I believe his email to me would suffice for any dispute. IANAL. But I agree that changing the license to Boost would be better.
June 30, 2010
Walter Bright wrote:
> Adam Ruppe wrote:
>> My network thing is very simple: it opens a socket, then wraps it up
>> in a File struct, via FILE*. Then, you can treat it the same way.
> 
> That's the traditional way to do it, but I'm not so sure it's the right way for the future. Wouldn't it be better to have an interface to it that is a range, rather than pretend it's a FILE* ?

I initially also thought that a file (or socket etc.) should be a range, but then I realized that that would overload roles. It's best to have a handle/ranges architecture in which the handle (e.g. File) is responsible for opening, closing, and managing the connection, and several ranges are responsible for fetching data in various ways (by character, by chunk, by line etc.)

BTW I'm virtually done implemented readf. I only need to parse floating-point numbers and strtod won't work. Walter, could you please email me your strtod implementation? Thanks.

The current issue with readf (and other similar formatted read routines) is that they require a primitive peek() that looks up the current character in a stream without swallowing it. This is not a FILE* primitive, but can be simulated (slow) by using getc() and ungetc(). Fortunately on GNU's I/O libs peek() is easy to define (actually they have an internal routine by that name), and on Windows dmd uses Walter's I/O library, which again has fast peek().


Andrei
June 30, 2010
Sean Kelly wrote:
> Walter Bright Wrote:
> 
>> Sean Kelly wrote:
>>> Walter Bright Wrote:
>>>
>>>> Adam Ruppe wrote:
>>>>> My network thing is very simple: it opens a socket, then wraps it up
>>>>> in a File struct, via FILE*. Then, you can treat it the same way.
>>>> That's the traditional way to do it, but I'm not so sure it's the right way for the future. Wouldn't it be better to have an interface to it that is a range, rather than pretend it's a FILE* ?
>>> And in either case, if the buffer is empty a get() operation will block, correct?
>> I'm no expert in network programming, but yes. I suspect there should be a settable timeout which should throw, too.
> 
> I've been thinking about this and I think this is probably what the majority of apps want anyway.  It isn't scalable, but server apps are a whole 'nother ball of wax.

To obtain asynchronous operation, an app can spawn a secondary thread using blocking I/O and passing stuff as messages. Indeed defining many secondary threads does become a scalability issue.

Andrei
June 30, 2010
Andrei Alexandrescu Wrote:

> Sean Kelly wrote:
> > Walter Bright Wrote:
> > 
> >> Sean Kelly wrote:
> >>> Walter Bright Wrote:
> >>>
> >>>> Adam Ruppe wrote:
> >>>>> My network thing is very simple: it opens a socket, then wraps it up in a File struct, via FILE*. Then, you can treat it the same way.
> >>>> That's the traditional way to do it, but I'm not so sure it's the right way for the future. Wouldn't it be better to have an interface to it that is a range, rather than pretend it's a FILE* ?
> >>> And in either case, if the buffer is empty a get() operation will block, correct?
> >> I'm no expert in network programming, but yes. I suspect there should be a settable timeout which should throw, too.
> > 
> > I've been thinking about this and I think this is probably what the majority of apps want anyway.  It isn't scalable, but server apps are a whole 'nother ball of wax.
> 
> To obtain asynchronous operation, an app can spawn a secondary thread using blocking I/O and passing stuff as messages. Indeed defining many secondary threads does become a scalability issue.

Yeah, this is where Java was 10 years ago.  It's an easy model to program for but scales terribly, assuming you're talking about kernel threads.  If the threads are replaced with fibers and context-switching takes place behind the scenes when a read or write is issued it's actually a pretty cool programming model that should scale quite well.  So I suppose it's not a bad model to expose to users, since we'll eventually be 64-bit and fibers will probably be used by spawn() at some point.
June 30, 2010
On 6/30/10, Sean Kelly <sean@invisibleduck.org> wrote:
> I've been thinking about this and I think this is probably what the majority of apps want anyway.  It isn't scalable, but server apps are a whole 'nother ball of wax.

Blocking calls are convenient for simple apps, since you just call the read and write functions and don't worry about the packets.

For servers, they are still pretty useful. You can use the select() call on unix to wait for any one of a set of connections to be ready for you, and when it is, you then call the same blocking read/write functions. Since  you know ahead of time that they are ready, it doesn't actually wait.

I imagine you could do the same with threads, but I've never actually tried it.

Does this scale well? Honestly, I don't know. Every network server I've ever written has only had a handful of concurrent users anyway.
June 30, 2010
Sean Kelly wrote:
> So I suppose it's not a bad model to
> expose to users, since we'll eventually be 64-bit and fibers will probably be
> used by spawn() at some point.

The 64 bit address space eliminates the stack size problems with threads. It's also why I think supporting stack chaining is a waste of time.
July 01, 2010
On Wed, 30 Jun 2010 13:13:33 -0400, Andrei Alexandrescu <SeeWebsiteForEmail@erdani.org> wrote:

> Walter Bright wrote:
>> Adam Ruppe wrote:
>>> My network thing is very simple: it opens a socket, then wraps it up
>>> in a File struct, via FILE*. Then, you can treat it the same way.
>>  That's the traditional way to do it, but I'm not so sure it's the right way for the future. Wouldn't it be better to have an interface to it that is a range, rather than pretend it's a FILE* ?
>
> I initially also thought that a file (or socket etc.) should be a range, but then I realized that that would overload roles. It's best to have a handle/ranges architecture in which the handle (e.g. File) is responsible for opening, closing, and managing the connection, and several ranges are responsible for fetching data in various ways (by character, by chunk, by line etc.)
>
> BTW I'm virtually done implemented readf. I only need to parse floating-point numbers and strtod won't work. Walter, could you please email me your strtod implementation? Thanks.
>
> The current issue with readf (and other similar formatted read routines) is that they require a primitive peek() that looks up the current character in a stream without swallowing it. This is not a FILE* primitive, but can be simulated (slow) by using getc() and ungetc(). Fortunately on GNU's I/O libs peek() is easy to define (actually they have an internal routine by that name), and on Windows dmd uses Walter's I/O library, which again has fast peek().

I really think D needs to replace FILE * with it's own buffering scheme.  That way we can control the underlying buffering and have access to it.  We can also take advantage of D features that aren't available in the underlying code, such as thread local storage to avoid taking global locks.

-Steve