View mode: basic / threaded / horizontal-split · Log in · Help
November 19, 2012
Is there interest in a std.http?
I'd like to see an HTTP module in Phobos, but I wanted to gauge 
interest first and see if this has been discussed before.

An example of how I would like to interface with it (for creating 
a server):

    interface HTTPHandler {
        void serveHTTP(Request, Response);
    }

    class CustomHandler : HTTPHandler {
        ....
        void serveHTTP(Request req, Response res) {
        }
    }

    auto handler = new CustomHandler(); // implements http.Handler
    auto server = new HttpServer(handler);
    server.listen("0.0.0.0", 80);

As long as serveHttp() is thread safe, the http server could be 
concurrent or evented (with libev or similar) and the custom 
handler code wouldn't have to change.

I'm willing to put in the lion's share of the work (I've already 
written a bunch of it), but I'd naturally like to get some 
community input so I don't go in a completely wrong direction. 
I'm thinking of emulating Go's http library 
(http://golang.org/pkg/net/http), but of course D style.

I think the following are necessary:

* Access to underlying TcpSocket (for protocol upgrades, like 
WebSocket)
* HTTP body is a stream
* Simple HTTP requests
* Errors can be recovered from (allow user-defined error 
handlers):
* User settable size limits
  * Size of request line (1024 bytes by default)
  * Size of each header (1024 bytes by default)
  * Total size of header block (4096 bytes by default)

So, basically I'm looking for two things:

1. Interest level: willing to writ code, willing to test, want to 
use
2. Suggestions for the API/behavior
November 19, 2012
Re: Is there interest in a std.http?
On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little 
wrote:
> I'd like to see an HTTP module in Phobos, but I wanted to gauge 
> interest first and see if this has been discussed before.

I've been asked to put my cgi.d in phobos before (which includes 
a http server as well as a cgi, fastcgi, and scgi implementation 
of its common interface), so there's probably some interest.

I haven't put mine in just because I'm not particularly motivated 
to go through the red tape.

The file is cgi.d in here:
https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff

Feel free to take whatever you want from it. There's also an 
extremely minimal http.d file in there, which is a very simple 
http client. (I didn't go far with this though because I tend to 
use curl for most my work anyway.)



The way I did it is with a function:

void handleRequest(Cgi cgi) { }

And then there's a mixin called GenericMain that calls your 
function.

mixin GenericMain!handleRequest;


Depending on how you compile it, the main function constructs the 
cgi object differently. -version=fastcgi means it is plugged into 
a FastCGI loop. version=scgi ditto. -version=embedded_httpd 
includes a simple little threaded http server. No version builds 
it as a standard CGI binary, one process per request. However it 
is compiled though, the Cgi methods all remain the same, so your 
function shouldn't have to change.


> * Access to underlying TcpSocket (for protocol upgrades, like 
> WebSocket)

My cgi.d I don't think can do this yet. I've been thinking about 
adding it though, but haven't.

> * HTTP body is a stream

class Cgi has a method onRequestBodyDataReceived and one 
handleIncomingDataChunk which can be overridden by subclasses to 
do this kind of thing.

It's a bit of a pain though, I made it thinking you'd want to 
handle it after it is all loaded.

> * Simple HTTP requests

This is where I think my cgi.d is a winner because most the data 
is available so easily

cgi.get["foo"] returns a string
cgi.getArray["foo"] returns an array of strings

ditto for post

cgi.request("foo", 0) returns with a default value, converting 
types as requested

There's also members like cookie, queryString, host, etc., that 
gives nice access to other headers.

> * Errors can be recovered from (allow user-defined error 
> handlers):
> * User settable size limits

eh, I'm partially there on these. There's some constructor args 
you can play with and some degree of exception catching but 
probably not fully what you have in mind.
November 19, 2012
Re: Is there interest in a std.http?
> I've been asked to put my cgi.d in phobos before (which 
> includes a http server as well as a cgi, fastcgi, and scgi 
> implementation of its common interface), so there's probably 
> some interest.
>
> I haven't put mine in just because I'm not particularly 
> motivated to go through the red tape.
>
> The file is cgi.d in here:
> https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff

Awesome. I assume this hasn't gone through rigorous testing, but 
has it been used in production?

>> * Errors can be recovered from (allow user-defined error 
>> handlers):
>> * User settable size limits
>
> eh, I'm partially there on these. There's some constructor args 
> you can play with and some degree of exception catching but 
> probably not fully what you have in mind.

I'm thinking of something a little more sophisticated than 
exceptions. Exceptions unwind the stack, so you can't just 
continue where you left off.

How do you feel about function pointer callbacks? The http parser 
in node.js (https://github.com/joyent/http-parser) takes a struct 
of function pointers to handle errors/events. When the parser 
encounters a recoverable error (max header length reached), the 
programmer could opt to ignore it and continue the parse. 
Unrecoverable errors could throw exceptions (like trying to parse 
garbage data).

If an error is recovered, it would just continue as if nothing 
happened.

Of course, implementing this would increase complexity in the 
parser, but I think it's justified. Thoughts?
November 20, 2012
Re: Is there interest in a std.http?
Before you spend too much time :
https://github.com/rejectedsoftware/vibe.d/tree/master/source/vibe/http



On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little 
wrote:
> I'd like to see an HTTP module in Phobos, but I wanted to gauge 
> interest first and see if this has been discussed before.
>
> An example of how I would like to interface with it (for 
> creating a server):
>
>     interface HTTPHandler {
>         void serveHTTP(Request, Response);
>     }
>
>     class CustomHandler : HTTPHandler {
>         ....
>         void serveHTTP(Request req, Response res) {
>         }
>     }
>
>     auto handler = new CustomHandler(); // implements 
> http.Handler
>     auto server = new HttpServer(handler);
>     server.listen("0.0.0.0", 80);
>
> As long as serveHttp() is thread safe, the http server could be 
> concurrent or evented (with libev or similar) and the custom 
> handler code wouldn't have to change.
>
> I'm willing to put in the lion's share of the work (I've 
> already written a bunch of it), but I'd naturally like to get 
> some community input so I don't go in a completely wrong 
> direction. I'm thinking of emulating Go's http library 
> (http://golang.org/pkg/net/http), but of course D style.
>
> I think the following are necessary:
>
> * Access to underlying TcpSocket (for protocol upgrades, like 
> WebSocket)
> * HTTP body is a stream
> * Simple HTTP requests
> * Errors can be recovered from (allow user-defined error 
> handlers):
> * User settable size limits
>   * Size of request line (1024 bytes by default)
>   * Size of each header (1024 bytes by default)
>   * Total size of header block (4096 bytes by default)
>
> So, basically I'm looking for two things:
>
> 1. Interest level: willing to writ code, willing to test, want 
> to use
> 2. Suggestions for the API/behavior
November 20, 2012
Re: Is there interest in a std.http?
On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little 
wrote:
> * HTTP body is a stream

No Content-Size, no multiple requests per connection (unless you 
use chunked encoding?).

> * User settable size limits
>   * Size of request line (1024 bytes by default)
>   * Size of each header (1024 bytes by default)
>   * Total size of header block (4096 bytes by default)

I think this is pointless. Use an appender and the RFC limits. 
Unless you are serving VERY simple pages, the cost of a few 
allocations for handling the HTTP protocol's overhead will not be 
noticeable compared to the application's. (Slowloris is something 
to keep in mind, though.)

My HTTP library, which is used for this forum and several other 
"production" projects, is here: https://github.com/CyberShadow/ae
The main problem with getting it into Phobos is that it's tied to 
a lot of other related code, such as the asynchronous sockets 
module, the unmanaged memory wrapper type, etc.

Also, what about vibe.d?
November 20, 2012
Re: Is there interest in a std.http?
On Monday, 19 November 2012 at 23:57:35 UTC, Tyler Jameson Little 
wrote:
> Awesome. I assume this hasn't gone through rigorous testing, 
> but has it been used in production?

I've been using it for almost all my work apps for the last... I 
think three years now. None of them so far have huge userbases, 
but it has been in steady use for a while.

The others in this thread have pointed out the other options that 
have some users: Vladimir's library that runs the forum, and 
vibe.d that seems to be the most popular. I haven't used either 
of them though.

> How do you feel about function pointer callbacks?

We could also do them as virtual functions... that's a little 
easier in the setup I have because the constructor arguments are 
messy, whereas subclassing is a lot simpler.

But yeah, it sounds like a decent plan.
November 20, 2012
Re: Is there interest in a std.http?
>> * HTTP body is a stream
>
> No Content-Size, no multiple requests per connection (unless 
> you use chunked encoding?).

Not sure what you mean. I meant incoming stream. There would be a 
request object with access to all other headers, it just wouldn't 
be read in until the user actually wanted it.

>> * User settable size limits
>>  * Size of request line (1024 bytes by default)
>>  * Size of each header (1024 bytes by default)
>>  * Total size of header block (4096 bytes by default)
>
> I think this is pointless. Use an appender and the RFC limits. 
> Unless you are serving VERY simple pages, the cost of a few 
> allocations for handling the HTTP protocol's overhead will not 
> be noticeable compared to the application's. (Slowloris is 
> something to keep in mind, though.)

I just made up those limits, but the point is I'd like the user 
to be able to tweak those. The default will work for most people 
though.

> My HTTP library, which is used for this forum and several other 
> "production" projects, is here: 
> https://github.com/CyberShadow/ae
> The main problem with getting it into Phobos is that it's tied 
> to a lot of other related code, such as the asynchronous 
> sockets module, the unmanaged memory wrapper type, etc.

Cool, I didn't know that this was served by D code. I'll take a 
look at it.

> Also, what about vibe.d?

vibe.d does a lot of things, and it probably does those things 
very well. It seems well maintained, and generally a good 
project. I think that a lot of the things it does well could be 
brought into the standard library. As this thread pointed out, 
there are several HTTP parsers floating around out there. For 
some, vibe.d might not be a perfect fit for whatever reason, but 
everyone can benefit from a simple HTTP library.

Maybe it shouldn't be as high-level as Go's http library, but it 
should at least make writing a simple HTTP server trivial.

Would a minor refactor of vibe.d be acceptable? This is pretty 
much what I'm looking for:

https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/common.d
https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d

Except that I'd remove some parts of it, like JSON parsing.

I think that vibe.d could benefit from moving some of the code 
there into Phobos. I guess it comes down to whether it makes 
sense to make a standard HTTP library.
November 20, 2012
Re: Is there interest in a std.http?
On 11/19/12 3:38 PM, Tyler Jameson Little wrote:
> I'd like to see an HTTP module in Phobos, but I wanted to gauge interest
> first and see if this has been discussed before.

I can say the following. We sorely need a server expert on board with 
the time and inclination to write a good server-side networking 
framework. We have a couple, but for various reasons I have been unable 
to convince them (through newsgroup and private exchanges) to invest 
time in such an endeavor. That's mostly because lack of time - speaking 
for myself, I wish I found the time to learn about all that, which as 
far as I understand gravitates around notions such as asynchronous I/O, 
libevent/libevt, select(), and the such. I've never used them so I'd 
need to start from scratch.

We really need an expert. Look at the Go programming language - it's not 
remarkable, but it benefits of full-time dedication of experts in 
server-side programming, so it has acquired excellent library support 
for networking servers. And they milk that for all it's worth: any 
discussion, article, or blog post about Go gravitates toward the 
five-lines HTTP server with the same implacable reach as conversations 
with ideologists, which inevitably converge towards their ideological 
stronghold.


Andrei
November 20, 2012
Re: Is there interest in a std.http?
On 11/19/12 11:26 PM, Andrei Alexandrescu wrote:
> On 11/19/12 3:38 PM, Tyler Jameson Little wrote:
>> I'd like to see an HTTP module in Phobos, but I wanted to gauge interest
>> first and see if this has been discussed before.
>
> I can say the following. We sorely need a server expert on board with
> the time and inclination to write a good server-side networking
> framework.

To clarify (lest this would be interpreted as excluding vibe.d from the 
category of good server-side networking frameworks): ... for inclusion 
in the standard library.

Andrei
November 20, 2012
Re: Is there interest in a std.http?
On 2012-11-19 21:38, Tyler Jameson Little wrote:

> I think the following are necessary:
>
> * Access to underlying TcpSocket (for protocol upgrades, like WebSocket)
> * HTTP body is a stream
> * Simple HTTP requests
> * Errors can be recovered from (allow user-defined error handlers):
> * User settable size limits
>    * Size of request line (1024 bytes by default)
>    * Size of each header (1024 bytes by default)
>    * Total size of header block (4096 bytes by default)

* HTTPS

-- 
/Jacob Carlborg
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home