I'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before.

An example of how I would like to interface with it (for creating a server):

    interface HTTPHandler {
        void serveHTTP(Request, Response);
    }

    class CustomHandler : HTTPHandler {
        ....
        void serveHTTP(Request req, Response res) {
        }
    }

    auto handler = new CustomHandler(); // implements http.Handler
    auto server = new HttpServer(handler);
    server.listen("0.0.0.0", 80);

As long as serveHttp() is thread safe, the http server could be concurrent or evented (with libev or similar) and the custom handler code wouldn't have to change.

I'm willing to put in the lion's share of the work (I've already written a bunch of it), but I'd naturally like to get some community input so I don't go in a completely wrong direction. I'm thinking of emulating Go's http library (http://golang.org/pkg/net/http), but of course D style.

I think the following are necessary:

* Access to underlying TcpSocket (for protocol upgrades, like WebSocket)
* HTTP body is a stream
* Simple HTTP requests
* Errors can be recovered from (allow user-defined error handlers):
* User settable size limits
  * Size of request line (1024 bytes by default)
  * Size of each header (1024 bytes by default)
  * Total size of header block (4096 bytes by default)

So, basically I'm looking for two things:

1. Interest level: willing to writ code, willing to test, want to use
2. Suggestions for the API/behavior

On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little wrote:
> I'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before.

I've been asked to put my cgi.d in phobos before (which includes a http server as well as a cgi, fastcgi, and scgi implementation of its common interface), so there's probably some interest.

I haven't put mine in just because I'm not particularly motivated to go through the red tape.

The file is cgi.d in here:
https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff

Feel free to take whatever you want from it. There's also an extremely minimal http.d file in there, which is a very simple http client. (I didn't go far with this though because I tend to use curl for most my work anyway.)



The way I did it is with a function:

void handleRequest(Cgi cgi) { }

And then there's a mixin called GenericMain that calls your function.

mixin GenericMain!handleRequest;


Depending on how you compile it, the main function constructs the cgi object differently. -version=fastcgi means it is plugged into a FastCGI loop. version=scgi ditto. -version=embedded_httpd includes a simple little threaded http server. No version builds it as a standard CGI binary, one process per request. However it is compiled though, the Cgi methods all remain the same, so your function shouldn't have to change.


> * Access to underlying TcpSocket (for protocol upgrades, like WebSocket)

My cgi.d I don't think can do this yet. I've been thinking about adding it though, but haven't.

> * HTTP body is a stream

class Cgi has a method onRequestBodyDataReceived and one handleIncomingDataChunk which can be overridden by subclasses to do this kind of thing.

It's a bit of a pain though, I made it thinking you'd want to handle it after it is all loaded.

> * Simple HTTP requests

This is where I think my cgi.d is a winner because most the data is available so easily

cgi.get["foo"] returns a string
cgi.getArray["foo"] returns an array of strings

ditto for post

cgi.request("foo", 0) returns with a default value, converting types as requested

There's also members like cookie, queryString, host, etc., that gives nice access to other headers.

> * Errors can be recovered from (allow user-defined error handlers):
> * User settable size limits

eh, I'm partially there on these. There's some constructor args you can play with and some degree of exception catching but probably not fully what you have in mind.

> I've been asked to put my cgi.d in phobos before (which includes a http server as well as a cgi, fastcgi, and scgi implementation of its common interface), so there's probably some interest.
>
> I haven't put mine in just because I'm not particularly motivated to go through the red tape.
>
> The file is cgi.d in here:
> https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff

Awesome. I assume this hasn't gone through rigorous testing, but has it been used in production?

>> * Errors can be recovered from (allow user-defined error handlers):
>> * User settable size limits
>
> eh, I'm partially there on these. There's some constructor args you can play with and some degree of exception catching but probably not fully what you have in mind.

I'm thinking of something a little more sophisticated than exceptions. Exceptions unwind the stack, so you can't just continue where you left off.

How do you feel about function pointer callbacks? The http parser in node.js (https://github.com/joyent/http-parser) takes a struct of function pointers to handle errors/events. When the parser encounters a recoverable error (max header length reached), the programmer could opt to ignore it and continue the parse. Unrecoverable errors could throw exceptions (like trying to parse garbage data).

If an error is recovered, it would just continue as if nothing happened.

Of course, implementing this would increase complexity in the parser, but I think it's justified. Thoughts?

Before you spend too much time :
https://github.com/rejectedsoftware/vibe.d/tree/master/source/vibe/http



On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little wrote:
> I'd like to see an HTTP module in Phobos, but I wanted to gauge interest first and see if this has been discussed before.
>
> An example of how I would like to interface with it (for creating a server):
>
>     interface HTTPHandler {
>         void serveHTTP(Request, Response);
>     }
>
>     class CustomHandler : HTTPHandler {
>         ....
>         void serveHTTP(Request req, Response res) {
>         }
>     }
>
>     auto handler = new CustomHandler(); // implements http.Handler
>     auto server = new HttpServer(handler);
>     server.listen("0.0.0.0", 80);
>
> As long as serveHttp() is thread safe, the http server could be concurrent or evented (with libev or similar) and the custom handler code wouldn't have to change.
>
> I'm willing to put in the lion's share of the work (I've already written a bunch of it), but I'd naturally like to get some community input so I don't go in a completely wrong direction. I'm thinking of emulating Go's http library (http://golang.org/pkg/net/http), but of course D style.
>
> I think the following are necessary:
>
> * Access to underlying TcpSocket (for protocol upgrades, like WebSocket)
> * HTTP body is a stream
> * Simple HTTP requests
> * Errors can be recovered from (allow user-defined error handlers):
> * User settable size limits
>   * Size of request line (1024 bytes by default)
>   * Size of each header (1024 bytes by default)
>   * Total size of header block (4096 bytes by default)
>
> So, basically I'm looking for two things:
>
> 1. Interest level: willing to writ code, willing to test, want to use
> 2. Suggestions for the API/behavior

On Monday, 19 November 2012 at 20:38:56 UTC, Tyler Jameson Little wrote:
> * HTTP body is a stream

No Content-Size, no multiple requests per connection (unless you use chunked encoding?).

> * User settable size limits
>   * Size of request line (1024 bytes by default)
>   * Size of each header (1024 bytes by default)
>   * Total size of header block (4096 bytes by default)

I think this is pointless. Use an appender and the RFC limits. Unless you are serving VERY simple pages, the cost of a few allocations for handling the HTTP protocol's overhead will not be noticeable compared to the application's. (Slowloris is something to keep in mind, though.)

My HTTP library, which is used for this forum and several other "production" projects, is here: https://github.com/CyberShadow/ae
The main problem with getting it into Phobos is that it's tied to a lot of other related code, such as the asynchronous sockets module, the unmanaged memory wrapper type, etc.

Also, what about vibe.d?

On Monday, 19 November 2012 at 23:57:35 UTC, Tyler Jameson Little wrote:
> Awesome. I assume this hasn't gone through rigorous testing, but has it been used in production?

I've been using it for almost all my work apps for the last... I think three years now. None of them so far have huge userbases, but it has been in steady use for a while.

The others in this thread have pointed out the other options that have some users: Vladimir's library that runs the forum, and vibe.d that seems to be the most popular. I haven't used either of them though.

> How do you feel about function pointer callbacks?

We could also do them as virtual functions... that's a little easier in the setup I have because the constructor arguments are messy, whereas subclassing is a lot simpler.

But yeah, it sounds like a decent plan.

>> * HTTP body is a stream
>
> No Content-Size, no multiple requests per connection (unless you use chunked encoding?).

Not sure what you mean. I meant incoming stream. There would be a request object with access to all other headers, it just wouldn't be read in until the user actually wanted it.

>> * User settable size limits
>>  * Size of request line (1024 bytes by default)
>>  * Size of each header (1024 bytes by default)
>>  * Total size of header block (4096 bytes by default)
>
> I think this is pointless. Use an appender and the RFC limits. Unless you are serving VERY simple pages, the cost of a few allocations for handling the HTTP protocol's overhead will not be noticeable compared to the application's. (Slowloris is something to keep in mind, though.)

I just made up those limits, but the point is I'd like the user to be able to tweak those. The default will work for most people though.

> My HTTP library, which is used for this forum and several other "production" projects, is here: https://github.com/CyberShadow/ae
> The main problem with getting it into Phobos is that it's tied to a lot of other related code, such as the asynchronous sockets module, the unmanaged memory wrapper type, etc.

Cool, I didn't know that this was served by D code. I'll take a look at it.

> Also, what about vibe.d?

vibe.d does a lot of things, and it probably does those things very well. It seems well maintained, and generally a good project. I think that a lot of the things it does well could be brought into the standard library. As this thread pointed out, there are several HTTP parsers floating around out there. For some, vibe.d might not be a perfect fit for whatever reason, but everyone can benefit from a simple HTTP library.

Maybe it shouldn't be as high-level as Go's http library, but it should at least make writing a simple HTTP server trivial.

Would a minor refactor of vibe.d be acceptable? This is pretty much what I'm looking for:

https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/common.d
https://github.com/rejectedsoftware/vibe.d/blob/master/source/vibe/http/server.d

Except that I'd remove some parts of it, like JSON parsing.

I think that vibe.d could benefit from moving some of the code there into Phobos. I guess it comes down to whether it makes sense to make a standard HTTP library.

On 11/19/12 3:38 PM, Tyler Jameson Little wrote:
> I'd like to see an HTTP module in Phobos, but I wanted to gauge interest
> first and see if this has been discussed before.

I can say the following. We sorely need a server expert on board with the time and inclination to write a good server-side networking framework. We have a couple, but for various reasons I have been unable to convince them (through newsgroup and private exchanges) to invest time in such an endeavor. That's mostly because lack of time - speaking for myself, I wish I found the time to learn about all that, which as far as I understand gravitates around notions such as asynchronous I/O, libevent/libevt, select(), and the such. I've never used them so I'd need to start from scratch.

We really need an expert. Look at the Go programming language - it's not remarkable, but it benefits of full-time dedication of experts in server-side programming, so it has acquired excellent library support for networking servers. And they milk that for all it's worth: any discussion, article, or blog post about Go gravitates toward the five-lines HTTP server with the same implacable reach as conversations with ideologists, which inevitably converge towards their ideological stronghold.


Andrei

On 11/19/12 11:26 PM, Andrei Alexandrescu wrote:
> On 11/19/12 3:38 PM, Tyler Jameson Little wrote:
>> I'd like to see an HTTP module in Phobos, but I wanted to gauge interest
>> first and see if this has been discussed before.
>
> I can say the following. We sorely need a server expert on board with
> the time and inclination to write a good server-side networking
> framework.

To clarify (lest this would be interpreted as excluding vibe.d from the category of good server-side networking frameworks): ... for inclusion in the standard library.

Andrei

On 2012-11-19 21:38, Tyler Jameson Little wrote:

> I think the following are necessary:
>
> * Access to underlying TcpSocket (for protocol upgrades, like WebSocket)
> * HTTP body is a stream
> * Simple HTTP requests
> * Errors can be recovered from (allow user-defined error handlers):
> * User settable size limits
>    * Size of request line (1024 bytes by default)
>    * Size of each header (1024 bytes by default)
>    * Total size of header block (4096 bytes by default)

* HTTPS

-- 
/Jacob Carlborg