I'm writing a network server with some specific requirements:
- 5-50 clients connected (almost) permanently (maybe a bit more, but
definitely not hundreds of them)
- possibly thousands of requests per seconds
- responses need to be returned within 5 seconds or the client will
disconnect and complain

Currently I have a Master thread (which is basically the main thread) which is handling connections/disconnections, socket operations, sends parsed requests for processing to single Worker thread, sends responses to clients. Interaction with Worker is done via message passing.

The problem with my approach is that I read as much data as possible from each ready client in order. As there are many requests this read phase might take a few seconds making the clients disconnect. Now I see 2 possible solutions:

1. Stay with the design I have, but change the workflow somewhat - instead of reading all the data from clients just read some requests and then send responses that are ready and repeat; the downside is that it's more complicated than current design, might be slower (more loop iterations with less work done in each iteration) and might require quite a lot of tweaking when it comes to how many requests/responses handle each time etc.

2. Create separate thread per each client connection. I think this could
result in a nice, clean setup, but I see some problems:
- I'm not sure how ~50 threads will do resource-wise (although they will
probably be mostly waiting on Socket.select)
- I can't initialize threads created via std.concurrency.spawn with a Socket
object ("Aliases to mutable thread-local data not allowed.")
- I already have problems with "interrupted system call" on Socket.select
due to GC kicking in; I'm restarting the call manually, but TBH it sucks I
have to do anything about that and would suck even more to do that with 50
or so threads

If anyone has any idea how to handle the problems I mentioned or has any idea for more suitable design I would be happy to hear it. It's also possible I'm approaching the issue from completely wrong direction, so you can correct me on that as well.

-- 
Marek Janukowicz

On Sunday, 4 August 2013 at 19:37:40 UTC, Marek Janukowicz wrote:
> I'm writing a network server with some specific requirements:
> - 5-50 clients connected (almost) permanently (maybe a bit more, but
> definitely not hundreds of them)
> - possibly thousands of requests per seconds
> - responses need to be returned within 5 seconds or the client will
> disconnect and complain
>
> Currently I have a Master thread (which is basically the main thread) which
> is handling connections/disconnections, socket operations, sends parsed
> requests for processing to single Worker thread, sends responses to clients.
> Interaction with Worker is done via message passing.
>
> The problem with my approach is that I read as much data as possible from
> each ready client in order. As there are many requests this read phase might
> take a few seconds making the clients disconnect. Now I see 2 possible
> solutions:
>
> 1. Stay with the design I have, but change the workflow somewhat - instead
> of reading all the data from clients just read some requests and then send
> responses that are ready and repeat; the downside is that it's more
> complicated than current design, might be slower (more loop iterations with
> less work done in each iteration) and might require quite a lot of tweaking
> when it comes to how many requests/responses handle each time etc.
>
> 2. Create separate thread per each client connection. I think this could
> result in a nice, clean setup, but I see some problems:
> - I'm not sure how ~50 threads will do resource-wise (although they will
> probably be mostly waiting on Socket.select)
> - I can't initialize threads created via std.concurrency.spawn with a Socket
> object ("Aliases to mutable thread-local data not allowed.")
> - I already have problems with "interrupted system call" on Socket.select
> due to GC kicking in; I'm restarting the call manually, but TBH it sucks I
> have to do anything about that and would suck even more to do that with 50
> or so threads
>
> If anyone has any idea how to handle the problems I mentioned or has any
> idea for more suitable design I would be happy to hear it. It's also
> possible I'm approaching the issue from completely wrong direction, so you
> can correct me on that as well.

Take a look at how vibe.d approaches the problem: http://vibed.org/

04-Aug-2013 23:38, Marek Janukowicz пишет:
> I'm writing a network server with some specific requirements:
> - 5-50 clients connected (almost) permanently (maybe a bit more, but
> definitely not hundreds of them)
> - possibly thousands of requests per seconds
> - responses need to be returned within 5 seconds or the client will
> disconnect and complain
>
> Currently I have a Master thread (which is basically the main thread) which
> is handling connections/disconnections, socket operations, sends parsed
> requests for processing to single Worker thread, sends responses to clients.
> Interaction with Worker is done via message passing.

Typical approach would be to separate responsibilities even more  and make a pool of threads per each stage.

You may want to make a Master thread only handle new connections selecting over an "accept socket" (or a few if multiple end-points).
Then it may distribute connected clients over I/O worker threads.

A pool of I/O workers would then only send/receive data passing parsed request to "real" workers and responses back. They handle disconnects and closing though.

The real workers could be again pooled to be more responsive (or e.g. just one per each I/O thread).

> The problem with my approach is that I read as much data as possible from
> each ready client in order. As there are many requests this read phase might
> take a few seconds making the clients disconnect. Now I see 2 possible
> solutions:
>
> 1. Stay with the design I have, but change the workflow somewhat - instead
> of reading all the data from clients just read some requests and then send
> responses that are ready and repeat; the downside is that it's more
> complicated than current design, might be slower (more loop iterations with
> less work done in each iteration) and might require quite a lot of tweaking
> when it comes to how many requests/responses handle each time etc.

Or split the clients across a group of threads to reduce maximum latency. See above, just determine the amount of clients per thread your system can sustain in time. A better way would be to dynamically load-balance clients between threads but it's far more complicated.

> 2. Create separate thread per each client connection. I think this could
> result in a nice, clean setup, but I see some problems:
> - I'm not sure how ~50 threads will do resource-wise (although they will
> probably be mostly waiting on Socket.select)

50 threads is not that big a problem. Around 100+ could be, 1000+ is a killer. The benefit with thread per client is that you don't even need Socket.select, just use blocking I/O and do the work per each parsed request in the same thread.

> - I can't initialize threads created via std.concurrency.spawn with a Socket
> object ("Aliases to mutable thread-local data not allowed.")

This can be hacked with casts to shared void* and back. Not pretty but workable.

> - I already have problems with "interrupted system call" on Socket.select
> due to GC kicking in; I'm restarting the call manually, but TBH it sucks I
> have to do anything about that and would suck even more to do that with 50
> or so threads

I'm not sure if that problem will surface with blocking reads.

> If anyone has any idea how to handle the problems I mentioned or has any
> idea for more suitable design I would be happy to hear it. It's also
> possible I'm approaching the issue from completely wrong direction, so you
> can correct me on that as well.
>


-- 
Dmitry Olshansky

John Colvin wrote:
> Take a look at how vibe.d approaches the problem: http://vibed.org/

Vibe.d uses fibers, which I don't find feasible for my particular
application for a number of reasons:
- I have constant number of ever-connected clients, not an ever-changing
number of random clients
- after I read and parse a request there is not much room for yielding
during processing (I don't do I/O or database calls, I have an in-memory
"database" for performance reasons)
- event-based programming generally looks complicated to me and (for the
reason mentioned above) I don't see much point in utilizing it in this case

-- 
Marek Janukowicz

On Sunday, 4 August 2013 at 20:37:43 UTC, Marek Janukowicz wrote:
> John Colvin wrote:
>> Take a look at how vibe.d approaches the problem:
>> http://vibed.org/
>
> Vibe.d uses fibers, which I don't find feasible for my particular
> application for a number of reasons:
> - I have constant number of ever-connected clients, not an ever-changing
> number of random clients
> - after I read and parse a request there is not much room for yielding
> during processing (I don't do I/O or database calls, I have an in-memory
> "database" for performance reasons)
> - event-based programming generally looks complicated to me and (for the
> reason mentioned above) I don't see much point in utilizing it in this case

You'd be surprised how easy it can be with vibe and D

Nonetheless, this isn't my area of expertise, I just thought it might be interesting, if you hadn't already seen it.

Dmitry Olshansky wrote:

> 04-Aug-2013 23:38, Marek Janukowicz пишет:
>> I'm writing a network server with some specific requirements:
>> - 5-50 clients connected (almost) permanently (maybe a bit more, but
>> definitely not hundreds of them)
>> - possibly thousands of requests per seconds
>> - responses need to be returned within 5 seconds or the client will
>> disconnect and complain
>>
>> Currently I have a Master thread (which is basically the main thread) which is handling connections/disconnections, socket operations, sends parsed requests for processing to single Worker thread, sends responses to clients. Interaction with Worker is done via message passing.
> 
> Typical approach would be to separate responsibilities even more  and make a pool of threads per each stage.
> 
> You may want to make a Master thread only handle new connections selecting over an "accept socket" (or a few if multiple end-points). Then it may distribute connected clients over I/O worker threads.
> 
> A pool of I/O workers would then only send/receive data passing parsed request to "real" workers and responses back. They handle disconnects and closing though.

This is basically approach "2." I mentioned in my original post, I'm glad you agree it makes sense :)

> The real workers could be again pooled to be more responsive (or e.g.
> just one per each I/O thread).

There are more things specific to this particular application that would play a role here. One is that such "real workers" would operate on a common data structure and I would have to introduce some synchronization. Single worker thread was not my first approach, but after some woes with other solutions I decided to take it, because the problem is really not in processing (where a single thread does just fine so far), but in socket read/write operations.

>> The problem with my approach is that I read as much data as possible from each ready client in order. As there are many requests this read phase might take a few seconds making the clients disconnect. Now I see 2 possible solutions:
>>
>> 1. Stay with the design I have, but change the workflow somewhat - instead of reading all the data from clients just read some requests and then send responses that are ready and repeat; the downside is that it's more complicated than current design, might be slower (more loop iterations with less work done in each iteration) and might require quite a lot of tweaking when it comes to how many requests/responses handle each time etc.
> 
> Or split the clients across a group of threads to reduce maximum latency. See above, just determine the amount of clients per thread your system can sustain in time. A better way would be to dynamically load-balance clients between threads but it's far more complicated.

Yeah, both approaches seem to be somewhat more complicated and I'd like to aovid this if possible. So one client per thread makes sense to me.

>> 2. Create separate thread per each client connection. I think this could
>> result in a nice, clean setup, but I see some problems:
>> - I'm not sure how ~50 threads will do resource-wise (although they will
>> probably be mostly waiting on Socket.select)
> 
> 50 threads is not that big a problem. Around 100+ could be, 1000+ is a killer.

Thanks for those numbers, it's great to know at least the ranges here.

> The benefit with thread per client is that you don't even need Socket.select, just use blocking I/O and do the work per each parsed request in the same thread.

Not really. This is something that Go (the language I also originally
considered for the project) has solved in much better way - you can "select"
on a number of "channels" and have both I/O and message passing covered by
those. In D I must react both to network data or message from worker
incoming, which means either self-pipe trick (which leads to Socket.select
again) or some quirky stuff with timeouts on socket read and message receive
(but this is basically a busy loop).

>> - I can't initialize threads created via std.concurrency.spawn with a Socket object ("Aliases to mutable thread-local data not allowed.")
> 
> This can be hacked with casts to shared void* and back. Not pretty but workable.

I'm using this trick elsewhere, was a bit reluctant to try it here. Btw. would it work if I pass a socket to 2 threads - reader and writer (by working I mean - not running into race conditions and other scary concurrent stuff)?

Also I'm really puzzled by the fact this common idiom doesn't work in some elegant way in D. I tried to Google a solution, but only found some weird tricks. Can anyone really experienced in D tell me why there is no nice solution for this (or correct me if I'm mistaken)?

>> - I already have problems with "interrupted system call" on Socket.select due to GC kicking in; I'm restarting the call manually, but TBH it sucks I have to do anything about that and would suck even more to do that with 50 or so threads
> 
> I'm not sure if that problem will surface with blocking reads.

Unfortunately it will (it precisely happens with blocking calls).

Thanks for your input, which shed some more light for me and also allowed me to explain the whole thing a bit more.

-- 
Marek Janukowicz

Am Sun, 04 Aug 2013 22:59:04 +0200
schrieb Marek Janukowicz <marek@janukowicz.net>:

> >> - I already have problems with "interrupted system call" on Socket.select due to GC kicking in; I'm restarting the call manually, but TBH it sucks I have to do anything about that and would suck even more to do that with 50 or so threads
> > 
> > I'm not sure if that problem will surface with blocking reads.
> 
> Unfortunately it will (it precisely happens with blocking calls).
> 
> Thanks for your input, which shed some more light for me and also allowed me to explain the whole thing a bit more.
> 

This is a bug in std.socket BTW. Blocking calls will get interrupted by the GC - there's no way to avoid that - but std.socket should handle this internally and just retry the interrupted operation. Please file a bug report about this.

(Partial writes is another issue that could/should be handled in std.socket so the user doesn't have to care about it)

On 2013-08-04 19:38:49 +0000, Marek Janukowicz said:

> ...
> If anyone has any idea how to handle the problems I mentioned or has any
> idea for more suitable design I would be happy to hear it. It's also
> possible I'm approaching the issue from completely wrong direction, so you
> can correct me on that as well.

Hi, I would take a look at the BEEP protocol idea and there at the Vortex library [1] it deals with everything you need. The idea of BEEP is, that you don't have to care about all the network pitfalls since these are always the same. Instead you can concentrate on your application level design. Where the time is spent much more valuable.

The lib is written in C and works very good. It's matured and multi-threaded to allow for maximum transfers.

[1] http://www.aspl.es/vortex/


-- 
Robert M. Münch
Saphirion AG

http://www.saphirion.com
smarter | better | faster

On Sun, 04 Aug 2013 20:38:49 +0100, Marek Janukowicz <marek@janukowicz.net> wrote:
> I'm writing a network server with some specific requirements:
> - 5-50 clients connected (almost) permanently (maybe a bit more, but
> definitely not hundreds of them)
> - possibly thousands of requests per seconds
> - responses need to be returned within 5 seconds or the client will
> disconnect and complain
>
> Currently I have a Master thread (which is basically the main thread) which
> is handling connections/disconnections, socket operations, sends parsed
> requests for processing to single Worker thread, sends responses to clients.
> Interaction with Worker is done via message passing.
>
> The problem with my approach is that I read as much data as possible from
> each ready client in order. As there are many requests this read phase might
> take a few seconds making the clients disconnect. Now I see 2 possible
> solutions:
>
> 1. Stay with the design I have, but change the workflow somewhat - instead
> of reading all the data from clients just read some requests and then send
> responses that are ready and repeat; the downside is that it's more
> complicated than current design, might be slower (more loop iterations with
> less work done in each iteration) and might require quite a lot of tweaking
> when it comes to how many requests/responses handle each time etc.
>
> 2. Create separate thread per each client connection. I think this could
> result in a nice, clean setup, but I see some problems:
> - I'm not sure how ~50 threads will do resource-wise (although they will
> probably be mostly waiting on Socket.select)
> - I can't initialize threads created via std.concurrency.spawn with a Socket
> object ("Aliases to mutable thread-local data not allowed.")
> - I already have problems with "interrupted system call" on Socket.select
> due to GC kicking in; I'm restarting the call manually, but TBH it sucks I
> have to do anything about that and would suck even more to do that with 50
> or so threads
>
> If anyone has any idea how to handle the problems I mentioned or has any
> idea for more suitable design I would be happy to hear it. It's also
> possible I'm approaching the issue from completely wrong direction, so you
> can correct me on that as well.

Option #2 should be fine, provided you don't intend to scale to a larger number of clients.  I have had loads of experience with server applications on Windows and a little less on the various flavours of UNIXen and 50 connected clients serviced by 50 threads should be perfectly manageable for the OS.

It sounds like only non-blocking sockets have the GC interrupt issue, if so use non-blocking sockets instead.  However, it occurs to me that the issue may rear it's head again on the call to select() on non-blocking sockets, so it is worth testing this first.

If there is no way around the GC interrupt issue then code up your own recv function and re-use it all your threads, not ideal but definitely workable.

In the case of non-blocking sockets your read operation needs to account for the /this would block/ error code, and should go something like this.. (using low level socket function call names because I have not used the D socket library recently)
1. Attempt recv(), expect either DATA or ERROR.
1a. If DATA, process data and handle possible partial request(s) - by buffering and going back to #1 for example
1b. If ERROR, check for code [WSA]EWOULDBLOCK and if so go to step #2
1c. If ERROR and not would block, fail/exit/disconnect.
2. Perform select() (**this may be interruptable by GC**) for a finite shortish timeout - if you want your client handlers to react quickly to the signal to shutdown then you want a shorter time - for example.
2a. If select returns a read indicator, go to #1
2b. If select returns an error, fail/exit/disconnect.
2c. Otherwise, go to #2

Do you have control of the connecting client code as well?  If so, think about disabling the Nagle algorithm:
http://en.wikipedia.org/wiki/Nagle's_algorithm

You will want to ensure the client writes it's requests in a single send() call but in this way you reduce the delay in receiving requests at the server side.  If the client writes multiple requests rapidly then with Nagle enabled it may buffer them on the client end and will delay the server seeing the first, but with it disabled the server will see the first as soon as it is written and can start processing it while the client writes.  So depending on how your clients send requests, you may see a performance improvement here.

I don't know how best to solve the "Aliases to mutable thread-local data not allowed.".  You will need to ensure the socket is allocated globally (not thread local) and because you know it's unique and not shared you can cast it as such to get it into the thread, once there you can cast it back to unshared/local/mutable.  Not ideal, but not illegal or invalid AFAICS.

FYI.. For a better more scaleable solution you would use async IO with a pool of worker threads, I am not sure if D has good support for this and it's been a while since I did this myself (outside of using the nice C# library support for it).

Regan

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/

On Monday, 5 August 2013 at 06:36:15 UTC, Johannes Pfau wrote:
> This is a bug in std.socket BTW. Blocking calls will get interrupted by
> the GC - there's no way to avoid that - but std.socket should handle
> this internally and just retry the interrupted operation. Please file a
> bug report about this.

I'm not sure whether we can do anything about Socket.select itself at this point, as it would be a breaking API change – interrupted calls returning a negative value is even mentioned explicitly in the docs.

There should, however, be a way to implement this in a platform-independent manner in client code, or even a second version that handles signal interruptions internally.

> (Partial writes is another issue that could/should be handled in
> std.socket so the user doesn't have to care about it)

I don't think that would be possible – std.socket by design is a thin wrapper around BSD sockets (whether that's a good idea or not is another question), and how to handle partial writes depends entirely on the environment the socket is used in (think event-based architecture using fibers vs. other designs).

In general, I wonder what the best way for going forward with std.socket is. Sure, we could try to slowly morph it into a "modern" networking implementation, but the current state also has its merits, as it allows people to use the familiar BSD sockets API without having to worry about all the trivial differences between the platforms (e.g. in symbol names).

We should definitely add a note to std.socket though that it is a low-level API and that there might be a better choice for most applications (e.g. vibe.d, Thrift, …).

David