On Sunday, 23 June 2013 at 18:24:05 UTC, Andrei Alexandrescu wrote:
> Would separate processes work better under high load? Educate me.

One nice thing is you can spread separate processes across several machines. Another advantage of *cgi|embedded_httpd is that it is portable to other web servers too, you can just drop it into a shop that uses nginx or Microsoft IIS and expect it to work, whereas an Apache module is generally apache only.

I think it is just an accident of history that mod_php ever got used. Classic cgi implementations were still slow enough (especially with an interpreted language) that people wanted to try something else, but the other world of options hadn't taken root yet either (I think mod_php even slightly predates fastcgi's introduction), and continues to exist just out of inertia.

On Sunday, 23 June 2013 at 17:54:01 UTC, Andrei Alexandrescu wrote:
> I assume it's a dynamic library.

PHP can work as fastcgi too.

>> But I kinda want to play with this now with shared libraries just for
>> something to do so maybe I will.
>
> Awesome!

Eh, not so much. I started playing and got a thing saying Runtime.loadLibrary is not yet implemented on posix... so went to do it myself and got random segfaults, relating to the garbage collector (and yes, I tried Runtime.initialize).

I've never done a shared library plugin before so I could be doing something wrong, but it seems to me that druntime said it wasn't implemented yet for a reason, must still have some work to do there. Oh well, maybe I'll try again next release.

BTW I should mention, I wasn't actually trying to do an Apache module. I wanted to do a D server that watches the files for changes, then recompiles them as needed and reloads the resulting file as a shared lib.

I could just run the compiled executable too, cgi style, but here I was more interested in playing with shared libraries than just getting it to work.

Am 20.06.2013 11:26, schrieb Jacob Carlborg:
> On 2013-06-20 00:13, Paulo Pinto wrote:
>
>> The issue is not to test third party libraries, far from it.
>>
>> The problem is that you cannot mock them, specially if you rely a lot on
>> non virtual methods or pure function calls. Or on framework code that
>> calls your code back, after certain events happened in the system.
>
> I would only mock an external service that needs to be available online.
> I don't mock third party code. That would be insane. There's no limit.
> Should I mock the standard library?

No, but if your code calls something that ends up calling again other part of your code, you might need to mock it somehow.

Specially if it is a library that does some IO (network, disk) in the process and is full of static methods.

--
Paulo

On 2013-06-23 21:50, Paulo Pinto wrote:

> No, but if your code calls something that ends up calling again other
> part of your code, you might need to mock it somehow.

I'm using same approach there. If I'm testing method "a" and it also calls method "b". I assume method "b" works and I don't mock it. Then I make sure to have a test for method "b" as well. A given test only tests one method.

> Specially if it is a library that does some IO (network, disk) in the
> process and is full of static methods.

Especially network IO is a good idea to mock.

-- 
/Jacob Carlborg

On Sunday, 23 June 2013 at 17:54:01 UTC, Andrei Alexandrescu wrote:
> Awesome!

I realized initializing the runtime might have been a mistake when loading the .so and moreover I was using the wrong calling convention. Fixed that and now the shared library thing works. On Linux at least, I hardcoded ".so" so don't try it anywhere else.

Here's some code:
http://arsdnet.net/dcode/dhp.zip

unzip it, hit make. dhp2.d is the main file here, it also uses some helper libraries you can see in the arsd folder.

Then run ./dhp2 in another window or something. It is a long running process that serves http.

Head over to your browser and surf to
http://localhost:8085/


It will take a couple seconds to load the first time, because it needs to compile the code. Subsequent loads will pull it from a cache and be faster.

Anyway it will show you some stuff and a link. These pull from the files in the zip called index.dhp and dhpt.dhp. index is just html, no D code, although it will compile it since this server compiles everything.


dhpt.dhp actually includes some D:

        <p>Hello, <% cgi.write(cgi.request("name", "user")); %>, happy to see you.</p>

Note: you access that by going to localhost:8085/dhpt, NOT /dhpt.dhp. It strips out dots from the url as a way of sanitizing the filename so keep it simple.

It uses ASP style <% %> tags instead of <?d ?> because my dom.d already understands them. (BTW this parses the .dhp files to be well-formed xml, so if you mismatch tags, it will throw. It might be fun to put the DOM node in scope to inspect too).

There's a Cgi cgi in scope in the function it builds here. Use it to do communication instead of writeln() etc., as seen in this example.



Here's where the shared library magic comes in: feel free to edit one of those .dhp files, or create your own, and go back to it in the browser. It will recompile and present it to you without having to restart the server. That's kinda cool. The downside is if you segfault in here it will take the whole server down so don't do that.

If you fail compiling though, it will actually read dmd's output and translate the filename and line number to match the .dhp file input, instead of the .d file dmd actually sees. So it feels less like a filthy hack then.

Feel free to look at index.d and dhpt.d in that same folder after you browse to them to see what the generated code looks like.




Soooo yeah. I'll probably never use this, dom templates rok so much more than anything asp/php style, but if you wanna play, feel free and let me know if you want more features or see bugs. Probably won't be too hard to fix up now that it is started.

On 6/23/13 11:51 AM, Adam D. Ruppe wrote:
> I think it is just an accident of history that mod_php ever got used.
> Classic cgi implementations were still slow enough (especially with an
> interpreted language) that people wanted to try something else, but the
> other world of options hadn't taken root yet either (I think mod_php
> even slightly predates fastcgi's introduction), and continues to exist
> just out of inertia.

OK so what's the way to go now? One process per request? Seems heavy to me seeing as most requests last very little.

Andrei

On Monday, 24 June 2013 at 02:39:30 UTC, Andrei Alexandrescu wrote:
> On 6/23/13 11:51 AM, Adam D. Ruppe wrote:
>> I think it is just an accident of history that mod_php ever got used.
>> Classic cgi implementations were still slow enough (especially with an
>> interpreted language) that people wanted to try something else, but the
>> other world of options hadn't taken root yet either (I think mod_php
>> even slightly predates fastcgi's introduction), and continues to exist
>> just out of inertia.
>
> OK so what's the way to go now? One process per request? Seems heavy to me seeing as most requests last very little.
>

That is what apache used to do when mod_php was introduced anyway. This whole conversation has gone out of control :D

mod_php was a way to reduce the communication overhead and avoid 2 process per request in the first place (instead of one).

It become plain obvious that no silver bullet exists here. The solution seems to be a mix of fibers/threads and processes where number of each depends on your workload and the actual hardware it is running on. Most high performance and high scalability system are converging toward one flavor of that pattern.

On Sunday, 23 June 2013 at 18:24:05 UTC, Andrei Alexandrescu wrote:
> On 6/23/13 11:04 AM, Dicebot wrote:
>> On Sunday, 23 June 2013 at 17:54:01 UTC, Andrei Alexandrescu wrote:
>>> On 6/23/13 10:34 AM, Adam D. Ruppe wrote:
>>> We should do what php does, it was very successful. I assume it's a
>>> dynamic library.
>>
>> And what reason behind this other than "millions of lemmings can't be
>> wrong"? This approach is disaster when it comes to high load.
>
> Would separate processes work better under high load? Educate me.
>
> Andrei

It is not about "process vs dynamic library", it is about basic threaded architecture used in Apache. Every single network server that I know that cares about performance uses some sort of event model and avoids any context switches at all possible costs. It does not matter where request processing code is, only how it fits in bigger picture.

Actually, server software with most impressive performance I have seen so far was implemented as single barebone process with all network stack and event library embedded. Slightly similar idea was propsoed here - http://highscalability.com/blog/2013/5/13/the-secret-to-10-million-concurrent-connections-the-kernel-i.html (found via reddit)

On Sunday, 23 June 2013 at 18:42:12 UTC, Adam D. Ruppe wrote:
> On Sunday, 23 June 2013 at 18:02:04 UTC, Dicebot wrote:
>> I can see reasons for wanting to keep it behind reverse proxy like nginx, but Apache?
>
> Generally, I don't trust random http servers connected to the open internet for correctness, stability, security, and logging.
>
> Especially not my code, There's a lot of the http protocol I never implemented, and putting apache, nginx, IIS, whatever, I'm just using Apache here because Andrei mentioned it, out in front will filter some of that out before it gets to my app.

Yes, that is why I have said that understand reasoning to use somthing like nginx or lighttpd - it is better to work behind something that has proper implementation of HTTP protocol and focus on application needs. I am using nginx for this on my own and perfectly satisfied with it.

But using Apache is just throwing away most part of performance achived by your backend, for nothing. And creating security issues in case your app is run via Apache module is it typically runs from http user in such scenarios and using app-specific UNIX access policies becomes much harder.