Hi,

I've noticed a couple of times (i.e very rarely, mind you), that the page generation of the forum had hiccups.
Have other people experienced such hiccups ?
If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault (or was it a network latency ?).

Anyhow, the forum seems to me a good opportunity to measure the D runtime in action on the server side, and in particular the GC.
A few questions :
Is the server regularly restarted or is it running smoothly without having to be restarted at all ?
Does it measure the time for the page generation ? Would it be interesting to see at the bottom of the page something like "Page generated in xxx ms", when I (or other people) come accross those hiccups, and to collect these data for statistics over a week/month ?

On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
> Hi,
>
> I've noticed a couple of times (i.e very rarely, mind you), that the page generation of the forum had hiccups.
> Have other people experienced such hiccups ?
> If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault (or was it a network latency ?).
>
> Anyhow, the forum seems to me a good opportunity to measure the D runtime in action on the server side, and in particular the GC.
> A few questions :
> Is the server regularly restarted or is it running smoothly without having to be restarted at all ?
> Does it measure the time for the page generation ? Would it be interesting to see at the bottom of the page something like "Page generated in xxx ms", when I (or other people) come accross those hiccups, and to collect these data for statistics over a week/month ?

So noone thinks this could be a good idea ?

"SomeDude" <lovelydear@mailmetrash.com> wrote in message news:mjectyruiwxebfrkegvt@forum.dlang.org...
> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
>> Hi,
>>
>> I've noticed a couple of times (i.e very rarely, mind you), that the page
>> generation of the forum had hiccups.
>> Have other people experienced such hiccups ?
>> If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault
>> (or was it a network latency ?).
>>
>> Anyhow, the forum seems to me a good opportunity to measure the D runtime
>> in action on the server side, and in particular the GC.
>> A few questions :
>> Is the server regularly restarted or is it running smoothly without
>> having to be restarted at all ?
>> Does it measure the time for the page generation ? Would it be
>> interesting to see at the bottom of the page something like "Page
>> generated in xxx ms", when I (or other people) come accross those
>> hiccups, and to collect these data for statistics over a week/month ?
>
> So noone thinks this could be a good idea ?

I like it. (And the StopWatch in std.datetime would make it super easy. StopWatch is freaking awesome.) But newgroup/forum posts don't always get responses after only a few hours. And even a total lack of replies shouldn't be mistaken for a "no" vote. So I wouldn't worry too much about that.

Could you be more specific what you mean by "hiccup"? I assume you mean that once in a while a page load will be slower than usual. If so, about how long?

My wild random guesses, aside from your ideas, are that maybe your request just happens to occur at the right time for the forum to decide to refresh its cache (if it even works that way). Or maybe it could be related to the newsgroup server (which the web forum is just a frontend for) being temporarily down due to high load (which unfortunately has been happening a lot lately). Vladimir would probably have a better idea what might be going on than I would, though.

On Apr 12, 2012 4:29 PM, "SomeDude" <lovelydear@mailmetrash.com> wrote
>
> So noone thinks this could be a good idea ?

I like it.  I usually put a comment in the bottom of my pages but since the form is implemented in D it would nice to actually display it somewhere.

On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
> Hi,
>
> I've noticed a couple of times (i.e very rarely, mind you), that the page generation of the forum had hiccups.
> Have other people experienced such hiccups ?
> If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault (or was it a network latency ?).

Long page loads are caused by a combination of high server load + RAM cache misses + SQLite queries that require a lot of seeks. I still haven't found the time for the migration to a more powerful server.

> Is the server regularly restarted or is it running smoothly without having to be restarted at all ?

It seems to run fine for weeks on end.

> Does it measure the time for the page generation ? Would it be interesting to see at the bottom of the page something like "Page generated in xxx ms", when I (or other people) come accross those hiccups, and to collect these data for statistics over a week/month ?

There are some numbers in the reddit discussion:

http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte

Le 13/04/2012 01:55, Nick Sabalausky a écrit :
> "SomeDude" <lovelydear@mailmetrash.com> wrote in message news:mjectyruiwxebfrkegvt@forum.dlang.org...
>> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: So noone thinks this could be a good idea ?
> 
> I like it. (And the StopWatch in std.datetime would make it super easy. StopWatch is freaking awesome.) But newgroup/forum posts don't always get responses after only a few hours. And even a total lack of replies shouldn't be mistaken for a "no" vote. So I wouldn't worry too much about that.
> 
OK

> Could you be more specific what you mean by "hiccup"? I assume you mean that once in a while a page load will be slower than usual. If so, about how long?
> 

Yeah, once I was surprised, because it took several seconds I believe.

> My wild random guesses, aside from your ideas, are that maybe your request just happens to occur at the right time for the forum to decide to refresh its cache (if it even works that way). Or maybe it could be related to the newsgroup server (which the web forum is just a frontend for) being temporarily down due to high load (which unfortunately has been happening a lot lately). Vladimir would probably have a better idea what might be going on than I would, though.
> 

Ah, I forgot about the newsgroup server.

Le 13/04/2012 03:40, Vladimir Panteleev a écrit :
> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
>> Hi,
>>
>> I've noticed a couple of times (i.e very rarely, mind you), that the
>> page generation of the forum had hiccups.
>> Have other people experienced such hiccups ?
>> If yes, I wonder if we saw the GC kicking in, or if it was SQLite's
>> fault (or was it a network latency ?).
> 
> Long page loads are caused by a combination of high server load + RAM cache misses + SQLite queries that require a lot of seeks. I still haven't found the time for the migration to a more powerful server.
> 

Well, it's not really urgent. It's fast enough.
Still, collecting statistics over time would have been nice, so that one
could analyse the general behaviour of the GC, for instance, by getting
an idea of the standard deviation and maximum response time over weeks
of use. I guess since it can handle much larger loads, the forum server
is not stressed enough to give very meaningful insight under heavy load.
But I have experienced a response of several seconds once, I don't know
where this comes from (although it's most likely one of your reasons).
So collecting real time data would have to be done for each component of
the chain (SQLite, cache, D generated page). Such data would be useful
for any web server anyway.

BTW, could you repost the source code address ? Thx.

>> Is the server regularly restarted or is it running smoothly without having to be restarted at all ?
> 
> It seems to run fine for weeks on end.

Nice.

> 
>> Does it measure the time for the page generation ? Would it be interesting to see at the bottom of the page something like "Page generated in xxx ms", when I (or other people) come accross those hiccups, and to collect these data for statistics over a week/month ?
> 
> There are some numbers in the reddit discussion:
> 
> http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
> 

Nice, now I remember having read this post.

* Somedude <lovelydear@mailmetrash.com> [2012-04-13 04:25:19 +0200]:
> Well, it's not really urgent. It's fast enough.
> Still, collecting statistics over time would have been nice, so that one
> could analyse the general behaviour of the GC, for instance, by getting
> an idea of the standard deviation and maximum response time over weeks
> of use. I guess since it can handle much larger loads, the forum server
> is not stressed enough to give very meaningful insight under heavy load.
> But I have experienced a response of several seconds once, I don't know
> where this comes from (although it's most likely one of your reasons).
> So collecting real time data would have to be done for each component of
> the chain (SQLite, cache, D generated page). Such data would be useful
> for any web server anyway.

I doubt that the GC is the source of any real problems. As a web developer (in PHP no less :S) I can say with authority that most slow downs are due to waiting on external services and data processing. Some of the slowest parts of our site are due to heavy database usage where we have to perform over 100 SQL queries to generate a page. Other slowdowns are caused by the need to process the data.

I understand that you want to see how the GC affects the server performance, but it will be so small that it will be lost in the noise, a blip in traffic because it's lunchtime and people are watching youtube in your area will cause a bigger delay than the GC kicking in.

--
James Miller

Le 13/04/2012 03:40, Vladimir Panteleev a écrit :
> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote: There are some numbers in the reddit discussion:
> 
> http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
> 
Oh, I just saw you wrote the max page generation was around 50 ms.

Le 13/04/2012 04:43, James Miller a écrit :
> I doubt that the GC is the source of any real problems. As a web developer (in PHP no less :S) I can say with authority that most slow downs are due to waiting on external services and data processing. Some of the slowest parts of our site are due to heavy database usage where we have to perform over 100 SQL queries to generate a page. Other slowdowns are caused by the need to process the data.
> 
> I understand that you want to see how the GC affects the server performance, but it will be so small that it will be lost in the noise, a blip in traffic because it's lunchtime and people are watching youtube in your area will cause a bigger delay than the GC kicking in.
> 
> --
> James Miller

I agree, you're probably right.