View mode: basic / threaded / horizontal-split · Log in · Help
April 12, 2012
Measuring the page generation of the forum
Hi,

I've noticed a couple of times (i.e very rarely, mind you), that 
the page generation of the forum had hiccups.
Have other people experienced such hiccups ?
If yes, I wonder if we saw the GC kicking in, or if it was 
SQLite's fault (or was it a network latency ?).

Anyhow, the forum seems to me a good opportunity to measure the D 
runtime in action on the server side, and in particular the GC.
A few questions :
Is the server regularly restarted or is it running smoothly 
without having to be restarted at all ?
Does it measure the time for the page generation ? Would it be 
interesting to see at the bottom of the page something like "Page 
generated in xxx ms", when I (or other people) come accross those 
hiccups, and to collect these data for statistics over a 
week/month ?
April 12, 2012
Re: Measuring the page generation of the forum
On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
> Hi,
>
> I've noticed a couple of times (i.e very rarely, mind you), 
> that the page generation of the forum had hiccups.
> Have other people experienced such hiccups ?
> If yes, I wonder if we saw the GC kicking in, or if it was 
> SQLite's fault (or was it a network latency ?).
>
> Anyhow, the forum seems to me a good opportunity to measure the 
> D runtime in action on the server side, and in particular the 
> GC.
> A few questions :
> Is the server regularly restarted or is it running smoothly 
> without having to be restarted at all ?
> Does it measure the time for the page generation ? Would it be 
> interesting to see at the bottom of the page something like 
> "Page generated in xxx ms", when I (or other people) come 
> accross those hiccups, and to collect these data for statistics 
> over a week/month ?

So noone thinks this could be a good idea ?
April 12, 2012
Re: Measuring the page generation of the forum
"SomeDude" <lovelydear@mailmetrash.com> wrote in message 
news:mjectyruiwxebfrkegvt@forum.dlang.org...
> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
>> Hi,
>>
>> I've noticed a couple of times (i.e very rarely, mind you), that the page 
>> generation of the forum had hiccups.
>> Have other people experienced such hiccups ?
>> If yes, I wonder if we saw the GC kicking in, or if it was SQLite's fault 
>> (or was it a network latency ?).
>>
>> Anyhow, the forum seems to me a good opportunity to measure the D runtime 
>> in action on the server side, and in particular the GC.
>> A few questions :
>> Is the server regularly restarted or is it running smoothly without 
>> having to be restarted at all ?
>> Does it measure the time for the page generation ? Would it be 
>> interesting to see at the bottom of the page something like "Page 
>> generated in xxx ms", when I (or other people) come accross those 
>> hiccups, and to collect these data for statistics over a week/month ?
>
> So noone thinks this could be a good idea ?

I like it. (And the StopWatch in std.datetime would make it super easy. 
StopWatch is freaking awesome.) But newgroup/forum posts don't always get 
responses after only a few hours. And even a total lack of replies shouldn't 
be mistaken for a "no" vote. So I wouldn't worry too much about that.

Could you be more specific what you mean by "hiccup"? I assume you mean that 
once in a while a page load will be slower than usual. If so, about how 
long?

My wild random guesses, aside from your ideas, are that maybe your request 
just happens to occur at the right time for the forum to decide to refresh 
its cache (if it even works that way). Or maybe it could be related to the 
newsgroup server (which the web forum is just a frontend for) being 
temporarily down due to high load (which unfortunately has been happening a 
lot lately). Vladimir would probably have a better idea what might be going 
on than I would, though.
April 13, 2012
Re: Measuring the page generation of the forum
On Apr 12, 2012 4:29 PM, "SomeDude" <lovelydear@mailmetrash.com> wrote
>
> So noone thinks this could be a good idea ?

I like it.  I usually put a comment in the bottom of my pages but since the
form is implemented in D it would nice to actually display it somewhere.
April 13, 2012
Re: Measuring the page generation of the forum
On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
> Hi,
>
> I've noticed a couple of times (i.e very rarely, mind you), 
> that the page generation of the forum had hiccups.
> Have other people experienced such hiccups ?
> If yes, I wonder if we saw the GC kicking in, or if it was 
> SQLite's fault (or was it a network latency ?).

Long page loads are caused by a combination of high server load + 
RAM cache misses + SQLite queries that require a lot of seeks. I 
still haven't found the time for the migration to a more powerful 
server.

> Is the server regularly restarted or is it running smoothly 
> without having to be restarted at all ?

It seems to run fine for weeks on end.

> Does it measure the time for the page generation ? Would it be 
> interesting to see at the bottom of the page something like 
> "Page generated in xxx ms", when I (or other people) come 
> accross those hiccups, and to collect these data for statistics 
> over a week/month ?

There are some numbers in the reddit discussion:

http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
April 13, 2012
Re: Measuring the page generation of the forum
Le 13/04/2012 01:55, Nick Sabalausky a écrit :
> "SomeDude" <lovelydear@mailmetrash.com> wrote in message 
> news:mjectyruiwxebfrkegvt@forum.dlang.org...
>> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
>> So noone thinks this could be a good idea ?
> 
> I like it. (And the StopWatch in std.datetime would make it super easy. 
> StopWatch is freaking awesome.) But newgroup/forum posts don't always get 
> responses after only a few hours. And even a total lack of replies shouldn't 
> be mistaken for a "no" vote. So I wouldn't worry too much about that.
> 
OK

> Could you be more specific what you mean by "hiccup"? I assume you mean that 
> once in a while a page load will be slower than usual. If so, about how 
> long?
> 

Yeah, once I was surprised, because it took several seconds I believe.

> My wild random guesses, aside from your ideas, are that maybe your request 
> just happens to occur at the right time for the forum to decide to refresh 
> its cache (if it even works that way). Or maybe it could be related to the 
> newsgroup server (which the web forum is just a frontend for) being 
> temporarily down due to high load (which unfortunately has been happening a 
> lot lately). Vladimir would probably have a better idea what might be going 
> on than I would, though.
> 

Ah, I forgot about the newsgroup server.
April 13, 2012
Re: Measuring the page generation of the forum
Le 13/04/2012 03:40, Vladimir Panteleev a écrit :
> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
>> Hi,
>>
>> I've noticed a couple of times (i.e very rarely, mind you), that the
>> page generation of the forum had hiccups.
>> Have other people experienced such hiccups ?
>> If yes, I wonder if we saw the GC kicking in, or if it was SQLite's
>> fault (or was it a network latency ?).
> 
> Long page loads are caused by a combination of high server load + RAM
> cache misses + SQLite queries that require a lot of seeks. I still
> haven't found the time for the migration to a more powerful server.
> 

Well, it's not really urgent. It's fast enough.
Still, collecting statistics over time would have been nice, so that one
could analyse the general behaviour of the GC, for instance, by getting
an idea of the standard deviation and maximum response time over weeks
of use. I guess since it can handle much larger loads, the forum server
is not stressed enough to give very meaningful insight under heavy load.
But I have experienced a response of several seconds once, I don't know
where this comes from (although it's most likely one of your reasons).
So collecting real time data would have to be done for each component of
the chain (SQLite, cache, D generated page). Such data would be useful
for any web server anyway.

BTW, could you repost the source code address ? Thx.

>> Is the server regularly restarted or is it running smoothly without
>> having to be restarted at all ?
> 
> It seems to run fine for weeks on end.

Nice.

> 
>> Does it measure the time for the page generation ? Would it be
>> interesting to see at the bottom of the page something like "Page
>> generated in xxx ms", when I (or other people) come accross those
>> hiccups, and to collect these data for statistics over a week/month ?
> 
> There are some numbers in the reddit discussion:
> 
> http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
> 

Nice, now I remember having read this post.
April 13, 2012
Re: Measuring the page generation of the forum
* Somedude <lovelydear@mailmetrash.com> [2012-04-13 04:25:19 +0200]:
> Well, it's not really urgent. It's fast enough.
> Still, collecting statistics over time would have been nice, so that one
> could analyse the general behaviour of the GC, for instance, by getting
> an idea of the standard deviation and maximum response time over weeks
> of use. I guess since it can handle much larger loads, the forum server
> is not stressed enough to give very meaningful insight under heavy load.
> But I have experienced a response of several seconds once, I don't know
> where this comes from (although it's most likely one of your reasons).
> So collecting real time data would have to be done for each component of
> the chain (SQLite, cache, D generated page). Such data would be useful
> for any web server anyway.

I doubt that the GC is the source of any real problems. As a web
developer (in PHP no less :S) I can say with authority that most slow
downs are due to waiting on external services and data processing. Some
of the slowest parts of our site are due to heavy database usage where
we have to perform over 100 SQL queries to generate a page. Other
slowdowns are caused by the need to process the data.

I understand that you want to see how the GC affects the server
performance, but it will be so small that it will be lost in the noise,
a blip in traffic because it's lunchtime and people are watching youtube
in your area will cause a bigger delay than the GC kicking in.

--
James Miller
April 13, 2012
Re: Measuring the page generation of the forum
Le 13/04/2012 03:40, Vladimir Panteleev a écrit :
> On Thursday, 12 April 2012 at 18:36:01 UTC, SomeDude wrote:
> There are some numbers in the reddit discussion:
> 
> http://www.reddit.com/r/programming/comments/ppre5/the_new_d_online_forum_software_written_in_d/c3rfwte
> 
Oh, I just saw you wrote the max page generation was around 50 ms.
April 13, 2012
Re: Measuring the page generation of the forum
Le 13/04/2012 04:43, James Miller a écrit :
> I doubt that the GC is the source of any real problems. As a web
> developer (in PHP no less :S) I can say with authority that most slow
> downs are due to waiting on external services and data processing. Some
> of the slowest parts of our site are due to heavy database usage where
> we have to perform over 100 SQL queries to generate a page. Other
> slowdowns are caused by the need to process the data.
> 
> I understand that you want to see how the GC affects the server
> performance, but it will be so small that it will be lost in the noise,
> a blip in traffic because it's lunchtime and people are watching youtube
> in your area will cause a bigger delay than the GC kicking in.
> 
> --
> James Miller

I agree, you're probably right.
« First   ‹ Prev
1 2
Top | Discussion index | About this forum | D home