Hey guys I wanted to discuss my first major depressing experience with D in a reallife project.

First off here is the according bug report : http://d.puremagic.com/issues/show_bug.cgi?id=4951

I have to mention that it was by far not the first bug i encountered in D but it has never been such a PITA. I spent hours and hours of searching, rewriting and crawling my code for the cause of this issue. Just to finally find out that it was not my code but some bug in druntime.

What i do:
I spawn multiple threads all periodically connecting to some webservers waiting for the request to finish. Every request is totally independent of the others. There is no data sharing in my code. Now i have a protocol that makes my process send one last important request when shutting down. For a clean shut down i wanted to make sure that all the other threads are shut down first. That way the final request of the process is the very last. So i wait for every thread to finish their request and stop them.
BUT it wont work like that. AFTER i stopped all other threads some strange behavior of druntime makes every creation of an InternetAddress instance (internally trying to resolve host) impossible (throws an exception). Additionally even if the Internet Addresses were created upfront the TcpSockets wont connect but throw.
I have no idea what the heck is going on inside of the druntime that could cause this but in my opinion it makes designing a big server environment pretty much impossible because it would mean that i cannot let any of my user generated threads exit unless i want to shutdown completely anyway.

Perhaps one side note: I am testing and having these issues under Windows.


I have never been so tempted to use another language while coding in D as i was this time and that makes me sad.


- Stephan


(Sorry for reposting but it belongs to this NG and not the Announcement NG)

On Tue, 19 Oct 2010 20:33:29 +0900, Stephan <spam@extrawurst.org> wrote:

> Hey guys I wanted to discuss my first major depressing experience with D in a reallife project.
>
> First off here is the according bug report : http://d.puremagic.com/issues/show_bug.cgi?id=4951

I think this bug is related to http://d.puremagic.com/issues/show_bug.cgi?id=4344
I replied similar post in NG.
http://lists.puremagic.com/pipermail/digitalmars-d/2010-September/082814.html

I want to know the best solution.

On Tue, 19 Oct 2010 07:33:29 -0400, Stephan <spam@extrawurst.org> wrote:

> Hey guys I wanted to discuss my first major depressing experience with D in a reallife project.
>
> First off here is the according bug report : http://d.puremagic.com/issues/show_bug.cgi?id=4951
>
> I have to mention that it was by far not the first bug i encountered in D but it has never been such a PITA. I spent hours and hours of searching, rewriting and crawling my code for the cause of this issue. Just to finally find out that it was not my code but some bug in druntime.
>
> What i do:
> I spawn multiple threads all periodically connecting to some webservers waiting for the request to finish. Every request is totally independent of the others. There is no data sharing in my code. Now i have a protocol that makes my process send one last important request when shutting down. For a clean shut down i wanted to make sure that all the other threads are shut down first. That way the final request of the process is the very last. So i wait for every thread to finish their request and stop them.
> BUT it wont work like that. AFTER i stopped all other threads some strange behavior of druntime makes every creation of an InternetAddress instance (internally trying to resolve host) impossible (throws an exception). Additionally even if the Internet Addresses were created upfront the TcpSockets wont connect but throw.
> I have no idea what the heck is going on inside of the druntime that could cause this but in my opinion it makes designing a big server environment pretty much impossible because it would mean that i cannot let any of my user generated threads exit unless i want to shutdown completely anyway.
>
> Perhaps one side note: I am testing and having these issues under Windows.
>
>
> I have never been so tempted to use another language while coding in D as i was this time and that makes me sad.

That is unfortunate.  These kinds of bugs need to be fixed ASAP, and they should be much easier than compiler bugs.

The issue at the moment is that nobody owns or wants to own the network API portion of phobos (it's not druntime btw).  So your pleas are falling on sympathetic ears, but not ones that will make changes.  These growing pains will pass, I think some have mentioned wanting to rewrite the network API, but streams in general are up in the air at the moment...

I'll try and spend a few minutes to see if I can spot the problem.  Having a minimal test case as you have posted helps.

-Steve

On 19.10.2010 14:22, Steven Schveighoffer wrote:
> On Tue, 19 Oct 2010 07:33:29 -0400, Stephan <spam@extrawurst.org> wrote:
>
>> Hey guys I wanted to discuss my first major depressing experience with
>> D in a reallife project.
>>
>> First off here is the according bug report :
>> http://d.puremagic.com/issues/show_bug.cgi?id=4951
>>
>> I have to mention that it was by far not the first bug i encountered
>> in D but it has never been such a PITA. I spent hours and hours of
>> searching, rewriting and crawling my code for the cause of this issue.
>> Just to finally find out that it was not my code but some bug in
>> druntime.
>>
>> What i do:
>> I spawn multiple threads all periodically connecting to some
>> webservers waiting for the request to finish. Every request is totally
>> independent of the others. There is no data sharing in my code. Now i
>> have a protocol that makes my process send one last important request
>> when shutting down. For a clean shut down i wanted to make sure that
>> all the other threads are shut down first. That way the final request
>> of the process is the very last. So i wait for every thread to finish
>> their request and stop them.
>> BUT it wont work like that. AFTER i stopped all other threads some
>> strange behavior of druntime makes every creation of an
>> InternetAddress instance (internally trying to resolve host)
>> impossible (throws an exception). Additionally even if the Internet
>> Addresses were created upfront the TcpSockets wont connect but throw.
>> I have no idea what the heck is going on inside of the druntime that
>> could cause this but in my opinion it makes designing a big server
>> environment pretty much impossible because it would mean that i cannot
>> let any of my user generated threads exit unless i want to shutdown
>> completely anyway.
>>
>> Perhaps one side note: I am testing and having these issues under
>> Windows.
>>
>>
>> I have never been so tempted to use another language while coding in D
>> as i was this time and that makes me sad.
>
> That is unfortunate. These kinds of bugs need to be fixed ASAP, and they
> should be much easier than compiler bugs.
>
> The issue at the moment is that nobody owns or wants to own the network
> API portion of phobos (it's not druntime btw). So your pleas are falling
> on sympathetic ears, but not ones that will make changes. These growing
> pains will pass, I think some have mentioned wanting to rewrite the
> network API, but streams in general are up in the air at the moment...
>
> I'll try and spend a few minutes to see if I can spot the problem.
> Having a minimal test case as you have posted helps.
>
> -Steve

Thanks for the reply and that you gonna take a look. I added the relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to the bug aswell because it reallys seems to have the same underlying root.

- Stephan

On Tue, 19 Oct 2010 08:35:09 -0400, Stephan <spam@extrawurst.org> wrote:
>
> Thanks for the reply and that you gonna take a look. I added the relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to the bug aswell because it reallys seems to have the same underlying root.

I fixed both with the suggested update in 4344.  I also found a race condition on Linux that as far as I know was never reported.

See changeset http://www.dsource.org/projects/phobos/changeset/2107

That was an easy one :)

-Steve

On 19.10.2010 15:27, Steven Schveighoffer wrote:
> On Tue, 19 Oct 2010 08:35:09 -0400, Stephan <spam@extrawurst.org> wrote:
>>
>> Thanks for the reply and that you gonna take a look. I added the
>> relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to
>> the bug aswell because it reallys seems to have the same underlying root.
>
> I fixed both with the suggested update in 4344. I also found a race
> condition on Linux that as far as I know was never reported.
>
> See changeset http://www.dsource.org/projects/phobos/changeset/2107
>
> That was an easy one :)
>
> -Steve

Awesome, thank you.
Is there a new release planned anytime soon that includes the patch ?

- Stephan

On Tue, 19 Oct 2010 23:02:31 +0900, Stephan <spam@extrawurst.org> wrote:

> On 19.10.2010 15:27, Steven Schveighoffer wrote:
>> On Tue, 19 Oct 2010 08:35:09 -0400, Stephan <spam@extrawurst.org> wrote:
>>>
>>> Thanks for the reply and that you gonna take a look. I added the
>>> relation of bug http://d.puremagic.com/issues/show_bug.cgi?id=4344 to
>>> the bug aswell because it reallys seems to have the same underlying root.
>>
>> I fixed both with the suggested update in 4344. I also found a race
>> condition on Linux that as far as I know was never reported.
>>
>> See changeset http://www.dsource.org/projects/phobos/changeset/2107
>>
>> That was an easy one :)
>>
>> -Steve
>
> Awesome, thank you.
> Is there a new release planned anytime soon that includes the patch ?
>
> - Stephan

Discussion is here.
http://lists.puremagic.com/pipermail/phobos/2010-October/002957.html


Masahiro

On 19.10.2010 13:33, Stephan wrote:
> BUT it wont work like that. AFTER i stopped all other threads some strange behavior of druntime makes every creation of an InternetAddress instance (internally trying to resolve host) impossible (throws an exception). Additionally even if the Internet Addresses were created upfront the TcpSockets wont connect but throw.

Terminating threads has always been iffy in D. As a simple workaround, use a threadpool to keep a set of threads around blocking on a semaphore (and consuming no resources), and rewrite your problem in terms of tasks (void delegate()).

I'm sorry you had such a bad experience with this. There really should be a list of common D pitfalls, and "Don't ever destroy threads" should be among the top ten.

On Tue, 19 Oct 2010 12:07:04 -0400, FeepingCreature <default_357-line@yahoo.de> wrote:

> On 19.10.2010 13:33, Stephan wrote:
>> BUT it wont work like that. AFTER i stopped all other threads some
>> strange behavior of druntime makes every creation of an InternetAddress
>> instance (internally trying to resolve host) impossible (throws an
>> exception). Additionally even if the Internet Addresses were created
>> upfront the TcpSockets wont connect but throw.
>
> Terminating threads has always been iffy in D. As a simple workaround, use a threadpool to keep a set of threads around blocking on a semaphore (and consuming no resources), and rewrite your problem in terms of tasks (void delegate()).
>
> I'm sorry you had such a bad experience with this. There really should be a list of common D pitfalls, and "Don't ever destroy threads" should be among the top ten.

This one was a simple bug.  He was not terminating threads, rather he was letting a thread exit normally.

-Steve

On 10/19/10 11:07 CDT, FeepingCreature wrote:
> On 19.10.2010 13:33, Stephan wrote:
>> BUT it wont work like that. AFTER i stopped all other threads some
>> strange behavior of druntime makes every creation of an
>> InternetAddress instance (internally trying to resolve host)
>> impossible (throws an exception). Additionally even if the Internet
>> Addresses were created upfront the TcpSockets wont connect but
>> throw.
>
> Terminating threads has always been iffy in D.

Terminating threads has always been iffy in all languages that use native threads. I've discussed with Sean a couple of times about ways to define a little protocol and API for orderly thread termination. We need something like a callback mechanism - atShutdown().

Andrei