So the subject implies that the problem is std.stream... but that's not necessarily true.

I'm relatively certain at this point the the freezing issues I've been having (plenty of posts on the dsource mango forum, and "SIGUSR1 Makes Program Hang" here) are due to std.stream... or some interaction between it and mango...  Whenever something std.stream gets linked in (even if it isn't used!) the HTTP server freezes after a certain number of requests. The number of HTTP requests before failing seems to be inversely proportional to the amount of code to process said requests.  That is with all my code, I get about 9 to 24 requests in... with a short "canned" response, and no reading of the request (sometimes mango responds before the client finishes sending!) I'm able to get up to about 1900 requests.... which is one of the reasons this one took so long to track down.

Now that I've figured this out, I'm switch over to all mango IO, which
I've been planning on doing anyway, but I thought I'd let you all know.
I haven't been able to pinpoint what in std.stream is causing it... I just
know that my problem went away when I converted one of my libraries to
mango, and the problem came back when I put xmld back in, which relies on
streams- MemoryStream, and the generic Stream.  The good news is there's a
test case below... the bad news is that it's a little long, and it relies
on Mango (not sure what version... I'm using SVN HEAD) and Andy's xmld,
which is available on his website.  I've also had the problem occur with
some of my own code using std.stream, so I don't think the issue is with
xmld, but I can't get the issue to manifest without it.  Sorry... but it's
as short as I could get it.  Haven't tested it on windows, since I don't
have a working windows box.

Anyway, on the 312th request- every time- all three processes freeze up, and the only way to kill it is to send them all a SIGKILL.  To test it, fire the rig up, and send a couple hundred POST requests to /RPC2.  I've been sending 2500 HTTP POSTs, since sometimes the issue only manifests itself at higher numbers.

I'm running DMD 0.101 (was also an issue with DMD 0.98) on Linux.

John

------------------------------------

import mango.servlet.ServletProvider;
import mango.servlet.Servlet;
import mango.log.Logger;
import mango.log.ConsoleAppender;
import mango.io.Socket;
import mango.http.server.HttpServer;

import xmld.xml;

import std.stream;

XmlNode readString(char[] string)
{
	string ~= " ";
	return readDocument(new MemoryStream(string));
}

char[] smpl = "<methodResponse>
	<params>
		<param>
		<value>
			<struct>
			<member>
				<name>ID</name>
				<value>
				<string>1@1</string>
				</value>
			</member>
			</struct>
		</value>
		</param>
	</params>
</methodResponse>";

char[] output = "<?xml version=\"1.0\"?>
	<methodResponse>
	<params>
		<param>
		<value>
			<struct>
			<member>
				<name>ID</name>
				<value>
				<string>1@1</string>
				</value>
			</member>
			</struct>
		</value>
		</param>
	</params>
</methodResponse>";

class TestServlet : MethodServlet
{
	public override void doPost (IServletRequest request, IServletResponse response)
	{
		try
		{
			XmlNode n = readString(smpl);

			response.setContentType("text/xml");
			response.getWriter().put(output);
		}
		catch (Exception e)
		{
			printf(e.toString() ~ "\n\0");
		}
	}

		public void install(ServletProvider provider)
		{
			IRegisteredServlet irs = provider.addServlet (this, "XmlRpc");
			provider.addMapping ("/RPC2", irs);
		}
}

void main()
{
	Logger hLogger = Logger.getLogger ("HTTP Server");
	hLogger.setLevel(Logger.Level.Trace);
	hLogger.addAppender(new ConsoleAppender());

	InternetAddress bindTo = new InternetAddress(8181);
	ServletProvider sp = new ServletProvider();

	TestServlet ts = new TestServlet();
	ts.install(sp);

	//The HTTP Server
	HttpServer httpServer = new HttpServer(sp, bindTo, 1, hLogger);

	//Go!
	httpServer.start();

	//Wait for input, then close it all up
	stdin.readLine();
}

well naturally I'm curious and will come to the defense of std.stream... I'll poke around some with the example below. I compiled it but when I run it I get a seg-v in _D5mango2io8FilePath8FilePath7getNameFZAa inside of mango's Logger.getLogger. Anyway, I'll keep debugging.

Since the GC deals with SIGUSER1 my assumption is that something is generating garbage like mad and kicking off a garbage collection. I'll try sprinkling in calls to gcstats and printing out the results and see what is going on memory-wise.

teqDruid wrote:

> So the subject implies that the problem is std.stream... but that's not necessarily true.
> 
> I'm relatively certain at this point the the freezing issues I've been having (plenty of posts on the dsource mango forum, and "SIGUSR1 Makes Program Hang" here) are due to std.stream... or some interaction between it and mango...  Whenever something std.stream gets linked in (even if it isn't used!) the HTTP server freezes after a certain number of requests. The number of HTTP requests before failing seems to be inversely proportional to the amount of code to process said requests.  That is with all my code, I get about 9 to 24 requests in... with a short "canned" response, and no reading of the request (sometimes mango responds before the client finishes sending!) I'm able to get up to about 1900 requests.... which is one of the reasons this one took so long to track down.
> 
> Now that I've figured this out, I'm switch over to all mango IO, which
> I've been planning on doing anyway, but I thought I'd let you all know.
> I haven't been able to pinpoint what in std.stream is causing it... I just
> know that my problem went away when I converted one of my libraries to
> mango, and the problem came back when I put xmld back in, which relies on
> streams- MemoryStream, and the generic Stream.  The good news is there's a
> test case below... the bad news is that it's a little long, and it relies
> on Mango (not sure what version... I'm using SVN HEAD) and Andy's xmld,
> which is available on his website.  I've also had the problem occur with
> some of my own code using std.stream, so I don't think the issue is with
> xmld, but I can't get the issue to manifest without it.  Sorry... but it's
> as short as I could get it.  Haven't tested it on windows, since I don't
> have a working windows box.
> 
> Anyway, on the 312th request- every time- all three processes freeze up, and the only way to kill it is to send them all a SIGKILL.  To test it, fire the rig up, and send a couple hundred POST requests to /RPC2.  I've been sending 2500 HTTP POSTs, since sometimes the issue only manifests itself at higher numbers.
> 
> I'm running DMD 0.101 (was also an issue with DMD 0.98) on Linux.
> 
> John
> 
> ------------------------------------
> 
> import mango.servlet.ServletProvider;
> import mango.servlet.Servlet;
> import mango.log.Logger;
> import mango.log.ConsoleAppender;
> import mango.io.Socket;
> import mango.http.server.HttpServer;
> 
> import xmld.xml;
> 
> import std.stream;
> 
> XmlNode readString(char[] string)
> {
> string ~= " ";
> return readDocument(new MemoryStream(string));
> }
> 
> char[] smpl = "<methodResponse>
> <params>
> <param>
> <value>
> <struct>
> <member>
> <name>ID</name>
> <value>
> <string>1@1</string>
> </value>
> </member>
> </struct>
> </value>
> </param>
> </params>
> </methodResponse>";
> 
> char[] output = "<?xml version=\"1.0\"?>
> <methodResponse>
> <params>
> <param>
> <value>
> <struct>
> <member>
> <name>ID</name>
> <value>
> <string>1@1</string>
> </value>
> </member>
> </struct>
> </value>
> </param>
> </params>
> </methodResponse>";
> 
> class TestServlet : MethodServlet
> {
> public override void doPost (IServletRequest request, IServletResponse
> response) {
> try
> {
> XmlNode n = readString(smpl);
> 
> response.setContentType("text/xml");
> response.getWriter().put(output);
> }
> catch (Exception e)
> {
> printf(e.toString() ~ "\n\0");
> }
> }
> 
> public void install(ServletProvider provider)
> {
> IRegisteredServlet irs = provider.addServlet (this, "XmlRpc");
> provider.addMapping ("/RPC2", irs);
> }
> }
> 
> void main()
> {
> Logger hLogger = Logger.getLogger ("HTTP Server");
> hLogger.setLevel(Logger.Level.Trace);
> hLogger.addAppender(new ConsoleAppender());
> 
> InternetAddress bindTo = new InternetAddress(8181);
> ServletProvider sp = new ServletProvider();
> 
> TestServlet ts = new TestServlet();
> ts.install(sp);
> 
> //The HTTP Server
> HttpServer httpServer = new HttpServer(sp, bindTo, 1, hLogger);
> 
> //Go!
> httpServer.start();
> 
> //Wait for input, then close it all up
> stdin.readLine();
> }

I got the example to compile and run. It sits there with
INFO HTTP Server - Server http::Servlet started on 0.0.0.0:8181 with 1
accept threads, 10 backlogs
until I hit enter and then it seg-v's. Is this what is supposed to happen? I
don't understand what it is doing or what it is supposed to be doing. How
do I reproduce the bug?

-Ben

well, you'll have to tell me how to send 1900 POST requests. When I point my
browser at http://localhost:8181 I get a 404 in the browser and "closing
socket handle... socket handle closed" echoed to the console of the server.
So something is happening but I have no idea how to use this server
(needless to say I'm not server-savvy).
Without more hints there isn't much more I can do to debug.

-Ben

On Mon, 13 Sep 2004 21:50:34 -0400, Ben Hinkle <bhinkle4@juno.com> wrote:
> I got the example to compile and run.

You got further than I did. I'd like to help too, however, mine simply says 'Access Violation' when I run it, I don't think I compiled it right. :)

I have the latest mango and xmld from andy's site. I need to produce a lib for each of them right, how?

> It sits there with
> INFO HTTP Server - Server http::Servlet started on 0.0.0.0:8181 with 1
> accept threads, 10 backlogs
> until I hit enter and then it seg-v's. Is this what is supposed to
> happen? I
> don't understand what it is doing or what it is supposed to be doing. How
> do I reproduce the bug?

I suspect you're supposed to connect and send a web POST request, attached is a Windows console program I wrote it's really quite neat/useful for debugging tcpip stuff.

Included in the zip you'll find post.txt and instructions.txt. To run it type:

   tcpbot post.txt localhost 8181

it will attempt to connect to localhost on port 8181 the it will carry out the instructions in post.txt.

See instructions.txt for a description of how to format post.txt - it's a rough guide.

Regan

-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/

note: one thing I noticed is that mango has destructors that reference other objects. This means when the app can do random things (though I've always just gotten a seg-v) during a GC. Destructors should only reference the object being destroyed or system resources (ie non-GC resources). Since a SIGUSR1 is indicating a paused thread during a GC I now even more strongly suspect the problem occurs during GC.

teqDruid wrote:

> So the subject implies that the problem is std.stream... but that's not necessarily true.
> 
> I'm relatively certain at this point the the freezing issues I've been having (plenty of posts on the dsource mango forum, and "SIGUSR1 Makes Program Hang" here) are due to std.stream... or some interaction between it and mango...  Whenever something std.stream gets linked in (even if it isn't used!) the HTTP server freezes after a certain number of requests. The number of HTTP requests before failing seems to be inversely proportional to the amount of code to process said requests.  That is with all my code, I get about 9 to 24 requests in... with a short "canned" response, and no reading of the request (sometimes mango responds before the client finishes sending!) I'm able to get up to about 1900 requests.... which is one of the reasons this one took so long to track down.
> 
> Now that I've figured this out, I'm switch over to all mango IO, which
> I've been planning on doing anyway, but I thought I'd let you all know.
> I haven't been able to pinpoint what in std.stream is causing it... I just
> know that my problem went away when I converted one of my libraries to
> mango, and the problem came back when I put xmld back in, which relies on
> streams- MemoryStream, and the generic Stream.  The good news is there's a
> test case below... the bad news is that it's a little long, and it relies
> on Mango (not sure what version... I'm using SVN HEAD) and Andy's xmld,
> which is available on his website.  I've also had the problem occur with
> some of my own code using std.stream, so I don't think the issue is with
> xmld, but I can't get the issue to manifest without it.  Sorry... but it's
> as short as I could get it.  Haven't tested it on windows, since I don't
> have a working windows box.
> 
> Anyway, on the 312th request- every time- all three processes freeze up, and the only way to kill it is to send them all a SIGKILL.  To test it, fire the rig up, and send a couple hundred POST requests to /RPC2.  I've been sending 2500 HTTP POSTs, since sometimes the issue only manifests itself at higher numbers.
> 
> I'm running DMD 0.101 (was also an issue with DMD 0.98) on Linux.
> 
> John
> 
> ------------------------------------
> 
> import mango.servlet.ServletProvider;
> import mango.servlet.Servlet;
> import mango.log.Logger;
> import mango.log.ConsoleAppender;
> import mango.io.Socket;
> import mango.http.server.HttpServer;
> 
> import xmld.xml;
> 
> import std.stream;
> 
> XmlNode readString(char[] string)
> {
> string ~= " ";
> return readDocument(new MemoryStream(string));
> }
> 
> char[] smpl = "<methodResponse>
> <params>
> <param>
> <value>
> <struct>
> <member>
> <name>ID</name>
> <value>
> <string>1@1</string>
> </value>
> </member>
> </struct>
> </value>
> </param>
> </params>
> </methodResponse>";
> 
> char[] output = "<?xml version=\"1.0\"?>
> <methodResponse>
> <params>
> <param>
> <value>
> <struct>
> <member>
> <name>ID</name>
> <value>
> <string>1@1</string>
> </value>
> </member>
> </struct>
> </value>
> </param>
> </params>
> </methodResponse>";
> 
> class TestServlet : MethodServlet
> {
> public override void doPost (IServletRequest request, IServletResponse
> response) {
> try
> {
> XmlNode n = readString(smpl);
> 
> response.setContentType("text/xml");
> response.getWriter().put(output);
> }
> catch (Exception e)
> {
> printf(e.toString() ~ "\n\0");
> }
> }
> 
> public void install(ServletProvider provider)
> {
> IRegisteredServlet irs = provider.addServlet (this, "XmlRpc");
> provider.addMapping ("/RPC2", irs);
> }
> }
> 
> void main()
> {
> Logger hLogger = Logger.getLogger ("HTTP Server");
> hLogger.setLevel(Logger.Level.Trace);
> hLogger.addAppender(new ConsoleAppender());
> 
> InternetAddress bindTo = new InternetAddress(8181);
> ServletProvider sp = new ServletProvider();
> 
> TestServlet ts = new TestServlet();
> ts.install(sp);
> 
> //The HTTP Server
> HttpServer httpServer = new HttpServer(sp, bindTo, 1, hLogger);
> 
> //Go!
> httpServer.start();
> 
> //Wait for input, then close it all up
> stdin.readLine();
> }

Yes... after firing it up, throw a couple hundred or thousand HTTP POSTS at it.  There's a sample HTTP Post that I've been throwing at it on the mango forum. The stdin.readLine is there to terminate the program- once one is done with it.  The segfault is a bug in mango's socket stuff, and is unrelated. Once the bug occurs, one cannot simply hit enter to kill it, as it is frozen.

> I have the latest mango and xmld from andy's site. I need to produce a lib for each of them right, how?

I extracted xmld into mango's directory, and added it to the mango makefile.  This way, all one has to do is run the mango makefile, and link the program with the mango lib, and xmld is automagically there.  Plus, it's only one more source path to give to dmd, if you're compiling the test file outside the mango directory.

John

On Tue, 14 Sep 2004 17:47:05 +1200, Regan Heath wrote:

> On Mon, 13 Sep 2004 21:50:34 -0400, Ben Hinkle <bhinkle4@juno.com> wrote:
>> I got the example to compile and run.
> 
> You got further than I did. I'd like to help too, however, mine simply says 'Access Violation' when I run it, I don't think I compiled it right. :)
> 
> I have the latest mango and xmld from andy's site. I need to produce a lib for each of them right, how?
> 
>> It sits there with
>> INFO HTTP Server - Server http::Servlet started on 0.0.0.0:8181 with 1
>> accept threads, 10 backlogs
>> until I hit enter and then it seg-v's. Is this what is supposed to
>> happen? I
>> don't understand what it is doing or what it is supposed to be doing. How
>> do I reproduce the bug?
> 
> I suspect you're supposed to connect and send a web POST request, attached is a Windows console program I wrote it's really quite neat/useful for debugging tcpip stuff.
> 
> Included in the zip you'll find post.txt and instructions.txt. To run it type:
> 
>    tcpbot post.txt localhost 8181
> 
> it will attempt to connect to localhost on port 8181 the it will carry out the instructions in post.txt.
> 
> See instructions.txt for a description of how to format post.txt - it's a rough guide.
> 
> Regan

Just write a quick program to open a connection to port 8181, and have it send something like the following:
---------------------------
POST /RPC2 HTTP/1.1
Content-Length: 157
Content-Type: text/xml
Cache-Control: no-cache
Pragma: no-cache
User-Agent: Java/1.4.2_05
Host: localhost:8181
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

<?xml version="1.0" encoding="ISO-8859-1"?><methodCall><methodName>nnCore.getNode</methodName><params><param><value>1@1</value></param></params></methodCall>
------------------------------

As you can tell, I'm using Java to send it.  I'd send you my app, but actually I'm using an XML-RPC library to send it, though I don't believe it need be that complicated.

I'm pretty sure the content is irrevelent, but that's what I've been testing it with.

You may also need to have your app read the response from the server... Mine does, but I'm not sure if it's necessary.

John

On Mon, 13 Sep 2004 22:35:38 -0400, Ben Hinkle wrote:

> well, you'll have to tell me how to send 1900 POST requests. When I point my
> browser at http://localhost:8181 I get a 404 in the browser and "closing
> socket handle... socket handle closed" echoed to the console of the server.
> So something is happening but I have no idea how to use this server
> (needless to say I'm not server-savvy).
> Without more hints there isn't much more I can do to debug.
> 
> -Ben

teqDruid wrote:

> Just write a quick program to open a connection to port 8181, and have it send something like the following:
> ---------------------------
> POST /RPC2 HTTP/1.1
> Content-Length: 157
> Content-Type: text/xml
> Cache-Control: no-cache
> Pragma: no-cache
> User-Agent: Java/1.4.2_05
> Host: localhost:8181
> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
> Connection: keep-alive
> 
> <?xml version="1.0"
>
encoding="ISO-8859-1"?><methodCall><methodName>nnCore.getNode</methodName><params><param><value>1@1</value></param></params></methodCall>
> ------------------------------

I have no idea how to do that. Please post all the code required to reproduce the bug or send me something bhinkle4 at juno.com (if you are interested in pursuing this that is).

> As you can tell, I'm using Java to send it.  I'd send you my app, but actually I'm using an XML-RPC library to send it, though I don't believe it need be that complicated.
> 
> I'm pretty sure the content is irrevelent, but that's what I've been testing it with.
> 
> You may also need to have your app read the response from the server... Mine does, but I'm not sure if it's necessary.

Don't underestimate my lack of knowledge about debugging web servers. :-)
The word "may" makes me nervous because that implies a decision I don't
know how to make.

> John
> 
> On Mon, 13 Sep 2004 22:35:38 -0400, Ben Hinkle wrote:
> 
>> well, you'll have to tell me how to send 1900 POST requests. When I point
>> my browser at http://localhost:8181 I get a 404 in the browser and
>> "closing socket handle... socket handle closed" echoed to the console of
>> the server. So something is happening but I have no idea how to use this
>> server (needless to say I'm not server-savvy).
>> Without more hints there isn't much more I can do to debug.
>> 
>> -Ben

On Tue, 14 Sep 2004 14:56:56 -0400, teqDruid <me@teqdruid.com> wrote:
> Yes... after firing it up, throw a couple hundred or thousand HTTP POSTS
> at it.  There's a sample HTTP Post that I've been throwing at it on the
> mango forum. The stdin.readLine is there to terminate the program- once
> one is done with it.  The segfault is a bug in mango's socket stuff, and
> is unrelated. Once the bug occurs, one cannot simply hit enter to kill it,
> as it is frozen.
>
>> I have the latest mango and xmld from andy's site. I need to produce a lib
>> for each of them right, how?
>
> I extracted xmld into mango's directory, and added it to the mango
> makefile.  This way, all one has to do is run the mango makefile, and link
> the program with the mango lib, and xmld is automagically there.

Good idea.. first I have to get mango to compile however.

> Plus,
> it's only one more source path to give to dmd, if you're compiling the
> test file outside the mango directory.

I tried..

D:\D\src\mango>d:\d\dm\bin\make -f win32.make lib
d:\d\dm\bin\lib -c -n -p256 obj/mango.lib @obj/mango.rsp
Digital Mars Librarian Version 8.00n
Copyright (C) Digital Mars 2000-2002 All Rights Reserved www.digitalmars.com

Error: cannot open response file

--- errorlevel 1

the file 'obj/mango.rsp' does not exist.

Regan

> John
>
> On Tue, 14 Sep 2004 17:47:05 +1200, Regan Heath wrote:
>
>> On Mon, 13 Sep 2004 21:50:34 -0400, Ben Hinkle <bhinkle4@juno.com> wrote:
>>> I got the example to compile and run.
>>
>> You got further than I did. I'd like to help too, however, mine simply
>> says 'Access Violation' when I run it, I don't think I compiled it right.
>> :)
>>
>> I have the latest mango and xmld from andy's site. I need to produce a lib
>> for each of them right, how?
>>
>>> It sits there with
>>> INFO HTTP Server - Server http::Servlet started on 0.0.0.0:8181 with 1
>>> accept threads, 10 backlogs
>>> until I hit enter and then it seg-v's. Is this what is supposed to
>>> happen? I
>>> don't understand what it is doing or what it is supposed to be doing. How
>>> do I reproduce the bug?
>>
>> I suspect you're supposed to connect and send a web POST request, attached
>> is a Windows console program I wrote it's really quite neat/useful for
>> debugging tcpip stuff.
>>
>> Included in the zip you'll find post.txt and instructions.txt.
>> To run it type:
>>
>>    tcpbot post.txt localhost 8181
>>
>> it will attempt to connect to localhost on port 8181 the it will carry out
>> the instructions in post.txt.
>>
>> See instructions.txt for a description of how to format post.txt - it's a
>> rough guide.
>>
>> Regan
>



-- 
Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/