Hi guys,

I've only been trying to learn D for a short while, and compared to the other more popular programming languages, I've been able to find very little helpful and updated documentation or tutorials on it.

I basically need help getting data from a website, just the HTML of a webpage, in this case a PHP page. I'm using the following code which compiles fine, but when run it just encounters a 'Socket.ERROR' with a value of -1.

I'm not sure what I'm doing wrong, but any help or suggestions would be appreciated. As I said, I'm quite new to D...

---
//readData.d
module main;

import std.stdio;
import std.socket;
import std.cstream;

int main(string[] args)
{
	string host = "www.dprogramming.com";
	ushort port = 80;
	
	Socket listener = new TcpSocket;
        assert(listener.isAlive);
        listener.blocking = false;
	
	listener.connect(new InternetAddress(host, port));
	
	char[] msg;
	char[] req = cast(char[]) "GET /internet.php HTTP/1.1\r\nHost: www.dprogramming.com\r\n\r\n";
	
	listener.send(req);
	
	while (1)
	{
		int bytes;
		char[1024] buf;
		bytes = listener.receive(buf);
		if (Socket.ERROR == bytes)
		{
			writeln("Connection Error ", Socket.ERROR);
			listener.close();
			din.readLine();
			return 1;
		}
		else if (0 == bytes)
		{
			try
			{
				writeln("Connection to target lost.");
			}
			catch (SocketException)
			{
				writeln("Connection closed.");
			}
			listener.close();
			din.readLine();
			return 1;
		}
		else
		{
			writeln("Recieved ", bytes, "bytes from ", listener.remoteAddress().toString(), ": \"", buf[0 .. bytes], "\"");
		}
	}
}
---

- Alex.

On Tuesday, 9 July 2013 at 09:03:32 UTC, AlexMcP wrote:
> 			writeln("Connection Error ", Socket.ERROR);

First tip: Socket.ERROR is a constant, so printing it doesn't  help much (as you probably noticed). More helpful is the function lastSocketError(), which returns a string with some details.

                        writeln("Connection Error ", Socket.ERROR, " ", lastSocketError());

Which tells you:

Connection Error -1 Resource temporarily unavailable



The reason that is an error is because you turned off blocking.

        listener.blocking = false;


So, since the connection hasn't had a chance to get established and transfer data yet, instead of waiting for it to finish when you called receive, it returned the error telling you to try again later.

The simplest thing to do here is to comment out that blocking = false line, letting receive just wait until data is available. Another option is to use Socket.select to notify you when data is available, and only receive then.

After that, make sure the req literal is all on one line to avoid http 400 errors (not sure if that was a result of copy/paste or not, probably) and you should get a response.


Another note: casting a string literal to char[] is usually a bad idea, because you can't write to them anyway. Best to just leave them typed as string. It should all still work the same after doing that.



Finally, if you want a higher level library to do this, check out std.net.curl: http://dlang.org/phobos/std_net_curl.html

BTW I also wrote a more complete lower level implementation too:

https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/http.d

My code is kinda ugly, but might be useful to refer to too. http has some other features you'll need to think about, like chunked data transfer, to have a more complete implementation, and I have some openssl code in there too. Of course, curl takes care of all this too.

Thank you! It worked, finally, and I can't believe that was the only thing stopping it from working. But yeah, I'll take your advice and make the changes.

I've also looked at curl and your implementation, but I wanted to give this a go manually instead of just using external libraries, I feel it's just better to learn how to do it all from scratch.

So thank you again, and you can probably expect me to be back here in the near future!

- Alex.