Thank you very much. I should invite you to a beer ;-)

For the other hand,

I get this error:

[Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640): Can't convert value `HTT' of type string to type uint]


if I only want the length:

//D 2.0
//gdmd-4.6 <fitxer> dhttpclient => surt el fitxer amb el mateix nom i .o
//Usa https://github.com/Bystroushaak/DHTTPClient
//versió 0.0.2
import std.stdio, std.string, std.conv, std.stream;
import std.socket, std.socketstream;
import dhttpclient;

int main(string [] args)
{
    if (args.length < 2) {
		writeln("Usage:");
		writeln("   ./spider {<url1>, <url2>, ...}");
		return 0;
	}
	else {
		try {
			string[string] capcalera = dhttpclient.FFHeaders;
			HTTPClient navegador = new HTTPClient();
			navegador.setClientHeaders(capcalera);

			foreach (a; args[1..$]) {
				auto tamany = cast(ubyte[]) navegador.get(a);
				writeln("[Contingut: ", tamany.length, "]");
			}
		}
		catch (Exception e) {
			writeln("[Excepció: ", e, "]");
		}
		return 0;
	}
}


In theory, tamany.length is completely defined.

Xan.

2012/1/20 Bystroushaak <bystrousak@kitakitsune.org>:
> rawWrite():
>
> stdout.rawWrite(cast(ubyte[]) navegador.get(a));
>
>
> On 20.1.2012 18:18, Xan xan wrote:
>>
>> Mmmm... I understand it. But is there any way of circumvent it? Perhaps I could write to one file, isn't?
>>
>>
>>
>> 2012/1/20 Bystroushaak<bystrousak@kitakitsune.org>:
>>>
>>> Thats because you are trying writeln binary data, and that is impossible, because writeln IMHO checks UTF8 validity.
>>>
>>>
>>> On 20.1.2012 18:08, Xan xan wrote:
>>>>
>>>>
>>>> Before and now, I get this error:
>>>>
>>>> $ ./spider http://static.arxiv.org/pdf/1109.4897.pdf
>>>> [Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640):
>>>> Can't convert value `HTT' of type string to type uint]
>>>>
>>>> The code:
>>>>
>>>> //D 2.0
>>>> //gdmd-4.6<fitxer>    =>    surt el fitxer amb el mateix nom i .o
>>>> //Usa https://github.com/Bystroushaak/DHTTPClient
>>>> import std.stdio, std.string, std.conv, std.stream;
>>>> import std.socket, std.socketstream;
>>>> import dhttpclient;
>>>>
>>>> int main(string [] args)
>>>> {
>>>>     if (args.length<    2) {
>>>>                writeln("Usage:");
>>>>                writeln("   ./spider {<url1>,<url2>, ...}");
>>>>                return 0;
>>>>        }
>>>>        else {
>>>>                try {
>>>>                        string[string] capcalera = dhttpclient.FFHeaders;
>>>>                        //capcalera["User-Agent"] = "arachnida yottiuma";
>>>>                        HTTPClient navegador = new HTTPClient();
>>>>                        navegador.setClientHeaders(capcalera);
>>>>
>>>>                        foreach (a; args[1..$]) {
>>>>                                writeln("[Contingut: ", cast(ubyte[])
>>>> navegador.get(a), "]");
>>>>                        }
>>>>                }
>>>>                catch (Exception e) {
>>>>                        writeln("[Excepció: ", e, "]");
>>>>                }
>>>>                return 0;
>>>>        }
>>>> }
>>>>
>>>>
>>>>
>>>> What happens?
>>>>
>>>>
>>>> 2012/1/20 Bystroushaak<bystrousak@kitakitsune.org>:
>>>>>
>>>>>
>>>>> It is unlimited, you just have to cast output to ubyte[]:
>>>>>
>>>>> std.file.write("logo3w.png", cast(ubyte[])
>>>>> cl.get("http://www.google.cz/images/srpr/logo3w.png"));
>>>>>
>>>>>
>>>
>

The same error with:
[...]
foreach (a; args[1..$]) {
|___|___|___|___write("[Longitud: ");
|___|___|___|___stdout.rawWrite(cast(ubyte[]) navegador.get(a));
|___|___|___|___writeln("]");
|___|___|___}
[...]

2012/1/20 Bystroushaak <bystrousak@kitakitsune.org>:
> rawWrite():
>
> stdout.rawWrite(cast(ubyte[]) navegador.get(a));
>
>
> On 20.1.2012 18:18, Xan xan wrote:
>>
>> Mmmm... I understand it. But is there any way of circumvent it? Perhaps I could write to one file, isn't?
>>
>>
>>
>> 2012/1/20 Bystroushaak<bystrousak@kitakitsune.org>:
>>>
>>> Thats because you are trying writeln binary data, and that is impossible, because writeln IMHO checks UTF8 validity.
>>>
>>>
>>> On 20.1.2012 18:08, Xan xan wrote:
>>>>
>>>>
>>>> Before and now, I get this error:
>>>>
>>>> $ ./spider http://static.arxiv.org/pdf/1109.4897.pdf
>>>> [Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640):
>>>> Can't convert value `HTT' of type string to type uint]
>>>>
>>>> The code:
>>>>
>>>> //D 2.0
>>>> //gdmd-4.6<fitxer>    =>    surt el fitxer amb el mateix nom i .o
>>>> //Usa https://github.com/Bystroushaak/DHTTPClient
>>>> import std.stdio, std.string, std.conv, std.stream;
>>>> import std.socket, std.socketstream;
>>>> import dhttpclient;
>>>>
>>>> int main(string [] args)
>>>> {
>>>>     if (args.length<    2) {
>>>>                writeln("Usage:");
>>>>                writeln("   ./spider {<url1>,<url2>, ...}");
>>>>                return 0;
>>>>        }
>>>>        else {
>>>>                try {
>>>>                        string[string] capcalera = dhttpclient.FFHeaders;
>>>>                        //capcalera["User-Agent"] = "arachnida yottiuma";
>>>>                        HTTPClient navegador = new HTTPClient();
>>>>                        navegador.setClientHeaders(capcalera);
>>>>
>>>>                        foreach (a; args[1..$]) {
>>>>                                writeln("[Contingut: ", cast(ubyte[])
>>>> navegador.get(a), "]");
>>>>                        }
>>>>                }
>>>>                catch (Exception e) {
>>>>                        writeln("[Excepció: ", e, "]");
>>>>                }
>>>>                return 0;
>>>>        }
>>>> }
>>>>
>>>>
>>>>
>>>> What happens?
>>>>
>>>>
>>>> 2012/1/20 Bystroushaak<bystrousak@kitakitsune.org>:
>>>>>
>>>>>
>>>>> It is unlimited, you just have to cast output to ubyte[]:
>>>>>
>>>>> std.file.write("logo3w.png", cast(ubyte[])
>>>>> cl.get("http://www.google.cz/images/srpr/logo3w.png"));
>>>>>
>>>>>
>>>
>

On 20.1.2012 18:42, Xan xan wrote:
> Thank you very much. I should invite you to a beer ;-)

Write me if you will be in prag/czech republic :)

> For the other hand,
>
> I get this error:
>
> [Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640):
> Can't convert value `HTT' of type string to type uint]

This is very strange error, because on my computer it works well. Can you remove try..catch and post full error list and program parameters?

The full code is:;

//D 2.0
//gdmd-4.6 <fitxer> dhttpclient => surt el fitxer amb el mateix nom i .o
//Usa https://github.com/Bystroushaak/DHTTPClient
//versió 0.0.3
import std.stdio, std.string, std.conv, std.stream;
import std.socket, std.socketstream;
import dhttpclient;

int main(string [] args)
{
    if (args.length < 2) {
		writeln("Usage:");
		writeln("   ./spider {<url1>, <url2>, ...}");
		return 0;
	}
	else {
		try {
			string[string] capcalera = dhttpclient.FFHeaders;
			HTTPClient navegador = new HTTPClient();
			navegador.setClientHeaders(capcalera);

			foreach (a; args[1..$]) {
				write("[Longitud: ");
				stdout.rawWrite(cast(ubyte[]) navegador.get(a));
				writeln("]");
			}
		}
		catch (Exception e) {
			writeln("[Excepció: ", e, "]");
		}
		return 0;
	}
}


I don't know what happens!!!

And no, I don't live in Czech Republic: we have to postpone the invitation ;-)




2012/1/20 Bystroushaak <bystrousak@kitakitsune.org>:
> On 20.1.2012 18:42, Xan xan wrote:
>>
>> Thank you very much. I should invite you to a beer ;-)
>
>
> Write me if you will be in prag/czech republic :)
>
>
>> For the other hand,
>>
>> I get this error:
>>
>> [Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640): Can't convert value `HTT' of type string to type uint]
>
>
> This is very strange error, because on my computer it works well. Can you remove try..catch and post full error list and program parameters?

With png works, with pdf not:

 ./spider2 http://www.google.com/intl/ca/images/logos/mail_logo.png
[a lot of output]

$ ./spider2 http://static.arxiv.org/pdf/1109.4897.pdf
[Longitud: [Excepció:
std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640): Can't
convert value `HTT' of type string to type uint]


2012/1/20 Bystroushaak <bystrousak@kitakitsune.org>:
> On 20.1.2012 18:42, Xan xan wrote:
>>
>> Thank you very much. I should invite you to a beer ;-)
>
>
> Write me if you will be in prag/czech republic :)
>
>
>> For the other hand,
>>
>> I get this error:
>>
>> [Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640): Can't convert value `HTT' of type string to type uint]
>
>
> This is very strange error, because on my computer it works well. Can you remove try..catch and post full error list and program parameters?

That is really strange - for me, it works with both files. Are you sure, that you can manually download that pdf file? Maybe your provider blocking your connection, or something like that.

What type of compiler did you used?

On 21.1.2012 13:14, Xan xan wrote:
> With png works, with pdf not:
>
>   ./spider2 http://www.google.com/intl/ca/images/logos/mail_logo.png
> [a lot of output]
>
> $ ./spider2 http://static.arxiv.org/pdf/1109.4897.pdf
> [Longitud: [Excepció:
> std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640): Can't
> convert value `HTT' of type string to type uint]
>
>
> 2012/1/20 Bystroushaak<bystrousak@kitakitsune.org>:
>> On 20.1.2012 18:42, Xan xan wrote:
>>>
>>> Thank you very much. I should invite you to a beer ;-)
>>
>>
>> Write me if you will be in prag/czech republic :)
>>
>>
>>> For the other hand,
>>>
>>> I get this error:
>>>
>>> [Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640):
>>> Can't convert value `HTT' of type string to type uint]
>>
>>
>> This is very strange error, because on my computer it works well. Can you
>> remove try..catch and post full error list and program parameters?

Al 21/01/12 14:28, En/na Bystroushaak ha escrit:
> That is really strange - for me, it works with both files. Are you sure, that you can manually download that pdf file? Maybe your provider blocking your connection, or something like that.

I don't think so. It's arxiv pdf.
>
> What type of compiler did you used?

I use gdmd-4.6 in ubuntu. Surely you use dmd, isn't?
Perhaps it's a bug on gdc. Can you help me to isolate this?

Thanks,
Xan.
>
> On 21.1.2012 13:14, Xan xan wrote:
>> With png works, with pdf not:
>>
>>   ./spider2 http://www.google.com/intl/ca/images/logos/mail_logo.png
>> [a lot of output]
>>
>> $ ./spider2 http://static.arxiv.org/pdf/1109.4897.pdf
>> [Longitud: [Excepció:
>> std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640): Can't
>> convert value `HTT' of type string to type uint]
>>
>>
>> 2012/1/20 Bystroushaak<bystrousak@kitakitsune.org>:
>>> On 20.1.2012 18:42, Xan xan wrote:
>>>>
>>>> Thank you very much. I should invite you to a beer ;-)
>>>
>>>
>>> Write me if you will be in prag/czech republic :)
>>>
>>>
>>>> For the other hand,
>>>>
>>>> I get this error:
>>>>
>>>> [Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640):
>>>> Can't convert value `HTT' of type string to type uint]
>>>
>>>
>>> This is very strange error, because on my computer it works well. Can you
>>> remove try..catch and post full error list and program parameters?

Fixed. Bug was caused by HTTP 1.0 'HTTP 1.0 200 OK' reply.

On 21.1.2012 13:14, Xan xan wrote:
> With png works, with pdf not:
>
>   ./spider2 http://www.google.com/intl/ca/images/logos/mail_logo.png
> [a lot of output]
>
> $ ./spider2 http://static.arxiv.org/pdf/1109.4897.pdf
> [Longitud: [Excepció:
> std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640): Can't
> convert value `HTT' of type string to type uint]
>
>
> 2012/1/20 Bystroushaak<bystrousak@kitakitsune.org>:
>> On 20.1.2012 18:42, Xan xan wrote:
>>>
>>> Thank you very much. I should invite you to a beer ;-)
>>
>>
>> Write me if you will be in prag/czech republic :)
>>
>>
>>> For the other hand,
>>>
>>> I get this error:
>>>
>>> [Excepció: std.conv.ConvException@/usr/include/d2/4.6/std/conv.d(1640):
>>> Can't convert value `HTT' of type string to type uint]
>>
>>
>> This is very strange error, because on my computer it works well. Can you
>> remove try..catch and post full error list and program parameters?