ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
	ubyte[] buffer = new ubyte[receiveCount];
	size_t count = socket.receive(buffer);
	return buffer[0 .. count];
}

string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
	ubyte[][] data;
	size_t count = 0;
	
	do
	{
		debug(1) writefln("Chunk %s", count);
		data ~= receiveBytes(socket, segmentSize);
		writeln(data[count]);
		if (!data)
			break;

	} while(data[count++]);

	char[] stringData;

	foreach (elem; data)
		stringData ~= elem;

	debug(1) writeln(`Exiting "receiveAll"`);

	return to!string(stringData);
}


I've tried many variations of the above code; both with the retrieve integrated into the do-while loop in receiveAll; however I was just seeing whether abstracting the code would let me spot my issues a bit faster because it won't require me to make two variables for the buffer and amount of bytes received, more like the typical interface you get.

Issue is; when receiving any size buffer from the end-point (tested with >1024 byte and <1024 byte buffers); there is always a superfluous chunk which is awaiting data.


Listening: 0.0.0.0:6969
Client: somebody:58769
Chunk 0
[123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101, 116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 32, 34, 51, 48, 50, 54, 57, 50, 48, 49, 55, 50, 51, 54, 56, 54, 57, 49, 50, 49, 95, 97, 97, 97, 97, 97, 34, 44, 32, 34, 100, 97, 116, 97, 34, 58, 32, 34, 34, 125]
Chunk 1
[PAUSE]


Listening: 0.0.0.0:6969
Client: somebody:58767
Chunk 0
[123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101, 116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 32, 34, ... 97]
Chunk 1
[97, ... 125]
Chunk 2
[PAUSE]


No matter what way I try; my code doesn't seem to know when to quit regardless of the check. Also for the arbitrary packet sizes, I would've expected that if I received N bytes X times, the first X-1 times would be perfectly N not some unusual integer.
Simply put, say I'm receiving 1024 bytes 5 times. The length of each item on the stack looks like:

[720,
 490,
 1024,
 103
]

Although I'd assume it'd be more like:

[1024,
 1024,
 289
]

What's up with this? It makes working with sockets so damn tedious since there's no room for assumptions; can anybody suggest an abstracted library for sockets in D or help with my former issue?

On 12/13/2017 11:39 AM, Unazed Spectaculum wrote:
> ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
> {
>      ubyte[] buffer = new ubyte[receiveCount];
>      size_t count = socket.receive(buffer);

Don't trust code you find on newsgroups. :o) You have to check the returned value first. According to documentation, it can return Socket.ERROR:

  https://dlang.org/phobos/std_socket.html#.Socket.receive

> there is always a superfluous chunk
> which is awaiting data.

Can you show with complete code? Perhaps the stream is in blocking mode?

> No matter what way I try; my code doesn't seem to know when to quit
> regardless of the check. Also for the arbitrary packet sizes, I would've
> expected that if I received N bytes X times, the first X-1 times would
> be perfectly N not some unusual integer.
> Simply put, say I'm receiving 1024 bytes 5 times. The length of each
> item on the stack looks like:
>
> [720,
>   490,
>   1024,
>   103
> ]

Posix read(2) man page says

"It is not an error if this number is smaller than the number of bytes
requested; this may happen for example because fewer bytes are actually
available right now (maybe because we were close to end-of-file, or because
we are reading from a pipe, or from a terminal), or because read() was
interrupted by a signal."

Ali

On Thursday, 14 December 2017 at 00:09:39 UTC, Ali Çehreli wrote:
> On 12/13/2017 11:39 AM, Unazed Spectaculum wrote:
> > ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
> > {
> >      ubyte[] buffer = new ubyte[receiveCount];
> >      size_t count = socket.receive(buffer);
>
> Don't trust code you find on newsgroups. :o) You have to check the returned value first. According to documentation, it can return Socket.ERROR:
>
>   https://dlang.org/phobos/std_socket.html#.Socket.receive
>
> > there is always a superfluous chunk
> > which is awaiting data.
>
> Can you show with complete code? Perhaps the stream is in blocking mode?
>
> > No matter what way I try; my code doesn't seem to know when
> to quit
> > regardless of the check. Also for the arbitrary packet sizes,
> I would've
> > expected that if I received N bytes X times, the first X-1
> times would
> > be perfectly N not some unusual integer.
> > Simply put, say I'm receiving 1024 bytes 5 times. The length
> of each
> > item on the stack looks like:
> >
> > [720,
> >   490,
> >   1024,
> >   103
> > ]
>
> Posix read(2) man page says
>
> "It is not an error if this number is smaller than the number of bytes
> requested; this may happen for example because fewer bytes are actually
> available right now (maybe because we were close to end-of-file, or because
> we are reading from a pipe, or from a terminal), or because read() was
> interrupted by a signal."
>
> Ali

void main()
{
	auto socket = new TcpSocket();
	setupSocket(socket, "0.0.0.0", 6969);
	
	writefln("Listening: %s", socket.localAddress);

	while(true)
	{
		Socket client = socket.accept();
		debug(1) writefln("Client: %s", client.remoteAddress);

		auto data = receiveAll(client);
		writeln(data);
		JSONValue json;

		try {
			json = parseJSON(data);
		} catch (JSONException e) {
			debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e);
			client.close();
			continue;
		} catch (Exception e) {
			debug(1) writefln("Client caused exception:\n||%s", e);
			client.close();
			continue;
		}

		if (!verifyValues(json))
		{
			debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress);
			client.close();
			continue;
		}

		debug(1) writeln("Client transacted successful JSON packet.");

		writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d",
				client.remoteAddress,
				json["filename"],
				json["method"],
				data.length
			);

		if (json["method"].str == "store")
			storeData(json["filename"].str, json["data"].str);
		else if (json["method"].str == "retrieve")
			retrieveData(client, json["filename"].str);

		client.close();
	}
}

This is the only function which has a call to `receiveAll`, also yeah I don't typically check return codes for error values, I just assume it'll all work and if it doesn't a fresh restart will fix it; but I'll include it in my code just in case.

On 12/14/2017 11:55 AM, Unazed Spectaculum wrote:

> This is the only function which has a call to `receiveAll`


I marked my changes with [Ali]:

import std.stdio;
import std.socket;
import std.conv;
import std.json;

ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
{
    ubyte[] buffer = new ubyte[receiveCount];
    size_t count = socket.receive(buffer);
    // [Ali] Return null when there is no data
    if (count == Socket.ERROR || count == 0) {
        return null;
    }
    return buffer[0 .. count];
}

string receiveAll(T)(T socket, size_t segmentSize = 1024)
{
    ubyte[][] data;
    size_t count = 0;

    do
    {
        debug(1) writefln("Chunk %s", count);
        auto part = receiveBytes(socket, segmentSize);
        // [Ali] Done when there is no data
        if (!part) {
            break;
        }
        data ~= part;
        writeln(data[count]);
        if (!data)
            break;

    } while(data[count++]);

    char[] stringData;

    foreach (elem; data)
        stringData ~= elem;

    debug(1) writeln(`Exiting "receiveAll"`);

    return to!string(stringData);
}

void main()
{
    auto socket = new TcpSocket();
    // [Ali]    setupSocket(socket, "0.0.0.0", 6969);
    // The following had worked in a test program:
    socket.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true);
    socket.bind(new InternetAddress(6969));
    socket.listen(1);

    writefln("Listening: %s", socket.localAddress);

    while(true)
    {
        Socket client = socket.accept();
        debug(1) writefln("Client: %s", client.remoteAddress);

        auto data = receiveAll(client);
        writeln(data);
        JSONValue json;

        try {
            json = parseJSON(data);
        } catch (JSONException e) {
            debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e);
            client.close();
            continue;
        } catch (Exception e) {
            debug(1) writefln("Client caused exception:\n||%s", e);
            client.close();
            continue;
        }

        /+ [Ali] Not important for this test
        if (!verifyValues(json))
        {
            debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress);
            client.close();
            continue;
        }
        +/

        debug(1) writeln("Client transacted successful JSON packet.");

        writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d",
                client.remoteAddress,
                json["filename"],
                json["method"],
                data.length
            );

        // [Ali] Not important for this test
        /+
        if (json["method"].str == "store")
            storeData(json["filename"].str, json["data"].str);
        else if (json["method"].str == "retrieve")
            retrieveData(client, json["filename"].str);
        +/
        client.close();
    }
}

1) Start the program

2) $ telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.

3) Paste expected json to terminal:

{"filename":"/tmp/foo","method":"bar"}

4) Press the escape character

^]

5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection

telnet> Connection closed.

Here is the output from the program:

Listening: 0.0.0.0:6969
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
{"filename":"/tmp/foo","method":"bar"}

127.0.0.1:47704:
	Filename: "\/tmp\/foo"
	Method: "bar"
	Data length: 40

I think it works! :)

Ali

On Thursday, 14 December 2017 at 20:27:36 UTC, Ali Çehreli wrote:
> On 12/14/2017 11:55 AM, Unazed Spectaculum wrote:
>
>> This is the only function which has a call to `receiveAll`
>
>
> I marked my changes with [Ali]:
>
> import std.stdio;
> import std.socket;
> import std.conv;
> import std.json;
>
> ubyte[] receiveBytes(T)(T socket, size_t receiveCount)
> {
>     ubyte[] buffer = new ubyte[receiveCount];
>     size_t count = socket.receive(buffer);
>     // [Ali] Return null when there is no data
>     if (count == Socket.ERROR || count == 0) {
>         return null;
>     }
>     return buffer[0 .. count];
> }
>
> string receiveAll(T)(T socket, size_t segmentSize = 1024)
> {
>     ubyte[][] data;
>     size_t count = 0;
>
>     do
>     {
>         debug(1) writefln("Chunk %s", count);
>         auto part = receiveBytes(socket, segmentSize);
>         // [Ali] Done when there is no data
>         if (!part) {
>             break;
>         }
>         data ~= part;
>         writeln(data[count]);
>         if (!data)
>             break;
>
>     } while(data[count++]);
>
>     char[] stringData;
>
>     foreach (elem; data)
>         stringData ~= elem;
>
>     debug(1) writeln(`Exiting "receiveAll"`);
>
>     return to!string(stringData);
> }
>
> void main()
> {
>     auto socket = new TcpSocket();
>     // [Ali]    setupSocket(socket, "0.0.0.0", 6969);
>     // The following had worked in a test program:
>     socket.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true);
>     socket.bind(new InternetAddress(6969));
>     socket.listen(1);
>
>     writefln("Listening: %s", socket.localAddress);
>
>     while(true)
>     {
>         Socket client = socket.accept();
>         debug(1) writefln("Client: %s", client.remoteAddress);
>
>         auto data = receiveAll(client);
>         writeln(data);
>         JSONValue json;
>
>         try {
>             json = parseJSON(data);
>         } catch (JSONException e) {
>             debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e);
>             client.close();
>             continue;
>         } catch (Exception e) {
>             debug(1) writefln("Client caused exception:\n||%s", e);
>             client.close();
>             continue;
>         }
>
>         /+ [Ali] Not important for this test
>         if (!verifyValues(json))
>         {
>             debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress);
>             client.close();
>             continue;
>         }
>         +/
>
>         debug(1) writeln("Client transacted successful JSON packet.");
>
>         writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d",
>                 client.remoteAddress,
>                 json["filename"],
>                 json["method"],
>                 data.length
>             );
>
>         // [Ali] Not important for this test
>         /+
>         if (json["method"].str == "store")
>             storeData(json["filename"].str, json["data"].str);
>         else if (json["method"].str == "retrieve")
>             retrieveData(client, json["filename"].str);
>         +/
>         client.close();
>     }
> }
>
> 1) Start the program
>
> 2) $ telnet 0.0.0.0 6969
> Trying 0.0.0.0...
> Connected to 0.0.0.0.
> Escape character is '^]'.
>
> 3) Paste expected json to terminal:
>
> {"filename":"/tmp/foo","method":"bar"}
>
> 4) Press the escape character
>
> ^]
>
> 5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection
>
> telnet> Connection closed.
>
> Here is the output from the program:
>
> Listening: 0.0.0.0:6969
> [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
> {"filename":"/tmp/foo","method":"bar"}
>
> 127.0.0.1:47704:
> 	Filename: "\/tmp\/foo"
> 	Method: "bar"
> 	Data length: 40
>
> I think it works! :)
>
> Ali

I've tried to integrate your code with mine; to no avail, I've directly tested your code; and yet again to no avail, so this should conclude the thread as it pinpoints an error with my actual OS.

Just in case, as a last resort in case I'm actually that dumb, here's me reproducing your steps:

1) Starting program
unazed@unazed  /home/d/storage-server  dmd -debug -run app.d
Listening: 0.0.0.0:6969

2) telnet to the server
 unazed@unazed  ~  telnet 0.0.0.0 6969
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.

3) paste '{"filename":"/tmp/foo","method":"bar"}'
 unazed@unazed  /home/d/storage-server  dmd -debug -run app.d
Listening: 0.0.0.0:6969
Client: 127.0.0.1:55014
Chunk 0
[123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
Chunk 1
[pause]

4-5) press ESC + CTRL+D (I'm on Linux)
Chunk 1
[27]
Chunk 2
[pause]

So, big thanks , and sorry for taking longer to respond; I just get anxious about solving things sometimes and now it's kind of annoying I have to do some more research to fix this issue.

On 12/16/2017 05:21 PM, Unazed Spectaculum wrote:

> 1) Starting program
> unazed@unazed  /home/d/storage-server  dmd -debug -run app.d

Although I don't normally use the -run switch, as expected, it works with -run as well. (More below.)

> Listening: 0.0.0.0:6969
>
> 2) telnet to the server
>   unazed@unazed  ~  telnet 0.0.0.0 6969
> Trying 0.0.0.0...
> Connected to 0.0.0.0.
> Escape character is '^]'.

Note that line! ;)

>
> 3) paste '{"filename":"/tmp/foo","method":"bar"}'

Please also press Enter after that. I don't know what code is at fault but the json parser was not happy without that newline at the end.

>   unazed@unazed  /home/d/storage-server  dmd -debug -run app.d
> Listening: 0.0.0.0:6969
> Client: 127.0.0.1:55014
> Chunk 0
> [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116,
> 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100,
> 34, 58, 34, 98, 97, 114, 34, 125, 13, 10]
> Chunk 1
> [pause]

Same here...

> 4-5) press ESC + CTRL+D (I'm on Linux)

Aha! That's the problem! You should enter the "Escape character", which is ^] as telnet indicates. You should press Ctrl-] (stay away from the ESC key). Otherwise, you're injecting an ESC character to the communication stream. (The 27 below.)

By the way, I'm on Linux as well! :) (Mint, Ubuntu based.)

And finally, press Ctrl-D to end the stream. It really works:

[...]
Chunk 1
Exiting "receiveAll"
{"filename":"/tmp/foo","method":"bar"}
[...]

Ali