Thread overview | ||||||||
---|---|---|---|---|---|---|---|---|
|
December 13, 2017 `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF | ||||
---|---|---|---|---|
| ||||
ubyte[] receiveBytes(T)(T socket, size_t receiveCount) { ubyte[] buffer = new ubyte[receiveCount]; size_t count = socket.receive(buffer); return buffer[0 .. count]; } string receiveAll(T)(T socket, size_t segmentSize = 1024) { ubyte[][] data; size_t count = 0; do { debug(1) writefln("Chunk %s", count); data ~= receiveBytes(socket, segmentSize); writeln(data[count]); if (!data) break; } while(data[count++]); char[] stringData; foreach (elem; data) stringData ~= elem; debug(1) writeln(`Exiting "receiveAll"`); return to!string(stringData); } I've tried many variations of the above code; both with the retrieve integrated into the do-while loop in receiveAll; however I was just seeing whether abstracting the code would let me spot my issues a bit faster because it won't require me to make two variables for the buffer and amount of bytes received, more like the typical interface you get. Issue is; when receiving any size buffer from the end-point (tested with >1024 byte and <1024 byte buffers); there is always a superfluous chunk which is awaiting data. Listening: 0.0.0.0:6969 Client: somebody:58769 Chunk 0 [123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101, 116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 32, 34, 51, 48, 50, 54, 57, 50, 48, 49, 55, 50, 51, 54, 56, 54, 57, 49, 50, 49, 95, 97, 97, 97, 97, 97, 34, 44, 32, 34, 100, 97, 116, 97, 34, 58, 32, 34, 34, 125] Chunk 1 [PAUSE] Listening: 0.0.0.0:6969 Client: somebody:58767 Chunk 0 [123, 34, 109, 101, 116, 104, 111, 100, 34, 58, 32, 34, 114, 101, 116, 114, 105, 101, 118, 101, 34, 44, 32, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 32, 34, ... 97] Chunk 1 [97, ... 125] Chunk 2 [PAUSE] No matter what way I try; my code doesn't seem to know when to quit regardless of the check. Also for the arbitrary packet sizes, I would've expected that if I received N bytes X times, the first X-1 times would be perfectly N not some unusual integer. Simply put, say I'm receiving 1024 bytes 5 times. The length of each item on the stack looks like: [720, 490, 1024, 103 ] Although I'd assume it'd be more like: [1024, 1024, 289 ] What's up with this? It makes working with sockets so damn tedious since there's no room for assumptions; can anybody suggest an abstracted library for sockets in D or help with my former issue? |
December 13, 2017 Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF | ||||
---|---|---|---|---|
| ||||
Posted in reply to Unazed Spectaculum | On 12/13/2017 11:39 AM, Unazed Spectaculum wrote: > ubyte[] receiveBytes(T)(T socket, size_t receiveCount) > { > ubyte[] buffer = new ubyte[receiveCount]; > size_t count = socket.receive(buffer); Don't trust code you find on newsgroups. :o) You have to check the returned value first. According to documentation, it can return Socket.ERROR: https://dlang.org/phobos/std_socket.html#.Socket.receive > there is always a superfluous chunk > which is awaiting data. Can you show with complete code? Perhaps the stream is in blocking mode? > No matter what way I try; my code doesn't seem to know when to quit > regardless of the check. Also for the arbitrary packet sizes, I would've > expected that if I received N bytes X times, the first X-1 times would > be perfectly N not some unusual integer. > Simply put, say I'm receiving 1024 bytes 5 times. The length of each > item on the stack looks like: > > [720, > 490, > 1024, > 103 > ] Posix read(2) man page says "It is not an error if this number is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now (maybe because we were close to end-of-file, or because we are reading from a pipe, or from a terminal), or because read() was interrupted by a signal." Ali |
December 14, 2017 Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ali Çehreli | On Thursday, 14 December 2017 at 00:09:39 UTC, Ali Çehreli wrote: > On 12/13/2017 11:39 AM, Unazed Spectaculum wrote: > > ubyte[] receiveBytes(T)(T socket, size_t receiveCount) > > { > > ubyte[] buffer = new ubyte[receiveCount]; > > size_t count = socket.receive(buffer); > > Don't trust code you find on newsgroups. :o) You have to check the returned value first. According to documentation, it can return Socket.ERROR: > > https://dlang.org/phobos/std_socket.html#.Socket.receive > > > there is always a superfluous chunk > > which is awaiting data. > > Can you show with complete code? Perhaps the stream is in blocking mode? > > > No matter what way I try; my code doesn't seem to know when > to quit > > regardless of the check. Also for the arbitrary packet sizes, > I would've > > expected that if I received N bytes X times, the first X-1 > times would > > be perfectly N not some unusual integer. > > Simply put, say I'm receiving 1024 bytes 5 times. The length > of each > > item on the stack looks like: > > > > [720, > > 490, > > 1024, > > 103 > > ] > > Posix read(2) man page says > > "It is not an error if this number is smaller than the number of bytes > requested; this may happen for example because fewer bytes are actually > available right now (maybe because we were close to end-of-file, or because > we are reading from a pipe, or from a terminal), or because read() was > interrupted by a signal." > > Ali void main() { auto socket = new TcpSocket(); setupSocket(socket, "0.0.0.0", 6969); writefln("Listening: %s", socket.localAddress); while(true) { Socket client = socket.accept(); debug(1) writefln("Client: %s", client.remoteAddress); auto data = receiveAll(client); writeln(data); JSONValue json; try { json = parseJSON(data); } catch (JSONException e) { debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e); client.close(); continue; } catch (Exception e) { debug(1) writefln("Client caused exception:\n||%s", e); client.close(); continue; } if (!verifyValues(json)) { debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress); client.close(); continue; } debug(1) writeln("Client transacted successful JSON packet."); writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d", client.remoteAddress, json["filename"], json["method"], data.length ); if (json["method"].str == "store") storeData(json["filename"].str, json["data"].str); else if (json["method"].str == "retrieve") retrieveData(client, json["filename"].str); client.close(); } } This is the only function which has a call to `receiveAll`, also yeah I don't typically check return codes for error values, I just assume it'll all work and if it doesn't a fresh restart will fix it; but I'll include it in my code just in case. |
December 14, 2017 Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF | ||||
---|---|---|---|---|
| ||||
Posted in reply to Unazed Spectaculum | On 12/14/2017 11:55 AM, Unazed Spectaculum wrote: > This is the only function which has a call to `receiveAll` I marked my changes with [Ali]: import std.stdio; import std.socket; import std.conv; import std.json; ubyte[] receiveBytes(T)(T socket, size_t receiveCount) { ubyte[] buffer = new ubyte[receiveCount]; size_t count = socket.receive(buffer); // [Ali] Return null when there is no data if (count == Socket.ERROR || count == 0) { return null; } return buffer[0 .. count]; } string receiveAll(T)(T socket, size_t segmentSize = 1024) { ubyte[][] data; size_t count = 0; do { debug(1) writefln("Chunk %s", count); auto part = receiveBytes(socket, segmentSize); // [Ali] Done when there is no data if (!part) { break; } data ~= part; writeln(data[count]); if (!data) break; } while(data[count++]); char[] stringData; foreach (elem; data) stringData ~= elem; debug(1) writeln(`Exiting "receiveAll"`); return to!string(stringData); } void main() { auto socket = new TcpSocket(); // [Ali] setupSocket(socket, "0.0.0.0", 6969); // The following had worked in a test program: socket.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); socket.bind(new InternetAddress(6969)); socket.listen(1); writefln("Listening: %s", socket.localAddress); while(true) { Socket client = socket.accept(); debug(1) writefln("Client: %s", client.remoteAddress); auto data = receiveAll(client); writeln(data); JSONValue json; try { json = parseJSON(data); } catch (JSONException e) { debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e); client.close(); continue; } catch (Exception e) { debug(1) writefln("Client caused exception:\n||%s", e); client.close(); continue; } /+ [Ali] Not important for this test if (!verifyValues(json)) { debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress); client.close(); continue; } +/ debug(1) writeln("Client transacted successful JSON packet."); writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d", client.remoteAddress, json["filename"], json["method"], data.length ); // [Ali] Not important for this test /+ if (json["method"].str == "store") storeData(json["filename"].str, json["data"].str); else if (json["method"].str == "retrieve") retrieveData(client, json["filename"].str); +/ client.close(); } } 1) Start the program 2) $ telnet 0.0.0.0 6969 Trying 0.0.0.0... Connected to 0.0.0.0. Escape character is '^]'. 3) Paste expected json to terminal: {"filename":"/tmp/foo","method":"bar"} 4) Press the escape character ^] 5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection telnet> Connection closed. Here is the output from the program: Listening: 0.0.0.0:6969 [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10] {"filename":"/tmp/foo","method":"bar"} 127.0.0.1:47704: Filename: "\/tmp\/foo" Method: "bar" Data length: 40 I think it works! :) Ali |
December 17, 2017 Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF | ||||
---|---|---|---|---|
| ||||
Posted in reply to Ali Çehreli | On Thursday, 14 December 2017 at 20:27:36 UTC, Ali Çehreli wrote: > On 12/14/2017 11:55 AM, Unazed Spectaculum wrote: > >> This is the only function which has a call to `receiveAll` > > > I marked my changes with [Ali]: > > import std.stdio; > import std.socket; > import std.conv; > import std.json; > > ubyte[] receiveBytes(T)(T socket, size_t receiveCount) > { > ubyte[] buffer = new ubyte[receiveCount]; > size_t count = socket.receive(buffer); > // [Ali] Return null when there is no data > if (count == Socket.ERROR || count == 0) { > return null; > } > return buffer[0 .. count]; > } > > string receiveAll(T)(T socket, size_t segmentSize = 1024) > { > ubyte[][] data; > size_t count = 0; > > do > { > debug(1) writefln("Chunk %s", count); > auto part = receiveBytes(socket, segmentSize); > // [Ali] Done when there is no data > if (!part) { > break; > } > data ~= part; > writeln(data[count]); > if (!data) > break; > > } while(data[count++]); > > char[] stringData; > > foreach (elem; data) > stringData ~= elem; > > debug(1) writeln(`Exiting "receiveAll"`); > > return to!string(stringData); > } > > void main() > { > auto socket = new TcpSocket(); > // [Ali] setupSocket(socket, "0.0.0.0", 6969); > // The following had worked in a test program: > socket.setOption(SocketOptionLevel.SOCKET, SocketOption.REUSEADDR, true); > socket.bind(new InternetAddress(6969)); > socket.listen(1); > > writefln("Listening: %s", socket.localAddress); > > while(true) > { > Socket client = socket.accept(); > debug(1) writefln("Client: %s", client.remoteAddress); > > auto data = receiveAll(client); > writeln(data); > JSONValue json; > > try { > json = parseJSON(data); > } catch (JSONException e) { > debug(1) writefln("Failed parsing data as JSON, aborting.\n|| %s", e); > client.close(); > continue; > } catch (Exception e) { > debug(1) writefln("Client caused exception:\n||%s", e); > client.close(); > continue; > } > > /+ [Ali] Not important for this test > if (!verifyValues(json)) > { > debug(1) writefln("Client missed out important key fields: %s", client.remoteAddress); > client.close(); > continue; > } > +/ > > debug(1) writeln("Client transacted successful JSON packet."); > > writefln("%s:\n\tFilename: %s\n\tMethod: %s\n\tData length: %d", > client.remoteAddress, > json["filename"], > json["method"], > data.length > ); > > // [Ali] Not important for this test > /+ > if (json["method"].str == "store") > storeData(json["filename"].str, json["data"].str); > else if (json["method"].str == "retrieve") > retrieveData(client, json["filename"].str); > +/ > client.close(); > } > } > > 1) Start the program > > 2) $ telnet 0.0.0.0 6969 > Trying 0.0.0.0... > Connected to 0.0.0.0. > Escape character is '^]'. > > 3) Paste expected json to terminal: > > {"filename":"/tmp/foo","method":"bar"} > > 4) Press the escape character > > ^] > > 5) Press Ctrl-D (Ctrl-Z on Windows) to terminate the connection > > telnet> Connection closed. > > Here is the output from the program: > > Listening: 0.0.0.0:6969 > [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10] > {"filename":"/tmp/foo","method":"bar"} > > 127.0.0.1:47704: > Filename: "\/tmp\/foo" > Method: "bar" > Data length: 40 > > I think it works! :) > > Ali I've tried to integrate your code with mine; to no avail, I've directly tested your code; and yet again to no avail, so this should conclude the thread as it pinpoints an error with my actual OS. Just in case, as a last resort in case I'm actually that dumb, here's me reproducing your steps: 1) Starting program unazed@unazed /home/d/storage-server dmd -debug -run app.d Listening: 0.0.0.0:6969 2) telnet to the server unazed@unazed ~ telnet 0.0.0.0 6969 Trying 0.0.0.0... Connected to 0.0.0.0. Escape character is '^]'. 3) paste '{"filename":"/tmp/foo","method":"bar"}' unazed@unazed /home/d/storage-server dmd -debug -run app.d Listening: 0.0.0.0:6969 Client: 127.0.0.1:55014 Chunk 0 [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, 34, 58, 34, 98, 97, 114, 34, 125, 13, 10] Chunk 1 [pause] 4-5) press ESC + CTRL+D (I'm on Linux) Chunk 1 [27] Chunk 2 [pause] So, big thanks , and sorry for taking longer to respond; I just get anxious about solving things sometimes and now it's kind of annoying I have to do some more research to fix this issue. |
December 16, 2017 Re: `Socket.receive` providing arbitrary packet sizes and hanging without sending EOF | ||||
---|---|---|---|---|
| ||||
Posted in reply to Unazed Spectaculum | On 12/16/2017 05:21 PM, Unazed Spectaculum wrote: > 1) Starting program > unazed@unazed /home/d/storage-server dmd -debug -run app.d Although I don't normally use the -run switch, as expected, it works with -run as well. (More below.) > Listening: 0.0.0.0:6969 > > 2) telnet to the server > unazed@unazed ~ telnet 0.0.0.0 6969 > Trying 0.0.0.0... > Connected to 0.0.0.0. > Escape character is '^]'. Note that line! ;) > > 3) paste '{"filename":"/tmp/foo","method":"bar"}' Please also press Enter after that. I don't know what code is at fault but the json parser was not happy without that newline at the end. > unazed@unazed /home/d/storage-server dmd -debug -run app.d > Listening: 0.0.0.0:6969 > Client: 127.0.0.1:55014 > Chunk 0 > [123, 34, 102, 105, 108, 101, 110, 97, 109, 101, 34, 58, 34, 47, 116, > 109, 112, 47, 102, 111, 111, 34, 44, 34, 109, 101, 116, 104, 111, 100, > 34, 58, 34, 98, 97, 114, 34, 125, 13, 10] > Chunk 1 > [pause] Same here... > 4-5) press ESC + CTRL+D (I'm on Linux) Aha! That's the problem! You should enter the "Escape character", which is ^] as telnet indicates. You should press Ctrl-] (stay away from the ESC key). Otherwise, you're injecting an ESC character to the communication stream. (The 27 below.) By the way, I'm on Linux as well! :) (Mint, Ubuntu based.) And finally, press Ctrl-D to end the stream. It really works: [...] Chunk 1 Exiting "receiveAll" {"filename":"/tmp/foo","method":"bar"} [...] Ali |
Copyright © 1999-2021 by the D Language Foundation