Thread overview
Reading data from the network without knowing the size of the buffer that is coming
Mar 21, 2019
Roman Sztergbaum
Mar 21, 2019
Andre Pany
Mar 21, 2019
Roman Sztergbaum
Mar 21, 2019
Adam D. Ruppe
Mar 22, 2019
Vladimir Panteleev
March 21, 2019
Hello !

in my function :

```D
  private config_load_answer load_config(string[] args)
    in(args !is null, "args cannot be null")
    in(args.length == 2, "need 1 arguments")
    out(r; r.state == "SUCCESS", "load_config should succeed")
    out(r; !r.config_name.empty, "config_name should not be empty")
    out(r; r.config_id > 0, "config_id should be different than 0")
    {
        string config_key;
        string readonly_config_key;
        getopt(args, "key", &config_key, "readonly_key", &readonly_config_key);
        auto cfg = deserialize_to!config_load("../API_doc/json_recipes/config_load.json");
        cfg.config_key = config_key;
        cfg.readonly_config_key = readonly_config_key;
        writeln(cfg);
        writeln(cfg.serializeToJsonPretty);
        client_.socket.send(cfg.serializeToJson);
        auto answer = new ubyte[256]; //
        client_.socket.receive(answer);
        (cast(string) answer).writeln;
        return (cast(string) answer).deserialize!config_load_answer;
    }
```

I would like to get rid of the "ubytes[256]" because I do not know the size of the data that is comming, I would like to read the entire buffer that I send at once. Can someone point me?

(i'm pretty new to D langage, but i'm familiar with C++, and for example in boost asio  we have `byte_transfered` so we can create a buffer with this size and read the incoming data)

Thank's a lot for your help !

March 21, 2019
On Thursday, 21 March 2019 at 16:54:01 UTC, Roman Sztergbaum wrote:
> Hello !
>
> in my function :
>
> ```D
>   private config_load_answer load_config(string[] args)
>     in(args !is null, "args cannot be null")
>     in(args.length == 2, "need 1 arguments")
>     out(r; r.state == "SUCCESS", "load_config should succeed")
>     out(r; !r.config_name.empty, "config_name should not be empty")
>     out(r; r.config_id > 0, "config_id should be different than 0")
>     {
>         string config_key;
>         string readonly_config_key;
>         getopt(args, "key", &config_key, "readonly_key", &readonly_config_key);
>         auto cfg = deserialize_to!config_load("../API_doc/json_recipes/config_load.json");
>         cfg.config_key = config_key;
>         cfg.readonly_config_key = readonly_config_key;
>         writeln(cfg);
>         writeln(cfg.serializeToJsonPretty);
>         client_.socket.send(cfg.serializeToJson);
>         auto answer = new ubyte[256]; //
>         client_.socket.receive(answer);
>         (cast(string) answer).writeln;
>         return (cast(string) answer).deserialize!config_load_answer;
>     }
> ```
>
> I would like to get rid of the "ubytes[256]" because I do not know the size of the data that is comming, I would like to read the entire buffer that I send at once. Can someone point me?
>
> (i'm pretty new to D langage, but i'm familiar with C++, and for example in boost asio  we have `byte_transfered` so we can create a buffer with this size and read the incoming data)
>
> Thank's a lot for your help !

I am also not a  expert in this area. If the message received from the server exceeds 256 bytes, you have to call the receive message several times until you know you received all bytes.

If the data received is smaller than 256 bytes, the array will contain NULL (character 0 value in the ascii table). This you might could use as indicator you received all values.

Is the server really without protocol? If the server code is under your control, you could switch to http, which has a message length attribute.

Ps. In case there is a network error, your program will terminate (out contract throws Error) without any chance to avoid this. You should throw an Exception at the end of method body instead for resource issues.

Kind regards
Andre
March 21, 2019
On Thursday, 21 March 2019 at 18:27:56 UTC, Andre Pany wrote:
> On Thursday, 21 March 2019 at 16:54:01 UTC, Roman Sztergbaum wrote:
>> [...]
>
> I am also not a  expert in this area. If the message received from the server exceeds 256 bytes, you have to call the receive message several times until you know you received all bytes.
>
> If the data received is smaller than 256 bytes, the array will contain NULL (character 0 value in the ascii table). This you might could use as indicator you received all values.
>
> Is the server really without protocol? If the server code is under your control, you could switch to http, which has a message length attribute.
>
> Ps. In case there is a network error, your program will terminate (out contract throws Error) without any chance to avoid this. You should throw an Exception at the end of method body instead for resource issues.
>
> Kind regards
> Andre
Hello,

I didn't precise it, but i use unix local socket communication, the layer is the KERNEL, and not TCP. I think i cannot use module http then.

But, i'm surprise that is not an other way that's is more simple, i will wait for some other answers i think.

Thank's for your help
March 21, 2019
On Thursday, 21 March 2019 at 18:27:56 UTC, Andre Pany wrote:
> If the data received is smaller than 256 bytes, the array will contain NULL (character 0 value in the ascii table). This you might could use as indicator you received all values.


That's wrong, the correct thing to check is the return value of receive(). It gives the length it actually filled in (or 0 if the connection is finished, or negative on error conditions)
March 22, 2019
On Thursday, 21 March 2019 at 16:54:01 UTC, Roman Sztergbaum wrote:
> I would like to get rid of the "ubytes[256]" because I do not know the size of the data that is comming, I would like to read the entire buffer that I send at once. Can someone point me?

If you do not know the size of the response, then how can you know when you should stop waiting for more bytes?

Some connections are stream-oriented, and some are datagram-oriented. With a datagram-oriented connection, each packet has a fixed size (though there is a maximum size). You mentioned this is a UNIX socket connection, and I believe UNIX sockets can be either stream-oriented or datagram-oriented.

If the socket is stream-oriented, there should generally be a way for the other end to signal when it's done sending bytes. This can be in the form of a fixed-size "header" containing the body length, or maybe the other end just closes the connection when it's done writing. Other approaches, like reading until there are no more bytes in the buffer (though more may yet to arrive one millisecond later) are generally not reliable.

Perhaps you should include more information like what program you are trying to talk to, or the protocol being used.