I have a project where I need to take and send UDP packets over the Internet. Only raw UDP - my application uses packets directly, with their starting `[0x5a, packet.length.to!ubyte]` included. And only communication with a single address, no need to communicate with multiple clients concurrently.

I understand that I could do it either with the Curl library bundled with Phobos, or use Vibe.D or Hunt instead. But it's the first time I'm dealing with low-level networking like this, and my knowledge about it is lacking. So seek opinions about what library I should use, and more importantly, why.

Other advice about projects like this is also welcome, should anyone wish to share it.

On Sat, Jul 18, 2020 at 04:00:09PM +0000, Dukc via Digitalmars-d-learn wrote:
> I have a project where I need to take and send UDP packets over the Internet. Only raw UDP - my application uses packets directly, with their starting `[0x5a, packet.length.to!ubyte]` included. And only communication with a single address, no need to communicate with multiple clients concurrently.
> 
> I understand that I could do it either with the Curl library bundled with Phobos, or use Vibe.D or Hunt instead. But it's the first time I'm dealing with low-level networking like this, and my knowledge about it is lacking.  So seek opinions about what library I should use, and more importantly, why.

If you already have the raw packets, there is no need for any library, just call the OS's C API directly (such as core.sys.posix.sys.socket). For UDP you don't even need to set up anything, just create a socket and fire the packets away.


--T

On Saturday, 18 July 2020 at 16:00:09 UTC, Dukc wrote:
> I have a project where I need to take and send UDP packets over the Internet. Only raw UDP - my application uses packets directly, with their starting `[0x5a, packet.length.to!ubyte]` included. And only communication with a single address, no need to communicate with multiple clients concurrently.
>
> I understand that I could do it either with the Curl library bundled with Phobos, or use Vibe.D or Hunt instead. But it's the first time I'm dealing with low-level networking like this, and my knowledge about it is lacking. So seek opinions about what library I should use, and more importantly, why.
>
> Other advice about projects like this is also welcome, should anyone wish to share it.

D has socket wrapper interfaces just as many other languages.

https://dlang.org/phobos/std_socket.html

On Saturday, 18 July 2020 at 16:00:09 UTC, Dukc wrote:
> I have a project where I need to take and send UDP packets over the Internet. Only raw UDP

I wrote an example using phobos on my blog a while ago that might help you get started:

http://dpldocs.info/this-week-in-d/Blog.Posted_2019_11_11.html#communication-by-datagram

On Saturday, 18 July 2020 at 16:00:09 UTC, Dukc wrote:
> I have a project where I need to take and send UDP packets over the Internet. Only raw UDP - my application uses packets directly, with their starting `[0x5a, packet.length.to!ubyte]` included. And only communication with a single address, no need to communicate with multiple clients concurrently.
>
> I understand that I could do it either with the Curl library bundled with Phobos, or use Vibe.D or Hunt instead. But it's the first time I'm dealing with low-level networking like this, and my knowledge about it is lacking. So seek opinions about what library I should use, and more importantly, why.
>
> Other advice about projects like this is also welcome, should anyone wish to share it.

Someone once wrote about a UDP library, which was used to sync data to somewhere in APAC (Hongkong?) and by doing so the data transfer was magnitudes faster then before (over TCP)...

I couldn't find this info now, neither in the forum not something meaningful in code.dlang.org...

Maybe the author is still around or someone else has this post at hand and could share it?!

Beside this, there is:
- https://dlang.org/blog/2016/10/21/project-highlight-libasync/

Hope this helps...

Thank you everybody - Especially for the links to the blogs. This is just the kind of stuff I seek (didn't give a close look yet, though).

I think I'm going to try std.socket first, since it's in the standard library. If it feels like it could be easier, I'll consider Libasync.

On Sunday, 19 July 2020 at 09:48:24 UTC, notna wrote:
> Someone once wrote about a UDP library, which was used to sync data to somewhere in APAC (Hongkong?) and by doing so the data transfer was magnitudes faster then before (over TCP)...
>

In the best case scenario, and orders of magnitude more unreliable otherwise.

Choosing UDP over TCP because speed is like choosing a hammer over a screwdriver to drive in a screw because it works and is faster. But it's still the wrong tool for the job.

UDP is a protocol for broadcasting messages which means it's connection less, unreliable (as in no guarantees for delivery of datagrams or that they'll be delivered only once, order i.e. datagrams sent in order A B C D can be delivered like e.g B C A D), and data integrity.
It's insecure (as in everyone who listens can receive it).

Once you need any of these features/guarantees you'll lose performance just as you would by using TCP, plus you pay the cost for re-inventing the wheel, bugs, testing, maintenance, support and all.

UDP is the protocol of choice when you want to broadcast, the data you send isn't important or the TCP overhead is bigger than the transmitted message. Like broadcasting radio/podcasts, weather updates for your status bar or some such or DNS queries.

If you need reliable transmission you need to use a reliable protocol.

On Tuesday, 21 July 2020 at 13:05:21 UTC, wjoe wrote:
> On Sunday, 19 July 2020 at 09:48:24 UTC, notna wrote:
>> Someone once wrote about a UDP library, which was used to sync data to somewhere in APAC (Hongkong?) and by doing so the data transfer was magnitudes faster then before (over TCP)...
>>
>
> In the best case scenario, and orders of magnitude more unreliable otherwise.
>
> Choosing UDP over TCP because speed is like choosing a hammer over a screwdriver to drive in a screw because it works and is faster. But it's still the wrong tool for the job.
>
> UDP is a protocol for broadcasting messages which means it's connection less, unreliable (as in no guarantees for delivery of datagrams or that they'll be delivered only once, order i.e. datagrams sent in order A B C D can be delivered like e.g B C A D), and data integrity.
> It's insecure (as in everyone who listens can receive it).
>
> Once you need any of these features/guarantees you'll lose performance just as you would by using TCP, plus you pay the cost for re-inventing the wheel, bugs, testing, maintenance, support and all.
>
> UDP is the protocol of choice when you want to broadcast, the data you send isn't important or the TCP overhead is bigger than the transmitted message. Like broadcasting radio/podcasts, weather updates for your status bar or some such or DNS queries.
>
> If you need reliable transmission you need to use a reliable protocol.

well, I guess all your remarks are true... and irrelevant at the same time.

please go back and read his first post.... starts with "I have a project where I need to take and send UDP packets over the Internet"...

On Tuesday, 21 July 2020 at 18:35:34 UTC, notna wrote:
> well, I guess all your remarks are true... and irrelevant at the same time.
>
> please go back and read his first post.... starts with "I have a project where I need to take and send UDP packets over the Internet"...

... and continues with:

On Saturday, 18 July 2020 at 16:00:09 UTC, Dukc wrote:
> [...] And only communication with a single address, no need to communicate with multiple clients concurrently.

Let me elaborate on why what I wrote is both, on topic and relevant at the same time:

It's a fundamental characteristic of UDP that you can't communicate with a single client only but you always are communicating with everyone who listens concurrently.

The combination of the Internet and theoretically only 65535 available ports means that (mis)communication with other clients is likely to happen.
OP's client may receive datagrams that are broadcast on that port by other programs (port forwarding).
Likewise, other programs, which have a socket bound to the same port, may receive OP's data.

Examples:
- You behaving correctly yourself doesn't make others behave correctly now or retroactively.

- Choosing a port which isn't in use right now isn't good enough because a few minutes later there may be another program using it, too, and for the same reason.

- Checking a magic byte at offset 0 followed by the size isn't good enough.
Let's say I decide my magic is a word because I want to be able to determine byte order - I choose 0x015a.
When I broadcast a packet, e.g. [[0x015a], [0x0, 0x5a, 0xff]], but because I sent this from a little endian machine, the datagram will be delivered like so [0x5a, 0x01, 0x0, 0x5a, 0xff].
When OP's client receives this message it goes like so:
Read 2 bytes: [0x5a, 0x1]
 0x5a, oh that's me, 0x01 aha length of 1 byte - except it's not even a valid packet in the context of their program...

Using UDP in a good way is far more complicated than it first appears to be.

On Wednesday, 22 July 2020 at 13:17:11 UTC, wjoe wrote:
> - Choosing a port which isn't in use right now isn't good enough because a few minutes later there may be another program using it, too, and for the same reason.

But doesn't the UDP header include the sender IP address? So together with the magic number, shouldn't it be good enough? I know it definitely is not going to hold against jamming or a man in the middle, but it isn't supposed to, at this stage. It's used only for simulations that have the criticality of a casual Tetris server.

I do acknowledge that the needs may rise later on. And if so, I understand that I'm much better off switching the protocol than trying to hardening the UDP.