Hi D

I'm about to start a small program to whose job is:

1. Connect to a server over a TCP socket
2. Read a packetized real-time data stream
3. Update/insert to a postgresql database as data arrive.

In general it should buffer data in RAM to avoid exerting back pressure on the input socket and to allow for dropped connections to the PG database. Since the data stream is at most 1.5 megabits/sec (not bytes) I can buffer for quite some time before running out of space.

So far, I've never written a multi-threaded D program where one thread writes a FIFO and the other reads it so I'm reading the last few chapters of Ali Cehreli's book as background. On top of that preparation, I'm looking for:

• general tips on which libraries to examine
• gotchas you've run into in your multi-threaded (or just concurrent) programs,
• other resources to consult
  etc.

Thanks for any advice you want to share.

Best,

On Saturday, 19 February 2022 at 20:13:01 UTC, Chris Piker wrote:

Most people will probably say this is crazy,
but as to PG, one can do without libraries.

I am doing so during years (in C, not D) and
did not expierienced extremely complex troubles.
I mean I do not use libpq - instead I implement some
subset of the protocol, which is needed for particular program.

What I do not like in all these libs for working
with widely used services (postgres, redis etc) is
the fact that they all hide inside them i/o stuff,
including TCP-connect.

Why have connect() in each library?
It is universal thing, as well as read() and write().
If I want several connection to DBMS in a program,
libraries like libpq compel me to use multithreading.

But what if I want to do many-many-many things concurrently
in a single thread?

Usually I design more or less complex (network) programs using
event-driven paradigm (reactor pattern) plus state machines.
In other words programs designed this way are, so to say,
hierarchical team of state machines, interacting with
each other as well as with outer world (signals,
timers, events from sockets etc)

On Sunday, 20 February 2022 at 15:20:17 UTC, eugene wrote:

Very interesting. I need to stand-up this program and two others in one week, so it looks like dpq2 and message passing is the good short term solution to reduce implementation effort. But I would like to return to your idea in a couple months so that I can try a fiber based implementation instead.

It sounds like you might have a rigorous way of defining and keeping track of your state machines. I could probably learn quite a bit from reading your source code, or the source for similarly implemented programs. Are there examples you would recommend?

On Sunday, 20 February 2022 at 16:55:44 UTC, Chris Piker wrote:

I thougt about implementing my engine using fibers but...
it seemed to me they are not very convinient because
coroutines yield returns to the caller, but I want
to return to a single event loop (after processing an event).

Yes, here is my engine with example (echo client/server pair):

• In C (for Linux)
• In D (for Linux & FreeBSD)

edsm = 'event driven state machines'

As to the program you are writing - I wrote a couple of dozens of programs
more or less similar to what you are going to do (data acqusition) using the engine above (C) for production systems and they all serve very well.

On Saturday, 19 February 2022 at 20:13:01 UTC, Chris Piker wrote:

If I get it right you want to restore connection
if it was closed by server for some reason.

I use special SM for that purpose, see this picture

In each state where this SM has to send/recv data, it takes
sending/receiving SM from a pool and commands them to
perform the task. Upon reaching IDLE state this machine
send some messsge to the user (another SM) of the connection
and seats in this state until the user detects connection lost
(in which case it sends M2 to DB-LINK SM). Then DB-LINK
goes to WAIT state, where it starts a timer and when it expires,
it goes to CONN state, where it tries to reconnect (using
sending SM - when connection is ready we get POLLOUT on socket).

You can have as many such connectors as you want,
so you have multiple connections within single thread.

I often use two connections, one for perform main task
(upload some data and alike) and the second for getting
notifications from PG, 'cause it very incovinient to
do both in a single connection.

On Sunday, 20 February 2022 at 18:00:26 UTC, eugene wrote:

The code is terse and clean, thanks for sharing :) I'm adverse to reading it closely since there was no license file and don't want to accidentally violate copyright.

I noticed there were no dub files in the package. Not surprised. Dub is such a restrictive tool compared to say, setup.py/.cfg in python.

On Sunday, 20 February 2022 at 18:36:21 UTC, eugene wrote:

Ah, a very handy tip. It would be convoluted to multiplex notifications
on the data connection.

On Monday, 21 February 2022 at 04:48:56 UTC, Chris Piker wrote:

I am remembering psql client behavior - it sees notifications only after some request. It is really inconvinient to perform regular tasks and be ready to peek up notifications at any moment in one connection.