Hi! Couple of weeks ago I was approached with a task of writing simple microservice, and I decided to give D a chance. The problem with performance became apparent very soon, and I started to look for the cause. I tried all existing libraries, and noted pretty mature dpq2 and ddb (and it's hb-ddb fork) libs. I didn't like the first one, because I didn't want to leave vibe-d eventloop, and the second one was native (big plus for me), supported vibe-d sockets. I forked it, multiplied performance by hacking in write buffer, but then the whole philosophy behind it's code structure started to bother me, it incapsulated too much IMHO. I needed something low-level, with access to message nature of the protocol.

So, here's another one, uses native PSQL EQ TCP protocol:
https://github.com/Boris-Barboris/dpeq
https://www.postgresql.org/docs/9.5/static/protocol.html

Barely any type support, barely tested, but my little service with it's two tables actually responds to requests. It's something.

Maybe something good will happen and it will find it's use, maybe someone will come and write good generic ORM\DB tool for D, and we'll have a good low-level client to hack into it as an PSQL adapter, maybe it will perish. Who knows. It will definetly go, if I won't publish it though.

If anyone here has some insight on typical architectural errors wich make ORM writer's life hard, please share.

On 2017-09-04 00:57, Boris-Barboris wrote:
> I tried all existing libraries, and noted pretty mature dpq2 and ddb (and it's hb-ddb fork) libs. I didn't like the first one, because I didn't want to leave vibe-d eventloop, and the second one was native (big plus for me), supported vibe-d sockets. I forked it, multiplied performance by hacking in write buffer, but then the whole philosophy behind it's code structure started to bother me, it incapsulated too much IMHO. I needed something low-level, with access to message nature of the protocol.

If would be great if you want to upstream your improvements.

I think it's a bit unfortunate that everyone is rolling their own implementations in this community instead of working together. I would say that it's rare to need direct access to the messages of the protocol. But perhaps the library can be better structured in layers/levels, allowing the user to pick the most suitable level for his/her needs.

-- 
/Jacob Carlborg

On Monday, 4 September 2017 at 06:40:09 UTC, Jacob Carlborg wrote:
>
> If would be great if you want to upstream your improvements.
>
> I think it's a bit unfortunate that everyone is rolling their own implementations in this community instead of working together. I would say that it's rare to need direct access to the messages of the protocol. But perhaps the library can be better structured in layers/levels, allowing the user to pick the most suitable level for his/her needs.

I did try (for example https://github.com/teamhackback/hb-ddb/pull/38, branch with the buffer: https://github.com/Boris-Barboris/hb-ddb/commits/buffered_sockets),
up until the point I understood I dislike it so much it would be easier for me to rewrite it. Amount of possible message combinations made me write some hack methods like prepare_and_bind, bind_and_query, bind_and_execute... I kinda gave up.

It is rare indeed, I would not recommend it to anyone sane, but abstractions leak, and when they do, they better not have everything marked private or belong to separate address space.

Could you give an example how to make connection object if I need access to it from several classes? Should it be global?

On Monday, 4 September 2017 at 09:30:06 UTC, Suliman wrote:
> Could you give an example how to make connection object if I need access to it from several classes? Should it be global?

"""Good way""" is probably to wrap it into some ConnectionPool (at least that's what I did).

Snipper from what I use:

final class PostgresStorage
{
    private ConnectionPool!PostgresConection m_pool;
...
    /// PostgresConection factory used by vibe-d connection pool
    private PostgresConection spawn_connection() @safe
    {
        return new PostgresConection(someparams);
    }

    /// Client code uses this to get connection
    auto get_connection()
    {
        log_debug("Trying to lock PSQL connection");
        auto con = m_pool.lockConnection();
        log_debug("PSQL connection locked");
        return con;
    }

    this()
    {
        m_pool = new ConnectionPool!PostgresConection(
            &spawn_connection, CONF.storageConfig.psql.poolsize);
    }


}

alias PSQLConnT = PSQLConnection!(SocketT);

/// PSQL connection that is user-friendly
final class PostgresConection
{
    private PSQLConnT m_con; // actual dpeq connection
    // here goes stuff you want to expose to client code (REST api handlers or anything else.
    ... some constructors and methods, for example:
    private this()
    {
        m_con = new PSQLConnT(m_conn_params); // from global parameters, taken from config
    }
}

Honestly, tastes differ, It's your wheel to invent.

I use global Storage object, that can give out connections from connection pool,
each connection has collection objects with methods to operate on persistent objects. Transactions are handled by connection, other stuff - from collections.
It's not like there is some golden standard.

If you are not writing anything concurrent, you can just make it global.

On 2017-09-04 10:12, Boris-Barboris wrote:
> On Monday, 4 September 2017 at 06:40:09 UTC, Jacob Carlborg wrote:
>>
>> If would be great if you want to upstream your improvements.
>>
>> I think it's a bit unfortunate that everyone is rolling their own implementations in this community instead of working together. I would say that it's rare to need direct access to the messages of the protocol. But perhaps the library can be better structured in layers/levels, allowing the user to pick the most suitable level for his/her needs.
> 
> I did try (for example https://github.com/teamhackback/hb-ddb/pull/38, branch with the buffer: https://github.com/Boris-Barboris/hb-ddb/commits/buffered_sockets),

Ah, ok. I didn't know about hb-ddb until you started this thread. I'm currently one of the maintainers of ddb and I haven't seen anything upstreamed there.

> up until the point I understood I dislike it so much it would be easier for me to rewrite it. Amount of possible message combinations made me write some hack methods like prepare_and_bind, bind_and_query, bind_and_execute... I kinda gave up.

Ok.

-- 
/Jacob Carlborg

On Monday, 4 September 2017 at 12:07:29 UTC, Jacob Carlborg wrote:

> Ah, ok. I didn't know about hb-ddb until you started this thread. I'm currently one of the maintainers of ddb and I haven't seen anything upstreamed there.

Me too...

/P

On Monday, 4 September 2017 at 12:07:29 UTC, Jacob Carlborg wrote:
> Ah, ok. I didn't know about hb-ddb until you started this thread. I'm currently one of the maintainers of ddb and I haven't seen anything upstreamed there.

Haha, yeah, I guess I've developed a habit of looking at github network graph of every project I touch first, before diving in.

On Sunday, 3 September 2017 at 22:57:39 UTC, Boris-Barboris wrote:

> I tried all existing libraries, and noted pretty mature dpq2 and ddb (and it's hb-ddb fork) libs. I didn't like the first one, because I didn't want to leave vibe-d eventloop,

https://github.com/denizzzka/vibe.d.db.postgresql uses dpq2 backend

On Monday, 4 September 2017 at 14:42:31 UTC, denizzzka wrote:
> https://github.com/denizzzka/vibe.d.db.postgresql uses dpq2 backend

https://github.com/denizzzka/vibe.d.db.postgresql/blob/master/source/vibe/db/postgresql/package.d#L92

Very interesting read, thank you.
Making this:
https://github.com/denizzzka/vibe.d.db.postgresql/blob/master/source/vibe/db/postgresql/package.d#L131
private is probably a crime against humanity though.

I mostly bothered with EQ protocol because I wanted to send PARSE + BIND + EXEC + SYNC without blocking. Is that possible with socket blocking flag manipulation and moderate source code modification?