I'm back to actively working on a std.database specification & implementation.  It's still unstable, minimally tested, and there is plenty of work to do, but I wanted to share an update on my progress.

The main focus of this project is to bring a standard interface for database clients.    This is similar to the purpose of JDBC (java) and DBI (perl).  While there is some existing work in place (ddbc, etc.c.odbc.sql, vibe.d and other newer projects), my goal, after a lengthly period of community review/revision, is to achieve Phobos inclusion for the interface standard and also some of the implementations. There is debate on whether the implementations belong there, but I have made the implementation Phobos compatible (by using templates) while this issue is sorted out.

The initial focus of this project is to first cover the synchronous interface details and get it to a refined state.  Asynchronous I/O is also important and it will take some time to work that into the model with a number of complicated aspects to consider.

Also, this is not an ORM tool, although it does provide a foundational layer for building one. This is also aimed at SQL databases, although I will consider how no-sql targets might fit in.

There are now implementations for 4 initial targets (sqlite, mysql, Oracle, and ODBC) and they are in a basic working state with a common test suite.

For usage examples and other limited project details, see the github page:
https://github.com/cruisercoder/dstddb

Dub link coming soon (dealing with dub zombie name) along with more progress updates.

Please feel free to comment or question on the design as evolves.

erik

On Tuesday, 1 March 2016 at 21:00:30 UTC, Erik Smith wrote:
>
> Please feel free to comment or question on the design as evolves.

Minor typo in the fluent style select section of readme.md.
the line
createDatabase("file:///demo.sqlite");
should be
createDatabase("file:///demo.sqlite")

Typo fixed - thanks.  Incidentally, I'm not 100% content with createDatabase.  With the library being template based, the types are no longer as easy to work with directly:

auto database = Database!DefaultPolicy();

alias cant be used because it instantiates.  The template argument can be defaulted, but the best I can do is this:

auto database = Database!()();

It would be nice if the compiler invoked the default type without the extra !().  I settled on using a no parameter template function named createDatabase.   I would have liked just database() to match the other member function style in the package, but I think that might be injecting a name that might be too common as a variable name.

Project minutiae.

erik

On Tuesday, 1 March 2016 at 21:00:30 UTC, Erik Smith wrote:
> I'm back to actively working on a std.database specification & implementation.  It's still unstable, minimally tested, and there is plenty of work to do, but I wanted to share an update on my progress.
>
> [...]

Hi Erik,

As soon as sqlite-d is at the point that it is usable.
I will see if I can support the API proposed by you.

Have you ever used it in production or semi-production code ?
What is the rational for your design ?

Okay I've found a problem.

Here is some code demonstrating it.
http://dpaste.dzfl.pl/022c9e610a18

Now take a look again at Database https://github.com/cruisercoder/dstddb/blob/master/src/std/database/poly/database.d#L37

Do you see the problem?
The solution is simple.

The client database type can be a struct or a class. It doesn't matter too much (assuming you're going the way of ranges).
But the intermediary representation must be on the heap and should probably use the constructor and not a static create method to get it.
This way people can use other memory management solutions and construct it however they like.

Hi Stefan,

It might be a challenge for CTFE compatibility in the API, but it would be interesting to see how much of it is workable.  There does need to be some optional API elements in cases where it's not supported by the underlying database client(array binding for example) and these compile time policies could also be used to define a restricted subset if needed for your design.

I don't have any D code in production unfortunately, but I have a similar design in C++ that has been in production for years.

As far as the design rationale, the primary aspect of the design is to ensure deterministic resource lifetimes for each of the resources commonly exposed by clients (connections, statements, rowsets, buffer allocations, etc), which is why structs are used.  This could be done with with simple non-copyable structs for a more limited design.  I went with the value type structs (using reference counting) which provides better composibility overall.  This can be seen in the call chaining examples, which I think significantly increase ease-of-use.  It also covers use cases in application code where, for example, lazy results are returned through code layers and they need to retain their underlying connections.

Thanks for the questions and the interest.

erik

Yes agree that the poly Database is broken - it isn't reference counted and I will fix that.

Your sample code had me wondering if I am missing something else, but I can't see another issue yet.  I think the use of classes would definitely lead to problems with resources being freed out of order or too late.

As far as memory management options, my plan is to work allocators into the design and that would seem to provide a lot of options.  I'm having a problem at the moment with MallocAllocator's shared interface.  I'm not sure why it's shared since malloc/free are thread safe and I can't seem to cast away the shared.  I'm sure there is a reason.

erik


On Wednesday, 2 March 2016 at 03:07:54 UTC, Rikki Cattermole wrote:
> Okay I've found a problem.
>
> Here is some code demonstrating it.
> http://dpaste.dzfl.pl/022c9e610a18
>
> Now take a look again at Database https://github.com/cruisercoder/dstddb/blob/master/src/std/database/poly/database.d#L37
>
> Do you see the problem?
> The solution is simple.
>
> The client database type can be a struct or a class. It doesn't matter too much (assuming you're going the way of ranges).
> But the intermediary representation must be on the heap and should probably use the constructor and not a static create method to get it.
> This way people can use other memory management solutions and construct it however they like.

On 02/03/16 4:48 PM, Erik Smith wrote:
> Yes agree that the poly Database is broken - it isn't reference counted
> and I will fix that.

My point was, you shouldn't handle that.

> Your sample code had me wondering if I am missing something else, but I
> can't see another issue yet.  I think the use of classes would
> definitely lead to problems with resources being freed out of order or
> too late.

Currently its use after free. E.g. destructor gets called but there is still a copy around.
At the very least it should be an explicit call.

If classes lead to problems, so will structs.

> As far as memory management options, my plan is to work allocators into
> the design and that would seem to provide a lot of options.  I'm having
> a problem at the moment with MallocAllocator's shared interface.  I'm
> not sure why it's shared since malloc/free are thread safe and I can't
> seem to cast away the shared.  I'm sure there is a reason.

Use IAllocator. Don't touch the structs unless you want pain.

Oh and btw final class is your friend.
As an FYI here is my managed memory concept https://github.com/rikkimax/alphaPhobos/blob/master/source/std/experimental/memory/managed.d its not finished but it mostly works.

Of course I would want to go the more OOP way, sure thats more allocations but over all I think there are enough wins that its worth it. So if you feel it doesn't fit well with your goal, say so :)

> erik
>
>
> On Wednesday, 2 March 2016 at 03:07:54 UTC, Rikki Cattermole wrote:
>> Okay I've found a problem.
>>
>> Here is some code demonstrating it.
>> http://dpaste.dzfl.pl/022c9e610a18
>>
>> Now take a look again at Database
>> https://github.com/cruisercoder/dstddb/blob/master/src/std/database/poly/database.d#L37
>>
>>
>> Do you see the problem?
>> The solution is simple.
>>
>> The client database type can be a struct or a class. It doesn't matter
>> too much (assuming you're going the way of ranges).
>> But the intermediary representation must be on the heap and should
>> probably use the constructor and not a static create method to get it.
>> This way people can use other memory management solutions and
>> construct it however they like.
>
>

I will look at your managed approach to understand it better.  One drawback I can think of is that destruction might not occur immediately.  This can be an issue for shared libraries when the GC hasn't run when the library call returns to the host application.

Maybe it's a C++ bias, but the RefCounted approach seems like such a natural fit for this use case and so far it's working well.  There is the race condition issue but I think there is an allocator solution for that coming.

erik


On Wednesday, 2 March 2016 at 04:11:10 UTC, Rikki Cattermole wrote:
> On 02/03/16 4:48 PM, Erik Smith wrote:
>> Yes agree that the poly Database is broken - it isn't reference counted
>> and I will fix that.
>
> My point was, you shouldn't handle that.
>
>> Your sample code had me wondering if I am missing something else, but I
>> can't see another issue yet.  I think the use of classes would
>> definitely lead to problems with resources being freed out of order or
>> too late.
>
> Currently its use after free. E.g. destructor gets called but there is still a copy around.
> At the very least it should be an explicit call.
>
> If classes lead to problems, so will structs.
>
>> As far as memory management options, my plan is to work allocators into
>> the design and that would seem to provide a lot of options.  I'm having
>> a problem at the moment with MallocAllocator's shared interface.  I'm
>> not sure why it's shared since malloc/free are thread safe and I can't
>> seem to cast away the shared.  I'm sure there is a reason.
>
> Use IAllocator. Don't touch the structs unless you want pain.
>
> Oh and btw final class is your friend.
> As an FYI here is my managed memory concept https://github.com/rikkimax/alphaPhobos/blob/master/source/std/experimental/memory/managed.d its not finished but it mostly works.
>
> Of course I would want to go the more OOP way, sure thats more allocations but over all I think there are enough wins that its worth it. So if you feel it doesn't fit well with your goal, say so :)
>
>> erik
>>
>>
>> On Wednesday, 2 March 2016 at 03:07:54 UTC, Rikki Cattermole wrote:
>>> Okay I've found a problem.
>>>
>>> Here is some code demonstrating it.
>>> http://dpaste.dzfl.pl/022c9e610a18
>>>
>>> Now take a look again at Database
>>> https://github.com/cruisercoder/dstddb/blob/master/src/std/database/poly/database.d#L37
>>>
>>>
>>> Do you see the problem?
>>> The solution is simple.
>>>
>>> The client database type can be a struct or a class. It doesn't matter
>>> too much (assuming you're going the way of ranges).
>>> But the intermediary representation must be on the heap and should
>>> probably use the constructor and not a static create method to get it.
>>> This way people can use other memory management solutions and
>>> construct it however they like.

On Tuesday, 1 March 2016 at 21:00:30 UTC, Erik Smith wrote:
> The main focus of this project is to bring a standard interface for database clients.    This is similar to the purpose of JDBC (java) and DBI (perl).  While there is some existing work in place (ddbc, etc.c.odbc.sql, vibe.d and other newer projects), my goal, after a lengthly period of community review/revision, is to achieve Phobos inclusion for the interface standard and also some of the implementations. There is debate on whether the implementations belong there, but I have made the implementation Phobos compatible (by using templates) while this issue is sorted out.
>

My quick comments:

1. In my opinion it should not be called std.database, but let's say "std.dbc".
This is because it looks like a wrapper over db clients.  Moreover one may say it's almost impossible to make a common and effective interface which would work with all databases. e.g. someone on Wikipedia argues ODBC is obolete (check "ODBC today")

2. I'm not against such a functionality in Phobos. However your project seems to be a duplication of the DDBC project. And it was not proposed for inclusion so often.

3. What I call a D Database API could be described as:
 - Database object
 - DbCollection object (equivalent to array)
 - ranges + std.algorithm

 And here is my implementation (experimental) of this API:
 https://github.com/PiotrekDlang/AirLock/tree/master/docs/database/design.md
 https://github.com/PiotrekDlang/AirLock/tree/master/src

Piotrek