The goal of this post is to measure the craziness of an idea to embed a database engine into the D language ;)

I think about a database engine which would meet my three main requirements:
  - integrated with D (ranges)
  - ACID
  - fast

Since the days when I was working on financing data SW I become allergic to SQL. I though that NoSQL databases would fill the bill. Unfortunately they didn't. And I want to have an ability to write a code like this without too much effort:

  struct Person
  {
   string name;
   string surname;
   ubyte age;
   Address address;
  }

 DataBase db = new DataBase("file.db");
 auto coll = db.collection!Person("NSA.Registry");
 auto visitationList = coll.filter!(p => p.name == "James");
 writeln (visitationList);

And other things like updating and deleting from db. I think you get my point.

So I started a PoC project based on SQLite design:
https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

The PoC code: https://github.com/PiotrekDlang/AirLock/tree/master/src/database

Can you please share your thoughts and experience on the topic? Has anyone tried similar things?

Piotrek

On 01/01/16 6:14 AM, Piotrek wrote:
> The goal of this post is to measure the craziness of an idea to embed a
> database engine into the D language ;)
>
> I think about a database engine which would meet my three main
> requirements:
>    - integrated with D (ranges)
>    - ACID
>    - fast
>
> Since the days when I was working on financing data SW I become allergic
> to SQL. I though that NoSQL databases would fill the bill. Unfortunately
> they didn't. And I want to have an ability to write a code like this
> without too much effort:
>
>    struct Person
>    {
>     string name;
>     string surname;
>     ubyte age;
>     Address address;
>    }
>
>   DataBase db = new DataBase("file.db");
>   auto coll = db.collection!Person("NSA.Registry");
>   auto visitationList = coll.filter!(p => p.name == "James");
>   writeln (visitationList);
>
> And other things like updating and deleting from db. I think you get my
> point.
>
> So I started a PoC project based on SQLite design:
> https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture
>
>
> The PoC code:
> https://github.com/PiotrekDlang/AirLock/tree/master/src/database
>
> Can you please share your thoughts and experience on the topic? Has
> anyone tried similar things?
>
> Piotrek

You've just introduced two topics.
The first is a database engine, abstracting away the drivers.
And second an ORM.

On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
> The goal of this post is to measure the craziness of an idea to embed a database engine into the D language ;)
>
> I think about a database engine which would meet my three main requirements:
>   - integrated with D (ranges)
>   - ACID
>   - fast
>
> Since the days when I was working on financing data SW I become allergic to SQL. I though that NoSQL databases would fill the bill. Unfortunately they didn't. And I want to have an ability to write a code like this without too much effort:
>
>   struct Person
>   {
>    string name;
>    string surname;
>    ubyte age;
>    Address address;
>   }
>
>  DataBase db = new DataBase("file.db");
>  auto coll = db.collection!Person("NSA.Registry");
>  auto visitationList = coll.filter!(p => p.name == "James");
>  writeln (visitationList);
>
> And other things like updating and deleting from db. I think you get my point.
>
> So I started a PoC project based on SQLite design:
> https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture
>
> The PoC code: https://github.com/PiotrekDlang/AirLock/tree/master/src/database
>
> Can you please share your thoughts and experience on the topic? Has anyone tried similar things?
>
> Piotrek

You know someone needs to maintain all that code base continuously. When SQLite is a separate project, it has its own developers and we just bind to its library; it is same for other DBs. Your proposal is nice, but creating another standard, and a group of people needs to take care of it in both documentation and coding wise continuously are biggest issues.

On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
>
>   struct Person
>   {
>    string name;
>    string surname;
>    ubyte age;
>    Address address;
>   }
>
>  DataBase db = new DataBase("file.db");
>  auto coll = db.collection!Person("NSA.Registry");
>  auto visitationList = coll.filter!(p => p.name == "James");
>  writeln (visitationList);

This example shows the difficulty of doing this in D. You can't really have something like `p.Name == "James"`, or `p.Age < 21` translate to SQL properly without language changes, which I believe Walter or Andrei were against. This has been the key problem when things like Linq to Sql for D have been brought up before.

On Fri, 2016-01-01 at 10:00 +0000, Kapps via Digitalmars-d wrote:
> On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
> > 
> >   struct Person
> >   {
> >    string name;
> >    string surname;
> >    ubyte age;
> >    Address address;
> >   }
> > 
> >  DataBase db = new DataBase("file.db");
> >  auto coll = db.collection!Person("NSA.Registry");
> >  auto visitationList = coll.filter!(p => p.name == "James");
> >  writeln (visitationList);
> 
> This example shows the difficulty of doing this in D. You can't really have something like `p.Name == "James"`, or `p.Age < 21` translate to SQL properly without language changes, which I believe Walter or Andrei were against. This has been the key problem when things like Linq to Sql for D have been brought up before.

Why does it need language changes?

Having the ability to have an internal DSL instead of SQL string fiddling is one of the major wins for SQLAlchemy. If it can be done in Python why can't it be done in D?

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Friday, 1 January 2016 at 10:26:14 UTC, Russel Winder wrote:
> On Fri, 2016-01-01 at 10:00 +0000, Kapps via Digitalmars-d wrote:
>> On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
>> > 
>> >   struct Person
>> >   {
>> >    string name;
>> >    string surname;
>> >    ubyte age;
>> >    Address address;
>> >   }
>> > 
>> >  DataBase db = new DataBase("file.db");
>> >  auto coll = db.collection!Person("NSA.Registry");
>> >  auto visitationList = coll.filter!(p => p.name == "James");
>> >  writeln (visitationList);
>> 
>> This example shows the difficulty of doing this in D. You can't really have something like `p.Name == "James"`, or `p.Age < 21` translate to SQL properly without language changes, which I believe Walter or Andrei were against. This has been the key problem when things like Linq to Sql for D have been brought up before.
>
> Why does it need language changes?
>
> Having the ability to have an internal DSL instead of SQL string fiddling is one of the major wins for SQLAlchemy. If it can be done in Python why can't it be done in D?

Someone else can explain better / more correctly than me, but I believe the issue lies with opCmp and opEquals. You can make expressions like p.Name.equals("James") work (I believe using opDispatch), but because all you have is opEquals, you can't know if the user put in 'p.Name == "James"` or `p.Name != "James"`, as they both simply call opEquals. In order to do that, you would need things like opLessThan, opEquals, opNotEquals, opGreaterThan, etc, which would (with improper use or bugs) cause other issues, like a < b && a > b and a == b && a != b to be true, or a == b || a != b to be false.

I'm also not certain how you could implement `p => p.Name == "James" || p.Name == "Bob"`, but there might be a way? I think this is the gist of it, but I'm likely wrong on some aspects of this, so it would be good if someone else clarified..

On Friday, 1 January 2016 at 10:26:14 UTC, Russel Winder wrote:
> Why does it need language changes?
>
> Having the ability to have an internal DSL instead of SQL string fiddling is one of the major wins for SQLAlchemy. If it can be done in Python why can't it be done in D?

In D1 Walter made a point about restricting operator overloading to discourage reuse of operators. In D2 there are many ways to create your own weird syntax, but Walter is locked on his D1 position, even though it makes little sense in D2.

This is an overarching theme in D: the design rationale, that goes a decade back and is based on making a restricted easy to use version of C++, does not change, even though the context is very different in 2015 and the premises has changed.

These things are unlikely to change without a fork. Which is a pitty as D needs a more coherent design to get out of stagnation.

On Friday, 1 January 2016 at 10:40:59 UTC, Kapps wrote:

> Someone else can explain better / more correctly than me, but I believe the issue lies with opCmp and opEquals. You can make expressions like p.Name.equals("James") work (I believe using opDispatch), but because all you have is opEquals, you can't know if the user put in 'p.Name == "James"` or `p.Name != "James"`, as they both simply call opEquals. In order to do that, you would need things like opLessThan, opEquals, opNotEquals, opGreaterThan, etc, which would (with improper use or bugs) cause other issues, like a < b && a > b and a == b && a != b to be true, or a == b || a != b to be false.
>
> I'm also not certain how you could implement `p => p.Name == "James" || p.Name == "Bob"`, but there might be a way? I think this is the gist of it, but I'm likely wrong on some aspects of this, so it would be good if someone else clarified..

That's exactly the problem. See an issue reported [1] and a related thread [2]. AST macros could solve the problem as well [3] (and many other problems).

[1] https://issues.dlang.org/show_bug.cgi?id=14593
[2] http://forum.dlang.org/thread/msvapl$2rmn$1@digitalmars.com
[3] http://wiki.dlang.org/DIP50

--
/Jacob Carlborg

On Fri, 2016-01-01 at 10:40 +0000, Kapps via Digitalmars-d wrote:
> […]
> 
> Someone else can explain better / more correctly than me, but I believe the issue lies with opCmp and opEquals. You can make expressions like p.Name.equals("James") work (I believe using opDispatch), but because all you have is opEquals, you can't know if the user put in 'p.Name == "James"` or `p.Name != "James"`, as they both simply call opEquals. In order to do that, you would need things like opLessThan, opEquals, opNotEquals, opGreaterThan, etc, which would (with improper use or bugs) cause other issues, like a < b && a > b and a == b && a != b to be true, or a == b || a != b to be false.
> 
> I'm also not certain how you could implement `p => p.Name == "James" || p.Name == "Bob"`, but there might be a way? I think this is the gist of it, but I'm likely wrong on some aspects of this, so it would be good if someone else clarified..

Hummm… so to put it another way, the D meta-object protocol is even
more broken than that of Java: at least in Java there isn't even a
pretence that you can create an internal DSL. This is very, very sad, I
had not realized D was this broken.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Fri, 2016-01-01 at 11:03 +0000, Jacob Carlborg via Digitalmars-d wrote:
> 
[…]
> That's exactly the problem. See an issue reported [1] and a related thread [2]. AST macros could solve the problem as well [3] (and many other problems).
> 
> [1] https://issues.dlang.org/show_bug.cgi?id=14593
> [2] http://forum.dlang.org/thread/msvapl$2rmn$1@digitalmars.com
> [3] http://wiki.dlang.org/DIP50

Or alternative 4, fix D so that proper operator definition can be
achieved.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder