I use this title at Andrei's suggestion, and repeat his idea that it be used as a prefix for discussions as we navigate toward a design. Unless there is resistance to the idea, I will on the job of implementing whatever we decide is appropriate. I am retired, and have the time to do it.

It seems that every man, and possibly his dog, has a private implementation for at least a favorite DB, so we should have plenty of material to build on.

At this point I would like to get responses from those who feel they are likely to contribute to the design through to completion.

I'd also like to get a feel for the magnitude of the task, so I'd like to ask what database systems you think should be supported.

I have started a github account, and will put my mysqld stuff there shortly, then you can kick me out if you don't like what you see.

Steve

I'm willing to try and contribute as best I can.

Steve Teale wrote:

> I use this title at Andrei's suggestion, and repeat his idea that it be used as a prefix for discussions as we navigate toward a design. Unless there is resistance to the idea, I will on the job of implementing whatever we decide is appropriate. I am retired, and have the time to do it.
I am fine with that as long as people can still dip in if they see bugs or potential improvements.
> 
> It seems that every man, and possibly his dog, has a private implementation for at least a favorite DB, so we should have plenty of material to build on.
> 
> At this point I would like to get responses from those who feel they are likely to contribute to the design through to completion.
> 
> I'd also like to get a feel for the magnitude of the task, so I'd like to ask what database systems you think should be supported.
mysql, postgresql, sqllite are the 3 I am aiming at in my personal implementation.
> 
> I have started a github account, and will put my mysqld stuff there shortly, then you can kick me out if you don't like what you see.
> 
> Steve

On 10/8/11 8:36 AM, Adam Burton wrote:
>> I'd also like to get a feel for the magnitude of the task, so I'd like to
>> ask what database systems you think should be supported.
> mysql, postgresql, sqllite are the 3 I am aiming at in my personal
> implementation.

I had lunch yesterday with a database expert and discussed the matter with him. He advised that we take a driver-oriented approach in which we define a common API for all databases (modeled at high level after e.g. JDBC to reuse that accumulated experience), and then define a few drivers for popular DBMSs in hope that users interested in supporting additional DBMSs will contribute additional drivers using the initial drivers as a template. Eventually when D and its database API become successful, DBMS providers themselves will write the drivers.

It would be great if we could leverage another already widespread driver architecture. There are two that come to mind, and I looked into both: JDBC and ODBC. The first requires either bridging into Java or translating Java driver code to D. The second is widespread on Windows but less so on other OSs. So probably we'll need to define our own drivers. (This also brings dynamic linking as a possibility.)


Andrei

Andrei,

I had a go at odbcd at about the time I first started on mysqld. I must dig it out and get it up to the same state so that I understand ODBC again

But basically from what you're saying, all we need is the ODBC header files translated into D. There appear to be driver managers and plenty of ODBC drivers for Linux.

But that does not get us to where I was thinking of. ODBC is not much easier to use than the native C apis for the databases. I had thought that ODBC was just one of the C database apis that we would have to cover.

Steve

On 10/8/11 11:49 AM, Steve Teale wrote:
> Andrei,
>
> I had a go at odbcd at about the time I first started on mysqld. I must dig it out and
> get it up to the same state so that I understand ODBC again
>
> But basically from what you're saying, all we need is the ODBC header files
> translated into D.

There's also the matter of dynamically linking with drivers I think.

> There appear to be driver managers and plenty of ODBC drivers
> for Linux.

I'm not seeing all that bonanza. First hit for ==ODBC linux== yields http://www.unixodbc.org/ and last update is from April 2010. Not sure how good it is or anything.

> But that does not get us to where I was thinking of. ODBC is not much easier to
> use than the native C apis for the databases. I had thought that ODBC was just one
> of the C database apis that we would have to cover.

1. If we build a D wrapper for ODBC, then we allow people to write code for any database that has an ODBC driver. This, assuming we commit to ODBC as D's standard database interface, would complete the project.

2. If we want to go the route of "one std.database API with drivers for each DBMS" and consider ODBC one of several DBMSs, then we need to define our own driver architecture, write a few drivers ourselves (including probably ODBC), and hope that people will add more drivers. That's a larger project but it unties us from ODBC.

3. If we want to go the route of "similar but not identical specialized APIs per database system" and consider ODBC again only one of the database systems, then we need to define one specialized API per DBMS and force users to essentially choose upfront what DBMS they'll use, and code for it. It's an even larger project and I don't see obvious advantages for it than less of a need for upfront design.


Andrei

Andrei,

It's not an easy one is it.

I think it has to be #2. If we just did #1 we would probably alienate the Linux community.

#3 is a recipe for maintenance problems.

It would be nice if the DB specific drivers could be mixed and matched with whatever we come up with as the standard D interface.

I'm more or less finished with populating individual variables and structs and arrays of structs with my mysqld implementation. I'm going to try to merge variants in as seamlessly as possible tomorrow.

Steve

On Saturday, October 08, 2011 06:43:29 Steve Teale wrote:
> I'd also like to get a feel for the magnitude of the task, so I'd like to ask what database systems you think should be supported.

sqlite, postgres, and mysql are the ones that come to mind for me, though outside of a corporate environment, I can't ever see myself using anything other than sqlite, since it uses a file instead of a server, which makes way more sense when having a database for an individual program.

- Jonathan M Davis

Microsoft SQL Server is important to cover too. I'm pretty sure ODBC works fine for that (there's ODBC bindings for D already, it's part of the Windows headers) and I wrote a little something for my database.d, but I haven't actually tested it yet!

(The project I work on for SQL server has a lot of legacy VB code, so  while I want to get D in on it, haven't really gotten into it yet.)

On Saturday, October 08, 2011 12:00:37 Andrei Alexandrescu wrote:
> 1. If we build a D wrapper for ODBC, then we allow people to write code for any database that has an ODBC driver. This, assuming we commit to ODBC as D's standard database interface, would complete the project.
> 
> 2. If we want to go the route of "one std.database API with drivers for each DBMS" and consider ODBC one of several DBMSs, then we need to define our own driver architecture, write a few drivers ourselves (including probably ODBC), and hope that people will add more drivers. That's a larger project but it unties us from ODBC.
> 
> 3. If we want to go the route of "similar but not identical specialized APIs per database system" and consider ODBC again only one of the database systems, then we need to define one specialized API per DBMS and force users to essentially choose upfront what DBMS they'll use, and code for it. It's an even larger project and I don't see obvious advantages for it than less of a need for upfront design.

I definitely vote for #2 or #3. One of our projects at work uses it (though not one that I personally work on), and I've never heard good things about it. Supporting it makes good sense, but I wouldn't want us to design Phobos' database solution around it.

We should probably explore #2 first, and then if it doesn't work well enough, then at least we have a solid base for the API's being similar with #3. However, given what I've heard about ODBC and its attempts to unify databases, I'm skeptical of how well we'll be able to have a unified DBMS API without harming performance. And from what I understand, it's pretty rare to change DBMSes for a project. You just pick one and use it. And then in the rare case where you have to change, you do the work to do it (and as long as the DB is appropriately modularized with regards to the rest of the program, it doesn't have a hugely negative affect on the rest of the program). So, I question that #2 gains us a whole lot over #3 ultimately (_especially_ if it ends up costing much of anything in terms of performance or usability), but I'm not a DB expert, and I do think that it's at least worth exploring #2 first - if nothing else because it could lead to the APIs for #3 being better unified without harming their performance or usability.

- Jonathan M Davis

The database API I wrote ages ago is built on ODBC and you're welcome to a copy if it would help. At the time (admittedly 15 years ago) the docs for ODBC were incomplete and wrong in places, so a reference can be handy.

Sent from my iPhone

On Oct 8, 2011, at 9:49 AM, Steve Teale <steve.teale@britseyeview.com> wrote:

> Andrei,
> 
> I had a go at odbcd at about the time I first started on mysqld. I must dig it out and get it up to the same state so that I understand ODBC again
> 
> But basically from what you're saying, all we need is the ODBC header files translated into D. There appear to be driver managers and plenty of ODBC drivers for Linux.
> 
> But that does not get us to where I was thinking of. ODBC is not much easier to use than the native C apis for the databases. I had thought that ODBC was just one of the C database apis that we would have to cover.
> 
> Steve