On Sat, 31 Dec 2016 19:24:31 -0800, Adam Wilson wrote:
> My idea: Each data store has it's own implementation with it's own
> naming convention. For example (ADO.NET):
> 	- SqlConnection (MSSQL)
> 	- NpgsqlConnection (Npgsql)
> 
> Yes, this means that you have to change names in your code if you switch data-stores

You *can* use classes and interfaces and type hierarchies. They do use the GC by default, unlike structs, but they're kind of handy, especially here.

Then you have a SqlConnection interface that most people use all the time and all people use most of the time. If you explicitly need some connection properties that are specific to Postgres, you cast to a PostgresConnection.

On Sunday, 1 January 2017 at 03:24:31 UTC, Adam Wilson wrote:
> 2. There are so many different types of data storage systems, how do you design a system generic enough for all of them?
>
> My answer: You don't. Nobody else has bothered trying, and I believe that our worry over that question is a large part of why we don't have anything substantive today.
>
> My idea: Split the data storage systems out by category of data-store.
> For example:
> 	- SQL: std.database.sql (PostgreSQL, MySQL, MSSQL, etc.)
> 	- Document: std.database.document (Mongo, CouchDB, etc.)
> 	- Key-Value: std.database.keyvalue (Redis, etcd2, etc.)

I think that these can all be seen as special cases of a hypegraph database. So on the face of it you probably can build some common interface (apparently the fellows in hypergraphdb.org are trying to do something in this spirit). You can then have specialized interfaces inheriting from it. But given how general hypergraph dbs are,  I'm not sure if this is a worthwhile abstraction.

By the way, what about XML? The documentaion on std.xml says that the module will be replaced at some point in the future. I wonder when and with what...

Mark wrote:
> On Sunday, 1 January 2017 at 03:24:31 UTC, Adam Wilson wrote:
>> 2. There are so many different types of data storage systems, how do
>> you design a system generic enough for all of them?
>>
>> My answer: You don't. Nobody else has bothered trying, and I believe
>> that our worry over that question is a large part of why we don't have
>> anything substantive today.
>>
>> My idea: Split the data storage systems out by category of data-store.
>> For example:
>>     - SQL: std.database.sql (PostgreSQL, MySQL, MSSQL, etc.)
>>     - Document: std.database.document (Mongo, CouchDB, etc.)
>>     - Key-Value: std.database.keyvalue (Redis, etcd2, etc.)
>
> I think that these can all be seen as special cases of a hypegraph
> database. So on the face of it you probably can build some common
> interface (apparently the fellows in hypergraphdb.org are trying to do
> something in this spirit). You can then have specialized interfaces
> inheriting from it. But given how general hypergraph dbs are,  I'm not
> sure if this is a worthwhile abstraction.
>

My experience with graph DB's is that the reality has never been anywhere close to the hype. I don't think it's a worthwhile abstraction. But that is my opinion. And if it is, we should be able to add it over the top of this layer if we really want too.

> By the way, what about XML? The documentaion on std.xml says that the
> module will be replaced at some point in the future. I wonder when and
> with what...

I have no idea. It's a great question! That said it's a bit outside the scope of this topic. :)

-- 
Adam Wilson
IRC: LightBender
//quiet.dlang.dev

Chris Wright wrote:
> On Sat, 31 Dec 2016 19:24:31 -0800, Adam Wilson wrote:
>> My idea: Each data store has it's own implementation with it's own
>> naming convention. For example (ADO.NET):
>> 	- SqlConnection (MSSQL)
>> 	- NpgsqlConnection (Npgsql)
>>
>> Yes, this means that you have to change names in your code if you switch
>> data-stores
>
> You *can* use classes and interfaces and type hierarchies. They do use
> the GC by default, unlike structs, but they're kind of handy, especially
> here.
>

That was my intention, the knee-jerk reaction that class and interfaces get here sometimes strikes me as a bit histrionic sometimes. They are a tool with a use case. :)

> Then you have a SqlConnection interface that most people use all the time
> and all people use most of the time. If you explicitly need some
> connection properties that are specific to Postgres, you cast to a
> PostgresConnection.
>

That is pretty much how it works in ADO.NET and JDO. And I think it works well.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

Chris Wright wrote:
> On Sun, 01 Jan 2017 10:29:28 +0100, Jacob Carlborg wrote:
>
>> On 2017-01-01 04:24, Adam Wilson wrote:
>>
>>> My idea: Each data store has it's own implementation with it's own
>>> naming convention. For example (ADO.NET):
>>>       - SqlConnection (MSSQL)
>>>       - NpgsqlConnection (Npgsql)
>>>
>>> Yes, this means that you have to change names in your code if you
>>> switch data-stores, but since you are already changing your queries,
>>> which is a much more difficult change, this isn't a significant
>>> additional cost.
>>
>> I don't think we should try to make implementations different just
>> because. If you have an SQL builder or an ORM on top of the interface
>> that abstract the differences in the SQL syntax, it's possible to switch
>> driver, within reason.
>
> Those both limit your ability to use the underlying database to its full
> potential. They offer a chance for queries that seem simple and efficient
> to become horribly inefficient.
>

I cannot state my agreement with this paragraph enough. Every ORM I've worked with generates some inexplicably horrific SQL in seemingly simple situations.

> I ran across a problem in NHibernate about a decade ago. We had a
> straightforward HQL query involving joins. It took over a minute to run.
> We wrote the simple equivalent in SQL and it completed in milliseconds.
> Fortunately, NHibernate had the ability to run raw SQL queries.
>

I've also seen Entity Framework 6/7 do the same thing.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

Chris Wright wrote:
> On Sat, 31 Dec 2016 19:24:31 -0800, Adam Wilson wrote:
>> My idea: Split the data storage systems out by category of data-store.
>> For example:
>> 	- SQL: std.database.sql (PostgreSQL, MySQL, MSSQL, etc.)
>
> This is doable; SQL is an ANSI and ISO standard, and it strongly
> constrains what you can do with your data.
>
>> 	- Document: std.database.document (Mongo, CouchDB, etc.)
>> 	- Key-Value: std.database.keyvalue (Redis, etcd2, etc.)
>
> I'm not so certain about this. CouchDB has a rather different approach to
> things than MongoDB -- possibly not as divergent as Mongo from MySQL, but
> far more than Postgres from MySQL.
>
> Likewise, there are many key/value stores in existence, and they support
> many different operations. For instance, it looks like etcd2 has a notion
> of directories, where you can list items in a directory. Redis just lets
> you list keys with a given prefix. Redis lets you modify values in-place;
> etcd2 doesn't.
>
> We could define a common subset of operations for document databases and
> key/value stores, but most people probably wouldn't be satisfied with it.
>
> There's also a question of where you'd put Cassandra in that, since it's
> decidedly not a SQL database but tries to pretend it is.
>
Given that the Cassandra folks wrote an ADO.NET provider for it, I would suggest that it is easiest to treat it as a SQL database from an interface standpoint.

>> 4. We should hide querying from the developer because they are bad at
>> it, security flaws, etc.
>>
>> My answer: While agree in principal, especially with the security
>> concerns, in reality what you are asking for is an ORM. In my opinion,
>> that is a separate concern from a database interface, and is typically
>> implemented as layer over the DB interface.
>
> We can encourage people to use prepared queries with documentation and
> naming.
>

Precisely, my focus would be an making the API as easy as possible to use with Parameterized Queries, and if that makes it harder to write non-parameterized queries, oh well. :D

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

Adam D. Ruppe wrote:
> On Sunday, 1 January 2017 at 03:24:31 UTC, Adam Wilson wrote:
>> interface(s) to a data-store an essential component of the D Standard
>> Library.
>
> Eh, I count it as would-be-nice just because it isn't that hard to just
> use the C ones, or another third party lib; it doesn't have to be Phobos
> itself.
>

On that I beg to differ. The C libraries are not @safe, they have wildly different API's, and they have high-complexity, which is a large risk-factor for bugs and/or security flaws. Any place where we can

> That said though, a basic db interface is quite simple and would be a
> nice batteries included bit - it is one of the things IMO that PHP did
> quite successfully (even if its interface sucked, it still just worked)

I'll agree that it isn't hard, and I think including it would help boost D's usability in the web service space.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

rikki cattermole wrote:
> On 01/01/2017 5:19 PM, Adam D. Ruppe wrote:
>> On Sunday, 1 January 2017 at 03:51:52 UTC, rikki cattermole wrote:
>>> Which is fine if all you use is c's sockets or only that database
>>> connection for a thread.
>>
>> The C drivers typically offer handles of some sort (Windows HANDLE, *nix
>> file descriptor, that kind of thing) that you can integrate into other
>> event loops.
>
> That's fine and all, but you've still got to deal with it on D's side so
> you can mix and match libraries that require access to the same event
> loop (such as Windows).

Vibe.d is working on a native D event loop. We would probably want to integrate with that.

EventCore: https://code.dlang.org/packages/eventcore

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;

On 02/01/2017 3:03 PM, Adam Wilson wrote:
> rikki cattermole wrote:
>> On 01/01/2017 5:19 PM, Adam D. Ruppe wrote:
>>> On Sunday, 1 January 2017 at 03:51:52 UTC, rikki cattermole wrote:
>>>> Which is fine if all you use is c's sockets or only that database
>>>> connection for a thread.
>>>
>>> The C drivers typically offer handles of some sort (Windows HANDLE, *nix
>>> file descriptor, that kind of thing) that you can integrate into other
>>> event loops.
>>
>> That's fine and all, but you've still got to deal with it on D's side so
>> you can mix and match libraries that require access to the same event
>> loop (such as Windows).
>
> Vibe.d is working on a native D event loop. We would probably want to
> integrate with that.
>
> EventCore: https://code.dlang.org/packages/eventcore
>

No, it isn't generic enough.
Nor can it handle windowing without a good bit of modifications.

Mine in SPEW[0] is however ready for this task.

[0] https://github.com/Devisualization/spew/tree/master/src/base/cf/spew/event_loop

rikki cattermole wrote:
> On 02/01/2017 3:03 PM, Adam Wilson wrote:
>> rikki cattermole wrote:
>>> On 01/01/2017 5:19 PM, Adam D. Ruppe wrote:
>>>> On Sunday, 1 January 2017 at 03:51:52 UTC, rikki cattermole wrote:
>>>>> Which is fine if all you use is c's sockets or only that database
>>>>> connection for a thread.
>>>>
>>>> The C drivers typically offer handles of some sort (Windows HANDLE,
>>>> *nix
>>>> file descriptor, that kind of thing) that you can integrate into other
>>>> event loops.
>>>
>>> That's fine and all, but you've still got to deal with it on D's side so
>>> you can mix and match libraries that require access to the same event
>>> loop (such as Windows).
>>
>> Vibe.d is working on a native D event loop. We would probably want to
>> integrate with that.
>>
>> EventCore: https://code.dlang.org/packages/eventcore
>>
>
> No, it isn't generic enough.
> Nor can it handle windowing without a good bit of modifications.
>
> Mine in SPEW[0] is however ready for this task.
>
> [0]
> https://github.com/Devisualization/spew/tree/master/src/base/cf/spew/event_loop
>

How much effort to make the changes? Their plans seem to indicate that they want to support UI integration. I ask because I need to use the library that is going to get the most support over time and that is vibe.d right now.

-- 
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;