On Monday, 4 January 2016 at 11:19:38 UTC, Walter Bright wrote:
>ou didn't address my points. Have you used C++ ETs for
> production? They fall apart in real world use for the reasons I mentioned. It's one thing to read a paper about something, and another to use it for something larger than will fit on a slide.

You are assuming template expressions. I am not. The main overhead when interacting with databases is network traffic, building the query should be done at runtime. No need for templates.

I am in favour of having a clean core language (minimal language after desugaring/lowering) that can be manipulated using term rewriting/symbolic manipulation (or ast macros). But that sounds more like D4 than D2...?

On 01/04/2016 08:09 AM, John Colvin wrote:
> On Monday, 4 January 2016 at 12:28:47 UTC, Russel Winder wrote:
>> I must now try creating a D version of the pytest.mark.parametrize
>> decorator – unless someone already has and I have just missed it.
>
> I quick look at pytest.mark.parametrize suggests it could be implemented
> with UDAs and a test-runner that finds all declarations a module
> (recursively) and does all the relevant logic (e.g. that's got more than
> one instance of parametrize, so do some sort of cartesian product of the
> inputs) and actually runs the test.
>
> The main thing that python has here over D is that D's UDAs can't
> directly modify the function they're attached to, but I don't think
> that's necessary for parametrize.
>
> Interestingly, functions can query their own attributes:
>
> @(3) auto attr()
> {
>      return __traits(getAttributes, attr)[0];
> }
>
> unittest
> {
>      assert(attr() == 3);
> }
>
> not sure when I'd use that though...

For computed attributes in template functions. -- Andrei

On 01/04/2016 07:39 AM, Jacob Carlborg wrote:
> If you can't use the host language, i.e. Person.where(e => e.name ==
> "John");, what's the point of inventing a new DSL? Why not just use SQL
> if the DSL needs to be in a string anyway.

I agree. A CTFE SQL parser with automatic variable binding would be pretty rad. Not to mention it would by design take care of things like string injection attacks etc. Also: finding SQL syntax errors during compilation of D code... priceless. -- Andrei

On 01/04/2016 01:49 AM, rumbu wrote:
> On Monday, 4 January 2016 at 00:49:29 UTC, Andrei Alexandrescu wrote:
>>
>> Second, ET as a mechanism for SQL interface has other inherent
>> limitations. Consider the "LIKE" operator in SQL, which has no ET
>> equivalent in C++ with similar syntax, and no direct equivalent in
>> LINQ. That doesn't mean the respective languages are broken.
>>
>
> LINQ transforms StartsWith, EndsWith and Contains methods in LIKE
> queries under the hood since Entity Framework 4.0 (8 years ago).
> Also, when in doubt, there is an entire namespace dedicated to LINQ
> queries with direct SQL equivalents (including LIKE) -
> https://msdn.microsoft.com/en-us/library/system.data.linq.sqlclient.sqlmethods(v=vs.110).aspx

Misunderstanding. Please re-read what I wrote. -- Andrei

On Saturday, 2 January 2016 at 20:47:37 UTC, Chris Wright wrote:
> So you want to create the following query:
>
>   people.filter!(x => x.surname == "Slughorn");
>
> And you've got ten million people in the collection, and you want this query to finish soonish. So you need to use an index. But a full index scan isn't so great; you want to do an index lookup if possible.


Correct.

> That's simple enough; we generate proxy types to record what properties you're using and what operations you're performing. PersonProxy records that you're accessing a field 'surname', gives a StringFieldProxy, and that records that you're checking for equality with the string "Slughorn". The lambda returns true when opEquals returns true.
>
> But people write queries that are more complex than that, like:
>
>   people.filter!(x => x.surname == "Slughorn" || x.age <= 17);
>
> First time you run this, x.surname.opEquals("Slughorn") returns true and the expression as a whole returns true. You missed the second part of the expression. That's bad.
> So we need to evaluate this lambda twice per parameter.

Hmm. Probably we don't think about the "proxy" term in the same way.
When I mentioned proxy I thought about a skeleton object to access parts of the original heavy object. So in fact, filter works on proxy and one evaluation is performed no matter how complex the condition is. The proxy is responsible to retrieve needed data. Additionally (I haven't mentioned it yet) we can provide mechanism for data integrity when joining separate objects.

> You might be able to support queries as large as ten comparisons in a reasonable timeframe. But for all but the most trivial queries, it'll be faster to use SQL.

SQL is no magic. You can write a code to beat any SQL planner. Also I think you had "joins" in mind which I guess are the biggest problem for performance.

But "What if I tell there is no SQL" !!!! Not even under the hood.

Piotrek

On Sunday, 3 January 2016 at 19:48:42 UTC, Abdulhaq wrote:
> My two pence, if you want it to be fast then it must have a good implementation of indices. Your filter functions should not actually start collecting real records, but instead should simply change the way that the cursor traverses the underlying data store. You will need good query 'compilation' like the big boys do, which work out which tables and indices to use and in which order, based on stats of the data / indices.

Agree (I mentioned about proxies before). But I'm not sure about "good query compilation" thing. Good storage management should do. Rest can be done in user code.

> If you want ACID then SQL seems like a good approach to me, certainly I wouldn't want anything ORM-like for updating / inserting data.

Well. Some people really like SQL. I'm not one of them. I'm neither a fan of ORM+SQL. But I think ACID is not only reserved for SQL.

> There a number of good libraries out there already, SQLite obviously springs to mind.

The SQLite project is great.

> It would be a fun project but perhaps a lot more work than you realised if you really want isolation levels, speed etc.

Yes, I know. But one step at a time. It's a some kind of fun for me as well ;)

Cheers
Piotrek

On Monday, 4 January 2016 at 07:59:40 UTC, Jacob Carlborg wrote:
> On 2016-01-04 00:50, Andrei Alexandrescu wrote:
>
>> This may in fact be good signal that an approach based on expression
>> templates is not the most appropriate for D. -- Andrei
>
> This whole thread has already discussed and showed that D operator overloading is lacking in terms of expression templates. The post by Chris assumes no language changes. If D supported overloading all operators the opEquals method would not return true, it would return a proxy that overloaded the || operator. The whole lambda expression would only generate a single query. The rest of the post falls apart from this mistake.

But most of the posts are related to DSL (->SQL). What do you think about solution where no DSL, VM or other intermediate layer is used?

Also I have a theory that SQL appeared because of limiting expressiveness of other programming languages (like C) ;)

Cheers
Piotrek

On Sunday, 3 January 2016 at 23:22:17 UTC, Jakob Jenkov wrote:
> You could just target your database at data analysis. Then you don't need to care about ACID, transactions etc. Just load all the data into memory, and start analyzing it.
>
> Also, you'd typically be scanning over large parts of the data set for each query, so you may not need to support a full query language. Just what is needed for data analysis.
>
> Later you can modify your engine to support ACID, more expressive query language etc.

That's the plan:) Except no dedicated query language is planned. At least that's my vision based on what I know about D and databases currently.

> On one of the projects I am working on right now, we will also implement our own database engine. We need it to integrate tightly with the rest our architecture, and the only way to do that is to roll our own. We will also not be using SQL because SQL is so limiting.
>
> So, I'd say "go ahead" - you can only learn something from the project. I've "reinvented a lot of wheels" over the years, and each time I came out smarter than before. Not every reinvention was a success, but I always learned something from the process.

Thanks! So at least one more soul believing that D can approach the SQL expressiveness in db domain.

Cheers
Piotrek

On Mon, 2016-01-04 at 13:09 +0000, John Colvin via Digitalmars-d wrote:
> 
[…]
> not sure when I'd use that though...

At a trivial level that nonetheless leads to functionality that can be used more sensibly, think of testing factorial. You want property-based testing, but also you want a (well usually two) table of examples.

Python via pytest.mark.parametrize does the latter very well. Hypothesis does the former. Spock enables both very easily.

It is know that the tests for the Java driver to MongoDB relies on table-driven and property-based testing, mostly using Spock.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Mon, 2016-01-04 at 01:44 -0800, Walter Bright via Digitalmars-d wrote:
> On 1/1/2016 3:33 AM, Russel Winder via Digitalmars-d wrote:
> > Or alternative 4, fix D so that proper operator definition can be achieved.
> 
> The way D's operator overloading is designed is not a mistake. It's
> limitations
> are a feature, not a bug. It was deliberately set up to:
> 
> 1. Make doing C++ style iostreams hard.

What is the problem with having the << and >> operators do input output. Very object-oriented.

> 2. Prevent clever use of operator overloading and expression
> templates to create
> languages that look like D, but are NOT.

But it would be D. Boost.Sprint code may look like EBNF, but it is C++.

> 3. Work well when using operator overloading to implement arithmetic types.

It is important that this works. But it should be possible to create an operator algebra for any type: arithmetic types are a very small subset of types used in computing.

> For example, I've seen operator overloading used in C++ to turn it
> into a
> sort-of regex language. The failures of it are:
> 
> 1. Sort-of because C++ operator precedence and prefix/postfix grammar
> is
> different than that of regex, so it can't be emulated correctly.
> 
> 2. It is visually indistinguishable from C++ code. You simply cannot
> look at a
> piece of code and tell it is regex with utterly different meaning,
> rather than
> the usual meanings.
> 
> 3. Any error messages from misuse are utterly and totally
> incomprehensible,
> because the compiler is designed to compile C++ and gives C++
> messages, not
> regex messages.
> 
> 4. C++ ETs are legendary in their tendency to consume all the memory
> in the
> computer and take incredibly long to compile. Most fail when the
> expressions
> exceed a rather small level of complexity because of this.
> 
> My not-so-humble opinion is these sorts of DSLs are technical
> demonstrations,
> but not useful nor desirable tools.
> 
> ***************************************
> So, what does D do?
> 
>     http://dlang.org/phobos/std_regex.html
> 
> D enables CTFE to write a PROPER compile time regex language, with
> correct regex
> grammar, correct regex tokens, regex-specific error messages, etc.
> And it works
> great! It's not a hack, compromise, or workaround. It's a real,
> embedded DSL.
> And it's faster than any other regex engine on the market.

I avoid regular expressions except when editing using ed, sed, or emacs, so I cannot properly comment on the above. However I find Boost.Sprint a very sensible way of using compile-time meta-object protocols. You write a grammar in EBNF in the source code, and the compiler writes the parser. Excellent use of internal DSL. And type safe.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder