Sean Kelly wrote:
> Andrei Alexandrescu (See Website For Email) wrote:
>> Sean Kelly wrote:
>>> Andrei Alexandrescu (See Website For Email) wrote:
>>>>
>>>> I'm not even thinking of stored procedures as logic vehicles. All stored procedures I've dealt with were little more than simply stored views (SELECT statements). If I want to see customer phone numbers and their orders, I'm glad to let the database do that efficiently by doing an inner join and passing me the information in one shot, instead of having me look at the two relevant tables for the sake of object orientedness. To say nothing about data integrity, which is best taken care of at the database level.
>>>
>>> Stored procedures have a few advantages over inline queries:
>>>
>>> - Speed.  Stored procedures are pre-compiled.  This can have a tremendous impact on performance for server applications.
>>> - Decoupling.  If the schema changes the app doesn't typically even need recompilation, it "just works" so long as the stored procs are updated as well.
>>> - Security.  Access to the tables can be restricted to admins so the only thing a user can do is run approved stored procs.
>>> - Encapsulation.  This is really an extension of decoupling, but from more of an OO perspective.  Hiding data access and manipulation behind structs has the same advantages of the same thing in OO programming languages.
>>>
>>> By comparison, views are pre-compiled and provide security, but not the other two features.  Quite simply, all interaction with a DB in applications I design is through stored procedures.  If a particular server doesn't support them then it's a toy.
>>
>> One can tell you work in finance. :o)
> 
> :-)
> 
>> Then probably a framework that does what RoR does, plus offers tighter binding to large-scale databases (in addition to better speed and checking), could fill a niche that today RoR does not address for a mix of practical and philosophical reasons?
> 
> Probably.  I'll admit to not being too terribly familiar with RoR, but it's an appealing concept.  I'll really have to read up on it so I can contribute more to the discussion.
> 
> For what it's worth, one of my first jobs was writing systems to do customized telephony processing for a switched long-distance carrier. Much of the actual logic, in addition to the call records, user data, etc, was stored in SQL.  It was a closed system so security wasn't much of an issue, but the necessary performance would have been impossible to achieve without stored procedures.  So I'll admit to being slightly baffled at why many web servers and other popular technologies nowadays don't seem to consider them important (MySQL didn't have them until recently AFAIK).  Is it simply that automated code generation is more valuable from a cost perspective?
> 
> 
> Sean

In my experience, there are several reasons why databases aren't ever used to their fullest potential in web applications:

1) DBA Hegemony in the workplace - devs aren't always given the rights or permissions to create stored procedures and views, even in a development environment.  YMMV.  The common attitude in this scenario is: "I'm Oracle 9i certified, you're not, it's my machine so it's my rules, and I golf regularly with the VP."  Just imagine trying to write an application with the BOFH in the way - you're going to draft a schema in Er-Win, test it 100 ways from Sunday, hand it to him once, and pray to Bob you can code around any mistakes.  Office politics and/or plain ol' bureaucracy is the key problem here.

2) Lack of experience/education.  Devs don't always know much beyond basic SQL and cartesian joins.  Also SQL is *weird*; things like "null != null" is hard for some folks to grasp, type conversion is hackish (or vendor-specific), and date/time wrangling is a chore so they stay where they're comfortable instead.

3) Lack of vendor conformance to features post SQL-99.   Really practical things, like Limit/Top, type conversion, and so-forth are non-standard.  As the SQL-99 spec ages, the major RDBMS vendors become more divergent in their application of new and useful stuff, which only hurts the educational landscape out there.  The result is that more and more, developers have to stick to the lowest-common denominator which is often shored up by middleware that can inject vendor-specific features as needed.

4) Abundance of solutions to the above.  With so many great middleware solutions for wrangling your database, why would you ever need to know more than basic SQL?  Sure, that's like burying your head in the sand, but if it's a time-saver, you're likely going to go this route.


-- 
- EricAnderton at yahoo

Pragma wrote:
> 
> In my experience, there are several reasons why databases aren't ever used to their fullest potential in web applications:
> 
> 1) DBA Hegemony in the workplace - devs aren't always given the rights or permissions to create stored procedures and views, even in a development environment.  YMMV.  The common attitude in this scenario is: "I'm Oracle 9i certified, you're not, it's my machine so it's my rules, and I golf regularly with the VP."  Just imagine trying to write an application with the BOFH in the way - you're going to draft a schema in Er-Win, test it 100 ways from Sunday, hand it to him once, and pray to Bob you can code around any mistakes.  Office politics and/or plain ol' bureaucracy is the key problem here.

I've heard as much but haven't ever found myself in this situation.  DB admins who think programmers aren't capable of writing efficient or reliable SQL so they accept work orders and implement everything themselves.  The official reason seems to be that the DB admin is more aware of the performance demands of different departments, etc.  I suppose this is a common case for internal applications in large firms.

> 2) Lack of experience/education.  Devs don't always know much beyond basic SQL and cartesian joins.  Also SQL is *weird*; things like "null != null" is hard for some folks to grasp, type conversion is hackish (or vendor-specific), and date/time wrangling is a chore so they stay where they're comfortable instead.

This one seems to pretty common, and I suppose gives rise to #1.  But then many programmers don't seem to adequately understand their language of choice, either.

> 3) Lack of vendor conformance to features post SQL-99.   Really practical things, like Limit/Top, type conversion, and so-forth are non-standard.  As the SQL-99 spec ages, the major RDBMS vendors become more divergent in their application of new and useful stuff, which only hurts the educational landscape out there.  The result is that more and more, developers have to stick to the lowest-common denominator which is often shored up by middleware that can inject vendor-specific features as needed.
> 
> 4) Abundance of solutions to the above.  With so many great middleware solutions for wrangling your database, why would you ever need to know more than basic SQL?  Sure, that's like burying your head in the sand, but if it's a time-saver, you're likely going to go this route.

And here I thought there was some practical reason why things were done this way :-)  But my experience as a customer for various online sites and products (MMORPGs come to mind) reinforces what you've said.  I'll admit to being occasionally baffled at the problems some seem to have, but I guess it's the same everywhere.  Well, good to know I haven't missed the boat, I suppose.


Sean

Robby wrote:
> 
> To play devils advocate for a bit:
> 
> There are several scenarios where writing procs would seem kinda over the top, I mean, mysql handles itself quite well in performance benchmarks (not saying it's the best) while going for years not implementing them, you can even go as far to say that while they're enabled they're not there by default (MyISAM is the default and doesn't have procs?) and considering how many 'big' sites have mysql implemented  at some level it would be fair to say that procs is not the only way to go.

Fair enough.  I'll admit that the performance demands of many websites simply aren't at the level I'm accustomed to, but I'd think the impact on maintenance would still be a factor.  However, I guess this is where middleware and code generation come in.

> So there's causes and cases for both concerns, RoR seems to only work with the 'no procs, I'll control my domain logic', there may be other implementations that sees procs and views as something that is first class I'm not sure. D from a performance and expressive standpoint could implement both. It's possible and should be considered.

Good to know that RoR does indeed generate inline SQL as I suspected.

> Another post mentions something about injects and if RoR handles it, it does. There are conventions built in to sanitize arguments going in to prevent such things.

Yup.  And I'll admit that dynamically generated queries can be very useful for some situations.  I suppose I just haven't been involved in developing products where this was the most appropriate way to go.


Sean

Bruno Medeiros wrote:
snipped
> Hum, in the context of the previous discussion of 1.005 features, I too have been trying to understand what makes Rails so special. After reading that article, it is my understanding that most of the goodness of Rails comes from the ability to generate Ruby code from the database's SQL schema, where that Ruby code handles all or most of the ORM logic. Is that correct? If so, is there anything special about Ruby about the language of Rails? Couldn't a similar framework be made for other languages, like Java for example, with similar results as Rails?
> 
In this context, yes. Matter in fact there is several implementations of active record (the ORM logic) coming in from other languages. Keep in mind that Rails is a MVC stack for the web, written in Ruby. The ORM is just one contribution.
> Also, from the article, I've identified two DSLs: SQL and rhtml (the equivalent of Java's JSPs). 
In the interest of accuracy there's also yaml (an xml like language, without the cruft), rjs (javascript templating etc). SQL isn't directly used as a DSL, the metadata via database apis is used to generate code.

In both cases, Ruby code is generated from
> code in these two DSLs, but that code generation is performed not by the Ruby compiler during compile-time, but by an external tool (similar to parser generators for example).
No, it's done with Ruby during run time, using Ruby's dynamic and reflective abilities.

 If so, that would be mean that the D
> 1.005 features are not necessarily required or useful for the "enabling of such applications", which I think was the point kris was making in the Derailed DSL thread ago. Good interoping with DSLs is decisive, but that doesn't mean the interoping needs to be done at compile-time, by the compiler.
> Is this correct?
> 
Yes

Dave wrote:
> Robby wrote:
>>> - Are there any well defined studies suggesting that RoR developed apps. are cheaper than, say, apps. developed with PHP, using a traditional development approach?
>> More enterprise speak?
> 
> Guilty as charged <g> Unfortunately, no matter how promising and cool a technology is many managers have to have a solid justification to try something like RoR even for a smallish project. Heck - I don't blame the managers - I'd have to have some solid justification to divert some of my personal time over to learning RoR, because there's so much to keep up with, and much of it ends up being nothing but a temporary phenom (you mentioned Struts for example).
> 
Oh, I know all about managers *knowing more* :), so no worries, just didn't want to answer those questions if they were in that context. And I agree about a lot to keep up with, which actually falls somewhat inline with the current discussion, if you've taken D as a language of choice wouldn't you want to be able to work on the web stack with D itself? The thing is the web is *so* complicated that there is always going to be generational fads and improvements.
>>> - Is RoR here to stay?
>> I'd say the adoption and the momentum is going to keep it in swing for quite a while, how long? no clue.. I mean, back in the day Struts was the thing (bleh), and it's slowly becoming defunct. Can you be productive in RoR, sure, can you improve it? Sure, it's all about tastes.. but if you want something in the terms of an independent report on how great it is, I can't help. I've used it, it's worked for me on a quite large intranet application, but nothing is the be all end all.
>>
>>>
>>> Then I've got to ask: How could D possibly improve on RoR enough to get people to move away from RoR to DeRailed?
>>>
>> It shouldn't be able moving people from, to d's implementation, it's about providing a strong stack of development tools to allow people who want to code in d on the client, on the server and indirectly, for the web.
> 
> You're right - but is DeRailed really what many of the talented [tango] lib. developers should be working on (if they were so inclined to take direction)?

I can't and won't speak for the Tango developers (I'm not one atm). However, one can assume that you can approach Tango as a toolkit, and considering that Mango's developers are also apart of Tango, so a toolkit for the web kinda makes sense.

Also, in my opinion the best part about trying to compete with technologies in problem domains allows you to open your eyes about some improvements that may help D as a whole. Consider some of the things that Ruby does to make it so easy, reflective abilities, dynamic generation and runtime type information.

Now obviously D isn't going to turn into a dynamic language, but exposing such traits that D could help a static typed langauge approach could help us all in the end. - such as runtime type information.

Sean Kelly wrote:
> Robby wrote:
>>
>> To play devils advocate for a bit:
>>
>> There are several scenarios where writing procs would seem kinda over the top, I mean, mysql handles itself quite well in performance benchmarks (not saying it's the best) while going for years not implementing them, you can even go as far to say that while they're enabled they're not there by default (MyISAM is the default and doesn't have procs?) and considering how many 'big' sites have mysql implemented  at some level it would be fair to say that procs is not the only way to go.
> 
> Fair enough.  I'll admit that the performance demands of many websites simply aren't at the level I'm accustomed to, but I'd think the impact on maintenance would still be a factor.  However, I guess this is where middleware and code generation come in.
> 
>> So there's causes and cases for both concerns, RoR seems to only work with the 'no procs, I'll control my domain logic', there may be other implementations that sees procs and views as something that is first class I'm not sure. D from a performance and expressive standpoint could implement both. It's possible and should be considered.
> 
> Good to know that RoR does indeed generate inline SQL as I suspected.
> 
>> Another post mentions something about injects and if RoR handles it, it does. There are conventions built in to sanitize arguments going in to prevent such things.
> 
> Yup.  And I'll admit that dynamically generated queries can be very useful for some situations.  I suppose I just haven't been involved in developing products where this was the most appropriate way to go.
> 
> 
> Sean
For the record I've been on both sides of the fence, tri weekly meetings with a DBA, and playing DBA myself. So I fully understand where you're coming from, just trying to give a point of view from the dark side ;o).

Robby wrote:
> Also, in my opinion the best part about trying to compete with technologies in problem domains allows you to open your eyes about some improvements that may help D as a whole. Consider some of the things that Ruby does to make it so easy, reflective abilities, dynamic generation and runtime type information.
> 
> Now obviously D isn't going to turn into a dynamic language, but exposing such traits that D could help a static typed langauge approach could help us all in the end. - such as runtime type information.

Definitely. RTTI and reflection are good to have will probably make it in D (as I mentioned, as a couple of the many applications of static introspection). The runtime code generation issue is much more problematic, so a alternative approach is to do compile-time generation for select modules. Running the compiler to generate code will be not much slower than running the interpreter of a competing language. The advantage is that the resulting code will be easier to make correct and faster.

Let me put it this way. If all you need to do is recompile a couple of modules when the database schema changes (a process that is easy to automate), then this might be more attractive than a scheme that does a lot of busywork at runtime to adapt itself dynamically to a structure that seldom changes, and to fail "late" when inconsistencies in the code are revealed.


Andrei

BLS wrote:
> Hi Andrei,
> 
> Andrei Alexandrescu (See Website For Email) schrieb:
> 
>> Sorry, I'm not sure I understand. My understanding of the mechanism is the following:
>>
>> 1. The app runs a SQL-to-target-language parser to build an idea about the database.
>>
>> 2. The database folk changes the database in any number of ways. This is not a process that automatically notifies the target language application.
>>
>> 3. The target language application must undergo some change to accommodate the change in the database.
>>
>> I did DB/financial work in 1998. This scenario was a total bitch because we didn't have small and fast test cases for all logic code to run when the database changed. Basically it was the customer (financial analysts) who let us know when something bombed, and they actually got so used to it that they even weren't pissed anymore.
>> Andrei
> 
> I understand and I know about the problem. Due to the fact that this is a serious task, which should not be answered (like before) within a simple statement, and the fact that I am not used to think and argue in english, requires that I have to write down my thoughts in german first and to translate them later. This may take a while, I hope you understand.
> 
> Allow me a few WHYs
> WHY a datastore called database has such an influence/impact  on consumer applications, f.i. your Software ? I mean a database should be
> a Black Box ONLY accessable through your application by using a public key in the sense of (PGP)... Your Application should master the database and not vice versa.
> 
> Since about 24 years I am in the database business and I have seen only one implemantation which keeps you away from this kind of trouble, let me quote /
> Suneido has an integrated client-server relational database. The database is accessed via a language which includes administration requests and update operations as well as actual queries. The query language is based on the relational algebra language in An Introduction to Database Systems by C.J.Date.
> /
> 
> D is young and refreshing: So WHY not implementing this kind of DB System in D ?
> 
> Since I know that you are very busy I hesitate to offer you a this link  but I think you will find some very interesting information worth spending a few minutes...
> http://www.suneido.com/index.php?option=com_content&task=view&id=49&Itemid=1 
> 
> 
> Further. a big step
> I could imagine that it could be a matter of interest to port the new MINIX  kernel into D having a DB system instead of "journaling file system"  (Beside, still wonder why this kind of filesys. is not allready reality) Why not doing  that in D ? This is indeed a  vision and as our old german chanceler "Helmut Schmidt" says : People having visions should see a doctor really quick.

I've seen Tanenbaum talking about Minix. It's a jewel; the kernel is only 4000 lines of code, and everything else is modularly implementable in user space. Writing Minix's kernel in D would definitely be an interesting D project.


Andrei

Andrei Alexandrescu (See Website For Email) wrote:
> Robby wrote:
>> Also, in my opinion the best part about trying to compete with technologies in problem domains allows you to open your eyes about some improvements that may help D as a whole. Consider some of the things that Ruby does to make it so easy, reflective abilities, dynamic generation and runtime type information.
>>
>> Now obviously D isn't going to turn into a dynamic language, but exposing such traits that D could help a static typed langauge approach could help us all in the end. - such as runtime type information.
> 
> Definitely. RTTI and reflection are good to have will probably make it in D (as I mentioned, as a couple of the many applications of static introspection). The runtime code generation issue is much more problematic, so a alternative approach is to do compile-time generation for select modules. Running the compiler to generate code will be not much slower than running the interpreter of a competing language. The advantage is that the resulting code will be easier to make correct and faster.
> 
> Let me put it this way. If all you need to do is recompile a couple of modules when the database schema changes (a process that is easy to automate), then this might be more attractive than a scheme that does a lot of busywork at runtime to adapt itself dynamically to a structure that seldom changes, and to fail "late" when inconsistencies in the code are revealed.
> 
> 
> Andrei
Yeah, fair enough.
I mentioned something similar in the DeRailed thread and posted a very very rough draft[1] how something could work with D's implementation now. And now that I think about it further, that very setup could even use the new mixin/import features.

What I'm currently working on is using derelict's loader setup and porting sqlite to use it, this way I'm not confined to where the sqlite.dll is on windows (ugh), then I'm going to start working on DeActive using sqlite as a test bed.

I'm actually not looking at this whole thing from a RubyOnRails perspective, but for now, from a ActiveRecord perspective. I want a nice ORM setup for a gui app I'm working on, and in turn an ORM to keep track of my poker hands + backtesting for my poker bot. I just happen to have extensive experience through my old job on RubyOnRails and its implemention.

[1]http://www.dsource.org/projects/tango/wiki/DeActive

Bruno Medeiros wrote:
> Andrei Alexandrescu (See Website For Email) wrote:
>> After yesterday's hubbub, Judge Judy called and punished me to read about RoR, in addition to the obligatory sentence of helping a little old lady cross the street five times a week.
>>
>> So I went and read the nice tutorial at:
>>
>> http://www.onlamp.com/pub/a/onlamp/2006/12/14/revisiting-ruby-on-rails-revisited.html 
>>
>>
>> I have a couple of questions that I assume will be easy to answer by anyone who actually has used RoR.
>>
>> On the second page of the tutorial, the authors describe how they write SQL code to create tables, and then how they invoke Ruby to parse that SQL code and generate (I assume) Ruby wrappers for it.
>>
>> Now consider that the database changes: new tables, new fields, different types for existing fields, etc.
>>
>> 1. Is now the generated Ruby code is out of sync with the database?
>>
>> 2. In case it is out of sync, what is the way to bring it back in sync? Manual editing of the Ruby code? Editing the SQL and then regenerating the wrappers? Some automated way?
>>
>> An additional question: most interesting work in databases is done through views (SELECT statements) and stored procedures. Can Ruby parse such stuff and generate appropriate wrappers? If so, what happens when the views and stored procedures change?
>>
>> I'm asking these questions because I want to figure whether automating the task of keeping in sync with a database, plus the additional type safety and speed, are significant advantages in the Web/DB domain. In such a scenario, error messages like the one in Part 2 (http://www.onlamp.com/pub/a/onlamp/2007/01/05/revisiting-ruby-on-rails-revisited-2.html?page=4) may be avoided; the code simply fails to compile. I know of domains where such advantages are very important, but I'm not sure how the Web/DB domain feels about it.
>>
>>
>> Andrei
> 
> Hum, in the context of the previous discussion of 1.005 features, I too have been trying to understand what makes Rails so special. After reading that article, it is my understanding that most of the goodness of Rails comes from the ability to generate Ruby code from the database's SQL schema, where that Ruby code handles all or most of the ORM logic. Is that correct? If so, is there anything special about Ruby about the language of Rails? Couldn't a similar framework be made for other languages, like Java for example, with similar results as Rails?
> 
> Also, from the article, I've identified two DSLs: SQL and rhtml (the equivalent of Java's JSPs). In both cases, Ruby code is generated from code in these two DSLs, but that code generation is performed not by the Ruby compiler during compile-time, but by an external tool (similar to parser generators for example). If so, that would be mean that the D 1.005 features are not necessarily required or useful for the "enabling of such applications", which I think was the point kris was making in the Derailed DSL thread ago. Good interoping with DSLs is decisive, but that doesn't mean the interoping needs to be done at compile-time, by the compiler.
> Is this correct?
> 

That is correct. But everybody is doing the run-time schtick, so doing it during compilation would bring an edge (error detection and faster execution).

Andrei