November 13, 2008
Julio César Carrascal Urquijo wrote:

> Hello Aarti_pl,
> 
>> ...and this high-level design is IMHO mistake. Especially mapping relations from db to objects.
>> 
>> Well maybe someone will give me examples where domain objects are more useful than relations? From my observations presentation layer (GUI) is also relational, so I don't see a sense with making conversions: relation -> object -> relation.
>> 
>> I am working on db access framework which makes use of relations rather than creating objects. And it makes it in typesafe way...
>> 
>> BR
>> Marcin Kuszczak
>> (aarti_pl)
> 
> Agree. I haven't been satisfied with any ORM I've tryied until now. The best until now has been LINQ to SQL but Microsoft it's killing it.
> 
> Can you provide us any details on your project? It sounds a lot like .NET typed datasets.
> 
> Thanks

Please see my answer in other thread....

-- 
Regards
Marcin Kuszczak (Aarti_pl)
-------------------------------------
Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl)
Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/
-------------------------------------

November 13, 2008
Christopher Wright wrote:

> Aarti_pl wrote:
>> ...and this high-level design is IMHO mistake. Especially mapping relations from db to objects.
>> 
>> Well maybe someone will give me examples where domain objects are more useful than relations? From my observations presentation layer (GUI) is also relational, so I don't see a sense with making conversions: relation -> object -> relation.
>> 
>> I am working on db access framework which makes use of relations rather than creating objects. And it makes it in typesafe way...
>> 
>> BR
>> Marcin Kuszczak
>> (aarti_pl)
> 
> It's fine if you're just doing CRUD operations. At my job, we do staff scheduling software -- I'd kill myself if I had to write the scheduling engine or rule violation code in terms of database rows. And that would be a major performance issue.

Well, really I don't get what the problem is here. Could you please elaborate more?

> A good ORM solution will help prevent data inconsistency issues. Let's say you need this superfast query in one area but it returns some data that you already have. If you manipulated that data in memory already and haven't committed it to the database already (because database access is slow, so you don't want to be committing everything all the time), your application will be in an inconsistent state.

Yes, probably there are ORM solutions which keeps data consistent . But please notice that "keeping data consistent" is not a property of ORM itself. It's just that people implemented something like that in ORM systems. I am sure that it is possible to make data consistency layer also in relational system. I just don't want object Person in my program, but instead just row with person's data.

> Also, touching the database is slow. It requires inter-process communication; it requires translating data from the database's internal format to whichever format it sends over the wire; it requires translating from that intermediate format to something the application can handle.

Sure. But mapping this resulting format into objects is even slower. That's in fact one of major arguments against ORM.

> If you can do your query in memory, that's often going to be faster.

Again. Queries in memory is not a property of ORM. In fact you make them always when you iterate specific collection e.g. searching for specific value.

> There are exceptions, such as when your database schema is significantly different than your objects (in which case this isn't very good object-relational mapping), or when the query returns very few rows and you have a huge amount of objects in memory, or when the database is indexed on rows and you don't replicate that in your application.
> 
> And doing in-memory queries isn't possible unless you have a collection of objects that is a guaranteed superset of the desired output.

As I said above - please notice that you can make queries in memory also in relational model. In fact I implemented it in my db access framework in quite universal way as 'agents'.

> In general I prefer ORM. At the very least to translate rows into objects, and to write basic CRUD queries for me; that keeps my database access layer up to date.
> 
> Also, it's a lot faster to write code with ORM than it is to write custom SQL. I've heard people saying that if you are writing a custom database layer in SQL, you're stealing from your employer. This presumes, of course, that you're working in a language with a suitable ORM library available.

Well I think that it is just because that there was a lot of man-power putted into ORM frameworks. And that was probably just because of philosophical reasons - not pragmatical. :-)

Please see also my other post about my Db access framework.

-- 
Regards
Marcin Kuszczak (Aarti_pl)
-------------------------------------
Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl)
Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/
-------------------------------------

November 14, 2008
Marcin Kuszczak wrote:
> Christopher Wright wrote:
> 
>> Aarti_pl wrote:
>>> ...and this high-level design is IMHO mistake. Especially mapping
>>> relations from db to objects.
>>>
>>> Well maybe someone will give me examples where domain objects are more
>>> useful than relations? From my observations presentation layer (GUI) is
>>> also relational, so I don't see a sense with making conversions:
>>> relation -> object -> relation.
>>>
>>> I am working on db access framework which makes use of relations rather
>>> than creating objects. And it makes it in typesafe way...
>>>
>>> BR
>>> Marcin Kuszczak
>>> (aarti_pl)
>> It's fine if you're just doing CRUD operations. At my job, we do staff
>> scheduling software -- I'd kill myself if I had to write the scheduling
>> engine or rule violation code in terms of database rows. And that would
>> be a major performance issue.
> 
> Well, really I don't get what the problem is here. Could you please elaborate more?

Scheduling, you assign people to shifts. That's a many-to-many relationship. If I just had database rows, assigning would be a matter of adding a row. With real objects, I have a set of people associated with each shift. This means it's O(1) to check if someone's assigned rather than O(number of assignments).

There are a lot of things like this. You can use the active record pattern, but that's essentially an implementation detail for ORM.

>> A good ORM solution will help prevent data inconsistency issues. Let's
>> say you need this superfast query in one area but it returns some data
>> that you already have. If you manipulated that data in memory already
>> and haven't committed it to the database already (because database
>> access is slow, so you don't want to be committing everything all the
>> time), your application will be in an inconsistent state.
> 
> Yes, probably there are ORM solutions which keeps data consistent . But please notice that "keeping data consistent" is not a property of ORM itself. It's just that people implemented something like that in ORM systems. I am sure that it is possible to make data consistency layer also in relational system. I just don't want object Person in my program, but instead just row with person's data.

You want a database interaction layer that will return the same row for multiple queries, assuming that the row has the same primary key.

Joins are an issue with this. You're not updating a row retrieved from a join, but you're getting it from the database, and your in-memory row might be dirty.

>> Also, touching the database is slow. It requires inter-process
>> communication; it requires translating data from the database's internal
>> format to whichever format it sends over the wire; it requires
>> translating from that intermediate format to something the application
>> can handle.
> 
> Sure. But mapping this resulting format into objects is even slower. That's in fact one of major arguments against ORM.
> 
>> If you can do your query in memory, that's often going to be
>> faster. 
> 
> Again. Queries in memory is not a property of ORM. In fact you make them always when you iterate specific collection e.g. searching for specific value.

True, thank you for correcting me.

There's still one problem with doing queries, even if you ignore database access. How are you going to arrange these rows? If you want to do interesting operations, not just CRUD, will it be sufficiently fast to have arrays or sets of rows? And will this be convenient from a programming point of view? If not, you're left with implementing active record yourself.

If you are merely doing CRUD operations, a decent ORM might be able to write your queries and database schema for you. If the result is reasonable, you've saved a fair bit of time. If not, you've wasted a small amount of time.

If you have significant business logic, in my experience, you're better off with ORM.

>> In general I prefer ORM. At the very least to translate rows into
>> objects, and to write basic CRUD queries for me; that keeps my database
>> access layer up to date.
>>
>> Also, it's a lot faster to write code with ORM than it is to write
>> custom SQL. I've heard people saying that if you are writing a custom
>> database layer in SQL, you're stealing from your employer. This
>> presumes, of course, that you're working in a language with a suitable
>> ORM library available.
> 
> Well I think that it is just because that there was a lot of man-power putted into ORM frameworks. And that was probably just because of philosophical reasons - not pragmatical. :-)

So what you're saying is that a lot of strangers put a lot of effort into ORM, and this means that I don't have to do as much work. I'm happy with that.

> Please see also my other post about my Db access framework. 
> 
November 16, 2008
Christopher Wright wrote:

Sorry, I missed your post before. My answers in-lined.

> Scheduling, you assign people to shifts. That's a many-to-many relationship. If I just had database rows, assigning would be a matter of adding a row. With real objects, I have a set of people associated with each shift. This means it's O(1) to check if someone's assigned rather than O(number of assignments).
> 
> There are a lot of things like this. You can use the active record pattern, but that's essentially an implementation detail for ORM.

Please remember that I am not saying that entities are solution for everything. I am just saying that in many cases (most?) it is sufficient enough to use entities instead objects. Especially when you have to use data in presentation layer.

In my program I needed graph analysis and to make it fast I used objects. But even in that case I put into these objects simple recordsets from database. Still no need for domain objects.

So if you need fast access - it's ok. - create hash table or use binary search on collection.

>>> A good ORM solution will help prevent data inconsistency issues. Let's say you need this superfast query in one area but it returns some data that you already have. If you manipulated that data in memory already and haven't committed it to the database already (because database access is slow, so you don't want to be committing everything all the time), your application will be in an inconsistent state.
>> 
>> Yes, probably there are ORM solutions which keeps data consistent . But please notice that "keeping data consistent" is not a property of ORM itself. It's just that people implemented something like that in ORM systems. I am sure that it is possible to make data consistency layer also in relational system. I just don't want object Person in my program, but instead just row with person's data.
> 
> You want a database interaction layer that will return the same row for multiple queries, assuming that the row has the same primary key.
> 
> Joins are an issue with this. You're not updating a row retrieved from a join, but you're getting it from the database, and your in-memory row might be dirty.

I implemented only simple static data container which doesn't change automatically its state after creation and changes done by user doesn't change automatically database. Changes to database must be implemented using sql statement like below:

Script script = Update(visitcards).Set(visitcards.name, "Yeti").Where(Equals(visitcards.id, 5));

In such a case to ensure consistency you need to implement simple consistency checks using e.g. optimistic locking:
.... .Where(Equals(visitcards.counter, dbmatrix.get(5, visitcards.counter)));

dbmatrix - container keeping data read before from database visitcards.counter - field incremented on every write to this record

If above statement will not be executed, as where clause will not match any record, it means that data in database have already changed and data in memory is inconsistent.

It is possible to create dynamic container, which will update database automatically when user is setting specific value in row. In my database access model ALL information about database structure is known when executing sql statement, so joins are not a problem. Db layer knows exactly which columns should be updated - it doesn't matter if table was created using joins or simple queries.

>>> Also, touching the database is slow. It requires inter-process communication; it requires translating data from the database's internal format to whichever format it sends over the wire; it requires translating from that intermediate format to something the application can handle.
>> 
>> Sure. But mapping this resulting format into objects is even slower. That's in fact one of major arguments against ORM.
>> 
>>> If you can do your query in memory, that's often going to be faster.
>> 
>> Again. Queries in memory is not a property of ORM. In fact you make them always when you iterate specific collection e.g. searching for specific value.
> 
> True, thank you for correcting me.
> 
> There's still one problem with doing queries, even if you ignore database access. How are you going to arrange these rows? If you want to do interesting operations, not just CRUD, will it be sufficiently fast to have arrays or sets of rows? And will this be convenient from a programming point of view? If not, you're left with implementing active record yourself.

I use array of rows. Every column in row is indexed by int (it's hashmap). I have done quite a bit of data processing on recordsets using virtual columns. The advantage of such a design that you can mix virtual columns with other virtual columns and with normal columns. So in one column you can calculate e.g. id of some row, and in other virtual column use this calculated id to get real data and process it somehow. It's very powerful concept.

It's also very convenient, especially for gui. Let's take simple combo box.
To fill combo box you usually need one column of recordset. With proper API you can get one column with simple method call. Then for every item in combo box you need some additional information. It's easy when it is e.g. only id of database record, but if you  need few columns of different data and some of these data must be calculated? With my framework it's easy also :-)

> If you are merely doing CRUD operations, a decent ORM might be able to write your queries and database schema for you. If the result is reasonable, you've saved a fair bit of time. If not, you've wasted a small amount of time.
> 
> If you have significant business logic, in my experience, you're better off with ORM.
> 
>>> In general I prefer ORM. At the very least to translate rows into objects, and to write basic CRUD queries for me; that keeps my database access layer up to date.
>>>
>>> Also, it's a lot faster to write code with ORM than it is to write custom SQL. I've heard people saying that if you are writing a custom database layer in SQL, you're stealing from your employer. This presumes, of course, that you're working in a language with a suitable ORM library available.
>> 
>> Well I think that it is just because that there was a lot of man-power putted into ORM frameworks. And that was probably just because of philosophical reasons - not pragmatical. :-)
> 
> So what you're saying is that a lot of strangers put a lot of effort into ORM, and this means that I don't have to do as much work. I'm happy with that.

That's very pragmatic. But you know that the fact that it works doesn't mean that it couldn't work better :-)

>> Please see also my other post about my Db access framework.
>>

-- 
Regards
Marcin Kuszczak (Aarti_pl)
-------------------------------------
Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl)
Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/
-------------------------------------

November 16, 2008
Marcin Kuszczak wrote:
> Please remember that I am not saying that entities are solution for everything. I am just saying that in many cases (most?) it is sufficient enough to use entities instead objects. Especially when you have to use data in presentation layer. 
> 
> In my program I needed graph analysis and to make it fast I used objects. But even in that case I put into these objects simple recordsets from database. Still no need for domain objects.

True -- it sounds like you could have done most of your work in SQL, if you were sufficiently masochistic.

My current project involves several thousand classes, each of which has to be persisted, so I'm obviously not using ORM for that; the database schema would be unmaintainably huge. I'll be using a database, but only with very minimal information and then a blob of JSON. A library like yours would be helpful in this situation.

> I implemented only simple static data container which doesn't change automatically its state after creation and changes done by user doesn't change automatically database. Changes to database must be implemented using sql statement like below:
> 
> Script script = Update(visitcards).Set(visitcards.name, "Yeti").Where(Equals(visitcards.id, 5));
> 
> In such a case to ensure consistency you need to implement simple consistency checks using e.g. optimistic locking:
> .... .Where(Equals(visitcards.counter, dbmatrix.get(5, visitcards.counter)));
> 
> dbmatrix - container keeping data read before from database
> visitcards.counter - field incremented on every write to this record
> 
> If above statement will not be executed, as where clause will not match any record, it means that data in database have already changed and data in memory is inconsistent.

Okay, manually implementing optimistic database concurrency. That only has to be done in a couple places, so it's reasonable.

> I use array of rows. Every column in row is indexed by int (it's hashmap). I have done quite a bit of data processing on recordsets using virtual columns. The advantage of such a design that you can mix virtual columns with other virtual columns and with normal columns. So in one column you can calculate e.g. id of some row, and in other virtual column use this calculated id to get real data and process it somehow. It's very powerful concept.
> 
> It's also very convenient, especially for gui. Let's take simple combo box.
> To fill combo box you usually need one column of recordset. With proper API you can get one column with simple method call. Then for every item in combo box you need some additional information. It's easy when it is e.g. only id of database record, but if you  need few columns of different data and some of these data must be calculated? With my framework it's easy also :-)

Data binding directly to the database. It's an interesting and appealing concept.
1 2
Next ›   Last »