On 2011-10-26 20:20, Jonny Dee wrote:
> One more use case for reflection is data binding with GUI components.
> This approach is heavily used in Windows Presentation Foundation library
> [3,4]. GUI components can update an object's properties by using
> reflection. You don't need to register listeners for this purpose anymore.

Runtime reflection is used quite a lot in GUI programming on Mac OS X (at least the libraries/tools under the hood).

-- 
/Jacob Carlborg

On Wed, 26 Oct 2011 11:20:47 -0700, Jonny Dee <jonnyd@gmx.net> wrote:

> Hello Robert,
>
> Am 26.10.11 07:16, schrieb Robert Jacques:
>> On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
>>> Am 25.10.11 16:41, schrieb Robert Jacques:
>>>> On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
>>>>> [...]
>>> Hi Robert,
>>>
>>> Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
>>> "
>>> [...]
>>> - Discover and modify source code constructions (such as code blocks,
>>> classes, methods, protocols, etc.) as a first-class object at runtime.
>>> - Convert a string matching the symbolic name of a class or function
>>> into a reference to or invocation of that class or function.
>>> [...]
>>> "
>>>
>>> Here is what I would dream of for arbitrary objects/classes (not
>>> necessarily known at compile-time):
>>> - Query an object for its list of methods together with their
>>> signatures. Select a method, bind some values to its arguments, call it,
>>> and retrieve the return type (if any).
>>> - Query an object for its public fields (at least), and provide a way to
>>> get/set their values.
>>> - Query an object's class for all implemented interfaces and its base
>>> class.
>>> - Query a module for all type definitions and provide a way to
>>> introspect these types in more detail. For instance, it would be really
>>> cool if I could find a class with name "Car" in module "cars", get a
>>> list of all defined constructors, select one, bind values to the
>>> constructor's parameters, and create a corresponding object.
>>>
>>> [...]
>>>
>>> Implementing such a DI container heavily depends on reflection, because
>>> the DI container component doesn't know anything about the objects to be
>>> created during runtime.
>>>
>>> Qt also extends C++ with a reflection mechanism through the help of its
>>> meta object compiler (moc). It analyses the C++ source code, generates
>>> meta class definitions [6,7] and weaves them into your Qt class. Hence,
>>> in Qt, you can query an object for fields, methods, interfaces, etc. and
>>> you can call methods with arbitrary parameters, or you can instantiate a
>>> class using an arbitrary constructor. Consequently, somone implemented a
>>> DI container for C++ which is based on Qt and works more or less the
>>> same way the Spring DI container does. You can build up object trees
>>> simply by specifying such trees in an XML file.
>>>
>>> I don't go into why dependency injection is a very powerful feature.
>>> This is Martin Fowler's [3] job ;) But when I program with C++ I miss
>>> such a flexible dependency injection mechanism a lot. And I hope this
>>> will eventually be available for D.
>>>
>>> Cheers,
>>> Jonny
>>>
>>> [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
>>> [2] http://en.wikipedia.org/wiki/Dependency_injection
>>> [3] http://martinfowler.com/articles/injection.html
>>> [4]
>>> http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
>>>
>>> [5] http://qtioccontainer.sourceforge.net/
>>> [6] http://doc.qt.nokia.com/stable/qmetaobject.html
>>> [7]
>>> http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx
>>>
>>
>> Hi Jonny,
>> Thank you for your informative (and well cited) post. It has provided me
>> with a new take on an old design pattern and some enjoyable reading. In
>> return, let me outline my opinion of reflection in D today, and
>> tomorrow, as it pertains to your wish list.
>
> Many thanks to you, too, for your very elaborate answer :)
>
>> Reflection in D today is very different from the host of VM languages
>> that have popularized the concept. Being a compiled systems language,
>> actual runtime self-modification is too virus like to become at a
>> language level feature. However, given the compilation speed of D,
>> people have made proof of concept libraries that essentially wrapped the
>> compiler and dynamically loaded the result. As LDC uses LLVM, which has
>> a jit backend, I'd expect to see something get into and D 'eval' library
>> into etc eventually. (phobos uses the BOOST license, which isn't
>> compatible with LLVM).
>
> I know, that "runtime self-modification" and runtime code generation is a "dangerous" feature. And there really are rare cases where using such an approach might justify the risc in using it. Although this feature is not on my wish list, it might be good for generating dynamic proxies to arbitrary object instances like they are used by some ORMs. See Hibernate/NHibernate, for example [1,2]. Another example is aspect-oriented programming. But while I can't see the exacty reason for it, such a feature might indeed be a feature which is more appropriate for VM languages.
>
>> Compile-time reflection and generation of code, on the other hand, is
>> something D does in spades. It fulfills your dream list, although I
>> think module level reflection might only be available in the github
>> version. The API design is still in flux and we are actively iterating /
>> improving it as find new uses cases and bugs. The current plan is to
>> migrate all the traits functions over to a special 'meta' namespace
>> (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
>> libraries for each of the concepts I listed, (prototype objects,
>> duck-typing/casting or serialization), have been written using the
>> compile-time meta-programming features in D. So that's the good.
>
> A absolutely agree! D's compile-time reflection is very good. And it's hard to top this.
>
>> On the other hand, D's runtime capabilities are limited to
>> object.factory, the under implemented RTTI and library solutions which
>> manually expose information gathered by D's compile-time mechanisms. And
>> so far, these tools have been more than enough, from a functionality
>> point of view. Most of our desire for better runtime reflection stems
>> from a desire for efficiency, composition, cleanliness of user syntax
>> and simplification of library code. These are all important issues for
>> the widespread use of reflection based libraries, but they're not 'I
>> can't implement X' issues.
>
> You are right, there IS certainly always some way to 'implement X'. But, as you know, there is always a consideration of the effort you need to implement X.
>
>> As for the future, I believe that the division in D between compile-time
>> and run-time reflection warrants a serious look at the design of the
>> run-time half of the system. To that end, I believe that implementing
>> reflection in a library would be the best way to experiment and iterate
>> an API. To that end, I have a proposal in the review queue to improve
>> std.variant which contains dynamic dispatch (i.e. the ability to get/set
>> public fields and call methods), duck-typeing/casting and
>> prototype-style objects. Notably, it's missing introspection
>> capabilities as thats what I'm most unsure about API wise, and simplest
>> to add. Designing reflection inside a library keeps Walter & Co's
>> bandwidth free for other bugs/features and provides a very good stress
>> test of D's meta-programming capabilities. (Both of which I think are
>> good things)
>
> I've got no problem with an approach which puts runtime reflection capabiities into a separate library. No matter were you look at, Java, C#, or Qt, all have a library for reflection purpose. I do not see, however, how this might be done without compiler support. As already mentioned, Qt has its own moc compiler, which parses your C++ source code and generates the necessary infra structure. I'm still a beginner with respect to D, so I don't know what is really already possible, and what not. But my current feeling is that a similar approach would also be needed for D. If the D compiler itself, or another post-compiler component should generate the meta information for runtime reflection is another question. I could live with both, although I'd prefer the former built-in one.
>
>> (I am soliciting feedback, if you care to take a look:
>> https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)
>
> Thanks for this link, I'll certainly have a look at it.
>
>> And there are many reasons we might want to experiment with D's runtime
>> reflection API instead of just copying someone. For example, take
>> Dependency Injection. Using Java-style reflection, DI suffers from the
>> Reflection injection and Unsafe Reflection security vulnerabilities. To
>> do DI safely, you have to validate your inputs and D's compile-time
>> reflection provides a perfect way to implement validated DI. Every time
>> I hear about some form of injection, be it SQL or JSON or Reflection,
>> hit the news, makes me think that 5-10 years from now well look back on
>> the use of unvalidated dynamic code constructs the same way we do about
>> null terminated arrays today.
>
> Considering security vulnerabilities is of course a very important matter. However, I think security must be assured a by software's design. Enforcing it solely by programming language constructs will not work for all cases. Particularly not, if this programming language allows direct access to a computer's memory, like C, C++, and D does. There is no sandbox out-of-the-box, where the compiled program runs in. So if you have a private field in a class that carries a password, one has to make sure it's only in memory as long as it is required. And if it is not required anymore one should clear it out with zeros, for example. So making a field private is by no means a secure solution. A hacker will not give up just because the official API declares a field to be private.
>
> I consider 'unsafe reflection', as you call it, as a tool. Let's compare it to a knife, for instance. It is really a useful tool, I think nobody doubts, but at the same time it can be very dangerous. But inspite of this fact, everybody has not only one at home, I guess. Pointers are a useful tool, too. But they are also dangerous and still available in D, which is a good thing, because you can use this tool where needed. And actually, I don't think runtime reflection must be unsafe. The reflection mechanism provides type information for an object's properties, functions, arguments, etc. So validated DI is even possible with runtime reflection. Let's consider the the XML configuration of object trees once again. If you want to store a string to an int-field, for instance, then the DI container can refuse to do this, because it has access to all information required to enforce correct value types.
>
> One more use case for reflection is data binding with GUI components. This approach is heavily used in Windows Presentation Foundation library [3,4]. GUI components can update an object's properties by using reflection. You don't need to register listeners for this purpose anymore.
>
> That said, I don't think dynamic code constructs will be old-fashioned in 5-10 years, because you don't have to go an "unvalidated dynamic" way with runtime reflection.
>
> BTW, as this thread also discusses an opt-in or opt-out implementation for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I guess the binaries are meant) is not as bad as it might sound with today's memory sizes. And if one wants to avoid it for optimization purpose, one can do it. The use of D as a systems programming language for embedded systems is, as I've read somewhere, not a first citizen anyway, because you'll get a lot of code for the garbage collection mechanism.
>
> Cheers,
> Jonny
>
> [1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29
> [2] http://en.wikipedia.org/wiki/Nhibernate
> [3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation
> [4] http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx

I am a first time poster but I've been lurking around here for a while now and this topic is something that is extremely near and dear to my heart. I have been working with .NET since the C# 1.0 Beta 1 days and cut my teeth on WPF while it was still called Avalon. I know you mention that WPF uses Reflection, and on that statement I can only say that you are correct, for better or worse, usually worse. However, thats not the complete story. WPF uses something called a DependencyObject to implement objects with data-binding. And this is important as literally EVERY WPF object inherits this class, you can't build a WPF object without it. And every property you set on a WPF object is really two properties, one of which is a "public static DependencyProperty PropertyName", a DependencyProperty is also being a class that requires a DependencyObject to initialize it. In WPF you absolutely MUST declare this object as public. And while not strictly required Microsoft Best Practices dictate that the actual property is public as well.

My point is that even in WPF you don't actually NEED reflection of private members. The reason for this is that back during the design of Silverlight Microsoft knew that SL's reflection would NOT be able to reflect to private members and they needed to make sure that SL XAML would still work with WPF XAML. Remember, SL was originally called WPF/Everywhere. WPF has taken a lot of flack for the amount of reflection that it does precisely because it slows it down, but I think that it has other bigger problems. Personally, I've found the Measure and Arrange calls to be even significantly more expensive, but those can be mitigated with experience, the reflection can't, so it gets blamed. WPF has many other perf issues too, how they draw rounded corners makes grown men cry and any game programmer who wrote code that bad would be fired on the spot, a one thousand polygon circle anyone?

It's also interesting that you bring up WPF because the project I lead, The Horizon Project, is an attempt to create a open-source multi-platform framework that is as easy to use as the .NET Framework and one of it's core pillars is a declarative XML based UI design language. I am evaluating D and so far it is blowing C++ out of the water, (we'll need to have long conversations about strings though, I think .NET has the right answer here) but I have to admit that I am seriously missing Reflection. The problem is that there is currently no way to do data-binding without member-level reflection because it is a purely run-time concept. That said, I am researching ways around this, and I do have ideas ... but that's not for this forum.

For D, I have two notes. First, data-binding is the way of the future, you can't ignore it, or say that because it doesn't fit with philosophy 'X' that it shouldn't be done. The biggest reason that open-source projects die is that the industry moves in a way that conflicts with the projects philosophy and leaves the project behind. And this is a good thing, we are collective discovering better ways of doing things and part of that process is rejecting old philosophies. Adapt or die. The second is that you can't protect the programmer from all possible mistakes, and in terms of security their are many worse potential exploits that are much easier to reach for than Reflection. And to be honest, I've never actually heard of a successful Reflection exploit via the .NET Framework, although I am sure someone here knows of one...

However, I see no particular reason to reflect on private members by default, if you need to get data out, you should be using a public property or method to filter the data as needed, that's kind of the point of scoping. Private member reflection is NOT a requirement for UX, WPF being the case in point. I personally like the solution that Reflection in D is opt-out with both command-line options and @noreflect. I would add that a @deepreflect option be made available for those incorrigible souls who absolutely must reflect private members. I would also add that reflection should be default disabled by the compiler for code the works directly with pointers and direct memory access (malloc/free, new/delete, etc.) and the developer can only enable it using a compiler option.

For most of us, code-bloat is not an issue even in the slightest, todays computers can open multi-megabyte executables running JIT's (think .NET) in less than half a second (we have a working example where I work). In this case, a little extra bloat to make my life significantly easier is a non-choice, I'll take the bloat every time. However, I completely agree that there are cases where bloat is bad, anything system level (kernels, drivers, etc.), mobile or power-limited devices (although more of a toss-up), and others like games. But in the end, I have products to deliver, I need to best tools for the job, if that means using reflection, so be it, ideology should never enter the equation. In the case of reflection, it is my opinion that the compiler should provide a full range of tools and let the developers decide how much they want to use; this would make D superior to even .NET, as .NET is extremely limited in it's reflection control.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.com/

On 2011-10-26 21:53, Adam Wilson wrote:
> It's also interesting that you bring up WPF because the project I lead,
> The Horizon Project, is an attempt to create a open-source
> multi-platform framework that is as easy to use as the .NET Framework
> and one of it's core pillars is a declarative XML based UI design
> language. I am evaluating D and so far it is blowing C++ out of the
> water, (we'll need to have long conversations about strings though, I
> think .NET has the right answer here) but I have to admit that I am
> seriously missing Reflection. The problem is that there is currently no
> way to do data-binding without member-level reflection because it is a
> purely run-time concept. That said, I am researching ways around this,
> and I do have ideas ... but that's not for this forum.

Are you saying that you consider using D for this Horizon project? I can recommend you take a look at DWT: www.dsource.org/projects/dwt

Somewhere down the road I've planed to create an interface/window builder for DWT using XML or something similar. I'm thinking something like how it works on Mac OS X using Interface Builder.

> However, I see no particular reason to reflect on private members by
> default, if you need to get data out, you should be using a public
> property or method to filter the data as needed, that's kind of the
> point of scoping. Private member reflection is NOT a requirement for UX,
> WPF being the case in point. I personally like the solution that
> Reflection in D is opt-out with both command-line options and
> @noreflect. I would add that a @deepreflect option be made available for
> those incorrigible souls who absolutely must reflect private members. I
> would also add that reflection should be default disabled by the
> compiler for code the works directly with pointers and direct memory
> access (malloc/free, new/delete, etc.) and the developer can only enable
> it using a compiler option.

I don't understand this one. Should the compiler disable reflection as soon as it sees malloc/free? On what level should it be disabled? I mean, the runtime needs to be able to use these functions to implement the memory manager, i.e. the garbage collector and other things as well.

-- 
/Jacob Carlborg

On Wed, 26 Oct 2011 14:20:47 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
[snip]
> Am 26.10.11 07:16, schrieb Robert Jacques:
[snip]

> I know, that "runtime self-modification" and runtime code generation is
> a "dangerous" feature.

As a point of semantics, actual "runtime self-modification" is no longer available outside a VM; the practice is heavily used in viruses, worms, etc and practically nowhere else, so both OS, hardware and anti-virus vendors have ganged up against this practice. :)

On Wed, 26 Oct 2011 14:20:47 -0400, Jonny Dee <jonnyd@gmx.net> wrote:
[snip]
> Am 26.10.11 07:16, schrieb Robert Jacques:
[snip]
> I've got no problem with an approach which puts runtime reflection
> capabiities into a separate library. No matter were you look at, Java,
> C#, or Qt, all have a library for reflection purpose. I do not see,
> however, how this might be done without compiler support.

Long ago, when reflection in was being discussed one of the big reasons we turned to compile-time reflection first, besides its synergy with other meta-programming features, is because compile-time reflection enables runtime-reflection. A decent portion of the community has always assumed that D's runtime reflection support would come from a library exploiting D's compile-time reflection. All it would take is someone willing to put in the effort and time to do so. So far, that someone has been me. :)

My new Variant (https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht) makes heavy use of compile-time reflection and provides runtime-reflection capabilities. Currently, I have focused on dispatch, i.e. the calling of member functions, the calling of static members and the setting of fields, as I don't know what the introspection API should look like. I have found that while __traits does provide all the information I need, it's often not in the format I wish to consume it in. So I have been holding back on implementation in order to solicit the feedback of the community.

As for ease of use, Variant generates reflection information for every type it is exposed to. That includes all return types, fields and member arguments types for every type assigned to or retrieved from Variant. Most of the time, exposure happens naturally and everything 'Just Works' and no effort is needed. For example:

Variant var = student;
assert(var.grade.mark == student.grade.mark)

Or

Variant var = Object.factory("example.Student");
assert(var.grade == (var.get!(Student)).grade);

Now it is possible for types, particularly sub-classes, to never be retrieved from or assigned to, in which case only they can only use their interfaces/base-classes type information, without some form of registration. For example,

Variant.__register!Student;

And with the new module reflection capabilities, the hassle / annoyance of registration can be reduced to that of an import statement. Or even a single statement that would reflect all the imports in a module. (Registration works via static ctors, so their's no worry about using reflection before registration, or duplicate registrations, etc)

Now, this probably gets to the heart of why I don't like the concept of @noreflect, @reflect or any other fixed reflection option: there is no right answer to whether or not a class should support reflection. Trying to find a single choice (or all/none) that fits the embedded, iOS/Driod/Windows 8, Desktop, Server, etc developer is fool hardy. Letting each developer choose how much or how little they need feels right to me. And in D we have the tools to provide the granularity of control to make this declaration of reflection intent as succinct as possible.

On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob@me.com> wrote:

> Are you saying that you consider using D for this Horizon project? I can recommend you take a look at DWT: www.dsource.org/projects/dwt
>
> Somewhere down the road I've planed to create an interface/window builder for DWT using XML or something similar. I'm thinking something like how it works on Mac OS X using Interface Builder.

I am, and I have looked at DWT. My problem with it is a one that is endemic to open-source UI framework. Microsoft recognized a decade ago that UI widgets whose look-and-feel is defined and controlled by the Operating System are going the way of the dodo. Out of that realization WPF was born. UI designers today want the ability to control every pixel of the UI's presentation. And the reasons for this are two-fold. The first is that it turns out that most OS designers are fantastically bad at UI and design in general. It takes epic piles of cash to pull of a decent one and even then there is a still a "programmers were here" look to it (with the notable exception of iOS/OSX where designers rule the roost). The second is product differentiation. Nobody wants an app that looks like every other app because it actually becomes impossible for the user to distinguish which app works best for them. Users ONLY look at the UI, and if the app doesn't look good, they wont "buy" it, even if it's free. This is non-negotiable. Users, when given two apps that do the same thing, even for different prices, will pick the prettier one every time, because the prettier one is perceived as being "better". It's called the Attractiveness Bias and it is a well-known principle in the design world. Who would you rather look at all day, Alessandra Ambrosio or Rosie O'Donnell? I rest my case.

I maintain that this is prime reason that Linux on the desktop has failed miserably, and I think Android proves my point. Android's key win is that it put a usable UI on top of Linux. People never had a problem with the price of Linux, they just couldn't stand to look at it. The Linux Desktop LOOKS industrial and it's apps for the most part look the same (I know of a few outliers that did a good job, but it isn't the norm). My point is that the day of cookie cutter apps is over. Anyone designing for that paradigm is history. Microsoft's latest UI paradigm, "Metro", is just a different cut-down version of WPF similar to Silverlight. Microsoft has no plans to go back to OS controlled UI styles; Metro and WPF are the plan for the next 15 years. (I attended the MS BUILD conference, they made this plan very ... nay, EXTREMELY clear).

Open-source is chronically behind the big boys with money, precisely because FOSS doesn't have the money to sling around for Testing and Usability Studies, and most FOSS guys don't want to mess around with that stuff anyways. You see FOSS guys tend to be engineers; they can put up with, and even like, industrial looking interfaces. But programmers also have a giant blind-spot when it comes to users. Most programmers view users as a lower species and assume that they will be delighted by whatever the programmer deigns to bequeath to them. But if you look at the successful people in the tech industry (*ahem* Steve Jobs) you'll find an attitude that is the exact opposite. Jobs was so focused on delivering what the user wanted that he would publicly berate any programmer who thought they knew better than the designer. While I don't necessarily agree with Jobs' management style, there is a reason why Apple is the second largest company in the world right now, and it has nothing to do with how good Apple's engineering is (which I hear is average at best). Despite programmers' best efforts, the world of technology is no longer controlled by programmers. For better or worse, users determine our course now. The open-source community would do well to embrace the user.

But without a first-class UI framework, that will never happen. In terms of capability and usability, both Apple and Microsoft have beat the best FOSS has to offer by a decade at least. I looked, searched, and scoured, but the fact of the matter is, even the usable FOSS UI offerings are pitiful when compared to the commercial offerings. The Horizon Project got it's start because there has been a trend in recent commercial UI offerings from towards increasing reliance on the operating system itself. Metro XAML just flat won't work on anything other than Windows 8, Silverlight is a second class citizen at MS now, Cocoa only works on Mac etc. My goal with The Horizon Project is to create First-Class UI Framework for multiple platforms so that programmers don't have to rewrite the UI from scratch for each new platform they want to support and then open-source it so that the commercial OS vendors can't pervert it for their own purposes. I want to put [some of] the power back into the programmers hands.

Apologies for the length, but this is a topic that is of some interest to me. :-)

> I don't understand this one. Should the compiler disable reflection as soon as it sees malloc/free? On what level should it be disabled? I mean, the runtime needs to be able to use these functions to implement the memory manager, i.e. the garbage collector and other things as well.

The idea is to automatically prevent reflection access to sections of the program where using direct memory manipulation could potentially result in security holes but provide a way out for the programmer if they really wanted it. C/C++ is famous for programmer bugs around memory handling. If the compiler automatically disabled reflection, by sticking @noreflect in front of a function that used malloc or new, it could potentially prevent those types of memory manipulation flaws and help keep the Reflection attacks to a minimum. It was an idea that I was throwing out there. But I don't know enough about D yet to know if it's the right way to handle it. And I have to admit I am little confused though as I would hope that Reflection would be disabled on the GC, because I have never personally had a reason to reflect into the GC...

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.com/

On 2011-10-27 20:42, Adam Wilson wrote:
> On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob@me.com> wrote:
>
>> Are you saying that you consider using D for this Horizon project? I
>> can recommend you take a look at DWT: www.dsource.org/projects/dwt
>>
>> Somewhere down the road I've planed to create an interface/window
>> builder for DWT using XML or something similar. I'm thinking something
>> like how it works on Mac OS X using Interface Builder.
>
> I am, and I have looked at DWT. My problem with it is a one that is
> endemic to open-source UI framework. Microsoft recognized a decade ago
> that UI widgets whose look-and-feel is defined and controlled by the
> Operating System are going the way of the dodo. Out of that realization
> WPF was born. UI designers today want the ability to control every pixel
> of the UI's presentation. And the reasons for this are two-fold. The
> first is that it turns out that most OS designers are fantastically bad
> at UI and design in general. It takes epic piles of cash to pull of a
> decent one and even then there is a still a "programmers were here" look
> to it (with the notable exception of iOS/OSX where designers rule the
> roost).

I can agree to that.

> The second is product differentiation. Nobody wants an app that
> looks like every other app because it actually becomes impossible for
> the user to distinguish which app works best for them.

I want that. Because I know how the GUI works and it will be easy for me to learn if it follows the guidelines of the platform. Also I don't need to figure out if I can use the scroll wheel on the mouse on this, what looks like a, scroll bar. If the application have a native look and feel I know how to use the widgets.

> Users ONLY look
> at the UI, and if the app doesn't look good, they wont "buy" it, even if
> it's free. This is non-negotiable. Users, when given two apps that do
> the same thing, even for different prices, will pick the prettier one
> every time, because the prettier one is perceived as being "better".
> It's called the Attractiveness Bias and it is a well-known principle in
> the design world. Who would you rather look at all day, Alessandra
> Ambrosio or Rosie O'Donnell? I rest my case.

That's why I use Mac OS X where the native applications look good :)

> I maintain that this is prime reason that Linux on the desktop has
> failed miserably, and I think Android proves my point. Android's key win
> is that it put a usable UI on top of Linux. People never had a problem
> with the price of Linux, they just couldn't stand to look at it. The
> Linux Desktop LOOKS industrial and it's apps for the most part look the
> same (I know of a few outliers that did a good job, but it isn't the
> norm).

I can agree to that.

> My point is that the day of cookie cutter apps is over. Anyone
> designing for that paradigm is history. Microsoft's latest UI paradigm,
> "Metro", is just a different cut-down version of WPF similar to
> Silverlight. Microsoft has no plans to go back to OS controlled UI
> styles; Metro and WPF are the plan for the next 15 years. (I attended
> the MS BUILD conference, they made this plan very ... nay, EXTREMELY
> clear).

I don't know how Metro or WPF is implemented but how says they can't be the native look and feel of the OS.

> Open-source is chronically behind the big boys with money, precisely
> because FOSS doesn't have the money to sling around for Testing and
> Usability Studies, and most FOSS guys don't want to mess around with
> that stuff anyways. You see FOSS guys tend to be engineers; they can put
> up with, and even like, industrial looking interfaces. But programmers
> also have a giant blind-spot when it comes to users. Most programmers
> view users as a lower species and assume that they will be delighted by
> whatever the programmer deigns to bequeath to them. But if you look at
> the successful people in the tech industry (*ahem* Steve Jobs) you'll
> find an attitude that is the exact opposite. Jobs was so focused on
> delivering what the user wanted that he would publicly berate any
> programmer who thought they knew better than the designer. While I don't
> necessarily agree with Jobs' management style, there is a reason why
> Apple is the second largest company in the world right now, and it has
> nothing to do with how good Apple's engineering is (which I hear is
> average at best). Despite programmers' best efforts, the world of
> technology is no longer controlled by programmers. For better or worse,
> users determine our course now. The open-source community would do well
> to embrace the user.

I have no trouble in letting a designer decide how GUI should look like, in fact, as you say it would be better if they did. But not everyone can have that luxury. Since I'm no designer you do the best I can.

> But without a first-class UI framework, that will never happen. In terms
> of capability and usability, both Apple and Microsoft have beat the best
> FOSS has to offer by a decade at least. I looked, searched, and scoured,
> but the fact of the matter is, even the usable FOSS UI offerings are
> pitiful when compared to the commercial offerings. The Horizon Project
> got it's start because there has been a trend in recent commercial UI
> offerings from towards increasing reliance on the operating system
> itself. Metro XAML just flat won't work on anything other than Windows
> 8, Silverlight is a second class citizen at MS now, Cocoa only works on
> Mac etc. My goal with The Horizon Project is to create First-Class UI
> Framework for multiple platforms so that programmers don't have to
> rewrite the UI from scratch for each new platform they want to support
> and then open-source it so that the commercial OS vendors can't pervert
> it for their own purposes. I want to put [some of] the power back into
> the programmers hands.
>
> Apologies for the length, but this is a topic that is of some interest
> to me. :-)

I's a topic of interest to me as well. But I, on the other hand, prefer a native look and fell of applications. Instead of having yet another application with yet another GUI that doesn't work properly. If I see something that looks like a scroll bar I assume I can use it like a scroll bar. But that's not true for many applications, instead they implement the bare minimum for having the scroll bar "work", i.e. I may not be able to scroll using the wheel on the mouse. A great example of this are scroll bars in games.

In my experience of non-native GUI's they always perform worse the native GUI's.

Anyway, you can still use DWT to draw your own widgets. Even if you don't use any native widgets you still need to be able to draw something on a surface. DWT has a platform independent way of doing that.

IBM Lotus Software and IBM Rational Software are built on SWT (which DWT is a port of) uses a lot a non-native widgets. There's an application called "Azureus" that had its own GUI that didn't look native at all, apparently it's called Vuze these days and it looks native.

>> I don't understand this one. Should the compiler disable reflection as
>> soon as it sees malloc/free? On what level should it be disabled? I
>> mean, the runtime needs to be able to use these functions to implement
>> the memory manager, i.e. the garbage collector and other things as well.
>
> The idea is to automatically prevent reflection access to sections of
> the program where using direct memory manipulation could potentially
> result in security holes but provide a way out for the programmer if
> they really wanted it. C/C++ is famous for programmer bugs around memory
> handling. If the compiler automatically disabled reflection, by sticking
> @noreflect in front of a function that used malloc or new, it could
> potentially prevent those types of memory manipulation flaws and help
> keep the Reflection attacks to a minimum. It was an idea that I was
> throwing out there. But I don't know enough about D yet to know if it's
> the right way to handle it. And I have to admit I am little confused
> though as I would hope that Reflection would be disabled on the GC,
> because I have never personally had a reason to reflect into the GC...

What happens when you use class A from class B and the compiler has added @noreflect to class A? Will it add it to B as well? If not, how does the compiler know that?

There has been a similar discussion about having the compiler insert "pure" automatic on functions. But what might happens is a function that is called by another function change it implementation making it no longer pure. Which means your function will no longer be pure and you have no idea about it.

-- 
/Jacob Carlborg

I think the compile-time reflection should be improved first, because
run-time reflection can be built upon that.
Currently D's compile-time reflection is not fully developed.

On Fri, 28 Oct 2011 00:07:57 -0700, Jacob Carlborg <doob@me.com> wrote:

> On 2011-10-27 20:42, Adam Wilson wrote:
>> On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob@me.com> wrote:
>>
>>> Are you saying that you consider using D for this Horizon project? I
>>> can recommend you take a look at DWT: www.dsource.org/projects/dwt
>>>
>>> Somewhere down the road I've planed to create an interface/window
>>> builder for DWT using XML or something similar. I'm thinking something
>>> like how it works on Mac OS X using Interface Builder.
>>
>> I am, and I have looked at DWT. My problem with it is a one that is
>> endemic to open-source UI framework. Microsoft recognized a decade ago
>> that UI widgets whose look-and-feel is defined and controlled by the
>> Operating System are going the way of the dodo. Out of that realization
>> WPF was born. UI designers today want the ability to control every pixel
>> of the UI's presentation. And the reasons for this are two-fold. The
>> first is that it turns out that most OS designers are fantastically bad
>> at UI and design in general. It takes epic piles of cash to pull of a
>> decent one and even then there is a still a "programmers were here" look
>> to it (with the notable exception of iOS/OSX where designers rule the
>> roost).
>
> I can agree to that.
>
>> The second is product differentiation. Nobody wants an app that
>> looks like every other app because it actually becomes impossible for
>> the user to distinguish which app works best for them.
>
> I want that. Because I know how the GUI works and it will be easy for me to learn if it follows the guidelines of the platform. Also I don't need to figure out if I can use the scroll wheel on the mouse on this, what looks like a, scroll bar. If the application have a native look and feel I know how to use the widgets.
>

Ahh, thats the true power of WPF. Presentation and Implementation are cleanly separated. I recent re-skinned every scrollbar in one of our WPF apps by writing the 30-line style once, placing it in the global resource dictionary with implicit styling enabled, and bam, every scrollbar in the app looks different, but functions in the exact same way any windows user would expect. There are cases where you can write code that calls up into the presentation layer and thus introducing a dependency on the presentation layer, but this is actively discouraged, and when I found one recently in a 3rd party vendors control, it was removed by the next release.
I can go on and on, but the point is, there are only two frameworks in existence that can do that are the WPF family and JavaFX. JavaFX is pathetic compared to WPF and is of course tied to an Oracle *shudder* controlled langauge. And a few months ago we had thousands of developers on the Silverlight forum looking for WPF family alternatives, all that they could find was JavaFX.

>> Users ONLY look
>> at the UI, and if the app doesn't look good, they wont "buy" it, even if
>> it's free. This is non-negotiable. Users, when given two apps that do
>> the same thing, even for different prices, will pick the prettier one
>> every time, because the prettier one is perceived as being "better".
>> It's called the Attractiveness Bias and it is a well-known principle in
>> the design world. Who would you rather look at all day, Alessandra
>> Ambrosio or Rosie O'Donnell? I rest my case.
>
> That's why I use Mac OS X where the native applications look good :)
>
>> I maintain that this is prime reason that Linux on the desktop has
>> failed miserably, and I think Android proves my point. Android's key win
>> is that it put a usable UI on top of Linux. People never had a problem
>> with the price of Linux, they just couldn't stand to look at it. The
>> Linux Desktop LOOKS industrial and it's apps for the most part look the
>> same (I know of a few outliers that did a good job, but it isn't the
>> norm).
>
> I can agree to that.
>
>> My point is that the day of cookie cutter apps is over. Anyone
>> designing for that paradigm is history. Microsoft's latest UI paradigm,
>> "Metro", is just a different cut-down version of WPF similar to
>> Silverlight. Microsoft has no plans to go back to OS controlled UI
>> styles; Metro and WPF are the plan for the next 15 years. (I attended
>> the MS BUILD conference, they made this plan very ... nay, EXTREMELY
>> clear).
>
> I don't know how Metro or WPF is implemented but how says they can't be the native look and feel of the OS.
>

Nobody, in fact WPF has three default looks out of the box, one for Windows 2000, another for WinXP, and another for Aero (Vista+). WPF defaults to the correct one for that system unless you override it. It's just that in my experience, the default look is a lowest common demoninator, and not a good one.

>> Open-source is chronically behind the big boys with money, precisely
>> because FOSS doesn't have the money to sling around for Testing and
>> Usability Studies, and most FOSS guys don't want to mess around with
>> that stuff anyways. You see FOSS guys tend to be engineers; they can put
>> up with, and even like, industrial looking interfaces. But programmers
>> also have a giant blind-spot when it comes to users. Most programmers
>> view users as a lower species and assume that they will be delighted by
>> whatever the programmer deigns to bequeath to them. But if you look at
>> the successful people in the tech industry (*ahem* Steve Jobs) you'll
>> find an attitude that is the exact opposite. Jobs was so focused on
>> delivering what the user wanted that he would publicly berate any
>> programmer who thought they knew better than the designer. While I don't
>> necessarily agree with Jobs' management style, there is a reason why
>> Apple is the second largest company in the world right now, and it has
>> nothing to do with how good Apple's engineering is (which I hear is
>> average at best). Despite programmers' best efforts, the world of
>> technology is no longer controlled by programmers. For better or worse,
>> users determine our course now. The open-source community would do well
>> to embrace the user.
>
> I have no trouble in letting a designer decide how GUI should look like, in fact, as you say it would be better if they did. But not everyone can have that luxury. Since I'm no designer you do the best I can.
>

Neither am I really, I am a programmer by trade who cut his teeth back in the C++ days before C# came along and made me actually productive, but design isn't that hard, if one is willing to let go of their preferences. More to the point though, my company has no on staff designer, so it was either pay large sums of money to a consultant, or I as team lead could do it. I lost. To some degree it's a totally different way of thinking and I find that after a good design session (a week or two), it takes me a couple of days to realign my brain to left-brain-progammer-mode. But hey, they pay me, I jump to. *shrug* There is no mystical powers that are given to designers, anyone can make a good design.

>> But without a first-class UI framework, that will never happen. In terms
>> of capability and usability, both Apple and Microsoft have beat the best
>> FOSS has to offer by a decade at least. I looked, searched, and scoured,
>> but the fact of the matter is, even the usable FOSS UI offerings are
>> pitiful when compared to the commercial offerings. The Horizon Project
>> got it's start because there has been a trend in recent commercial UI
>> offerings from towards increasing reliance on the operating system
>> itself. Metro XAML just flat won't work on anything other than Windows
>> 8, Silverlight is a second class citizen at MS now, Cocoa only works on
>> Mac etc. My goal with The Horizon Project is to create First-Class UI
>> Framework for multiple platforms so that programmers don't have to
>> rewrite the UI from scratch for each new platform they want to support
>> and then open-source it so that the commercial OS vendors can't pervert
>> it for their own purposes. I want to put [some of] the power back into
>> the programmers hands.
>>
>> Apologies for the length, but this is a topic that is of some interest
>> to me. :-)
>
> I's a topic of interest to me as well. But I, on the other hand, prefer a native look and fell of applications. Instead of having yet another application with yet another GUI that doesn't work properly. If I see something that looks like a scroll bar I assume I can use it like a scroll bar. But that's not true for many applications, instead they implement the bare minimum for having the scroll bar "work", i.e. I may not be able to scroll using the wheel on the mouse. A great example of this are scroll bars in games.
>
> In my experience of non-native GUI's they always perform worse the native GUI's.
>

Actually we found that by re-skinning our app intelligently we were able to load a large list of complex data 4x faster than the default skin. The reason for this is that in WPF all drawing is done the same way, it doesn't matter if it's a system skin or not, it's just a skin, there are no special shortcuts for the system skin to take. At it's heart WPF is based on DirectX9. This means that screen complexity is the ONLY limiting factor.

The other thing that WPF allows is data / presentation separation. I can pass a list box a chunk of data, and the data does not have to concern itself with how the presentation looks or vice versa. The listbox will display the data according to however the style is setup it. And THEN if I really want to I can change the presentation style on the fly based on user input. Can DWT do that in 2 lines of code? WPF Can. That's the power of reflection based data-binding. In fact I spend most of my time writing (much more compact) XAML code to do presentation layer work than I do write C#, to the point where the C# coding makes up less than half of the work done on a screen. It's usually things like load this item based on a selection changed event or some such that XAML has no way to do (and shouldn't, thats implementation).

> Anyway, you can still use DWT to draw your own widgets. Even if you don't use any native widgets you still need to be able to draw something on a surface. DWT has a platform independent way of doing that.
>

Yea, but how much work is it? Can I create a completely new look for a control in 30 lines or less without modifying the implementation code at all to make the new look work? That's the power I'm after.

> IBM Lotus Software and IBM Rational Software are built on SWT (which DWT is a port of) uses a lot a non-native widgets. There's an application called "Azureus" that had its own GUI that didn't look native at all, apparently it's called Vuze these days and it looks native.
>

That's IBM, they have supertanker loads of money, the open-source world does not. I expect IBM to re-skin anything they want to their specifications, they can afford it. WPF type apps are game changers in that they allow very small teams to create powerful, beautiful, apps in something less than a decade, and without requiring multi-million dollar investments. As an example, our flagship app (which we are keeping under tight wraps for the following reason) has the potential to completely wipe out multiple billions of dollars worth of corporate market capitalization and probably sink some long-time industry names in the process, all with a team  of less than 10 guys. At least, thats our goal. But that goal is only achievable because we can move orders of magnitude faster than they can. It also doesn't hurt that their interface looks like it was designed in 1999 (seriously) and is definitely a traditional type of framework (the standard Java one IIRC). Thank you WPF, it simply wouldn't be possible without you. So when I say the world is leaving behind traditional UI frameworks, I am not kidding; it just wont happen that quickly as corp IT loathes change, but it's already starting. (See: iOS and Android apps and CxO's demanding their internal apps on those platforms.)

Now imagine a world where not only can open-source compete with the big-boys, but take serious chunks out of their consumer mind-share. I want that. But, the only way that is happening is with a framework that allows open-source to compete at the same speed as the big boys can afford while delivering what the customer wants. Any app is only as good as the number of people using it.

It's not the big who eat the small, it's the fast who eat the slow. :-)

>>> I don't understand this one. Should the compiler disable reflection as
>>> soon as it sees malloc/free? On what level should it be disabled? I
>>> mean, the runtime needs to be able to use these functions to implement
>>> the memory manager, i.e. the garbage collector and other things as well.
>>
>> The idea is to automatically prevent reflection access to sections of
>> the program where using direct memory manipulation could potentially
>> result in security holes but provide a way out for the programmer if
>> they really wanted it. C/C++ is famous for programmer bugs around memory
>> handling. If the compiler automatically disabled reflection, by sticking
>> @noreflect in front of a function that used malloc or new, it could
>> potentially prevent those types of memory manipulation flaws and help
>> keep the Reflection attacks to a minimum. It was an idea that I was
>> throwing out there. But I don't know enough about D yet to know if it's
>> the right way to handle it. And I have to admit I am little confused
>> though as I would hope that Reflection would be disabled on the GC,
>> because I have never personally had a reason to reflect into the GC...
>
> What happens when you use class A from class B and the compiler has added @noreflect to class A? Will it add it to B as well? If not, how does the compiler know that?
>
> There has been a similar discussion about having the compiler insert "pure" automatic on functions. But what might happens is a function that is called by another function change it implementation making it no longer pure. Which means your function will no longer be pure and you have no idea about it.
>


-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.com/

On 2011-10-28 21:36, Adam Wilson wrote:
> Ahh, thats the true power of WPF. Presentation and Implementation are
> cleanly separated. I recent re-skinned every scrollbar in one of our WPF
> apps by writing the 30-line style once, placing it in the global
> resource dictionary with implicit styling enabled, and bam, every
> scrollbar in the app looks different, but functions in the exact same
> way any windows user would expect. There are cases where you can write
> code that calls up into the presentation layer and thus introducing a
> dependency on the presentation layer, but this is actively discouraged,
> and when I found one recently in a 3rd party vendors control, it was
> removed by the next release.
> I can go on and on, but the point is, there are only two frameworks in
> existence that can do that are the WPF family and JavaFX. JavaFX is
> pathetic compared to WPF and is of course tied to an Oracle *shudder*
> controlled langauge. And a few months ago we had thousands of developers
> on the Silverlight forum looking for WPF family alternatives, all that
> they could find was JavaFX.

It's not the look of the, in this example, scroll bar, it's the behavior. It may look like a scroll bar but it may not behave in all the ways as a real, native, scroll bar would.

> Nobody, in fact WPF has three default looks out of the box, one for
> Windows 2000, another for WinXP, and another for Aero (Vista+). WPF
> defaults to the correct one for that system unless you override it. It's
> just that in my experience, the default look is a lowest common
> demoninator, and not a good one.

To me it sounds like WPF is the native GUI. Then why are you complaining about native GUI's?

> Neither am I really, I am a programmer by trade who cut his teeth back
> in the C++ days before C# came along and made me actually productive,
> but design isn't that hard, if one is willing to let go of their
> preferences. More to the point though, my company has no on staff
> designer, so it was either pay large sums of money to a consultant, or I
> as team lead could do it. I lost. To some degree it's a totally
> different way of thinking and I find that after a good design session (a
> week or two), it takes me a couple of days to realign my brain to
> left-brain-progammer-mode. But hey, they pay me, I jump to. *shrug*
> There is no mystical powers that are given to designers, anyone can make
> a good design.

I'm just saying that I'm not a good designer and if a designer is available I would prefer that they did the design.

> Actually we found that by re-skinning our app intelligently we were able
> to load a large list of complex data 4x faster than the default skin.
> The reason for this is that in WPF all drawing is done the same way, it
> doesn't matter if it's a system skin or not, it's just a skin, there are
> no special shortcuts for the system skin to take. At it's heart WPF is
> based on DirectX9. This means that screen complexity is the ONLY
> limiting factor.

If the changing the skin of an application can increase the data load by four times it sounds like something is seriously  wrong.

On Mac OS X a list only loads the data that is displayed so I don't think the amount of data matters. The same can be done using virtual tables in DWT.

> The other thing that WPF allows is data / presentation separation. I can
> pass a list box a chunk of data, and the data does not have to concern
> itself with how the presentation looks or vice versa. The listbox will
> display the data according to however the style is setup it. And THEN if
> I really want to I can change the presentation style on the fly based on
> user input. Can DWT do that in 2 lines of code? WPF Can. That's the
> power of reflection based data-binding. In fact I spend most of my time
> writing (much more compact) XAML code to do presentation layer work than
> I do write C#, to the point where the C# coding makes up less than half
> of the work done on a screen. It's usually things like load this item
> based on a selection changed event or some such that XAML has no way to
> do (and shouldn't, thats implementation).

No, probably not. It's not as easy to change the theme in SWT/DWT since it's supposed to use the native look and feel.

It appears that you can at least set colors and similar properties using CSS: http://jaxenter.com/css-styling-in-eclipse-helios-32465.html

>> Anyway, you can still use DWT to draw your own widgets. Even if you
>> don't use any native widgets you still need to be able to draw
>> something on a surface. DWT has a platform independent way of doing that.
>>
>
> Yea, but how much work is it? Can I create a completely new look for a
> control in 30 lines or less without modifying the implementation code at
> all to make the new look work? That's the power I'm after.

No you probably can't. I don't know what you want to achieve with your project but I got the impression that you were looking to build a new GUI library that behaves something like WPF.

Regardless of how this GUI library works you need somehow to draw the widgets on the screen. I'm just saying that DWT has a platform independent way of doing this. On the other hand, if you're creating a completely new GUI library you probably would not want to depend on DWT. But instead creating bindings directly to the underlying drawing operations for Cairo, Quartz and DirectX or whatever Windows uses these days.

-- 
/Jacob Carlborg