On Tuesday, 26 November 2019 at 12:15:57 UTC, rikki cattermole wrote:
> - Render pipeline is from AAA games, the only person I trust with designing it is Manu

I don't understand what you mean by this. A game rendering pipeline and a desktop UI rendering pipeline are fundamentally very, very different beasts. One shouldn't be used to emulate the other as the use cases are far too dissimilar.

> - UI automation (if you don't have this you can literally be sued and that ignores the customers you will loose and bad press in general)

Let's not resort to hyperbole here, please. I only know of lawsuits against websites lacking accessibility features. Has any such lawsuit ever been filed for a desktop application? The real risk of a lawsuit is low.

Having said that, there's applications that definitely should be accessible. There's others where a certain kind of accessibility support is just not possible. Screen readers and Photoshop obviously make little sense in combination, for example. And accessibility is different from automation. There's different APIs for that on different platforms.

So, in general, I'm still not seeing a proper justification for not building on top of the work that already exists in dlangUI.

On 28/11/2019 5:06 AM, Gregor Mückl wrote:
> On Tuesday, 26 November 2019 at 12:15:57 UTC, rikki cattermole wrote:
>> - UI automation (if you don't have this you can literally be sued and that ignores the customers you will loose and bad press in general)
> 
> Let's not resort to hyperbole here, please. I only know of lawsuits against websites lacking accessibility features. Has any such lawsuit ever been filed for a desktop application? The real risk of a lawsuit is low.

It covers mobile applications as well. Different API, same scope, same code pretty much.

> Having said that, there's applications that definitely should be accessible. There's others where a certain kind of accessibility support is just not possible. Screen readers and Photoshop obviously make little sense in combination, for example. And accessibility is different from automation. There's different APIs for that on different platforms.
> 
> So, in general, I'm still not seeing a proper justification for not building on top of the work that already exists in dlangUI.

UI automation API's is how accessibility programs like screen readers work.

"Microsoft UI Automation is an accessibility framework that enables Windows applications to provide and consume programmatic information about user interfaces (UIs). It provides programmatic access to most UI elements on the desktop. It enables assistive technology products, such as screen readers, to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI." - https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32

On Wednesday, 27 November 2019 at 16:34:33 UTC, rikki cattermole wrote:
> On 28/11/2019 5:06 AM, Gregor Mückl wrote:
>> On Tuesday, 26 November 2019 at 12:15:57 UTC, rikki cattermole wrote:
>>> - UI automation (if you don't have this you can literally be sued and that ignores the customers you will loose and bad press in general)
>> 
>> Let's not resort to hyperbole here, please. I only know of lawsuits against websites lacking accessibility features. Has any such lawsuit ever been filed for a desktop application? The real risk of a lawsuit is low.
>
> It covers mobile applications as well. Different API, same scope, same code pretty much.
>

Sorry, I don't follow.

>> Having said that, there's applications that definitely should be accessible. There's others where a certain kind of accessibility support is just not possible. Screen readers and Photoshop obviously make little sense in combination, for example. And accessibility is different from automation. There's different APIs for that on different platforms.
>> 
>> So, in general, I'm still not seeing a proper justification for not building on top of the work that already exists in dlangUI.
>
> UI automation API's is how accessibility programs like screen readers work.
>
> "Microsoft UI Automation is an accessibility framework that enables Windows applications to provide and consume programmatic information about user interfaces (UIs). It provides programmatic access to most UI elements on the desktop. It enables assistive technology products, such as screen readers, to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI." - https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32

No, this is only how Microsoft chose to name their accessibility interface. Google and Apple use accessibility as the relevant term, including in their actual API naming. The same goes for Gtk and Qt, which are the most common implementations of accessibility features on top of X11 (sadly, the X protocol doesn't have any notion of accessibility itself). Please don't use the single outlier's unique terms.

On 28/11/2019 6:08 AM, Gregor Mückl wrote:
> On Wednesday, 27 November 2019 at 16:34:33 UTC, rikki cattermole wrote:
>>> Having said that, there's applications that definitely should be accessible. There's others where a certain kind of accessibility support is just not possible. Screen readers and Photoshop obviously make little sense in combination, for example. And accessibility is different from automation. There's different APIs for that on different platforms.
>>>
>>> So, in general, I'm still not seeing a proper justification for not building on top of the work that already exists in dlangUI.
>>
>> UI automation API's is how accessibility programs like screen readers work.
>>
>> "Microsoft UI Automation is an accessibility framework that enables Windows applications to provide and consume programmatic information about user interfaces (UIs). It provides programmatic access to most UI elements on the desktop. It enables assistive technology products, such as screen readers, to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI." - https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32
> 
> No, this is only how Microsoft chose to name their accessibility interface. Google and Apple use accessibility as the relevant term, including in their actual API naming. The same goes for Gtk and Qt, which are the most common implementations of accessibility features on top of X11 (sadly, the X protocol doesn't have any notion of accessibility itself). Please don't use the single outlier's unique terms.

"Action methods. The NSAccessibility protocol also defines a number of methods that simulate button presses, mouse clicks and selections in your view or control. By implementing these methods, you give accessibility clients the ability to drive your view or control."

1. The user says, “Open Preferences window.”
2. The screen reader sends a message to the app’s accessible element, asking for a reference to the menu bar accessible element. It then queries the menu bar for a list of its children and queries each child for its title. As soon as it finds the one whose title matches app’s name (that is, the application menu). A second iteration lets it find the Preferences menu item within the application menu. Finally, the screen reader tells the Preferences menu item to perform the press action.

https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/OSXAXmodel.html#//apple_ref/doc/uid/TP40001078-CH208-TPXREF101

Sounds like UI automation to me. Regardless of what they named it.



And for ATK which is Gtk's library: https://developer.gnome.org/atk/stable/AtkAction.html

"AtkAction should be implemented by instances of AtkObject classes with which the user can interact directly, i.e. buttons, checkboxes, scrollbars, e.g. components which are not "passive" providers of UI information."

Sounds like UI automation to me even if accessibility centric.



From QT with regards to accessibility support on Windows:

"Also, the new UI Automation support in Qt may become useful for application testing, since it can provide metadata and programmatic control of UI elements, which can be leveraged by automated test suites and other tools."

https://www.qt.io/blog/2018/02/20/qt-5-11-brings-new-accessibility-backend-windows

Linux support is hard to find anything solid on for QT, but they do have support for ATK bundled with it as part of AT-SPI2 specification.

On Wednesday, 27 November 2019 at 16:06:38 UTC, Gregor Mückl wrote:
> On Tuesday, 26 November 2019 at 12:15:57 UTC, rikki cattermole wrote:
>> - Render pipeline is from AAA games, the only person I trust with designing it is Manu
>
> I don't understand what you mean by this. A game rendering pipeline and a desktop UI rendering pipeline are fundamentally very, very different beasts. One shouldn't be used to emulate the other as the use cases are far too dissimilar.

Stop saying this. It's thoroughly incorrect. Cripes.

If you think the desktop layout engine introduced in Windows Vista, or even the layout engines used in mobile browsers and current desktop browsers, doesn't have a ton in common with a game rendering pipeline then your knowledge is well outdated.

On Thursday, 28 November 2019 at 15:29:21 UTC, Ethan wrote:
> On Wednesday, 27 November 2019 at 16:06:38 UTC, Gregor Mückl wrote:
>> On Tuesday, 26 November 2019 at 12:15:57 UTC, rikki cattermole wrote:
>>> - Render pipeline is from AAA games, the only person I trust with designing it is Manu
>>
>> I don't understand what you mean by this. A game rendering pipeline and a desktop UI rendering pipeline are fundamentally very, very different beasts. One shouldn't be used to emulate the other as the use cases are far too dissimilar.
>
> Stop saying this. It's thoroughly incorrect. Cripes.
>
> If you think the desktop layout engine introduced in Windows Vista, or even the layout engines used in mobile browsers and current desktop browsers, doesn't have a ton in common with a game rendering pipeline then your knowledge is well outdated.

As an example of it, Windows 10 composition layer is basically a thin wrapper around DirectX, exposing most of its features to WinUI.

https://docs.microsoft.com/en-us/windows/uwp/composition/visual-layer

https://channel9.msdn.com/Events/TechDays/Techdays-2016-The-Netherlands/The-Visual-Layer-Animations-Effects--More-in-Windows-10-Anniversary-Edition

On Thursday, 28 November 2019 at 15:42:29 UTC, Paulo Pinto wrote:
> As an example of it, Windows 10 composition layer is basically a thin wrapper around DirectX, exposing most of its features to WinUI.

Well, you could've said that for Amiga too, but you do things differently for a UI than a game. Then again you do things differently for different types of graphics engines and hardware too, and that does change over time... so the argument is kinda moot from the get go.

Anyway, if you are building a portable UI toolkit for existing windowing systems you should not open the most expressive graphics context on all platforms. You should adopt to the platform (save resources, get better interop with native toolkits etc). AND be future proof. Meaning: be conservative. If you want to run on embedded slow monochrome LCDs, networked x-windows, os-x, windows... well, you won't get very far with a game engine.

Besides there is no way the D community can fully sustain a portable UI implemented on the hardware level anyway... Just use Skia conservatively and save yourself a few man years of design, development, porting, debugging and maintenance... and that is only for the lowest level.

On Thursday, 28 November 2019 at 15:29:21 UTC, Ethan wrote:
> On Wednesday, 27 November 2019 at 16:06:38 UTC, Gregor Mückl wrote:
>> On Tuesday, 26 November 2019 at 12:15:57 UTC, rikki cattermole wrote:
>>> - Render pipeline is from AAA games, the only person I trust with designing it is Manu
>>
>> I don't understand what you mean by this. A game rendering pipeline and a desktop UI rendering pipeline are fundamentally very, very different beasts. One shouldn't be used to emulate the other as the use cases are far too dissimilar.
>
> Stop saying this. It's thoroughly incorrect. Cripes.
>
> If you think the desktop layout engine introduced in Windows Vista, or even the layout engines used in mobile browsers and current desktop browsers, doesn't have a ton in common with a game rendering pipeline then your knowledge is well outdated.

Do you have any more information on the topic? I remember digging through Qt and there are sections that completely avoid the GPU all together as it is too inaccurate for the computation that was required. Can't recall exactly what it was.

On Thursday, 28 November 2019 at 19:37:47 UTC, Jab wrote:
> Do you have any more information on the topic? I remember digging through Qt and there are sections that completely avoid the GPU all together as it is too inaccurate for the computation that was required. Can't recall exactly what it was.

This would have been an accurate statement when GPUs were entirely fixed function. But then this little technology called "shaders" was introduced to consumer hardware in 2001.

GPUs these days are little more than programmable number crunchers that work *REALLY FAST* in parallel.

On Thursday, 28 November 2019 at 15:29:21 UTC, Ethan wrote:
> On Wednesday, 27 November 2019 at 16:06:38 UTC, Gregor Mückl wrote:
>> On Tuesday, 26 November 2019 at 12:15:57 UTC, rikki cattermole wrote:
>>> - Render pipeline is from AAA games, the only person I trust with designing it is Manu
>>
>> I don't understand what you mean by this. A game rendering pipeline and a desktop UI rendering pipeline are fundamentally very, very different beasts. One shouldn't be used to emulate the other as the use cases are far too dissimilar.
>
> Stop saying this. It's thoroughly incorrect. Cripes.
>
> If you think the desktop layout engine introduced in Windows Vista, or even the layout engines used in mobile browsers and current desktop browsers, doesn't have a ton in common with a game rendering pipeline then your knowledge is well outdated.

I don't want to belabor that point too much, but I can say a few things in response to that:

Yes, compositors are implemented using 3D rendering APIs these days because they slap together textured quads on screen. They don't concern themselves with how the contents of these quads came to be.

And rendering the window contents is where things start to diverge a lot. A game engine is a fundamentally different beast from a renderer for the kind of graphics a UI draws. The graphics primitives that GUI code wants to deal map awkwardly to the GPU rendering pipeline. Sure, there are ways (some of them quite impressive), but it's a pain. There's no explicit scene graph. You can construct a sort of implied scene graph elements from the draw calls the widgets make in their paint event handlers and go from there. But UI code sometimes requests state changes like crazy, switches primitive types, enables and disabling blending, depends a lot on clipping etc... and you can't simply go and reorder most of that. As a result, renderer for that is quite different from a renderer for a 3D scene, and hard to do in its own right. They can use the same GPU rendering API, but the algorithms on top of that are quite different. If you don't believe me, you can go and read some code: ImGUI, cairo-gl, Qt, WPF...

As for browsers: an HTML page is essentially a pretty static scene graph with quite simple constituent elements, with the exception of a few outliers like canvas. The range of styles possible through CSS is limited in such a way that a HTML rendering engine can do a lot of reasoning about that. That's a luxury of not having paint event handlers executing arbitrary code. And typical engines spend quite some time on that - hundreds of ms aren't uncommon on a page load as far as I know. DOM changes through JS can also be surprisingly slow for that reason. All that processing is too heavy for an application that has paint event handlers and wants to refresh quickly.