I haven't posted in announce for a while, so I figured I'd give you guys an update on what's been happening in my libs. This message is a big long, so feel free to just skim to see if you are interested in anything.

Here's my three main public repos:

https://github.com/adamdruppe/arsd
https://github.com/adamdruppe/terminal-emulator
https://github.com/adamdruppe/adrdox

 * * * * *

Going bottom to top: adrdox is the documentation generator for my http://dpldocs.info/ website. In the last few months, it:

* Is easier to use by third parties. If you download it and hit `make`, then run it from its own source directory (that's important right now) `./doc2 path/to/your/project`, it will generate your docs. So you don't have to be on my computer to run it anymore!

* It has a javascript search option, so you can use it out-of-the-box with search enabled on a static site, like  github pages.

* It has gained a lot of automatic cross referencing in source code.

* The documentation generating language got a few more features like footnotes. See the syntax doc here: http://dpldocs.info/experimental-docs/adrdox.syntax.html

 * * * * *

terminal-emulator is my custom Linux terminal emulator. It includes four programs:

* a Linux program that works like xterm, just with my preferences and features. For example, it adjusts the palette so if a program prints yellow on white, it will adjust to a darker yellow to make it easier to read.

* a Windows program that works just like Linux, but instead of running a local shell, it ssh connects to a Linux box and runs the shell there.

* a Linux program that enables tabbing and detaching of terminal sessions, similar to GNU Screen, but forwarding my custom features. For example, scrollback with shift+pageup works through all levels of nesting, and it can do bracketed pastes through the levels.

* and a nested terminal emulator that I rarely use, but could be used to split screens and such.

It also holds the stand-alone terminalemulator.d core, which is independent of any UI.

It has mostly gotten bug fixes over the last several months, including a major one on scrolling back long lines. I use it for all my terminal emulation need (which is almost everything I do on computers!), so the more I use it, the more fixes it gets.

 * * * * *

arsd contains my generally useful library modules. See the docs here: http://dpldocs.info/experimental-docs/arsd.html

Some of the modules are obsolete or experimental, but many of them are stable for many years, including:

* cgi.d can be used for D web programs using existing libraries - it uses simple process and thread model rather than the newfangled async event loops, which means it is compatible with almost any code you write. It works with cgi, fastcgi, scgi, or you can run an embedded http server in it. It stands alone, though web.d (I haven't used it for a while so it might need some updating) and others can build on top of it.

It hasn't changed for a while, no new updates here.

* dom.d is a html or xml dom parser, capable of reading tag soup. Its API is similar to Javascript, so if you have used the browser apis before, you should be able to dive in pretty quickly. It has no required dependencies.

In recent months, it has had a few small bug fixes and gained a new api: `optionSelector` http://dpldocs.info/experimental-docs/arsd.dom.Element.optionSelector.html

It uses the mostly-CSS3-compatible selector syntax, same as querySelector and requireSelector, but returns a null-safe object, meaning you can use it for scrapes.

optionSelector(".foo").innerText; // instead of throwing or segfaulting if .foo does not exist, it just returns an empty string

It has also gained a Document.fromUrl function which will download and parse some web site. This requires characterencodings.d and http2.d as well.


dom.d has also received a few more bug fixes in reading tag soup as I use it for more and more web scraping. See also: htmltotext.d which uses dom.d to convert html to plain text and html.d for more html helper functions. It includes a css expand thing similar to scss. It hasn't changed much recently, either.


* database.d, postgres.d, sqlite.d, mysql.d, and mssql.d are wrappers of the C libraries. They haven't changed much recently.


* http2.d is a http/1.1 client library. (It is the successor to my http.d lib and does not yet have http/2 support. And maybe never will since 1.1 client is plenty good enough for everything I use it for).

It has gotten some new stuff recently:

  - HttpClient is a class that works kinda like a browser, with cookie storage, connection reuse, and relative link handling. It has received a number of bug fixes in the last few months, especially with regard to SSL (for which it requires OpenSSL library btw).

  - The HttpApiClient template has convenience methods for working with Oauth2 json apis. It is still a work in progress but you can read some about it here: http://dpldocs.info/experimental-docs/arsd.http2.HttpApiClient.html It requires jsvar.d


* jsvar.d and script.d provide a Javascript-esque type for D with easy interop into the rest of the language. jsvar itself just provides the data type, and script.d expands it with a dynamic scripting language interpreter too (no outside dependencies btw). They've changed a bit recently:

  - jsvar is even more loosely typed than it was before: accessing a member of a null object simply returns null too. This is convenient for accessing nested JSON objects where they may not be present (similar to the use case for optionSelector in dom.d), though makes errors harder to find. I'll probably go back to some kind of middle ground on that eventually.

  - script has more bug fixes and gains from every jsvar change. Notably, it can now more reliably wrap native D class objects, but it does so in an opaque way, just holding and returning the object on demand, unless the object's specific methods are marked with a UDA `@"arsd_jsvar_compatible"`. That UDA has nicer aliases in a few of my modules like `@scriptable`. The reason it is a string though is I can mark it in other modules without introducing a dependency on jsvar so the user doesn't pay for it if they don't use it.

  - The script language now allows identifiers-with-dashes, since I like how they look in scripts a lot. To do subtraction, you must put spaces around the - character.


* terminal.d is a terminal client library, similar to ncurses. It allows you to move the cursor around the terminal, go to full screen mode, output color, input all keys and mouse events.

In recent months, it has gained 24-bit color output support (with automatic approximation to lower color fallback), though I generally suggest you keep to 8 color output for best results in terminal programs.

It also got a `getline` function with user editing and autocompletion available, unified some of the key input events, and a ScrollbackBuffer which is clunky to use, but allows you to have individual scroll buffers scattered around the screen.

It also received a few small bug fixes for Mac OS X.


* simpleaudio, joystick, and gamehelpers haven't been changed yet, but it is my plan to almost completely redo them. I am reasonably happy with joystick.d, but simpleaudio.d is probably going to be gutted and rewritten. If you actually use it as-is, let me know and I'll work with you on compatibility, but I doubt that...


* simpledisplay.d has gotten a LOT of work lately.

  - Its ScreenPainter is now more reliable if you pass it to other functions. It does refcounting but also sets the impl values back in dtors so it acts like a hybrid value/reference type.

  - The NotificationAreaIcon now works on Windows and Linux, with balloon notifications.

  - There's a new postEvent API with collapsing and timeouts, allowing multithreaded programs to run efficiently.

  - It has support for creating OpenGL 3+ contexts.

  - There are more exposed low level events - simpledisplay aims to be simple for many things, but not get in your way if you want to do more with it than it was specifically designed for.

  - It has timer and Linux file read support in its event loop now, though postEvent can be better than Timer for game purposes, and the Linux file read is use-at-your-own-risk experimental.

  - ScreenPainter can do clipping and transformations.

  - Sprite can now be drawn upon, just like SimpleWindow. Sprite is a class that contains a display-side image for quick blitting (kinda like a 2d version of a opengl texture).

  - SimpleWindow can construct more window types like popup menus, for use building custom UIs, among other things.

  - OperatingSystemFont gives a way to load a font off the OS and draw with it in a ScreenPainter.

  - You can captureInput to a window, getting mouse tracking or even confinement.

  - There are platform-specific functions for handling global hotkeys. Still work in progress in unifying the APIs.

  - simpledisplay.d can now survive an X server disconnect, and can even migrate windows from one X server to another! Think GNU Screen for GUI apps.

  - The documentation has been expanded, though it still needs cleanup, it includes more and more examples. See: http://dpldocs.info/experimental-docs/arsd.simpledisplay.html

  - Other little features and bug fixes.


I haven't used SDL or anything similar for years thanks to simpledisplay.d. Its only dependency is color.d, leading to simple builds.


* And the huge one in the last month, minigui.d is a small GUI widget library that aims to use native widgets on Windows and a small set of custom ones on Linux (in the future, it might also support GTK widgets, though I don't plan to write that myself, there is just another user who said he might). It has no dependencies except for simpledisplay.d and color.d and the operating system - no complicated library build process, no huge dlls to distribute with your program.

http://dpldocs.info/experimental-docs/arsd.minigui.html

Before now, it was fairly usable on Windows, but only had buttons on Linux. Now, it is about equally featured on both with widgets like scroll areas, checkboxes, buttons, progress bars, menus, drop down selects, textareas, and more. It aims to cover the basic boxes that HTTP 4 forms did, while also being easy to define your own widgets - you just subclass it and draw in the area it assigns you. It now even has an OpenGL widget.

It automatically sizes and lays out your widgets and has an event system similar to Javascript. With add on modules like minigui_xml, you can create windows from XML definitions, and it is script.d compatible. When I'm ready, I'll post that too, and hopefully you can take basic web knowledge over to desktop land... without just embedding browsers and calling them desktop apps. My largest program is about 800 KB, and that's all it needs.

Since it is built on simpledisplay, it also gets X connection disconnect survival or migration!

Old screenshot: http://arsdnet.net/guitest.png
New screenshot: http://arsdnet.net/minigui.windows.new.png
New screenshot, Linux version: http://arsdnet.net/minigui.linux.d.png
an app I'm writing with it: http://arsdnet.net/minigui-sprite.png
xml app screenshot: http://arsdnet.net/minigui-scripted-app.png


Demo program source code: http://arsdnet.net/guitest.d


I'll write more about minigui.d and do a separate announcement for it when it reaches version 1.0, which will probably be in about another month.

The look on Linux is based on Windows 95, I think it is elegant and usable, and is also very efficient for remote X connections, so I can use them from my laptop on slow wifi.

minigui also has more add on widgets: https://github.com/adamdruppe/arsd/tree/master/minigui_addons

In the core is a file picker, which is very fast on Linux and native on Windows. In addons, we add a color picker and a terminal emulator widget (depends on the terminal emulator core, mentioned above in this message).


Look forward to more on this, it is close to being ready for general use, and then we'll have another chapter in the D gui ecosystem. dlangui <https://github.com/buggins/dlangui>, I hope you like some friendly competition :)

On Thu, Apr 13, 2017 at 04:04:18PM +0000, Adam D. Ruppe via Digitalmars-d-announce wrote: [...]
> * terminal.d is a terminal client library, similar to ncurses. It allows you to move the cursor around the terminal, go to full screen mode, output color, input all keys and mouse events.

I've been using terminal.d for a lot of my projects.  It's a really nice terminal library that has a much better (IMO) API than the crawling mess that is ncurses. Thanks so much for sharing it!


> In recent months, it has gained 24-bit color output support (with automatic approximation to lower color fallback), though I generally suggest you keep to 8 color output for best results in terminal programs.

Hooray!  Finally!  Maybe now I can play videos over SSH (albeit
ultra-pixelated). :-P


> It also got a `getline` function with user editing and autocompletion available, unified some of the key input events, and a ScrollbackBuffer which is clunky to use, but allows you to have individual scroll buffers scattered around the screen.

Very nice. So far I've been rolling my own for user input, but it's nice to have built-in support.

And on that note, I've been playing around with my own range-like abstractions over Terminal (variously called Display or Canvas), that lets you treat rectangular subsets of the Terminal as sub-terminals in their own right (as least as far as output is concerned).  Basically, there's an isCanvas template that acknowledges anything that has moveTo and writef as a canvas-like type (and obviously designed so that Terminal automatically qualifies), along with a series of introspection templates to identify if a canvas has a clear method, a flush method, etc.. On top of this there's a subcanvas UFCS function that lets you create arbitrary rectangular views over any canvas (with automatic moveTo coordinate translations, etc.).  So I can write code that will work either directly with a Terminal, or any other Terminal-like backends as well as arbitrary rectangular subregions.


[...]
> * simpledisplay.d has gotten a LOT of work lately.

I've been eyeing SDL (via Derelict) recently... but maybe if simpledisplay.d does what I need, I might just use it instead.  Leaner and meaner is always better, I think!


[...]
>   - simpledisplay.d can now survive an X server disconnect, and can
>   even migrate windows from one X server to another! Think GNU Screen
>   for GUI apps.

Now *this* really caught my eye.  I'm gonna hafta check it out just for this feature alone.  It has always annoyed me that although such a thing has always been *possible* for a network-oriented GUI protocol like X11, hardly any programs (if at all!) are written in a way that would work this way.  This is a really awesome feature and now I'm *really* tempted to try it out!


[...]
> I haven't used SDL or anything similar for years thanks to simpledisplay.d.  Its only dependency is color.d, leading to simple builds.

Now I'm very tempted to look into simpledisplay.d instead of SDL.


> * And the huge one in the last month, minigui.d is a small GUI widget library that aims to use native widgets on Windows and a small set of custom ones on Linux (in the future, it might also support GTK widgets, though I don't plan to write that myself, there is just another user who said he might). It has no dependencies except for simpledisplay.d and color.d and the operating system - no complicated library build process, no huge dlls to distribute with your program.

Awesome!!!  I always hated those bulky toolkit libraries that includes everything and the kitchen sink -- 90% of which you don't actually need in your application, which usually means you have a ton of dependencies that aren't actually necessary and library dependency hell when you ship your binaries.


[...]
> It automatically sizes and lays out your widgets and has an event system similar to Javascript. With add on modules like minigui_xml, you can create windows from XML definitions, and it is script.d compatible. When I'm ready, I'll post that too, and hopefully you can take basic web knowledge over to desktop land... without just embedding browsers and calling them desktop apps. My largest program is about 800 KB, and that's all it needs.

That's cool, man.  Now I gotta check this out too! :-P


> Since it is built on simpledisplay, it also gets X connection disconnect survival or migration!

This is a killer feature, IMO.


[...]
> I'll write more about minigui.d and do a separate announcement for it when it reaches version 1.0, which will probably be in about another month.

Awesome!


[...]
> Look forward to more on this, it is close to being ready for general use, and then we'll have another chapter in the D gui ecosystem. dlangui <https://github.com/buggins/dlangui>, I hope you like some friendly competition :)

This looks like a potential awesome trend, especially that bit about X server disconnection survival / migration.  I think we could write a killer app on top of this.  Say a terminal that travels with you wherever you go -- migratable from desktop to laptop on wifi and back, perhaps even to/from your {smart/dumb}phone or tablet, so don't have to bother with reconnecting your current code hacking session, but can just pick up and keep going from where you left off instantaneously.

Or a lightweight mini-browser that follows you wherever you go. Etc.

We could start a new trend based on migratable apps that can move around from device to device seamlessly. Sure, browser apps have this to some extent, but we're talking about not even needing to start your browser and relogin and all that jazz -- just resume in exactly the same state you left it on the previous device.  Or ping-pong a terminal window between you and a colleague for instantaneous collaboration.  Best of all, the programmers only need to put in minimal effort to support it.


T

-- 
VI = Visual Irritation

On Thursday, 13 April 2017 at 18:11:10 UTC, H. S. Teoh wrote:
> Hooray!  Finally!  Maybe now I can play videos over SSH (albeit
> ultra-pixelated). :-P

Hah.... though actually, my terminal emulator has support for outputting png images...

But truth is, this is one reason why I prefer things like `attach` or the new simpledisplay feature to VNC and the like - I can then combine programs from different computers on one desktop. So if I do need to play videos, I get them in the compressed format over whatever link and play them locally on the laptop.

I rarely run browsers remotely, even though sometimes I wish I could just because the local one tends to be much easier to use.


> And on that note, I've been playing around with my own range-like abstractions over Terminal (variously called Display or Canvas), that lets you treat rectangular subsets of the Terminal as sub-terminals in their own right (as least as far as output is concerned).

Nice. I actually thought about doing something like that (basically nested ScrollbackBuffers with a nicer api) too, but haven't gotten around to it yet.

> Now *this* really caught my eye.  I'm gonna hafta check it out just for this feature alone.  It has always annoyed me that although such a thing has always been *possible* for a network-oriented GUI protocol like X11, hardly any programs (if at all!) are written in a way that would work this way.


Indeed. So the way it works right now is there's an XDisconnectException that is thrown. If you want to recover, you must catch it OUTSIDE the event loop, then adjust the display string if appropriate and run XDisplayConnection.discardAndRecreate. If it succeeds, you can reenter your event loop and pick up where you left off.

Any X resources you had may need to be recreated. The library will automatically handle things it has inside (SimpleWindow right now, I'll add support to the other classes soon) but if you go outside it, you might have to do it yourself.

Server-side fonts, in particular, may need attention because they might no longer be available. But I might automate that too (keep a list of fallbacks inside the class).

Well, anyway, the big point is it isn't 100% automatic, you do need to catch the exception, possibly fix some stuff, and reconnect, but it is almost automatic; in simple apps, it is as simple as `catch(...) discardAndRecreate(); loopAgain;`

But it DOES work! :P

> Awesome!!!  I always hated those bulky toolkit libraries that includes everything and the kitchen sink -- 90% of which you don't actually need in your application, which usually means you have a ton of dependencies that aren't actually necessary and library dependency hell when you ship your binaries.

Yeah, I avoid distributing most programs because I can't stand having 20 MB + of stuff and the poor users have to manage that crap. Gah.

And I avoid *compiling* most gui programs because that's a huge pain too. But not with minigui, you've used my libs, you know how it is

dmd yourapp.d minigui simpledisplay color

Boom, it works. There's a few tweaks you can do, you might want some of the optional add on modules (which I am keeping each one strictly independent - they are not allowed to import each other to keep that decoupled), but the basic of it is right there.

And it builds in a fraction of a second. No meta-object compiler, no bindings generation, no dlls. And the generated file can be as small as 200 KB if you strip it right, though my test programs tend to hover closer to 600 on Windows, ~800 on Linux. I can handle that a lot easier than 20,000 KB.

Adding widgets is just newing the class. The automatic layout is usually ok. Custom widgets? Just subclass (actually until yesterday, it was all done via delegates, but subclassing is actually a bit easier, thanks D's anonymous classes) and implement paint and events. Again, no multiple files to generate and compile.


> I think we could write a killer app on top of this.  Say a terminal that travels with you wherever you go -- migratable from desktop to laptop on wifi and back, perhaps even to/from your

Actually, I already have that: `attach` in the terminal emulator repo (it is similar to GNU screen, just my style, including things like scrollback, window icons, mouse support, and more).

My "IDE" tends to be an attach session running everything I want in the terminal tabs and I grab them by name when moving around.

That's the big reason why I have spent more time on terminal.d than minigui.d until now: my terminal sessions follow me on the laptop and such, so I wrote a LOT of terminal programs. But now, I can do GUI too! So we'll see how it goes.

The one problem I have with a quick migration is that the gui one is still somewhat bandwidth inefficient. Just rolling your mouse over a window generates several KB of network traffic. On a real connection... no big deal, but sometimes I use my laptop on a mobile metered connection, and I pay by the kilobyte there.

The good news though is I should be able to optimize that. I'll keep the api being fat by default, but have the ability to turn off mouse motion reporting on demand, and will optimize the redraws and atom querying and others (that's a reason why the styles are so simple btw, other people transfer big pngs over, I transfer over small XDrawRectangle packets).


> just resume in exactly the same state you left it on the previous device.  Or ping-pong a terminal window between you and a colleague for instantaneous collaboration.  Best of all, the programmers only need to put in minimal effort to support it.

Indeed.

On Thursday, 13 April 2017 at 16:04:18 UTC, Adam D. Ruppe wrote:
> I haven't posted in announce for a while, so I figured I'd give you guys an update on what's been happening in my libs. This message is a big long, so feel free to just skim to see if you are interested in anything.
>
> [...]
>
> * simpledisplay.d has gotten a LOT of work lately.

Thanks particularly for this one, which is certainly a reference about how to use the nightmare-ish X11 API in D.

On Friday, 14 April 2017 at 06:23:00 UTC, Basile B. wrote:
> Thanks particularly for this one, which is certainly a reference about how to use the nightmare-ish X11 API in D.

I actually don't think X is that bad for what it does..... just that it is ridiculously underpowered compared to the competition.

If you want a real nightmare, take a look at terminal.d's innards. Eeek.

You have mixed spaces and tabs, not in the good way either. https://github.com/adamdruppe/arsd/blob/master/simpledisplay.d#L3642

On Friday, 14 April 2017 at 19:55:54 UTC, Jerry wrote:
> You have mixed spaces and tabs, not in the good way either. https://github.com/adamdruppe/arsd/blob/master/simpledisplay.d#L3642

Some code is copied from C headers, some come from open source contributors who may use different styles than I do. I accept these gifts in the spirit in which they are given.

If I edit that code, I might reformat it, but it isn't important to me.