Thread overview
Redub v1.17.1: Walking on its own direction
Nov 20
Hipreme
Nov 20
Hipreme
Nov 20
jmh530
Nov 28
Hipreme
November 20

After redub being in the shadow of dub on its older versions, redub has now took a step forward, going in a new direction bringing to the table new features that were not inside dub.

For that to happen, redub has now severed its ties with dub by removing it completely from its code. For accomplishing that, redub did:

  • Implemented dub registry package fetching code and simulated dub behavior
  • Implemented SDL -> JSON parser
  • Implemented --single file parser

Major Additions

Redub update

Redub is now able to automatically update itself, build itself and replace itself by calling redub update. If you use it from a redub repo, it will git pull, if you do it from a .exe file, it will download the latest release tag and build it. Your current redub.exe will be replaced at the end, guaranteeing instantaneous updates.

Redub Plugins

You won't need to use rdmd anymore as redub seeks for more performance and flexibility. With redub plugins, you won't be able to keep writing platform specific code as you can just use the old D code you have and directly interface with redub painlessly.

Parallel Fetching:

Redub will also fetch your dependencies in parallel making it even faster

Parallel pkg-config evaluation:

I was able to reduce from 180ms to 30ms one of the projects here in dub, so, it is really way better

-- Also, I'm testing right now my new tool for automatically generating changelog, be sure to check its capabilities right in the footer of this post

Changelog for redub

Added

  • redub update command - Automatically updates redub to the latest version on your system
  • Fetch: Redub is now able to fetch without dub.
  • Fetch: Redub now fetches in parallel for newer packages, bringing a speed gain up to 80% compared to the single thread one.
  • Fully added --single support for redub, with advanced caching support
  • SDL->JSON parser to Redub, removing one more dependency from dub
  • Support for wasm/webassembly OS and now the json cache is cleared after parsing a project, hide cache write log if it is too small
  • 'darwin' platform filter, fixed run execution by escaping its command
  • Filter out hidden files from adv cache and fixed cache invalidation for linking step
  • Caching solution with print-ups with copy enough and up-to-date builds
  • Use hardlink instead of copy making it much faster, also added cache for linker
  • Initial binary caching
  • pkg-config on posix
  • Pkg-config execution is fully parallelized, added more timings on vverbose mode
  • defaultCompiler, compilers cache for faster inference, and fixed #14

Redub Plugins

Redub has now new additions, starting from v1.14.0. Those are called plugins.
For using it, on the JSON, you must specify first the plugins that are usable. For doing that, you need to add:

"plugins": {
  "getmodules": "C:\\Users\\Hipreme\\redub\\plugins\\getmodules"
}

That line will both build that project and load it inside the registered plugins (That means the same name can't be specified twice)

The path may be either a .d module or a dub project

>

WARNING: This may be a subject of change and may also only support redub projects in the future, since that may only complicate code with a really low gain

Redub will start distributing some build plugins in the future. Currently, getmodules plugins is inside this repo as an example only but may be better used.
Only preBuild is currently supported since I haven't found situations yet to other cases.
For it to be considered a redub plugin to be built, that is the minimal code:

module getmodules;
import redub.plugin.api;

class GetModulePlugin : RedubPlugin {}
mixin PluginEntrypoint!(GetModulePlugin);

For using it on prebuild, you simply specify the module and its arguments:

"preBuildPlugins": {
  "getmodules": ["source/imports.txt"]
}

Useful links regarding plugins:

Update

  • Redub now includes version of the compiler used to be built, throws an exception if an inexistent configuration is sent, defaults plugin compiler to be the same as the used one to build
  • Fixed #30 and #29. Also fixed issue when specifying a custom compiler with arch not triggering LDC anymore. Fixed issue where specifying global compiler would fai
  • Plugins won't include environment variables in the process of building itself
  • Do not output deps by default, this is causing compile time slowdown rather than good
  • Less memory allocation inside adv_diff and inference for simplified hashing when file bigger than 2MB
  • Added simplified hashing for output files bringing a much afster cache writing
  • Faster copy cache formula using a simplified process of checking
  • Fixed dynamic lib name for linux and osx
  • Fix linux build
  • Adv diff will now use content hash for cache formulas instead of modification time, this will fix a problem where it wasn't able to get name changes on file
  • Remove start group from macOS
  • Now redub will always specify to handle circular linker dependencies
  • Fixed some cases where a dependency could be double registered when using subpackages
  • Fixed postBuildCommands to be used after .exe gen
  • Fixed on can't build for windows when having different drive disks
  • Improved logging to print less things, also fixed an issue where sometimes a sub package would get a wrong name
  • Fixed problem getting wrong cache when cross compiling, fixed the problem where the run arguments were not being sent correctly
  • Added getOutputName for redub api, added verbose build type
  • Filter -preview= from the dflags
  • Added time-trace and mixin-check to the build
  • Improvements on redub library api
  • Improved build to save cache even when it fails
  • Improved source imports
  • API should now get ISA for building
  • Minor performance improvement
  • Improved a little more the compiler finding so it doesn't try to search always on global path
  • Reduced memory allocations inside adv_diff and improved hexstring parsing algorithm
  • Improved compilation API so now it doesn't require a full cache status
  • Improve performance of diffstatus
  • Improvement for showing which file is dirty
  • Improve info on which files weren't found on adv_diff and initial deps parsing
  • Added targetPath and targetName from CLI, while keeping the cache calculation smarter
  • Improved redub clean to try cleaning all the files that were generated in the process
  • Added redub test #10 support, and now the cache invalidation only occurs before building since it gives the opportunity to change the tree
  • Now the cache is being able to be calculated in parallel
  • Auto clear compiler cache when changing redub version
  • Smarter log in every build type
  • Made the caching calculation be done in a separate thread for static libraries, making the program fully efficient
  • Better support on preGenerateCommands + handling --single + more environment being parsed

Fixed

  • Better error information when incompatible semver is found
  • targetName is now being correctly used
  • Support for linux
  • Remove comments from SDL before converting to JSON, handle buildTypes correctly from SDL converter, clean test files
  • Pending merge requirements weren't being processed with environment
  • #27, #28, expected artifacts now will only be evaluated at requirement time, removed it from build requirements, copy attributes on dir copy, fixed target name
  • Dynamic library names
  • Crash on empty adv cache formula and redub plugins are now global
  • Correctly infer staticLibrary for non root packages
  • A problem where some files would not be copied
  • Clean will now also clear the cache folder so it will always cause a rebuild (solves the trying to run inexistent file problem), and now also outputs libraries when they are root
  • Now the copy cache will also look for object files
  • Bug wehre some times the creation of cache files would not happen and still would cache the file
  • Bug on LDC where it would not allow to build if the directory for object dir didn't exist
  • Verbose output for posix
  • Regression bug on compiling for posix
  • Checking compiler for null return
  • satisfies now correctly uses matchAll
  • Even more speed on the up to date status
  • old build warnings
  • Linking on LDC, LDC not polluting anymore user folder, simplified compilation API, added clean on force builds
  • Cache for the new entries
  • Removed loggings
  • Windows is now using correct d_dependencies
  • Not working on windows d_dependencies
  • d_dependencies received in some cases more information that wasn't being handled
  • Bug where sometimes it would incorrectly identify as up to date
  • subPackages would sometimes not get the correct version
  • Now supporting glob matching copy on copyFilse
  • index out of range
  • Now none projects should not build and only run its commands. Added invalid targetType and fixed preGenerateCommands when inside configuration
  • Now the link files always are sent to their parents
  • Now differing dependency version from different depth level are updated
  • Now correctly adding environment variables per project
  • Redub now handles up to date builds much better
  • copyDir now make create a directory
  • Bug when pkg-config does not exists
  • Now sourceLibrary with dependencies can be handled better
  • Issue #19 lflags were not being merged
  • Wrong bug which made not be able to build single projects
  • More flexibility when assigning a defaultCompiler
  • Faster dependency resolution by logging less information
  • Build issue on linux
  • Made the output shorter when uusing up to date builds [ignore time took]
  • #16 compilation error for old compiler
Changelog Metadata
Generated with changelog-d
Contribute at https://github.com/MrcSnm/changelog-d
November 20

On Wednesday, 20 November 2024 at 02:18:07 UTC, Hipreme wrote:

>

After redub being in the shadow of dub on its older versions, redub has now took a step forward, going in a new direction bringing to the table new features that were not inside dub.

Unfortunately all links are broken. And this is the actual redub project link: https://github.com/MrcSnm/redub

Never heard of it before, looks interesting

November 20

On Wednesday, 20 November 2024 at 08:29:22 UTC, Anton Pastukhov wrote:

>

On Wednesday, 20 November 2024 at 02:18:07 UTC, Hipreme wrote:

>

After redub being in the shadow of dub on its older versions, redub has now took a step forward, going in a new direction bringing to the table new features that were not inside dub.

Unfortunately all links are broken. And this is the actual redub project link: https://github.com/MrcSnm/redub

Never heard of it before, looks interesting

Shoot, sorry, I'll post all the link again.

Redub Package: You can run with dub run redub. Though I would recommend building it and adding to PATH since it will run faster :)

Redub Project: Open issues or pull requests

GetModules plugin: Add to your build process a step of generating a file containing all the modules

Plugin Example Usage: How to use that as a plugin instead of an external script

changelog-d package: Package used to generate this changeling

November 20

On Wednesday, 20 November 2024 at 02:18:07 UTC, Hipreme wrote:

>

[snip]

I haven't tried this yet, but thanks for all the work on this.

November 25

On Wednesday, 20 November 2024 at 02:18:07 UTC, Hipreme wrote:

>

After redub being in the shadow of dub on its older versions, redub has now took a step forward, going in a new direction bringing to the table new features that were not inside dub.
...

Nice. In terms of other plugins, I'd be interested in having code signing and convenient packaging so that simply running the build tool can produce an RPM for Fedora. Calling through to external tools is fine.

November 28

On Monday, 25 November 2024 at 19:21:36 UTC, singingbush wrote:

>

On Wednesday, 20 November 2024 at 02:18:07 UTC, Hipreme wrote:

>

After redub being in the shadow of dub on its older versions, redub has now took a step forward, going in a new direction bringing to the table new features that were not inside dub.
...

Nice. In terms of other plugins, I'd be interested in having code signing and convenient packaging so that simply running the build tool can produce an RPM for Fedora. Calling through to external tools is fine.

That is quite simple with the new plugin situation on redub. The good thing is that it is easily extensible for any user case one may have. One of the reasons I've done plugins is that the way to achieve that in dub, is that you would only be able to spawn processes which is slower and platform dependent. While with plugins, you can write D code, modify some of your build recipe inside the plugin AND still get cache working.

Continuing the announce:

Redub v1.19.0 has also improved how the cache is stored. From my engine, I was able to reduce it from 550KB to 78KB. That difference can make a lot of difference in people using HD instead of SSD and also improve the JSON parsing speed.

Beyond that, I've also reduced the syscalls when comparing hashes, thus making it even faster (I got almost a 30% speedup when doing that)

Some parts of the code were made @nogc which also reduced a bunch of allocations.

In the end, both up to date builds AND non up to date builds got way faster on the caching side, so, even slower processors can now handle the work just fine.