View mode: basic / threaded / horizontal-split · Log in · Help
July 13, 2011
Re: Re Build tools for D [ was Re: Prototype buildsystem "Drake"]
On 2011-07-13 19:27, Jonathan M Davis wrote:
> On 2011-07-13 08:36, Jacob Carlborg wrote:
>> On 2011-07-13 07:28, Jonathan M Davis wrote:
>>> On Wednesday 13 July 2011 06:12:58 Russel Winder wrote:
>>>> And, of course, I should have mentioned CMake and CMakeD.
>>>>
>>>> The fact that I forgot, shows my prejudice against Makefile-based
>>>> systems and for direct DAG-based systems such as Gradle, SCons and Waf.
>>>> This though should not stop CMakeD being a part of this debate.
>>>>
>>>>  From previous discussions, it seems that one of the primary reasons for
>>>> having
>>>
>>> a D build tool in many people's minds is to also handle package
>>> management of D libraries (like Haskell's cabal or rubygems for ruby).
>>> And as great as cmaked, scons, gradle, waf, and other such tools may be,
>>> they don't do that.
>>>
>>> - Jonathan M Davis
>>
>> I don't agree with that. I think a build tool should deal with single
>> files and building. A package manager should deal with packages (of
>> files). In Ruby, RubyGems is the package manager and Rake is the build
>> tool.
>
> Well, I'm not advocating anything in particular. I was just pointing out that
> a big part of the discussions on build tools has been package management of
> libraries, and any build tool solution which doesn't at least integrate with
> some sort of package management solution is likely to not be what at least
> some people are looking for.

Ok, I see. I think as well that the build tool and package manager 
should interact with each other. For example, specifying package 
dependencies in the build script. But, I think they need to be separate 
and be usable on their own.

> Personally, I don't generally use package management tools for handling
> libraries even with languages that have such tools, and I don't generally use
> much in the way of build tools either beyond simple scripts (primarily because
> I don't generally have projects large enough for it to be an issue). As it
> stands, if I were to choose a build tool for a larger project, I'd probably
> choose CmakeD, but I'm not super-familiar with all of the tools out there and
> haven't generally found much use for them.
>
> I was just trying to point out that a fair bit of the discussion for such
> tools in this list has related to package management, and Nick's solution
> doesn't address that at all AFAIK.
>
> - Jonathan m Davis

Ok, I see.

-- 
/Jacob Carlborg
July 13, 2011
Re: Re Build system requirements [ was Re: Prototype buildsystem "Drake" ]
On 2011-07-13 19:18, Russel Winder wrote:
> On Wed, 2011-07-13 at 17:33 +0200, Jacob Carlborg wrote:
>
>> target("main.d"); // builds "main.d" as an executable
>
> The SCons equivalent is:
>
> 	Program ( 'main.d' )
>
>> Or:
>>
>> target("foobar"); // builds the directory "foobar" as a library
>
> SCons (and I think Waf as well) does not support this idea (at least
> currently).  In SCons you have to say:
>
> 	SharedLibrary ( 'libraryRootName' , Glob ( 'foobar/*.d' ) )
> 	StaticLibrary ( 'libraryRootName' , Glob ( 'foobar/*.d' ) )
>
> D support in Waf is currently far more sophisticated than that of SCons,
> but is still based on being a bit more explicit that your proposal
> above.

I think that one of the problem with these language independent build 
tools is that they don't make it as easy as it could, because they 
usually don't know enough about a given language.

DSSS works just the same, you specify a D file or a directory and it 
will compile it as an executable or a library. I don't think that a new 
build tool should do any worse than, for example, DSSS.

-- 
/Jacob Carlborg
July 13, 2011
Re: Prototype buildsystem "Drake"
"Chris Molozian" <chris@cmoz.me> wrote in message 
news:mailman.1595.1310554903.14074.digitalmars-d@puremagic.com...
>I asked about build tools for D on the mailing list a while ago. I
> needed a solution that allowed me to mix C++ and D builds in a
> cross-platform way with minimum fuss. You can find the discussion about
> it here
> <http://www.digitalmars.com/d/archives/digitalmars/D/Best_build_tool_for_D_projects_136103.html>
> (you were also a part of it IIRC). My biggest requirement at the time was:
>
>  * Keeping platform checks e.g. IF (MAC) {} ELSE IF(LINUX) {} ... etc.
>    to an absolute minimum. What's the point in a cross-platform
>    language if when you build projects in it you need to write a short
>    essay for the build system...

FWIW, my Drake system takes that as a high priority, too. For example, if 
you have project "foo", then you can get the cross-platform binary filename, 
object filename, shared lib filename, static lib filename, etc like this:

"foo".exe  // "foo.exe" or "foo"
"foo".obj  // "foo.obj" or "foo.o"
"foo".lib  // "foo.lib" or "foo.a"
"foo".slib  // "foo.dll" or "foo.so"
"foo".bat  // "foo.bat" or "foo"
"foo".sh   // "foo.bat" or "foo.sh"

And new ones are easy to define:



I also intend to make sure that invoking "./blah" works on Windows.

>  * Pre-built binaries available to all platforms (or as many as possible).
>

With Drake, the only pre-built binaries that are needed are DMD. The drake 
command itself is a trivial one-line shell/batch script that invokes RDMD to 
build+run the drakefile (which imports the rest of drake).

Of course, something still needs to be done to set up the DRAKE env var to 
point to the Drake directory (or at least do something to get the path to 
Drake into the -I arg sent to RDMD...) Not entirely sure how I want to do 
that, although I think I have one idea.

> In the end I settled for Jam (ftjam)
> <http://www.freetype.org/jam/index.html>, it works tremendously well.
> It's easy to get binaries for almost any operating system (especially
> with projects like homebrew <http://mxcl.github.com/homebrew/> for Mac).
> I have very few gripes with it:
>
>  * The build description language is very simple (once you fully grok
>    it), but could have done with a lot more example-based documentation.
>  * It's missing a few features like recursive-directory scanning (for
>    source files), and the ability to pipe the output from a program
>    into a variable in the build script (e.g. C++FLAGS = `llvm-config
>    --cxxflags`).
>  * A convenient way to force all generated object and library files
>    into a dedicated build folder. It's possible but not very easy to do.
>  * I'm a big fan of convention over configuration, I know it's a very
>    subjective topic but I like the way Maven3 and Gradle assume a
>    project structure (that you can deviate from if you need to). This
>    usually requires very good supporting documentation.
>
> The build script I put together for my project looked like this,
> http://mysticpaste.com/private/TCcTE6KGxn .
>

Interesting points to keep in mind, thanks. After recently looking at Waf, I 
do agree with you that it'd be nice to, as you say, "assume a project 
structure (that you can deviate from if you need to)". Definitely some 
benefits to be gained from that.
July 13, 2011
Re: Prototype buildsystem "Drake"
"Nick Sabalausky" <a@a.a> wrote in message 
news:ivkoms$7d9$1@digitalmars.com...
>
> "foo".exe  // "foo.exe" or "foo"
> "foo".obj  // "foo.obj" or "foo.o"
> "foo".lib  // "foo.lib" or "foo.a"
> "foo".slib  // "foo.dll" or "foo.so"
> "foo".bat  // "foo.bat" or "foo"
> "foo".sh   // "foo.bat" or "foo.sh"
>
> And new ones are easy to define:
>

[Forgot to finish that thought:]

alias crossPlatformSuffix!(".obj", ".o") obj;
July 13, 2011
Re: Prototype buildsystem "Drake"
On 2011-07-13 20:31, Nick Sabalausky wrote:
> "Chris Molozian"<chris@cmoz.me>  wrote in message
> news:mailman.1595.1310554903.14074.digitalmars-d@puremagic.com...
>> I asked about build tools for D on the mailing list a while ago. I
>> needed a solution that allowed me to mix C++ and D builds in a
>> cross-platform way with minimum fuss. You can find the discussion about
>> it here
>> <http://www.digitalmars.com/d/archives/digitalmars/D/Best_build_tool_for_D_projects_136103.html>
>> (you were also a part of it IIRC). My biggest requirement at the time was:
>>
>>   * Keeping platform checks e.g. IF (MAC) {} ELSE IF(LINUX) {} ... etc.
>>     to an absolute minimum. What's the point in a cross-platform
>>     language if when you build projects in it you need to write a short
>>     essay for the build system...
>
> FWIW, my Drake system takes that as a high priority, too. For example, if
> you have project "foo", then you can get the cross-platform binary filename,
> object filename, shared lib filename, static lib filename, etc like this:
>
> "foo".exe  // "foo.exe" or "foo"
> "foo".obj  // "foo.obj" or "foo.o"
> "foo".lib  // "foo.lib" or "foo.a"
> "foo".slib  // "foo.dll" or "foo.so"
> "foo".bat  // "foo.bat" or "foo"
> "foo".sh   // "foo.bat" or "foo.sh"

Don't forget "foo.dylib" on Mac OS X.

-- 
/Jacob Carlborg
July 13, 2011
Re: Prototype buildsystem "Drake"
Den 13-07-2011 03:02, Nick Sabalausky skrev:
> The recent discussions about package managers and buildsystems has prompted
> me to get off my ass (or rather, *on* my ass as per how programming is
> usually performed...) and start on the D-based rake-inspired buildtool I've
> been meaning to make for awhile now. It's not actually usable yet, but
> there's a sample drakefile demonstrating everything, and it does actually
> compile and run (but just doesn't build the dependency tree or actually run
> any of the build steps). *Should* work on Posix, but I only tested on
> Windows, so I may have fucked up the Posix version...
>
> Apologies to Jacob Carlborg for the name being so close to "dake". Didn't
> really know what else to call it ("Duck", maybe?) Like dake, it's inspired
> by Ruby's Rake. But unlike dake, the buildscript is written in D instead of
> Ruby, which was my #1 reason for making a Rake alternative.
>
> Before I go implemeting everything, I'd like to get input on it. Is it
> something that could be a major D tool?
>
> Overview of Drake and links to all the code are here:
>
> https://bitbucket.org/Abscissa256/drake/wiki/Home

A good start I think. The first thing that I think should be fixed is 
the naming.

Drake should be called dbuild
Orb should be called dpack

You read the name and you have an idea of what it does - nothing fancy.

People coming from ruby may recognize the drake name, but I guess many 
c++ peoply have no idea.
Orb is not derived from anything I guess.

Regarding drake itself:

I really like how general it is but at the same time miss some 
simplicity. "Make the common task easy and the uncommon possible"

Some steps that you could take in that direction is to predefine 
commonly used modes like "build", "clean" etc. And the define 
buildMode(), cleanMode() as builtins. That would get rid of the 
"immutable modes = ..." lines in simple projects.

Maybe let the wrapper that loads the drakefile insert the "import 
drake.all" and also wrap the "void drakefile() { ... }" around the file. 
Maybe that is what your have already listed on the todo?

As suggested elsewhere you could get rid of target!Task or target!File 
and just defined the target() and file() functions.

Also a convenience could be to just include all .d files found for an 
exe target if no .d files are explicitly listed.

I guess this would make it possible to have a drakefile as simple as:

task("appA".exe);


It seems to lack the configure part of the build process e.g. locate the 
libfoobar.a file for linking or locate a D compiler.

Thanks,
/Jonas
July 13, 2011
Re: Prototype buildsystem "Drake"
"Jacob Carlborg" <doob@me.com> wrote in message
news:ivke5k$2m78$1@digitalmars.com...
>
> First I have to say that I know you are doing this because you want to use 
> D as the language for the build scripts. The reason I did choose Ruby 
> because I think D will be too verbose and when I'm looking at drakefile.d 
> I do think it's too verbose.

Yea, D is likely to be a little more verbose than what could be done in Ruby 
(or Python). Personally, I think that's well worth it, though. I don't know 
how many others would agree or not.


> But instead of starting a debate between Ruby and D I'm going to give you 
> a few suggestions and comments instead.
>
> The minimal drakefile doesn't look that minimal.

I think I may be able to make the "immutable modes = [];" optional by 
checking for it with some compile-time reflecion.

The other parts I'll address further below...


> This is the idea I have for a minimal build script file:
>
> target("main.d"); // builds "main.d" as an executable
>
> Or:
>
> target("foobar"); // builds the directory "foobar" as a library
>
> DSSS did this very good.
>

I've never been very comfortable with build systems magically understanding 
and inferring all those details about a language and filetypes. It just 
feels too "magic", it's much more difficult to expand to new 
languages/tools, and every time I use a system like that I feel completely 
lost every time I need to include even just one custom buildstep.

However, Drake's design is expandable. Add-on modules can be created that 
define new target types, so you could do:

   module drake_dlang;
   class DLang : Target (or File)
   {
       //...etc
   }

   target!DLang("main.d");

And that would essentially wrap a File target, adding in any D-specific 
things such as "The target is 'main.exe'". And if you really wanted you 
could take it one more step and do:

   void targetD(string source)
   {
       target!DLang(source);
   }

   targetD("main.d");


Good add-ons could even be folded into the standard Drake.


> Now looking in the drakefile.d. You have targets and tasks all over the 
> place, almost every target has "Task" as a template parameter. Why is this 
> necessary? Have one function for targets and one for tasks. Example from 
> the drakefile.d:
>
>         target!Task("upload", "all",
>                 (Target t)
>                 {
>                         system("ftp ...");
>                 }
>         );
>
> Seems this could look like this:
>
> task("upload", "all", {
>     system("ftp...");
> }); // if D just could get a better looking delegate syntax
>

Some Drake terminology first: Every "node" in the dependency tree is a 
"Target". There are three pre-defined types of targets: "Task", "File" and 
"Dir". Other new target types can also be defined. So the template param is 
due to that.

However, I suppose it may be a good idea for every target type to come with 
a matching shortcut:

   alias target!Task task;
   alias target!File file;
   alias target!Dir  dir;

Then your "task("upload", ...);" example should work.

I agree that D's delegate syntax could be better. I think something like 
your example would be possible, but I ran into what seemed like some type 
inference limitations with all the overloading and templates, etc.


> "getDCompiler" seems unnecessary, both ldc and gdc have a dmd compatible 
> wrapper, ldmd and gdmd. These wrappers takes the same flags as dmd and 
> translate them to the correct "native" flags.
>

That's great, I didn't know that. But, you still have to say "ldmd" or 
"gdmd" instead of "dmd". And since the idea of "getDCompiler" is to allow 
the user to select which compiler to use (if the drakefile's author wants to 
support that), something roughly like "getDCompiler" may still be needed, 
albeit in a greatly simplified form.

In any case, it was just an example of using the "modes" feature.


> The "drakfile" function seems unnecessary. I would use a string import (or 
> what it's called), importing the whole build script into a method in a 
> class. All code in the "drakefile" function would instead be at top level.
>

I'm not entirely opposed to that idea, but here are the (perhaps minor?) 
reasons I did it this way:

- Using "import" inside functions is brand-new and likely still buggy. And 
unless I'm mistaken, I think there's other problems that still exist with 
using the same code inside a function as outside (such as some 
order-of-declaration limitations, IIRC). I wanted to get moving on something 
that would work reliably right away without being too sensitive to bleeding 
edge-cases. I figured if all that gets sorted out later on, then maybe a 
Drake v2 could go that route.

- I wasn't sure if a D file that isn't strictly a proper D file would be too 
weird, too confusing, too magical, or would confuse the fuck out of advanced 
IDE's.

If those are really just totally bullshit reasons, then I'm certainly open 
to the idea of doing it as you suggest.


> Most of the functions called in the build script should be instance 
> methods, this will allow to use threading, possible invoke multiple 
> targets/tasks simultaneously and running several build scripts 
> simultaneously.
>

My understanding is that with thread-local being the default, they *don't* 
need to be instance methods to be thread-safe.

Also, dependency-checking and task-invocation (once they're implemented) 
won't occur at all until *after* the drakefile() function completes. The 
target!...(...) function merely defines the targets, adding them to an 
internal list - nothing more. So I'm not sure that's not really a 
thread-appropriate matter.

Not really sure what you mean about running several build scripts 
simultaneously. The buildscript gets compiled into an exe, so the 
buildscripts all run in separate processes anyway.


> I see installation related code in the build script. I think a package 
> manager should be responsible for that. A build tool should deal with 
> single files and a package manager should deal with packages (of files).
>

Those are just there as examples. You can do it however you want.


-------------------------------
Not sent from an iPhone.
July 13, 2011
Re: Prototype buildsystem "Drake"
"jdrewsen" <jdrewsen@nospam.com> wrote in message 
news:ivl1gp$o1v$1@digitalmars.com...
>
> A good start I think. The first thing that I think should be fixed is the 
> naming.
>
> Drake should be called dbuild
> Orb should be called dpack
>
> You read the name and you have an idea of what it does - nothing fancy.
>

I think that's an excellent point. Only thing: Is "dbuild" already taken? 
Also I'd hate to claim that name if nobody ended up liking it ;)


> People coming from ruby may recognize the drake name, but I guess many c++ 
> peoply have no idea.
> Orb is not derived from anything I guess.
>
> Regarding drake itself:
>
> I really like how general it is but at the same time miss some simplicity. 
> "Make the common task easy and the uncommon possible"
>

Yea, that's a fair point. But I figure I should start with general and then 
build the simple/structured on top of that, rather than trying to go the 
other way around.


> Some steps that you could take in that direction is to predefine commonly 
> used modes like "build", "clean" etc. And the define buildMode(), 
> cleanMode() as builtins. That would get rid of the "immutable modes = ..." 
> lines in simple projects.
>
> Maybe let the wrapper that loads the drakefile insert the "import 
> drake.all" and also wrap the "void drakefile() { ... }" around the file. 
> Maybe that is what your have already listed on the todo?
>
> As suggested elsewhere you could get rid of target!Task or target!File and 
> just defined the target() and file() functions.
>
> Also a convenience could be to just include all .d files found for an exe 
> target if no .d files are explicitly listed.
>
> I guess this would make it possible to have a drakefile as simple as:
>
> task("appA".exe);
>

Thanks, definitely things to think about.

>
> It seems to lack the configure part of the build process e.g. locate the 
> libfoobar.a file for linking or locate a D compiler.
>

Hmm, yea, configure didn't even occur to me. I think it could acually be 
done as just another File target that everything else depends on. Only thing 
is then you'd have to save/load the configuration file manually, so maybe 
there should be some convenience mechnism for that (Maybe utilizing Jacob's 
Orange serialization library once that rewrite is done? Or maybe that would 
be overkill? Is there some INI-file util, or am I just imagining that?)
July 13, 2011
Re: Prototype buildsystem "Drake"
"Nick Sabalausky" <a@a.a> wrote in message 
news:ivl4f8$t29$1@digitalmars.com...
> "jdrewsen" <jdrewsen@nospam.com> wrote in message 
> news:ivl1gp$o1v$1@digitalmars.com...
>>
>> A good start I think. The first thing that I think should be fixed is the 
>> naming.
>>
>> Drake should be called dbuild
>> Orb should be called dpack
>>
>> You read the name and you have an idea of what it does - nothing fancy.
>>
>
> I think that's an excellent point. Only thing: Is "dbuild" already taken? 
> Also I'd hate to claim that name if nobody ended up liking it ;)
>

I mean, if nobody ended up liking the tool. A tool with an official-sounding 
name like "dbuild" should be something with a strong following.
July 13, 2011
Re: Prototype buildsystem "Drake"
I'm glad the feedback helped, this may be of use to help in abstracting 
away platform specific configuration: 
http://public.perforce.com/public/jam/src/Jambase

Cheers,

Chris


On 07/13/11 19:31, Nick Sabalausky wrote:
> "Chris Molozian"<chris@cmoz.me>  wrote in message
> news:mailman.1595.1310554903.14074.digitalmars-d@puremagic.com...
>> I asked about build tools for D on the mailing list a while ago. I
>> needed a solution that allowed me to mix C++ and D builds in a
>> cross-platform way with minimum fuss. You can find the discussion about
>> it here
>> <http://www.digitalmars.com/d/archives/digitalmars/D/Best_build_tool_for_D_projects_136103.html>
>> (you were also a part of it IIRC). My biggest requirement at the time was:
>>
>>   * Keeping platform checks e.g. IF (MAC) {} ELSE IF(LINUX) {} ... etc.
>>     to an absolute minimum. What's the point in a cross-platform
>>     language if when you build projects in it you need to write a short
>>     essay for the build system...
> FWIW, my Drake system takes that as a high priority, too. For example, if
> you have project "foo", then you can get the cross-platform binary filename,
> object filename, shared lib filename, static lib filename, etc like this:
>
> "foo".exe  // "foo.exe" or "foo"
> "foo".obj  // "foo.obj" or "foo.o"
> "foo".lib  // "foo.lib" or "foo.a"
> "foo".slib  // "foo.dll" or "foo.so"
> "foo".bat  // "foo.bat" or "foo"
> "foo".sh   // "foo.bat" or "foo.sh"
>
> And new ones are easy to define:
>
>
>
> I also intend to make sure that invoking "./blah" works on Windows.
>
>>   * Pre-built binaries available to all platforms (or as many as possible).
>>
> With Drake, the only pre-built binaries that are needed are DMD. The drake
> command itself is a trivial one-line shell/batch script that invokes RDMD to
> build+run the drakefile (which imports the rest of drake).
>
> Of course, something still needs to be done to set up the DRAKE env var to
> point to the Drake directory (or at least do something to get the path to
> Drake into the -I arg sent to RDMD...) Not entirely sure how I want to do
> that, although I think I have one idea.
>
>> In the end I settled for Jam (ftjam)
>> <http://www.freetype.org/jam/index.html>, it works tremendously well.
>> It's easy to get binaries for almost any operating system (especially
>> with projects like homebrew<http://mxcl.github.com/homebrew/>  for Mac).
>> I have very few gripes with it:
>>
>>   * The build description language is very simple (once you fully grok
>>     it), but could have done with a lot more example-based documentation.
>>   * It's missing a few features like recursive-directory scanning (for
>>     source files), and the ability to pipe the output from a program
>>     into a variable in the build script (e.g. C++FLAGS = `llvm-config
>>     --cxxflags`).
>>   * A convenient way to force all generated object and library files
>>     into a dedicated build folder. It's possible but not very easy to do.
>>   * I'm a big fan of convention over configuration, I know it's a very
>>     subjective topic but I like the way Maven3 and Gradle assume a
>>     project structure (that you can deviate from if you need to). This
>>     usually requires very good supporting documentation.
>>
>> The build script I put together for my project looked like this,
>> http://mysticpaste.com/private/TCcTE6KGxn .
>>
> Interesting points to keep in mind, thanks. After recently looking at Waf, I
> do agree with you that it'd be nice to, as you say, "assume a project
> structure (that you can deviate from if you need to)". Definitely some
> benefits to be gained from that.
>
>
>
1 2 3 4 5 6 7
Top | Discussion index | About this forum | D home