On Tue, 2012-10-30 at 10:08 +0100, Rob T wrote:
[…]

> Also scons has no built-in ability to scan subfolders for source files. You can manually specify sub-level source files, but that's unreasonable to do for large projects. You can write custom Python code to scan subfolders, but that is a lot more complicated than it should be. Do you know of a decent solution to scan sub-folders?

Using os.walk is quite natural in SCons since SCons is just a Python internal DSL. SConstruct and SConscript are internal DSL files (not external DSL as Makefiles are), thus they are just Python programs.

> Scons does look like a very powerful tool, and I really do appreciate your input and the other posters as well. I am determined not to continue with Make, so maybe I will have to keep trying to get scons to do what I want.

Mayhap then what you need to do is look at Parts. This is Jason Kenny's superstructure over SCons to deal with huge projects scattered across everywhere. This is the build framework Intel uses for their stuff. Intel have more and bigger C and C++ projects than I think anyone else around

> One more question: I am wondering how the scons D support locates dependencies from the imports specfied in the source files? So far I have not been able to get automatic dependency inclusions to work, so I end up manually specifying the import files. With C++ and Make, I could get gcc to scan source files for the #include files, and dump the list to a dependency file (.dep), and then I could include the .dep file(s) into the make process. Can scons with D support do something similar, or deal with it better?

Now this is a whole different issue!

I have no idea. If it is a problem it needs fixing.

The general philosophy in SCons is to have scanners that find the dependencies. In C and C++ this is the awful #include. In D it's import. The code should create a graph of the dependencies that can be walked to create the list of inputs. There is a bit of code in the D tooling that is supposed to do this. If it doesn't then we need a bug report and preferably a small project exhibiting the problem that can be entered into the test suite — SCons development is obsessively TDD.

I can only do stuff for Linux and OS X, I cannot do anything for Windows, and Windows is the "big problem" for the SCons D tooling at the moment which is why it has not got merged into the SCons mainline for release in 2.3.0.



A plea for help: in order for SCons to have sane support for D, it needs people to step up and actively assist evolve the tools either by providing small test projects exhibiting fails, or running the tests on Windows and providing patches for fails.

Currently I am not working in D, just in Python, Java, Groovy, Scala, Ceylon, Kotlin, so I have no D code activity I can use to evolve the D support in SCons. But if the D community can provide test cases, I can get SCons evolved to do the needful.  But only on Linux and OS X, I definitely need a collaborator interested in working on Windows.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Tuesday, 30 October 2012 at 09:42:50 UTC, Russel Winder wrote:
> I would suggest you haven't given SCons long enough to get into the
> SCons idioms, but you have to go with the route that is most comfortable
> for you.

You are right, I'll shut up and keep at it. I get frustrated sometimes, but that's no reason to vent in here. Sorry.

> I do not see why you need to scan subfolders recursively. This is a
> traditional Make approach. SCons does things differently even for highly
> hierarchical systems. In particular use of SConscript files handles
> everything. So I do not think this is a SCons problem.

The only reason is to automate the construction of a list of source files for building. I'm used to using automated build scripts, which require only a minimal of manual input. For example, if I add a new source file to the mix, then I am expecting to not have to modify a SConstruct file to include it.

I see mention of placing a SConsript file in each subdir, but that's a fair amount of manual overhead to bother with. so there must be another way?

> Of course if you have to do things recursively then os.walk is the
> function you need.

The solutions I looked at did not mention os.walk, so I'll have a look. Thanks for the tip.

> As noted in the exchanges to which I cc you in everything I sent, SCons
> does insist on having all directories in use under the directory with
> the SConstruct – unless you have with Default. On reflection I am now
> much less worried about this that I was at first.  Out of source tree I
> find essential in any build, SCons does this well, Make less so. Out of
> project tree builds I am now not really worried about.

Yes, I see I can build out of the source tree, that's half the battle solved, but it still insists on building in the project tree, which I was hoping to also do away with. There's a disadvantage for me doing it in this way, so it's something of a step backwards for me (in terms of organizing things), which I'd rather not have to do, hence the fustration I've expressed. There must be a way to solve it somehow.

> I disagree. I find Make totally rigid and unyielding. Not to mention
> rooted in 1977 assumptions of code.

Yes I agree that Make sucks, and I hope I won't offend anyone by saying that. ;)

> You don't need to install SCons to use it, you can use it from a clone
> directly using the bootstrap system.  I have an alias
>
>         alias scons='python /home/users/russel/Repositories/Mercurial/Masters/SCons_D_Tooling/bootstrap.py'

Sounds great, but my lack of Python expertise means that I do not fully understanding how this will work for me. I'll diginto it ...

Thanks for the input.

--rt

On Tuesday, 30 October 2012 at 09:56:08 UTC, Russel Winder wrote:
> Mayhap then what you need to do is look at Parts. This is Jason Kenny's
> superstructure over SCons to deal with huge projects scattered across
> everywhere. This is the build framework Intel uses for their stuff.
> Intel have more and bigger C and C++ projects than I think anyone else
> around

For the record, my projects are not very small but also not very big (whatever that means), mid size I suppose? I'll have a look at Parts, but I have a hard case of Bleeding Edge Syndrome, and I may not survive.

> Now this is a whole different issue!
>
> I have no idea. If it is a problem it needs fixing.
>
> The general philosophy in SCons is to have scanners that find the
> dependencies. In C and C++ this is the awful #include. In D it's import.
> The code should create a graph of the dependencies that can be walked to
> create the list of inputs. There is a bit of code in the D tooling that
> is supposed to do this. If it doesn't then we need a bug report and
> preferably a small project exhibiting the problem that can be entered
> into the test suite — SCons development is obsessively TDD.

So far I've had no indication that it's working. Perhaps it is only working if the import files are not located in subfolders?

Will it account for something like this?

import a.b;

There must be something you did with the D support to give scons the ability to scan for imports. I know that #include can have sub-folders, which is common to see, such as

#include <a/b.h>

So scons must be able to include dependencies located in sub-folders.

> I can only do stuff for Linux and OS X, I cannot do anything for
> Windows, and Windows is the "big problem" for the SCons D tooling at the
> moment which is why it has not got merged into the SCons mainline for
> release in 2.3.0.

Currently I'm Linux only, but I have in the past built fairly complicated projects in Windows and I expect to be doing this again. This is another reason why I considered scons as a build solution because it is cross-platform.

> A plea for help: in order for SCons to have sane support for D, it needs
> people to step up and actively assist evolve the tools either by
> providing small test projects exhibiting fails, or running the tests on
> Windows and providing patches for fails.

Once I get things working well enough on Linux, I'll give it a try. Right now though, I'm bogged down with Linux, I need to make some breathing space first.

BTW, I use virtualbox as well as KVM extensively to run different OS's on same machine, so maybe you can try that route to get a Windows VM running for tests?

--rt

Rob T wrote:
> On Tuesday, 30 October 2012 at 09:42:50 UTC, Russel Winder wrote:
>> I would suggest you haven't given SCons long enough to get into the SCons idioms, but you have to go with the route that is most comfortable for you.
> 
> You are right, I'll shut up and keep at it. I get frustrated sometimes, but that's no reason to vent in here. Sorry.
> 
>> I do not see why you need to scan subfolders recursively. This is a traditional Make approach. SCons does things differently even for highly hierarchical systems. In particular use of SConscript files handles everything. So I do not think this is a SCons problem.
> 
> The only reason is to automate the construction of a list of source files for building. I'm used to using automated build scripts, which require only a minimal of manual input. For example, if I add a new source file to the mix, then I am expecting to not have to modify a SConstruct file to include it.
> 
> I see mention of placing a SConsript file in each subdir, but that's a fair amount of manual overhead to bother with. so there must be another way?
> 

	The following SConstruct will scan all subfolders of the current
folder for d sources and compile them into a "foo" program.

==============================8<------------------------------
import os
sources = []
for dirpath, dirnames, filenames in os.walk ("."):
    sources += [ os.path.join (dirpath, f)
                 for f in filenames
                 if f.endswith (".d") ]
Program ("foo", sources)
------------------------------>8==============================

		Jerome
-- 
mailto:jeberger@free.fr
http://jeberger.free.fr
Jabber: jeberger@jabber.fr

On Wed, Oct 31, 2012 at 02:55:29PM +0000, Russel Winder wrote:
> On Tue, 2012-10-30 at 09:53 -0700, H. S. Teoh wrote:
> […]
> > That's not a bad idea. I also noticed that gdc tends to produce smaller executables when compiling in this way (I'm not sure why -- are identical template instances not getting merged when compiling separately?).
> 
> Is it fair to assume that DMD, LDC, and GDC all have the same behaviour in this respect?
[...]

Interesting, I just tested this with dmd (git HEAD), and it exhibits the same effect too. Perhaps it's a good idea to do a single compile step then.


T

-- 
What are you when you run out of Monet? Baroque.

On Tuesday, 30 October 2012 at 09:56:08 UTC, Russel Winder wrote:
> On Tue, 2012-10-30 at 10:08 +0100, Rob T wrote:
> […]
>
>> Also scons has no built-in ability to scan subfolders for source files. You can manually specify sub-level source files, but that's unreasonable to do for large projects. You can write custom Python code to scan subfolders, but that is a lot more complicated than it should be. Do you know of a decent solution to scan sub-folders?
>
> Using os.walk is quite natural in SCons since SCons is just a Python
> internal DSL. SConstruct and SConscript are internal DSL files (not
> external DSL as Makefiles are), thus they are just Python programs.
>
>> Scons does look like a very powerful tool, and I really do appreciate your input and the other posters as well. I am determined not to continue with Make, so maybe I will have to keep trying to get scons to do what I want.
>
> Mayhap then what you need to do is look at Parts. This is Jason Kenny's
> superstructure over SCons to deal with huge projects scattered across
> everywhere. This is the build framework Intel uses for their stuff.
> Intel have more and bigger C and C++ projects than I think anyone else
> around
>
>> One more question: I am wondering how the scons D support locates dependencies from the imports specfied in the source files? So far I have not been able to get automatic dependency inclusions to work, so I end up manually specifying the import files. With C++ and Make, I could get gcc to scan source files for the #include files, and dump the list to a dependency file (.dep), and then I could include the .dep file(s) into the make process. Can scons with D support do something similar, or deal with it better?
>
> Now this is a whole different issue!
>
> I have no idea. If it is a problem it needs fixing.
>
> The general philosophy in SCons is to have scanners that find the
> dependencies. In C and C++ this is the awful #include. In D it's import.
> The code should create a graph of the dependencies that can be walked to
> create the list of inputs. There is a bit of code in the D tooling that
> is supposed to do this. If it doesn't then we need a bug report and
> preferably a small project exhibiting the problem that can be entered
> into the test suite — SCons development is obsessively TDD.
>
> I can only do stuff for Linux and OS X, I cannot do anything for
> Windows, and Windows is the "big problem" for the SCons D tooling at the
> moment which is why it has not got merged into the SCons mainline for
> release in 2.3.0.
>
>
>
> A plea for help: in order for SCons to have sane support for D, it needs
> people to step up and actively assist evolve the tools either by
> providing small test projects exhibiting fails, or running the tests on
> Windows and providing patches for fails.
>
> Currently I am not working in D, just in Python, Java, Groovy, Scala,
> Ceylon, Kotlin, so I have no D code activity I can use to evolve the D
> support in SCons. But if the D community can provide test cases, I can
> get SCons evolved to do the needful.  But only on Linux and OS X, I
> definitely need a collaborator interested in working on Windows.

I also belong to the JVM/.NET languages at work camp, but since I use both Windows and UNIX based systems, I could execute Windows tests if required.

--
Paulo