Hello all,

Given the recent chatter and movement on CMake D support, I've decided to go public with a project of mine earlier than I had intended.

Before I go further, a request: do *not* post this to HN/reddit/etc just yet. It is still in early stages and an upstream CMake 3.0 release (without D support) is incoming. Announcing this far and wide will only yield confusion at this time.

So, what is this?

It's CMake, with various modifications to work toward making D a first-class citizen of the CMake world.

While other projects exist that attempt to add D support, they all do so without touching CMake's C++ sources. This means that they will inevitably fall short of the mark.

Additionally, when I first started toying with this several months ago, there were a lot of implementation/design issues in the existing projects, that went against the way CMake's internals expect things to be done. I'm not sure how the current scene is in that regard.

Status

* GDC is fully supported, as is DMD master
* LDC and older DMD's will work for simpler projects, but won't handle linking external libraries at the moment. I had a hack workaround for this before, but recently removed it when restructuring things a bit.
* 32-bit DMD on Windows can't really be used for a C/C++/D mixed project right now because of problems I'm having with Optlink
* VisualD generation works (tested on VS 2010 and VS 2012)
* Makefile generation (and similar generators) work
* Works on Windows and Linux. OS X ought to work, but is untested, as I don't have my OS X dev environment set up at the moment
* D is listed on the CMake Qt GUI, which is nice

Github: https://github.com/trentforkert/cmake
Wiki: https://github.com/trentforkert/cmake/wiki
Binaries: https://drive.google.com/folderview?id=0B5vzzNch4TtET09HM0NLWURKV1U&usp=drive_web#list


As is tradition here, destroy!

 - Trent

On Tue, Mar 25, 2014 at 17:50:30 +0000, Trent Forkert wrote:
> It's CMake, with various modifications to work toward making D a first-class citizen of the CMake world.
> 
> While other projects exist that attempt to add D support, they all do so without touching CMake's C++ sources. This means that they will inevitably fall short of the mark.

For full-featured support, I agree.

> Additionally, when I first started toying with this several months ago, there were a lot of implementation/design issues in the existing projects, that went against the way CMake's internals expect things to be done. I'm not sure how the current scene is in that regard.

The compiler detection and such hasn't changed in CMake much (AFAICT), so if they weren't working as expected before, nothing has changed.

> * GDC is fully supported, as is DMD master
> * LDC and older DMD's will work for simpler projects, but won't
> handle linking external libraries at the moment. I had a hack
> workaround for this before, but recently removed it when
> restructuring things a bit.
> * 32-bit DMD on Windows can't really be used for a C/C++/D mixed
> project right now because of problems I'm having with Optlink
> * VisualD generation works (tested on VS 2010 and VS 2012)
> * Makefile generation (and similar generators) work
> * Works on Windows and Linux. OS X ought to work, but is untested, as
> I don't have my OS X dev environment set up at the moment
> * D is listed on the CMake Qt GUI, which is nice

Much more comprehensive than my attempts :) . The depfile support would be nice to add in so that dependencies are calculated properly. Make will be left out with DMD/LDC, but GDC will work with Ninja and Make already (LDC/DMD+Ninja has a pull request with Ninja).

I'll try it out when I get a chance.

--Ben

On Tue, Mar 25, 2014 at 14:12:45 -0400, Ben Boeckel wrote:
> I'll try it out when I get a chance.

I've pushed some fixes to my CMake fork to work with GDC on Linux. I've also fixed up some things I saw and opened a bug on your fork (which isn't linked with Kitware/CMake, so PRs are unavailable…an unfortunate GitHub limitation; I've contacted support about this). I now have it working for a "dummy" project.

I've also commented on the branch as a whole on that issue as well.

--Ben

On Wednesday, 26 March 2014 at 06:17:57 UTC, Ben Boeckel wrote:
> On Tue, Mar 25, 2014 at 14:12:45 -0400, Ben Boeckel wrote:
>> I'll try it out when I get a chance.
>
> I've pushed some fixes to my CMake fork to work with GDC on Linux. I've
> also fixed up some things I saw and opened a bug on your fork (which
> isn't linked with Kitware/CMake, so PRs are unavailable…an unfortunate
> GitHub limitation; I've contacted support about this). I now have it
> working for a "dummy" project.
>
> I've also commented on the branch as a whole on that issue as well.
>
> --Ben

As I told Ben on Github, I didn't realize CMake had a github mirror. So, I waved my magic wand and made trentforkert/cmake a github fork of Kitware/CMake so that PRs should work now.

The original (non-github-fork) repo is now preserved at trentforkert/cmake_old, just in case we need it for something.

 - Trent

I have a quandary you might be able to help with, Ben.

I've been working on moving dependency resolution things to cmDependsD.cxx, and I've got it mostly working.

However, if we have gdc produce make-style dependencies (which will still require processing to get CMake to be happy), that requires me to put a special case in the C++ that I'd rather not have.

The way I see it, there are three possibilities here:
1. Keep gdc producing make-style deps, and deal with that as needed.
2. Have gdc produce dmd-style deps, and let the patched Ninja you mentioned use that.
3. Have a separate flag registered for the two different styles of dependencies.

I'm quite partial to number 3, as I could then use that to check which flags are defined for which style is supported, and act accordingly. I prefer that to hardcoding "gdc does this. Anything else does this other thing."

It should be noted that (AFAICT), the Ninja generator is the only thing that even considers ${CMAKE_DEPFILE_FLAGS_<lang>}. All the Makefile generators (basically everything but Ninja, VS, and possibly XCode) use the cmDepends system.

Thoughts?

 - Trent

On Wed, Mar 26, 2014 at 22:52:43 +0000, Trent Forkert wrote:
> However, if we have gdc produce make-style dependencies (which will still require processing to get CMake to be happy), that requires me to put a special case in the C++ that I'd rather not have.

I'm fine with either, but make-style is preferred since it can exclude system files (which I've asked LDC about supporting as well; should ask dmd too).

> The way I see it, there are three possibilities here:
> 1. Keep gdc producing make-style deps, and deal with that as needed.
> 2. Have gdc produce dmd-style deps, and let the patched Ninja you
> mentioned use that.
> 3. Have a separate flag registered for the two different styles of
> dependencies.
> 
> I'm quite partial to number 3, as I could then use that to check which flags are defined for which style is supported, and act accordingly. I prefer that to hardcoding "gdc does this. Anything else does this other thing."

What about:

  4. Add depfile support to Makefile generators.

It seems that it shouldn't be *too* hard[1] to do. Obviously, dmd-style deps will still have problems with make, but we could try and ask the dmd and ldc upstream to at least support Make-style dependency files (this would mean old dmd/ldc + make isn't supported, but at least Ninja would be an option there). If that fails, some CMake code to generate Make-style .d files from dmd-style shouldn't be too bad (some regex matching and character escaping should do the trick).

> It should be noted that (AFAICT), the Ninja generator is the only thing that even considers ${CMAKE_DEPFILE_FLAGS_<lang>}. All the Makefile generators (basically everything but Ninja, VS, and possibly XCode) use the cmDepends system.

Yeah, seems so. If we could get rid of the cmDepends* stuff, that'd be nice. I only see it included from Source/cmLocalUnixMakefileGenerator3.* which means if the Makefile learns about depfiles, we can start removing cmDepends* (yay!). I guess the IDE generators rely on IDE magic to get dependencies correct?

So...it looks like this is what we're aiming for:

  - DMD/LDC
    - Support make-style depfiles (optional; unlikely)
    - Support filtering out excess dependencies (private import, system
      files) (preferred)
  - Ninja
    - dmd depfile support (preferred; no comment from martine yet)
  - CMake
    - Add depfile support to Make (preferred)
    - Add dmd -> make depfile translator  (likely necessary[2])

--Ben

[1]http://stackoverflow.com/a/16969086
[2]The dmd-depfile format probably won't fly with make upstream since
make-depfiles aren't actually a thing with make, but gcc.

On Wednesday, 26 March 2014 at 23:17:31 UTC, Ben Boeckel wrote:
>
> What about:
>
>   4. Add depfile support to Makefile generators.

That's basically what I'm doing, though only in the context of D.
cmDependsD::WriteDependencies() gets called for every D object,
and has the D compiler produce a dmd-style depfile of the relevant sources.
It then scans each of the

    foo (.../foo.d) : public : object (.../object.di)

lines, pulling the filepaths from the () to construct the file
depends.make in the correct CMakeFiles/<target>.dir, which contains a list
of lines like:

    .../foo.d.o: .../foo.d
    .../foo.d.o: .../object.di

> Yeah, seems so. If we could get rid of the cmDepends* stuff, that'd be
> nice. I only see it included from Source/cmLocalUnixMakefileGenerator3.*
> which means if the Makefile learns about depfiles, we can start removing
> cmDepends* (yay!). I guess the IDE generators rely on IDE magic to get
> dependencies correct?

Y'know, I didn't really know. So, I spent longer than I'd care to admit in a find|grep loop examining the source. Raw notes as I progressed are appended to the bottom of this post, but the takeaway is that there are only really four kinds of generators:

* Makefiles
* Ninja (Some depfile solution)
* Visual Studio (maybe VisualD will help handle things for us)
* XCode (absolutley no idea)

Everything else is implemented on top of those. All the non-VS non-XC generators ultimately go through cmLocalUnixMakefile3 and cmDepends, or they go through Ninja.

So, we'll still need to get Ninja, VS, and XCode dependency separately, but cmDepends handles the all other generator's dependency resolution.


>
> So...it looks like this is what we're aiming for:
>
>   - DMD/LDC
>     - Support make-style depfiles (optional; unlikely)
>     - Support filtering out excess dependencies (private import, system
>       files) (preferred)

Right now I'm doing direct dependencies only, so if module foo imports std.stdio, it adds a dependency on stdio.di, but nothing else. I think we might actually want full (recursive) dependency listings though, because of how templates work. Alternatively, I can just check the module names and filter out std.* and similar.

>   - Ninja
>     - dmd depfile support (preferred; no comment from martine yet)
>   - CMake
>     - Add depfile support to Make (preferred)
>     - Add dmd -> make depfile translator  (likely necessary[2])

Like I said above, that's what I'm doing in cmDependsD at the moment. Since Ninja, VS and XCode all do their own thing, I think its fine to leave that translator inside cmDependsD.

>
> --Ben
>
> [1]http://stackoverflow.com/a/16969086
> [2]The dmd-depfile format probably won't fly with make upstream since
> make-depfiles aren't actually a thing with make, but gcc.

Whatever CMake does with their depends.make (I didn't go that deep), it works on all of the different Makefile generators CMake supports.

 - Trent

What follows is my notes as I sorted things out:

The method cmLocalUnixMakefileGenerator3::WriteDependLanguageInfo() calls into the cmDepends* system. This method is called by cmMakefileTargetGenerator::WriteTargetDependRules(). This in turn is called by WriteRuleFiles in cmMakefileExecutableTargetGenerator and cmMakefileLibraryTargetGenerator, which is called by cmLocalUnixMakefileGenerator3::Generate().

The cmLUMG3 will be set as the local generator for every cmGlobal*Generator except for Kdevelop, Ninja, VisualStudio and XCode.

Upon inspection:

* Kdevelop uses the Unix Makefile generator behind the scenes
* Ninja does its own thing
* VisualStudio does its own thing
* XCode appears to do its own thing

There are also cmExtra*Generators, for completeness, I'll inspect them too.

* CodeBlocks: uses another Makefile generator
* CodeLight: uses a Makefile generator or Ninja
* Eclipse: uses a Makefile generator or Ninja
* Kate: Makefile or Ninja
* Sublime: Makefile or Ninja

On Thu, Mar 27, 2014 at 00:38:05 +0000, Trent Forkert wrote:
> On Wednesday, 26 March 2014 at 23:17:31 UTC, Ben Boeckel wrote:
> >  4. Add depfile support to Makefile generators.
> 
> That's basically what I'm doing, though only in the context of D.

No, I meant, using the DEPFLAGS during the make build (like ninja does) rather than doing it at configure time.

> Y'know, I didn't really know. So, I spent longer than I'd care to admit in a find|grep loop examining the source.

'git grep' is your friend :) . ('git grep -l cmDepends')

> So, we'll still need to get Ninja, VS, and XCode dependency separately, but cmDepends handles the all other generator's dependency resolution.

Ninja is fine with the 'deps = dmd' solution. VS will need VisualD I imagine (probably fine to require), and XCode will need someone who cares enough to look into it (not me...) ;) . A cursory search finds a[1] few[2] solutions[3], but that's about the limit of my attention span for XCode+D right now.

> Right now I'm doing direct dependencies only, so if module foo imports std.stdio, it adds a dependency on stdio.di, but nothing else. I think we might actually want full (recursive) dependency listings though, because of how templates work. Alternatively, I can just check the module names and filter out std.* and similar.

Recursive public (non-static?) imports is the proper, minimal way to do it; you may be getting an import implicitly via another forwarding module.

> Like I said above, that's what I'm doing in cmDependsD at the moment. Since Ninja, VS and XCode all do their own thing, I think its fine to leave that translator inside cmDependsD.

I think I was unclear: I'd like to see the dependency resolution done at build time, not configure time. That's why there'd be something like:

    %.o: %.d
    %.d: %.dmd.d
        -cmake -DINPUT="$<" -DOUTPUT="$>" -P cmake_root/CMakeDMDToMakeDeps.cmake
    -include $(wildcard *.d)

or something in the Makefile if depfile support is detected (that snippet is a circular dependency, but maybe the CMakeDMDToMakeDeps.cmake could be placed into the %.o recipe after the compile step and make knows nothing of .dmd.d files. I'll have to ask Brad why Make was never converted off of cmDependsC when DEPFILE support was added.

> The method cmLocalUnixMakefileGenerator3::WriteDependLanguageInfo() calls into the cmDepends* system. This method is called by cmMakefileTargetGenerator::WriteTargetDependRules(). This in turn is called by WriteRuleFiles in cmMakefileExecutableTargetGenerator and cmMakefileLibraryTargetGenerator, which is called by cmLocalUnixMakefileGenerator3::Generate().

My suggestion is to remove the cmDepends calls with rules to chain up a .d depends rule supplement when available.

> There are also cmExtra*Generators, for completeness, I'll inspect them too.
> 
> * CodeBlocks: uses another Makefile generator
> * CodeLight: uses a Makefile generator or Ninja
> * Eclipse: uses a Makefile generator or Ninja
> * Kate: Makefile or Ninja
> * Sublime: Makefile or Ninja

My understanding (without looking): the extra generators basically do some scaffolding to write IDE files which just tell the IDE how to run the internal generator, possibly with a list of targets, sources, and whatnot for those which can't manually inspect the build files themselves.

--Ben

[1]http://dblog.aldacron.net/2010/03/22/d-for-xcode-now-with-dmd-support/ [2]http://michelf.ca/projects/d-for-xcode/ [3]http://prowiki.org/wiki4d/wiki.cgi?EditorSupport/AppleXcode

On Thursday, 27 March 2014 at 01:16:57 UTC, Ben Boeckel wrote:
> On Thu, Mar 27, 2014 at 00:38:05 +0000, Trent Forkert wrote:
>> On Wednesday, 26 March 2014 at 23:17:31 UTC, Ben Boeckel wrote:
>> >  4. Add depfile support to Makefile generators.
>> 
>> That's basically what I'm doing, though only in the context of D.
>
> No, I meant, using the DEPFLAGS during the make build (like ninja does)
> rather than doing it at configure time.

I've tested this by creating a wrapper script around dmd to log calls to it in a file. Using that, I can confirm that cmDependsD does nothing at configure time. Granted it didn't refresh the deps file when I updated my D code, but...

I've also tested with a simple C project (to confirm I no D-related bugs get in the way) that depend.make is generated and updated at build time. After building, I went and updated my C file to point to a new header it hadn't touched before, re-ran make (not cmake), and checked that depend.make listed now the header as a dependency (it did). To be extra sure, I reverted the change to the C code, and ran make again. depend.make was updated to remove the header dependency.

The reason cmDependsD didn't update deps at build-time when I tried appears to be a matter of implementing another method I haven't got to yet.

You're right that examine_d_source only worked at configure time, but cmDepends works at build time.

> Ninja is fine with the 'deps = dmd' solution.

I will go with that then.

> VS will need VisualD I
> imagine (probably fine to require)

It absolutely does.

> and XCode will need someone who
> cares enough to look into it (not me...) ;) . A cursory search
> finds a[1] few[2] solutions[3], but that's about the limit of my
> attention span for XCode+D right now.

Yeah, I'm set up to work on Windows and Linux at the moment. Despite using a Macbook, I almost never touch OS X ^_^.

> Recursive public (non-static?) imports is the proper, minimal way to do
> it; you may be getting an import implicitly via another forwarding
> module.
>
>> Like I said above, that's what I'm doing in cmDependsD at the moment.
>> Since Ninja, VS and XCode all do their own thing, I think its fine to
>> leave that translator inside cmDependsD.
>
> I think I was unclear: I'd like to see the dependency resolution done at
> build time, not configure time. That's why there'd be something like:
>
>     %.o: %.d
>     %.d: %.dmd.d
>         -cmake -DINPUT="$<" -DOUTPUT="$>" -P cmake_root/CMakeDMDToMakeDeps.cmake
>     -include $(wildcard *.d)

I can do something like that if its needed, I think. cmDependsFortran appears to generate dependency rules that call CMake, and 'cmake -E cmake_depends ...' is the command that is actually used to generate a depend.make. But, as I said above, cmDepends is a build-time thing. It's more obtuse about it than the Ninja generator is, but it is still build-time.

> My understanding (without looking): the extra generators basically do
> some scaffolding to write IDE files which just tell the IDE how to run
> the internal generator, possibly with a list of targets, sources, and
> whatnot for those which can't manually inspect the build files
> themselves.

Correct.

 - Trent

On Thu, Mar 27, 2014 at 01:52:00 +0000, Trent Forkert wrote:
> I've tested this by creating a wrapper script around dmd to log calls to it in a file. Using that, I can confirm that cmDependsD does nothing at configure time. Granted it didn't refresh the deps file when I updated my D code, but...
> 
> I've also tested with a simple C project (to confirm I no D-related bugs get in the way) that depend.make is generated and updated at build time. After building, I went and updated my C file to point to a new header it hadn't touched before, re-ran make (not cmake), and checked that depend.make listed now the header as a dependency (it did). To be extra sure, I reverted the change to the C code, and ran make again. depend.make was updated to remove the header dependency.
> 
> The reason cmDependsD didn't update deps at build-time when I tried appears to be a matter of implementing another method I haven't got to yet.

Well, that shows my ignorance of now cmDepends works. I now see that it is hooked through 'cmake -E cmake_depends' and not run during configure (as it living within a generator led me to believe). So the goal of using the compiler for this rather than C++ code would be to not have the C++ code in the first place. I'd guess it'd also have make build a lot faster (CMake performance being another recent and ongoing focus of mine)...

> Yeah, I'm set up to work on Windows and Linux at the moment. Despite using a Macbook, I almost never touch OS X ^_^.

XCode updates are a cause for dread from CMake developers. Lots of things always break :( . I don't know if I'll be able to find any willing victims (currently, Ninja bails ASAP on Fortran; making XCode do so for D is probably workable until someone cares enough to write patches).

> I can do something like that if its needed, I think. cmDependsFortran appears to generate dependency rules that call CMake, and 'cmake -E cmake_depends ...' is the command that is actually used to generate a depend.make. But, as I said above, cmDepends is a build-time thing. It's more obtuse about it than the Ninja generator is, but it is still build-time.

I'll have to ask Brad what he perfers for cmDepends* going forward. I don't know the full ramifications of removing it.

--Ben