Hi everyone, I'm responsible to implement MLIR support to LDC, just in case you missed my presentation post about the project this is the post that I wrote last week: https://forum.dlang.org/thread/hinceoqtdfcxhahbuwru@forum.dlang.org

This first week I finished the first topic of Milestone 1:
  Milestone 1. Compilation of D through the LLVM dialect of MLIR:
    - Build and set up as the newest version available: LLVM (10.0.0svn), MLIR, LDC (1.17.0) and DMD (v2.087.1).

Thanks to my mentor Nicholas Wilson and other LDC developers I'm now able to work with LDC and LLVM 10, which is the version that MLIR it's built.

For now, I have just two questions:
I've already started to play with LDC source code but I still don't know how to run and test the modifications that I writing. How can I see a "printf" that I put in a file just to test a variable?
How is the best way to debug LDC on MacOS? And how to enable all flags of debug and log?

I'm currently working on irstate.* and declarations.cpp, any tips or warnings about these files?

Regards,
Roberto Rosmaninho

On Wednesday, 25 September 2019 at 01:47:43 UTC, Roberto Rosmaninho wrote:
> Hi everyone, I'm responsible to implement MLIR support to LDC, just in case you missed my presentation post about the project this is the post that I wrote last week: https://forum.dlang.org/thread/hinceoqtdfcxhahbuwru@forum.dlang.org
>

IMO, it's better to keep updates on one thread (e.g. your presentation thread).
That way, one can open only one thread and see all the history.

>
> For now, I have just two questions:
> I've already started to play with LDC source code but I still don't know how to run and test the modifications that I writing. How can I see a "printf" that I put in a file just to test a variable?
> How is the best way to debug LDC on MacOS? And how to enable all flags of debug and log?

So, if you just want to print something for debugging purposes,
you can just use printf. However, I would recommend getting used to
Logger utilities. Let me explain:
You can see across LDC this "Logger" thing everywhere
There is code like:
IF_LOG Logger::println("InterfaceDeclaration::codegen: '%s'",
                       decl->toPrettyChars())

(That specific code is in visit(InterfaceDeclaration) in declarations.cpp).

These work like printf but with some added features. First, if you put
this IF_LOG thing in the front, then the message is printed only if logging
is enabled (don't worry, I will explain how to enable logging). Otherwise,
always of course. That helps so that you can leave logging info around
(when done debugging and you feel that there's something that is useful
to be printed when one requests a log).

The second thing which for me is super smart and useful is that these
utilities have a sense of scope. Think of this example:

void b() {
  Logger::println("roberto2");
}

void a() {
  Logger::println("roberto1");
  b();
}

With normal printf, you would see:
roberto1
roberto2

But with Logger::println, you see:
roberto1
* roberto2

Notice the indentation and the star. ;)
That is super useful because it decouples messages and helps you pinpoint
where your message is in the bigger picture.

(That thing btw is controlled with LOG_SCOPE which you will also see
around and you can learn more about how it works either by just grepping
it or reading this [1]).

So, now that you have a nicer way to print, how one enables logging?

Logging is of course useful not in order to see _your_ message, but all the
other messages and how yours is placed among them.
There are 2 kinds of logging (that I am aware of. I'm pretty happy with the second
so I didn't search more).
The one is -v which is "Verbose". What that means basically is it logs
info not for the LDC programmer but for the LDC user, who happens to care
to see some specific info about what LDC does.
There is also -vv which has a weird description but basically means: "Info
for the LDC developers that care about the internal things that happen".

You can see these and all the other options here [2]

You probably want the second, -vv. Since we're on the subject, the other
thing you probably will like is -output-ll which outputs the LLVM IR.
You may even end up to put a specific flag for MLIR. But for the time being,
seeing the LLVM IR I think will help.

Lastly, the usual workflow for LDC is Johan's workflow [3].
You don't necessarily want to follow this for _every single change_. That will
kill productivity and it's already kind of boring to wait for compilation
when you change even a single variable.
What you probably want though is having a specific (or a bunch of specific)
test case that you run (instead of running the whole suite) continuously.
and _definitely_, you run the whole suite when you're done. I'd suggest
to run it every so often, I usually do it every 10-20 "small" changes.

Now, notice this note I have added on the bottom of the page:

Note: `ninja` builds the whole LDC suite. Most of the time, you'll want to only rebuild the ldc2 binary. You can do that with `ninja ldc2`.

So, when you do a change on a single file, you want to do `ninja ldc2`. Trust me,
this is day and night in how much time it takes to compile.

Unfortunately, I can't give you any info about MacOS because I have not used
one for any serious programming. I _think_ David Nadlinger
uses Mac but I may very well be wrong.

AFAIK, you can have GDB. My guess is that Mac will have a more Visual Studio kind
of debugger because the terminal interface of GDB is just nerve-racking.
But since it's better than nothing for some situations, if I can't target
something easily, I do "info functions" while first I have made gdb output
to a file and then I grep this file for function that I want. This is
more complicated than you would expect because of name mangling and stuff.
I described this very roughly since I don't know if you're interested. Let
me know if you, but I would also recommend to ask Mac users for more info.

Lastly, make sure you have read this page [4] and that you have installed
LLVM and LDC with -DCMAKE_BUILD_TYPE=RelWithDebInfo. And bear in mind
that LDC debug info is not the best :).

Best of luck,
Stefanos Baziotis

>
> I'm currently working on irstate.* and declarations.cpp, any tips or warnings about these files?
>
> Regards,
> Roberto Rosmaninho

[1] https://forum.dlang.org/post/xbfsztzkrqtxmfgcflug@forum.dlang.org
[2] https://wiki.dlang.org/Using_LDC
[3] https://wiki.dlang.org/LDC_Lit-based_testsuite#My_workflow_when_working_on_LDC_.28Johan.29
[4] https://wiki.dlang.org/Building_LDC_from_source#Building_LLVM_manually

On Wednesday, 25 September 2019 at 01:47:43 UTC, Roberto Rosmaninho wrote:
>
> For now, I have just two questions:
> I've already started to play with LDC source code but I still don't know how to run and test the modifications that I writing. How can I see a "printf" that I put in a file just to test a variable?
> How is the best way to debug LDC on MacOS? And how to enable all flags of debug and log?

I forgot to remind you: Don't hesitate to ask LDC-specific questions in
the LDC gitter [1]. Apart from the fact that it's very welcome,
you also have a better chance that seasoned developers will see it and/or
answer like kinke and Johan (although they will probably see it here too).

> I'm currently working on irstate.* and declarations.cpp, any tips or warnings about these files?

I personally don't have any experience with those files. Bear in mind that
in my experience, questions about constrained situations (like, oh in the visit of
ClassDeclaration I don't understand the x) are easier to answer than broad ones.

Now that you mentioned it though, you reminded me that LDC does not have
a header description for a file (like on the top of every file, describing
what the file does roughly). I find this useful. I also find useful
a source guide like DMD's [2]. I'm trying to construct one on my own bit by bit.
If along the course of your project you're interested to contribute to the
documentation, let me know.

For now, you can safely ignore the last paragraph, I just left it there
for future reference. One thing to have in mind though is that you probably
will need to get somewhat comfortable with the DMD source code, so this
DMD guide might help.

Best regards,
Stefanos Baziotis

[1] http://gitter.im/ldc-developers/main
[2] https://wiki.dlang.org/DMD_Source_Guide

On Wednesday, 25 September 2019 at 10:42:46 UTC, Stefanos Baziotis wrote:

> IMO, it's better to keep updates on one thread (e.g. your presentation thread).
> That way, one can open only one thread and see all the history.
>
All right, I'll keep updating this thread them, thanks!

> So, if you just want to print something for debugging purposes,
> you can just use printf. However, I would recommend getting used to
> Logger utilities.
> . . .
>
> There is also -vv which has a weird description but basically means: "Info
> for the LDC developers that care about the internal things that happen".
>
> You can see these and all the other options here [2]
>
> You probably want the second, -vv. Since we're on the subject, the other

Thanks a lot!! It'll make my work much easier!!

> thing you probably will like is -output-ll which outputs the LLVM IR.
> You may even end up to put a specific flag for MLIR. But for the time being,
> seeing the LLVM IR I think will help.

Yeah! The plan is to have a -output-mlir at the end of the project!

> Lastly, the usual workflow for LDC is Johan's workflow [3].

This workflow makes sense even for this project that I still have to change a lot of files to get some output, just to know if I'm crashing something. Thanks again!

> Note: `ninja` builds the whole LDC suite. Most of the time, you'll want to only rebuild the ldc2 binary. You can do that with `ninja ldc2`.
> So, when you do a change on a single file, you want to do `ninja ldc2`. Trust me,
> this is day and night in how much time it takes to compile.

This seems powerful due the huge time to compile LDC just because of single modification. I'll try it!

> Unfortunately, I can't give you any info about MacOS because I have not used
> one for any serious programming. I _think_ David Nadlinger
> uses Mac but I may very well be wrong.

I have set up my project on Ubuntu as well, so if you know something about debugging on Linux and can explain it in greater detail it'll help anyway!

> Lastly, make sure you have read this page [4] and that you have installed
> LLVM and LDC with -DCMAKE_BUILD_TYPE=RelWithDebInfo. And bear in mind
> that LDC debug info is not the best :).

Done!

> Best of luck,
> Stefanos Baziotis
>
Thanks again,
Roberto Rosmaninho

On Wednesday, 25 September 2019 at 11:02:35 UTC, Stefanos Baziotis wrote:

> Now that you mentioned it though, you reminded me that LDC does not have
> a header description for a file (like on the top of every file, describing
> what the file does roughly). I find this useful. I also find useful
> a source guide like DMD's [2]. I'm trying to construct one on my own bit by bit.
> If along the course of your project you're interested to contribute to the
> documentation, let me know.

Yeah, I'm trying to understand the code and without some explanation about the files it has been really difficult, this lack of documentation is annoying but certainly, I'd contribute to improving this, I'm on Slack and Gitter just ping me there!


Regards,
Roberto Rosmaninho

On Wednesday, 25 September 2019 at 13:30:20 UTC, Roberto Rosmaninho wrote:
> On Wednesday, 25 September 2019 at 10:42:46 UTC, Stefanos Baziotis wrote:
> Thanks a lot!! It'll make my work much easier!!

I'm glad!

> This seems powerful due the huge time to compile LDC just because of single modification. I'll try it!

Yes, for me it was crucial. You can do the same thing for more changes.
Basically, as long as you change only LDC files.

> I have set up my project on Ubuntu as well, so if you know something about debugging on Linux and can explain it in greater detail it'll help anyway!

IMO, for the most part the debugging experience on Linux is horrible. :)
If you're lucky, because LDC is mostly C++, you may be able to get
CLion to work. Or make Visual Studio Code work as a GUI for GDB (I have not
done either of those, so I can't provide further info, but there's info
on the Internet).
The common case though is that you're stuck with GDB, which while powerful,
its terminal interface personally drives me crazy.

But, enough with bashing on GDB. I debug LDC mostly with printfs, but when
these don't work, here's what I do:

Note: I will cover some basics in case you know nothing about GDB.
I'm not a GDB guru anyway, but these will help you in most situations.

== Case 1 - You want to break in LDC / LLVM code ==

This is the easiest case. You just run the executable through gdb, then
break on a specific line by specifying the _full_ path. For example, I
have my ldc repo in: /home/stefanos/dlang/ldc/

So:
$ gdb /home/stefanos/dlang/ldc/build/bin/ldc2
... GDB opens. Inside GDB:
$ break /home/stefanos/dlang/ldc/gen/statements.cpp:289

to break on line 289 on gen/statements.cpp

You can similarly break into a function, e.g.

$ break ToIRVisitor::visit(IfStatement*)

using tab for autocomplete.

Similarly, you can break in LLVM code, e.g.:

$ break llvm::BasicBlock::Create(llvm::LLVMContext&, llvm::Twine const&, llvm::Function*, llvm::BasicBlock*)

again using autocomplete.

== Case 2 - You want to break in DMD code ==

For that, I'd recommend using the dmd executable, built with dmd.
Basically, doing what you would do if you were working on DMD. So, you download
the DMD version you want and do the same debugging steps.
You just have better debugging experience when DMD is built with DMD.

For now, you can forget what I said about listing functions and stuff. I can't remember why I had needed that.

Best regards,
Stefanos

On Wednesday, 25 September 2019 at 14:16:36 UTC, Roberto Rosmaninho wrote:
>
> Yeah, I'm trying to understand the code and without some explanation about the files it has been really difficult

There's not been much time since I started contributing to LDC so I know
how it feels. Don't hesitate to ask questions! I hope I will be able to
provide some feedback if kinke, johan etc. don't have time.

LDC is kind of a beast, with minimal commenting (for my taste), so just
stick with it. :)

> this lack of documentation is annoying but certainly, I'd contribute to improving this, I'm on Slack and Gitter just ping me there!

I'm on both Slack and Gitter too, so you can reach me if you have any problem.
For now, I think the best things to do if you want to help with doc are (these things are the ones I do as well and that require minimal effort so they
shouldn't take valuable time):
a) Ask about what you don't understand! It's obvious that to be selected
for SAoC, you're good. Meaning, if you don't understand something, it's probably
not something that is obvious and / or not something that one would understand
easily (or at all) from context. So, IMO, we better add some clarification
for future contributors like you and me.

b) Whenever you feel you have understood something non-trivial, add a clarification
comment. Even if it's wrong, it will be caught in PR review (it's better to
actually pinpoint to the comment on github so that reviewers can spot it easily
and verify it).

What I plan to do other than those micro-changes is roughly:
a) Add a header description in files.
b) Write some articles describing the architecture of LDC, maybe some diagrams
etc. Sort of like the DMD guide, but more visual and extended.

These actions require 2 things that I currently don't have:
a) Knowing more about LDC.
b) Time.

When I get around to do any of what I described, I hope that LDC people
will have the time to review them. In any case, if you feel that in the course
of your project, you can provide any contributions in what is described above,
be my guest!

Kind regards,
Stefanos Baziotis

Hi everyone,
I'm here just to report the status of my project on its second week:

 - I Read all documentation from MLIR
 - I Started to implement mlirstate.h and mlirstate.cpp
 - I Started to implement mlircodegenerator.h mlircodegenerator.cpp

The main issue that I stuck this week was to compile ldc with my modifications due to problems with mlir headers and dmd/globals.h.
So I had these 2 questions and I hope someone could help me:

 - What is the best way to refer mlir (a project inside llvm, like clang) into ldc CMakeLists.txt? I need the headers provided by this project to write my codes.

 - I'm trying to do the same things that ldc does to get LLVM IR, so I tried to create a "DArray<const char> mlir_ext;" in dmd/globals.h:296, but when I compile ldc with -vv it stops at " Targeting 'x86_64-apple-macosx' (CPU 'core2' with features '') " and tell me that the process finished with exit code 1. How can I fix this? And if I'm doing something wrong, what is the best way to get these variables for ldc and what is its function on the program? I thought it was setting the file extension, but I'm not sure.

Regards,
Roberto Rosmaninho

On Wednesday, 2 October 2019 at 18:43:14 UTC, Roberto Rosmaninho wrote:
> Hi everyone,
> I'm here just to report the status of my project on its second week:
>
>  - I Read all documentation from MLIR
>  - I Started to implement mlirstate.h and mlirstate.cpp
>  - I Started to implement mlircodegenerator.h mlircodegenerator.cpp
>
> The main issue that I stuck this week was to compile ldc with my modifications due to problems with mlir headers and dmd/globals.h.
> So I had these 2 questions and I hope someone could help me:
>
>  - What is the best way to refer mlir (a project inside llvm, like clang) into ldc CMakeLists.txt? I need the headers provided by this project to write my codes.

As far as I can tell, if you checkout MLIR into your local LLVM checkout [1], then it is automatically built with LLVM and the header files will be part of the LLVM install. So to get to the headers you probably don't have to add anything in LDC's CMakeLists.txt.
For linking correctly, you probably have to add similar CMake code as is done for SPIR-V, see the cmake code around `LLVM_SPIRV_FOUND`.

>  - I'm trying to do the same things that ldc does to get LLVM IR, so I tried to create a "DArray<const char> mlir_ext;" in dmd/globals.h:296, but when I compile ldc with -vv it stops at " Targeting 'x86_64-apple-macosx' (CPU 'core2' with features '') " and tell me that the process finished with exit code 1. How can I fix this? And if I'm doing something wrong, what is the best way to get these variables for ldc and what is its function on the program? I thought it was setting the file extension, but I'm not sure.

globals.h _must exactly_ match with globals.d and there is no tool that will check that for you ('.h' is used by the C++ compiler, '.d' by the D compiler and they don't talk to each other). One way to look at it is that `globals.d` is the `.cpp` file belonging to `globals.h`.
For your particular issue: is you add `mlir_ext` to `globals.h`, you have to add it to `globals.d` in _exactly the same_ location in the `Global` struct.

Do you aim to take a similar approach as the SPIR-V/dcompute mechanism?

cheers,
  Johan

[1] https://github.com/tensorflow/mlir

On Wednesday, 2 October 2019 at 20:31:02 UTC, Johan Engelen wrote:
> As far as I can tell, if you checkout MLIR into your local LLVM checkout [1], then it is automatically built with LLVM and the header files will be part of the LLVM install.

Yes! I found a solution: create a cmake find for mlir, ldc have to mlir and it fixes the problem: http://bit.ly/2n7xEJS

> globals.h _must exactly_ match with globals.d and there is no tool that will check that for you ('.h' is used by the C++ compiler, '.d' by the D compiler and they don't talk to each other). One way to look at it is that `globals.d` is the `.cpp` file belonging to `globals.h`.
> For your particular issue: is you add `mlir_ext` to `globals.h`, you have to add it to `globals.d` in _exactly the same_ location in the `Global` struct.

Works here! Thanks!!

> Do you aim to take a similar approach as the SPIR-V/dcompute mechanism?

I'm new to ecosystem D, so I don't know these approaches in D, do you mind explaining me about it or giving me a guide to study?

Best regards,
Roberto Rosmaninho