Hi everyone, I'm here for the last update of this 1st Milestone!

So, this week I refactored the code! Unfortunately, I wrote a bunch of useless code while I was trying to understand how ldc does some things, but that ok...the code is much better readable now, so if you have some interest in review the code I'm still updating this branch:
https://github.com/Robertorosmaninho/ldc/tree/fix-llvm-10

Since my last update I worked on the following tasks:
 - Refactoring the code.
    - I was doing some weird stuff to get access to all statements, so now it's possible to declare more than 1 function on a program.
    - I changed the name of the functions for the mangled name.
    - I had some problems with function parameters as well, but finally, I could fix.
 - I made mlir less verbose (HUGE improvements here!!)
 - Implement calls
 - Implement one-dimensional tensors (as a static array)

So far my implementation can deal with:
 - Simple VarDeclarations
 - Operations: +, - , x, /
 - Declare undimensional vectors
 - Function calls
 - All types are integer except the output of arrays

Issues:
 - MLIR is growing too fast I am more than 150 commits behind HEAD and at least yesterday wasn't compiling on my Linux with LLVM trunk.
 - LLVM trunk changed some stuff and I wasn't able to compile with my version of MLIR.

Where I'm at each project:
- LLVM: commit f79d8a064ce4d8846377e4abbc9a389b62f90d43 |  Wed Aug 28 15:40:34 2019.
- MLIR: I'm working on fix-llvm-10 branch so I'm behind commit b50775af52281d9f38aae6c359b72dc715408df0 of Aug 30.

This should be fixed next week, I will try to port my code to the next stable version of each project.

For the next steps:
 - Make CMake FindMLIR more general, it seems like it's not possible to reach projects directory by LLVM_ROOT_DIR on Linux so I have to figure out a new way to find MLIR.
 - Extend the size of tensor and work on support for all types...
 - Create the LDC Dialect

On Tuesday, 15 October 2019 at 23:09:06 UTC, Roberto Rosmaninho wrote:
> Hi everyone, I'm here for the last update of this 1st Milestone!

Hey Roberto,
  It's fun to read your updates, thanks :)

> So, this week I refactored the code! Unfortunately, I wrote a bunch of useless code while I was trying to understand how ldc does some things, but that ok...the code is much better readable now, so if you have some interest in review the code I'm still updating this branch:
> https://github.com/Robertorosmaninho/ldc/tree/fix-llvm-10

I hope to get the LLVM 10 fixes into LDC master soon. Then you can rebase on top of that to make the specific MLIR changes more obvious and easier to view.

> Since my last update I worked on the following tasks:
>  - Refactoring the code.
>     - I was doing some weird stuff to get access to all statements, so now it's possible to declare more than 1 function on a program.
>     - I changed the name of the functions for the mangled name.
>     - I had some problems with function parameters as well, but finally, I could fix.
>  - I made mlir less verbose (HUGE improvements here!!)
>  - Implement calls
>  - Implement one-dimensional tensors (as a static array)
>
> So far my implementation can deal with:
>  - Simple VarDeclarations
>  - Operations: +, - , x, /
>  - Declare undimensional vectors
>  - Function calls
>  - All types are integer except the output of arrays

(I didn't check if you already did this) I find it very helpful to look at test cases to see what functionality you are adding. The lit-based testsuite is very convenient for checking IR output. Did you already add MLIR tests there?

> Issues:
>  - MLIR is growing too fast I am more than 150 commits behind HEAD and at least yesterday wasn't compiling on my Linux with LLVM trunk.
>  - LLVM trunk changed some stuff and I wasn't able to compile with my version of MLIR.

Don't stress out too much about not following LLVM trunk exactly. But also don't lag too long behind, such that the changes you need to make are not too big and that you keep up to date on impactful MLIR changes. So perhaps only pull MLIR trunk once in 2 weeks or so.
You can try to offload some of the general LLVM trunk compile issues to others, by making us aware of LLVM trunk compile issues. I can try to help with that.

> Where I'm at each project:
> - LLVM: commit f79d8a064ce4d8846377e4abbc9a389b62f90d43 |  Wed Aug 28 15:40:34 2019.
> - MLIR: I'm working on fix-llvm-10 branch so I'm behind commit b50775af52281d9f38aae6c359b72dc715408df0 of Aug 30.
>
> This should be fixed next week, I will try to port my code to the next stable version of each project.

LLVM trunk compilation fixes (not related to MLIR) you can submit as PRs to LDC. That way your branch is solely about MLIR changes, and not cluttered by the general LLVM changes needed.

> For the next steps:
>  - Make CMake FindMLIR more general, it seems like it's not possible to reach projects directory by LLVM_ROOT_DIR on Linux so I have to figure out a new way to find MLIR.

I was hoping you had solved these issues. I'm still confused by "reach projects directory", is it not possible to `ninja install` MLIR ?

>  - Extend the size of tensor and work on support for all types...
>  - Create the LDC Dialect

Looking forward to the next update.

cheers,
  Johan

On Thursday, 17 October 2019 at 22:05:30 UTC, Johan wrote:
> Hey Roberto,
>   It's fun to read your updates, thanks :)

Hey Johan, I'm glad to hear that, thank you!!

> I hope to get the LLVM 10 fixes into LDC master soon. Then you can rebase on top of that to make the specific MLIR changes more obvious and easier to view.

Nic told me about your PR and I'm already trying to rebase my code on it!

> (I didn't check if you already did this) I find it very helpful to look at test cases to see what functionality you are adding. The lit-based testsuite is very convenient for checking IR output. Did you already add MLIR tests there?

Not yet, I'll use this test suite on this next phase, maybe I can put some tests that I'm already writing there. :)

> Don't stress out too much about not following LLVM trunk exactly. But also don't lag too long behind, such that the changes you need to make are not too big and that you keep up to date on impactful MLIR changes. So perhaps only pull MLIR trunk once in 2 weeks or so.

Great idea, I'll follow it, thanks!

> You can try to offload some of the general LLVM trunk compile issues to others, by making us aware of LLVM trunk compile issues. I can try to help with that.

Ok!

> LLVM trunk compilation fixes (not related to MLIR) you can submit as PRs to LDC. That way your branch is solely about MLIR changes, and not cluttered by the general LLVM changes needed.

I'm currently trying to use Linux to update the project and find the projects directory to improve my FindMLIR.cmake but I'm having the following problem over and over and I coudn't figure out what I'm doing wrong:

Commands:
$ git clone http://github.com/ldc-developers/ldc
$ cd ldc
$ git checkout llvm10cmake
$ mkdir build && cd build
$ export DMD=~/dlang/dmd-2.087.1/linux/bin64/dmd
$ cmake -G Ninja ..  -DCMAKE_BUILD_TYPE=Release
# Cmake works ok and LLVM10 was found, I can put the log of CMake here later if I request.
$ ninja
$ ninja install
#It gives me the following error:

CMake Error at cmake_install.cmake:44 (file):
  file INSTALL cannot copy file "/home/roberto/dlang/ldc/build/bin/ldc2" to
  "/usr/local/bin/ldc2".


FAILED: CMakeFiles/install.util
cd /home/roberto/dlang/ldc/build && /usr/bin/cmake -P cmake_install.cmake
ninja: build stopped: subcommand failed.

#When I try to compile a simple program: int main(){int a = 10; return 0;}
$ ~/ldc/build/bin/ldc2 sum.d

Error: cannot find source code for runtime library file 'object.d'
       ldc2 might not be correctly installed.
       Please check your ldc2.conf configuration file.
       Installation instructions can be found at http://wiki.dlang.org/LDC.
Specify path to file 'object.d' with -I switch

> I was hoping you had solved these issues. I'm still confused by "reach projects directory", is it not possible to `ninja install` MLIR ?

Well, the instructions on MLIR git just ask for a "cmake --build . --target check-mlir" when you're compiling LLVM with MLIR.

The problem is that doesn't exist any flags such as $LLVM_SOURCE_DIR or $LLVM_PROJECTS_DIR. So unless I pass the path of $MLIR_ROOT_DIR BY command line I'm don't know any way to reach the source code of llvm to get MLIR files. Unfortunately, the project's dir inside llvm build doesn't provide all the files that I need.

Just to illustrate my problem with LLVM_ROOT_DIR I added the following line on my cmake:
      message("LLVM ROOT DIR IS: ${LLVM_ROOT_DIR}")
Then when I run the CMake I get the following result:
      LLVM ROOT DIR IS: /home/roberto/llvm-project/build


Hope I can solve this soon!

Best regards,
Roberto

On Friday, 18 October 2019 at 21:30:16 UTC, Roberto Rosmaninho wrote:
> On Thursday, 17 October 2019 at 22:05:30 UTC, Johan wrote:
>> Hey Roberto,
>>   It's fun to read your updates, thanks :)
>
> Hey Johan, I'm glad to hear that, thank you!!
>
>> I hope to get the LLVM 10 fixes into LDC master soon. Then you can rebase on top of that to make the specific MLIR changes more obvious and easier to view.
>
> Nic told me about your PR and I'm already trying to rebase my code on it!

I just merged it into LDC master, so you can rebase on LDC master again.

>> (I didn't check if you already did this) I find it very helpful to look at test cases to see what functionality you are adding. The lit-based testsuite is very convenient for checking IR output. Did you already add MLIR tests there?
>
> Not yet, I'll use this test suite on this next phase, maybe I can put some tests that I'm already writing there. :)

Exactly. I find it very helpful to immediately add the tests in the lit test suite while implementing things.

>> LLVM trunk compilation fixes (not related to MLIR) you can submit as PRs to LDC. That way your branch is solely about MLIR changes, and not cluttered by the general LLVM changes needed.
>
> I'm currently trying to use Linux to update the project and find the projects directory to improve my FindMLIR.cmake but I'm having the following problem over and over and I coudn't figure out what I'm doing wrong:
>
> Commands:
> $ git clone http://github.com/ldc-developers/ldc
> $ cd ldc
> $ git checkout llvm10cmake
> $ mkdir build && cd build
> $ export DMD=~/dlang/dmd-2.087.1/linux/bin64/dmd
> $ cmake -G Ninja ..  -DCMAKE_BUILD_TYPE=Release
> # Cmake works ok and LLVM10 was found, I can put the log of CMake here later if I request.
> $ ninja
> $ ninja install

For LDC, you don't have to do a `ninja install`. If you do, you should specify where to install it when running cmake. What I do is have a `run_cmake.sh` that runs cmake for me so I can remember what options I passed to cmake. For reference, this is my `run_cmake.sh` for building with LLVM trunk:
```
cmake -G Ninja -DRT_SUPPORT_SANITIZERS=ON -DLDC_DYNAMIC_COMPILE=False -DD_COMPILER="/Users/johan/ldc/ldc2-1.12.0-osx-x86_64/bin/ldmd2" -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_INSTALL_PREFIX="/Users/johan/ldc/johan/installtrunk" -DLLVM_ROOT_DIR="/Users/johan/llvm/llvmtrunkinstall" ..
```

Note that I explicitly pass LLVM_ROOT_DIR where my LLVM trunk build is installed.

>> I was hoping you had solved these issues. I'm still confused by "reach projects directory", is it not possible to `ninja install` MLIR ?
>
> Well, the instructions on MLIR git just ask for a "cmake --build . --target check-mlir" when you're compiling LLVM with MLIR.

The instructions don't say it, because I think most people will already know what to do and there are many cases where one would _not_ want to install LLVM.
This is what my `run_cmake.sh` looks like for building LLVM:
```
cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;polly" -DCMAKE_INSTALL_PREFIX="/Users/johan/llvm/llvmtrunkinstall" -DLLVM_TARGETS_TO_BUILD="AArch64;ARM;Mips;MSP430;NVPTX;PowerPC;X86" -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="RISCV;WebAssembly" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_APPEND_VC_REV=ON -DLLVM_INSTALL_UTILS=ON ..
```
Note the "-DCMAKE_INSTALL_PREFIX", that's where LLVM will be installed. And that is what you need to pass to LDC CMake "-DLLVM_ROOT_DIR".

Hope that helps,
  Johan

On Friday, 18 October 2019 at 21:43:53 UTC, Johan Engelen wrote:
> I just merged it into LDC master, so you can rebase on LDC master again.

Great, I already did the git pull here.

> For LDC, you don't have to do a `ninja install`. If you do, you should specify where to install it when running cmake. What I do is have a `run_cmake.sh` that runs cmake for me so I can remember what options I passed to cmake. For reference, this is my `run_cmake.sh` for building with LLVM trunk:
> ```
> cmake -G Ninja -DRT_SUPPORT_SANITIZERS=ON -DLDC_DYNAMIC_COMPILE=False -DD_COMPILER="/Users/johan/ldc/ldc2-1.12.0-osx-x86_64/bin/ldmd2" -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_INSTALL_PREFIX="/Users/johan/ldc/johan/installtrunk" -DLLVM_ROOT_DIR="/Users/johan/llvm/llvmtrunkinstall" ..
> ```
>
> Note that I explicitly pass LLVM_ROOT_DIR where my LLVM trunk build is installed.

I just tried to compile it here using your script as baseline:

$ cmake -G Ninja -DRT_SUPPORT_SANITIZERS=ON -DLDC_DYNAMIC_COMPILE=False -DD_COMPILER="/home/roberto/dlang/dmd-2.087.1/linux/bin64/dmd" -DCMAKE_BUILD_TYPE="Debug" -DCMAKE_INSTALL_PREFIX="/home/roberto/dlang/ldc" -DLLVM_ROOT_DIR="/home/roberto/llvm-project" ..

$ ninja

Then I tried to compile this example: https://run.dlang.io/is/ulDVcv

$ ~/dlang/ldc/build/bin/ldc2 test.d

And get the same error:

Error: cannot find source code for runtime library file 'object.d'
       ldc2 might not be correctly installed.
       Please check your ldc2.conf configuration file.
       Installation instructions can be found at http://wiki.dlang.org/LDC.
Specify path to file 'object.d' with -I switch

I'm not trying my code yet, this is just the master branch that I'm trying to use.
Any ideas or should I report this as a issue?

>>> I was hoping you had solved these issues. I'm still confused by "reach projects directory", is it not possible to `ninja install` MLIR ?
>>
>> Well, the instructions on MLIR git just ask for a "cmake --build . --target check-mlir" when you're compiling LLVM with MLIR.
>
> The instructions don't say it, because I think most people will already know what to do and there are many cases where one would _not_ want to install LLVM.

Right!

> This is what my `run_cmake.sh` looks like for building LLVM:
> ```
> cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;lld;polly" -DCMAKE_INSTALL_PREFIX="/Users/johan/llvm/llvmtrunkinstall" -DLLVM_TARGETS_TO_BUILD="AArch64;ARM;Mips;MSP430;NVPTX;PowerPC;X86" -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="RISCV;WebAssembly" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_APPEND_VC_REV=ON -DLLVM_INSTALL_UTILS=ON ..
> ```
> Note the "-DCMAKE_INSTALL_PREFIX", that's where LLVM will be installed. And that is what you need to pass to LDC CMake "-DLLVM_ROOT_DIR".

I will test it as soon as I can compile LDC without my code!
Best regards,
Roberto

On Friday, 18 October 2019 at 22:18:29 UTC, Roberto Rosmaninho wrote:
> $ ~/dlang/ldc/build/bin/ldc2 test.d
>
> And get the same error:
>
> Error: cannot find source code for runtime library file 'object.d'
>        ldc2 might not be correctly installed.
>        Please check your ldc2.conf configuration file.
>        Installation instructions can be found at http://wiki.dlang.org/LDC.
> Specify path to file 'object.d' with -I switch
>
> I'm not trying my code yet, this is just the master branch that I'm trying to use.
> Any ideas or should I report this as a issue?

Looks like you simply haven't cloned the git submodules. Add --recursive to the git clone command, or do a `git submodule update --init` in an existing LDC repo. Note that CMake spits out a warning when the druntime isn't found and informs that only the compiler will be built.

On Friday, 18 October 2019 at 22:25:00 UTC, kinke wrote:
> Looks like you simply haven't cloned the git submodules. Add --recursive to the git clone command, or do a `git submodule update --init` in an existing LDC repo. Note that CMake spits out a warning when the druntime isn't found and informs that only the compiler will be built.

Oh God, I knew it was just a small problem, but I wasn't expected a dumb one!!
So sorry and thanks Kinke and Johan!

Hi everyone, here goes another update from my project!

This week I was able to build the support for more BinOps:
 - ModExp
 - AndExp
 - OrExp
 - XorExp
 - EqualExp
 - CmpExp(slt,sle,sgt,sge,ult,ule,ugt,uge,neq,eq)

And the last three days I worked to implement Ifstatement witch cost me a lot of time to figure out the right way to do that due the lack of documentation of implementation (like a doxygen) and none examples provided by MLIR. The next week I plan to finish those BinOps, replace the opaque BinOps for those provided by a StandardDialect from MLIR and start to build the D Dialect. My primary goal for now it will be build the ForStatement support so we will be able to have some cool tests like matrix multiplication or some like that to put on D test suit.

Hopefully next week I'll bring this test to show you, but for now you can track my work on https://github.com/Robertorosmaninho/ldc/tree/fix-llvm-10 and maybe provide some ideas or show me something that I'm not doing right.

About the cmake problem on FindMLIR.cmake the one way that I could think about solve was passing the MLIR path as cmake argument with -DMLIR_ROOT_DIR=/path/to/dir and then all the directories necessaries for MLIR works will be set.

Best regards,
Roberto Rosmaninho

On Saturday, 26 October 2019 at 00:00:51 UTC, Roberto Rosmaninho wrote:
> Hi everyone, here goes another update from my project!
>
> About the cmake problem on FindMLIR.cmake the one way that I could think about solve was passing the MLIR path as cmake argument with -DMLIR_ROOT_DIR=/path/to/dir and then all the directories necessaries for MLIR works will be set.

I wouldn't waste too much more time on that, MLIR will soon be moving to the mono-repo[1], and so will pretty much always be in the same place.

[1]: https://github.com/llvm/llvm-project

Hi everyone, here I know that I'm not writing for a while and I apologize for that, but here it's what I've been working on the past few weeks:

 - Add the notion of blocks and control flow: Support to IfStatement and ForStatement
 - Add support to fill different blocks
 - Refactoring of MLIRDeclaration to infer the type of each declaration as is used on MLIRStatement
 - Support for Template Interface
 - Improving on AssignExp
 - Support for PostExp: MinusMinus and PlusPlus
 - Support for Assign Bin Ops
 - Introduction to D Dialect: Creation of Dialect Class
 - Dialect Registration
 - Creation of Ops.td a file written in mlir-tablegen style (ODS[1]) to define Ops on D Dialect, for now only AddExp and MulExp are implemented for int{8,16,32,64} and float{16,32,64} types
 - Support to bool, float and different sizes of int on the entire project, other types depend on the creation of new types, MLIR doesn't support char, null, none by default.

Hopefully, I'll finish writing the Ops.td until next week, then I'll try to build my first MLIR pass to improve ForStatement with arguments on the block which replace the phi functions of LLVM.

Suggestions, ideas, and code reviews are more than welcome!
[1]Table-driven Operation Definition Specification (ODS) - https://github.com/tensorflow/mlir/blob/master/g3doc/OpDefinitions.md#table-driven-operation-definition-specification-ods

GitHub of the project: https://github.com/Robertorosmaninho/ldc/tree/fix-llvm-10

Thanks for your attention,
Roberto Rosmaninho