My naïveté often gets the best of me, so if I seem to be jumping off the deep end, please pull me back before I hurt myself.

I've been wondering why so much effort is necessary to make DMD available as a library. Isn't DMD already a library? If you remove the module containing main(), all you've got left are library files. So why not merely expose them to the masses and call it a day?

Thanks to @alphaglosined on IRC, I started thinking that to make dmd accessible as a library, all that's required is to provide a libdmd.a file in the appropriate directory and expose the dmd directory for use with import statements.
Today I did just that to gain access to the lexer and can use it without any problems.

Here's what I did:

1) modify dmd.conf to include -I%@P%/../../src/dmd

2) navigate to src/dmd/dmd and execute the command
   dmd -lib -J../../../src -of=libdmd.a {root/*, console, \
entity, errors, filecache, globals, id, identifier, \
lexer, tokens, utf, utils, astbase, parse, transitivevisitor, \
permissivevisitor, strictvisitor}.d

2) move the file to the lib folder: mv libdmd.a ~/dlang/dmd-2.093.0/osx/lib/

The result? You be the judge:

(dmd-2.093.0)edwarac@rukongai ~> dmd diced         //[1]
(dmd-2.093.0)edwarac@rukongai ~> ./diced diced.d
pragma
(
lib
,
"dmd"
)
;
import
dmd
.
lexer
;
import
dmd
.
tokens
;
import
std
.
file
;
import
std
.
stdio
;
import
std
.
conv
:
to
;
void
main
(
string
[
]
args
)
{
args
=
args
[
1
..
$
]
;
if
(
args
.
length
>
1
)
{
writeln
(
"Usage: diced [script]"
)
;
}
else
if
(
args
.
length
==
1
)
{
runFile
(
args
[
0
]
)
;
}
else
{
runPrompt
(
)
;
}
}
void
runFile
(
string
path
)
{
path
.
readText
.
run
;
}
void
runPrompt
(
)
{
for
(
;
;
)
{
write
(
"> "
)
;
auto
line
=
stdin
.
readln
(
)
;
if
(
line
is
null
)
break
;
run
(
line
)
;
}
}
void
run
(
string
source
)
{
scope
lex1
=
new
Lexer
(
null
,
source
.
ptr
,
0
,
source
.
length
,
0
,
0
)
;
TOK
tok
=
lex1
.
nextToken
(
)
;
while
(
tok
!=
TOK
.
endOfFile
)
{
writeln
(
lex1
.
token
.
toChars
(
)
.
to
!
string
)
;
tok
=
lex1
.
nextToken
(
)
;
}
}

Could it be that simple, or have I completely missed something? The compiler frontend is missing from this demo, but it shouldn't be too difficult to fix. So please let me know, other than that, what have I missed/misunderstood?

--Andrew

[1] I could have used -L-ldmd here but chose to use a pragma instead (both works)

On Sunday, 6 September 2020 at 11:55:17 UTC, Andrew Edwards wrote:
> My naïveté often gets the best of me, so if I seem to be jumping off the deep end, please pull me back before I hurt myself.
>
> [...]

As far as I remember, the current version schema of dmd (2.0xx.x) isn't semver compatible but that will be solved soon with dmd 2.100.0.

Kind regards
Andre

On Sunday, 6 September 2020 at 12:48:30 UTC, Andre Pany wrote:
> On Sunday, 6 September 2020 at 11:55:17 UTC, Andrew Edwards wrote:
>> My naïveté often gets the best of me, so if I seem to be jumping off the deep end, please pull me back before I hurt myself.
>>
>> [...]
>
> As far as I remember, the current version schema of dmd (2.0xx.x) isn't semver compatible but that will be solved soon with dmd 2.100.0.
>
> Kind regards
> Andre

It's been a running complaint in the forum for over a decade now. I wasn't aware there were efforts underway to resolve the issue, though. Thanks for pointing that out. Just curious, how does semver (non-)compatibility prevent/aid in using the compiler as a library?

Thanks,
Andrew

On 2020-09-06 13:55, Andrew Edwards wrote:
> My naïveté often gets the best of me, so if I seem to be jumping off the deep end, please pull me back before I hurt myself.
> 
> I've been wondering why so much effort is necessary to make DMD available as a library. Isn't DMD already a library? If you remove the module containing main(), all you've got left are library files. So why not merely expose them to the masses and call it a day?
That's correct. And it's already available to the masses through Dub [1][2]. Or is your goal to avoiding using Dub?

If you're referring to the conversations you can see here from time to time about DMD as a library, it's usually about make it easier and more usable as a library. There are a lot of things in DMD that makes it difficult to use as a library.

[1] https://code.dlang.org/packages/dmd
[2] https://github.com/dlang/dmd/blob/master/dub.sdl

-- 
/Jacob Carlborg

On Sunday, 6 September 2020 at 13:15:10 UTC, Jacob Carlborg wrote:
> On 2020-09-06 13:55, Andrew Edwards wrote:
>> My naïveté often gets the best of me, so if I seem to be jumping off the deep end, please pull me back before I hurt myself.
>> 
>> I've been wondering why so much effort is necessary to make DMD available as a library. Isn't DMD already a library? If you remove the module containing main(), all you've got left are library files. So why not merely expose them to the masses and call it a day?
> That's correct. And it's already available to the masses through Dub [1][2]. Or is your goal to avoiding using Dub?

That is indeed the goal. I just want to install the compiler and automatically have this capability. I shouldn't need to use dub for this because the source code included with every compiler install.

> If you're referring to the conversations you can see here from time to time about DMD as a library, it's usually about make it easier and more usable as a library. There are a lot of things in DMD that makes it difficult to use as a library.

Yes. Those are the conversations I am referring to. Ok got you, thanks for the explanation. I always thought it was simply that the user was not able to access the compiler for use in their code. Is there a list somewhere capturing the improvements desired or problems encountered in the as-a-library experience?

On Sunday, 6 September 2020 at 13:13:43 UTC, Andrew Edwards wrote:
> On Sunday, 6 September 2020 at 12:48:30 UTC, Andre Pany wrote:
>> On Sunday, 6 September 2020 at 11:55:17 UTC, Andrew Edwards wrote:
>>> My naïveté often gets the best of me, so if I seem to be jumping off the deep end, please pull me back before I hurt myself.
>>>
>>> [...]
>>
>> As far as I remember, the current version schema of dmd (2.0xx.x) isn't semver compatible but that will be solved soon with dmd 2.100.0.
>>
>> Kind regards
>> Andre
>
> It's been a running complaint in the forum for over a decade now. I wasn't aware there were efforts underway to resolve the issue, though. Thanks for pointing that out. Just curious, how does semver (non-)compatibility prevent/aid in using the compiler as a library?
>
> Thanks,
> Andrew

Afaik there is no active development regarding semver compatible of dmd. It is only a technical detail that 2.099.0 is not a valid semver version while 2.100.0 is. Therefore it could be added as dub package starting with 2.100.0.

Kind regards
Andre

On 2020-09-06 15:47, Andrew Edwards wrote:

> That is indeed the goal. I just want to install the compiler and automatically have this capability. I shouldn't need to use dub for this because the source code included with every compiler install.

You can have a look at the Dub file to see how it's built.

> Is there a list somewhere capturing the improvements desired or problems encountered in the as-a-library experience?

Not really. It's kind of spread out over newsgroup posts. Here are some of the problems:

* There's a lot of global state in the compiler. This might need to be reset when using the compiler in an IDE/editor LSP server

* There's no end location for tokens or AST nodes. This will make source code refactorings difficult/impossible

* Some AST nodes don't have location information at all

* Location information does not contain a buffer position/offset

* It's not possible to analyze source code that is only in memory. Imports and string imports will be looked up on disk

* The error handling/reporting is difficult to customize

* Information is lost between the various compilation phases

* The semantic analyzer is destructive

[1] https://github.com/dlang/dmd/blob/master/dub.sdl

-- 
/Jacob Carlborg