Thread overview | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
August 14, 2014 Recompiling D code | ||||
---|---|---|---|---|
| ||||
Hello everyone. As you all may know I've been working on recompiling D for web services last few weeks. Its both good news and bad news. Good: Reloading definitely possible. With dependency handling using dub. Bad: Its slow. And not in my code sort of way. You're welcome to atest my speeds that I have gotten so far (older well used hdd, Win7 x64), its around 2-3 seconds for a basic go download webpage and not much more. I've done a bit of working with it and with 2.066-rc2 got it to the lower end of 2s. Atleast from my experience with this, its dmd thats actually taking the time. I know my code isn't the best and optimized, so I have compared mine against dmd itself. 100-200ms difference normally. For those who don't know why I'm making a fuss over 2 seconds, its because you actually do notice this when doing web dev stuff. Without including start up time, this could become a lot larger. [0] https://github.com/rikkimax/livereload |
August 14, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to Rikki Cattermole | On Thursday, 14 August 2014 at 11:54:57 UTC, Rikki Cattermole wrote:
> Atleast from my experience with this, its dmd thats actually taking the time.
I can't glean this from looking at the code, but are you recompiling the entire program? Web development is a perfect fit for incremental compilation, so when changing one web page's template or view code, you should need to compile only one .d file.
ld and gold seem to support incremental linking, that could help speed things up as well.
|
August 14, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to Vladimir Panteleev | On 15/08/2014 12:47 a.m., Vladimir Panteleev wrote:
> On Thursday, 14 August 2014 at 11:54:57 UTC, Rikki Cattermole wrote:
>> Atleast from my experience with this, its dmd thats actually taking
>> the time.
>
> I can't glean this from looking at the code, but are you recompiling the
> entire program? Web development is a perfect fit for incremental
> compilation, so when changing one web page's template or view code, you
> should need to compile only one .d file.
>
> ld and gold seem to support incremental linking, that could help speed
> things up as well.
Dub automatically handles caching of dependencies such as vibe-d. So they are not rebuilt.
The only things that get recompiled for example is a single code unit. This is defined as being the dependency between a route file and template files (simplified).
The necessary dependencies such as vibe-d then should be added as part of the build process.
Incremental builds of things such as routes and templates are not a good idea. Same goes for data models. It could change the code unit output a little too much. Possibly even cause a corrupt binary from missing symbols ext.
But in this case none of this matters much. Pretty much I'm only testing against a single route module. No dependencies outside phobos and vibe-d.
|
August 14, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to Rikki Cattermole | On Thursday, 14 August 2014 at 12:56:10 UTC, Rikki Cattermole wrote:
> On 15/08/2014 12:47 a.m., Vladimir Panteleev wrote:
>> On Thursday, 14 August 2014 at 11:54:57 UTC, Rikki Cattermole wrote:
>>> Atleast from my experience with this, its dmd thats actually taking
>>> the time.
>>
>> I can't glean this from looking at the code, but are you recompiling the
>> entire program? Web development is a perfect fit for incremental
>> compilation, so when changing one web page's template or view code, you
>> should need to compile only one .d file.
>>
>> ld and gold seem to support incremental linking, that could help speed
>> things up as well.
>
> Dub automatically handles caching of dependencies such as vibe-d. So they are not rebuilt.
>
> The only things that get recompiled for example is a single code unit. This is defined as being the dependency between a route file and template files (simplified).
>
> The necessary dependencies such as vibe-d then should be added as part of the build process.
>
> Incremental builds of things such as routes and templates are not a good idea. Same goes for data models. It could change the code unit output a little too much. Possibly even cause a corrupt binary from missing symbols ext.
>
> But in this case none of this matters much. Pretty much I'm only testing against a single route module. No dependencies outside phobos and vibe-d.
My experience with these sort of things suggests that it'll be the linker taking the time. Dynamic libraries are the solution.
Dub needs proper support for dynamic library dependencies.
|
August 14, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to Rikki Cattermole | On Thursday, 14 August 2014 at 12:56:10 UTC, Rikki Cattermole wrote: > Dub automatically handles caching of dependencies such as vibe-d. So they are not rebuilt. > > The only things that get recompiled for example is a single code unit. This is defined as being the dependency between a route file and template files (simplified). How many D modules / object files is that? > The necessary dependencies such as vibe-d then should be added as part of the build process. > > Incremental builds of things such as routes and templates are not a good idea. Same goes for data models. It could change the code unit output a little too much. Possibly even cause a corrupt binary from missing symbols ext. That approach sounds much more high-level than what I meant, and I don't see why you'd need to abstract things to that level. I'm not familiar with how dub builds things, but inter-module dependencies are a solved problem, thanks to DMD's -v and -deps switches. Incremental compilation was implemented in older (D1-era) build tools, such as xfbuild, however has never picked up due to DMD bugs with incrementally compiling more than one object file at a time (which might very well be fixed by now). |
August 14, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to John Colvin | On Thu, 14 Aug 2014 14:58:20 +0000, John Colvin wrote:
> My experience with these sort of things suggests that it'll be the linker taking the time. Dynamic libraries are the solution.
>
> Dub needs proper support for dynamic library dependencies.
I'll second that suggestion. Can you run the timings using -c? Obviously you won't get binaries out the end, but it'd help diagnose the problem.
|
August 14, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to Rikki Cattermole | On 2014-08-14 13:54, Rikki Cattermole wrote: > Hello everyone. > As you all may know I've been working on recompiling D for web services > last few weeks. > Its both good news and bad news. > > Good: > Reloading definitely possible. With dependency handling using dub. > > Bad: > Its slow. And not in my code sort of way. Has anyone thought of trying to JIT using LDC. -- /Jacob Carlborg |
August 15, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to Justin Whear | On 15/08/2014 3:09 a.m., Justin Whear wrote:
> On Thu, 14 Aug 2014 14:58:20 +0000, John Colvin wrote:
>
>> My experience with these sort of things suggests that it'll be the
>> linker taking the time. Dynamic libraries are the solution.
>>
>> Dub needs proper support for dynamic library dependencies.
>
> I'll second that suggestion. Can you run the timings using -c?
> Obviously you won't get binaries out the end, but it'd help diagnose the
> problem.
Yes its definitely a lot faster. Around 600ms - 800ms. Even with other applications running.
|
August 15, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to Vladimir Panteleev | On 15/08/2014 2:58 a.m., Vladimir Panteleev wrote: > On Thursday, 14 August 2014 at 12:56:10 UTC, Rikki Cattermole wrote: >> Dub automatically handles caching of dependencies such as vibe-d. So >> they are not rebuilt. >> >> The only things that get recompiled for example is a single code unit. >> This is defined as being the dependency between a route file and >> template files (simplified). > > How many D modules / object files is that? I haven't gone into that, I don't really want to go around modifying dub if I can help it. >> The necessary dependencies such as vibe-d then should be added as part >> of the build process. >> >> Incremental builds of things such as routes and templates are not a >> good idea. Same goes for data models. It could change the code unit >> output a little too much. Possibly even cause a corrupt binary from >> missing symbols ext. > > That approach sounds much more high-level than what I meant, and I don't > see why you'd need to abstract things to that level. I'm not familiar > with how dub builds things, but inter-module dependencies are a solved > problem, thanks to DMD's -v and -deps switches. Incremental compilation > was implemented in older (D1-era) build tools, such as xfbuild, however > has never picked up due to DMD bugs with incrementally compiling more > than one object file at a time (which might very well be fixed by now). |
August 15, 2014 Re: Recompiling D code | ||||
---|---|---|---|---|
| ||||
Posted in reply to Jacob Carlborg | On 15/08/2014 6:06 a.m., Jacob Carlborg wrote:
> On 2014-08-14 13:54, Rikki Cattermole wrote:
>> Hello everyone.
>> As you all may know I've been working on recompiling D for web services
>> last few weeks.
>> Its both good news and bad news.
>>
>> Good:
>> Reloading definitely possible. With dependency handling using dub.
>>
>> Bad:
>> Its slow. And not in my code sort of way.
>
> Has anyone thought of trying to JIT using LDC.
Supposedly the llvm is slower for generation then dmd's backend. Atleast what was said during the SDC DCONF talk.
But I'm all for this, could also use it to cache code that is generated ext. in memory. Less usage on the hdd can only be a good thing.
I'll leave this to somebody who does know this area of things.
|
Copyright © 1999-2021 by the D Language Foundation