Hi, I'm learning D, and trying to develop a website using vibe.d.

I'm using Digital Ocean's VPS server, with 1GB memory. Many a time when I do `dub`, the memory consumption soars and forces dmd to quit, sometimes even causes the remote machine droplet to freeze.

I understand that due to template and static analysis in the compiler, the memory usage is huge for template heavy code. But I heard that there was effort going on and progress is already made in recent releases.

My question is, is there a plan for this effort? What is the general prediction of an ideal D compiler, some hundreds of MBs or less? I'd like to switch to 512MB VPS droplet, as it's much cheaper. When (which release) would the memory cost be in reasonable level?

As an alternative, I can build the project locally and rsync the binaries to the VPS, but a simple binary in vibe.d yields a ~12MB file, it's somewhat slow to upload it. So I wonder can we refer vibe.d/phobos as a dynamic library?

Thanks,
Puming.

Puming:

> My question is, is there a plan for this effort?

There are plans to reduce memory used by CTFE, but template bloat is less easy to face.


> What is the  general prediction of an ideal D compiler,
> some hundreds of MBs or less?

Non one knows, I think.


> but a simple binary in vibe.d yields a ~12MB file,
> it's somewhat slow to upload it.

There are programs to compress binaries a lot.


> So I wonder can we refer vibe.d/phobos as a dynamic library?

This is being worked on in the latest D releases.

Bye,
bearophile

On Wed, Dec 04, 2013 at 05:12:26PM +0100, bearophile wrote:
> Puming:
[...]
> >but a simple binary in vibe.d yields a ~12MB file,
> >it's somewhat slow to upload it.
> 
> There are programs to compress binaries a lot.

Take a look at UPX: http://upx.sourceforge.net/

In my own projects, a typical D program is easily 5-10MB, or, if it's template-heavy, 15-20MB. But if I run 'strip' on the executable and then run 'upx' on it, I can quite easily get it down to less than 1MB.

You won't be able to debug it, of course, since there would be no symbols, etc., left, but if you're deploying it to your production server anyway, presumably you've already debugged it beforehand, so this shouldn't be a problem.


T

-- 
Claiming that your operating system is the best in the world because more people use it is like saying McDonalds makes the best food in the world. -- Carl B. Constantine

While in general one may expect CTFE/template memory consumption to drop considerably, in vibe.d case it is unlikely to make that much of an impact. vibe.d does really lot of computations during compile-time, for example, all Diet templates get loaded and processed during that phase. I can't imagine optimizations that will make possible to fit compilation of any vibe.d project of notable size in 512Mb memory. To be honest, I'd hate to do any DMD + vibe.d development on anything less than 4Gb of RAM.

In that sense I think rsync'ing binary from development box is most reasonable approach in the long term. There is already working dynamic Phobos library for Linux, using it + stripping the binaries shall reduce their sizes quite a lot. Officially shared Phobos is still not very tested but I have not had any issues with it so far.

On Wednesday, 4 December 2013 at 16:12:28 UTC, bearophile wrote:
> Puming:
>
>> My question is, is there a plan for this effort?
>
> There are plans to reduce memory used by CTFE, but template bloat is less easy to face.

Vibe's diet templates would benefit from this I guess.

>
>> What is the  general prediction of an ideal D compiler,
>> some hundreds of MBs or less?
>
> Non one knows, I think.

That is a pity.

>
>
>> but a simple binary in vibe.d yields a ~12MB file,
>> it's somewhat slow to upload it.
>
> There are programs to compress binaries a lot.
>

Thanks for the tip, now I'll just need a little script to do the compress/rsync/decompress automation.

Did you mean tar/zip or some specialized tool? I found that tar makes it to ~2.5MB, which is fairly good

>
>> So I wonder can we refer vibe.d/phobos as a dynamic library?
>
> This is being worked on in the latest D releases.

That is great news. Would it be out in 2.065? ore 2.066?

>
> Bye,
> bearophile

Thanks bearophile,

So hopefully the situation might be much better earlier next year :-)

On Wednesday, 4 December 2013 at 16:26:43 UTC, H. S. Teoh wrote:
> On Wed, Dec 04, 2013 at 05:12:26PM +0100, bearophile wrote:
>> Puming:
> [...]
>> >but a simple binary in vibe.d yields a ~12MB file,
>> >it's somewhat slow to upload it.
>> 
>> There are programs to compress binaries a lot.
>
> Take a look at UPX: http://upx.sourceforge.net/
>
> In my own projects, a typical D program is easily 5-10MB, or, if it's
> template-heavy, 15-20MB. But if I run 'strip' on the executable and then
> run 'upx' on it, I can quite easily get it down to less than 1MB.
>
> You won't be able to debug it, of course, since there would be no
> symbols, etc., left, but if you're deploying it to your production
> server anyway, presumably you've already debugged it beforehand, so this
> shouldn't be a problem.
>
>
> T

Thanks, I'll try that :)

Puming:

> Did you mean tar/zip or some specialized tool?

I mean specialized programs that compile executables and keep them executables.


> I found that tar makes it to ~2.5MB, which is fairly good

I think that tar is not a compressor, it's a packager.


> That is great news. Would it be out in 2.065? ore 2.066?

It's already partially present. The next D release (2.065) will be about bug fixing, especially ICEs.

Bye,
bearophile

On Wednesday, 4 December 2013 at 16:28:07 UTC, Dicebot wrote:
> While in general one may expect CTFE/template memory consumption to drop considerably, in vibe.d case it is unlikely to make that much of an impact. vibe.d does really lot of computations during compile-time, for example, all Diet templates get loaded and processed during that phase. I can't imagine optimizations that will make possible to fit compilation of any vibe.d project of notable size in 512Mb memory. To be honest, I'd hate to do any DMD + vibe.d development on anything less than 4Gb of RAM.

Other than diet, is there many other CTFE/template heavy parts?

I hope there would be a non-CTFE version of diet template, similar to regex/CTRegex. If we use the non-CTFE version in dev mode, we would have a better memory & compile time turnaround in dev mode. And in the release mode the diet templates would be optimized with CTFE. Is that a viable approach?

>
> In that sense I think rsync'ing binary from development box is most reasonable approach in the long term. There is already working dynamic Phobos library for Linux, using it + stripping the binaries shall reduce their sizes quite a lot. Officially shared Phobos is still not very tested but I have not had any issues with it so far.

Yes, after some trying and struggle, I came to the same conclusion. Will try the compression tools mentioned by Teoh :)

Thanks,

Puming.

On Wednesday, 4 December 2013 at 16:38:28 UTC, bearophile wrote:
> Puming:
>
>> Did you mean tar/zip or some specialized tool?
>
> I mean specialized programs that compile executables and keep them executables.
>
>
>> I found that tar makes it to ~2.5MB, which is fairly good
>
> I think that tar is not a compressor, it's a packager.

Sorry, I mean gzip.

>
>
>> That is great news. Would it be out in 2.065? ore 2.066?
>
> It's already partially present. The next D release (2.065) will be about bug fixing, especially ICEs.

Thanks, I'll try that :) are there docs about how to use phobos as a shared lib?

>
> Bye,
> bearophile

On Wednesday, 4 December 2013 at 16:41:14 UTC, Puming wrote:
> Other than diet, is there many other CTFE/template heavy parts?

Also vibe.http.rest and vibe.http.form create really lot of template instances because of heavy usage of reflection + code generation. This is one of domains though that can potentially improve a lot by better compiler implementation,

> I hope there would be a non-CTFE version of diet template, similar to regex/CTRegex. If we use the non-CTFE version in dev mode, we would have a better memory & compile time turnaround in dev mode. And in the release mode the diet templates would be optimized with CTFE. Is that a viable approach?

That should be possible and relatively simple (because of CTFE nature a lot of code can be reused). It may considerably impact Diet performance but I can see desire for that in development mode as project size increases. As usual, it is mostly matter of manpower - vibe.d has only one persistent contributor right now and probably 5-8 guys doing occasional pull request, definitely not enough for project of that scale.