> 3) Performance can be improved to (near) native speeds with a JIT
> compiler. But then you might as well as go native to begin with. Why
> wait till runtime to do compilation, when it can be done beforehand?

The point though is that with a JIT, you can transmit source code (or byte code which is smaller in size) over a wire and have it execute natively on a client machine. You cannot do that with native machine code because the client machine is always an unknown target.

> But then again, even if we never do this, it makes no difference to *me*
> -- the current situation is good enough for *me*. The question is
> whether or not we want to D to be better received by enterprises.

Exactly, the *we* part of all this doesn't matter in the slightest, it's what the end user wants that matters. If many potential D users want to hide their code (even if it's trivially hidden), but D won't let them, then they won't use D. It's a very simple equation, but holding on to idealisms will often get in the way of good sense.

We already had one corporate user complain in here about the issue, and for everyone who complains there are dozens more who will say nothing at all and just walk away.

--rt

On Tuesday, 18 December 2012 at 02:17:25 UTC, Walter Bright wrote:
> On 12/17/2012 6:13 PM, Rob T wrote:
>> Your suggestion concerning the use of zip files is a good idea, although you
>> mention the encryption algo is very weak, but is there any reason to use a weak
>> encryption algo, and is there even a reason to bother maintaining compatibility
>> with the common zip format?
>
> Using standard zip tools is a big plus.

Yes, but why limit yourself in this way? I suppose you could provide a choice between different formats, but that's the wrong approach. The compiler should instead be restructured to allow D users to supply their own functionality in the form of user defined plugins, that way you won't have to bother second guessing what people need or don't need, or provide generic one size fits all solutions that no one likes, and you'll gain an army of coders who'll take D into very surprising directions that no one could possibly predict.

Another nice fix would be to separate the CTFE interpreter out of the compiler as a loadable library so it can be used outside of the compiler for embedded D scripting, and possibly even for JIT applications.

I expect there are a few more significant improvements that could be made simply by making the compiler less monolithic and more modularized.

Easier said than done, but it should be done at some point because the advantages are very significant.

--rt

On 12/17/2012 6:40 PM, Rob T wrote:
> On Tuesday, 18 December 2012 at 02:17:25 UTC, Walter Bright wrote:
>> On 12/17/2012 6:13 PM, Rob T wrote:
>>> Your suggestion concerning the use of zip files is a good idea, although you
>>> mention the encryption algo is very weak, but is there any reason to use a weak
>>> encryption algo, and is there even a reason to bother maintaining compatibility
>>> with the common zip format?
>>
>> Using standard zip tools is a big plus.
>
> Yes, but why limit yourself in this way?

The easy answer is look at the problems stemming from dmd using an object file format on Win32 that nobody else uses.

On Tuesday, 18 December 2012 at 02:48:05 UTC, Walter Bright wrote:
>>> Using standard zip tools is a big plus.
>>
>> Yes, but why limit yourself in this way?
>
> The easy answer is look at the problems stemming from dmd using an object file format on Win32 that nobody else uses.

I didn't say dmd should use a format that no one uses. What I did say, is that you should not be limiting the choices people want to make for themselves.

The current approach is a self-limiting approach that is unable to make effective use of the resourcefulness of the D community.

DMD may be open source, but it's a monolithic system that is very unfriendly to extensibility and re-use. Look at this thread of discussion, it is caused by the inability to make effective use out of the compiler in ways should be of absolutely no concern to you.

You should be looking at ways of enabling users to be as free and creative as they can be, and that means you must let them make their own mistakes, not the opposite.

--rt

On Mon, 17 Dec 2012 00:19:51 -0800, deadalnix <deadalnix@gmail.com> wrote:

> On Monday, 17 December 2012 at 08:02:12 UTC, Adam Wilson wrote:
>> With respect to those who hold one ideology above others, trying to impose those ideals on another is a great way to ensure animosity. What a business does with their code is entirely up to them, and I would guess that even Richard Stallman himself would take issue with trying to impose an ideology on another person. What does that mean for D practically? Using a close-to-home example, imagine if Remedy decided that shipping their ENTIRE codebase in .DI files with the product would cause them to give away some new rendering trick that they came up with that nobody else had. And they decided that this was unacceptable. What would they most likely do? Rewrite the project in C++ and tell the D community to kindly pound sand.
>>
>> A license agreement is not enough to stop a thief. And once the new trick makes it into the wild, as long as a competitor can honestly say they had no idea how they got it (and they probably really don't, as they saw it on a legitimate game development website) the hands of the legal system are tied.
>>
>
> But that what I say !
>
> I can't stop myself laughing at people that may think any business can be based on java, PHP or C#. That is a mere dream ! Such technology will simply never get used in companies, because bytecode can be decoded !

I use C# everyday at my job. We have expensive obfuscater's to protect the bytecode. Even then it isn't perfect. But it's good enough. With the metadata model of .NET this isn't a problem for public API's, just tell the obfuscater to ignore everything marked 'public'. However, with DI's being a copy of the plaintext source and NOT bytecode you run the risk of changing the meaning of program in unintended ways. You see, in CIL (.NET bytecode) there are no auto's (var's) or templates (generics) the C# compiler does the work of figuring out what the auto type really should be or what the templates really are is BEFORE it writes out the IL, so later when the obfuscater does its job, there are no templates or auto's for it to deal with.

In D, we don't have this option, you either have plaintext, or you have binary code, there is no intermediate step like CIL. Hence we can't use the obfuscation approach.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/

On Tuesday, 18 December 2012 at 02:48:05 UTC, Walter Bright wrote:
>>> Using standard zip tools is a big plus.
>>
>> Yes, but why limit yourself in this way?
>
> The easy answer is look at the problems stemming from dmd using an object file format on Win32 that nobody else uses.

I definitely didn't say dmd should use a format that no one else uses. In fact I actually said the opposite.

Dmd should not be limiting the choices people want to make for themselves. In other words, it should allow people to use whatever formats they wish to use, and not a format imposed on them, such as was pointed out by yourself with the Win32 object file format.

When I look at what dmd is, it's monolithic and is very unfriendly to extensibility and re-use, so I was suggesting that you look at ways to free dmd from the current set of restraints that are holding back its use and adoption.

My suggestion was that the compiler should be restructured into a re-usable modularized system with built-in user extensibility features. Doing that would be a massive improvement and a very big step forward.

--rt

On 12/17/12 6:11 PM, Rob T wrote:
> On Monday, 17 December 2012 at 22:12:01 UTC, Walter Bright wrote:
>> dmd xx foo.zip
>>
>> is equivalent to:
>>
>> unzip foo
>> dmd xx a.d b/c.d d.obj
>>
>> P.S. I've also wanted to use .zip files as the .lib file format (!),
>> as the various .lib formats have nothing over .zip files.
>
> Wow, I think that's exactly what we could use! It serves multiple
> optional use cases all at once!
>
> Was there a technical reason for you not getting around towards
> implementing, or just a lack of time?

The latter. I wanted to do it in rdmd for ages.

Andrei

On 2012-12-18 01:13, H. S. Teoh wrote:

> The problem is not so much the structure preprocessor -> compiler ->
> assembler -> linker; the problem is that these logical stages have been
> arbitrarily assigned to individual processes residing in their own
> address space, communicating via files (or pipes, whatever it may be).
>
> The fact that they are separate processes is in itself not that big of a
> problem, but the fact that they reside in their own address space is a
> big problem, because you cannot pass any information down the chain
> except through rudimentary OS interfaces like files and pipes. Even that
> wouldn't have been so bad, if it weren't for the fact that user
> interface (in the form of text input / object file format) has also been
> conflated with program interface (the compiler has to produce the input
> to the assembler, in *text*, and the assembler has to produce object
> files that do not encode any direct dependency information because
> that's the standard file format the linker expects).
>
> Now consider if we keep the same stages, but each stage is not a
> separate program but a *library*. The code then might look, in greatly
> simplified form, something like this:
>
> 	import libdmd.compiler;
> 	import libdmd.assembler;
> 	import libdmd.linker;
>
> 	void main(string[] args) {
> 		// typeof(asmCode) is some arbitrarily complex data
> 		// structure encoding assembly code, inter-module
> 		// dependencies, etc.
> 		auto asmCode = compiler.lex(args)
> 			.parse()
> 			.optimize()
> 			.codegen();
>
> 		// Note: no stupid redundant convert to string, parse,
> 		// convert back to internal representation.
> 		auto objectCode = assembler.assemble(asmCode);
>
> 		// Note: linker has direct access to dependency info,
> 		// etc., carried over from asmCode -> objectCode.
> 		auto executable = linker.link(objectCode);
> 		File output(outfile, "w");
> 		executable.generate(output);
> 	}
>
> Note that the types asmCode, objectCode, executable, are arbitrarily
> complex, and may contain lazy-evaluated data structure, references to
> on-disk temporary storage (for large projects you can't hold everything
> in RAM), etc.. Dependency information in asmCode is propagated to
> objectCode, as necessary. The linker has full access to all info the
> compiler has access to, and can perform inter-module optimization, etc.,
> by accessing information available to the *compiler* front-end, not just
> some crippled object file format.
>
> The root of the current nonsense is that perfectly-fine data structures
> are arbitrarily required to be flattened into some kind of intermediate
> form, written to some file (or sent down some pipe), often with loss of
> information, then read from the other end, interpreted, and
> reconstituted into other data structures (with incomplete info), then
> processed. In many cases, information that didn't make it through the
> channel has to be reconstructed (often imperfectly), and then used. Most
> of these steps are redundant. If the compiler data structures were
> already directly available in the first place, none of this baroque
> dance is necessary.

I couldn't agree more.

-- 
/Jacob Carlborg

On 2012-12-17 00:09, Walter Bright wrote:

> Figure out the cases where it happens and fix those cases.

How is it supposed to work? Could there be some issue with the dependency tracker that should otherwise have indicated that more modules should have been recompiled.

-- 
/Jacob Carlborg

On 2012-12-17 23:12, Walter Bright wrote:

> I have toyed with the idea many times, however, of having dmd support
> zip files. Zip files can contain an arbitrary file hierarchy, with
> individual files in compressed, encrypted, or plaintext at the selection
> of the zip builder. An entire project, or library, or collection of
> source modules can be distributed as a zip file, and could be compiled
> with nothing more than:

I think that a package manager should handle this. Example:

https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

Yes I can change the orbfiles to be written in D.

-- 
/Jacob Carlborg