January 05, 2007
Frits van Bommel wrote:
> Alexander Panek wrote:
>> TPJ wrote:

>>> 2) From FAQ: Since D can call C functions, any GUI library with a
>>> C interface is accessible from D. Does it mean that I can access
>>> *any* C library from D? If so, it a great feature!
>> 
>> Yes, any C library. 
> 
> You need to convert C headers to D import modules to use them though.
> It shouldn't be too hard unless too much preprocessor tricks are
> used, but it might be a bit of work for large files.

It's actually a lot easier than it sounds. :-)

And, there's no use converting everything, just convert the things in a C header that your program _actually_ uses! (Most people feel an inexplicable urge to convert the whole header file "just in case", or "later I'm going to need it anyway". Don't.)

Also, if the header file looks like "an exuberant show-off of preprocessor wizardry", then you might want to run it thorugh the preprocessor, and compare it with the original.
January 05, 2007
Walter Bright wrote:
> TPJ wrote:
>> 1) In order to run a program written in D I have to compile it on the target OS.
> 
> I haven't talked about this much, but it is technically feasible to create a D runtime that enables "compile once, run everywhere". It is not necessary to have a VM to do it.

How would you implement a garbage collector in an architecture-independent way?

Or are you talking about using version(arch_1) to version(arch_N)? [1]
Because I'm pretty sure the garbage collector has to know some architecture-specific things. Start of stack, direction of stack, location of static data, threading API (if any), that sort of thing.
So you'd need to perform at least _some_ porting to get it to run on a new architecture, if only to tell the GC what its properties are.

Though I guess stack direction could be experimentally determined at runtime/install time, as could endianness... Threading API is going to be hard to determine in that way, I'd think.


[1] or "version(StackGrowsUp), version(BigEndian), version(HasFeatureX), version(HasFeatureY), ..." or something like that.
January 05, 2007
Frits van Bommel wrote:
> Walter Bright wrote:
>> TPJ wrote:
>>> 1) In order to run a program written in D I have to compile it on the target OS.
>>
>> I haven't talked about this much, but it is technically feasible to create a D runtime that enables "compile once, run everywhere". It is not necessary to have a VM to do it.
> 
> How would you implement a garbage collector in an architecture-independent way?

I don't think you would.  Rather, I think you'd have a separate runtime library for each platform that handled all the necessary back-end tasks, but which then called through to the same user-level D program.  This would allow you to deal with platform-specific startup requirements, memory allocation scheme, executable file layout, etc.  So long as the ABI or whatever were sufficiently strict regarding how exception handling occurs and a few other details, I think the compiled user code would be portable across OSes.  But I'm not sure how this would interact with things like calling C libraries built by other compilers.  For example, DMD/Linux appears to pad doubles differently than DMD/Win32, so how would the D user code know what to do re: pushing the values onto the stack?


Sean
January 05, 2007
Walter Bright wrote:
> TPJ wrote:
> 
>> 1) In order to run a program written in D I have to compile it on the target OS.
> 
> I haven't talked about this much, but it is technically feasible to create a D runtime that enables "compile once, run everywhere". It is not necessary to have a VM to do it.

1. To run on any OS on one specific processor, either those OSes have to have different entry points to the "exe", or there has to be a way of probing the OS within "common" code, the result of which would cause a jump to an OS specific routine.

2. To run on different processor architectures, (most probably with some of the same OSes), the worst case scenario would be that the same OS with the same jump target gets invocated on a different processor. This would mean that one (er, Walter) would have to invent a series of binary instructions that would not cause an (unwanted) exception on any of the processors, would not have any harmful side effects, and would eventually result in a jump to some processor specific version of the code.

Both of these seem like formidable tasks. OTOH, there's nothing that directly says they'd be theoretically impossible. So, in practice, the fewer processors we want to support, the less likely this is to be impossible. But, the more processors, specifically: for each additional processor (of a different architecture philosophy) we want to support, the harder it gets. And IMHO exponentially. (In English: pretty hard, and it gets _vastly_ harder for each single new processor!)

----

Not right now having the additional patience to think this entirely through, I got to thinking about the following:

At first sight ("hearing" ?), it sounds too good to be true. Then one starts to suspect that either Walter has figured out a runtime gimmick that does it (thus rendering this (btw, patentable idea) independent of D), or it is some kind of result of the specific way D and its source code (more to the point, syntax domain) relate to the semantics of the source code, taken as a whole.

----

If there's any remotest chance that this may be patentable, I DEFINITELY don't want to hear a word about it for the next 18 months!

If not, I'd kill to hear more! :-)
January 06, 2007
Sean Kelly wrote:
> Frits van Bommel wrote:
>> Walter Bright wrote:
>>> TPJ wrote:
>>>> 1) In order to run a program written in D I have to compile it on the target OS.
>>>
>>> I haven't talked about this much, but it is technically feasible to create a D runtime that enables "compile once, run everywhere". It is not necessary to have a VM to do it.
>>
>> How would you implement a garbage collector in an architecture-independent way?
> 
> I don't think you would.  Rather, I think you'd have a separate runtime library for each platform that handled all the necessary back-end tasks, but which then called through to the same user-level D program.  This would allow you to deal with platform-specific startup requirements, memory allocation scheme, executable file layout, etc.  So long as the ABI or whatever were sufficiently strict regarding how exception handling occurs and a few other details, I think the compiled user code would be portable across OSes.  But I'm not sure how this would interact with things like calling C libraries built by other compilers.  For example, DMD/Linux appears to pad doubles differently than DMD/Win32, so how would the D user code know what to do re: pushing the values onto the stack?

Oh, I must've read it wrong. I thought this was about a runtime that would automagically compile on any platform... <blush>

That's different from "independent of OS" though, which seems to be what you're thinking of. ("run everywhere" does not equal "run on any x86". Not yet anyway ;) )

So this is about a single compiled form that'll work on any supported platform. Well, then it comes down to your definition of VM and 'compiled'.
You'd need some representation of the program that's independent of the platform. There are two basic forms of this: some form of bytecode or source code.
Since it's "compile once", I find it reasonable to assume there should be some compiling involved ;). That would indicate byte code. Unless you would care to argue that Winzip is a compiler? :P

So then you have several choices:
- Interpret the bytecode. This is basically the simplest VM possible. (Is this what Python does? At the very least, I'm pretty sure it used to do this)
- Perform another compilation at runtime: bytecode to machine code. This would basically be a JIT-compiling VM.
- Perform above compilation[1] at install time. This is the only option I see that doesn't require a VM... (For Java, this is possible through the Java bytecode frontend for GCC IIRC)
- {{ Did I miss anything? }}
- Some mixture of the above (such as: interpret uncommon parts, runtime-compile the most-ran code)

All options but the first require that the user environment has some form of compiler though, which might not be optimal either.


[1]: For this purpose, unpacking the source and performing a normal compilation would also work. It might take longer though.
January 06, 2007
Frits van Bommel wrote:
> Sean Kelly wrote:
>> Frits van Bommel wrote:
>>> Walter Bright wrote:
>>>> TPJ wrote:
>>>>> 1) In order to run a program written in D I have to compile it on the target OS.
>>>>
>>>> I haven't talked about this much, but it is technically feasible to create a D runtime that enables "compile once, run everywhere". It is not necessary to have a VM to do it.
>>>
>>> How would you implement a garbage collector in an architecture-independent way?
>>
>> I don't think you would.  Rather, I think you'd have a separate runtime library for each platform that handled all the necessary back-end tasks, but which then called through to the same user-level D program.  This would allow you to deal with platform-specific startup requirements, memory allocation scheme, executable file layout, etc.  So long as the ABI or whatever were sufficiently strict regarding how exception handling occurs and a few other details, I think the compiled user code would be portable across OSes.  But I'm not sure how this would interact with things like calling C libraries built by other compilers.  For example, DMD/Linux appears to pad doubles differently than DMD/Win32, so how would the D user code know what to do re: pushing the values onto the stack?
> 
> Oh, I must've read it wrong. I thought this was about a runtime that would automagically compile on any platform... <blush>
> 
> That's different from "independent of OS" though, which seems to be what you're thinking of. ("run everywhere" does not equal "run on any x86". Not yet anyway ;) )

True :)  But it still means quite a bit of portability.  And I could have misunderstood as well.  I simply couldn't imagine a binary app compiled for, say, x86 (one instruction set, big-endian) actually working on PPC (another instruction set, little-endian) without some sort of JIT compile step.

> So this is about a single compiled form that'll work on any supported platform. Well, then it comes down to your definition of VM and 'compiled'.
> You'd need some representation of the program that's independent of the platform. There are two basic forms of this: some form of bytecode or source code.

I'm far from an expert here, but I think the bulk of the actual binary program is probably consistent across OSes for the same architecture. The differences are probably in the executable startup code, the thread code, memory allocation, some portions of garbage collection, etc, all of which could theoretically be abstracted to a runtime library (in a sense, they already are: Phobos).  The issue would be figuring out how to go from the command prompt to running the proper version of everything.  Would a link step be required first?  Would the runtime library actually contain multiple versions of the code for different OSes and call the appropriate one dynamically?  Or perhaps the runtime could be in a dynamic lib that's loaded by some startup code linked into the app?  Note that none of these are Virtual Machine situations because the code is running natively in all cases.


Sean
January 06, 2007
Sean Kelly wrote:
> Frits van Bommel wrote:
>> Oh, I must've read it wrong. I thought this was about a runtime that would automagically compile on any platform... <blush>
>>
>> That's different from "independent of OS" though, which seems to be what you're thinking of. ("run everywhere" does not equal "run on any x86". Not yet anyway ;) )
> 
> True :)  But it still means quite a bit of portability.  And I could have misunderstood as well.  I simply couldn't imagine a binary app compiled for, say, x86 (one instruction set, big-endian) actually working on PPC (another instruction set, little-endian) without some sort of JIT compile step.

x86 is little-endian, PPC can IIRC be set to either, but is typically (i.e. on PPC Macs) set to big-endian.

>> So this is about a single compiled form that'll work on any supported platform. Well, then it comes down to your definition of VM and 'compiled'.
>> You'd need some representation of the program that's independent of the platform. There are two basic forms of this: some form of bytecode or source code.
> 
> I'm far from an expert here, but I think the bulk of the actual binary program is probably consistent across OSes for the same architecture. 

I was talking about multiple architectures here. Sorry if that wasn't clear enough.

> The differences are probably in the executable startup code, the thread code, memory allocation, some portions of garbage collection, etc, all of which could theoretically be abstracted to a runtime library (in a sense, they already are: Phobos).  The issue would be figuring out how to go from the command prompt to running the proper version of everything.  Would a link step be required first?  Would the runtime library actually contain multiple versions of the code for different OSes and call the appropriate one dynamically?  Or perhaps the runtime could be in a dynamic lib that's loaded by some startup code linked into the app?  Note that none of these are Virtual Machine situations because the code is running natively in all cases.

Before you consider how the runtime is linked to the app, first consider how the app starts in the first place. Let's say the bare minimum is x86 Linux and Windows compatibility.

I don't think there's currently any way to have it run directly on Windows and Linux simply because there's AFAIK no executable format that's supported by both OSs (ELF is supported on multiple OSs, but Windows doesn't support it). Furthermore, all executable formats I know of (except for plain binary, which AFAIK Linux doesn't support[1]) have some (unique) signature in the first few bytes, so a file can't be multiple of these formats at the same time.

So you'd need to either using some sort of custom loader binary for each OS or modify OSs to support a common binary format (i.e. a custom loader, but integrated into the OS).


[1]: Windows "supports" these for programs that fit in 64k: .com files are plain binaries. They're not really useful for anything but backwards compatibility though.
January 06, 2007
On Fri, 05 Jan 2007 11:54:09 -0800, Walter Bright wrote:

> TPJ wrote:
>> 1) In order to run a program written in D I have to compile it on the target OS.
> 
> I haven't talked about this much, but it is technically feasible to create a D runtime that enables "compile once, run everywhere". It is not necessary to have a VM to do it.


Isn't this something that LLVM can be used for?

http://llvm.org/

-JJR
January 06, 2007
Frits van Bommel wrote:
> Sean Kelly wrote:
>> Frits van Bommel wrote:
>>> Oh, I must've read it wrong. I thought this was about a runtime that would automagically compile on any platform... <blush>
>>>
>>> That's different from "independent of OS" though, which seems to be what you're thinking of. ("run everywhere" does not equal "run on any x86". Not yet anyway ;) )
>>
>> True :)  But it still means quite a bit of portability.  And I could have misunderstood as well.  I simply couldn't imagine a binary app compiled for, say, x86 (one instruction set, big-endian) actually working on PPC (another instruction set, little-endian) without some sort of JIT compile step.
> 
> x86 is little-endian, PPC can IIRC be set to either, but is typically (i.e. on PPC Macs) set to big-endian.

Oops, I had it backwards, you're right.  A bigger issue is 64 vs. 32-bit code.  At some point, the architectural differences are simply too great for binary portability of even user code.

>> The differences are probably in the executable startup code, the thread code, memory allocation, some portions of garbage collection, etc, all of which could theoretically be abstracted to a runtime library (in a sense, they already are: Phobos).  The issue would be figuring out how to go from the command prompt to running the proper version of everything.  Would a link step be required first?  Would the runtime library actually contain multiple versions of the code for different OSes and call the appropriate one dynamically?  Or perhaps the runtime could be in a dynamic lib that's loaded by some startup code linked into the app?  Note that none of these are Virtual Machine situations because the code is running natively in all cases.
> 
> Before you consider how the runtime is linked to the app, first consider how the app starts in the first place. Let's say the bare minimum is x86 Linux and Windows compatibility.
> 
> I don't think there's currently any way to have it run directly on Windows and Linux simply because there's AFAIK no executable format that's supported by both OSs (ELF is supported on multiple OSs, but Windows doesn't support it). Furthermore, all executable formats I know of (except for plain binary, which AFAIK Linux doesn't support[1]) have some (unique) signature in the first few bytes, so a file can't be multiple of these formats at the same time.

Hrm, right.  So there would probably have to be either a configuration step before the initial run or some sort of translation or linking that took place.  I'll be curious to see what comes of this.  If Walter can sort it out in a nice clean manner, it would be a killer feature.


Sean
January 06, 2007
Sean Kelly wrote:
> Walter Bright wrote:
>> TPJ wrote:
>>> 1) In order to run a program written in D I have to compile it on the target OS.
>>
>> I haven't talked about this much, but it is technically feasible to create a D runtime that enables "compile once, run everywhere". It is not necessary to have a VM to do it.
> 
> This would still be limited to a specific architecture, wouldn't it?

Nope. But it would require one to code in a portable subset of D.

> ie. "This app runs on x86, regardless of OS."  Given that there are really very few architectures in the marketplace though, this is still a killer feature.
> 
> 
> Sean