On Sat, 27 Apr 2002 20:58:32 +0400, "Pavel Minayev" <evilone@omen.ru> wrote:

>"Burton Radons" <loth@users.sourceforge.net> wrote in message news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg@4ax.com...
>
>> What do you mean by bloat?  If the parser's not linked in, it takes no space.  The parser part of the compiler shouldn't take much space at all even when it is linked in.
>
>Parser of what? The program is already compiled to machine code!

Parser of D.  You're missing what this would provide, simply: put the parser and compiler in a library and provide access to it.  That's it, nothing weird about the language or anything, just a refactoring and an interface on the D side.

[snip]
>> The only thing I need from here is a compiler and parser, actually, for in-game entities with code.  But I'll let it simmer until I'm actually at the point that I could use it.
>
>Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed a standard interpreter module could be done. Seems like a worthy idea.

Well, let's see.  Of the 1930kb of C code in my current project, 1230kb is actual implementation and 700.39kb is devoted to nothing more than interfacing Python; that's 36% of the body.  The Python code itself sums up to only 375kb.  The balance is all wacky, definitely not acceptable, but very typical.

With runtime access to the D compiler and full introspection, on the other hand, getting the compiler to interact is pretty trivial, and anything I do will come with the guarantee that it will run as fast as my machine and compiler can make it.

>... wanna try?

Been there, done that, have many, many T-shirts.  If I were to do this I would write a compiler to machine code; apart from floating point they're not particularly hard to do dumbly and are easier to interact with the rest of the system.  But then there's a lot of duplication of effort; it'd be like using a Tonka truck to dig a swimming pool when you have the keys to the backhoe in your pocket.

"Burton Radons" <loth@users.sourceforge.net> wrote in message news:br5qcuo879plqcnr60avcraeamqu318phr@4ax.com...

> >Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed a standard interpreter module could be done. Seems like a worthy idea.
>
> Well, let's see.  Of the 1930kb of C code in my current project, 1230kb is actual implementation and 700.39kb is devoted to nothing more than interfacing Python; that's 36% of the body.  The Python code itself sums up to only 375kb.  The balance is all wacky, definitely not acceptable, but very typical.
>
> With runtime access to the D compiler and full introspection, on the other hand, getting the compiler to interact is pretty trivial, and anything I do will come with the guarantee that it will run as fast as my machine and compiler can make it.

This sounds like a special kind of computer language, quite different by the very concept from traditional all-purpose languages. Personally, I wouldn't like D to turn into something like that. It might be a useful tool, but not for everybody; not for me, at least.

"Burton Radons" <loth@users.sourceforge.net> wrote in message news:br5qcuo879plqcnr60avcraeamqu318phr@4ax.com...
> On Sat, 27 Apr 2002 20:58:32 +0400, "Pavel Minayev" <evilone@omen.ru> wrote:
>
> >Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed a standard interpreter module could be done. Seems like a worthy idea.
>
> With runtime access to the D compiler and full introspection, on the other hand, getting the compiler to interact is pretty trivial, and anything I do will come with the guarantee that it will run as fast as my machine and compiler can make it.

Script languages usually have a set of problems to solve that differ from the problems of compiled languages.  For in-game code, there is an additional difficulty: keeping the game player safe.

Quake C and UnrealScript solve this problem.  It's not possible to harm the gamer's computer with these languages.  He/she can download mods (or have them automatically downloaded during on-line play) with a high level of comfort.  If in-game code is in a native-compiled language, then you've got an opportunity to spread worms and viruses and cause real harm.

Do you *really* want to subject your audience to malware like that?  Just so that you have the convenience of coding the core game engine and logic in the same language?

And I might even suggest another alternative: Java.  The Java sandbox works, and JIT compilation works.  You can use a Java native-code compiler for game engine performance, and load Java .class files for mods, which are then JIT-compiled.

(Having said that, I know id Software experimented using Java for Quake 3 game logic, and performance considerations caused them to shelve it. Instead, Quake 3 builds in a C compiler; game mods are programmed in a safe C subset, which is compiled at load time to actual native machine code.)

--
Richard Krehbiel, Arlington, VA, USA
rich@kastle.com (work) or krehbiel3@comcast.net  (personal)

"Richard Krehbiel" <rich@kastle.com> wrote in message news:aajmkt$qtu$1@digitaldaemon.com...
> (Having said that, I know id Software experimented using Java for Quake 3 game logic, and performance considerations caused them to shelve it. Instead, Quake 3 builds in a C compiler; game mods are programmed in a
safe
> C subset, which is compiled at load time to actual native machine code.)

That's an interesting idea. Does this C subset not allow pointers?

Walter wrote:
> "Richard Krehbiel" <rich@kastle.com> wrote in message
> news:aajmkt$qtu$1@digitaldaemon.com...
> 
>>(Having said that, I know id Software experimented using Java for Quake 3
>>game logic, and performance considerations caused them to shelve it.
>>Instead, Quake 3 builds in a C compiler; game mods are programmed in a
> 
> safe
> 
>>C subset, which is compiled at load time to actual native machine code.)
> 
> 
> That's an interesting idea. Does this C subset not allow pointers?

Quake 1 used QuakeC, which didn't allow pointers (and
actually didn't allow a lot of things -- I wouldn't
call it a C subset in fact; it was a vaguely C looking
scripting language). If I remember rightly it was
compiled to a tokenized or "bytecode"[1] form at game
build time (or maybe at load time, it's been a while).

Quake 2 used native DLLs.

According to 5 minutes of Google research, it looks to
me like Quake 3 also uses native DLLs:

http://www.planetquake.com/code3arena/tutorials/tutorial2.shtml

I believe they decided to leave security as a "caveat
player" thing in favor of performance and the ready
availability of tools.

-Russell B

[1] Anyone else hate this term as much as I do?

Burton Radons <loth@users.sourceforge.net> wrote in news:br5qcuo879plqcnr60avcraeamqu318phr@4ax.com:

[snip]
> Parser of D.  You're missing what this would provide, simply: put the parser and compiler in a library and provide access to it.  That's it, nothing weird about the language or anything, just a refactoring and an interface on the D side.

You could use this to build a scripting engine into you program and
it would be very fast.  However there is a security issue with it
depending on what you want to use it for.  For example with games
you may want to auto download game mods much like Unreal Tournament
does.  How do you prevent the mod writer from doing something malicious?
You can certainly filter the functions that downloaded script can call
but the main issue is that D still allows pointer manipulation. Given
a pointer a clever coder can gain access to system level functions.

So you need a more sandbox friendly version of D which ends up being a different language anyway.

I do like the idea of having the compiler modules as libraries.  I talked about something similar in a different thread here but I had different reasons for wanting it.  Building the compiler in a modular way allows user defined processing to done between the compilation stages.

[snip]

> Well, let's see.  Of the 1930kb of C code in my current project, 1230kb is actual implementation and 700.39kb is devoted to nothing more than interfacing Python; that's 36% of the body.  The Python code itself sums up to only 375kb.  The balance is all wacky, definitely not acceptable, but very typical.

I've done a lot of C++/Python integration too.  It's true that code to
hook the two together is a pain.  I eventually invented my own IDL
language.
I would run it through a script that would auto generate 99.9% of the
hook
code for me.  Of course the runtime bulk is still there.

> Been there, done that, have many, many T-shirts.  If I were to do this I would write a compiler to machine code; apart from floating point they're not particularly hard to do dumbly and are easier to interact with the rest of the system.  But then there's a lot of duplication of effort; it'd be like using a Tonka truck to dig a swimming pool when you have the keys to the backhoe in your pocket.

Still I think a separate system is needed for this type of application. What I've really looking for is an embedable JIT compiler that also takes care of sandbox issues.

Patrick Down wrote:

> Burton Radons <loth@users.sourceforge.net> wrote in news:br5qcuo879plqcnr60avcraeamqu318phr@4ax.com:
>
> [snip]
> > Parser of D.  You're missing what this would provide, simply: put the parser and compiler in a library and provide access to it.  That's it, nothing weird about the language or anything, just a refactoring and an interface on the D side.
>
> You could use this to build a scripting engine into you program and
> it would be very fast.  However there is a security issue with it
> depending on what you want to use it for.  For example with games
> you may want to auto download game mods much like Unreal Tournament
> does.  How do you prevent the mod writer from doing something malicious?
> You can certainly filter the functions that downloaded script can call
> but the main issue is that D still allows pointer manipulation. Given
> a pointer a clever coder can gain access to system level functions.
>
> So you need a more sandbox friendly version of D which ends up being a different language anyway.

Hey, it can be D--   :)

D, without pointers, without asm, with only access to a very carefully limited subset of the library, could potentially be a very good safe scripting language.  I proposed (something very like) this back on 8/20/01 in "A Java-style sandbox without a Virtual Machine?", but there was a lot of confusion on what I meant by it :(

I did get a pointer to Bochs from that thread, which was a real win :)



Even if Walter doesn't feel like supporting it...we could do it.

--
The Villagers are Online! villagersonline.com

.[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ]
.[ (a version.of(English).(precise.more)) is(possible) ]
?[ you want.to(help(develop(it))) ]

"Russell Borogove" <kaleja@estarcion.com> wrote in message news:3CCD7F65.20705@estarcion.com...
> Walter wrote:
> > "Richard Krehbiel" <rich@kastle.com> wrote in message news:aajmkt$qtu$1@digitaldaemon.com...
> >
> >>(Having said that, I know id Software experimented using Java for Quake
3
> >>game logic, and performance considerations caused them to shelve it. Instead, Quake 3 builds in a C compiler; game mods are programmed in a
> >
> > safe
> >
> >>C subset, which is compiled at load time to actual native machine code.)
> >
> >
> > That's an interesting idea. Does this C subset not allow pointers?
>
> Quake 1 used QuakeC, which didn't allow pointers (and
> actually didn't allow a lot of things -- I wouldn't
> call it a C subset in fact; it was a vaguely C looking
> scripting language). If I remember rightly it was
> compiled to a tokenized or "bytecode"[1] form at game
> build time (or maybe at load time, it's been a while).
>
> Quake 2 used native DLLs.
>
> According to 5 minutes of Google research, it looks to
> me like Quake 3 also uses native DLLs:
>
> http://www.planetquake.com/code3arena/tutorials/tutorial2.shtml
>

Um, I had forgotten that Quake 3 also allows game mods using .DLLs.

But read the very bottom where it talks about QVMs.

> I believe they decided to leave security as a "caveat player" thing in favor of performance and the ready availability of tools.

Quake 2 was definitely programmed that way; game modes *are* done in native-code .DLLs.

Here's a link that goes into more detail on Quake 3's mod system:

http://firingsquad.gamers.com/hardware/q3adrivers/default.asp

(I dimly recall John Carmack describing his QVM to native code JIT-compiler, but I can't find any links...)

--
Richard Krehbiel, Arlington, VA, USA
rich@kastle.com (work) or krehbiel3@comcast.net  (personal)

"Pavel Minayev" <evilone@omen.ru> wrote in news:aael3n$18ql$1 @digitaldaemon.com:

> 
> Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed a standard interpreter module could be done. Seems like a worthy idea.
> 
> ... wanna try?

I like the idea of a standard interpreter module although as I said in a different thread here what I really want is a embeddable JIT compiler module.  But in the mean time I'll take a regular old script interpreter.  I've thought about doing this for D if I have some spare time.

I've had a lot of success building applications that mix C++ and Python. I usually let the Python code handle the GUI stuff like dialogs and it also acts as a glue layer between the C++ modules.

There are a couple of problems.

1) The amount of interface code that is needed to tie the interpreter to the C++ code.  I have scripts that will automatically generate a lot of the code.  But the interface code is still bulky.

2) Python's objects are garbage collected via reference counting.  Any C++ code that references one of these objects had to manually deal with the reference count.  And C++ object that is reference from Python has to supply a reference count for the interpreter.

3) This is less of a problem and more of a personal preference on my
part.
I like Python's power I still don't like it's syntax.  I just can't
handle using indention as the only indicator of code blocks. :-)

Some of these problems could be solved with a scripting language interpreter written in D that uses extension modules written in D.

Number 2 is easy because the interpreter will use D's garbage collector.  All the memory management problems go away because the interpreter and D will use common memory management.

Number 1 is not quite so easy to solve.  Having the memory management issue solved does away with some of the interface code but there are still issues with marshalling function parameters between the interpreter and D code.  There are two sides to this.  Calling D functions from the script and calling script functions from D.

Having a script language construct like the "native" directive in Unreal
Script would allow the interpreter to automatically generate stack frames
for calls from script into D.  The interpreter could also use reflection
to find the function pointers in a module.  However, there are issues
with
controlling which functions you want exported to the script interpreter.
If D allowed user defined attribute tagging of classes and functions like
C# then this problem could be solved by adding a "scriptexport" attribute
to things in D that you wanted to export to the interpreter.  Another way
is just to hand a list of exported functions and classes to the
interpreter
when it is initialized.

Calling script functions from D is still messy but manageable with some helper functions.

Russ Lewis <spamhole-2001-07-16@deming-os.org> wrote in news:3CCD9200.64D5035D@deming-os.org:

> Patrick Down wrote:
> 
>> Burton Radons <loth@users.sourceforge.net> wrote in news:br5qcuo879plqcnr60avcraeamqu318phr@4ax.com:
>>
>> [snip]
>> > Parser of D.  You're missing what this would provide, simply: put the parser and compiler in a library and provide access to it. That's it, nothing weird about the language or anything, just a refactoring and an interface on the D side.
>>
>> You could use this to build a scripting engine into you program and it would be very fast.  However there is a security issue with it depending on what you want to use it for.  For example with games you may want to auto download game mods much like Unreal Tournament does.  How do you prevent the mod writer from doing something malicious? You can certainly filter the functions that downloaded script can call but the main issue is that D still allows pointer manipulation. Given a pointer a clever coder can gain access to system level functions.
>>
>> So you need a more sandbox friendly version of D which ends up being a different language anyway.
> 
> Hey, it can be D--   :)
> 
> D, without pointers, without asm, with only access to a very carefully limited subset of the library, could potentially be a very good safe scripting language.  I proposed (something very like) this back on 8/20/01 in "A Java-style sandbox without a Virtual Machine?", but there was a lot of confusion on what I meant by it :(
> 
> I did get a pointer to Bochs from that thread, which was a real win :)
> 
> 
> 
> Even if Walter doesn't feel like supporting it...we could do it.
> 

I would be cool to do this with D but it would require some restrictions.

1. Heavy control over imports
2. No pointers
3. No asm
4. Array bounds checking is always turned on.