ffs, if I don't release it now I'll _never_ release it.

After over a year of work, a few drastic changes, and a completely new implementation, I give you: MiniD 2.

http://dsource.org/projects/minid

---------------------------------
What's new in the language?

- Lots.

- Several syntactic changes and additions leading to much cleaner-looking, terser code.

- Classes and instances have been greatly improved, thanks to a one-year trip into the land of prototype-based objects.

- More metamethods for more object customizability.  In particular, field access and method calls can be intercepted, and there is now a full complement of arithmetic operator overloads (including reverse versions).

- Weak references.

- A new type which can hold instances of D classes, for use by native code.

- Sixty-four (64) bit integers.  What will *you* do with all those bits?

- A much-expanded and improved standard library.

- Much more well-defined semantics for most language constructs and operations.

- A more flexible module loading system.

---------------------------------
What's this new implementation?

- It's waaaaaaaaaAAAy better than the old one.  This is really the killer feature of MiniD 2.

- It has its own garbage collector, meaning that your scripts won't trigger native app GC cycles (well.. much, much less often), and MiniD code also runs much faster.

- It has a completely new API that is similar to the Lua or Squirrel APIs.

- There Is One Way To Do It.  The old implementation and API were hole-y and inconsistent.  Now, you're pretty much locked into doing things one way, meaning much less chance for error and much less to learn.

- Virtually anything you can do in the language, you can do in the API without much difficulty.  The new API is much more well-rounded than the old one.

- Performancewise, it beats the pants off of Python, though that's not _necessarily_ saying much.  I don't know how it measures up to Lua as I don't have a way of measuring Lua's performance from within the language (so as to exclude startup and compilation times), but I get the impression that it's pretty close in some cases.

---------------------------------
What do I need to compile it?

- Tango, SVN revision 4048.  I *told* you this was a tentative release candidate.  I don't know if it'll work with any older/newer ones.

- DMD 1.034 or newer, or equivalent.  LDC seems to work as well.  GDC does not.  Sorry Mac users, go get David to work on GDC again ;)

- Some kind of build tool, DSSS/rebuild or bu[il]d.

- MiniD itself.  Just check out the head revision of the MiniD trunk at http://svn.dsource.org/projects/minid/trunk.

- The understanding that if you are on Linux and try to do all sorts of stupid fancy stuff with symbolic links and putting things in nonstandard directories and making directories read-only, I will not have any sympathy for you when you can't get D, Tango, DSSS, or MiniD to work correctly.  The D toolchain is fragile enough as it is, please don't bend it.

---------------------------------
Why am I calling this a tentative release candidate?

- Some aspects of the library (THE BINDING LIBRARY, which is much better than MiniD 1's, and the compiler to an extent) have not been very heavily tested and are likely to be somewhat buggy.  But I'm one guy with limited time and so if you could start using it and making it fail that'd be just great.

- Some of the docs are not yet complete.  The section on modules is not yet complete, but not a whole lot has changed there anyway.  The DDocs for the individual API modules are basically complete but are somewhat lacking in more complex examples, which is what the tutorials are supposed to be for.

- I haven't compiled it on Linux recently.  Yeah, I should do that more often.  But it worked last month or so.

---------------------------------
What is needed before an "official" release is made?

- Time, something I don't have a lot of this week.  After that, I have three weeks of bliss.

- Testing.

- Finished docs.

- Maybe a couple more additions/changes.

---------------------------------
Anything else?

- Mudkip.

Great stuff, but I have some trouble with the binding lib.

1. If I don't wrap a constructor, I get an error:
minid/bind.d(1101): static assert  "Cannot call default constructor for class Foo; please wrap a constructor explicitly"

2. If I wrap a constructor with no parameters (i.e. WrapCtors!(void function())), I get an error:
minid/bind.d(1066): constructor minid.bind.WrappedClass!(Foo,"Foo","",WrapCtors!(void function()),WrapMethod!(Call)).WrappedClass.ClassCtorShims!(void function()).this (MDVM*,()) does not match parameter types ()
minid/bind.d(1066): Error: expected 1 arguments, not 0

3. I can't find an obvious way to initialize global variables, so I try superPush, which gives a runtime error: minid.types.MDException: <no location available>: newInstance - expected 'class' for base, not 'null'

Test code below:

import minid.bind;
import minid.api;

class Foo {
    this() {}
    this(int i) {}
    void Call() {}
}

void main() {
    MDVM vm;
    auto t = openVM(&vm);
    loadStdlibs(t);
    WrapGlobals!(
        WrapType!(
            Foo, "Foo",
            WrapCtors!(void function(int)),
            WrapMethod!(Foo.Call)
        )
    )(t);
    superPush(t, new Foo());
    newGlobal(t, "myfoo");
}

On Mon, Dec 8, 2008 at 3:44 AM, mpt <foo@invalid.x> wrote:
> Great stuff, but I have some trouble with the binding lib.

Good!  Because I wasn't ;)

> 1. If I don't wrap a constructor, I get an error:
> minid/bind.d(1101): static assert  "Cannot call default constructor for class Foo; please wrap a constructor explicitly"

Fixed.

> 2. If I wrap a constructor with no parameters (i.e. WrapCtors!(void function())), I get an error:
> minid/bind.d(1066): constructor minid.bind.WrappedClass!(Foo,"Foo","",WrapCtors!(void function()),WrapMethod!(Call)).WrappedClass.ClassCtorShims!(void function()).this (MDVM*,()) does not match parameter types ()
> minid/bind.d(1066): Error: expected 1 arguments, not 0

Fixed.

> 3. I can't find an obvious way to initialize global variables, so I try superPush, which gives a runtime error:

You're actually doing it right, unless you wanted to instantiate the wrapped MiniD class, in which case it'd be more like:

   pushGlobal(t, "Foo"); // get the global class Foo
   pushNull(t); // make room for 'this'
   rawCall(t, -2, 1); // call the class to instantiate it (1 return
for the instance)
   newGlobal(t, "myfoo"); // save it in the global myfoo


> minid.types.MDException: <no location available>: newInstance - expected 'class' for base, not 'null'

Fixed.

> Test code below:
>
> import minid.bind;
> import minid.api;
>
> class Foo {
>    this() {}
>    this(int i) {}
>    void Call() {}
> }
>
> void main() {
>    MDVM vm;
>    auto t = openVM(&vm);
>    loadStdlibs(t);
>    WrapGlobals!(
>        WrapType!(
>            Foo, "Foo",
>            WrapCtors!(void function(int)),
>            WrapMethod!(Foo.Call)
>        )
>    )(t);
>    superPush(t, new Foo());
>    newGlobal(t, "myfoo");
> }

This code now works, just svn up your MiniD installation :)

Jarrett Billingsley wrote:
> On Mon, Dec 8, 2008 at 3:44 AM, mpt <foo@invalid.x> wrote:
>> Great stuff, but I have some trouble with the binding lib.
> 
> Good!  Because I wasn't ;)
> 
>> 1. If I don't wrap a constructor, I get an error:
>> minid/bind.d(1101): static assert  "Cannot call default constructor for class Foo; please wrap a constructor explicitly"
> 
> Fixed.

Fix compiles for the explicitly defined ctor, but not for the default. This code gives:
minid/bind.d(1110): Error: no super class constructor for Foo

import minid.bind;
import minid.api;

class Foo {}

void main() {
    MDVM vm;
    auto t = openVM(&vm);
    loadStdlibs(t);
    WrapGlobals!(
        WrapType!(
            Foo, "Foo"
        )
    )(t);
}

>> 3. I can't find an obvious way to initialize global variables, so I try superPush, which gives a runtime error:
> 
> You're actually doing it right, unless you wanted to instantiate the
> wrapped MiniD class, in which case it'd be more like:
> 
>    pushGlobal(t, "Foo"); // get the global class Foo
>    pushNull(t); // make room for 'this'
>    rawCall(t, -2, 1); // call the class to instantiate it (1 return
> for the instance)
>    newGlobal(t, "myfoo"); // save it in the global myfoo
> 

I can't see the difference between superPush-created myfoo and one created by your example. superPush doc says it creates MiniD-converted objects, but they're not usable.

superPush(t, new Foo());
newGlobal(t, "myfoo");
runString(t, "myfoo.Call();");

This gives: "tango.core.Exception.AssertException@../extern/minid/minid/bind.d(1397): Invalid 'this' parameter passed to method Foo.Call"

Your example works, but it's not what I need. I want to create the Foo object in D myself and pass it to MiniD and call its methods.

Ideally, I'd like to have/write a function like
void setGlobal(Type)(MDThread* t, char[] name, Type value) to create and modify global variables.

On Mon, Dec 8, 2008 at 3:22 PM, mpt <mpt@iki.fi> wrote:
> Fix compiles for the explicitly defined ctor, but not for the default. This
> code gives:
> minid/bind.d(1110): Error: no super class constructor for Foo
>
> import minid.bind;
> import minid.api;
>
> class Foo {}
>
> void main() {
>    MDVM vm;
>    auto t = openVM(&vm);
>    loadStdlibs(t);
>    WrapGlobals!(
>        WrapType!(
>            Foo, "Foo"
>        )
>    )(t);
> }

Fixed.

> I can't see the difference between superPush-created myfoo and one created by your example. superPush doc says it creates MiniD-converted objects, but they're not usable.

There shouldn't be much different, at least from the point of view of the MiniD code.  If you want to know, the difference is that instances of the wrapped class created using MiniD are actually instances of a class that is _derived_ from the class you wrapped, whereas instances of the class you push are instances of the original class.  But the binding library is *supposed* to abstract that away ;)

> superPush(t, new Foo());
> newGlobal(t, "myfoo");
> runString(t, "myfoo.Call();");
>
> This gives: "tango.core.Exception.AssertException@../extern/minid/minid/bind.d(1397): Invalid 'this' parameter passed to method Foo.Call"

Fixed.

> Your example works, but it's not what I need. I want to create the Foo object in D myself and pass it to MiniD and call its methods.
>
> Ideally, I'd like to have/write a function like
> void setGlobal(Type)(MDThread* t, char[] name, Type value) to create and
> modify global variables.

The implementation of that might be:

void SetGlobal(Type)(MDThread* t, char[] name, Type value)
{
    superPush(t, value);

    if(findGlobal(t, name))
    {
        swap(t);
        fielda(t, -2, name);
        pop(t);
    }
    else
        newGlobal(t, name);
}

This will create a global if it doesn't already exist, and will update an existing global.  If you want it to always update an existing global, it simply becomes "superPush(t, value); setGlobal(t, name);".

I've learned a bit minid today.

I'm curious why foreach opApply don't use the yield instead of the current implementation?
and another curious thing is how can I serialize/deserialize minid object in minid?
any .tupleof for class available?

在 Mon, 08 Dec 2008 13:50:47 +0800，Jarrett Billingsley <jarrett.billingsley@gmail.com> 写道:

> ffs, if I don't release it now I'll _never_ release it.
>
> After over a year of work, a few drastic changes, and a completely new implementation, I give you: MiniD 2.
>
> http://dsource.org/projects/minid
>
> ---------------------------------
> What's new in the language?
>
> - Lots.
>
> - Several syntactic changes and additions leading to much cleaner-looking, terser code.
>
> - Classes and instances have been greatly improved, thanks to a one-year trip into the land of prototype-based objects.
>
> - More metamethods for more object customizability.  In particular, field access and method calls can be intercepted, and there is now a full complement of arithmetic operator overloads (including reverse versions).
>
> - Weak references.
>
> - A new type which can hold instances of D classes, for use by native code.
>
> - Sixty-four (64) bit integers.  What will *you* do with all those bits?
>
> - A much-expanded and improved standard library.
>
> - Much more well-defined semantics for most language constructs and operations.
>
> - A more flexible module loading system.
>
> ---------------------------------
> What's this new implementation?
>
> - It's waaaaaaaaaAAAy better than the old one.  This is really the killer feature of MiniD 2.
>
> - It has its own garbage collector, meaning that your scripts won't trigger native app GC cycles (well.. much, much less often), and MiniD code also runs much faster.
>
> - It has a completely new API that is similar to the Lua or Squirrel APIs.
>
> - There Is One Way To Do It.  The old implementation and API were hole-y and inconsistent.  Now, you're pretty much locked into doing things one way, meaning much less chance for error and much less to learn.
>
> - Virtually anything you can do in the language, you can do in the API without much difficulty.  The new API is much more well-rounded than the old one.
>
> - Performancewise, it beats the pants off of Python, though that's not _necessarily_ saying much.  I don't know how it measures up to Lua as I don't have a way of measuring Lua's performance from within the language (so as to exclude startup and compilation times), but I get the impression that it's pretty close in some cases.
>
> ---------------------------------
> What do I need to compile it?
>
> - Tango, SVN revision 4048.  I *told* you this was a tentative release candidate.  I don't know if it'll work with any older/newer ones.
>
> - DMD 1.034 or newer, or equivalent.  LDC seems to work as well.  GDC does not.  Sorry Mac users, go get David to work on GDC again ;)
>
> - Some kind of build tool, DSSS/rebuild or bu[il]d.
>
> - MiniD itself.  Just check out the head revision of the MiniD trunk at http://svn.dsource.org/projects/minid/trunk.
>
> - The understanding that if you are on Linux and try to do all sorts of stupid fancy stuff with symbolic links and putting things in nonstandard directories and making directories read-only, I will not have any sympathy for you when you can't get D, Tango, DSSS, or MiniD to work correctly.  The D toolchain is fragile enough as it is, please don't bend it.
>
> ---------------------------------
> Why am I calling this a tentative release candidate?
>
> - Some aspects of the library (THE BINDING LIBRARY, which is much better than MiniD 1's, and the compiler to an extent) have not been very heavily tested and are likely to be somewhat buggy.  But I'm one guy with limited time and so if you could start using it and making it fail that'd be just great.
>
> - Some of the docs are not yet complete.  The section on modules is not yet complete, but not a whole lot has changed there anyway.  The DDocs for the individual API modules are basically complete but are somewhat lacking in more complex examples, which is what the tutorials are supposed to be for.
>
> - I haven't compiled it on Linux recently.  Yeah, I should do that more often.  But it worked last month or so.
>
> ---------------------------------
> What is needed before an "official" release is made?
>
> - Time, something I don't have a lot of this week.  After that, I have three weeks of bliss.
>
> - Testing.
>
> - Finished docs.
>
> - Maybe a couple more additions/changes.
>
> ---------------------------------
> Anything else?
>
> - Mudkip.
> 

On Tue, Dec 9, 2008 at 3:58 AM, davidl <davidl@126.com> wrote:
> I've learned a bit minid today.
>
> I'm curious why foreach opApply don't use the yield instead of the current implementation?

Ah, but you can.  Coroutines have an opApply, and are therefore
iterable using foreach.  See the section on coroutines
(http://www.dsource.org/projects/minid/wiki/LanguageSpec2/Functions#Coroutines)
for some info on that.  Using coroutines for iteration is more
powerful, but of course the price you pay is that they're a lot more
expensive than a function closure.

> and another curious thing is how can I serialize/deserialize minid object in minid? any .tupleof for class available?

Sure, have a look of fieldsOf/allFieldsOf in the base library (http://www.dsource.org/projects/minid/wiki/StdLib2/BaseLib).  They'll give you functions as well, but it's easy to skip over them.

在 Tue, 09 Dec 2008 21:07:46 +0800，Jarrett Billingsley <jarrett.billingsley@gmail.com> 写道:

> On Tue, Dec 9, 2008 at 3:58 AM, davidl <davidl@126.com> wrote:
>> I've learned a bit minid today.
>>
>> I'm curious why foreach opApply don't use the yield instead of the current implementation?
>
> Ah, but you can.  Coroutines have an opApply, and are therefore
> iterable using foreach.  See the section on coroutines
> (http://www.dsource.org/projects/minid/wiki/LanguageSpec2/Functions#Coroutines)
> for some info on that.  Using coroutines for iteration is more
> powerful, but of course the price you pay is that they're a lot more
> expensive than a function closure.
>

I got an impression that a scripting lang coroutine is less expensive than a function call? I guess coroutine in a scripting interpreted lang won't require any thread involved, and the suspension of the execution maybe even more costless compared to a return statement and reexecution of that function in the case of opApply.
And I'm still not able to get opApply to work with coroutines, any example available?
I always get the error message of "Iterated coroutine must be in the initial state", I've no clue about what it's going on.

>> and another curious thing is how can I serialize/deserialize minid object in minid? any .tupleof for class available?
>
> Sure, have a look of fieldsOf/allFieldsOf in the base library (http://www.dsource.org/projects/minid/wiki/StdLib2/BaseLib).  They'll give you functions as well, but it's easy to skip over them.
> 

thanks, that's helpful. MiniD is way more powerful than I've thought.

On Wed, Dec 10, 2008 at 12:12 AM, davidl <davidl@126.com> wrote:
>
> I got an impression that a scripting lang coroutine is less expensive than a function call? I guess coroutine in a scripting interpreted lang won't require any thread involved, and the suspension of the execution maybe even more costless compared to a return statement and reexecution of that function in the case of opApply.

The cost of a coroutine object in MiniD, as well as the cost of calling it, depends on the compilation options (http://www.dsource.org/projects/minid/wiki/API2/CompilationOptions). If you don't specify any version flags when compiling MiniD, it will use tango Fibers for coroutines created from native functions, and will use its own system of pausing and resuming for coroutines created from MiniD functions.

The cost of a coroutine resume/yield is more than a normal function
call, but it's still not a lot.  For example, using MiniD coroutines,
I can perform around 1.6 million resume/yield pairs per second on my
Pentium M.  Fibers aren't much slower; I still get about 1.25 million.
 That's not going to be a bottleneck considering the speed of most
other operations.

Rather, the biggest cost with a coroutine is memory.  A coroutine
object is relatively large compared to a function closure, and
depending on the compilation options, can also allocate D heap memory.
 So for simple iteration tasks, it might be in the interest of
performance to try to write it in the form of a function iterator, and
if it gets to be too complex to be worth it, you can write it as a
coroutine.

> And I'm still not able to get opApply to work with coroutines, any example available?
> I always get the error message of "Iterated coroutine must be in the initial state", I've no clue about what it's going on.

When you use foreach on a coroutine, it has to be in the "initial"
state.  The only times a coroutine can be in an initial state is (1)
immediately after they are created, before they are called even once,
and (2) after they have died and then had their .reset() method
called.  A simple example:

function count(x) =
    coroutine function()
    {
        yield() // first "empty" yield is important for foreach-able coroutines
        for(i: 1 .. x + 1)
            yield(i) // yield values after that
        // when this coroutine returns here, iteration stops
    }

foreach(v; count(4))
    writeln(v) // prints 1 through 4

Notice that count is returning a new coroutine object each time you call it, so the coroutine is always in the "initial" state.

There's more info on coroutines, including showing some iterators and explaining their various states, at the following link:

http://www.dsource.org/projects/minid/wiki/LanguageSpec2/Functions#Coroutines

在 Wed, 10 Dec 2008 13:52:18 +0800，Jarrett Billingsley <jarrett.billingsley@gmail.com> 写道:

> On Wed, Dec 10, 2008 at 12:12 AM, davidl <davidl@126.com> wrote:
>>
>> I got an impression that a scripting lang coroutine is less expensive than a function call? I guess coroutine in a scripting interpreted lang won't require any thread involved, and the suspension of the execution maybe even more costless compared to a return statement and reexecution of that function in the case of opApply.
>
> The cost of a coroutine object in MiniD, as well as the cost of calling it, depends on the compilation options (http://www.dsource.org/projects/minid/wiki/API2/CompilationOptions). If you don't specify any version flags when compiling MiniD, it will use tango Fibers for coroutines created from native functions, and will use its own system of pausing and resuming for coroutines created from MiniD functions.
>
> The cost of a coroutine resume/yield is more than a normal function
> call, but it's still not a lot.  For example, using MiniD coroutines,
> I can perform around 1.6 million resume/yield pairs per second on my
> Pentium M.  Fibers aren't much slower; I still get about 1.25 million.
>  That's not going to be a bottleneck considering the speed of most
> other operations.
>
> Rather, the biggest cost with a coroutine is memory.  A coroutine
> object is relatively large compared to a function closure, and
> depending on the compilation options, can also allocate D heap memory.
>  So for simple iteration tasks, it might be in the interest of
> performance to try to write it in the form of a function iterator, and
> if it gets to be too complex to be worth it, you can write it as a
> coroutine.
>
>> And I'm still not able to get opApply to work with coroutines, any example available?
>> I always get the error message of "Iterated coroutine must be in the initial state", I've no clue about what it's going on.
>
> When you use foreach on a coroutine, it has to be in the "initial"
> state.  The only times a coroutine can be in an initial state is (1)
> immediately after they are created, before they are called even once,
> and (2) after they have died and then had their .reset() method
> called.  A simple example:
>
> function count(x) =
>     coroutine function()
>     {
>         yield() // first "empty" yield is important for foreach-able coroutines
>         for(i: 1 .. x + 1)
>             yield(i) // yield values after that
>         // when this coroutine returns here, iteration stops
>     }
>
> foreach(v; count(4))
>     writeln(v) // prints 1 through 4
>

It's cool it could be used in that syntax. But I find nowhere of docs describing this feature? Do I miss something?

Actually I tried something like:
module test
global v=coroutine function()
{
local opApply=coroutine function opApply(m)
{
local i=0
 if (i<3) {
 i++
 yield(i)
}
return null
}
yield()
}
v()
foreach(m;v)writefln(m)

In the case I wrote, the coroutine foreach just conflicts with the coroutine foreach way you illustrate
in your example. As I didn't have that information, I tried to perform my coroutine foreach in a standardized
opApply way. That's somewhat confusing for people not having the knowledge of coroutine foreach.
Intuitively, the foreach is presumably working in the way of calling instance's opApply func and passing the arg
of reverse or not to it. While the truth is I'm wrong.

The inconsistency seems to be a tradeoff. Yes, without opApply it looks clean. While it's inconsistent. That means
developers who use minid need to learn more.

The following case illustrates a easily misread code for me at least. In minid, I thought the parentheses
were compulsive, cause the coroutinefunc.state won't work, but coroutinefunc.state(). So in such case, I guess the
"count" refers to the function object, while the result shows it just endless loop and results stack overflow.
Even that "count" actually a function call without the arg, the calling is performed. that's somehow misleading.

function count(x)
{
    function opApply(m) = coroutine function opApply(m)
    {
        for(i: 1 .. x + 1)
            yield(i) // yield values after that
        return null
    }
    foreach(v;count)
        writeln(v) // prints 1 through 4
}
count(4)
object.Exception: Stack Overflow

MDCL stack overflowed out, maybe mdcl should do something to protect itself from stack overflowing.

I'm trying to use coroutine foreach in a class.

class count
{
    x =3
    function opApply(m) = coroutine function opApply(m)
    {
        yield()
        for(i: 1 .. m.x + 1)
            yield(i)
    }

}
global c= count()
    foreach(v;c.opApply(c))
        writeln(v) // prints 1 through 4

it causes of runtime error: Error: opApply(7): Attempting to access field 'x' from a value of type 'null'

The problem here is I misuseing the function definition. function opApply(m) = coroutine function opApply() works OK.

But that's inconsistent from a strong type user's view. Because the left part is function with 1 arg, right part is a coroutine function without any arg. That's pretty error-prone for newbies.

Also the class coroutine foreach tries as follows failed:

class count
{
    x =3
    function opApply(m) = coroutine function opApply()
    {
        writefln(m) // prints null, that's pretty astonishing for me.
        for(i: 1 .. x + 1)
            yield(i)
    }

}
global c= count()
    foreach(v;c)
        writeln(v) // prints 1 through 4
the result is printing nothing, and no runtime error.


If I modify the loop to
    for(i: 1 .. :x + 1)
            yield(i)
runtime error shows up:
Error: opApply(7): Attempting to access field 'x' from a value of type 'null'

Seems there's no easy way of using coroutine foreach with class instance?

> Notice that count is returning a new coroutine object each time you call it, so the coroutine is always in the "initial" state.
>
> There's more info on coroutines, including showing some iterators and explaining their various states, at the following link:
>
> http://www.dsource.org/projects/minid/wiki/LanguageSpec2/Functions#Coroutines
>