In wrapping Python, I have had cause to update David Rushby's version of the Python API header for D.

Aside from some bug fixes, one thing I've been trying to do is remove the need to call _loadPythonSupport() at the start of every module. This is not much of a burden with the wrapper I'm writing (it can simply call it in the module_init function), but I have an idea which I think would work better. The trouble is, it seems to cause Python to crash, which always seems to put a damper on things.

The _loadPythonSupport function just loads a whole mess of singleton objects. Py_None, Py_True, Py_False, all of the type objects, and all of the exception types, and some others. Rather than load all of these things when the module is loaded (as a given module will only need to
use a few of them), I am trying to load them lazily:

(Excuse the formatting, I didn't pay close attention to width when I wrote it.) :-)

private {

PyObject* m_Py_None;
PyObject* m_Py_True;
// ... and so on for all of the singletons

PyObject* m_builtins, m_types, m_weakref;

// Note the new function template syntax. Ohh, ahh.
typeof(Ptr) lazy_sys(alias Ptr, char[] name) () {
    if (Ptr is null) {
        PyObject* sys_modules = PyImport_GetModuleDict();
        Ptr = PyDict_GetItemString(sys_modules, name ~ \0);
        Py_DECREF(sys_modules);
        assert (Ptr !is null,
            "python.d couldn't load " ~ name ~ " attribute!");
    }
    return Ptr;
}

alias lazy_sys!(m_builtins, "__builtin__") builtins;
alias lazy_sys!(m_types, "types") types;
alias lazy_sys!(m_weakref, "weakref") weakref;

typeof(Ptr) lazy_load(alias from, alias Ptr, char[] name) () {
    if (Ptr is null) {
        Ptr = cast(typeof(Ptr)) PyObject_GetAttrString(from(),
            name ~ \0);
        assert (Ptr !is null,
            "python.d couldn't load " ~ name ~ " attribute!");
    }
    return Ptr;
}

} /* end private */

alias lazy_load!(builtins, m_Py_None, "None") Py_None;
alias lazy_load!(builtins, m_Py_True, "True") Py_True;
// ... and so on for all of the singletons

The loading code is based on David Rushby's.

Note that, though these are functions, you don't actually need to call them. Viz. you don't need to say "Py_None()", just "Py_None", as D seems to treat them like properties, or something. I'm still not totally clear on why that works, I only know that it does. (The "def" template in my Python wrapper makes use of the same oddity.)

Thus, the singletons are loaded the first time they are needed, and only the first time they are needed. There is no need to call Py_DECREF on these references in (say) a static destructor, as Python cannot "unload" a module; and besides, they cannot be deallocated by Python anyway.

This does, in fact, work, except for the wee problem that Python, once it has loaded a module compiled with this header, crashes horribly when you try to quit it. This makes me suspect something very wrong is happening when Python tries to unload the module. I'm not sure precisely what I did here that would cause this, however, and so I throw this out for ideas.

-Kirk McDonald

Ha! I fixed it. It was a stupid problem. I managed to decrement a reference count when I shouldn't have, causing Python to free() something twice when the interpreter closed down.

Kirk McDonald wrote:
> // Note the new function template syntax. Ohh, ahh.
> typeof(Ptr) lazy_sys(alias Ptr, char[] name) () {
>     if (Ptr is null) {
>         PyObject* sys_modules = PyImport_GetModuleDict();
>         Ptr = PyDict_GetItemString(sys_modules, name ~ \0);
This one right here; GetModuleDict returns a borrowed reference!
>         Py_DECREF(sys_modules);
>         assert (Ptr !is null,
>             "python.d couldn't load " ~ name ~ " attribute!");
>     }
>     return Ptr;
> }
> 

Anyway, the updated (and now working) header can be found at:

http://216.190.88.10/media/python.d

-Kirk McDonald

Kirk McDonald wrote:
> Ha! I fixed it. It was a stupid problem. I managed to decrement a reference count when I shouldn't have, causing Python to free() something twice when the interpreter closed down.
> 
> Kirk McDonald wrote:
> 
>> // Note the new function template syntax. Ohh, ahh.
>> typeof(Ptr) lazy_sys(alias Ptr, char[] name) () {
>>     if (Ptr is null) {
>>         PyObject* sys_modules = PyImport_GetModuleDict();
>>         Ptr = PyDict_GetItemString(sys_modules, name ~ \0);
> 
> This one right here; GetModuleDict returns a borrowed reference!
> 
>>         Py_DECREF(sys_modules);
>>         assert (Ptr !is null,
>>             "python.d couldn't load " ~ name ~ " attribute!");
>>     }
>>     return Ptr;
>> }
>>
> 
> Anyway, the updated (and now working) header can be found at:
> 
> http://216.190.88.10/media/python.d
> 
> -Kirk McDonald

Durrrr, that's not the right URL. It's this:

http://216.190.88.10:8087/media/python.d

-Kirk McDonald

Kirk McDonald wrote:
> Durrrr, that's not the right URL. It's this:
> 
> http://216.190.88.10:8087/media/python.d


Kirk,

Do you need a place to host this project of yours?  Might I offer up dsource.org for you?  I would need a project name (hopefully more exciting than Dython, but if that's all you got...) and a short description to get you set up.

It seems as if it's gone beyond just being a binding, but if not, and you don't want a full-blown project, you can always use the Bindings project to get SVN source control on your files.

http://www.dsource.org/projects/bindings

Cheers,
Brad

Brad Anderson wrote:
> Kirk McDonald wrote:
> 
>>Durrrr, that's not the right URL. It's this:
>>
>>http://216.190.88.10:8087/media/python.d
> 
> 
> 
> Kirk,
> 
> Do you need a place to host this project of yours?  Might I offer up
> dsource.org for you?  I would need a project name (hopefully more exciting
> than Dython, but if that's all you got...) and a short description to get you
> set up.
> 
> It seems as if it's gone beyond just being a binding, but if not, and you
> don't want a full-blown project, you can always use the Bindings project to
> get SVN source control on your files.
> 
> http://www.dsource.org/projects/bindings
> 
> Cheers,
> Brad

I've been calling it "Pyd" (pronounced "pied"), but I am open to suggestions. :-)

I would certainly enjoy some dsource space. My webserver is just my poor little Linux box. (Although my logs show only 10 people have grabbed that file so far, so I think it's been up to the task.) :-)

The long-term goal is something analagous to Boost.Python. At the moment, all I have is a nifty function wrapper and a class that manages a PyObject*, overloading very nearly every single overloadable operator, passing them along to various Python API functions. (The class, DPyObject, is largely working, but I need to clean up the opCall overload somewhat.) I also have a couple functions that can convert values to and from arbitrary Python objects, which support both the function wrapper and the DPyObject class. Some next steps include better support for lists, tuples, and dicts. (An array or AA of a convertable type can easily be converted to a Python list or dict. A Python list or dict can easily be converted to an array or AA of DPyObjects. It would also be nice to have a Boost.Python-style make_tuple function.)

So! A short description might be: "An object-oriented wrapper around the Python API."

-Kirk McDonald

Kirk McDonald wrote:
> I've been calling it "Pyd" (pronounced "pied"), but I am open to
> suggestions. :-)
> 
> I would certainly enjoy some dsource space. My webserver is just my poor little Linux box. (Although my logs show only 10 people have grabbed that file so far, so I think it's been up to the task.) :-)
> 
> The long-term goal is something analagous to Boost.Python. At the moment, all I have is a nifty function wrapper and a class that manages a PyObject*, overloading very nearly every single overloadable operator, passing them along to various Python API functions. (The class, DPyObject, is largely working, but I need to clean up the opCall overload somewhat.) I also have a couple functions that can convert values to and from arbitrary Python objects, which support both the function wrapper and the DPyObject class. Some next steps include better support for lists, tuples, and dicts. (An array or AA of a convertable type can easily be converted to a Python list or dict. A Python list or dict can easily be converted to an array or AA of DPyObjects. It would also be nice to have a Boost.Python-style make_tuple function.)
> 
> So! A short description might be: "An object-oriented wrapper around the Python API."
> 
> -Kirk McDonald

You need a username at dsource (register in the Forums section of the site) and then I'll get it set up.

BA

Brad Anderson wrote:
> Kirk McDonald wrote:
>>
>>So! A short description might be: "An object-oriented wrapper around the
>>Python API."
>>
>>-Kirk McDonald
> 
> 
> You need a username at dsource (register in the Forums section of the site)
> and then I'll get it set up.
> 
> BA

Done! I'm "KirkMcDonald".