On Tuesday, 16 June 2015 at 13:11:51 UTC, John Colvin wrote:
>> Looks like progress is being made on planting the seeds of a matrix implementation and computation library on top.
>
> Ilya Yaroshenko is doing great work here.

Yes - Ilya's work looks great, and Vlad Levenfeld has some interesting ideas too (less specifically on matrices, but relating to them).

> Well, as i briefly mentioned in my talk, it's really very easy to do, Jupyter/IPython makes it simple. Of course, behind the scenes Pyd is doing the hard work.

Yes - so often it's the last bit of integration that isn't so hard but removes frictions that makes it possible to realize the value created by all the foundational meta work.  And solving one's own problems is often a route to completing that integration.

> Visualisation is one of those things that no-one seems to have got right, despite the huge efforts that have gone in to it. I have ideas about D-based visualisation and already my own basic 3-D plotting library for huge datasets, but for now the link to python is key to get quick and easy access to the latest tools that the scientific and finance world are creating.

Would love to see it when it is ready (although I haven't used 3d plots much myself).  Agree about python, for better or for worse.

> There is http://nbviewer.ipython.org/gist/rossant/9463955 (see https://www.youtube.com/watch?v=_JZFSFR6Yeo for demo) which looks like a good starting point, but it's very basic.
>
> If https://github.com/quantopian/qgrid supported editing (perhaps it does already, but I can't see any mention of it) then that would probably be better.

Thank you for these - I will take a look.


Laeeth.

On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
> Bare-bones at the moment, but hopefully of interest to some people
>
> https://github.com/DlangScience/PydMagic
>
> This allows you to write D extensions to python, as inline cells in an ipython/jupyter notebook.

v0.1.0 is released, with substantial improvements. This is more like a usable tool and less like a proof of concept now, although I would definitely say it is still alpha quality due to lack of testing.

Highlights:

Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem

As a result of the above, the %%pyd magic interface has changed, see the README.md or %%pyd? for info on the new system

Extended the @pdef!() UDA mechanism to work for not only functions, but pretty much everything else that pyd supports. You can now use it to automatically expose types, members, properties and more.

Many bugs fixed.

On Tuesday, 30 June 2015 at 18:20:21 UTC, John Colvin wrote:
> On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
>> [...]
>
> v0.1.0 is released, with substantial improvements. This is more like a usable tool and less like a proof of concept now, although I would definitely say it is still alpha quality due to lack of testing.
>
> Highlights:
>
> Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem
>
> As a result of the above, the %%pyd magic interface has changed, see the README.md or %%pyd? for info on the new system
>
> Extended the @pdef!() UDA mechanism to work for not only functions, but pretty much everything else that pyd supports. You can now use it to automatically expose types, members, properties and more.
>
> Many bugs fixed.

Please report any bugs, suggestions, problems, anything. It all helps.

On Tuesday, 30 June 2015 at 18:20:21 UTC, John Colvin wrote:
> On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
>> Bare-bones at the moment, but hopefully of interest to some people
>>
>> https://github.com/DlangScience/PydMagic
>>
>> This allows you to write D extensions to python, as inline cells in an ipython/jupyter notebook.
>
> v0.1.0 is released, with substantial improvements. This is more like a usable tool and less like a proof of concept now, although I would definitely say it is still alpha quality due to lack of testing.
>
> Highlights:
>
> Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem
>
> As a result of the above, the %%pyd magic interface has changed, see the README.md or %%pyd? for info on the new system
>
> Extended the @pdef!() UDA mechanism to work for not only functions, but pretty much everything else that pyd supports. You can now use it to automatically expose types, members, properties and more.
>
> Many bugs fixed.

This is great - looking forward to playing with it.

One nice idea I have is to be able to flip between the notebook for charting and specifying reports and Excel for viewing and exploring the results.  Not just for me, but some people are much more comfortable with Excel.

(would use a backend server that is communicated with via nanomsg IPC so excel can see same data)

On Tue, 2015-06-30 at 18:20 +0000, John Colvin via Digitalmars-d
-announce
 […]
> 
> Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem

I am not convinced by this, though cleary my feeling carry no weight in decision making :-)

For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for:

	python3 setup.py build

	python3 setup.py install

SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one.

-- 
Russel. ============================================================================= Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net 41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.uk London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:
> On Tue, 2015-06-30 at 18:20 +0000, John Colvin via Digitalmars-d
> -announce
>  […]
>> 
>> Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem
>
> I am not convinced by this, though cleary my feeling carry no weight in decision making :-)
>
> For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for:
>
> 	python3 setup.py build
>
> 	python3 setup.py install
>
> SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one.

What is the benefit from using distutils for working with D in a notebook?  There are two standards - the Python one, and the D one.  The advantage of using dub is that it becomes wonderfully easy to pull in D projects from code.dlang.org and to compile your own work developed under dub.  (And dub itself continues to improve).  Linking to a D project of decent complexity via distutils is not my idea of fun.

I do agree that for pyd itself it would be nice to retain the option of distutils (although I would love to see dub added also),

One should look at this as an alpha, extremely promising project.  It is not any rough edges that are important at this stage, but that it has been done at all (in a form that is already very valuable).

The set of people that want to use computers to explore larger data sets than lately considered comfortable in an iterative manner is not small, even confining It just to finance.  It's of real value to be able to hook in to a proper back end that manages market and static data, with also any data processing and analytics also in D, and then to have a pretty and friendly front end that can just talk to this code on the back with minimal messing around to get there.  It's also very nice to have both Jupyter and an Excel spreadsheet at windows onto the data on your local machine or in the enterprise cloud.

The frictions to starting to play with D in a notebook are much lower than going via the command line, so I really am not sure if we should be worried about making it marketable at this stage rather than useful for doing real work.  It's potentially a nice debugging tool where you are trying to make sense of things that don't tidily fit in a debugging window, and where there is just too much stuff to do it via writefln or logging without putting lots of effort into the infrastructure to find events first.

I do like the simpler syntax (than PyD).  One question is whether you need to wrap every member of a struct.

But it's a very useful start as it stands.

On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:
> On Tue, 2015-06-30 at 18:20 +0000, John Colvin via Digitalmars-d
> -announce
>  […]
>> 
>> Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem
>
> I am not convinced by this, though cleary my feeling carry no weight in decision making :-)
>
> For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for:
>
> 	python3 setup.py build
>
> 	python3 setup.py install
>
> SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one.

As an addendum: not many people prefer to do non system type things in C than a higher level language when time is money.  So the mindset is you build a C extension to address the worst bits where Python's true colours shine through (ie when you are in a tight loop within Python interpreter and not spending most of the time in its C libraries.  Or writing C glue to connect Python to another library.

In my view, D is different.  I would rather write in D than Python, and to me it's much better Tor doing serious work.  Even for parsing a CSV I prefer it (although one could debate the point).  In any case D is not especially slower to write in than Python (particularly when you include time spent getting the bugs out), and in a decent number of cases it may be more productive.

So why bother with Python at all ?  Better ecosystem for charting and exploring data, and an interpreter is better suited generally for certain kinds of tasks.  Also in some areas more libraries, which can be helpful to get a quick result, so that one can go back and do it properly later.

It would be v helpful to have a Datetime conversion from D.  Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily.

On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:
> On Tue, 2015-06-30 at 18:20 +0000, John Colvin via Digitalmars-d
> -announce
>  […]
>> 
>> Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem
>
> I am not convinced by this, though cleary my feeling carry no weight in decision making :-)
>
> For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for:
>
> 	python3 setup.py build
>
> 	python3 setup.py install
>
> SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one.

There are 2 seperate points where a build/install system is needed:

1) Installing the extension. It's just one python file, which makes it a good fit for %install_ext but of course I could wrap it in a trivial setup.py to make people feel more comfortable / make it fit in with easy_install/pip or related tools.

2) building the inline D code and sorting out linking to other libraries etc. The basics of this can be done with distutils (it's how pyd traditionally does it), but seeing as it's all D code from this point, dub is a better fit, otherwise people have to have manually install any D libraries they want to use.

I originally modelled PydMagic on how the %%cython inline magic works. %%cython uses distutils under the scenes with minor leakage for the compilation options and PydMagic uses dub, with similar levels of leakage.


The recently-spun-off-from-PydMagic work of making a nicer API for pyd ( github.com/John-Colvin/ppyd which PydMagic pulls in via dub) is independent of the build system, so you can use pyd's distutils extensions or dub. This is the project you would use if you want to write a proper python extension in D, as opposed to little snippets in a Jupyter notebook.

On Thursday, 2 July 2015 at 19:51:19 UTC, Laeeth Isharc wrote:
> What is the benefit from using distutils for working with D in a notebook?  There are two standards - the Python one, and the D one.  The advantage of using dub is that it becomes wonderfully easy to pull in D projects from code.dlang.org and to compile your own work developed under dub.  (And dub itself continues to improve).  Linking to a D project of decent complexity via distutils is not my idea of fun.

This was my train of thought too.

> I do agree that for pyd itself it would be nice to retain the option of distutils (although I would love to see dub added also),

pyd does have rudimentary dub support, but it's only for embedding python in D, not the other way around. Extension support is on my TODO list, it's not complicated.

> One should look at this as an alpha, extremely promising project.
>  It is not any rough edges that are important at this stage, but that it has been done at all (in a form that is already very valuable).
>
> The set of people that want to use computers to explore larger data sets than lately considered comfortable in an iterative manner is not small, even confining It just to finance.  It's of real value to be able to hook in to a proper back end that manages market and static data, with also any data processing and analytics also in D, and then to have a pretty and friendly front end that can just talk to this code on the back with minimal messing around to get there.  It's also very nice to have both Jupyter and an Excel spreadsheet at windows onto the data on your local machine or in the enterprise cloud.
>
> The frictions to starting to play with D in a notebook are much lower than going via the command line, so I really am not sure if we should be worried about making it marketable at this stage rather than useful for doing real work.  It's potentially a nice debugging tool where you are trying to make sense of things that don't tidily fit in a debugging window, and where there is just too much stuff to do it via writefln or logging without putting lots of effort into the infrastructure to find events first.
>
> I do like the simpler syntax (than PyD).  One question is whether you need to wrap every member of a struct.

@pdefRecursive!() or @pdef!(Recursive.yes) or @pdef!(Recursive) ? It's totally doable.

> But it's a very useful start as it stands.
>
> As an addendum: not many people prefer to do non system type things in C than a higher level language when time is money.  So the mindset is you build a C extension to address the worst bits where Python's true colours shine through (ie when you are in a tight loop within Python interpreter and not spending most of the time in its C libraries.  Or writing C glue to connect Python to another library.
>
> In my view, D is different.  I would rather write in D than Python, and to me it's much better Tor doing serious work.  Even for parsing a CSV I prefer it (although one could debate the point).  In any case D is not especially slower to write in than Python (particularly when you include time spent getting the bugs out), and in a decent number of cases it may be more productive.
>
> So why bother with Python at all ?  Better ecosystem for charting and exploring data, and an interpreter is better suited generally for certain kinds of tasks.  Also in some areas more libraries, which can be helpful to get a quick result, so that one can go back and do it properly later.

Libraries, libraries, libraries. In a perfect world we can reimplement everything in D and it'll be awesome, but in the "getting shit done" world that normal people inhabit they need it to work, today. Bridges to other languages gives us that. E.g. I have a 3D plotting library in D (still in stealth mode on that one) that uses matplotlib via pyd to generate tick labels using either the builtin TeX parser & renderer (OK, fast) or actually calling out to latex (perfect, slow). Could I have done that all myself in D? Sure. Would I? No, i would still have terribly rendered tick labels with no TeX support. I don't have time to write a TeX implementation, even one as rudimentary and limited as matplotlib's.

In short, you're right.

> It would be v helpful to have a Datetime conversion from D.  Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily.

I know just enough about that topic to be very scared.

>> It would be v helpful to have a Datetime conversion from D.  Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily.
>
> I know just enough about that topic to be very scared.

I should be able to call these from D without changing PyD, I think, for D to Python.  Not sure about Python to D, but maybe.

https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d

From D to Python datetime.datetime
===================================
PyObject *PyDate_FromDate()(int year, int month, int day) {
  return PyDateTimeAPI.Date_FromDate(year, month, day, PyDateTimeAPI.DateType);
}
PyObject *PyDateTime_FromDateAndTime()(int year, int month, int day, int hour, int min, int sec, int usec) {
  return PyDateTimeAPI.DateTime_FromDateAndTime(year, month, day, hour,
    min, sec, usec, Py_None, PyDateTimeAPI.DateTimeType);
}

From Python datetime.datetime to D
===================================
// D translations of C macros:
int PyDateTime_GET_YEAR()(PyObject *o) {
  PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
  return (ot.data[0] << 8) | ot.data[1];
}
int PyDateTime_GET_MONTH()(PyObject *o) {
  PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
  return ot.data[2];
}
int PyDateTime_GET_DAY()(PyObject *o) {
  PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
  return ot.data[3];
}

int PyDateTime_DATE_GET_HOUR()(PyObject *o) {
  PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
  return ot.data[4];
}
int PyDateTime_DATE_GET_MINUTE()(PyObject *o) {
  PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
  return ot.data[5];
}
int PyDateTime_DATE_GET_SECOND()(PyObject *o) {
  PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
  return ot.data[6];
}
int PyDateTime_DATE_GET_MICROSECOND()(PyObject *o) {
  PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
  return (ot.data[7] << 16) | (ot.data[8] << 8) | ot.data[9];
}