On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
>>> It would be v helpful to have a Datetime conversion from D.  Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily.
>>
>> I know just enough about that topic to be very scared.
>
> I should be able to call these from D without changing PyD, I think, for D to Python.  Not sure about Python to D, but maybe.
>
> https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d

Doh!

It's already in deimos, but you need to call PyDateTime_IMPORT(); before calling the conversion function.

So all that is needed is the following:

module example;
import std.datetime;
import std.stdio;
import pyd.pyd;
import std.conv;
import deimos.python.datetime;


DateTime foo()
{
    return DateTime(1999,1,1);
}
extern(C) void PydMain() {
    ex_d_to_python((DateTime dt) => PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));
    def!foo();
    module_init();
    PyDateTime_IMPORT();
}

Ideally, shouldn't PyD do this for you ?  I think the code is already there in make_object.d, but it doesn't work - maybe because PyDateTime_IMPORT() has not been called if it needs to be.

Thanks.


Laeeth.

On Friday, 3 July 2015 at 10:23:14 UTC, Laeeth Isharc wrote:
> On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
>>>> It would be v helpful to have a Datetime conversion from D.  Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily.
>>>
>>> I know just enough about that topic to be very scared.
>>
>> I should be able to call these from D without changing PyD, I think, for D to Python.  Not sure about Python to D, but maybe.
>>
>> https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
>
> Doh!
>
> It's already in deimos, but you need to call PyDateTime_IMPORT(); before calling the conversion function.
>
> So all that is needed is the following:
>
> module example;
> import std.datetime;
> import std.stdio;
> import pyd.pyd;
> import std.conv;
> import deimos.python.datetime;
>
>
> DateTime foo()
> {
>     return DateTime(1999,1,1);
> }
> extern(C) void PydMain() {
>     ex_d_to_python((DateTime dt) => PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));
>     def!foo();
>     module_init();
>     PyDateTime_IMPORT();
> }
>
> Ideally, shouldn't PyD do this for you ?  I think the code is already there in make_object.d, but it doesn't work - maybe because PyDateTime_IMPORT() has not been called if it needs to be.
>
> Thanks.
>
>
> Laeeth.

Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date?

Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?

On Friday, 3 July 2015 at 10:52:38 UTC, John Colvin wrote:
>
> Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?

Woops, I means milliseconds.

On Friday, 3 July 2015 at 10:52:38 UTC, John Colvin wrote:
> On Friday, 3 July 2015 at 10:23:14 UTC, Laeeth Isharc wrote:
>> On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
>>>>> It would be v helpful to have a Datetime conversion from D.
>>>>>  Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily.
>>>>
>>>> I know just enough about that topic to be very scared.
>>>
>>> I should be able to call these from D without changing PyD, I think, for D to Python.  Not sure about Python to D, but maybe.
>>>
>>> https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
>>
>> Doh!
>>
>> It's already in deimos, but you need to call PyDateTime_IMPORT(); before calling the conversion function.
>>
>> So all that is needed is the following:
>>
>> module example;
>> import std.datetime;
>> import std.stdio;
>> import pyd.pyd;
>> import std.conv;
>> import deimos.python.datetime;
>>
>>
>> DateTime foo()
>> {
>>     return DateTime(1999,1,1);
>> }
>> extern(C) void PydMain() {
>>     ex_d_to_python((DateTime dt) => PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));
>>     def!foo();
>>     module_init();
>>     PyDateTime_IMPORT();
>> }
>>
>> Ideally, shouldn't PyD do this for you ?  I think the code is already there in make_object.d, but it doesn't work - maybe because PyDateTime_IMPORT() has not been called if it needs to be.
>>
>> Thanks.
>>
>>
>> Laeeth.
>
> Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date?
>
> Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?

I just put 0 in the microseconds field as I was too tired to look up how D does that for now and it wasn't so important for my proof of concept.  But python has microseconds (though not smaller, I think).  Numpy and pandas have their own datetime64 concept - I have not so nice memories of using these, but I am sure it must be possible to convert at C API level.  Worst case if there are no system calls to do the conversion, just hack up the current binary representation of the struct and deal with the mess when they change the version.  But I wouldn't suggest you put that in PyD unless we can do it via official call!  I do think regular datetime.datetime should be in there, though.  I think it would probably only need a handful of lines added to the file I mentioned - I thought they were already there, but that was my own change!

I am not sure where the init date call should go, and I am wary of tinkering with PyD when I don't yet know the codebase, nor the Python C API.  I think someone who knows it could do it in a few minutes.

Is it a problem not having sub micro second resolution?  It might be for some people, but then it's a core problem, and they probably have the budget to do it themselves and might have their own date representation anyway (which nonetheless might most conveniently be mapped to datetime64 given all the python libraries).

However, it's not my use case.  (I guess those who need are scientists and investors who analyze data at a high-frequency level, which is not me - 1 minute bars are enough for now, and my holding period will be much longer).

It's incrementally a big improvement to start by adding datetime.datetime...


Laeeth.

> Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date?
>
> Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?

https://docs.python.org/3/c-api/datetime.html

The Python API has microseconds passed as an int in last argument.  You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better.

On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free.  There are libraries that implement datetimes with a timezone.

So sensible options are:

1. datetime.datetime<->std.datetime.DateTime throwing away fractions of a second for python -> D.
2. datetime.datetime<->std.datetime.SysTime presuming python time is UTC and
2a) throwing away SysTime timezone and assuming it is UTC
2b) converting SysTime to a UTC time (and presuming python time is UTC).

I would go for 1 as default and allowing 2b as an option.  The problem is that if you implement the mapping in the pyd main code then user cannot override it in custom conversion code because it is only called if the inbuilt conversion fails.

So maybe 2b is the only sensible feasible option.

On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
>> Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date?
>>
>> Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?
>
> https://docs.python.org/3/c-api/datetime.html
>
> The Python API has microseconds passed as an int in last argument.  You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better.
>
> On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free.

both datetime.datetime and numpy.datetime64 have timezone support.

On Friday, 3 July 2015 at 21:46:09 UTC, John Colvin wrote:
> On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
>>> Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date?
>>>
>>> Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?
>>
>> https://docs.python.org/3/c-api/datetime.html
>>
>> The Python API has microseconds passed as an int in last argument.  You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better.
>>
>> On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free.
>
> both datetime.datetime and numpy.datetime64 have timezone support.


Great - in that case 2b makes sense.

On Saturday, 4 July 2015 at 00:14:51 UTC, Laeeth Isharc wrote:
> On Friday, 3 July 2015 at 21:46:09 UTC, John Colvin wrote:
>> On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
>>>> Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date?
>>>>
>>>> Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?
>>>
>>> https://docs.python.org/3/c-api/datetime.html
>>>
>>> The Python API has microseconds passed as an int in last argument.  You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better.
>>>
>>> On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free.
>>
>> both datetime.datetime and numpy.datetime64 have timezone support.
>
>
> Great - in that case 2b makes sense.

or rather, not 2b, but just mapping python + numpy to a SysTime