http://d.puremagic.com/issues/show_bug.cgi?id=2917

           Summary: std.date fails for all years before 1970
           Product: D
           Version: 2.029
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: Phobos
        AssignedTo: bugzilla@digitalmars.com
        ReportedBy: ghaecker@idworld.net


For years prior to 1970, conversion from string to d_time returns d_time_nan. Strings are parsed correctly, but conversion to d_time fails the test at line 440 of date.d in the makeDay() function.

Commenting out lines 440 through 445 allows a d_time to be returned, but the value is skewed.  Line 447 works for years >= 1970:

    return day(t) + date - 1;

but for years < 1970, this adjustment:

    return day(t) + date;

returns the expected value.

There is also another problem: attempting to convert a negative d_time back to a string requires adding ticksPerSecond to d_time to return the original value from secFromTime(), which, of course, trashes the ms value.  Adding the value of  ticksPerSecond appears to work for all negative d_time values, except for -1000.

NOTE: I was only working with a resolution of seconds, so the effect on milliseconds/ticks was not taken into account.

I'm new to D, so please don't be too critical of my code.  I'm including it so you can see the kludges I added to work around these issues.  BTW, an addMonths() function (more elegant than mine) would be a welcome addition to std.date.  My attempt at writing one along with a unittest section was what revealed this bug.  Unittest is a brilliant addition to the language.

import std.stdio,
    std.date,
    std.conv,
    std.random;

int main(char[][] args)
{
    bool pass;
    int adj;
    string tscmp;
    d_time tt, rtt;

    string[] testdates = [
        "1969-12-31 23:59:58",
        "1969-12-31 23:59:59",
        "1970-01-01 00:00:00",
        "1970-01-01 00:00:01",
        "1970-01-01 00:00:02"
        ];
    writefln("Crossing the 1970 epoch...\n");
    foreach(int i, string s; testdates) {
        tt = dbgParse(s ~ " GMT");
        // KLUDGE
        adj = (i < 2) ? ticksPerSecond - 5: 0;
        rtt = tt + adj;
        tscmp = toDbDateTime(rtt);
        pass = (tscmp == s);
        writefln("%s -> %5d", s, tt);
        writefln("%s <- %5d   %s\n", tscmp, rtt, pass);
    }

    // Random tests
    auto gen = Mt19937(unpredictableSeed);
    int yr, mh, dy, hr, me, sd, days;
    int yr2, mh2, dy2, hr2, me2, sd2;
    int tested, passed, before1970, show = 7;
    string ts;
    bool first;
    for (int i; i < 10000; i++) {
        tested++;
        yr = uniform(1601, 2400, gen);
        if (yr < 1970) before1970++;
        mh = uniform(1, 12, gen);
        days = daysInMonth(yr, mh);
        dy = uniform(1, days, gen);
        hr = uniform(0, 23, gen);
        me = uniform(0, 59, gen);
        sd = uniform(0, 59, gen);
        ts = std.string.format("%d-%02d-%02d %02d:%02d:%02d", yr, mh, dy, hr,
me, sd);
        tt = dbgParse(ts ~ " GMT");
        // KLUDGE
        adj = (yr < 1970) ? ticksPerSecond - 5: 0;
        rtt = tt + adj;
        tscmp = toDbDateTime(rtt);
        pass = (ts == tscmp);
        if (! pass || show > 0) {
            if (! first) {
                first = true;
                writefln("Showing first %d random samples...\n", show);
            }
            show--;
            writefln("%s -> %16d", ts, tt);
            writefln("%s <- %16d   %s\n", tscmp, rtt, pass);
        }
        if (pass) passed++;
    }

    writefln("");
    writefln("Tested: %7d", tested);
    writefln("< 1970: %7d", before1970);
    writefln("Passed: %7d", passed);

    return 0;
}

string toDbDateTime(d_time t) {
    auto yr = yearFromTime(t);
    auto mh = monthFromTime(t) + 1;
    auto dy = dateFromTime(t);
    auto hr = hourFromTime(t);
    auto me = minFromTime(t);
    auto sd = secFromTime(t);
    return std.string.format("%d-%02d-%02d %02d:%02d:%02d", yr, mh, dy, hr, me,
sd);
}

d_time dbgParse(string s)
{
    try
    {
        Date dp;
        dp.parse(s);
        auto time = makeTime(dp.hour, dp.minute, dp.second, dp.ms);
        if (dp.tzcorrection == int.min)
            time -= localTZA;
        else
        {
            time += cast(d_time)(dp.tzcorrection / 100) * msPerHour +
                    cast(d_time)(dp.tzcorrection % 100) * msPerMinute;
        }
        auto day = dbgMakeDay(dp.year, dp.month - 1, dp.day);
        d_time result = makeDate(day,time);
        return timeClip(result);
    }
    catch
    {
        return d_time_nan;                // erroneous date string
    }
}

d_time dbgMakeDay(d_time year, d_time month, d_time date)
{

    int[12] mdays =
        [ 0,31,59,90,120,151,181,212,243,273,304,334 ];
    const y = cast(int)(year + floor(month, 12));
    const m = dmod(month, 12);

    const leap = leapYear(y);
    auto t = timeFromYear(y) + cast(d_time) mdays[m] * msPerDay;
    if (leap && month >= 2)
        t += msPerDay;

    /* DISABLED: this test failes for y < 1970

    if (yearFromTime(t) != y ||
        monthFromTime(t) != m ||
        dateFromTime(t) != 1)
    {
        writefln("Bad match");
        writefln("%d : %d, %d : %d, %d : %d", y, yearFromTime(t), m,
monthFromTime(t), 1, dateFromTime(t));

        return  d_time_nan;
    }

    */

    // KLUDGE
    int adj = (y < 1970) ? 0 : 1;

    return day(t) + date - adj;

}

d_time addMonths(d_time dt, int months) {
    int days = 0;
    if (months <> 0) {
        int mon = monthFromTime(dt); // zero-based month
        int yr = yearFromTime(dt);
        while (months > 0) {
            days += daysInMonth(yr, mon + 1); // one-based month
            mon++;
            if (mon > 11) {
                mon = 0;
                yr++;
            }
            months--;
        }
        while (months < 0) {
            mon--;
            if (mon < 0) {
                mon = 11;
                yr--;
            }
            days -= daysInMonth(yr, mon + 1);  // one-based month
            months++;
        }
    }
    return dt += days * cast(d_time) ticksPerDay;
}

unittest {

    d_time dt1, dt2, dt3, dt4;
    dt1 = dbgParse("1979-10-31 13:14:15 GMT");
    dt2 = dbgParse("1981-05-31 13:14:15 GMT");
    dt3 = addMonths(dt1, 19);
    assert(dt3 == dt2, text(dt3, " != ", dt2));
    dt4 = addMonths(dt3, -19);
    assert(dt4 == dt1, text(dt4, " != ", dt1));

    dt1 = dbgParse("1969-10-31 13:14:15 GMT");
    dt2 = dbgParse("1971-05-31 13:14:15 GMT");
    dt3 = addMonths(dt1, 19);
    assert(dt3 == dt2, text(dt3, " != ", dt2));
    dt4 = addMonths(dt3, -19);
    assert(dt4 == dt1, text(dt4, " != ", dt1));

    dt1 = dbgParse("1599-10-31 13:14:15 GMT");
    dt2 = dbgParse("1601-05-31 13:14:15 GMT");
    dt3 = addMonths(dt1, 19);
    assert(dt3 == dt2, text(dt3, " != ", dt2));
    dt4 = addMonths(dt3, -19);
    assert(dt4 == dt1, text(dt4, " != ", dt1));

    dt1 = dbgParse("2399-10-31 13:14:15 GMT");
    dt2 = dbgParse("2401-05-31 13:14:15 GMT");
    dt3 = addMonths(dt1, 19);
    assert(dt3 == dt2, text(dt3, " != ", dt2));
    dt4 = addMonths(dt3, -19);
    assert(dt4 == dt1, text(dt4, " != ", dt1));

}


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2917





------- Comment #1 from ghaecker@idworld.net  2009-05-04 05:39 -------
Created an attachment (id=348)
 --> (http://d.puremagic.com/issues/attachment.cgi?id=348&action=view)
patch for std.date.d v. 2.029 fixes issues with negative time values

Rather than whine about the date issues, I decided to lend a hand and address the issues myself.  If you find the modifications useful you are welcome to use the code in any way you see fit.  I only added my name as a modifier so that Walter won't get blamed for any mistakes I made.  You may remove my name or leave it, as you see fit.  The changes I made aren't anything special; just some grunt work that needed doing.

I tinkered with some of the more vital functions in this module.  In
particular, I modified floor(), dmod(), day(), and dayFromYear().  The latter
got a little messy with year value 0 and lower.  Of course, pushing the
Gregorian calendar back that far back is purely hypothetical.  I added unittest
sections to day() and dayFromYear, as the proper working of these functions is
vital to the entire module.

I threw in a bit of trivial tidying up as well.  The calls to toInteger() and timeClip() were removed, since they don't do anything.  I left the functions in place, in case some existing code calls them.  I changed the name of the 'day' variable that called the day() function, as it seems like a bad idea to duplicate the name in the same scope.

I added an overload for dateFromTime(d_time t, int months).  In most cases,
where it's used in the module, the value for month is already known, so there's
no need for dateFromTime() to recalculate it.

My addition of toDbDateTime() is trivial, but I found it useful in testing, and it has practical value.  It's a fairly common format, and it doesn't add much weight to the module.

I also added two other functions: addMonths() and addYears().  These are both
common date/time-manipulation tasks, which aren't as straightforward as adding
fixed values like days or weeks.  addMonths() is rather clunky, but it gets the
job done.  It never cranks through more than 11 months on any call.  Beyond 11,
it calls addYears() so that adding 60 months is almost as quick as adding 5
years.

I've rebuilt the 2.029 library with this patch to std.date on linux.  All appears to be working well.  Someone needs to test the DosDate functions to make sure my changes didn't break them.  There were no unit tests for them.  I don't have D installed on a Windows box.

Thanks in advance for considering these changes.


-- 

http://d.puremagic.com/issues/show_bug.cgi?id=2917





--- Comment #2 from Glenn Haecker <ghaecker@idworld.net>  2009-05-14 16:20:06 PDT ---
Created an attachment (id=366)
 --> (http://d.puremagic.com/issues/attachment.cgi?id=366)
patch for std.date.d v. 2.030 fixes issues with negative time values

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=2917





--- Comment #3 from Glenn Haecker <ghaecker@idworld.net>  2009-05-14 17:06:15 PDT ---
This bit of code shows the bug.  There are larger effects as well.

-------------------------------------------
    string[] testdt = [
        "1969-12-31 23:59:58 UTC",
        "1969-12-31 23:59:59 UTC",
        "1970-01-01 00:00:00 UTC", // epoch
        "1970-01-01 00:00:01 UTC",
        "1970-01-01 00:00:02 UTC"];

    foreach(int i, string s; testdt)
        writefln("%s = %5d", s, parse(s));

    d_time[] times = [-2000,-1000,0,1000,2000];
    foreach(tm; times)
        writefln("%5d = %s", tm, toUTCString(tm));
-------------------------------------------

Unpatched results:

1969-12-31 23:59:58 UTC = -9223372036854775808 (d_time_nan)
1969-12-31 23:59:59 UTC = -9223372036854775808 (d_time_nan)
1970-01-01 00:00:00 UTC =     0
1970-01-01 00:00:01 UTC =  1000
1970-01-01 00:00:02 UTC =  2000
-2000 = Wed, 31 Dec 1969 23:59:57 UTC  (off by 1 sec)
-1000 = Wed, 31 Dec 1969 23:59:58 UTC  (off by 1 sec)
    0 = Thu, 01 Jan 1970 00:00:00 UTC
 1000 = Thu, 01 Jan 1970 00:00:01 UTC
 2000 = Thu, 01 Jan 1970 00:00:02 UTC

Patched results:

1969-12-31 23:59:58 UTC = -2000
1969-12-31 23:59:59 UTC = -1000
1970-01-01 00:00:00 UTC =     0
1970-01-01 00:00:01 UTC =  1000
1970-01-01 00:00:02 UTC =  2000
-2000 = Wed, 31 Dec 1969 23:59:58 UTC
-1000 = Wed, 31 Dec 1969 23:59:59 UTC
    0 = Thu, 01 Jan 1970 00:00:00 UTC
 1000 = Thu, 01 Jan 1970 00:00:01 UTC
 2000 = Thu, 01 Jan 1970 00:00:02 UTC

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=2917



--- Comment #4 from Glenn Haecker <ghaecker@idworld.net> 2010-01-25 02:59:44 PST ---
Created an attachment (id=555)
patch for std.date.d v.2.039 fixes issues with negative time values

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=2917



--- Comment #5 from Glenn Haecker <ghaecker@idworld.net> 2010-01-25 03:12:00 PST ---
Created an attachment (id=556)
date_assert.d contains assertions for date.d v2.039 patch

date_assert.d contains test cases, both to demonstrate what's wrong with std.date and to validate modifications by the supplied patch.

Please note that this file and the provided patch were created on Linux and contain UNIX-style line endings.

In order to compile date_assert.d with v2.039 phobos before the patch is applied, you'll need to comment out lines 68 through 84.  These lines contain tests that call addMonths(), which is not available before the patch is applied.

You can search for "fails w/o patch" to identify the assert() tests that fail
with out the patch.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=2917


downs <default_357-line@yahoo.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |default_357-line@yahoo.de


--- Comment #6 from downs <default_357-line@yahoo.de> 2010-05-22 11:07:48 PDT ---
This bug report is symptomatic for what's wrong with the D development process, in my opinion. There's a bug, there's a fix, there's a patch, there's a distinct lack of feedback from Walter/Andrei. No wonder so few people work on D; a situation like this would drive any contributor away.

My apologies to Mr. Haecker.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=2917


Lars T. Kyllingstad <bugzilla@kyllingen.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |bugzilla@kyllingen.net
         Resolution|                            |WONTFIX


--- Comment #7 from Lars T. Kyllingstad <bugzilla@kyllingen.net> 2011-02-03 00:13:24 PST ---
std.date has now been superseded by std.datetime.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------