Maybe the answer is in the valgrind logs, its alot to read. But isn't it possible to put a write breakpoint in the accounts arrays and step through the code until a zero is written.

On 10/21/20 12:24 PM, donallen wrote:
> The problem is that at some point, the verifier starts spewing bogus error messages about what it is seeing in the tree. Oddly, putting in debugging writelns results in the error messages not occurring -- a Heisenbug.  But working with gdb, I found that the account structs after the error messages start are zeroed. Turning on gc profiling tells me that 3 gcs have occurred. Disabling the gc results in the program running correctly -- no error messages (and I know the database has no errors because the C version, which has been around for awhile, confirms that the db is well formed).
> 
> I could post a PR, but I'm not sure that this is a bug. It could easily be a misunderstanding by me of D and its memory management. So I thought I'd try this post first, hoping that one of you who knows the language better than I do could point out a problem with my code. I do need to resolve this or abandon this project.
> 
> Thanks in advance for any help.

There seems to be nothing wrong in this code that I can see.

But of course, it's out of context, so it's hard to see if there are issues elsewhere.

These kinds of corruption bugs are really hard to track down.

What does your Account struct look like?

-Steve

On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:
> I tried this. With the gc on, the problem still occurs.
>
> I am working on 64-bit Arch Linux systems, up-to-date, DMD64 D Compiler v2.094.0

Have you tried using a different compiler? I know with the big languages like C or Java the meme is "it's never a compiler bug", but D is developed with *far* fewer resources, and compiler bugs are, sadly, not rare.

For example, DMD 2.094+ have this still-unfixed back-end regression that may possibly be causing your problem:

https://issues.dlang.org/show_bug.cgi?id=21325

The LDC and GDC back-ends are much more reliable (and better in other ways) than DMD's because they are tested and maintained by much larger communities.

I recommend giving the latest LDC a try, as well as a significantly older compiler of your choice, before assuming that the problem is in your code, or in the garbage collector.

On Thursday, 22 October 2020 at 18:24:57 UTC, Steven Schveighoffer wrote:
> On 10/21/20 12:24 PM, donallen wrote:
>> The problem is that at some point, the verifier starts spewing bogus error messages about what it is seeing in the tree. Oddly, putting in debugging writelns results in the error messages not occurring -- a Heisenbug.  But working with gdb, I found that the account structs after the error messages start are zeroed. Turning on gc profiling tells me that 3 gcs have occurred. Disabling the gc results in the program running correctly -- no error messages (and I know the database has no errors because the C version, which has been around for awhile, confirms that the db is well formed).
>> 
>> I could post a PR, but I'm not sure that this is a bug. It could easily be a misunderstanding by me of D and its memory management. So I thought I'd try this post first, hoping that one of you who knows the language better than I do could point out a problem with my code. I do need to resolve this or abandon this project.
>> 
>> Thanks in advance for any help.
>
> There seems to be nothing wrong in this code that I can see.
>
> But of course, it's out of context, so it's hard to see if there are issues elsewhere.
>
> These kinds of corruption bugs are really hard to track down.
>
> What does your Account struct look like?

    struct Account
    {
        string name;
        string path;
        string guid;
        string commodity_guid;
        int flags;
    }


>
> -Steve

On Thursday, 22 October 2020 at 18:54:24 UTC, tsbockman wrote:
> On Thursday, 22 October 2020 at 04:02:10 UTC, donallen wrote:
>> I tried this. With the gc on, the problem still occurs.
>>
>> I am working on 64-bit Arch Linux systems, up-to-date, DMD64 D Compiler v2.094.0
>
> Have you tried using a different compiler?

Yes -- to no avail:
dca@franz:~/Software/newcash_d/verifier$ make
cd ../library ; make
make[1]: Entering directory '/home/dca/Software/newcash_d/library'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/dca/Software/newcash_d/library'
#dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d ../library/lib.o
ldc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o
../library/lib.o:lib.d:function _D3std9exception__T7bailOutHTC9ExceptionZQwFNaNfAyamMAxaZv: error: undefined reference to '_d_throwdwarf'
../library/lib.o:lib.d:function _D3std3utf__T10decodeImplVbi1VEQBd8typecons__T4FlagVAyaa19_7573655265706c6163656d656e744463686172ZQCai0TAxwZQDrFNaQkKmZw: error: undefined reference to '_d_throwdwarf'
../library/lib.o:lib.d:function _D3std6format__T14formattedWriteTSQBg5array__T8AppenderTAyaZQoTaTAaZQCaFNaNfKQBsMxAaQtZk: error: undefined reference to '_d_throwdwarf'
../library/lib.o:lib.d:function _D3std6format__T6getNthVAyaa13_696e7465676572207769647468SQCe6traits10isIntegralTiTAaZQCsFNaNfkQmZi: error: undefined reference to '_d_throwdwarf'
../library/lib.o:lib.d:function _D3std6format__T13formatElementTSQBf5stdio4File17LockingTextWriterTAyaTaZQCfFNfKQBwQqMKxSQDjQDi__T10FormatSpecTaZQpZv: error: undefined reference to '__dmd_begin_catch'
../library/lib.o:lib.d:function _D3std6format__T15formatValueImplTSQBh5stdio4File17LockingTextWriterTwTaZQCfFNfKQBuwMKxSQDiQDh__T10FormatSpecTaZQpZv: error: undefined reference to '_memset32'
../library/lib.o:lib.d:DW.ref.__dmd_personality_v0: error: undefined reference to '__dmd_personality_v0'
../library/lib.o:lib.d:function _D3std9algorithm9iteration__T8splitterVAyaa6_61203d3d2062TQtTQwZQBjFQBdQBgZ6Result11__xopEqualsFKxSQDtQDsQDl__TQDeVQCya6_61203d3d2062TQDrTQDvZQEjFQEdQEgZQDaKxQCiZb: error: undefined reference to '_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb'
../library/lib.o:lib.d:function _D3std9algorithm9iteration__T8splitterVAyaa6_61203d3d2062TQtTQwZQBjFQBdQBgZ6Result11__xopEqualsFKxSQDtQDsQDl__TQDeVQCya6_61203d3d2062TQDrTQDvZQEjFQEdQEgZQDaKxQCiZb: error: undefined reference to '_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb'
../library/lib.o:lib.d:function _D3std4conv__T10emplaceRefTAyaTQeTQhZQxFKQoKQrZ1S11__xopEqualsFKxSQCmQCl__TQCjTQCaTQCeTQCiZQCzFKQCrKQCvZQCfKxQBsZb: error: undefined reference to '_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb'
../library/lib.o:lib.d:function _D4core8internal5array8equality__T8__equalsTxAyaTxQfZQtFNaNbNiNfMAxQwMQfZb: error: undefined reference to '_D4core8internal5array8equality__T8__equalsTyaTyaZQqFNaNbNiNfMAyaMQeZb'
collect2: error: ld returned 1 exit status
Error: /usr/bin/cc failed with status: 1
make: *** [makefile:11: verifier] Error 1
dca@franz:~/Software/newcash_d/verifier$ make
cd ../library ; make
make[1]: Entering directory '/home/dca/Software/newcash_d/library'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/dca/Software/newcash_d/library'
#dmd -g -profile=gc -I=../library -L='-lsqlite3' verifier.d ../library/lib.o
gdc -g -I=../library -L='-lsqlite3' verifier.d ../library/lib.o
verifier.d:4:8: error: module lib is in file 'lib.d' which cannot be read
    4 | import lib;
      |        ^
import path[0] = /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/include/d
make: *** [makefile:11: verifier] Error 1
dca@franz:~/Software/newcash_d/verifier$ ls -l ../library/lib.d
-rw-r--r-- 1 dca allen 12244 Oct 20 11:19 ../library/lib.d
dca@franz:~/Software/newcash_d/verifier$

I hate to say this, because so much of this project makes sense to me -- the design of the language, the effort that has been put into documentation, the helpfulness of the community -- but I think I have to throw in the towel. I can't spend any more time on this and it appears that D is just not solid enough for the work I'm trying to do with it.

On Thursday, 22 October 2020 at 19:34:18 UTC, donallen wrote:
>> What does your Account struct look like?
>
>     struct Account
>     {
>         string name;
>         string path;
>         string guid;
>         string commodity_guid;
>         int flags;
>     }

Do you pass any variable from D to C? and how does those variables look like?

And if you are passing any D object to C, do you always hold a reference to them before the C returns? I mean: the D's gc think the object is no longer used, but actually is used by the C side.

On Thursday, 22 October 2020 at 19:57:08 UTC, mw wrote:
> Do you pass any variable from D to C? and how does those variables look like?
>
> And if you are passing any D object to C, do you always hold a reference to them before the C returns? I mean: the D's gc

s/before/after

> think the object is no longer used, but actually is used by the C side.

On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:
> [snip]
>
> I hate to say this, because so much of this project makes sense to me -- the design of the language, the effort that has been put into documentation, the helpfulness of the community -- but I think I have to throw in the towel. I can't spend any more time on this and it appears that D is just not solid enough for the work I'm trying to do with it.

From the community's perspective, if there is a compiler bug that is causing your problem, then it will neither be found nor fixed, which is disappointing.

In general, you will always get better help when you are able to provide either the complete code or a reduced example and what commands you used to run it, as was mentioned above.

On Thursday, 22 October 2020 at 21:00:41 UTC, jmh530 wrote:
> On Thursday, 22 October 2020 at 19:38:42 UTC, donallen wrote:
>> [snip]
>>
>> I hate to say this, because so much of this project makes sense to me -- the design of the language, the effort that has been put into documentation, the helpfulness of the community -- but I think I have to throw in the towel. I can't spend any more time on this and it appears that D is just not solid enough for the work I'm trying to do with it.
>
> From the community's perspective, if there is a compiler bug that is causing your problem, then it will neither be found nor fixed, which is disappointing.
>
> In general, you will always get better help when you are able to provide either the complete code or a reduced example and what commands you used to run it, as was mentioned above.

I've already provided valgrind output that shows the gc referencing an uninitialized variable.

Please understand -- I don't have infinite time to spend on this and I do have alternatives for this work, e.g., nim or go or chez scheme. Given the valgrind output indicating what looks like a serious error, I'm not going to put the effort into trying to reduce this error to an example that I can share fully unless and until someone convinces me that what valgrind is saying doesn't indicate a bug in the gc.

On 23/10/2020 4:56 AM, donallen wrote:
> ==3854==    by 0x1B7E98: void verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account, int) (verifier.d:272)
> ==3854==  Uninitialised value was created by a stack allocation
> ==3854==    at 0x1B77FC: void verifier.main(immutable(char)[][]).walk_account_tree(verifier.main(immutable(char)[][]).Account, int) (verifier.d:84)
> ==3854==
> ==3854== Conditional jump or move depends on uninitialised value(s)

That is coming up an awful lot.

We'd need walk_account_tree source, to know if its doing something funky.

But upon reviewing how the entire thing is working, I think we need one_row and next_row_available_p source as well. Oh and reset_stmt too.