Jump to page: 1 2 3
Thread overview
Dynamic Linking & Memory Management
Jan 26, 2005
Benji Smith
Jan 26, 2005
Kris
Jan 26, 2005
Matthew
Jan 26, 2005
Kris
Jan 27, 2005
Matthew
Jan 27, 2005
Matthew
Jan 27, 2005
John Reimer
Jan 27, 2005
Matthew
Jan 26, 2005
Matthew
Jan 26, 2005
Walter
Jan 27, 2005
Kris
Jan 27, 2005
John Reimer
Jan 27, 2005
Matthew
Jan 27, 2005
Kris
Jan 27, 2005
Matthew
Jan 27, 2005
Kris
Jan 27, 2005
Matthew
Jan 27, 2005
Kris
Jan 28, 2005
Matthew
Jan 28, 2005
Kris
Jan 28, 2005
Walter
Jan 28, 2005
Kris
Jan 28, 2005
Walter
Jan 28, 2005
Dave
Jan 28, 2005
pragma
Jan 28, 2005
Kris
Jan 28, 2005
pragma
Jan 28, 2005
Kris
Jan 27, 2005
Benji Smith
January 26, 2005
I've been interested to read some of the recent discussion about the D garbage collector, and I'd like to describe my current Java project to give some perspective on what I think is ideal memory management.

I'm working on a technical analysis and simulation application for historical stock market data. And despite the fact that I've written about ten thousand lines of code myself, I'm using third-party libraries for many aspects of the project. First of all, I'm using JDBC drivers for MySQL and MS SQL Server. The MySQL drivers consist of 3 different JAR files that I include in my application's classpath. The MS SQL Server drivers are contained in another 3 JAR files. I'm also using the Xerces XML parser from the Apache group (3 more JARs), the JFreeChart graphical charting compontents (5 more JARs), the JUnit testing framework (1 JAR), a GNU commandline parsing library (1 JAR), and a few other miscellaneous libraries. All told, I'm importing functionality from more than fifteen different libraries. And the application is still very small; by the time I’ve finished developing it, I’ll probably be using twice as many libraries.

But when I write my code, I can write it as though I'm statically linking with each of those libraries. I don't need to use special export semantics when I need to call code from any of those third party vendors. And with a few of those libraries (the commandline parsing library in particular), I may end up writing my own implementation. When I do, I won't have to change the semantics to reflect the fact that I'm no longer using a compiled library. The rest of my code can be completely agnostic to whether I'm linking with source files, compiled class files, class files bundled into a JAR package, classes generated at runtime through reflection hooks, or classes loaded dynamically using a custom classloader. Since my application needs to support third-party plug-in development (users can load their own classes as custom charting indicators), dynamic runtime loading of classes is essential to my design.

Ubiquitous static-linking would not be an option for me with this application.

But there’s another important issue, too: debugging.

Last week I discovered a memory leak somewhere in my application. If I allowed some of the analysis code to run for a few hours--combing through all 18 million data points from the last 25 years of stock market data--the heap would grow from its initial allocation (8 MB) to its maximum allocation (256 MB). Luckily for me, all of those allocations take place within a single virtual machine, which uses a single garbage collector to manage all of the memory from all of the libraries I'm using. That allows me to use a profiling application to monitor the allocations of all the objects in the heap and--much more importantly--to find out which objects are holding references to which other objects. Within moments, I could see that the JDBC allocations were getting cleaned up properly, but one of my custom collection classes was failing—about 2% of the time—to release object references it was no longer using. After an hour or so of tinkering with the profiler, I was able to track down and fix that memory leak. The application now uses a steady 12 MB of heap memory, no matter how long it runs.

If I were building the same application with D, there would be fifteen different garbage collectors operating in fifteen different heap-spaces, and the objects allocated within one heap space might be referenced by objects in another heap space, each managed by a different garbage collector. It would be much more difficult to develop a heap profiling tool that could successfully allow a developer to navigate through such a fragmented heap space, particularly if the developer needed to figure out which GC was supposed to collect each out-of-reach object. Tracking down and fixing that memory leak probably would have taken a lot longer than an hour and a half.

Consequently, I strongly support the development of a model within D that allows for a single GC instance per process. Any other scenario sounds like a development & debugging nightmare.

January 26, 2005
The gods be praised! Good example, Benji.

This is why it's been noted that D would not pass muster, in much of the all-important commercial field.

- Kris


In article <ce4gv0dk6v1qgde6o94msecgi4huasdhga@4ax.com>, Benji Smith says...
>
>I've been interested to read some of the recent discussion about the D garbage collector, and I'd like to describe my current Java project to give some perspective on what I think is ideal memory management.
>
>I'm working on a technical analysis and simulation application for historical stock market data. And despite the fact that I've written about ten thousand lines of code myself, I'm using third-party libraries for many aspects of the project. First of all, I'm using JDBC drivers for MySQL and MS SQL Server. The MySQL drivers consist of 3 different JAR files that I include in my application's classpath. The MS SQL Server drivers are contained in another 3 JAR files. I'm also using the Xerces XML parser from the Apache group (3 more JARs), the JFreeChart graphical charting compontents (5 more JARs), the JUnit testing framework (1 JAR), a GNU commandline parsing library (1 JAR), and a few other miscellaneous libraries. All told, I'm importing functionality from more than fifteen different libraries. And the application is still very small; by the time I’ve finished developing it, I’ll probably be using twice as many libraries.
>
>But when I write my code, I can write it as though I'm statically linking with each of those libraries. I don't need to use special export semantics when I need to call code from any of those third party vendors. And with a few of those libraries (the commandline parsing library in particular), I may end up writing my own implementation. When I do, I won't have to change the semantics to reflect the fact that I'm no longer using a compiled library. The rest of my code can be completely agnostic to whether I'm linking with source files, compiled class files, class files bundled into a JAR package, classes generated at runtime through reflection hooks, or classes loaded dynamically using a custom classloader. Since my application needs to support third-party plug-in development (users can load their own classes as custom charting indicators), dynamic runtime loading of classes is essential to my design.
>
>Ubiquitous static-linking would not be an option for me with this application.
>
>But there’s another important issue, too: debugging.
>
>Last week I discovered a memory leak somewhere in my application. If I allowed some of the analysis code to run for a few hours--combing through all 18 million data points from the last 25 years of stock market data--the heap would grow from its initial allocation (8 MB) to its maximum allocation (256 MB). Luckily for me, all of those allocations take place within a single virtual machine, which uses a single garbage collector to manage all of the memory from all of the libraries I'm using. That allows me to use a profiling application to monitor the allocations of all the objects in the heap and--much more importantly--to find out which objects are holding references to which other objects. Within moments, I could see that the JDBC allocations were getting cleaned up properly, but one of my custom collection classes was failing—about 2% of the time—to release object references it was no longer using. After an hour or so of tinkering with the profiler, I was able to track down and fix that memory leak. The application now uses a steady 12 MB of heap memory, no matter how long it runs.
>
>If I were building the same application with D, there would be fifteen different garbage collectors operating in fifteen different heap-spaces, and the objects allocated within one heap space might be referenced by objects in another heap space, each managed by a different garbage collector. It would be much more difficult to develop a heap profiling tool that could successfully allow a developer to navigate through such a fragmented heap space, particularly if the developer needed to figure out which GC was supposed to collect each out-of-reach object. Tracking down and fixing that memory leak probably would have taken a lot longer than an hour and a half.
>
>Consequently, I strongly support the development of a model within D that allows for a single GC instance per process. Any other scenario sounds like a development & debugging nightmare.
>


January 26, 2005
Hear, hear! (And thanks for spending the time to write such an eloquent and persuasive post.)

I not only agree with everything you say, I think we should overtly state that without support such as this, D is Doomed: Dead, Duck-like, save for small self-contained utility programs (for which C, never mind C++, suffices adequately, IMO).


"Benji Smith" <dlanguage@xxagg.com> wrote in message news:ce4gv0dk6v1qgde6o94msecgi4huasdhga@4ax.com...
> I've been interested to read some of the recent discussion about the D garbage collector, and I'd like to describe my current Java project to give some perspective on what I think is ideal memory management.
>
> I'm working on a technical analysis and simulation application for historical stock market data. And despite the fact that I've written about ten thousand lines of code myself, I'm using third-party libraries for many aspects of the project. First of all, I'm using JDBC drivers for MySQL and MS SQL Server. The MySQL drivers consist of 3 different JAR files that I include in my application's classpath. The MS SQL Server drivers are contained in another 3 JAR files. I'm also using the Xerces XML parser from the Apache group (3 more JARs), the JFreeChart graphical charting compontents (5 more JARs), the JUnit testing framework (1 JAR), a GNU commandline parsing library (1 JAR), and a few other miscellaneous libraries. All told, I'm importing functionality from more than fifteen different libraries. And the application is still very small; by the time I've finished developing it, I'll probably be using twice as many libraries.
>
> But when I write my code, I can write it as though I'm statically linking with each of those libraries. I don't need to use special export semantics when I need to call code from any of those third party vendors. And with a few of those libraries (the commandline parsing library in particular), I may end up writing my own implementation. When I do, I won't have to change the semantics to reflect the fact that I'm no longer using a compiled library. The rest of my code can be completely agnostic to whether I'm linking with source files, compiled class files, class files bundled into a JAR package, classes generated at runtime through reflection hooks, or classes loaded dynamically using a custom classloader. Since my application needs to support third-party plug-in development (users can load their own classes as custom charting indicators), dynamic runtime loading of classes is essential to my design.
>
> Ubiquitous static-linking would not be an option for me with this application.
>
> But there's another important issue, too: debugging.
>
> Last week I discovered a memory leak somewhere in my application. If I allowed some of the analysis code to run for a few hours--combing through all 18 million data points from the last 25 years of stock market data--the heap would grow from its initial allocation (8 MB) to its maximum allocation (256 MB). Luckily for me, all of those allocations take place within a single virtual machine, which uses a single garbage collector to manage all of the memory from all of the libraries I'm using. That allows me to use a profiling application to monitor the allocations of all the objects in the heap and--much more importantly--to find out which objects are holding references to which other objects. Within moments, I could see that the JDBC allocations were getting cleaned up properly, but one of my custom collection classes was failing-about 2% of the time-to release object references it was no longer using. After an hour or so of tinkering with the profiler, I was able to track down and fix that memory leak. The application now uses a steady 12 MB of heap memory, no matter how long it runs.
>
> If I were building the same application with D, there would be fifteen different garbage collectors operating in fifteen different heap-spaces, and the objects allocated within one heap space might be referenced by objects in another heap space, each managed by a different garbage collector. It would be much more difficult to develop a heap profiling tool that could successfully allow a developer to navigate through such a fragmented heap space, particularly if the developer needed to figure out which GC was supposed to collect each out-of-reach object. Tracking down and fixing that memory leak probably would have taken a lot longer than an hour and a half.
>
> Consequently, I strongly support the development of a model within D that allows for a single GC instance per process. Any other scenario sounds like a development & debugging nightmare.
> 


January 26, 2005
Indeed.

I, for one, can live with D 1.0 not having dynamic class loading, but that would be a sine qua non for 2.0. Furthermore, I think 1.0 must have a cooperative/unified GC architecture between link units, otherwise, again, we'll just be writing our DLLs in C and all non-compiled in code to an app will be via D extensions (which, while fun to write occasionaly, will get *really* tiresome).

Or maybe I'm missing some deeper truth on the viability of D. If so, can someone please enlighten me so I can stop carping on like a harbinger of Doom.

The Dr .....


"Kris" <Kris_member@pathlink.com> wrote in message news:ct95fa$2tv5$1@digitaldaemon.com...
> The gods be praised! Good example, Benji.
>
> This is why it's been noted that D would not pass muster, in much of
> the
> all-important commercial field.
>
> - Kris
>
>
> In article <ce4gv0dk6v1qgde6o94msecgi4huasdhga@4ax.com>, Benji Smith says...
>>
>>I've been interested to read some of the recent discussion about the D garbage collector, and I'd like to describe my current Java project to give some perspective on what I think is ideal memory management.
>>
>>I'm working on a technical analysis and simulation application for historical stock market data. And despite the fact that I've written about ten thousand lines of code myself, I'm using third-party libraries for many aspects of the project. First of all, I'm using JDBC drivers for MySQL and MS SQL Server. The MySQL drivers consist of 3 different JAR files that I include in my application's classpath. The MS SQL Server drivers are contained in another 3 JAR files. I'm also using the Xerces XML parser from the Apache group (3 more JARs), the JFreeChart graphical charting compontents (5 more JARs), the JUnit testing framework (1 JAR), a GNU commandline parsing library (1 JAR), and a few other miscellaneous libraries. All told, I'm importing functionality from more than fifteen different libraries. And the application is still very small; by the time I've finished developing it, I'll probably be using twice as many libraries.
>>
>>But when I write my code, I can write it as though I'm statically linking with each of those libraries. I don't need to use special export semantics when I need to call code from any of those third party vendors. And with a few of those libraries (the commandline parsing library in particular), I may end up writing my own implementation. When I do, I won't have to change the semantics to reflect the fact that I'm no longer using a compiled library. The rest of my code can be completely agnostic to whether I'm linking with source files, compiled class files, class files bundled into a JAR package, classes generated at runtime through reflection hooks, or classes loaded dynamically using a custom classloader. Since my application needs to support third-party plug-in development (users can load their own classes as custom charting indicators), dynamic runtime loading of classes is essential to my design.
>>
>>Ubiquitous static-linking would not be an option for me with this application.
>>
>>But there's another important issue, too: debugging.
>>
>>Last week I discovered a memory leak somewhere in my application. If I allowed some of the analysis code to run for a few hours--combing through all 18 million data points from the last 25 years of stock market data--the heap would grow from its initial allocation (8 MB) to its maximum allocation (256 MB). Luckily for me, all of those allocations take place within a single virtual machine, which uses a single garbage collector to manage all of the memory from all of the libraries I'm using. That allows me to use a profiling application to monitor the allocations of all the objects in the heap and--much more importantly--to find out which objects are holding references to which other objects. Within moments, I could see that the JDBC allocations were getting cleaned up properly, but one of my custom collection classes was failing-about 2% of the time-to release object references it was no longer using. After an hour or so of tinkering with the profiler, I was able to track down and fix that memory leak. The application now uses a steady 12 MB of heap memory, no matter how long it runs.
>>
>>If I were building the same application with D, there would be fifteen different garbage collectors operating in fifteen different heap-spaces, and the objects allocated within one heap space might be referenced by objects in another heap space, each managed by a different garbage collector. It would be much more difficult to develop a heap profiling tool that could successfully allow a developer to navigate through such a fragmented heap space, particularly if the developer needed to figure out which GC was supposed to collect each out-of-reach object. Tracking down and fixing that memory leak probably would have taken a lot longer than an hour and a half.
>>
>>Consequently, I strongly support the development of a model within D that allows for a single GC instance per process. Any other scenario sounds like a development & debugging nightmare.
>>
>
>



January 26, 2005
"Benji Smith" <dlanguage@xxagg.com> wrote in message news:ce4gv0dk6v1qgde6o94msecgi4huasdhga@4ax.com...
> If I were building the same application with D, there would be fifteen different garbage collectors operating in fifteen different heap-spaces, and the objects allocated within one heap space might be referenced by objects in another heap space, each managed by a different garbage collector. It would be much more difficult to develop a heap profiling tool that could successfully allow a developer to navigate through such a fragmented heap space, particularly if the developer needed to figure out which GC was supposed to collect each out-of-reach object. Tracking down and fixing that memory leak probably would have taken a lot longer than an hour and a half.
>
> Consequently, I strongly support the development of a model within D that allows for a single GC instance per process. Any other scenario sounds like a development & debugging nightmare.

I agree. I'm working on it.


January 26, 2005
Encore!

And lest we forget: D would happily support the all-important unified GC (per process) if the GC were simply moved to a shared-lib. Further, Sean has already invested the majority of effort in carefully extracting the GC from the runtime, such that this is a near reality.

Walter notes that he's had customer-support difficulties in the past over shared-libs, due to the vagaries of Win32. Unfortunately, that negative experience is being reflected directly in the range of valid programming models effectively supported by the D language.

We /really/ need to move forward on this issue. Perhaps we can start (yet again) by asking why Walter feels we're all so much better off with a proliferation of GC instances instead of just one, easily manageable, instance?

- Kris


In article <ct968s$2uv3$1@digitaldaemon.com>, Matthew says...
>
>Indeed.
>
>I, for one, can live with D 1.0 not having dynamic class loading, but that would be a sine qua non for 2.0. Furthermore, I think 1.0 must have a cooperative/unified GC architecture between link units, otherwise, again, we'll just be writing our DLLs in C and all non-compiled in code to an app will be via D extensions (which, while fun to write occasionaly, will get *really* tiresome).
>
>Or maybe I'm missing some deeper truth on the viability of D. If so, can someone please enlighten me so I can stop carping on like a harbinger of Doom.
>
>The Dr .....
>
>
>"Kris" <Kris_member@pathlink.com> wrote in message news:ct95fa$2tv5$1@digitaldaemon.com...
>> The gods be praised! Good example, Benji.
>>
>> This is why it's been noted that D would not pass muster, in much of
>> the
>> all-important commercial field.
>>
>> - Kris
>>
>>
>> In article <ce4gv0dk6v1qgde6o94msecgi4huasdhga@4ax.com>, Benji Smith says...
>>>
>>>I've been interested to read some of the recent discussion about the D garbage collector, and I'd like to describe my current Java project to give some perspective on what I think is ideal memory management.
>>>
>>>I'm working on a technical analysis and simulation application for historical stock market data. And despite the fact that I've written about ten thousand lines of code myself, I'm using third-party libraries for many aspects of the project. First of all, I'm using JDBC drivers for MySQL and MS SQL Server. The MySQL drivers consist of 3 different JAR files that I include in my application's classpath. The MS SQL Server drivers are contained in another 3 JAR files. I'm also using the Xerces XML parser from the Apache group (3 more JARs), the JFreeChart graphical charting compontents (5 more JARs), the JUnit testing framework (1 JAR), a GNU commandline parsing library (1 JAR), and a few other miscellaneous libraries. All told, I'm importing functionality from more than fifteen different libraries. And the application is still very small; by the time I've finished developing it, I'll probably be using twice as many libraries.
>>>
>>>But when I write my code, I can write it as though I'm statically linking with each of those libraries. I don't need to use special export semantics when I need to call code from any of those third party vendors. And with a few of those libraries (the commandline parsing library in particular), I may end up writing my own implementation. When I do, I won't have to change the semantics to reflect the fact that I'm no longer using a compiled library. The rest of my code can be completely agnostic to whether I'm linking with source files, compiled class files, class files bundled into a JAR package, classes generated at runtime through reflection hooks, or classes loaded dynamically using a custom classloader. Since my application needs to support third-party plug-in development (users can load their own classes as custom charting indicators), dynamic runtime loading of classes is essential to my design.
>>>
>>>Ubiquitous static-linking would not be an option for me with this application.
>>>
>>>But there's another important issue, too: debugging.
>>>
>>>Last week I discovered a memory leak somewhere in my application. If I allowed some of the analysis code to run for a few hours--combing through all 18 million data points from the last 25 years of stock market data--the heap would grow from its initial allocation (8 MB) to its maximum allocation (256 MB). Luckily for me, all of those allocations take place within a single virtual machine, which uses a single garbage collector to manage all of the memory from all of the libraries I'm using. That allows me to use a profiling application to monitor the allocations of all the objects in the heap and--much more importantly--to find out which objects are holding references to which other objects. Within moments, I could see that the JDBC allocations were getting cleaned up properly, but one of my custom collection classes was failing-about 2% of the time-to release object references it was no longer using. After an hour or so of tinkering with the profiler, I was able to track down and fix that memory leak. The application now uses a steady 12 MB of heap memory, no matter how long it runs.
>>>
>>>If I were building the same application with D, there would be fifteen different garbage collectors operating in fifteen different heap-spaces, and the objects allocated within one heap space might be referenced by objects in another heap space, each managed by a different garbage collector. It would be much more difficult to develop a heap profiling tool that could successfully allow a developer to navigate through such a fragmented heap space, particularly if the developer needed to figure out which GC was supposed to collect each out-of-reach object. Tracking down and fixing that memory leak probably would have taken a lot longer than an hour and a half.
>>>
>>>Consequently, I strongly support the development of a model within D that allows for a single GC instance per process. Any other scenario sounds like a development & debugging nightmare.
>>>
>>
>>
>
>
>


January 27, 2005
In article <ct9afo$2l8$1@digitaldaemon.com>, Walter says...
>
>
>"Benji Smith" <dlanguage@xxagg.com> wrote in message news:ce4gv0dk6v1qgde6o94msecgi4huasdhga@4ax.com...
>> If I were building the same application with D, there would be fifteen different garbage collectors operating in fifteen different heap-spaces, and the objects allocated within one heap space might be referenced by objects in another heap space, each managed by a different garbage collector. It would be much more difficult to develop a heap profiling tool that could successfully allow a developer to navigate through such a fragmented heap space, particularly if the developer needed to figure out which GC was supposed to collect each out-of-reach object. Tracking down and fixing that memory leak probably would have taken a lot longer than an hour and a half.
>>
>> Consequently, I strongly support the development of a model within D that allows for a single GC instance per process. Any other scenario sounds like a development & debugging nightmare.
>
>I agree. I'm working on it.
>
>

Hallelujah!

I will now shutup about this for a while :-)

(can I get another Hallelujah?)


January 27, 2005
On Thu, 27 Jan 2005 00:16:37 +0000, Kris wrote:

> In article <ct9afo$2l8$1@digitaldaemon.com>, Walter says...

>>I agree. I'm working on it.
>>
>>
> 
> Hallelujah!
> 
> I will now shutup about this for a while :-)
> 
> (can I get another Hallelujah?)

:-D

This is indeed good news!

And Walter, thanks for listening.  Nice to see that you can be so
resilient despite being pounded like a fence post (but it was for a good
cause). :-)

- John R.



January 27, 2005
> And lest we forget: D would happily support the all-important unified GC (per process) if the GC were simply moved to a shared-lib. Further, Sean has already invested the majority of effort in carefully extracting the GC from the runtime, such that this is a near reality.

As I've said before, I want both models supportable, and I don't accept that this is technically infeasible.

Notwithstanding, if both are not, then we must go for a separation between the pure statically linked model and the dynamically linked GC model. If we stay with static linking only, D's a joke, isn't it?

> Walter notes that he's had customer-support difficulties in the past over shared-libs, due to the vagaries of Win32. Unfortunately, that negative experience is being reflected directly in the range of valid programming models effectively supported by the D language.

I agree. And I think this will kill D. As I've whinged and whined on, I can't understand how Walter thinks that D will be viable with the status quo.

Alas, though Walter has huge amounts of valuable experience and insight (more than mine, I would hazard), I think he fails to recognise, or at least act on, two important facts:

    1. he doesn't have *all* experience. None of us do. And, much more importantly, ...
    2. many of us do not have any serious problems doing *very successful* (see below) work in C/C++. If D is not a
quantum leap forward, _without_ new hassles, then why the hell is anyone ever going to use it? Because it's better than
Java?!? Pah!

> We /really/ need to move forward on this issue. Perhaps we can start (yet again) by asking why Walter feels we're all so much better off with a proliferation of GC instances instead of just one, easily manageable, instance?

We're not better off with that. We're nowhere with that! Someone turn out the lights on their way out ...

The Dr .....

(below to be seen): I've worked on several highly commercially important projects over the last several years, most of which have been (primarily) implemented in C++. All the guff that people generally whinge on about as problems in C++ have proved either non-existant, irrelevant, or easily amenable to good practice. Some of these are still in production, 2, 4, 5 years later, and have never had a millisecond of downtime. So why do we need D, if it's going to be hassle-bundled?



>
> In article <ct968s$2uv3$1@digitaldaemon.com>, Matthew says...
>>
>>Indeed.
>>
>>I, for one, can live with D 1.0 not having dynamic class loading, but that would be a sine qua non for 2.0. Furthermore, I think 1.0 must have a cooperative/unified GC architecture between link units, otherwise, again, we'll just be writing our DLLs in C and all non-compiled in code to an app will be via D extensions (which, while fun to write occasionaly, will get *really* tiresome).
>>
>>Or maybe I'm missing some deeper truth on the viability of D. If so, can someone please enlighten me so I can stop carping on like a harbinger of Doom.
>>
>>The Dr .....
>>
>>
>>"Kris" <Kris_member@pathlink.com> wrote in message news:ct95fa$2tv5$1@digitaldaemon.com...
>>> The gods be praised! Good example, Benji.
>>>
>>> This is why it's been noted that D would not pass muster, in much of
>>> the
>>> all-important commercial field.
>>>
>>> - Kris
>>>
>>>
>>> In article <ce4gv0dk6v1qgde6o94msecgi4huasdhga@4ax.com>, Benji Smith says...
>>>>
>>>>I've been interested to read some of the recent discussion about the D garbage collector, and I'd like to describe my current Java project to give some perspective on what I think is ideal memory management.
>>>>
>>>>I'm working on a technical analysis and simulation application for historical stock market data. And despite the fact that I've written about ten thousand lines of code myself, I'm using third-party libraries for many aspects of the project. First of all, I'm using JDBC drivers for MySQL and MS SQL Server. The MySQL drivers consist of 3 different JAR files that I include in my application's classpath. The MS SQL Server drivers are contained in another 3 JAR files. I'm also using the Xerces XML parser from the Apache group (3 more JARs), the JFreeChart graphical charting compontents (5 more JARs), the JUnit testing framework (1 JAR), a GNU commandline parsing library (1 JAR), and a few other miscellaneous libraries. All told, I'm importing functionality from more than fifteen different libraries. And the application is still very small; by the time I've finished developing it, I'll probably be using twice as many libraries.
>>>>
>>>>But when I write my code, I can write it as though I'm statically linking with each of those libraries. I don't need to use special export semantics when I need to call code from any of those third party vendors. And with a few of those libraries (the commandline parsing library in particular), I may end up writing my own implementation. When I do, I won't have to change the semantics to reflect the fact that I'm no longer using a compiled library. The rest of my code can be completely agnostic to whether I'm linking with source files, compiled class files, class files bundled into a JAR package, classes generated at runtime through reflection hooks, or classes loaded dynamically using a custom classloader. Since my application needs to support third-party plug-in development (users can load their own classes as custom charting indicators), dynamic runtime loading of classes is essential to my design.
>>>>
>>>>Ubiquitous static-linking would not be an option for me with this application.
>>>>
>>>>But there's another important issue, too: debugging.
>>>>
>>>>Last week I discovered a memory leak somewhere in my application. If I allowed some of the analysis code to run for a few hours--combing through all 18 million data points from the last 25 years of stock market data--the heap would grow from its initial allocation (8 MB) to its maximum allocation (256 MB). Luckily for me, all of those allocations take place within a single virtual machine, which uses a single garbage collector to manage all of the memory from all of the libraries I'm using. That allows me to use a profiling application to monitor the allocations of all the objects in the heap and--much more importantly--to find out which objects are holding references to which other objects. Within moments, I could see that the JDBC allocations were getting cleaned up properly, but one of my custom collection classes was failing-about 2% of the time-to release object references it was no longer using. After an hour or so of tinkering with the profiler, I was able to track down and fix that memory leak. The application now uses a steady 12 MB of heap memory, no matter how long it runs.
>>>>
>>>>If I were building the same application with D, there would be fifteen different garbage collectors operating in fifteen different heap-spaces, and the objects allocated within one heap space might be referenced by objects in another heap space, each managed by a different garbage collector. It would be much more difficult to develop a heap profiling tool that could successfully allow a developer to navigate through such a fragmented heap space, particularly if the developer needed to figure out which GC was supposed to collect each out-of-reach object. Tracking down and fixing that memory leak probably would have taken a lot longer than an hour and a half.
>>>>
>>>>Consequently, I strongly support the development of a model within D that allows for a single GC instance per process. Any other scenario sounds like a development & debugging nightmare.
>>>>
>>>
>>>
>>
>>
>>
>
> 


January 27, 2005
"Matthew" <admin.hat@stlsoft.dot.org> wrote in message news:ct9gl9$9k4$1@digitaldaemon.com...
>    2. many of us do not have any serious problems doing *very successful* (see below) work in C/C++. If D is not a
> quantum leap forward, _without_ new hassles, then why the hell is anyone ever going to use it? Because it's better
> than Java?!? Pah!

btw, Java-phobic hyperbole aside, I should point out that until D can handle scenarios such as outlined in Benji's excellent post, D isn't even fit to kiss the bloated arse of Java. And that's a sad position to be in, to be sure ...



« First   ‹ Prev
1 2 3