January 09, 2013
On Wednesday, 9 January 2013 at 03:05:34 UTC, H. S. Teoh wrote:
> I haven't been screaming yet because (so far) I haven't gotten to
> writing applications that need dynamic loading in D. But I did tell
> myself that I will be screaming when I do get to it, because it's a pain
> to have to recompile the entire application just to add a single addon.

Unfortunately for me, I'm getting to that point.

>
>> If there's another better way, I'd sure like to know about it!
> [...]
>
> Another way, yes. Better, I don't know. You *could* load plugins as
> separate processes and communicate via some kind of IPC mechanism, like
> Unix pipes. But that's a royal pain (requires serialization /
> deserialization, with the associated overhead, and a network stack or
> equivalent, just to interface with each other).
>
>
> T

The messaging concept does have some advantages. For example, if your external "plugin" fails for any reason (for example due to a segfault), the rest of your main application can continue operating just fine. There are other advantages, such as distributed processing, but if you can benefit from those advantages depends entirely on what you are attempting to achieve. The simplest use case that gives you a good subset of the main advantages, is through runtime loaded plugins.

In my case, I will making use of both methods, but I really prefer having the plug-in capability to start with.

--rt
January 09, 2013
On Wednesday, 9 January 2013 at 19:34:19 UTC, Andrei Alexandrescu wrote:
> At the end of the day if references are part of the language and programs can build arbitrary reference topologies, safety entails GC.


It looks like a non sequitur to me... wouldn't this work?

A language X has a built-in data type called Reference, and no classes.

The only thing you can do with it are using these functions:

Reference CreateObject(Reference typename);
Reference DeleteValue(Reference object, Reference field);
Reference GetValue(Reference object, Reference field);
Reference SetValue(Reference object, Reference field, Reference value);

Given _just_ these functions you can build _any_ arbitrary reference topology whatsoever. There's no need for a GC to be running, and it's completely manual memory management.


It's memory-safe too. What am I missing here?
January 09, 2013
On 1/9/2013 11:40 AM, Rob T wrote:
> Yes you are correct. I was thinking of nullable references when I made that
> comment. When you dereference on a nulled reference, I suppose that's not really
> referencing deallocated memory. I'm not sure what exactly happens behind the
> scenes when you dereference a null pointer, but obviously bad things happen and
> it's not safe, also it is a memory related problem.

Null pointer faults are not a memory safety issue. Memory safety is not defined as "no bugs", it is defined as "no bugs that corrupt memory".

> I did qualify what I said by mentioning that it depends on the definition of
> memory safety. According to my definition of memory safety, a memory leak is
> still a memory leak no matter how it happens.

If we each define our own meanings for words, we cannot understand each other. Memory safety is a standard piece of jargon, with a standard definition.

January 09, 2013
On Wednesday, 9 January 2013 at 09:35:28 UTC, deadalnix wrote:
> On Wednesday, 9 January 2013 at 08:46:20 UTC, Paulo Pinto wrote:
>> You can write plugins without dynamic loading, they just are a bit more cumbersome to write.
>>
>> Like I used to do back in the late 90's, by making use of UNIX's IPC.
>>
>> The IPC to use (shared memory, pipes, mailbox, sockets) depends on what is required from the plugin. With shared memory being the closest to what dynamic loading achieves.
>>
>> Of course this raises another set of issues like:
>>
>> - Take care what happens when a plugin dies
>> - Too many loaded plugins can stress the scheduler
>> - You might have synchronization issues when using shared memory
>> - It is a bit more painful to code for
>>
>> This is the school of thought of the Plan9/Go guys and most operating system micro-kernel architectures.
>>
>
> Such a solution cause the same kind of issue for the runtime. For instance, how to handle the garbage collection of the shared memory ? What do happen if I get the typeid of an object of a type that only exists in the plugin in the core app ?
>
> It quite don't solve problems we have.

Usually you serialize the data into the shared memory when using stronger type languages and only use public types defined in the common interface between the main application and plugins.

--
Paulo
January 09, 2013
On 1/9/13 12:09 PM, Mehrdad wrote:
> On Wednesday, 9 January 2013 at 19:34:19 UTC, Andrei Alexandrescu wrote:
>> At the end of the day if references are part of the language and
>> programs can build arbitrary reference topologies, safety entails GC.
>
>
> It looks like a non sequitur to me... wouldn't this work?
>
> A language X has a built-in data type called Reference, and no classes.
>
> The only thing you can do with it are using these functions:
>
> Reference CreateObject(Reference typename);
> Reference DeleteValue(Reference object, Reference field);
> Reference GetValue(Reference object, Reference field);
> Reference SetValue(Reference object, Reference field, Reference value);
>
> Given _just_ these functions you can build _any_ arbitrary reference
> topology whatsoever. There's no need for a GC to be running, and it's
> completely manual memory management.
>
>
> It's memory-safe too. What am I missing here?

What you're missing is that you define a store that doesn't model object references with object addresses. That's what I meant by "references are part of the language". If store is modeled by actual memory (i.e. accessing an object handle takes you to the object), you must have GC for the language to be safe. If store is actually indirected and gives up on the notion of address, then sure you can implement safety checks. The thing is everybody wants for references to model actual object addresses; indirect handles as the core abstraction are uninteresting.


Andrei
January 09, 2013
On Monday, 7 January 2013 at 15:01:27 UTC, Gor Gyolchanyan wrote:
> Hello, folks!
>
> I'm on to a project, which requires manual memory management using custom allocators, but I can't seem to get dynamic arrays and associative arrays to work.
>
> The std.conv.emplace only allocates the pointer and the size of the dynamic array and pointer to the associative array, which is half the issue.
>
> I can work around the dynamic array by manually allocating the elements of the array and returning a slice to the result (although I'd be really glad if I could directly use arrays with my custom allocators).


 I got to Page 3 but I really wanted to comment on some of this; perhaps a step in the non-GC portion.

 A thought coming to mind is to modify the existing D language to include custom allocator/deallocator. The idea of sorta borrowing from java will do the job. So 'new' can be a function which's only purpose is to allocate memory, renew can be resizing (appending?), and release is to handle deallocation.

  struct S {
    //similar to java's 'new' (C++ as well?) function.
    //'this' referencing the struct's current storage/pointer location.
    new(this, int size = S.sizeof) {
      assert(size >= S.sizeof);
      this = cast(S) GC.malloc(size);
      //ctor's called after new ends
    }

    //array (multidimentional?) allocation handling.
    //new[] or newArray?. Probably no ctor afterwards
    new(this, int size, int[] multiDimentional ...);

    //handles deallocation (if applicable), with GC it's probably empty
    //coinsides with already used releasing of memory.
    void release();

    //handles resizing (in place?), probably arrays,
    //could be realloc() and referring to the current object already...?
    //void realloc(int newSize);
    renew(this, int newSize) {
      S old = this.ptr;
      this.ptr = GC.realloc(this.ptr, newSize);
      if (old !is this.ptr)
        {} //reallocated not appended
    }

    //potentially above new could be used for reallocation as well... So
    //rewritten instead as:
    new(this, int size = S.sizeof) {
      assert(size >= S.sizeof);
      this = cast(S) GC.realloc(this, size);
      //ctor's called after new ends
    }
  }

 Just a thrown together gist. It's kinda how I remember with zlib's setup where you could specify the allocator/deallocator, and if you didn't it used the default. I know it's got problems as it is, but that doesn't mean it can't be used for brainstorming to handle alternate memory management.
January 09, 2013
On Wednesday, 9 January 2013 at 20:00:28 UTC, Dan wrote:
> On Tuesday, 8 January 2013 at 23:04:48 UTC, Paulo Pinto wrote:
>>
>> Besides Web applications, I also took part in projects that ported high
>> performance C++ daemons to Java.
>>
> Curious as to why? What was to be gained/overcome?

The architects decided it was cool to replace server code done in
a mixture of C++/CORBA/SNMP/Perl by simple Java based servers.

>
>> These were servers doing millions of data processing manipulations per
>> second of telecommunication data used in mobile networks.
>>
>
> Did it prove a worthwhile move?

It was a lengthy process with quite some victims along the way,
but it did improve the overall quality in the end.

- Way shorter build times;
- Better tooling infrastructure;
- Thanks to the JVM, the application monitoring improved
- Unit tests, code coverage and static analysis tools are much
better
- Better support of the operating systems in use without #ifdefs everywhere

To be honest, it would have been better to refactor the C++ code to improve the overall quality, but there was another issue related with the language change, as you can read in the next answer.

> Did the move relieve any issues with C++?

Yes, because in 300+ team projects not everyone is a C++ guru and you end up chasing language issues and pointer problems all the time. Not fun when you get a mobile operator complaining that their customers cannot do proper calls.

So moving to Java allowed to improve overall code quality at the time, although I must say it also allowed them to easily outsource parts of the applications to replaceable developers a few years later.

> Was GC an issue in the end?

In the beginning, yes. Most developers did not understand that in GC aware languages you need to change the way you code. So that was a learning process for many of them.

But the JVM has lots of nice monitoring tools to help you track down the issues, which allowed us to refactor the places that were giving problems.

Just some small parts were later also written in C, for deployment into network elements that were not capable to run a JVM with the desired hardware requirements.

So depending on where in the globe you're located, your phone calls data might be processed by those applications.

>
> Thanks,
> Dan


January 09, 2013
On Tuesday, 8 January 2013 at 22:19:56 UTC, Walter Bright wrote:
> Interestingly, carefully written code using a GC can be *faster* than manual memory management, for a number of rather subtle reasons.

 One being calling the OS to allocate memory is an expensive operation (freeing as well?). I would think a smart GC once it identifies a free memory block may not free it to the OS but hold onto it, then give it to another process when asked for memory, thereby skipping the OS step.
January 09, 2013
On Wed, Jan 09, 2013 at 09:03:17PM +0100, Rob T wrote:
> On Wednesday, 9 January 2013 at 03:05:34 UTC, H. S. Teoh wrote:
[...]
> >You *could* load plugins as separate processes and communicate via some kind of IPC mechanism, like Unix pipes. But that's a royal pain (requires serialization / deserialization, with the associated overhead, and a network stack or equivalent, just to interface with each other).
[...]

Another disadvantage is the need for every plugin to have a runtime,
since they'll need to start on their own (as a process, that is, even if
it's the parent app that actually exec()s the process).


> The messaging concept does have some advantages. For example, if your external "plugin" fails for any reason (for example due to a segfault), the rest of your main application can continue operating just fine.

Hmm. I was leaning against using separate processes, but this may just convince me to favor it. I have been the victim of way too many browser plugin malfunctions, which inevitably brings down the entire browser. Recent browsers started using plugin wrappers that can handle plugin crashes, etc., which is an indication of the need of this kind of insulation. The fact that plugin wrappers don't always work very well is another factor that's pushing me in favor of using completely separate processes.


> There are other advantages, such as distributed processing, but if you can benefit from those advantages depends entirely on what you are attempting to achieve. The simplest use case that gives you a good subset of the main advantages, is through runtime loaded plugins.
[...]

True. Using separate processes also incurs performance overhead -- for example, I have a C++ program that dynamically loads plugins that compute the value of some opaque mathematical function, from which a rather large sample set is repeatedly taken.  The reason I elected to use plugins is because the precompiled computation code runs faster -- every sample incurs just the cost of a single function call. If I were forced to make these plugins external processes, the performance hit would be so bad that I might as well just resort to using a boring old expression interpreter instead.


T

-- 
MACINTOSH: Most Applications Crash, If Not, The Operating System Hangs
January 09, 2013
On Wednesday, 9 January 2013 at 20:16:04 UTC, Andrei Alexandrescu wrote:
> What you're missing is that you define a store that doesn't model object references with object addresses. That's what I meant by "references are part of the language". If store is modeled by actual memory (i.e. accessing an object handle takes you to the object), you must have GC for the language to be safe. If store is actually indirected and gives up on the notion of address, then sure you can implement safety checks. The thing is everybody wants for references to model actual object addresses; indirect handles as the core abstraction are uninteresting.
>
> Andrei



But why can't Reference hold the actual address too?
Seems like a perfectly reasonable implementation to me, and there's no extra indirection involved that way, right?