On 10/9/2014 7:15 AM, Dicebot wrote:
> That can possibly be done though it will take some efforts to formalize issues
> from the casual chat rants. More important issue is - what will happen next? I
> am pretty sure many of those easy wins are not easy at all in a sense that
> breaking changes are needed.

That's why I'd like to know what Don has in mind.

On 09/10/14 18:41, Andrei Alexandrescu wrote:

> With structured outputs there are a lot more issues to address: one can
> think of a JSONObject as an output range with put() but that's only
> moving the real issues around. How would the JSONObject allocate memory
> internally, give it out to its own users, and dispose of it timely, all
> in good safety?

The XML DOM module in Tango uses, if I recall correctly, a free list internally for the nodes. It will reuse the nodes, if you want to keep some information you need to copy it yourself.

-- 
/Jacob Carlborg

On Thursday, 9 October 2014 at 17:29:01 UTC, Dicebot wrote:
> On Thursday, 9 October 2014 at 16:22:52 UTC, Andrei Alexandrescu wrote:
>>>> To clarify: calling GC.free does remove the root, correct?
>>>
>>> Not before it creates one. When I mean "avoid creating new GC roots" I
>>> mean "no GC activity at all other than extending existing chunks"
>>
>> That's interesting. So GC.malloc followed by GC.free does actually affect things negatively?
>
> Yes and quite notably so as GC.malloc can potentially trigger collection. With concurrent GC collection is not a disaster but it still affects the latency and should be avoided.

Is it just the potentially triggered collection, or is the actual allocation+deallocation too expensive?

Because the effects of the former can of course be reduced greatly by tweaking the GC to not collect every time the heap needs to grow, at the cost of slightly more memory consumption.

If it's the latter, that would indicate that maybe a different allocator with less overhead needs to be used.

>
>> Also let's note that extending existing chunks may result in new allocations.
>
> Yes. But as those chunks never get free'd it comes to O(1) allocation count over process lifetime with most allocations happening during program startup / warmup.

Hmm... but shouldn't this just as well apply to the temporary allocations? After some warming up phase, the available space on the heap should be large enough that all further temporary allocations can be satisfied without growing the heap.

On Friday, 10 October 2014 at 08:45:38 UTC, Marc Schütz wrote:
> On Thursday, 9 October 2014 at 17:29:01 UTC, Dicebot wrote:
>> On Thursday, 9 October 2014 at 16:22:52 UTC, Andrei Alexandrescu wrote:

I think the worst of D is summarized quite well by the following:

http://forum.dlang.org/post/m15i9c$51b$1@digitalmars.com

http://forum.dlang.org/post/54374DE0.6040405@digitalmars.com

And that is: the focus is no longer to do the right things about D, but to make D do the right thing for some shadowy customers that don't even care enough to come here and grumble.

Let's put D2 in maintenance mode and take on D3 with this nice motto, taken from here:

http://forum.dlang.org/post/3af85684-7728-4165-acf9-520a240f65e0@me.com

"why didn't work like that from the beginning?"

Tofu Ninja:

> What do you think are the worst parts of D?

There are several problems in D/Phobos, but I think the biggest one is the development process, that is currently toxic:
http://forum.dlang.org/thread/54374DE0.6040405@digitalmars.com

In my opinion an Open Source language with such development problem goes nowhere, so I think this needs to be improved.

There are several possible ways to improve this situation, but perhaps the may one that can work is: Walter has to delegate another slice of its authority to the community of core dmd developers. This will cause some problems, perhaps an increase in confusion and pull reversions, Walter may lose a bit of his grasp of the dmd codebase (but this can be avoided if he reads the pull requests code), but I think overall the advantages in the long term are going to be bigger than the disadvantages.

Regarding the product (and not the process), the design of a principled, sound, and flexible memory ownership system is an important part of D design to work on in the following months. It's going to add some complexity to D, but this is not the kind of complexity that kills a language, because if it's well designed it's going to give easy to read compile-time errors, it's not going to limit too much the variety of things that are currently allowed in D, and it's going to statically remove a significant amount of run-time bugs that are currently easy to do in D. The complexity that causes problems is not the one that requires you to understand a part of a well designed type system: it's the pile of arbitrary special cases and special cases of special cases you see in C++. D considers safety and correctness very important points of its design, and I agree with the importance of such points.

Bye,
bearophile

On Friday, 10 October 2014 at 08:45:38 UTC, Marc Schütz wrote:
>> Yes and quite notably so as GC.malloc can potentially trigger collection. With concurrent GC collection is not a disaster but it still affects the latency and should be avoided.
>
> Is it just the potentially triggered collection, or is the actual allocation+deallocation too expensive?

collection - for sure.
allocation+deallocation - maybe, I have never measured it. It is surely slower than not allocating at all though.

> Because the effects of the former can of course be reduced greatly by tweaking the GC to not collect every time the heap needs to grow, at the cost of slightly more memory consumption.

This is likely to work better but still will be slower than our current approach because of tracking many small objects. Though of course it is just speculation until RC stuff is implemented for experiments.

>>> Also let's note that extending existing chunks may result in new allocations.
>>
>> Yes. But as those chunks never get free'd it comes to O(1) allocation count over process lifetime with most allocations happening during program startup / warmup.
>
> Hmm... but shouldn't this just as well apply to the temporary allocations? After some warming up phase, the available space on the heap should be large enough that all further temporary allocations can be satisfied without growing the heap.

I am not speaking about O(1) internal heap increases but O(1) GC.malloc calls
Typical pattern is to encapsulate "temporary" buffer with the algorithm in a single class object and never release it, reusing with new incoming requests (wiping the buffer data each time). Such buffer quickly gets to the point where it is large enough to contain all algorithm temporaries for a single request and never touches GC from there.

In a well-written program which follows such pattern there are close to zero temporaries and GC only manages more persistent entities like cache elements.

On Wednesday, 8 October 2014 at 19:00:44 UTC, Jonathan wrote:
> 3) Taking a hint from the early success of Flash, add Derelict3 (or some basic OpenGL library) directly into Phobos. Despite some of the negatives (slower update cycle versus external lib), it would greatly add to D's attractiveness for new developers. I nearly left D after having a host issues putting Derelict3 into my project. If I had this issue, we may be missing out from attracting newbies looking to do gfx related work.


This reminds of an idea I've been pondering over for a while now.
What if there was a language, that came with a standard toolkit
for the more fun stuff such as OpenGL (There could be one already
and I just don't know of it). But if we take that idea and try to
apply it to D, we sortof get Deimos. Problem is Deimos, is pretty
disjointed and is only updated every now and then, so then as an
alternative I suppose there is Derelict. However, Derelict is
maintained primarily by one person it seems (he does a great job
though!), but Derelict isn't a standard feature (I know Deimos
isn't either) and I *personally* don't care much for it's heavy
leaning on dub. +Derelict isn't always a walk in the park to get
running

The alternative I'm suggesting, not by any means a top priority,
is give Deimos a makeover (Derelict could possibly be a big part
of this) and turn it into a semi-standard feature. So you can
import phobos modules to do the stuff phobos normally does, but
if you feel like making a quick tool or two, you can import
deimos to get Tcl/tk like you would in python, or call OpenGl, or
whatever other tool you need (doesn't have to be a graphics
thing). Then at compile time the compiler could just copy or
build the required dll's\so's & object files into the specified
directory, or whatever works best.

On Wednesday, 8 October 2014 at 19:47:05 UTC, Jeremy Powers via
Digitalmars-d wrote:
>
> Personally I take the opposite view - I'd much prefer a strong and easily
> consumed third-party library ecosystem than to shove everything into
> Phobos.  Dub is a wonderful thing for D, and needs to be so good that
> people use it by default.

Not to initiate my biweekly "not a fan of dub" conversation, but
just wanna say real quick: Not everyone really likes to use dub.
The only thing I like about it, is using it as the build script
for a library to get the .lib files and whatnot. Though I don't
feel like it really adds a drastic improvement over a .d build
script. However, I don't work on any open source libraries, but
maybe if I did, I'd like it better then..? Not something I would
have an answer to right now, soo... yea:P


----

Aside from what I mentioned above, I'm not sure where I'd like D
to be at next to be perfectly honest. Stuff like no GC or C++
integration sound cool, but for me personally they just seem like
'neat' feature, not something that I feel like would change my
current workflow for better or worse.
Refinement of what's already in place sounds the best if anything.


So those are just some passing thoughts of a stranger.. carry on,
this thread has been very interesting so far ;P

On Saturday, 11 October 2014 at 03:39:10 UTC, Dicebot wrote:
> I am not speaking about O(1) internal heap increases but O(1) GC.malloc calls
> Typical pattern is to encapsulate "temporary" buffer with the algorithm in a single class object and never release it, reusing with new incoming requests (wiping the buffer data each time). Such buffer quickly gets to the point where it is large enough to contain all algorithm temporaries for a single request and never touches GC from there.
>
> In a well-written program which follows such pattern there are close to zero temporaries and GC only manages more persistent entities like cache elements.

I understand that. My argument is that the same should apply to the entire heap: After you've allocated and released a certain amount of objects via GC.malloc() and GC.free(), the heap will have grown to a size large enough that any subsequent allocations of temporary objects can be satisfied from the existing heap without triggering a collection, so that only the overhead of actual allocation and freeing should be relevant.

> Exactly. C++ support is of no interest at all, and GC is something we contribute to, rather than something we expect from the community.
> Interestingly we don't even care much about libraries, we've done everything ourselves.
>
> So what do we care about? Mainly, we care about improving the core product.
>
> In general I think that in D we have always suffered from spreading ourselves too thin. We've always had a bunch of cool new features that don't actually work properly. Always, the focus shifts to something else, before the previous feature was finished.
>
> At Sociomantic, we've been successful in our industry using only the features of D1. We're restricted to using D's features from 2007!! Feature-wise, practically nothing from the last seven years has helped us!
>
> With something like C++ support, it's only going to win companies over when it is essentially complete. That means that working on it is a huge investment that doesn't start to pay for itself for a very long time. So although it's a great goal, with a huge potential payoff, I don't think that it should be consuming a whole lot of energy right now.
>
> And personally, I doubt that many companies would use D, even if with perfect C++ interop, if the toolchain stayed at the current level.
>
> As I said in my Dconf 2013 talk -- I advocate a focus on Return On Investment.
> I'd love to see us chasing the easy wins.

disclaimer: i am rather new to D and thus have a bit of a distant view.

i think the above touches an important point. One thing GO does right is that they focused on feature rich stdlib/library ecosystem even though the language was very young. i'm coming from Ruby/Python and the reason i use those languages is that they have two things:
a) they are fun to use (as andrei said in the floss interview: the creators had "taste").
b) huge set of libs that help me to get stuff done.

now i think a) is fine, but with b) i am not sure if the strategy to get full C/C++ interop will not take too long and scare those people off that are not coming from C/C++.

i think D is a fantastic tool to write expressive, fast and readable code. I don't need much more language features (again, look at GO...) but a solid foundation of libs to gain "competitive advantage" in my daily work.

On Saturday, 11 October 2014 at 09:26:28 UTC, Marc Schütz wrote:
> I understand that. My argument is that the same should apply to the entire heap: After you've allocated and released a certain amount of objects via GC.malloc() and GC.free(), the heap will have grown to a size large enough that any subsequent allocations of temporary objects can be satisfied from the existing heap without triggering a collection, so that only the overhead of actual allocation and freeing should be relevant.

But it still requires GC to check its pool state upon each request and make relevant adjustments for malloc/free combo. For something like a hundred of temporary allocations per request it accumulates into notable time (and milliseconds matter). In absence of collection it is cheap enough to not care about single malloc call on its own but still not cheap enough to ignore costs of many calls.

You have interested me in doing some related benchmarks though.