On Saturday, 23 August 2014 at 04:36:34 UTC, Walter Bright wrote:
> On 8/22/2014 9:01 PM, Ola Fosheim Gr wrote:
>> Does this mean that D is getting resizable stack allocations in lower stack
>> frames? That has a lot of implications for code gen.
>
> scopebuffer does not require resizeable stack allocations.

So you cannot use the stack for resizable allocations.

That would however be a nice optimization. Iff an algorithm only have one alloca, can be inlined in a way which does not extend the stack and use a resizable buffer that grows downwards in memory then you can have a resizable buffer on the stack:

HIMEM
...
Algorihm stack frame vars
Inlined vars
Buffer head/book keeping vars
Buffer end
Buffer front
...add to front here...
End of stack
LOMEM

On 8/22/2014 9:48 PM, Ola Fosheim Gr wrote:
> On Saturday, 23 August 2014 at 04:36:34 UTC, Walter Bright wrote:
>> On 8/22/2014 9:01 PM, Ola Fosheim Gr wrote:
>>> Does this mean that D is getting resizable stack allocations in lower stack
>>> frames? That has a lot of implications for code gen.
>>
>> scopebuffer does not require resizeable stack allocations.
>
> So you cannot use the stack for resizable allocations.

Please, take a look at how scopebuffer works.

On Saturday, 23 August 2014 at 05:28:55 UTC, Walter Bright wrote:
> On 8/22/2014 9:48 PM, Ola Fosheim Gr wrote:
>> On Saturday, 23 August 2014 at 04:36:34 UTC, Walter Bright wrote:
>>> On 8/22/2014 9:01 PM, Ola Fosheim Gr wrote:
>>>> Does this mean that D is getting resizable stack allocations in lower stack
>>>> frames? That has a lot of implications for code gen.
>>>
>>> scopebuffer does not require resizeable stack allocations.
>>
>> So you cannot use the stack for resizable allocations.
>
> Please, take a look at how scopebuffer works.

I have? It requires an upperbound to stay on the stack, that creates a big hole in the stack. I don't think wasting the stack or moving to the heap is a nice predictable solution. It would be better to just have a couple of regions that do "reverse" stack allocations, but the most efficient solution is the one I outlined.

With json you might be able to create an upperbound of say 4-8 times the size of the source iff you know the file size. You don't if you are streaming.

(scopebuffer is too unpredictable for real time, a pure stack solution is predictable)

On 8/22/2014 11:25 PM, Ola Fosheim Gr wrote:
> On Saturday, 23 August 2014 at 05:28:55 UTC, Walter Bright wrote:
>> On 8/22/2014 9:48 PM, Ola Fosheim Gr wrote:
>>> On Saturday, 23 August 2014 at 04:36:34 UTC, Walter Bright wrote:
>>>> On 8/22/2014 9:01 PM, Ola Fosheim Gr wrote:
>>>>> Does this mean that D is getting resizable stack allocations in lower stack
>>>>> frames? That has a lot of implications for code gen.
>>>>
>>>> scopebuffer does not require resizeable stack allocations.
>>>
>>> So you cannot use the stack for resizable allocations.
>>
>> Please, take a look at how scopebuffer works.
>
> I have? It requires an upperbound to stay on the stack, that creates a big hole
> in the stack. I don't think wasting the stack or moving to the heap is a nice
> predictable solution. It would be better to just have a couple of regions that
> do "reverse" stack allocations, but the most efficient solution is the one I
> outlined.

Scopebuffer is extensively used in Warp, and works very well. The "hole" in the stack is not a significant problem.


> With json you might be able to create an upperbound of say 4-8 times the size of
> the source iff you know the file size. You don't if you are streaming.
>
> (scopebuffer is too unpredictable for real time, a pure stack solution is
> predictable)

You can always implement your own buffering system and pass it in - that's the point, it's under user control.

On Saturday, 23 August 2014 at 06:41:11 UTC, Walter Bright wrote:
> Scopebuffer is extensively used in Warp, and works very well. The "hole" in the stack is not a significant problem.

Well, on a webserver you don't want to push out the caches for no good reason.

> You can always implement your own buffering system and pass it in - that's the point, it's under user control.

My point is that you need compiler support to get good buffering options on the stack. Something like an @alloca_inline:

auto buffer = @alloca_inline getstuff();
process(buffer);

I think all memory allocation should be under compiler control, the library solutions are bound to be suboptimal, i.e. slower.

On 8/22/14, Sönke Ludwig <digitalmars-d@puremagic.com> wrote:
> Hmmm, but it *is* a string. Isn't the problem more the use of with in this case?

Yeah, maybe so. I thought for a second it was a tuple, but then I saw the square brackets and was left scratching my head. :)

Am 23.08.2014 03:05, schrieb Walter Bright:
> On 8/22/2014 2:27 PM, Sönke Ludwig wrote:
>> Am 22.08.2014 20:08, schrieb Walter Bright:
>>> 1. There's no mention of what will happen if it is passed malformed JSON
>>> strings. I presume an exception is thrown. Exceptions are both slow and
>>> consume GC memory. I suggest an alternative would be to emit an "Error"
>>> token instead; this would be much like how the UTF decoding algorithms
>>> emit a "replacement char" for invalid UTF sequences.
>> The latest version now features a LexOptions.noThrow option which
>> causes an
>> error token to be emitted instead. After popping the error token, the
>> range is
>> always empty.
>
> Having a nothrow option may prevent the functions from being attributed
> as "nothrow".

It's a compile time option, so that shouldn't be an issue. There is also just a single "throw" statement in the source, so it's easy to isolate.

Am 23.08.2014 04:23, schrieb deadalnix:
> First thank you for your work. std.json is horrible to use right now, so
> a replacement is more than welcome.
>
> I haven't played with your code yet, so I may be asking for somethign
> that already exists, but did you had a look to jsvar by Adam ?
>
> You can find it here:
> https://github.com/adamdruppe/arsd/blob/master/jsvar.d
>
> One of the big pain when one work with format like JSON is that you go
> from the untyped world to the typed world (the same problem occurs with
> XML and various config format as well).
>
> I think Adam got the right balance in jsvar. It behave closely enough to
> javascript so it is convenient to manipulate, while removing the most
> dangerous behavior (concatenation is still done using ~and not + as in JS).
>
> If that is not already the case, I'd love that the element I get out of
> my JSON behave that way. If you can do that, you have a user.

Setting the issue of opDispatch aside, one of the goals was to use Algebraic to store values. It is probably not completely as flexible as jsvar, but still transparently enables a lot of operations (with those pull requests merged at least). But it has another big advantage, which is that we can later define other types based on Algebraic, such as BSONValue, and those can be transparently runtime converted between each other in a generic way. A special case type on the other hand produces nasty dependencies between the formats.

Main issues of using opDispatch:

 - Prone to bugs where a normal field/method of the JSONValue struct is accessed instead of a JSON field
 - On top of that the var.field syntax gives the wrong impression that you are working with static typing, while var["field"] makes it clear that runtime indexing is going on
 - Every interface change of JSONValue would be a silent breaking change, because the whole string domain is used up for opDispatch

On Saturday, 23 August 2014 at 09:22:01 UTC, Sönke Ludwig wrote:
> Main issues of using opDispatch:
>
>  - Prone to bugs where a normal field/method of the JSONValue struct is accessed instead of a JSON field
>  - On top of that the var.field syntax gives the wrong impression that you are working with static typing, while var["field"] makes it clear that runtime indexing is going on
>  - Every interface change of JSONValue would be a silent breaking change, because the whole string domain is used up for opDispatch

I have seen similar issues to these with simplexml in PHP. Using opDispatch to match all possible names except a few doesn't work so well.

I'm not sure if you've changed it already, but I agree with the earlier comment about changing the flag for pretty printing from a boolean to an enum value. Booleans in interfaces is one of my pet peeves.

Am 23.08.2014 14:19, schrieb w0rp:
> I'm not sure if you've changed it already, but I agree with the earlier
> comment about changing the flag for pretty printing from a boolean to an
> enum value. Booleans in interfaces is one of my pet peeves.

It's split into two separate functions now. Having to type out a full enum value I guess would be too distracting in this case, since they will be pretty frequently used.