Hi.

What's the best way to pass the contents of a file to the stream parser without reading the whole thing into memory first?  I get an error if using byLine because the kind of range this function returns is not what the stream parser is expecting.

There is an optional filename argument to parseJSONStream, but I am not sure what this is for, since it still requires the first argument for the input if the filename is passed.


Thanks.


Laeeth.

On Wednesday, 29 April 2015 at 18:48:22 UTC, Laeeth Isharc wrote:
> Hi.
>
> What's the best way to pass the contents of a file to the stream parser without reading the whole thing into memory first?  I get an error if using byLine because the kind of range this function returns is not what the stream parser is expecting.
>
> There is an optional filename argument to parseJSONStream, but I am not sure what this is for, since it still requires the first argument for the input if the filename is passed.
>
>
> Thanks.
>
>
> Laeeth.

some more suggestions for enhancements here:
https://github.com/s-ludwig/std_data_json/issues/12

a big advantage of languages like python is that there are end-to-end examples of using the language+libraries to accomplish real work.  (they don't need to be more than a screenful of code, but for the newcomer having it all set out makes a big difference).  since I believe a significant reason for people to switch to D will be a realisation that actually CPU time isn't free, it makes sense to be accommodating to people who previously used scripting types of languages for these jobs.



Laeeth.

Current std.stdio is deprecated. This ad-hoc should works.

	auto json = File("fileName")
		.byChunk(1024 * 1024) //1 MB. Data cluster equals 1024 * 4
		.map!(ch => ch.idup)
		.joiner
		.map!(b => cast(char)b)
		.parseJSON;


On Friday, 1 May 2015 at 14:19:26 UTC, Laeeth Isharc wrote:
> On Wednesday, 29 April 2015 at 18:48:22 UTC, Laeeth Isharc wrote:
>> Hi.
>>
>> What's the best way to pass the contents of a file to the stream parser without reading the whole thing into memory first?  I get an error if using byLine because the kind of range this function returns is not what the stream parser is expecting.
>>
>> There is an optional filename argument to parseJSONStream, but I am not sure what this is for, since it still requires the first argument for the input if the filename is passed.
>>
>>
>> Thanks.
>>
>>
>> Laeeth.
>
> some more suggestions for enhancements here:
> https://github.com/s-ludwig/std_data_json/issues/12
>
> a big advantage of languages like python is that there are end-to-end examples of using the language+libraries to accomplish real work.  (they don't need to be more than a screenful of code, but for the newcomer having it all set out makes a big difference).  since I believe a significant reason for people to switch to D will be a realisation that actually CPU time isn't free, it makes sense to be accommodating to people who previously used scripting types of languages for these jobs.
>
>
>
> Laeeth.

This line can be removed:
                .map!(ch => ch.idup)


On Friday, 1 May 2015 at 20:02:46 UTC, Ilya Yaroshenko wrote:
> Current std.stdio is deprecated. This ad-hoc should works.
>
> 	auto json = File("fileName")
> 		.byChunk(1024 * 1024) //1 MB. Data cluster equals 1024 * 4
> 		.map!(ch => ch.idup)
> 		.joiner
> 		.map!(b => cast(char)b)
> 		.parseJSON;

On Friday, 1 May 2015 at 20:04:58 UTC, Ilya Yaroshenko wrote:
> This line can be removed:
>                 .map!(ch => ch.idup)
>
>
> On Friday, 1 May 2015 at 20:02:46 UTC, Ilya Yaroshenko wrote:
>> Current std.stdio is deprecated. This ad-hoc should works.
>>
>> 	auto json = File("fileName")
>> 		.byChunk(1024 * 1024) //1 MB. Data cluster equals 1024 * 4
>> 		.map!(ch => ch.idup)
>> 		.joiner
>> 		.map!(b => cast(char)b)
>> 		.parseJSON;

Thanks for this.  I am trying to call parseJSONStream, so I will see if that works.

(Sample code here:
https://github.com/s-ludwig/std_data_json/blob/master/source/stdx/data/json/parser.d
)

It doesn't like it.  Any thoughts ?

lexer.d(257): Error: safe function 'stdx.data.json.parser.JSONLexerRange!(MapResult!(__lambda3, Result), cast(LexOptions)0, __lambda31).JSONLexerRange.empty' cannot call system function 'app.lookupTickers.MapResult!(__lambda3, Result).MapResult.empty'

string[2][] lookupTickers(string dataSource,string[] searchItems)
{
	import stdx.data.json;
	import std.conv:to;
	import std.algorithm:canFind,countUntil,joiner,map;
	import std.string:toLower;
	bool found=false;
	bool checkedCode=false;
	bool checkedName=false;
	string[2][] ret;
	string buf;
	auto filename="../importquandl/"~dataSource~".json";
	//auto data=cast(string)std.file.read(filename);
	auto data = File("fileName")
		.byChunk(100*1024 * 1024) //1 MB. Data cluster equals 1024 * 4
		// .map!(ch => ch.idup)
		.joiner
		.map!(b => cast(char)b);
	auto range1=parseJSONStream(data);
}

You can use std.json or create TrustedInputRangeShell template with @trasted methods:


struct TrustedInputRangeShell(Range)
{
    Range* data;

    auto front() @property @trusted { return (*data).front; }

    //etc
}

But I am not sure about other parseJSONStream bugs.