Simply put, I'd like a D styled dataformat alike JSON.
Somehow saying that I would like it D styled makes it difficult to
understand :)

"Georg Wrede" <georg.wrede@iki.fi> wrote in message news:gta2r9$1vls$1@digitalmars.com...
> Saaa wrote:
>>>> I'm still dangling between a full parsing at load and parse on demand.
>>>> A parse on demand (get) would do the following steps:
>>>> Get the type of the variable in which the data must be stored (in
>>>> string format)
>>>> Search for this type in every line of the char[][], when found check
>>>> whether the name is the same
>>>> and then convert the chars to that type and place the data in the
>>>> variable.
>>> This sounds complicated. Can you decide the input file format? Or is it from some other program that you can't change?
>>
>> The input file format would be like D
>> Like file.dat from the original post.
>> I just often have the need to save large arrays and other variables and I
>> thought
>> why not just save them like the way I would use them in my modules.
>
> Err, ok, lemme assume this is kind-of like a scrapbook thing. You collect nice arrays, filled data structures and the like. And the you'd want to read them into your programs.
Yes, except that the program creating them is the same as the one loading
them.
It's a simple safe (with very large multi arrays)

>
> Now, to know how to use the stuff, the program would need to not only be able to handle the data structures, but also know their names. Sure, you could have a parser that returns first the name of the thing, and then the thing itself. But then, to use it, you probably couldn't have an Universal program. Rather, you'd have to write the program different for each time you decide to use a particular item from the scrap book.
>
The JSON parser does the same thing. You give it the name of the array you want to read.

This is what the corresponding data files look like
- JSON version
"array" :
[
[
[
1,
2,
],
[
3,
4,
]
]
[
... etc :)
--

-D styled version
int[2][2][2] array = [[
[1,2],
[3,4]],
[
[5,6],
[7,8]];
--

> Which actually leads us to another simple solution. Why not write your scrapbook simply in total D format, and then include it in the program?
I really can't ask my customers to rebuild their app everytime they save something :)

>
>>> Suppose you already had this get function. How would you use it? An example would help.
>>
>> I use the get functions a few posts back, in the main:
>>
>> ---
>> void main()
>> {
>>  char[] filename = `data.dat`;
>>  char[][] file;
>>
>>  try{
>>   file = splitlines( cast(char[])read(filename) );
>>  }
>>  catch{
>>   throw new Exception("Couldn't load : " ~ filename);
>>  }
>>
>>  DData file_dd = new DData(file);
>>  int i;
>>  file_dd.get(`i`, i);
>> }
>> ---
>>
>> The data.dat should have a line like this to work:
>> --- data.dat
>> int i = 10;
>> ---
>
> So here main has to know that there will be a variable i in the data file, then get it, then presumably assign the value to the here defined "int i;". This way you have to do everything (and more) you would if you were to rename data.dat to data.d, include it, and then just use the variables.
The get function doesn't need to know that there will be an i in the data, it will just search for it.

>
>> I hope it all makes sense :)
>
> So, either I still don't get it, or you're doing Double Thinking. A picture might clarify:
>
> http://ocw.mit.edu/NR/rdonlyres/A166EB14-E7FD-473F-98FB-741A9817540C/0/chp_triangle.jpg
>
> It's very often that programmers spend time attempting to do something that is possible, but when it comes down to it, it either isn't, or it takes an inordinate amount of work /compared/ to the value of the goal.
>
I know it is possible because
1. it isn't much different than the JSON parser from tango
2. the compiler does the same

If the JSON writer won't write every item (which will be thousands of items) on a new line and reading in multi arrays, then I think I need to witch to Tango :D

Saaa wrote:
> Simply put, I'd like a D styled dataformat alike JSON.
> Somehow saying that I would like it D styled makes it difficult to understand :)
> 
> "Georg Wrede" <georg.wrede@iki.fi> wrote in message news:gta2r9$1vls$1@digitalmars.com...
>> Saaa wrote:
>>>>> I'm still dangling between a full parsing at load and parse on demand.
>>>>> A parse on demand (get) would do the following steps:
>>>>> Get the type of the variable in which the data must be stored (in string format)
>>>>> Search for this type in every line of the char[][], when found check whether the name is the same
>>>>> and then convert the chars to that type and place the data in the variable.
>>>> This sounds complicated. Can you decide the input file format? Or is it from some other program that you can't change?
>>> The input file format would be like D
>>> Like file.dat from the original post.
>>> I just often have the need to save large arrays and other variables and I thought
>>> why not just save them like the way I would use them in my modules.
>> Err, ok, lemme assume this is kind-of like a scrapbook thing. You collect nice arrays, filled data structures and the like. And the you'd want to read them into your programs.
> Yes, except that the program creating them is the same as the one loading them.
> It's a simple safe (with very large multi arrays)

Well, then were really talking about Classic Serialzation. In other words, a program writes data somewhere (possibly in human readable format), and then later (as in tomorrow) reads it in.

>> Now, to know how to use the stuff, the program would need to not only be able to handle the data structures, but also know their names. Sure, you could have a parser that returns first the name of the thing, and then the thing itself. But then, to use it, you probably couldn't have an Universal program. Rather, you'd have to write the program different for each time you decide to use a particular item from the scrap book.
>>
> The JSON parser does the same thing. You give it the name of the array you want to read.
> 
> This is what the corresponding data files look like
> - JSON version
> "array" :
> [
> [
> [
> 1,
> 2,
> ],
> [
> 3,
> 4,
> ]
> ]
> [
> ... etc :)
> --
> 
> -D styled version
> int[2][2][2] array = [[
> [1,2],
> [3,4]],
> [
> [5,6],
> [7,8]];

So, are we talking basically saving the state of the program for the next run?

>> Which actually leads us to another simple solution. Why not write your scrapbook simply in total D format, and then include it in the program?
> I really can't ask my customers to rebuild their app everytime they save something :)
> 
>>>> Suppose you already had this get function. How would you use it? An example would help.
>>> I use the get functions a few posts back, in the main:
>>>
>>> ---
>>> void main()
>>> {
>>>  char[] filename = `data.dat`;
>>>  char[][] file;
>>>
>>>  try{
>>>   file = splitlines( cast(char[])read(filename) );
>>>  }
>>>  catch{
>>>   throw new Exception("Couldn't load : " ~ filename);
>>>  }
>>>
>>>  DData file_dd = new DData(file);
>>>  int i;
>>>  file_dd.get(`i`, i);
>>> }
>>> ---
>>>
>>> The data.dat should have a line like this to work:
>>> --- data.dat
>>> int i = 10;
>>> ---
>> So here main has to know that there will be a variable i in the data file, then get it, then presumably assign the value to the here defined "int i;". This way you have to do everything (and more) you would if you were to rename data.dat to data.d, include it, and then just use the variables.
> The get function doesn't need to know that there will be an i in the data,
> it will just search for it.

Exactly. The get function doesn't have to know. But the rest of the program will have to know what to do with what get gives it. Right? Actually, /that/ is what I'd see as the hard part here.

>>> I hope it all makes sense :)
>> So, either I still don't get it, or you're doing Double Thinking. A picture might clarify:
>>
>> http://ocw.mit.edu/NR/rdonlyres/A166EB14-E7FD-473F-98FB-741A9817540C/0/chp_triangle.jpg
>>
>> It's very often that programmers spend time attempting to do something that is possible, but when it comes down to it, it either isn't, or it takes an inordinate amount of work /compared/ to the value of the goal.
>>
> I know it is possible because
> 1. it isn't much different than the JSON parser from tango
> 2. the compiler does the same
> 
> If the JSON writer won't write every item (which will be thousands of items)
> on a new line and reading in multi arrays, then I think I need to witch to Tango :D 

No problem. I'm not here to "save you from Tango". :-)


Okay, (I still feel I don't get it entirely), now I assume the thing is to save the state for the next invocation tomorrow.

Then I'd assume, you've basically got two things to do. First, write the code to save basic data types, and then the code to save compound types (as in structs, classes, trees, etc.). An OO approach would be to have each class have methods to write and to read the data.

I can't help feeling that either using e.g. the Json parser, or writing your own IO routines, should be the easy part. But then I feel using this get function would be the harder thing.

Incidentally, Andrei is about to publish printing arbitrary data structures in D (as part of Phobos), and if I suspect correctly, this will also be done in a way that you can both write /and/ read stuff from files. A little like Json. But of course, this might not be soon enough for any real program for a customer.

> Well, then were really talking about Classic Serialzation. In other
> words, a program writes data somewhere (possibly in human readable
> format), and then later (as in tomorrow) reads it in.
Ok, lets start from there :D

>
>>> Now, to know how to use the stuff, the program would need to not only be able to handle the data structures, but also know their names. Sure, you could have a parser that returns first the name of the thing, and then the thing itself. But then, to use it, you probably couldn't have an Universal program. Rather, you'd have to write the program different for each time you decide to use a particular item from the scrap book.
>>>
>> The JSON parser does the same thing. You give it the name of the array you want to read.
>>
>> This is what the corresponding data files look like
>> - JSON version
>> "array" :
>> [
>> [
>> [
>> 1,
>> 2,
>> ],
>> [
>> 3,
>> 4,
>> ]
>> ]
>> [
>> ... etc :)
>> --
>>
>> -D styled version
>> int[2][2][2] array = [[
>> [1,2],
>> [3,4]],
>> [
>> [5,6],
>> [7,8]];
>
> So, are we talking basically saving the state of the program for the next run?
Yes (also within a single run, but that doesn't really change things: same functions)


>>> So here main has to know that there will be a variable i in the data file, then get it, then presumably assign the value to the here defined "int i;". This way you have to do everything (and more) you would if you were to rename data.dat to data.d, include it, and then just use the variables.
>> The get function doesn't need to know that there will be an i in the
>> data,
>> it will just search for it.
>
> Exactly. The get function doesn't have to know. But the rest of the program will have to know what to do with what get gives it. Right? Actually, /that/ is what I'd see as the hard part here.
Well at the moment that is the easy part: a class just has a load function
which
sets all kinds of vars to the loaded values.
I already have my custom file loader, which isn't nice nor safe.

>> If the JSON writer won't write every item (which will be thousands of
>> items)
>> on a new line and reading in multi arrays, then I think I need to witch
>> to Tango :D
>
> No problem. I'm not here to "save you from Tango". :-)
Took me a while to not read the oppisite.

>
> Okay, (I still feel I don't get it entirely), now I assume the thing is to save the state for the next invocation tomorrow.
>
> Then I'd assume, you've basically got two things to do. First, write the code to save basic data types, and then the code to save compound types (as in structs, classes, trees, etc.). An OO approach would be to have each class have methods to write and to read the data.
I have this (except the OO approach)
It can read arrays will a certain dept (3 deep I think)
but the implementation is awefull, which made me think of what the
more general solution would be. Which was a D styled dataformat :-D

>
> I can't help feeling that either using e.g. the Json parser, or writing your own IO routines, should be the easy part. But then I feel using this get function would be the harder thing.
Why?
In the load method within a class you would first have the correct file
loaded
and then a list of vars to set.

>
> Incidentally, Andrei is about to publish printing arbitrary data structures in D (as part of Phobos), and if I suspect correctly, this will also be done in a way that you can both write /and/ read stuff from files. A little like Json. But of course, this might not be soon enough for any real program for a customer.
Wow, isn't this exactly what I wanted :D
How do you know about this? If within a month that would be ok.

Saaa wrote:
> If the JSON writer won't write every item (which will be thousands of items)
> on a new line and reading in multi arrays, then I think I need to witch to Tango :D 

Tango's JSON printer has options for pretty-printing and condensed printing. The default is condensed; everything will be on one line. If you choose to use the pretty printing style, you get a nicely indented file that's easy for humans to read.

This is what I came up with to save a multidimentional array in JSON. It compiles . . but does not work : )


int[100][100][10] bigArray;

//set the array to something other then zero
foreach( int[100][100] arr; bigArray)
  foreach(int index, int[100] miniarr; arr)
    miniarr[] = index;

auto json = new Json!(char);

with (json)
foreach(int index, int[100][100] arr; bigArray)
  foreach(int index2, int[100] miniarr; arr)
    value = object (pair("data_"~Conv.toString(index)~Conv.toString(index2),
array(miniarr[0])));
//I want to save the whole array of course but array() doesn't take an array
as such, I think.
//Also I only get one pair (which is nicely named data_999 )

auto outfile = new File("/data.dat");
json.print ((char[] s) {outfile.write(s);});
json.print ((char[] s) {Cout(s);});

"Christopher Wright" <dhasenan@gmail.com> wrote in message news:gtb02h$e4g$1@digitalmars.com...
> Saaa wrote:
>> If the JSON writer won't write every item (which will be thousands of
>> items)
>> on a new line and reading in multi arrays, then I think I need to witch
>> to Tango :D
>
> Tango's JSON printer has options for pretty-printing and condensed printing. The default is condensed; everything will be on one line. If you choose to use the pretty printing style, you get a nicely indented file that's easy for humans to read.
Ok, that's nice. If I can get it to do what I want I think I'll switch to
Tango then.
Not using that many std functions anyways although phobos2 looks really
neat.
Hope to see Tango and Phobos living together in dmd2 when it is announced
stable.

Saaa wrote:
> "Christopher Wright" <dhasenan@gmail.com> wrote in message news:gtb02h$e4g$1@digitalmars.com...
>> Saaa wrote:
>>> If the JSON writer won't write every item (which will be thousands of items)
>>> on a new line and reading in multi arrays, then I think I need to witch to Tango :D
>> Tango's JSON printer has options for pretty-printing and condensed printing. The default is condensed; everything will be on one line. If you choose to use the pretty printing style, you get a nicely indented file that's easy for humans to read.
> Ok, that's nice. If I can get it to do what I want I think I'll switch to Tango then.
> Not using that many std functions anyways although phobos2 looks really neat.
> Hope to see Tango and Phobos living together in dmd2 when it is announced stable. 

If your licensing requirements allow, you can probably just extract tango.text.json and use that -- it shouldn't have significant requirements on the rest of tango.

Can somebody help me out here?

> This is what I came up with to save a multidimentional array in JSON. It compiles . . but does not work : )
>
>
> int[100][100][10] bigArray;
>
> //set the array to something other then zero
> foreach( int[100][100] arr; bigArray)
>  foreach(int index, int[100] miniarr; arr)
>    miniarr[] = index;
>
> auto json = new Json!(char);
>
> with (json)
> foreach(int index, int[100][100] arr; bigArray)
>  foreach(int index2, int[100] miniarr; arr)
>    value = object
> (pair("data_"~Conv.toString(index)~Conv.toString(index2),
> array(miniarr[0])));
> //I want to save the whole array of course but array() doesn't take an
> array as such, I think.
> //Also I only get one pair (which is nicely named data_999 )
>
> auto outfile = new File("/data.dat");
> json.print ((char[] s) {outfile.write(s);});
> json.print ((char[] s) {Cout(s);});
>
> 

> If your licensing requirements allow
Commercial use should allow this I think.

>, you can probably just extract tango.text.json and use that -- it shouldn't have significant requirements on the rest of tango.

These are the dependencies:

module tango.text.json.Json;
 private import tango.core.Vararg;
   //va_arg!()()
   //va_list
 private import Float = tango.text.convert.Float;
   //Float.parse ()
   //Float.format ()

module tango.text.json.JsonEscape;
 private import Util = tango.text.Util;
   //Util.indexOf()
 private import Utf = tango.text.convert.Utf;
   //Utf.fromString32()

module tango.text.json.JsonParser;
 private import tango.util.container.more.Stack;
   //private Stack!(State, 16)       state;

I'm not sure whether all these have a Phobos equivalent.