Dear D community!

I would like to give you some information about status of doost library, as it got few substantial updates recently.

*NEW MODULES*
*doost.storage.Storage/FileStorage*
I want to start from this one as this module as it introduces concept
which is used in other modules. Basically Storage is continuous array of
elements of same type. String and array of ints are also kinds of storage.
The difference is that Storage can be much bigger than computer memory.
Storage can be also mapped to real physical devices such as File,
Network connection etc. In fact it is very similar concept to Stream,
but it has additional requirement that all elements of Storage are of
the same type. Storage have narrower set of operations than arrays.
Supported operations are:
- checking if element with given index is in storage or outside of
stream of elements - eos
- consuming first or few first elements from storage - get
- peeking into any element in storage (only forward) - peek
- it is also possible to operate directly on 'working window' - frame

You can move across storage in one direction and operate on 'working
window'. First element of storage have always index of '0'. Below simple
visualization of storage:
sink -----------|O--working window--O|----------source
O - optional buffers for input and output

It might look complicated in a first look but, it's not in fact. Using this module it is possible to write all operations on arrays in a such a way that it will be possible to use any other Storage than string and have it read data from e.g. file.

In storage module there are definitions of above operations for strings,
so you can use:
char[] text;
text.put("blabla");
char c= text.get()
if (text.eos(10)) throw new Exception("Error");

When using in you program you can template it in such a way that it will
work nevertheless user will put as Storage string or e.g. FileStorage,
which will read data from file.
When using for strings there should be no performance penalty as all
functions should be inlined Unfortunately DMD does not optimize
functions which takes arguments by ref. (See: bugzilla issue: 2008).
There is one concrete Storage implemented for files in package
doost.storage.

This module is already quite important in my library, but I still think it needs some more conceptual work.

*doost.text.Scanner*
Module Scanner is intended to be used for parsing texts. It consists from
set of functions, which uses internally only one operation: scan. Every
function scan returnes struct Matcher, which keeps information about
results of scanning: status of scanning, error message, matched position,
index of match (in case of matching against sets of sequences or sets of
characters).

position - position in input
repeatno - how many times repeat scanning; for uint.max functions scan
for more than zero matches
All below functions works also with instances of Storages (see above).

Following template functions are available:
Matcher scan(input, CharClass, position, repeatno)  //CharClass ->
basically checks if input from 'position' begins with one of characters
defined in CharClass
Matcher scan(input, SequenceSet, position, repeatno) //SequenceSet ->
same as CharClass, but for sequences
Matcher scan(input, RegExp, position, repeatno)
Matcher scan(input, array, position, repeatno)
Matcher scan(input, element_of_array, position, repeatno)

For below functions condition can be any of types for scan  + Matcher;
uint skip(ref T input, S condition, uint repeatno=1) // returns number
of skipped characters
StorageType!(T) munch(T, S)(ref T input, S condition, uint repeatno=1)
// consumes characters/sequences/etc. returning them from function

Additional high level functions:
Matcher scanEscapeSequence - scans for escape sequence
Matcher scanString - scans for string
Matcher scanRegion - scans for region; region is delimited by matching
characters e.g. [5,5,4]
Matcher scanNumber - scans for number; needs some more work
Matcher scanDate - scans for number; needs some more work

Scanner uses internally among others CharClass by Jascha Wetzel from Tango ported to Phobos. Module needs some additional work to make it feature complete.

*doost.text.Escaper*
This is quite simple module, which should be used for
escaping/unescaping strings. It should work for every escape sequence
defined by D. It has only two functions:

escape(input)
unescape(input)

Escaper module should work also for Storage (see above).

*doost.util.serializer*
This is automatic serialization library. Currently it has two compile
time 'backends': TextArchive and JsonArchive.
Some of most important features:
- can work with arrays and as well with storages
- highly customizable
- thread/exception safe
- tracking of references (classes/pointers)
- user defined types versioning
- importing of old classes
- user defined constructors
- user defined loaders/dumpers
- different archive classes

There is quite big suite of unit tests for serialization. You can see serializer features in usage there. Serializer works also with storages (see above).

*doost.api.Common/doost.core*
- doost.api is meant as something similar to tango.groups; Common is for
commonly used definitions/imports etc.
- doost.core: Any (not many changes here), Traits (templates mostly
taken from Phobos/Tango)


*UPDATED MODULES*
*doost.util.DUnit (doost.DUnit)*
This module was improved to make tests visualization in console more
readable and usefull. Currently it shows nice table with all necessary
informations. You can set width of this table, you can say if you want to
get timing or just result of unit test, you can also disable one or more
tests. It is possible to define groups of unittests. There are printed some
useful statistics at the end of table. Traces for tests are shown only in
case of failures.

*doost.util.config.ProgramOptions*
Program Options now internally uses serializer to store information. It
means that every type you will use to configure your program can be
automatically stored as text. I think that this module needs some redesign.
It is especially visible after introducing serializer, which obsolete few
features and force to change implementation. Currently few unit tests (not
most important) doesn't pass, but making them work needs rethinking
architecture and redesign. I will keep it for later :-)

*WORK IN PROGRESS*
Currently I am working on database package. Idea is to make operation on
database (almost) completely type-safe. I want to achieve this using Sql
Object pattern. Thanks to this it will be possible to send such an
object to database layer and get recordset as result.

*HOW CAN YOU HELP (if you feel it is worthy)*
- comment about design - especially about interfaces and use-cases.
- prepare patches and attach them as tickets
- documentation: it is rather sparse. I will be happy to get some help here
- adding unit tests - fixing bugs
- adding support for Tango. In this I would hope for some help from
Tango team especially with Storage concept.
- improving build system. Currently I am using CodeBlocks, but its far from
perfect solution. There is a need for better/more universal builder. Right
now it should be dsss, later maybe something other.
- making Sqlite database from DDBI project working in latest revision
:-) It's terribly broken right now.

Few modules of doost have dependencies on ddbi and std2, so you will need to play a little bit with your compiler configuration. I used ddbi revision 62, as trunk version is broken for Sqlite. I successfully run doost libraries on Tango with Tangobos (I had to add string aliases to object.di), but doost itself doesn't contain any dependencies on Tango. CodeBlocks project files might not work in your environment, so your probably will have to adjust them to.

If you think, you can help with some tasks regarding library please let me know e.g. by e-mail.

-- 
Regards
Marcin Kuszczak (Aarti_pl)
-------------------------------------
Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl)
Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/
-------------------------------------

Marcin Kuszczak wrote:
> - making Sqlite database from DDBI project working in latest revision :-) It's terribly broken right now.
I had a small play with this recently. It seems to be missing StatementBinder. Removing the prepared statements code will let it compile and run (without prepared statements), or you could implement the StatementBinder and it may be ok I guess (not sure where that is up to).

For tango you also need to rename fromUtf8z to fromStringz and toUtf8z to toStringz

Since doost is based on boost, it'd be nice to see a simple list of which boost modules have been ported.  This helps those of us already familiar with various boost libraries...

PS: I'm glad to see doost is alive and well!

Marcin Kuszczak wrote:

> Dear D community!
> 
> I would like to give you some information about status of doost library, as it got few substantial updates recently.
> 
> *NEW MODULES*
> *doost.storage.Storage/FileStorage*
> I want to start from this one as this module as it introduces concept
> which is used in other modules. Basically Storage is continuous array of
> elements of same type. String and array of ints are also kinds of storage.
> The difference is that Storage can be much bigger than computer memory.
> Storage can be also mapped to real physical devices such as File,
> Network connection etc. In fact it is very similar concept to Stream,
> but it has additional requirement that all elements of Storage are of
> the same type. Storage have narrower set of operations than arrays.
> Supported operations are:
> - checking if element with given index is in storage or outside of
> stream of elements - eos
> - consuming first or few first elements from storage - get
> - peeking into any element in storage (only forward) - peek
> - it is also possible to operate directly on 'working window' - frame
> 
> You can move across storage in one direction and operate on 'working
> window'. First element of storage have always index of '0'. Below simple
> visualization of storage:
> sink -----------|O--working window--O|----------source
> O - optional buffers for input and output
> 
> It might look complicated in a first look but, it's not in fact. Using this module it is possible to write all operations on arrays in a such a way that it will be possible to use any other Storage than string and have it read data from e.g. file.
> 
> In storage module there are definitions of above operations for strings,
> so you can use:
> char[] text;
> text.put("blabla");
> char c= text.get()
> if (text.eos(10)) throw new Exception("Error");
> 
> When using in you program you can template it in such a way that it will
> work nevertheless user will put as Storage string or e.g. FileStorage,
> which will read data from file.
> When using for strings there should be no performance penalty as all
> functions should be inlined Unfortunately DMD does not optimize
> functions which takes arguments by ref. (See: bugzilla issue: 2008).
> There is one concrete Storage implemented for files in package
> doost.storage.
> 
> This module is already quite important in my library, but I still think it needs some more conceptual work.
> 
> *doost.text.Scanner*
> Module Scanner is intended to be used for parsing texts. It consists from
> set of functions, which uses internally only one operation: scan. Every
> function scan returnes struct Matcher, which keeps information about
> results of scanning: status of scanning, error message, matched position,
> index of match (in case of matching against sets of sequences or sets of
> characters).
> 
> position - position in input
> repeatno - how many times repeat scanning; for uint.max functions scan
> for more than zero matches
> All below functions works also with instances of Storages (see above).
> 
> Following template functions are available:
> Matcher scan(input, CharClass, position, repeatno)  //CharClass ->
> basically checks if input from 'position' begins with one of characters
> defined in CharClass
> Matcher scan(input, SequenceSet, position, repeatno) //SequenceSet ->
> same as CharClass, but for sequences
> Matcher scan(input, RegExp, position, repeatno)
> Matcher scan(input, array, position, repeatno)
> Matcher scan(input, element_of_array, position, repeatno)
> 
> For below functions condition can be any of types for scan  + Matcher;
> uint skip(ref T input, S condition, uint repeatno=1) // returns number
> of skipped characters
> StorageType!(T) munch(T, S)(ref T input, S condition, uint repeatno=1)
> // consumes characters/sequences/etc. returning them from function
> 
> Additional high level functions:
> Matcher scanEscapeSequence - scans for escape sequence
> Matcher scanString - scans for string
> Matcher scanRegion - scans for region; region is delimited by matching
> characters e.g. [5,5,4]
> Matcher scanNumber - scans for number; needs some more work
> Matcher scanDate - scans for number; needs some more work
> 
> Scanner uses internally among others CharClass by Jascha Wetzel from Tango ported to Phobos. Module needs some additional work to make it feature complete.
> 
> *doost.text.Escaper*
> This is quite simple module, which should be used for
> escaping/unescaping strings. It should work for every escape sequence
> defined by D. It has only two functions:
> 
> escape(input)
> unescape(input)
> 
> Escaper module should work also for Storage (see above).
> 
> *doost.util.serializer*
> This is automatic serialization library. Currently it has two compile
> time 'backends': TextArchive and JsonArchive.
> Some of most important features:
> - can work with arrays and as well with storages
> - highly customizable
> - thread/exception safe
> - tracking of references (classes/pointers)
> - user defined types versioning
> - importing of old classes
> - user defined constructors
> - user defined loaders/dumpers
> - different archive classes
> 
> There is quite big suite of unit tests for serialization. You can see serializer features in usage there. Serializer works also with storages (see above).
> 
> *doost.api.Common/doost.core*
> - doost.api is meant as something similar to tango.groups; Common is for
> commonly used definitions/imports etc.
> - doost.core: Any (not many changes here), Traits (templates mostly
> taken from Phobos/Tango)
> 
> 
> *UPDATED MODULES*
> *doost.util.DUnit (doost.DUnit)*
> This module was improved to make tests visualization in console more
> readable and usefull. Currently it shows nice table with all necessary
> informations. You can set width of this table, you can say if you want to
> get timing or just result of unit test, you can also disable one or more
> tests. It is possible to define groups of unittests. There are printed
> some useful statistics at the end of table. Traces for tests are shown
> only in case of failures.
> 
> *doost.util.config.ProgramOptions*
> Program Options now internally uses serializer to store information. It
> means that every type you will use to configure your program can be
> automatically stored as text. I think that this module needs some
> redesign. It is especially visible after introducing serializer, which
> obsolete few features and force to change implementation. Currently few
> unit tests (not most important) doesn't pass, but making them work needs
> rethinking architecture and redesign. I will keep it for later :-)
> 
> *WORK IN PROGRESS*
> Currently I am working on database package. Idea is to make operation on
> database (almost) completely type-safe. I want to achieve this using Sql
> Object pattern. Thanks to this it will be possible to send such an
> object to database layer and get recordset as result.
> 
> *HOW CAN YOU HELP (if you feel it is worthy)*
> - comment about design - especially about interfaces and use-cases.
> - prepare patches and attach them as tickets
> - documentation: it is rather sparse. I will be happy to get some help
> here - adding unit tests - fixing bugs
> - adding support for Tango. In this I would hope for some help from
> Tango team especially with Storage concept.
> - improving build system. Currently I am using CodeBlocks, but its far
> from perfect solution. There is a need for better/more universal builder.
> Right now it should be dsss, later maybe something other.
> - making Sqlite database from DDBI project working in latest revision
> :-) It's terribly broken right now.
> 
> Few modules of doost have dependencies on ddbi and std2, so you will need to play a little bit with your compiler configuration. I used ddbi revision 62, as trunk version is broken for Sqlite. I successfully run doost libraries on Tango with Tangobos (I had to add string aliases to object.di), but doost itself doesn't contain any dependencies on Tango. CodeBlocks project files might not work in your environment, so your probably will have to adjust them to.
> 
> If you think, you can help with some tasks regarding library please let me know e.g. by e-mail.
> 

Daniel Giddings wrote:

> Marcin Kuszczak wrote:
>> - making Sqlite database from DDBI project working in latest revision :-) It's terribly broken right now.
> I had a small play with this recently. It seems to be missing StatementBinder. Removing the prepared statements code will let it compile and run (without prepared statements), or you could implement the StatementBinder and it may be ok I guess (not sure where that is up to).
> 
> For tango you also need to rename fromUtf8z to fromStringz and toUtf8z to toStringz

Yes, probably it's not difficult, but the problem is that you have to distract yourself from your goal to fix some other library. Currently AFAIK ddbi has no active maintainer. And it is really pity as it's important project for D.

-- 
Regards
Marcin Kuszczak (Aarti_pl)
-------------------------------------
Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl)
Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/
-------------------------------------

Jason House wrote:

> Since doost is based on boost, it'd be nice to see a simple list of which boost modules have been ported.  This helps those of us already familiar with various boost libraries...
> 
> PS: I'm glad to see doost is alive and well!

Well, currently scope of library is much wider than only libraries ported from Boost or inspired by Boost. While I was porting program_options I noticed that originall library can be easily extended with e.g. writing options back into physicall storage. D just allows to do much more, still having implementation quite simple.

Now I am working on D version of part of U++ library, which is intended to allow type safe database queries. (http://www.ultimatepp.org/src$Sql$SqlExp$en-us.html)

Imagine that you put database schema into file: http://www.dsource.org/projects/doost/browser/trunk/examples/database/TestSchema.schema

and then you use in your program typesafe versions of sql: http://www.dsource.org/projects/doost/browser/trunk/examples/database/PackageTest.d

And simple cases work already quite well :-)

I just see problem with sql expressions. In U++ it is done like below:
Select(BOOK.AUTHOR).WHERE((BOOK.ID == 100) && (BOOK.TITLE == "Dot"));

In D it is not possible (unfortunately) and must be written like below:
Select(BOOK.AUTHOR).WHERE(land(leq(BOOK.ID, 100), leq(BOOK.TITLE, "Dot")));

I hope that maybe Walter someday will reconsider adding extended operators overloading as addition to the current system.

-- 
Regards
Marcin Kuszczak (Aarti_pl)
-------------------------------------
Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl)
Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/
-------------------------------------