Request for List directory contents

This was discussed here.

Do you guys have a working version for both
target operating systems?
Can you post it here so anybody can use it
and maybe we can persuade Walter to include them
in the next phobos release?

Thanks
Ant

Ant

I've got several STL-like directory system search classes in STLSoft. (Read
"http://www.windevnet.com/documents/win0303a/" for more info. Note that you
*must* put in the trailing slash in that URL, or the WinDevNet doc finder
will fail). There is the WinSTL basic_findfile_sequence<> template (uses
FindFirst/NextFile(Ex)(), ANSI or Unicode), and the UNIXSTL readdir_sequence
(uses opendir()/readdir()) and findfile_sequence (uses glob()) classes. If
you've already got STLSoft, check 'em out. (If you've not, why not? Maybe
you don't like C++. Try http://stlsoft.org/downloads.html or
http://www.digitalmars.com/download/freecompiler.html)

These three classes follow the STLSoft philosophy of adapting operating system APIs to the STL way of doing things (e.g. for_each-able iterator ranges), but all three do things slightly differently, reflecting the considerable differences between the methods of searching on Win32 & UNIX.

If there was a petition got up, I've no doubt that I could do some porting, putting a D (i.e. foreach) face onto them. If it was up to me, I'd probably still go for separate types. Otherwise, we'd need to use only the common functionality, which would be to return all entries, all file entries, all directory entries (with or without dots). It may be pretty simple to add in the regexp code, and then platform-independent pattern matching could be done. (Hmmm, this is starting to sound tempting ...)

The other option, which I'd planned to bring up in a couple of months, is to
use a new library that I'm writing for my new C/C++ User's Journal. (It
features in the November issue, which should be available just about now.)
The library is called recls (for recursive ls), and it is a
platform-independent file-searching C library (implemented in C++). The
purpose of the column, called Positive Integration, is to show how to
integrate C and C++ with other languages/technologies. So the first few
instalments will feature recls, and provide mappings to a host of languages,
including C++ (wrapper classes), C++ (STL-like sequences), C#, COM (IEnumXXX
and Automation collections), D (foreach-able), Delphi, Java, Managed C++,
Perl, Python and Ruby. (I might do more, if the mood takes me ... :)

The first instalment contains the first version of the library, and the C++ wrapper classes. The second instalment (already written, coming out in January's issue) contains the C++ (STL) and C# mappings. The third will contain COM and D. Naturally the mappings can be made available when ready, rather than waiting for the column instalments.

I'd planned to suggest recls be a D standard library outer (i.e. non core) module, that would be treated in the same way other C components, such as D.c.stdio, i.e.. D.c.recls. Or, it might just be an external library that people would (hopefully) use, though this is less likely to happen than if it's in the standard library.

At this point, I'm stupidly busy as usual, so would rather not be doing any more D until the registry library is into Phobos and happy, but would be interested in everyone's opinions on this issue. My ideal is to wait for the D recls mapping (which I'll be doing in a couple of weeks), since once that's done there's not really a need for any other search components: it can do recursive or non-recursive searching, handles patterns, can search for files and/or directories (and/or links), and will be platform-independent, so no-one would have to care about platform-dependent client code hassles.All you do is specify a root directory to search, a search pattern (it will eventually support multiple search patterns, e.g. "*.cpp;r*.d;*.xl?"), and search flags. I even have plans for letting it handle searching FTP sites, if I can wrangle the URI syntax in a simple and unambiguous fashion.

I'd be interested in everyone's thoughts on possible directions:

(1) Crib the WinSTL/UNIXSTL sequence types, and D-ise them into separate
types (e.g. D.win32.filesys.readdir, D.linux.filesys.readdir).
(2) Option (1), but abstract the common features, and D-ise into a single
type (e.g. D.filesys.readdir).
(3) Use the recls D mapping, as a part of the (outer) standard library (e.g.
D.c.(filesys.)recls)
(4) Use the recls D mapping, as a SynSoft offering (e.g. synsoft.recls).
(5) P**s off and leave it to the Java-ites. :)

Technically speaking, I'd prefer option (3). If people preferred recls (and,
importantly, Walter approved it!), I guess I could put some effort into it
soonish, and have a Win32 version working within the next couple of weeks.
Options (1) & (2) will take longer, since there'll be a lot more work
involved (and it's work I had not planned to do, unlike the recls D
mapping). Option (4) would be much less likely to get used in the main, and
will therefore still require a standard library system file-searcher, so one
wonders what the point would be. Option (5) is easiest (for me). :)

Cheers

-- 
Matthew Wilson

STLSoft moderator and C++ monomaniac       (http://www.stlsoft.org)
Contributing editor, C/C++ Users Journal
(www.synesis.com.au/articles.html#columns)

"But if less is more, think how much more more will be!" -- Dr Frazier Crane

----------------------------------------------------------------------------
---




"Ant" <Ant_member@pathlink.com> wrote in message news:bls8qm$2935$1@digitaldaemon.com...
> Request for List directory contents
>
> This was discussed here.
>
> Do you guys have a working version for both
> target operating systems?
> Can you post it here so anybody can use it
> and maybe we can persuade Walter to include them
> in the next phobos release?
>
> Thanks
> Ant
>
>

In article <blsn7k$2v3p$1@digitaldaemon.com>, Matthew Wilson says...
>
>Ant
>
>I've got several STL-like directory system search classes in STLSoft.

You are very generous, however I wouldn't know where to start.

Let me propose another way of doing this:
(I'll repost this in a separate post)
Someone (Walter or any other smart guy) should make public the
desired interface for D and them some hands on guy (Ant or any other guy)
could make the implementation of that interface
(we will use the version() not the interface/implementation).
even a less then good (I don't mean buggy)
implementation is better then nothing (I guess).

bellow is what I mean by directory contents:
(this might be a
"pointless wrappers around C runtime library functions or OS API functions",
let me know please)

First let me ask a couple of questions (see code bellow):

technical:
- why do I get segfault on free(void*) ??? I thought I need it...
- why is diret.d_name a D char[]? (I printf it with %.*s not %s)
- if I get a string from C "char *cString" do I need to free it after
   converting it to a D array with char[] s = string.toString(cString); ?

filosofical:
- do we need this?
- is this the way to go? I don't mean the specifics of my implementation
   but the general idea.
- should we try to extract a common interface with what Burton posted on
   http://www.digitalmars.com/drn-bin/wwwnews?D/4753
   so we can have a window and linux implementation?
- should I just drop it as it has no interest?

community:
- does anybody have something better that want's to share?

finally here it is what I waisted the entire night with:
(I believe is waisted time because some of you must
have already done it)

//###########################################################

module utils.Directory;

private import string;

public:
enum FileTypes
{
DIRECTORY = 4,
FILE = 8,
S_LINK = 10
}

extern(C)
{

struct DIR;

// this structure will not work on all systems
// the the size of the integer will vary
struct dirent
{
uint d_ino;
uint d_off;
ushort d_reclen;
char d_type;
char[256] d_name;  /*this limitation is imposed by the system*/
}
;

DIR* opendir(char* name);
dirent* readdir(DIR* dir);
void rewinddir(DIR *dir);

void free(void*);
}

/**
* A directory error
* No error codes for now, just inadequate descriptive message
* \todo get the errno?
* \todo define some code errors as windows and un*x will be different?
*/
class DirectoryError : Error
{
this(char[] message)
{
super(message);
}
}

/**
* A directory content browser.
* \todo get meaninfull information on the directory entries
*/
public:
class Directory
{
int count = 0;

DIR* dir;
dirent* entry;

dirent*[] dirs;
dirent*[] files;


public:

/**
* Creates a Directory object for the directory
* @param directory must be a directory name
*/
this(char[] directory)
{
if ( directory === null )
{
throw new DirectoryError("Directory name not received");
}
dir = opendir((char*)directory);
if ( dir === null )
{
throw new DirectoryError("can't open directory ");
}
}

~this()
{
if ( dir !== null )
{
//free(dir);
}
if ( entry !== null )
{
//free(entry);
}
//freeAndClearDirs();
//freeAndClearFiles();
}

bit next()
{
if ( entry !== null )
{
free(entry);
entry = null;
}
//printf("\n");
dirent* entry = readdir(dir);
if ( entry !== null )
{
//printf("entry d_ino    %d\n",entry.d_ino);
//printf("entry d_off    %d\n",entry.d_off);
//printf("entry d_reclen %d\n",entry.d_reclen);
//printf("entry d_type   %c\n",entry.d_type);
printf("%d %d %.*s\n",count++,(int)entry.d_type,entry.d_name);
}
//return entry;
return entry !== null;
}

/**
* Resets the read pointer to the first entry
*/
void rewind()
{
rewinddir(dir);
}

private dirent*[] getEntries(FileTypes type)
{
dirent*[] entries;
dirent* entry;
do
{
entry = readdir(dir);
if ( entry !== null )
{
if ( entry.d_type == type )
{
entries ~= entry;
}
else
{
//free(entry);
}
}
} while (entry !== null);
return entries;
}

/**
* get and array with all the subdirectories of this directory
* @return and array with all the subdirectories
* \todo include the simbolic links
* \todo avoid scan the directory twice to get the subdirs and the files
*/
dirent*[] getDirs()
{
//freeAndClearDirs();
rewinddir(dir);
dirs = getEntries(FileTypes.DIRECTORY);
return dirs;
}

/**
* get and array with all the files of this directory
* @return and array with all the files
* \todo include the simbolic links
* \todo avoid scan the directory twice to get the subdirs and the files
*/
dirent*[] getFiles()
{
//freeAndClearDirs();
rewinddir(dir);
files = getEntries(FileTypes.FILE);
return dirs;
}

/**
* Gets a sorted array with all the subdirectory names
* @return a sorted array with all the subdirectory names
*/
char[][] getNames(FileTypes type)
{
char[][] names;
rewinddir(dir);
dirent* entry = readdir(dir);
while ( entry !== null )
{
if ( entry.d_type == type )
{
names[] ~= string.toString(entry.d_name);
}
//free(entry);
entry = readdir(dir);
}
return names.sort;
}

/**
* Gets a sorted array of the subdirectories
* @return a sorted array of the subdirectories
*/
char[][] getDirNames()
{
return getNames(FileTypes.DIRECTORY);
}

/**
* Gets a sorted array of the files on this directory
* @return a sorted array of the files on this directory
*/
char[][] getFileNames()
{
return getNames(FileTypes.FILE);
}

/**
* Frees the content of the dirs array and sets it's length to 0
*/
private void freeAndClearDirs()
{
foreach(dirent* d ; dirs)
{
free(d);
}
dirs.length = 0;
}

/**
* Frees the content of the files array and setst it;s length to 0
*/
private void freeAndClearFiles()
{
foreach(dirent* f ; files)
{
free(f);
}
files.length = 0;
}


}

/**
* just testing the thing
*/
void main(char[][] argv)
{

char [] dirName;

if ( argv.length < 2 )
{
dirName = ".";
}
else
{
dirName = argv[1];
}

Directory dir = new Directory(dirName);
bit d;

printf("\n###################\n");
printf("################### All entries of %.*s\n",dirName);
printf("###################\n\n");

do
{
d = dir.next();
} while(d);

printf("\n###################\n");
printf("################### All subdirectories of %.*s\n",dirName);
printf("###################\n\n");

foreach(dirent* entry; dir.getDirs() )
{
printf("\t%.*s\n",entry.d_name);
}

printf("\n###################\n");
printf("################### All files of %.*s\n",dirName);
printf("###################\n\n");

foreach(dirent* entry; dir.getFiles() )
{
printf("\t%.*s\n",entry.d_name);
}

printf("\n###################\n");
printf("################### directory names of %.*s\n",dirName);
printf("###################\n\n");

foreach(char[] name; dir.getDirNames() )
{
printf("\t%.*s\n",name);
}

printf("\n###################\n");
printf("################### file names of %.*s\n",dirName);
printf("###################\n\n");

foreach(char[] name; dir.getFileNames() )
{
printf("\t%.*s\n",name);
}

}

//##########################################################

thanks

Ant

> Let me propose another way of doing this:
> (I'll repost this in a separate post)
> Someone (Walter or any other smart guy) should make public the
> desired interface for D and them some hands on guy (Ant or any other guy)
> could make the implementation of that interface
> (we will use the version() not the interface/implementation).
> even a less then good (I don't mean buggy)
> implementation is better then nothing (I guess).

Sorry, I misunderstood. I thought you were asking for whether an implementation was, or is soon likely to be, available; I didn't realise you were wanting to implement it yourself. Go for it! :)

> Let me propose another way of doing this:
> (I'll repost this in a separate post)
> Someone (Walter or any other smart guy) should make public the
> desired interface for D and them some hands on guy (Ant or any other guy)
> could make the implementation of that interface
> (we will use the version() not the interface/implementation).
> even a less then good (I don't mean buggy)
> implementation is better then nothing (I guess).

Sorry, I misunderstood. I thought you were asking for whether an implementation was, or is soon likely to be, available; I didn't realise you were wanting to implement it yourself.

Go for it! :)

That's really cool Matthew.  I vote for option 2 (but 3 sounds nice too)

Sean

"Matthew Wilson" <matthew@stlsoft.org> wrote in message news:blsn7k$2v3p$1@digitaldaemon.com...
> At this point, I'm stupidly busy as usual, so would rather not be doing
any
> more D until the registry library is into Phobos and happy, but would be interested in everyone's opinions on this issue. My ideal is to wait for
the
> D recls mapping (which I'll be doing in a couple of weeks), since once that's done there's not really a need for any other search components: it can do recursive or non-recursive searching, handles patterns, can search for files and/or directories (and/or links), and will be platform-independent, so no-one would have to care about
platform-dependent
> client code hassles.All you do is specify a root directory to search, a search pattern (it will eventually support multiple search patterns, e.g. "*.cpp;r*.d;*.xl?"), and search flags. I even have plans for letting it handle searching FTP sites, if I can wrangle the URI syntax in a simple
and
> unambiguous fashion.
>
> I'd be interested in everyone's thoughts on possible directions:
>
> (1) Crib the WinSTL/UNIXSTL sequence types, and D-ise them into separate
> types (e.g. D.win32.filesys.readdir, D.linux.filesys.readdir).
> (2) Option (1), but abstract the common features, and D-ise into a single
> type (e.g. D.filesys.readdir).
> (3) Use the recls D mapping, as a part of the (outer) standard library
(e.g.
> D.c.(filesys.)recls)
> (4) Use the recls D mapping, as a SynSoft offering (e.g. synsoft.recls).
> (5) P**s off and leave it to the Java-ites. :)
>
> Technically speaking, I'd prefer option (3). If people preferred recls
(and,
> importantly, Walter approved it!), I guess I could put some effort into it
> soonish, and have a Win32 version working within the next couple of weeks.
> Options (1) & (2) will take longer, since there'll be a lot more work
> involved (and it's work I had not planned to do, unlike the recls D
> mapping). Option (4) would be much less likely to get used in the main,
and
> will therefore still require a standard library system file-searcher, so
one
> wonders what the point would be. Option (5) is easiest (for me). :)

Good to hear. :)

Option 2 will take longer, and will not do the recursive stuff, but will be
100% pure D. (I don't know whether this issue is going to be important to
the anal, not to say ludicrous, degree it is with the 100% pure Java
movement. I hope not.)
Option 3 will be sooner, and will recurse, but will be a D+C lib. (However,
since we'll always have dependencies on C libs in D, I personally don't
think it's an issue).

My suspicion is that this will come down to a Walter-decision, and that he'll want option 2.

I think the way to go is for me to do the recls-D mapping, and advertise it here, and people can decide from use, rather than conversation, what they feel about it.


"Sean L. Palmer" <palmer.sean@verizon.net> wrote in message news:blts2r$2g3n$1@digitaldaemon.com...
> That's really cool Matthew.  I vote for option 2 (but 3 sounds nice too)
>
> Sean
>
> "Matthew Wilson" <matthew@stlsoft.org> wrote in message news:blsn7k$2v3p$1@digitaldaemon.com...
> > At this point, I'm stupidly busy as usual, so would rather not be doing
> any
> > more D until the registry library is into Phobos and happy, but would be interested in everyone's opinions on this issue. My ideal is to wait for
> the
> > D recls mapping (which I'll be doing in a couple of weeks), since once that's done there's not really a need for any other search components:
it
> > can do recursive or non-recursive searching, handles patterns, can
search
> > for files and/or directories (and/or links), and will be platform-independent, so no-one would have to care about
> platform-dependent
> > client code hassles.All you do is specify a root directory to search, a search pattern (it will eventually support multiple search patterns,
e.g.
> > "*.cpp;r*.d;*.xl?"), and search flags. I even have plans for letting it handle searching FTP sites, if I can wrangle the URI syntax in a simple
> and
> > unambiguous fashion.
> >
> > I'd be interested in everyone's thoughts on possible directions:
> >
> > (1) Crib the WinSTL/UNIXSTL sequence types, and D-ise them into separate
> > types (e.g. D.win32.filesys.readdir, D.linux.filesys.readdir).
> > (2) Option (1), but abstract the common features, and D-ise into a
single
> > type (e.g. D.filesys.readdir).
> > (3) Use the recls D mapping, as a part of the (outer) standard library
> (e.g.
> > D.c.(filesys.)recls)
> > (4) Use the recls D mapping, as a SynSoft offering (e.g. synsoft.recls).
> > (5) P**s off and leave it to the Java-ites. :)
> >
> > Technically speaking, I'd prefer option (3). If people preferred recls
> (and,
> > importantly, Walter approved it!), I guess I could put some effort into
it
> > soonish, and have a Win32 version working within the next couple of
weeks.
> > Options (1) & (2) will take longer, since there'll be a lot more work
> > involved (and it's work I had not planned to do, unlike the recls D
> > mapping). Option (4) would be much less likely to get used in the main,
> and
> > will therefore still require a standard library system file-searcher, so
> one
> > wonders what the point would be. Option (5) is easiest (for me). :)
>
>

Matthew Wilson wrote:
> Good to hear. :)
> 
> Option 2 will take longer, and will not do the recursive stuff, but will be
> 100% pure D. (I don't know whether this issue is going to be important to
> the anal, not to say ludicrous, degree it is with the 100% pure Java
> movement. I hope not.)
> Option 3 will be sooner, and will recurse, but will be a D+C lib. (However,
> since we'll always have dependencies on C libs in D, I personally don't
> think it's an issue).
> 
> My suspicion is that this will come down to a Walter-decision, and that
> he'll want option 2.
> 
> I think the way to go is for me to do the recls-D mapping, and advertise it
> here, and people can decide from use, rather than conversation, what they
> feel about it.
> ...
> 

Walter, or whoever wrote the D language specs, has already indicated that (paraphrased) "D will link easily with C, as it is rediculous to think of translating all of the C libraries".  So those who are purists should probably think again.

That said, I believe that it is beneficial that as much as feasible of the language features and tools be implemented in D.  D doesn't have the same reasons that Java has for being purist about it.  It's not aiming at full portability for compiled code.  But code in C often makes use of things like doubly indirect pointers, inintelligible macros, lots and lots of casts, etc., which makes it difficult to read and understand.  Of course, I'm sure that obfuscated code could be written in D, even without dropping into assembler.  But the language doesn't positively *encourage* the practice.

"Charles Hixson" <charleshixsn@earthlink.net> wrote in message news:blvjdn$1r94$1@digitaldaemon.com...
> Matthew Wilson wrote:
> > Good to hear. :)
> >
> > Option 2 will take longer, and will not do the recursive stuff, but will
be
> > 100% pure D. (I don't know whether this issue is going to be important
to
> > the anal, not to say ludicrous, degree it is with the 100% pure Java
> > movement. I hope not.)
> > Option 3 will be sooner, and will recurse, but will be a D+C lib.
(However,
> > since we'll always have dependencies on C libs in D, I personally don't think it's an issue).
> >
> > My suspicion is that this will come down to a Walter-decision, and that he'll want option 2.
> >
> > I think the way to go is for me to do the recls-D mapping, and advertise
it
> > here, and people can decide from use, rather than conversation, what
they
> > feel about it.
> > ...
> >
>
> Walter, or whoever wrote the D language specs, has already indicated that (paraphrased) "D will link easily with C, as it is rediculous to think of translating all of the C libraries".  So those who are purists should probably think again.

Agreed. It makes perfect sense to me to use a C library that is available just about now, rather than wait for one that may take a long time just to get pure D.

> That said, I believe that it is beneficial that as much as feasible of the language features and tools be implemented in D.  D doesn't have the same reasons that Java has for being purist about it.  It's not aiming at full portability for compiled code.  But code in C often makes use of things like doubly indirect pointers, inintelligible macros, lots and lots of casts, etc., which makes it difficult to read and understand.  Of course, I'm sure that obfuscated code could be written in D, even without dropping into assembler.  But the language doesn't positively *encourage* the practice.

You've not made your preference clear. Option (2) or (3)? (Or (5)?)

:)

btw, I've just started playing around with the recls-D mapping, and I reckon I can have it done in a very short space of time indeed.

Matthew