I have recently finished compiling the zillion or so different software packages that will be needed to host the new D source-code repository site. Along the way, I experienced significant enough frustration that I'm proposing that we change the world. I'll tell you about my frustrating experience first. Then I'll tell you about what I think we should do about it.

<irrelevant>
I'm hosting the new site on a "virtual dedicated server". There are lots of ways
of doing this (especially in a unix environment (and especially especially in a
linux environment)), but my particular hosting provider does it in such a way
that I get full root access, including the ability to compile and install
software, rwx permissions to every directory in the tree, and the ability to
create arbitrary shell accounts. I can host as many domains as I want on my
server, and I can essentially use it for whatever purposes I like. I get all of
that, plus 3GB of storage space and 25GB of bandwidth each month, for only $15.
The name of the hosting company is http://www.tektonic.net . (The only reason I
mention their name at all is because it's such a good deal, I figure there'll be
someone else in the ng who will be impressed by that price as much as I was. I
don't make any money from tooting their horn. )
</irrelevant>

Anyhow, I've been installing lots of software that will be necessary to get the repository site up and running. I've had to compile pretty much everything from source. And I've got some big gripes:

1) You can extract your source archives pretty much anywhere in the directory tree, as long as you run 'make' and 'make install' from within these directories. This can lead to source directories all over the place. Some of the software packages that came preinstalled on my server left their source directories in some pretty weird places.

2) There is no standard unix location for the installation of binaries, library files, and include (header) files. Binaries might be installed to '/bin' or 'sbin' or '/usr/bin' or '/usr/sbin' or '/usr/local/bin'. Libraries might be installed into analagous directories (just substitute 'lib' for 'bin' in any of these examples) or they might be installed into '/var/lib'. Header files are even more troublesome to locate. They might be in an analagous directory (replace 'bin' and 'lib' in any of the preceding examples with 'include'), but they could also be in a 'src' directory instead of an 'include' directory. Finally, any software package might choose to install its 'bin', 'lib' and 'include' directories under a single application-specific subdirectory. This leads to things like '/usr/local/mysql/lib' or '/usr/kerberos/bin'.

When I need to compile new software, and that software needs to link with existing libraries and header files, it's a huge pain in the ass to track down the location of the installed files and then set up a CFLAGS environment variable with a '-I' option pointing to the directory in question.

***************************

I'd like to avoid this type of mayhem in the D community by establishing a standard location for source code, libraries, and include files.

I don't have much to say about libraries and include files, because I don't have much experience in the C/C++ world, and I think some of you guys will have more interesting stuff to say than me about that, but I do have some opinions about a canonical organization structure for source code. I think that Java got it right: all source code should be kept in a single code heirarchy, using a deep directory structure to provide unique canonical names to packages and modules. All java code written by Sun gets installed into the com > sun directory. Xml stuff written by the apache group gets installed into the org > apache > xerces directory. And so on and so on. Using reversed domain names in the package/module naming convention ensures that my XML api exists in an entirely different namespace as someone else's XML api.

I know that Burton has imposed this type of canonical package organization to the dig sources, with module names like 'net.BurtonRadons.dig.main' and I think we need to move to an organizational structure like that. In fact, I think the phobos sources should be imported using 'com.digitalmars.phobos.*' just like any other package. There's no reason they should be special enough to be at the top of the package hierarchy.

The trouble with this type of organizational structure is that it relies on everyone having a unique domain name in order to prevent conflicts in canonical names. And not everybody has a domain name. Especially when the primary members of the development community are individual developers rather than software companies.

I have a solution to this problem as well. In a few weeks time, I'll be opening the gates to the first public source code repository for the D language. All source code will be available through public cvs access (well, actally it's svn rather than cvs, but it's the same idea), and I want to organize all of the projects into a single repository that can be imported using 'com.XXXXX' (where 'XXXXX' will be the domain name of the D repository site, which I'm keeping under wraps until after I've finished alpha testing). So, when I host a new project for my XML parsing API, it will be in the 'com.XXXXX.spinnerette' package, since 'spinnerette' is the name of my XML api.

I think the repository site will also be the best place to host a standard library, like phobos. So, even though phobos would (at the moment) be imported using 'com.digitalmars.phobos', I think it should eventually be moved to 'com.XXXXX.phobos', once its development has become a community effort rather than being primarily under Walter's control.

Do these ideas mesh well with everyone else's conception of how source code should be organized into hierarchies of packages? Does anyone strongly disagree here?

We need to hash this out NOW because I need to build the repository structure within the next two weeks, and whatever gets implemented at that point will be difficult to reverse.

I'm not a unix admin so probably the following
might be wrong, hopefully some one will correct it
if necessary.

> Binaries might be installed to
'/bin' - system programs
'sbin' - system programs super user only (root)
/bin and /sbin must be on the boot device

tipicaly /usr is on a separated device that is mounted after the system boots up.

'/usr/bin' - application programs
'/usr/sbin' - application programs super user only (root)

'/usr/local/bin' - applications programs
'/usr/local/bin' - applications programs super user only (root)

from /usr/ to /usr/local the difference might be what you
want to make public on same kind of remote access
(this one I'm just guessing)

lib, includes and src probably follow the same idea.

There, now you know more about unix then I do.

> All of the projects into a single repository that
> can be imported using 'com.XXXXX'

that's why I wanted to know what was the domain a month ago.

I have one simple problem with that,
my project is not commercial, couldn't we have
com.XXXXX
net.XXXXX
org.XXXXX
..

Ant

In article <bjovb9$1f22$1@digitaldaemon.com>, Ant says...
>> All of the projects into a single repository that
>> can be imported using 'com.XXXXX'
>
>that's why I wanted to know what was the domain a month ago.

I would love to tell everyone the domain name right now, but I know that people would show up to the site and start messing with stuff. I can't allow that until I've finished with alpha testing. You'll know soon enough.

>I have one simple problem with that,
>my project is not commercial, couldn't we have
>com.XXXXX
>net.XXXXX
>org.XXXXX

I don't really care whether stuff is commercial or not. The D repository site is certainly not a for-profit or commercial venture. And yet I'm using a ".com" tld rather than a ".org". Why? I just like it better, that's why.

Any packages that are hosted strictly using the repository website can use the package naming convention of 'com.XXXXX.xyz' so that they're gaurunteed a canonical place to live. However, anyone else who hosts their own code (or anyone with a domain name, whether they host their code there or not) can use whatever TLD their domain falls under. So we'll have things like:

com.digitalmars.phobos
org.opend.dli
net.somebodysdsite.wxd
co.uk.xyz.zlib

and so on and so on. My point is that all of these packages should live under a single parent directory. So, when you checkout a new package from somebody's cvs repository, you can put it into:

C:\projects\development\d\src_root\org\opend\dli\

Somewhere in your DMD settings, you'd set up your source directory as "C:\projects\development\d\src_root", and you'll know that your D sources will always fall somewhere beneath this directory.

--Benji

Just a few comment on various topics....

Benji Smith wrote:
> 
> I have recently finished compiling the zillion or so different software packages that will be needed to host the new D source-code repository site. Along the way, I experienced significant enough frustration that I'm proposing that we change the world. I'll tell you about my frustrating experience first. Then I'll tell you about what I think we should do about it.

I'm very grateful that you do this.

> Anyhow, I've been installing lots of software that will be necessary to get the repository site up and running. I've had to compile pretty much everything from source. And I've got some big gripes:
> 
> 1) You can extract your source archives pretty much anywhere in the directory tree, as long as you run 'make' and 'make install' from within these directories. This can lead to source directories all over the place. Some of the software packages that came preinstalled on my server left their source directories in some pretty weird places.

You can, but you needn't. Typically - I think - a single place like e.g.
   /usr/src/packages/SOURCES
is used.

> 2) There is no standard unix location for the installation of binaries, library files, and include (header) files. Binaries might be installed to '/bin' or 'sbin' or '/usr/bin' or '/usr/sbin' or '/usr/local/bin'. Libraries might be installed into analagous directories (just substitute 'lib' for 'bin' in any of these examples) or they might be installed into '/var/lib'. Header files are even more troublesome to locate. They might be in an analagous directory (replace 'bin' and 'lib' in any of the preceding examples with 'include'), but they could also be in a 'src' directory instead of an 'include' directory. Finally, any software package might choose to install its 'bin', 'lib' and 'include' directories under a single application-specific subdirectory. This leads to things like '/usr/local/mysql/lib' or '/usr/kerberos/bin'.
> 
> When I need to compile new software, and that software needs to link with existing libraries and header files, it's a huge pain in the ass to track down the location of the installed files and then set up a CFLAGS environment variable with a '-I' option pointing to the directory in question.

It might be easier to have a single -I option and to put a symbolic link at that place if you choose to have the directory tree elsewhere.

For me /usr/bin feels best.

> ***************************
> 
> I'd like to avoid this type of mayhem in the D community by establishing a standard location for source code, libraries, and include files.

I agree.
And it should be done the sooner the better for it will break any existing code.
Even the small semantic change (in 0.70?) of 'c' was/is a PITA.

> I don't have much to say about libraries and include files, because I don't have much experience in the C/C++ world, and I think some of you guys will have more interesting stuff to say than me about that, but I do have some opinions about a canonical organization structure for source code. I think that Java got it right: all source code should be kept in a single code heirarchy, using a deep directory structure to provide unique canonical names to packages and modules. All java code written by Sun gets installed into the com > sun directory. Xml stuff written by the apache group gets installed into the org > apache > xerces directory. And so on and so on. Using reversed domain names in the package/module naming convention ensures that my XML api exists in an entirely different namespace as someone else's XML api.

It's possible to go this Java path, it's simple and proven. It may be a bit of an overkill enforcing long module names and heavy use of aliases.

But Burton seems to have allready walked this way, so it seems fruitless to
discuss this any more.

> I know that Burton has imposed this type of canonical package organization to the dig sources, with module names like 'net.BurtonRadons.dig.main' and I think we need to move to an organizational structure like that. In fact, I think the phobos sources should be imported using 'com.digitalmars.phobos.*' just like any other package. There's no reason they should be special enough to be at the top of the package hierarchy.

We could use an organized namespace of invented top level domains and
replace
  com.digitalmars.phobos.
by just
  d.

> The trouble with this type of organizational structure is that it relies on everyone having a unique domain name in order to prevent conflicts in canonical names. And not everybody has a domain name. Especially when the primary members of the development community are individual developers rather than software companies.

Individual programmers could have
  name.TomJones
without even owing such a domain.

> I have a solution to this problem as well. In a few weeks time, I'll be opening the gates to the first public source code repository for the D language. All source code will be available through public cvs access (well, actally it's svn rather than cvs, but it's the same idea), and I want to organize all of the projects into a single repository that can be imported using 'com.XXXXX' (where 'XXXXX' will be the domain name of the D repository site, which I'm keeping under wraps until after I've finished alpha testing). So, when I host a new project for my XML parsing API, it will be in the 'com.XXXXX.spinnerette' package, since 'spinnerette' is the name of my XML api.
> 
> I think the repository site will also be the best place to host a standard library, like phobos. So, even though phobos would (at the moment) be imported using 'com.digitalmars.phobos', I think it should eventually be moved to 'com.XXXXX.phobos', once its development has become a community effort rather than being primarily under Walter's control.
> 
> Do these ideas mesh well with everyone else's conception of how source code should be organized into hierarchies of packages? Does anyone strongly disagree here?

As I said, I would prefer a
  d.   instead of com.XXXXX.phobos or com.digitalmars.phobos
as it wouldn't collide with any existing or future domain.

> We need to hash this out NOW because I need to build the repository structure within the next two weeks, and whatever gets implemented at that point will be difficult to reverse.

I agree. Thank you for your work.

-- 
Helmut Leitner    leitner@hls.via.at
Graz, Austria   www.hls-software.com

> ***************************
>
> I'd like to avoid this type of mayhem in the D community by establishing a standard location for source code, libraries, and include files.
>
> I don't have much to say about libraries and include files, because I
don't have
> much experience in the C/C++ world, and I think some of you guys will have
more
> interesting stuff to say than me about that, but I do have some opinions
about a
> canonical organization structure for source code. I think that Java got it right: all source code should be kept in a single code heirarchy, using a
deep
> directory structure to provide unique canonical names to packages and
modules.
> All java code written by Sun gets installed into the com > sun directory.
Xml
> stuff written by the apache group gets installed into the org > apache >
xerces
> directory. And so on and so on. Using reversed domain names in the package/module naming convention ensures that my XML api exists in an
entirely
> different namespace as someone else's XML api.

I recall that when I suggested this I was shouted down (boo hoo). However, I
(still) think it's the way to go, so you can count me in.

> I know that Burton has imposed this type of canonical package organization
to
> the dig sources, with module names like 'net.BurtonRadons.dig.main' and I
think
> we need to move to an organizational structure like that. In fact, I think
the
> phobos sources should be imported using 'com.digitalmars.phobos.*' just
like any
> other package. There's no reason they should be special enough to be at
the top
> of the package hierarchy.

All my stuff is currently synsoft.*, but I think I'll probably move it to org.synsoft.* if we see a groundswell of support.

> The trouble with this type of organizational structure is that it relies
on
> everyone having a unique domain name in order to prevent conflicts in
canonical
> names. And not everybody has a domain name. Especially when the primary
members
> of the development community are individual developers rather than
software
> companies.
>
> I have a solution to this problem as well. In a few weeks time, I'll be
opening
> the gates to the first public source code repository for the D language.
All
> source code will be available through public cvs access (well, actally
it's svn
> rather than cvs, but it's the same idea), and I want to organize all of
the
> projects into a single repository that can be imported using 'com.XXXXX'
(where
> 'XXXXX' will be the domain name of the D repository site, which I'm
keeping
> under wraps until after I've finished alpha testing). So, when I host a
new
> project for my XML parsing API, it will be in the 'com.XXXXX.spinnerette' package, since 'spinnerette' is the name of my XML api.

I like the fact that you're willing to supply them, but ...

> I think the repository site will also be the best place to host a standard library, like phobos. So, even though phobos would (at the moment) be
imported
> using 'com.digitalmars.phobos', I think it should eventually be moved to 'com.XXXXX.phobos', once its development has become a community effort
rather
> than being primarily under Walter's control.

... I strongly disagree with this. If we like

  org.blah.lib-main.lib-individual.*
  com.digitalmars.phobos.

isn't putting everything into

  com.XXXXX.*

utterly pointless, since everything will just have com.XXXXX in front of it.

The second objection is people will simply not take to being forced/coerced/obliged to go into "the one true way".

The third reason is that I can't see Walter going with it in a month of Sundays.


For my part I think it's useful and important to:

 (i) Have a consistent, coherent, predictable and inclusive structure.
 (ii) Have a standard library (owned by Digital Mars until the language gets
all ISO'd) that has a distinguisable identity.

The com.domain.lib.sub-lib.* format is very good for this, it's just not the whole answer. I think that Phobos should be under

  D

as in

  D.rtl.stdio
  D.lang.except

Note that this distingushes between language and standard rtl features.

(I personally don't want Phobos as part of the package names, though I do
want to keep it as the (un)official name of the rtl distribution, but I'm
not sure it's up to me. ;)

> Do these ideas mesh well with everyone else's conception of how source
code
> should be organized into hierarchies of packages? Does anyone strongly
disagree
> here?

Yes and no

> We need to hash this out NOW because I need to build the repository
structure
> within the next two weeks, and whatever gets implemented at that point
will be
> difficult to reverse.

> > I'd like to avoid this type of mayhem in the D community by establishing
a
> > standard location for source code, libraries, and include files.
>
> I agree.
> And it should be done the sooner the better for it will break any existing
code.
> Even the small semantic change (in 0.70?) of 'c' was/is a PITA.

Hear, hear. I am itching to get the win32 registry library written, but this is one of the things that's tempering my enthusiasm (along with those static properties, Walter).

> > I don't have much to say about libraries and include files, because I
don't have
> > much experience in the C/C++ world, and I think some of you guys will
have more
> > interesting stuff to say than me about that, but I do have some opinions
about a
> > canonical organization structure for source code. I think that Java got
it
> > right: all source code should be kept in a single code heirarchy, using
a deep
> > directory structure to provide unique canonical names to packages and
modules.
> > All java code written by Sun gets installed into the com > sun
directory. Xml
> > stuff written by the apache group gets installed into the org > apache >
xerces
> > directory. And so on and so on. Using reversed domain names in the package/module naming convention ensures that my XML api exists in an
entirely
> > different namespace as someone else's XML api.
>
> It's possible to go this Java path, it's simple and proven. It may be a
bit of
> an overkill enforcing long module names and heavy use of aliases.

It is, but it's unambiguous.

> But Burton seems to have allready walked this way, so it seems fruitless
to
> discuss this any more.
>
> > I know that Burton has imposed this type of canonical package
organization to
> > the dig sources, with module names like 'net.BurtonRadons.dig.main' and
I think
> > we need to move to an organizational structure like that. In fact, I
think the
> > phobos sources should be imported using 'com.digitalmars.phobos.*' just
like any
> > other package. There's no reason they should be special enough to be at
the top
> > of the package hierarchy.
>
> We could use an organized namespace of invented top level domains and
> replace
>   com.digitalmars.phobos.
> by just
>   d.

Yes, I think the standard library should not be at com.digitalmars.phobos. Presumably we all want this language to succeed and expand. At such time, would an implementor, e.g. Microsoft, be keen to be writing their modules within "com.digitalmars.*"? I think not.

> > The trouble with this type of organizational structure is that it relies
on
> > everyone having a unique domain name in order to prevent conflicts in
canonical
> > names. And not everybody has a domain name. Especially when the primary
members
> > of the development community are individual developers rather than
software
> > companies.
>
> Individual programmers could have
>   name.TomJones
> without even owing such a domain.

Good idea. Would Benji's organisation arbitrate on these, i.e. when we get multiple people with the same name, or do we assume that there'll be a D Libraries Group together by then?

> > I think the repository site will also be the best place to host a
standard
> > library, like phobos. So, even though phobos would (at the moment) be
imported
> > using 'com.digitalmars.phobos', I think it should eventually be moved to 'com.XXXXX.phobos', once its development has become a community effort
rather
> > than being primarily under Walter's control.
> >
> > Do these ideas mesh well with everyone else's conception of how source
code
> > should be organized into hierarchies of packages? Does anyone strongly
disagree
> > here?
>
> As I said, I would prefer a
>   d.   instead of com.XXXXX.phobos or com.digitalmars.phobos
> as it wouldn't collide with any existing or future domain.
>
> > We need to hash this out NOW because I need to build the repository
structure
> > within the next two weeks, and whatever gets implemented at that point
will be
> > difficult to reverse.
>
> I agree. Thank you for your work.

Yes, and yes. Looking forward to seeing it. :)

Matthew Wilson wrote:
> 
> > > I'd like to avoid this type of mayhem in the D community by establishing
> a
> > > standard location for source code, libraries, and include files.
> >
> > I agree.
> > And it should be done the sooner the better for it will break any existing
> code.
> > Even the small semantic change (in 0.70?) of 'c' was/is a PITA.
> 
> Hear, hear. I am itching to get the win32 registry library written, but this is one of the things that's tempering my enthusiasm (along with those static properties, Walter).

Don't want to be misunderstood. I'm very happy with the semantical change of 'c'.

But it broke almost any code and library out there. There are some, that are not actively maintained. People downloading and testing such code (perhaps even in a year) without knowing the internals, might not become happy.

-- 
Helmut Leitner    leitner@hls.via.at
Graz, Austria   www.hls-software.com

"Benji Smith" <dlanguage@xxagg.com> ha scritto nel messaggio news:bjosk1$1aut$1@digitaldaemon.com...
> I have recently finished compiling the zillion or so different software
packages
> that will be needed to host the new D source-code repository site.

Oh, BTW... I had missed your first posts about the repository site. Although late, please accept my thanks and congratulations.

> I don't have much to say about libraries and include files, because I
don't have
> much experience in the C/C++ world, and I think some of you guys will have
more
> interesting stuff to say than me about that, but I do have some opinions
about a
> canonical organization structure for source code. I think that Java got it right: all source code should be kept in a single code heirarchy, using a
deep
> directory structure to provide unique canonical names to packages and
modules.

If you're in a hurry, skip next paragraph: it's just to say that I agree. :)

I'm not a Java programmer, but I've seen some Java code and sure, it has the
best code organization I've ever seen. And if setting up for compilation
under Linux is a pain because of too many "standard" paths, how about DOS
and Windows, that have none?! Every time I have to set up a new development
machine, or make some modifications to old code and thus install those four
or five libraries I haven't been using for a couple of years, I get asked
why a modification so trivial took me a whole week... but it usually took 4
days just to have it recompile as-is!
And don't even mention "smart" component-based IDEs! Yes, for the first time
ever, this ng is seeing me complaining about Delphi! :)

> The trouble with this type of organizational structure is that it relies
on
> everyone having a unique domain name in order to prevent conflicts in
canonical
> names. And not everybody has a domain name. Especially when the primary
members
> of the development community are individual developers rather than
software
> companies.

I can't believe that! What sense does it make to constrain the source tree structure to actual Internet registered domains? It may be right for Java, so the VM may download at runtime a class used by an applet and know exactly where to save it for later retrieval, but maybe it could be not so strict a requirement for D.

> I have a solution to this problem as well. [...] So, when I host a new project for my XML parsing API, it will be in the 'com.XXXXX.spinnerette' package, since 'spinnerette' is the name of my XML api.

I'm with Ant here: org.XXXXX would be better. But I'm not going to start a holy war on that: there are too many more important things that are to be dealt with, one of the most important ones being having the repository site up and running, so it could even be biz.XXXXX for what it's worth.

Thanks again for your precious work
Ric

"Matthew Wilson" <matthew@stlsoft.org> ha scritto nel messaggio news:bjp6oc$1pie$1@digitaldaemon.com...
> [...] If we like
>
>   org.blah.lib-main.lib-individual.*
>   com.digitalmars.phobos.
>
> isn't putting everything into
>
>   com.XXXXX.*
>
> utterly pointless, since everything will just have com.XXXXX in front of
it.

That could in fact make less people willing to host their projects on the repository site.There are companies which release open source code; I don't think they'd go with anything else than com.companyname for their libraries. They spend money on it, leave them the glory at least. :)

> The third reason is that I can't see Walter going with it in a month of Sundays.

The worst thing about Walter, IMHO, is that he doesn't have a copy constructor so we can clone him. :-)

> The com.domain.lib.sub-lib.* format is very good for this, it's just not
the
> whole answer. I think that Phobos should be under
>
>   D
>
> as in
>
>   D.rtl.stdio
>   D.lang.except
>
> Note that this distingushes between language and standard rtl features.

Agreed completely. Let Phobos remain as a codename, but the D RTL is part of D, just as Windows is not Whistler, OpenDOS is OpenDOS and not any of its fancy release codenames, etc.

Ric

I don't think i like lengthy names, including arbitrary domains. I would propose that:

 - There is only one repository site.
 - Project names *are* namespace names. No "com.blahblah" in front;
 - To avoid collisions, "d" and "c" should be pre-registered, and all commercial companies developing their code, should register their projects or namespaces on the repository site. They could use webspace as redirection, as well as file download facility, but they need not use svn.
 - "d" stands for standard libraries - currently Phobos.

I can't recall any name collision when using Delphi, even though it uses next to no namespace protection.

-eye


PS. Ricardo: you do not need to register components with the IDE just to make Delphi compile some project which uses them. :) Just show it the path and it shall figure out. :) Much less pain than i'm having with C++, which relies on the actual file positions, almost manual linking, and so on...