I recently came across a design situation where the concept of 'hidden' modules would prove useful, particularly since we can't have package and  modules with the same at the same level. Consider this package structure:

mypackage.platform;
mypackage.platformimpl.win32;
mypackage.platformimpl.linux;
mypackage.platformimpl.sdl;
mypackage.platformimpl.glfw;

Platform is setup to import the proper implementation based upon a version conditional. The problem is that all of the modules in the platofrmimpl package are visible to the client and can be imported freely. One might consider this as a solution:

mypackage.platform.platform;
mypackage.platform.win32;
mypackage.platform.linux;
...

Then the implementation modules could each be wrapped with package protection so that they are only visible to the platform.platform module. This works well if you are dealing with predefined interfaces. But in this case, there's no sense in using an interface - an app will only be compiled to use one platform at a time. From a design standpoint, it makes more sense in this case that each module just implement the necessary interface without an actual interface definition - perhaps as a struct with static methods. Then, the platform module just does this:

version(Win32App)
   import mypackage.platformimpl.win32;
else version(LinuxApp)
   import mypackage.platformimpl.linux;

Package protection doesn't work in this case because the contents of each platform file need to be exposed to the client, sense there is no abstract interface defined. Without some mechanism to allow you to hide a particular module from the client, there is no way around the need to use an interface + package protection if you wish to hide the imlementation.

So why not a new level of module protection that hides a given module from the client - meaning, it can't be imported at all outside of a particular part of the package tree. Perhaps something along these lines:

// module declaration using new 'hidden' keyword
hidden module mypackage.platformimpl.win32;

This would make the win32 module visible only to modules that meet the following criteria:

* any module that is in a subpackage of platformimpl can import it. ex: mypackage.platformimpl.anotherpackage.somemodule

* any module that is in the same package as the hidden module can import it. ex: mypackage.platformimpl.somemodule

* any module that is exactly one level above the hidden module's package can import it. ex: mypackage.somemodule

Regarding the third point, it might be useful to allow any module above the hidden module up to the root package to import it. For example:

foo.bar.baz.mymodule (hidden)
can be imported by foo.bar.module and foo.module

However, I think it better if it would be restricted only to one level above the hidden module's package. This allows you to provide an 'adapter module' which imports the hidden modules, and any module that needs to use the hidden module's functionality must go through the adapter. This allows better encapsulation and avoids issues that might arise from importing the hidden modules directly in the root package.

Just something I thought might prove useful in providing a little more design flexibility over what we have already (and adds more weight to the need for a module statement, which some people have questioned recently).

That didn't come out as clearly as it did in my head. If you're confused, I'll try to clarify it :)

On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71@yahoo.com> wrote:
> That didn't come out as clearly as it did in my head. If you're confused, I'll try to clarify it :)

I may be confused. What is wrong with:

[mypackage.platform.d]
module mypackage.platform
version(Win32) import mypackage.platformimpl.win32;
version(linux) import mypackage.platformimpl.linux;
version(sdl) import mypackage.platformimpl.sdl;
version(glfw) mypackage.platformimpl.glfw;

[mypackage.platformimpl.win32.d]
module mypackage.platformimpl.win32
version(Win32):
..etc..

and so on for each module.

This prevents an app compiled on "linux" from importing "mypackage.platformimpl.win32".

Or is the real problem with hiding parts of the implementation? Does "mypackage.platform" need access to parts of "mypackage.platformimpl.win32"? are you trying to prevent users from having access to those same parts?

Perhaps a small example will help, then I can experiment too.

Regan

Regan Heath wrote:
> On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71@yahoo.com>  wrote:
> 
>> That didn't come out as clearly as it did in my head. If you're  confused, I'll try to clarify it :)
> 
> 
> I may be confused. What is wrong with:
> 
> [mypackage.platform.d]
> module mypackage.platform
> version(Win32) import mypackage.platformimpl.win32;
> version(linux) import mypackage.platformimpl.linux;
> version(sdl) import mypackage.platformimpl.sdl;
> version(glfw) mypackage.platformimpl.glfw;
> 
> [mypackage.platformimpl.win32.d]
> module mypackage.platformimpl.win32
> version(Win32):
> ..etc..
> 
> and so on for each module.
> 
> This prevents an app compiled on "linux" from importing  "mypackage.platformimpl.win32".

No it doesn't. An app compiled on linux could still directly import mypackage.platformimpl.win32 directly, bypassing mypackage.platform altogether.

> 
> Or is the real problem with hiding parts of the implementation? 

That's it exactly. The platform stuff was probably not the best example, but it's what I was working on when the idea hit me. To break it down simply:

mypackage.foo.bar.module1
mypackage.foo.bar.module2
mypackage.foo.bar.module3
mypackage.foo.moduleselector

The goal is to prevent any module outside of mypackage.foo (including anything in mypackage) from importing module1, module2, and module3 directly, forcing them to access that functionality through moduleselector.

Obviously, you could wrap module1, module2, and module3 all in package protection statements and do this:

mypackage.foo.bar.moduleselector

With moduleselector in the same package, it is allowed access to declarations in all 3 modules. Outside modules will be forced to go through moduleselector since package protection hides everything. Really this solution is fine. But there's a minor drawback.

All modules that moduleselector manages must be in the same package. From a design perspective, this prevents logical grouping of modules when multiple modules are involved. Consider this:

mypackage.foo.bar.module1a
mypackage.foo.bar.module1b
mypackage.foo.bar.module1c
mypackage.foo.bar.module1

If modules 2 & 3 have multiple modules that go together, now you have a big jumbled mess. With the hidden module concept as I proposed it, you could do this:

mypackage.foo.implone.module1a
mypackage.foo.implone.module1b
mypackage.foo.implone.module1c
mypackage.foo.implone.module1
mypackage.foo.impltwo.module2a
...
mypackage.foo.moduleselector

So the benefits from such a mechanism are design flexibility, and the ability to prevent some modules from being imported outside of a particular package tree. This can be very useful when designing library.

It's not something that we can't live without, but I do think it's something that could prove useful.

On Mon, 11 Jul 2005 10:40:10 +0900, Mike Parker <aldacron71@yahoo.com> wrote:
> Regan Heath wrote:
>> On Sun, 10 Jul 2005 16:38:32 +0900, Mike Parker <aldacron71@yahoo.com>  wrote:
>>
>>> That didn't come out as clearly as it did in my head. If you're  confused, I'll try to clarify it :)
>>   I may be confused. What is wrong with:
>>  [mypackage.platform.d]
>> module mypackage.platform
>> version(Win32) import mypackage.platformimpl.win32;
>> version(linux) import mypackage.platformimpl.linux;
>> version(sdl) import mypackage.platformimpl.sdl;
>> version(glfw) mypackage.platformimpl.glfw;
>>  [mypackage.platformimpl.win32.d]
>> module mypackage.platformimpl.win32
>> version(Win32):
>> ..etc..
>>  and so on for each module.
>>  This prevents an app compiled on "linux" from importing  "mypackage.platformimpl.win32".
>
> No it doesn't. An app compiled on linux could still directly import mypackage.platformimpl.win32 directly, bypassing mypackage.platform altogether.

True, I wasn't meant to say that exactly. What I meant was, they can do it but it causes no problems. No compile errors. No access to things that are supposed to be hidden/protected (assuming "package" is used).

>>  Or is the real problem with hiding parts of the implementation?
>
> That's it exactly. The platform stuff was probably not the best example, but it's what I was working on when the idea hit me. To break it down simply:
>
> mypackage.foo.bar.module1
> mypackage.foo.bar.module2
> mypackage.foo.bar.module3
> mypackage.foo.moduleselector
>
> The goal is to prevent any module outside of mypackage.foo (including anything in mypackage) from importing module1, module2, and module3 directly, forcing them to access that functionality through moduleselector.
>
> Obviously, you could wrap module1, module2, and module3 all in package protection statements and do this:
>
> mypackage.foo.bar.moduleselector
>
> With moduleselector in the same package, it is allowed access to declarations in all 3 modules. Outside modules will be forced to go through moduleselector since package protection hides everything. Really this solution is fine. But there's a minor drawback.

This is the solution I was suggesting/hinting at.

> All modules that moduleselector manages must be in the same package.  From a design perspective, this prevents logical grouping of modules when multiple modules are involved. Consider this:
>
> mypackage.foo.bar.module1a
> mypackage.foo.bar.module1b
> mypackage.foo.bar.module1c
> mypackage.foo.bar.module1
>
> If modules 2 & 3 have multiple modules that go together, now you have a big jumbled mess.

I must be dense today.. why is it a jumbled mess?

[mypackage.foo.bar.module1.d]
module mypackage.foo.bar.module1
version(1A) import mypackage.foo.bar.module1a

[mypackage.foo.bar.module1a.d]
version(1A): package:
..etc..

[mypackage.foo.bar.module2.d]
module mypackage.foo.bar.module1
version(2A) import mypackage.foo.bar.module1a

[mypackage.foo.bar.module2a.d]
version(2A): package:
..etc..

[program.d]
import mypackage.foo.bar.module1;
import mypackage.foo.bar.module2;

Seems fine to me.

> With the hidden module concept as I proposed it, you could do this:
>
> mypackage.foo.implone.module1a
> mypackage.foo.implone.module1b
> mypackage.foo.implone.module1c
> mypackage.foo.implone.module1
> mypackage.foo.impltwo.module2a
> ...
> mypackage.foo.moduleselector
>
> So the benefits from such a mechanism are design flexibility, and the ability to prevent some modules from being imported outside of a particular package tree. This can be very useful when designing library.
>
> It's not something that we can't live without, but I do think it's something that could prove useful.

I'm not sure we need to "prevent" people importing things, we just have to have a method of protecting code while also making it possible to divide a module into several logical units (files). "package" is the method which was added to facilitate this.

Perhaps I'm just not getting it? In which case can you give a more concrete example of a specific problem?

Regan

On Sun, 10 Jul 2005 03:35:29 -0400, Mike Parker <aldacron71@yahoo.com> wrote:

> So why not a new level of module protection that hides a given module from the client - meaning, it can't be imported at all outside of a particular part of the package tree.

I did think of something like this before, but the syntax I thought of was:
   package module foo.bar;
meaning the module itself has package access.

Vathix wrote:

> I did think of something like this before, but the syntax I thought of was:
>    package module foo.bar;
> meaning the module itself has package access.

That works too, particularly since we already have the package keyword. But 'package' implies that it is only available in the same package and subpackages. What I'm looking for is giving access one level up as well, but I suppose either would be acceptable.

Regan Heath wrote:

>>
>> mypackage.foo.bar.module1a
>> mypackage.foo.bar.module1b
>> mypackage.foo.bar.module1c
>> mypackage.foo.bar.module1
>>
>> If modules 2 & 3 have multiple modules that go together, now you have a  big jumbled mess.
> 
> 
> I must be dense today.. why is it a jumbled mess?

Because if you have multiple 'groups' of modules (1a/b/c/d, 2/a/b/c/d, 3a/b/c/d) in the same package, there's no logical organization - all of them are jumbled together in the same package (necessary when using package protection). What is much cleaner is this:

mypackage.implone.module1a/b/c/d
mypackage.impltwo.module2a/b/c/d
mypackage.implthree.module3a/b/c/d
mypackage.moduleselector

By allowing the modules in the implone/two/three packages to be hidden to oustsiders (except those below and one level above), you can now logically structure your packages and keeo a clean separation of module groups, rather than having them all bunched up in one package.


> Perhaps I'm just not getting it? In which case can you give a more  concrete example of a specific problem?

It's not a problem, per se. Nothing's really broken here. It's about adding more flexibility to the design through package structure. The package keyword is a big improvement over when we didn't have it, but it is still limited in that it forces everything to be in the same package or lower, and other modules can still access anything not protected by the package keyword when they import a particular module.

On Mon, 11 Jul 2005 14:41:35 +0900, Mike Parker <aldacron71@yahoo.com> wrote:
> Regan Heath wrote:
>
>>>
>>> mypackage.foo.bar.module1a
>>> mypackage.foo.bar.module1b
>>> mypackage.foo.bar.module1c
>>> mypackage.foo.bar.module1
>>>
>>> If modules 2 & 3 have multiple modules that go together, now you have a  big jumbled mess.
>>   I must be dense today.. why is it a jumbled mess?
>
> Because if you have multiple 'groups' of modules (1a/b/c/d, 2/a/b/c/d, 3a/b/c/d) in the same package, there's no logical organization - all of them are jumbled together in the same package (necessary when using package protection).

So, you're saying you cannot do this:

mypackage/implone/a
mypackage/implone/b
mypackage/implone/c

mypackage/impltwo/a
mypackage/impltwo/b
mypackage/impltwo/c

..

because implone and impltwo require access to each others internal data?

This seems odd to me, no offense but it suggests either bad design, or that they should be in the same package or even the same file if they're that tightly bound.

Without a concrete example I can only speculate.

> What is much cleaner is this:
>
> mypackage.implone.module1a/b/c/d
> mypackage.impltwo.module2a/b/c/d
> mypackage.implthree.module3a/b/c/d
> mypackage.moduleselector
>
> By allowing the modules in the implone/two/three packages to be hidden to oustsiders (except those below and one level above), you can now logically structure your packages and keeo a clean separation of module groups, rather than having them all bunched up in one package.

Making them 'hidden' would probably achieve your goal, but, there are likely several solutions and I think I'd prefer one that was more like "package". I mean there is no point prohibiting an import if you can instead ensure it is meaningless (i.e. they cannot gain access to things they should't) thats my opinion anyway.

>> Perhaps I'm just not getting it? In which case can you give a more  concrete example of a specific problem?
>
> It's not a problem, per se. Nothing's really broken here.

I realise that.

> It's about adding more flexibility to the design through package structure. The package keyword is a big improvement over when we didn't have it, but it is still limited in that it forces everything to be in the same package or lower,

I haven't had a problem with this restriction as yet, I think a concrete example i.e. show me exactly what you tried to do when you found this.

> and other modules can still access anything not protected by the package keyword when they import a particular module.

That is because without "package" or another protection attribute it defaults to "public". We should probably get into the habit of putting:

package:

at the top of all our source (which suggests to me that this should be the default, not public)

Regan

Regan Heath wrote:

>> Regan Heath wrote:
>>
>>>>
>>>> mypackage.foo.bar.module1a
>>>> mypackage.foo.bar.module1b
>>>> mypackage.foo.bar.module1c
>>>> mypackage.foo.bar.module1
>>>>
>>>> If modules 2 & 3 have multiple modules that go together, now you have  a  big jumbled mess.
>>>
>>>   I must be dense today.. why is it a jumbled mess?
>>
>>
>> Because if you have multiple 'groups' of modules (1a/b/c/d, 2/a/b/c/d,  3a/b/c/d) in the same package, there's no logical organization - all of  them are jumbled together in the same package (necessary when using  package protection).
> 
> 
> So, you're saying you cannot do this:
> 
> mypackage/implone/a
> mypackage/implone/b
> mypackage/implone/c
> 
> mypackage/impltwo/a
> mypackage/impltwo/b
> mypackage/impltwo/c
> 
> ..
> 
> because implone and impltwo require access to each others internal data?

No, that's not what I'm saying at all. I'm not sure if I'm not explaining myself clearly or not, but you are completely missing my point. Imagine you have a single module that provides an interface to implone and impltwo in mypackage.myinterfacemodule. You only want clients of your package to access implone and impltwo through the interface module, and not allow them to import them directly, thereby hiding the implementation from the client. Right now, there is no mechanism in D that allows this. The workaround is to put the interface module and all of the implementatoins in the same directory, but then you lose logical separation of the implementations.

Imagine a team of programmers - Bob, Joe and Bill - who are working on a D library. One of the subsytems in the library is the foo subsystem, of which there could be several possible implementations. So, they develop a foo interface and each programmer develops his own custom implementation. This is the resulting package structure:

teamlib.foomodule
teamlib.bobfoo.modulesa/b/c/d/e
teamlib.joefoo.modulea/b/c
teamlib.billfoo.modulea/b/c/d

Perhaps the implemetation can be chosen at compile time, perhaps at runtime, it doesn' matter. Currently in D, any part of the implementation that is not package protected can be imported directly and accessed by the library client. But, for whatever reason, the team doesn't want the client to access anything directly - they want would prefer if the client goes through foomodule instead. Because there is currently no way in D to achieve this goal with the given package structure, they'll need to wrap their implementation modules in package protection and do this:

teamlib.foo.foomodule
teamlib.foo.bobfooa/b/c/d/e
teamlib.foo.joefooa/b/c
teamlib.foo.billfooa/b/c/d

So now instead of a logical separation of implementations, you have all 12 modules in the same package. Additionally, the implementaion modules have access to each other (which may not be desirable). The first method is structurally cleaner, easier to maintain (when you are talking about numerous files), and hides implementations from each other.

That's what I'm trying to get across, and I hope I have succeeded this time. Having the ability to prevent modules from being imported outside of a particular scope could come in handy for library designs, I think. I know I can see uses for such a feature. It's something I can live without, but would love to have.