http://d.puremagic.com/issues/show_bug.cgi?id=11330

           Summary: Directory named as imported module should not stop
                    module search
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: regression
          Priority: P2
         Component: DMD
        AssignedTo: nobody@puremagic.com
        ReportedBy: thecybershadow@gmail.com


--- Comment #0 from Vladimir Panteleev <thecybershadow@gmail.com> 2013-10-23 17:47:53 EEST ---
This is a regression that appears to be caused by implementation of DIP37.

Directory structure:
├───b
│       (empty directory)
└───src
        a.d - contains "import b;"
        b.d - empty file

Command line:
dmd -c -Isrc src/a

DMD 2.063.2 succeeds.

DMD from git complains:

src\a.d(1): Error: importing package 'b' requires a 'package.d' file which cannot be found in 'b\package.d'

DMD shouldn't stop searching for a b.d or b\package.d just because it finds a "b" directory first.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330


Kenji Hara <k.hara.pg@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID


--- Comment #1 from Kenji Hara <k.hara.pg@gmail.com> 2013-10-23 08:30:30 PDT ---
This is an invalid issue, as same as bug 11241 and bug 11243.

In your project, the module/package name 'b' should be resolved to the first-found file system entry "./b/". The new package module feature looks for "./b/package.d", then compiler correctly fails to search the file.

In other words: if a fully-qualified module name 'xx.yy.zz' would match to multiple file/directories through the whole import paths, it is essentially not good.

See also: http://d.puremagic.com/issues/show_bug.cgi?id=11241#c1

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330


Vladimir Panteleev <thecybershadow@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|INVALID                     |


--- Comment #2 from Vladimir Panteleev <thecybershadow@gmail.com> 2013-10-23 18:37:39 EEST ---
If the behavior is by design, then the design is broken and needs to be fixed. Reopening.

Compilers should not treat source files, which use an extension specifically chosen for the programming language, the same as directories, which can contain anything at all.

The current behavior is equivalent to the compiler complaining that there is a file called "b.txt", because its base name is the same as a module/package used in the program. It's not the compiler's business what directories there are in the search path, because they can contain arbitrary files completely unrelated to the task of building the program.

The generalized version of my situation is:

Let's say you have a program that manages foos. Your program consists of several entry point modules, such as mangle-foos.d and unmangle-foos.d, and a module which contains code that manages foos, foos.d. If you build the program from its source directory, everything is fine. However, if you go to your project directory, which incidentally contains a directory called "foos" containing the data to be managed, and you run "rdmd foosmangler/mangle-foos.d", the build breaks. In my opinion, this is not acceptable.

My specific use case is a build tool used to configure, build and deploy a software project. The tool contains a module called "source", which manages the project's source code. If I run it (using rdmd) from the project root, which contains a directory called "source", it breaks. I don't think I need to rename my modules or my source directory to work around this design flaw.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330


Dicebot <public@dicebot.lv> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |public@dicebot.lv


--- Comment #3 from Dicebot <public@dicebot.lv> 2013-10-23 08:53:05 PDT ---
It is also important to remember that DIP37 was intended as breakage-free transparent change.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330



--- Comment #4 from Kenji Hara <k.hara.pg@gmail.com> 2013-10-23 10:00:36 PDT ---
Hmm. OK, confirmed your point. But I don't think that the current behavior/design is broken.

Up to 2.063, d and di files has been considered as a source of the compiler. From 2.064, directory name will be added to the compiler's input by implementing DIP37. That's intended change.

And, I'd like to explain why the current behavior is necessary. If you have the following project:

./
 +--src1/
 |  +--a/
 |     +--...
 +--src2/
 |  +--a.d
 +--src3/
    +--b.d  (contains "import a;")

And you use the command line "dmd -Isrc1 -Isrc2 -Isrc3 ..." to compile project.
If compiler ignores the directory "src1/a/" when resolving "import a;", it
would be used to import "src2/a.d".
Then, what happen if you add a new file src1/a/package.d by using DIP37
feature? It will break existing the code "import a;'in "src3/b.d". The occurred
problem is contrary to the principle in DIP37- adding package.d should be
breakage-free transparent change.

To avoid the problem, if the directory "a" does not contain package.d file, compiler should raise an error on "import a;" from the beginning.

========

On the other hand, in the OP case, I think the root issue is that dmd implicitly adds the current directory in import paths by default. When you use dmd as just a part of build-tool, the implicit addition could cause problems like your case.

An idea based on my thought is: add a new compiler switch "--I." which removes current directory from the import path list.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330



--- Comment #5 from Vladimir Panteleev <thecybershadow@gmail.com> 2013-10-23 20:06:50 EEST ---
(In reply to comment #4)
> It will break existing the code "import a;'in "src3/b.d". The occurred problem is contrary to the principle in DIP37- adding package.d should be breakage-free transparent change.

I understand that the goal is to avoid hijacking when a package.d file is added. If that's the case, then the issue can be resolved by making the compiler complain if it finds both a.d and a/package.d in its search paths.

However, if you think about it more, shouldn't the compiler also complain if it finds more than one match in its search path? Because that is also a hijacking opportunity - remove the first match, and suddenly the compiler "silently" is using some other version of the module. But would making the compiler complain about ambiguous search path matches break too many environments?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330



--- Comment #6 from Kenji Hara <k.hara.pg@gmail.com> 2013-10-23 10:20:26 PDT ---
(In reply to comment #5)
> However, if you think about it more, shouldn't the compiler also complain if it finds more than one match in its search path? Because that is also a hijacking opportunity - remove the first match, and suddenly the compiler "silently" is using some other version of the module. But would making the compiler complain about ambiguous search path matches break too many environments?

Essentially yes. But it would introduce extra cost to search filesystem. If import path list is huge - I think it's a possible case if you manage thousands of third party projects by using package system - I cannot say the extra cost is enough small. I'm not sure it is a compiler's work.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330


Walter Bright <bugzilla@digitalmars.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
                 CC|                            |bugzilla@digitalmars.com
         Resolution|                            |WONTFIX


--- Comment #7 from Walter Bright <bugzilla@digitalmars.com> 2013-10-26 14:02:10 PDT ---
1. The compiler has always relied on a "first found" algorithm when searching the import paths. That's like using nested scopes when searching for a symbol. This is a valuable feature as it allows for overriding (and I use this, especially when debugging). There's a clear hierarchy of this. (The anti-hijacking support in D comes into play for cases where there is no clear hierarchy.)

2. The import system is designed to map onto the file system. Using file/directory names that match module/package names is how it is supposed to work, if there are matching names that have nothing to do with D will cause problems.

I don't see any reasonable resolution for the regression issue you identified. The only thing I can think of is don't name things with the same names as D modules/packages when they are in the import path.

I'm going to resolve as wontfix for now. If a workable solution appears, please reopen.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330



--- Comment #8 from Vladimir Panteleev <thecybershadow@gmail.com> 2013-10-27 00:04:50 EEST ---
> If a workable solution appears, please reopen.

What conditions must be satisfied for a solution to be deemed as "workable"?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------

http://d.puremagic.com/issues/show_bug.cgi?id=11330



--- Comment #9 from Vladimir Panteleev <thecybershadow@gmail.com> 2013-10-27 00:37:00 EEST ---
(In reply to comment #7)
> 1. The compiler has always relied on a "first found" algorithm when searching the import paths. That's like using nested scopes when searching for a symbol. This is a valuable feature as it allows for overriding (and I use this, especially when debugging). There's a clear hierarchy of this. (The anti-hijacking support in D comes into play for cases where there is no clear hierarchy.)

I don't see how a list of search paths is like a number of nested scopes. One is a flat list, the other has clear hierarchy.

This is besides my argument, anyway, so I understand you're replying about the suggestion for module hijack protection? This quote:

> I understand that the goal is to avoid hijacking when a package.d file is added.

If that's not the case, then it's even more unclear to me why this change was introduced, and why it is considered beneficial.

> 2. The import system is designed to map onto the file system. Using file/directory names that match module/package names is how it is supposed to work, if there are matching names that have nothing to do with D will cause problems.

This does not address my arguments in comment #2.

Also, without context, your quote can be used to defend a hypothetical change in DMD to reject "modulename.txt" files, if they exist in the search path and "modulename" is imported somewhere, which is absurd. Which is the main point of my argument: D compilers should not treat filesystem directories that match imported D modules as if they are made for use for D. ".d" files are for D compilers, directories are for everyone.

As far as I know, no other implementation of a package system in any other language behaves as DMD behaves now.

> I'm going to resolve as wontfix for now. If a workable solution appears, please reopen.

I hope this isn't an attempt to sweep a regression under the rug to get 2.064
out quicker:
- I have stumbled upon this problem twice, in two different situations.
- This change affected two other users (issue 11241 and issue 11243).

I would appreciate a prompt formulation of the problem that this change was supposed to solve, and the constraints that solutions must meet, so we can work on a solution that satisfies all use cases and have it implemented before 2.064's release. Right now I don't have much to work with.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------