On Sat, 14 Sep 2013 08:32:04 +0200
Rainer Schuetze <r.sagitario@gmx.de> wrote:
> 
> The generated sc.ini in dmd/src is no good, the LIB environment variable should be set like this:
> 
> LIB="%@P%\..\..\phobos"
> 
> Even if you do so, it does not work because optlink is also looking for an sc.ini (contrary or in addition to the documentation that mentions optlink.cfg) and extracts the LIB environment variable. If the link.exe in the PATH is from a different version of dmd, the wrong phobos.lib is taken.
> 
> It is easiest to always put link.exe into the same directory as dmd.exe.

Aha! That was it! I don't know why my experiments doing that didn't work before (I must have messed up something else at the time), but fixing sc.ini and copying OPTLINK to the same dir fixed that problem!

I still have another small issue with the chm generation, and some thorough testing to do, but things are looking REALLY good now!


> >> >- the working directory defaultWorkDir is never created
> > Really? Hmm:
> >
> > - Is it possible that might be a permissions issue? Since this isn't
> >    really an end-user tool, I didn't feel a need to make sure it
> >    worked with a limited user account. What temp directory
> >    is reported when you do "create_dmd_release --help"?
> >
> > - Are you using --use-clone=path? If so, then it won't create
> >    defaultWorkDir unless it needs to - although I might have missed
> >    something there, so I'll check into that.
> 
> Actually I could not reproduce this problem now. Could it be that this happened with a version from a few days ago and was already fixed?
> 

That turned out to be a regression I had introduced with a recent change. I didn't catch it because, for me, the directory was already there. It went away for you because the directory must have gotten created on another run.

It's fixed on GitHub now.


> >> >- I get an error that '.' cannot be removed, but I can continue answering 'n' to the following question for retry
> > Ok now that seems bizarre. Can you post a log or screenshot of that? I have no idea what part of the process might be doing that - definitely something involving an external tool being invoked, because I didn't add any interactive prompts to it.
> >
> > You're not running it*from*  the [your temp dir]/.create_dmd_release directory, are you?
> >
> >
> 
> I using an user account with Admin rights. Here is what I do up to the error:
> 
> C:\tmp\d\installer\create_dmd_release>set
> path=c:\l\dmd2.063\windows\bin;c:\l\dmc\bin;%path%
> C:\tmp\d\installer\create_dmd_release>dmd create_dmd_release.d
> C:\tmp\d\installer\create_dmd_release>md x
> C:\tmp\d\installer\create_dmd_release>cd x
> C:\tmp\d\installer\create_dmd_release\x>md junk
> C:\tmp\d\installer\create_dmd_release\x>..\create_dmd_release master
> --extras=junk --only-32 -v
> Checking: git --help
> Checking: unzip --help
> Checking: zip -v
> Checking: 7za --help
> Checking: dmc -c .create_release_dummy
> Checking: make -f .create_release_dummy
> Running: dmc .create_release_dummy.c -L/?
> > .create_release_optlink_help Checking:
> > type .create_release_optlink_help
> Checking: type .create_release_optlink_help
> Creating dir: C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release
> Entering dir: C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release
> Creating dir: dmd
> Entering dir: dmd
> Cloning: git@github.com:D-Programming-Language/dmd.git
> Running: git clone "git@github.com:D-Programming-Language/dmd.git" .
> Cloning into '.'...
> Permission denied (publickey).
> fatal: Could not read from remote repository.
> 
> Please make sure you have the correct access rights
> and the repository exists.
> Deletion of directory '.' failed. Should I try again? (y/n) n
> Entering dir: C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release
> Couldn't do git protocol, falling back to 'https://'...
> Removing dir: dmd
> Creating dir: dmd
> Entering dir: dmd
> Cloning: https://github.com/D-Programming-Language/dmd.git
> ...

Ok, I see what's happening. It's git that's doing that:

This create_dmd_release tool *tries* to clone using git/ssh protocol because it's supposed to be faster, albeit less compatible. But if that fails (as it does for both you and me...some rights/certificate issue apparently), then the script tries again using the slower, but more reliable HTTPS protocol.

But here's the thing: When git tries to clone, and the clone fails, then git's nice about it and tries to clean up the mess it left behind by deleting the directory with the failed clone. Unfortunately, Windows likes to randomly hold file locks open (or maybe it's a bug in your version of git that's holding an old file handle open). So git's unable to delete the failed-clone directory and asks if you want to try again.

I can't find a cmdline switch to tell "git clone" to always retry (or to never retry), so I don't think there's much create_dmd_release can do about it, short of reading git's output, scanning for that question, and sending a 'y'. But that seems like overkill. It may be better to just let people answer on their own if they get that.

On 20.09.2013 09:29, Nick Sabalausky wrote:
>> fatal: Could not read from remote repository.
>> >
>> >Please make sure you have the correct access rights
>> >and the repository exists.
>> >Deletion of directory '.' failed. Should I try again? (y/n) n
>> >Entering dir: C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release
>> >Couldn't do git protocol, falling back to 'https://'...
>> >Removing dir: dmd
>> >Creating dir: dmd
>> >Entering dir: dmd
>> >Cloning:https://github.com/D-Programming-Language/dmd.git
>> >...
> Ok, I see what's happening. It's git that's doing that:
>
> This create_dmd_release tool*tries*  to clone using git/ssh protocol
> because it's supposed to be faster, albeit less compatible. But if that
> fails (as it does for both you and me...some rights/certificate issue
> apparently), then the script tries again using the slower, but more
> reliable HTTPS protocol.
>
> But here's the thing: When git tries to clone, and the clone fails,
> then git's nice about it and tries to clean up the mess it left behind
> by deleting the directory with the failed clone. Unfortunately, Windows
> likes to randomly hold file locks open (or maybe it's a bug in your
> version of git that's holding an old file handle open). So git's unable
> to delete the failed-clone directory and asks if you want to try again.
>
> I can't find a cmdline switch to tell "git clone" to always retry
> (or to never retry), so I don't think there's much create_dmd_release
> can do about it, short of reading git's output, scanning for that
> question, and sending a 'y'. But that seems like overkill. It may be
> better to just let people answer on their own if they get that.
>

I noticed that it is the git process that has a handle on that directory. It seems to able to delete the directory if you run git from the parent directory and pass "dmd" instead of ".".

On Fri, 20 Sep 2013 21:38:06 +0200
Rainer Schuetze <r.sagitario@gmx.de> wrote:
> On 20.09.2013 09:29, Nick Sabalausky wrote:
> 
> I noticed that it is the git process that has a handle on that directory. It seems to able to delete the directory if you run git from the parent directory and pass "dmd" instead of ".".

Hmm, well it's been working fine on my Win machine as-is, but I'll look into doing that. I think I had some reason for doing it this way, but it probably wasn't anything major.

On Fri, 20 Sep 2013 17:39:22 -0400
Nick Sabalausky <SeeWebsiteToContactMe@semitwist.com> wrote:

> On Fri, 20 Sep 2013 21:38:06 +0200
> Rainer Schuetze <r.sagitario@gmx.de> wrote:
> > 
> > I noticed that it is the git process that has a handle on that directory. It seems to able to delete the directory if you run git from the parent directory and pass "dmd" instead of ".".
> 
> Hmm, well it's been working fine on my Win machine as-is, but I'll look into doing that. I think I had some reason for doing it this way, but it probably wasn't anything major.
> 

Just before I was about to make that fix, I started experiencing the same problem, too.

I've made the change you suggested and it seems to work for me. Can you verify?

On 22.09.2013 04:54, Nick Sabalausky wrote:
> On Fri, 20 Sep 2013 17:39:22 -0400
> Nick Sabalausky <SeeWebsiteToContactMe@semitwist.com> wrote:
>
>> On Fri, 20 Sep 2013 21:38:06 +0200
>> Rainer Schuetze <r.sagitario@gmx.de> wrote:
>>>
>>> I noticed that it is the git process that has a handle on that
>>> directory. It seems to able to delete the directory if you run git
>>> from the parent directory and pass "dmd" instead of ".".
>>
>> Hmm, well it's been working fine on my Win machine as-is, but I'll
>> look into doing that. I think I had some reason for doing it this
>> way, but it probably wasn't anything major.
>>
>
> Just before I was about to make that fix, I started experiencing the
> same problem, too.
>
> I've made the change you suggested and it seems to work for me. Can you
> verify?
>

I retried and had the issue with access to %TEMP%/".create_dmd_release problem problem during startup again. This time it was TortoiseGitCache holding a handle to that directory. Killing that process helped.

Then, the 32-bit build ran flawlessly if I

- add the dmc binary path to the PATH environment variable

- add the path to some existing dmd installation to the path. rdmd is needed to run get_dlibcurl32.bat. [OT: I noticed it uses a VB script to do the download. Shudder ;-) Here is a http-get function without needing curl: https://github.com/D-Programming-Language/visuald/blob/master/stdext/httpget.d , though I don't know how usable it is. ]

I usually don't have these paths added because the folders contain too many executables that try to replace more useful versions of them (lib,link,shell,make). I'd prefer if you could specify the path to dmc as an option, but it's not a big deal as the task of building the release is pretty special anyway.

It would be nice if rdmd is used from the current build, though.

After the successful build I tried the 64-bit build. It stopped here:

Building Druntime 64-bit
Entering dir: C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release\druntime
Running: make MODEL=64 DMD=../dmd/src/dmd -f win64.mak
mkdir import\core\stdc
A subdirectory or file import\core\stdc already exists.
mkdir import\core\sys\freebsd\sys
A subdirectory or file import\core\sys\freebsd\sys already exists.
mkdir import\core\sys\linux\sys
A subdirectory or file import\core\sys\linux\sys already exists.
mkdir import\core\sys\osx\mach
A subdirectory or file import\core\sys\osx\mach already exists.
mkdir import\core\sys\posix\arpa
A subdirectory or file import\core\sys\posix\arpa already exists.
mkdir import\core\sys\posix\net
A subdirectory or file import\core\sys\posix\net already exists.
mkdir import\core\sys\posix\netinet
A subdirectory or file import\core\sys\posix\netinet already exists.
mkdir import\core\sys\posix\sys
A subdirectory or file import\core\sys\posix\sys already exists.
mkdir import\core\sys\windows
A subdirectory or file import\core\sys\windows already exists.
mkdir import\etc\linux
A subdirectory or file import\etc\linux already exists.
"\Program Files (x86)\Microsoft Visual Studio 10.0\VC"\bin\amd64\cl -c /Z7 /I"\Program Files (x86)\Microsoft Visual Studio 10.0\VC"\INCLUDE /I"\Program Files (x86)\Microsoft SDKs\Windows\v7.0A"\Include src\core\stdc\errno.c -Foerrno_c.obj
Error: '\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\cl' not found
Entering dir: C:\tmp\d\installer2\create_dmd_release\x
create_dmd_release: Error: Command failed (ran from dir 'C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release\druntime'): make MODEL=64 DMD=../dmd/src/dmd -f win64.mak

This happens because the VCDIR variable is not passed to the makefile (to work with VS2012+ you'll have to pass CC and AR explicitely as the path to the binaries inside the VS installation has changed).

On Wed, 04 Sep 2013 10:13:48 +0200
Jacob Carlborg <doob@me.com> wrote:

> On 2013-09-04 02:53, Nick Sabalausky wrote:
> 
> > - OSX 10.7: Works fine. (At least it did last time I tried it, a couple weeks ago. It *could* have regressed but I won't have access to the machine again until maybe Sunday.)
> 
> When I unpack the generate zip none of the binaries have executable permission set.
> 

Fixed.

On Sun, 22 Sep 2013 08:59:04 +0200
Rainer Schuetze <r.sagitario@gmx.de> wrote:
> 
> I retried and had the issue with access to %TEMP%/".create_dmd_release problem problem during startup again. This time it was TortoiseGitCache holding a handle to that directory. Killing that process helped.
> 

Yea, Windows is a pain with file/directory locking :/

> [OT: I noticed it uses a VB script
> to do the download. Shudder ;-) Here is a http-get function without
> needing curl:
> https://github.com/D-Programming-Language/visuald/blob/master/stdext/httpget.d
> , though I don't know how usable it is. ]
> 

I admit, the VBScript thing is a bit of a hack. On the upside, the actual connection/dload is handled by a lib that's been in Windows for some time, so hopefully it should be pretty battle-hardened. I'll definitely keep yours in mind in the future, I think that's awesome to have. But in the interest of getting this (and some other things) out the door I think I'll keep it as-is for now.


> I usually don't have these paths added because the folders contain too many executables that try to replace more useful versions of them (lib,link,shell,make). I'd prefer if you could specify the path to dmc as an option, but it's not a big deal as the task of building the release is pretty special anyway.
> 

It's a nice idea, but as you say it's not real critical right now.


> It would be nice if rdmd is used from the current build, though.
> 

Agreed, and now fixed.


> After the successful build I tried the 64-bit build. It stopped here:
> 
> Building Druntime 64-bit
> Entering dir:
> C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release\druntime
> Running: make MODEL=64 DMD=../dmd/src/dmd -f win64.mak
>[...]
> "\Program Files (x86)\Microsoft Visual Studio 10.0\VC"\bin\amd64\cl
> -c /Z7 /I"\Program Files (x86)\Microsoft Visual Studio
> 10.0\VC"\INCLUDE /I"\Program Files (x86)\Microsoft
> SDKs\Windows\v7.0A"\Include src\core\stdc\errno.c -Foerrno_c.obj
> Error: '\Program Files (x86)\Microsoft Visual Studio
> 10.0\VC\bin\amd64\cl' not found
> Entering dir: C:\tmp\d\installer2\create_dmd_release\x
> create_dmd_release: Error: Command failed (ran from dir
> 'C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release\druntime'):
> make MODEL=64 DMD=../dmd/src/dmd -f win64.mak
> 
> This happens because the VCDIR variable is not passed to the makefile (to work with VS2012+ you'll have to pass CC and AR explicitely as the path to the binaries inside the VS installation has changed).

Ugh, I wish MS would just pick a design as stick with it :/... I'll try to take care of that. On the bright side though, this *should* be the final critical issue blocking create_dmd_release from meeting its basic charter and being "pull request"-ready. Everything else seems to be working fine now on Windows and various Posixen in all my tests so far, so fingers crossed...

On 25.09.2013 15:36, Nick Sabalausky wrote:
> On Sun, 22 Sep 2013 08:59:04 +0200
> Rainer Schuetze <r.sagitario@gmx.de> wrote:
>>
>> I retried and had the issue with access to
>> %TEMP%/".create_dmd_release problem problem during startup again.
>> This time it was TortoiseGitCache holding a handle to that directory.
>> Killing that process helped.
>>
>
> Yea, Windows is a pain with file/directory locking :/

Well, actually Windows is way better than POSIX in this regard, because
Windows like a few other systems does implement locking at the kernel level.

Whereas POSIX locking relies in a form of userspace cooperative locking that does not prevent someone to come in between and mess up with the files.

Nothing like deleting a file or write to it, while it is in use by other
processes, without any means to prevent such actions in a portable way.

--
Paulo

On Wed, 25 Sep 2013 16:58:48 +0200
Paulo Pinto <pjmlp@progtools.org> wrote:

> On 25.09.2013 15:36, Nick Sabalausky wrote:
> > On Sun, 22 Sep 2013 08:59:04 +0200
> > Rainer Schuetze <r.sagitario@gmx.de> wrote:
> >>
> >> I retried and had the issue with access to %TEMP%/".create_dmd_release problem problem during startup again. This time it was TortoiseGitCache holding a handle to that directory. Killing that process helped.
> >>
> >
> > Yea, Windows is a pain with file/directory locking :/
> 
> Well, actually Windows is way better than POSIX in this regard, because Windows like a few other systems does implement locking at the kernel level.
> 
> Whereas POSIX locking relies in a form of userspace cooperative locking that does not prevent someone to come in between and mess up with the files.
> 
> Nothing like deleting a file or write to it, while it is in use by other processes, without any means to prevent such actions in a portable way.
> 

Oh right, that's certainly true, which I why I have a bit of a love/hate thing for how Windows does it. It's safer and more "correct", but also frequently a big pain. Stoopid tradeoffs...

On 2013-09-25 15:36, Nick Sabalausky wrote:

> I admit, the VBScript thing is a bit of a hack.

If I recall correctly, you can use JavaScript as an alternative. I would consider that slightly better. It would of course be better if both could be avoided.

-- 
/Jacob Carlborg