msys2 building issue

[Phillip Lord]

I'm struggling with an issue with the snapshot build for windows. I have
a small script that in turn runs admin/nt/dist-build/build-zips.sh. The
build times (based on when the output files are created) make no sense.

x86_64
no-deps zip 17min
Deps-zip 3min
installer 25min

i686
no deps zip 31min
deps zip 47min
installer 61min

So, the installer is really slow to build (probably because of the
compression I guess). But the second i686 step is really slow,
especially the building the zips and installer.

I don't think this is the Emacs build per se; it looks to me like a
memory leak; the build process just gets slower and slower. In fact,
I've had to reduce the parallalism of make or the whole process crashes
with a resource allocation error. But Process Manager reports that the
VM has memory left.

I ask about all this, because I have no idea how to debug this further,
and I thought someone might have an idea.


My launch script is below, and build-zips.sh is in the repo.


#!/bin/bash

set -o errexit

## List the ssh-keys -- this has the crashes the script if there are none
ssh-add -l

pushd master/admin/nt/dist-build

## Update repo
git pull

./build-zips.sh -sn | tee ../../../../build.log

popd

pushd master
## Make the source so we can distribute it, but don't update because
## we've build in a different directory. --no-changelog is also needed
## because otherwise we run make which runs configure
./make-dist --snapshot --no-check --no-changelog
rename .tar _`date +%Y-%m-%d`.tar emacs*.tar.gz
mv emacs-*.tar.gz ~/emacs-upload
popd

[Eli Zaretskii]

Date: Tue, 27 Mar 2018 22:20:34 +0100
x86_64
no-deps zip 17min
Deps-zip 3min
installer 25min
i686
no deps zip 31min
deps zip 47min
installer 61min
So, the installer is really slow to build (probably because of the
compression I guess). But the second i686 step is really slow,
especially the building the zips and installer.

Does this happen only when building both the 64-bit and the 32-bit
zips? What if you build only the 32-bit one -- do you get the same
slow times?

My next suspect is the way MSYS2 lets you build 32-bit binaries. I
know nothing about that, all I see is that you set 2 environment
variables, but could that cause Bash or Make or some other component
to run out of memory due to a bug or a memory leak?

I don't think this is the Emacs build per se; it looks to me like a
memory leak; the build process just gets slower and slower. In fact,
I've had to reduce the parallalism of make or the whole process crashes
with a resource allocation error.

Which program reports the error?

But Process Manager reports that the VM has memory left.

What does Task Manager say about memory consumption of the various
programs involved in the build, like Bash, Make, and Python? Do you
see their memory growing to values that are unreasonably large?

The slow-down could be caused by memory consumption, so it's possible
the actual problem is with memory, not with slow builds.

[Phillip Lord]

Post by Eli Zaretskii

Date: Tue, 27 Mar 2018 22:20:34 +0100
x86_64
no-deps zip 17min
Deps-zip 3min
installer 25min
i686
no deps zip 31min
deps zip 47min
installer 61min
So, the installer is really slow to build (probably because of the
compression I guess). But the second i686 step is really slow,
especially the building the zips and installer.

Does this happen only when building both the 64-bit and the 32-bit
zips? What if you build only the 32-bit one -- do you get the same
slow times?
My next suspect is the way MSYS2 lets you build 32-bit binaries. I
know nothing about that, all I see is that you set 2 environment
variables, but could that cause Bash or Make or some other component
to run out of memory due to a bug or a memory leak?

I've tested this out since. And I am having a hard time making head nor
tail of it. Building on it's own, the i686 will build fast
enough. Building x86_64 sometimes give very inconsistent behaviour
between the two builds (with the second slower). I could, of course,
build the x86 and i686 separately, but I want to launch the build once
and let it finish.

I've had occasions when the build time goes up to many hours.

Post by Eli Zaretskii

I don't think this is the Emacs build per se; it looks to me like a
memory leak; the build process just gets slower and slower. In fact,
I've had to reduce the parallalism of make or the whole process crashes
with a resource allocation error.

Which program reports the error?

But Process Manager reports that the VM has memory left.

What does Task Manager say about memory consumption of the various
programs involved in the build, like Bash, Make, and Python? Do you
see their memory growing to values that are unreasonably large?
The slow-down could be caused by memory consumption, so it's possible
the actual problem is with memory, not with slow builds.

Actually, no Task Manager reports that CPU is 100%, memory is between 50
and 70%.

I'm coming to the conclusion that it's an oddity of the VM
infrastructure. I think it's going to be hard to test out further,
because the slow build makes experiments a bit of a PITA.

Never mind, even if it is slow it always seems to build now that I am
using only dual threaded build. I guess that is ultimately all I care
about.

Phil

[Eli Zaretskii]

Date: Tue, 10 Apr 2018 13:00:20 +0100
I'm coming to the conclusion that it's an oddity of the VM
infrastructure.

What is "VM" in this context?

[Phillip Lord]

Post by Eli Zaretskii

Date: Tue, 10 Apr 2018 13:00:20 +0100
I'm coming to the conclusion that it's an oddity of the VM
infrastructure.

What is "VM" in this context?

"Virtual Machine". I used to build Emacs on one of my own computers, but
then I can't use that machine for anything else while it's building.

Phil

[Eli Zaretskii]

Date: Tue, 10 Apr 2018 19:52:09 +0100

Post by Eli Zaretskii
What is "VM" in this context?

"Virtual Machine". I used to build Emacs on one of my own computers, but
then I can't use that machine for anything else while it's building.

Does that mean you build not natively on Windows, but in VM running on
Unix?

[Phillip Lord]

Date: Tue, 10 Apr 2018 19:52:09 +0100

For Emacs-25, windows was built on a physical (dual boot) machine, but
the window partition started to run out of space. I switched to an
externally hosted VM in November.

Generally, it's been a positive experience, except that the windows VM
is low-powered because I am too tight-fisted to pay for anything
faster. Perhaps this is the core of the issue.

Phil