buildbot.git
8 years agophase1: run maketag.sh under exclusive lock
Jo-Philipp Wich [Wed, 14 Dec 2016 00:26:08 +0000 (01:26 +0100)]
phase1: run maketag.sh under exclusive lock

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: adjust change filter
Jo-Philipp Wich [Fri, 2 Dec 2016 21:44:18 +0000 (22:44 +0100)]
phase2: adjust change filter

Instead of hardcoding the master branch, filter incoming changes by the branch
used for the corresponding feed.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agomakebranch.sh: use -SNAPSHOT instead of -CURRENT for untagged branch builds
Jo-Philipp Wich [Fri, 2 Dec 2016 21:18:47 +0000 (22:18 +0100)]
makebranch.sh: use -SNAPSHOT instead of -CURRENT for untagged branch builds

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: use -SNAPSHOT instead of -CURRENT as build designation
Jo-Philipp Wich [Fri, 2 Dec 2016 21:16:20 +0000 (22:16 +0100)]
phase1: use -SNAPSHOT instead of -CURRENT as build designation

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agomaketag.sh: use the output of getver.sh verbatim
Jo-Philipp Wich [Fri, 2 Dec 2016 14:33:49 +0000 (15:33 +0100)]
maketag.sh: use the output of getver.sh verbatim

Do not append Git commit hashes to the getver.sh reported version but take the
value verbatim. Also change the Git author to "Release Management".

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agomakebranch.sh: use YY.MM-CURRENT designation for branch versions
Jo-Philipp Wich [Fri, 2 Dec 2016 14:32:59 +0000 (15:32 +0100)]
makebranch.sh: use YY.MM-CURRENT designation for branch versions

Also use "Release Management" as Git author.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: use YY.MM-CURRENT as branch directory
Jo-Philipp Wich [Fri, 2 Dec 2016 12:30:19 +0000 (13:30 +0100)]
phase1: use YY.MM-CURRENT as branch directory

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: don't fail if Git is not yet cloned
Jo-Philipp Wich [Wed, 30 Nov 2016 20:17:46 +0000 (21:17 +0100)]
phase1: don't fail if Git is not yet cloned

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: add base version suffix to directory for non-master builds
Jo-Philipp Wich [Mon, 28 Nov 2016 19:06:26 +0000 (20:06 +0100)]
phase2: add base version suffix to directory for non-master builds

Also support persistent operation where previous build artifacts are not
cleared out.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: create version directories
Jo-Philipp Wich [Mon, 28 Nov 2016 19:03:15 +0000 (20:03 +0100)]
phase1: create version directories

For tagged or non-master builds, emit version subdirectories for the binary
artifacts and symlink the shared feed repositories to a common location.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: support creating tags on demand
Jo-Philipp Wich [Mon, 28 Nov 2016 18:36:18 +0000 (19:36 +0100)]
phase1: support creating tags on demand

If a build is forcibly invoked with a custom tag=x.y.z property, the builder
will prepare and push the appropriate Git repository tag.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: add helper scripts for creating branches and tags
Jo-Philipp Wich [Mon, 28 Nov 2016 18:35:40 +0000 (19:35 +0100)]
phase1: add helper scripts for creating branches and tags

These scripts will be used to prepare release branches and repository tags.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: switch away from deprecated slave side Git step
Jo-Philipp Wich [Mon, 28 Nov 2016 18:31:46 +0000 (19:31 +0100)]
phase1: switch away from deprecated slave side Git step

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: improve Git cloning on the master
Jo-Philipp Wich [Mon, 28 Nov 2016 18:28:07 +0000 (19:28 +0100)]
phase1: improve Git cloning on the master

Avoid switching branches later and checkout the proper branch right away.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoRemove unsupported logEnviron from master command calls
Jo-Philipp Wich [Tue, 15 Nov 2016 20:10:48 +0000 (21:10 +0100)]
Remove unsupported logEnviron from master command calls

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoAllow overriding GnuPG home directory
Jo-Philipp Wich [Tue, 15 Nov 2016 20:02:28 +0000 (21:02 +0100)]
Allow overriding GnuPG home directory

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: use a different name for the Git clone work directory
Jo-Philipp Wich [Tue, 15 Nov 2016 13:28:41 +0000 (14:28 +0100)]
phase1: use a different name for the Git clone work directory

Upon first start, the build master expects a clean directory to issue
git init --bare in, therfore we cannot use the clone used for bootstrapping.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: move initial .config into external file
Jo-Philipp Wich [Mon, 14 Nov 2016 17:46:34 +0000 (18:46 +0100)]
phase1: move initial .config into external file

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: remove duplicate branch argument
Jo-Philipp Wich [Mon, 14 Nov 2016 14:38:21 +0000 (15:38 +0100)]
phase2: remove duplicate branch argument

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: honour branch references in feeds.conf
Jo-Philipp Wich [Mon, 14 Nov 2016 14:36:40 +0000 (15:36 +0100)]
phase2: honour branch references in feeds.conf

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: fix typo in previous commit
Jo-Philipp Wich [Mon, 14 Nov 2016 14:21:47 +0000 (15:21 +0100)]
phase1: fix typo in previous commit

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: support specifying repository source branch
Jo-Philipp Wich [Mon, 14 Nov 2016 14:20:37 +0000 (15:20 +0100)]
phase1: support specifying repository source branch

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: also honour branch in Git() step
Jo-Philipp Wich [Mon, 14 Nov 2016 14:17:46 +0000 (15:17 +0100)]
phase1: also honour branch in Git() step

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: avoid creating nested dl/ symlinks
Jo-Philipp Wich [Mon, 14 Nov 2016 14:01:20 +0000 (15:01 +0100)]
phase2: avoid creating nested dl/ symlinks

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: avoid creating nested dl/ symlinks
Jo-Philipp Wich [Mon, 14 Nov 2016 14:00:24 +0000 (15:00 +0100)]
phase1: avoid creating nested dl/ symlinks

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: make sdk_password setting optional
Jo-Philipp Wich [Mon, 14 Nov 2016 13:46:02 +0000 (14:46 +0100)]
phase2: make sdk_password setting optional

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: remove unused cleanup script
Jo-Philipp Wich [Mon, 14 Nov 2016 13:35:04 +0000 (14:35 +0100)]
phase1: remove unused cleanup script

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: pass buildbot url to cleanup.sh
Jo-Philipp Wich [Mon, 14 Nov 2016 13:33:52 +0000 (14:33 +0100)]
phase2: pass buildbot url to cleanup.sh

Instead of hardcoding the REST url in the cleanup script, pass the value
derived from the configuration.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: make SDK URL configurable
Jo-Philipp Wich [Mon, 14 Nov 2016 13:18:57 +0000 (14:18 +0100)]
phase2: make SDK URL configurable

Also allow to specify the SDK file name pattern and make the decompression
format agnostic.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: remove handling of unused repo url setting
Jo-Philipp Wich [Mon, 14 Nov 2016 12:48:15 +0000 (13:48 +0100)]
phase2: remove handling of unused repo url setting

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: make slave port configurable
Jo-Philipp Wich [Mon, 14 Nov 2016 12:46:07 +0000 (13:46 +0100)]
phase2: make slave port configurable

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: make slave port configurable
Jo-Philipp Wich [Mon, 14 Nov 2016 12:43:55 +0000 (13:43 +0100)]
phase1: make slave port configurable

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: build local tar before running downloads
Jo-Philipp Wich [Fri, 4 Nov 2016 16:54:04 +0000 (17:54 +0100)]
phase1: build local tar before running downloads

The download make target requires a modern GNU tar with --sort support in order
to pack SCM archives.

As Debian still ships an older tar, explicitely build the local tools/tar
before invoking "make download".

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: always allow user clean targets
Jo-Philipp Wich [Fri, 4 Nov 2016 15:45:31 +0000 (16:45 +0100)]
phase1: always allow user clean targets

Also allow user clean targets if a tree expiry has been configured.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: remove automatic triggering of clean targets
Jo-Philipp Wich [Fri, 4 Nov 2016 14:59:50 +0000 (15:59 +0100)]
phase1: remove automatic triggering of clean targets

Remove the automatic triggering of clean targets depending on the changed
files since the code for that is incompatible with BuildBot 0.8.9.

Instead, implement the ability to trigger specific clean steps through
build properties.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: remove isChangeBuiltin() handling
Jo-Philipp Wich [Fri, 4 Nov 2016 14:58:46 +0000 (15:58 +0100)]
phase1: remove isChangeBuiltin() handling

The builtin file detection logic is stubbed since quite a while already so
simply drop it now.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: initialize source.git on first start
Jo-Philipp Wich [Fri, 4 Nov 2016 14:56:02 +0000 (15:56 +0100)]
phase1: initialize source.git on first start

Since phase1 needs an existing source.git to bootstrap its config, prepare
the git clone further processing.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: rsync based on checksum, not mtime
Jo-Philipp Wich [Fri, 4 Nov 2016 13:19:25 +0000 (14:19 +0100)]
phase2: rsync based on checksum, not mtime

Instruct rsync to compare the file checksums, and not their modification times.
This should decrease the source and package upload volume significantly.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: rsync based on checksum, not mtime
Jo-Philipp Wich [Fri, 4 Nov 2016 13:16:32 +0000 (14:16 +0100)]
phase1: rsync based on checksum, not mtime

Instruct rsync to compare the file checksums, and not their modification times.
This should decrease the source upload volume significantly.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: parallelize target/install phase
Jo-Philipp Wich [Fri, 4 Nov 2016 13:07:35 +0000 (14:07 +0100)]
phase1: parallelize target/install phase

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: fix issues in previous commit
Jo-Philipp Wich [Fri, 4 Nov 2016 12:06:45 +0000 (13:06 +0100)]
phase1: fix issues in previous commit

 - Fix syntax error in expire.sh script
 - Download expire.sh into parent directory
 - Ensure that lifetime argument is passed as string when invoking expire.sh

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: add logic to purge trees after maximum lifetime
Jo-Philipp Wich [Fri, 4 Nov 2016 11:47:31 +0000 (12:47 +0100)]
phase1: add logic to purge trees after maximum lifetime

Add logic to the phase1 build config to purge persistent build trees if a
configurable maximum age in seconds is reached.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: adjust for .tar.xz packed SDK archives
Jo-Philipp Wich [Fri, 4 Nov 2016 10:43:11 +0000 (11:43 +0100)]
phase2: adjust for .tar.xz packed SDK archives

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: remove package/feeds before installing feeds
Jo-Philipp Wich [Fri, 4 Nov 2016 10:35:43 +0000 (11:35 +0100)]
phase1: remove package/feeds before installing feeds

This removal step is needed to get rid of dangling symlinks and to ensure
that core package overrides are effective.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: execute new checksum target
Jo-Philipp Wich [Mon, 1 Aug 2016 16:13:59 +0000 (18:13 +0200)]
phase1: execute new checksum target

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: enable new image building code
Jo-Philipp Wich [Fri, 29 Jul 2016 11:56:05 +0000 (13:56 +0200)]
phase1: enable new image building code

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: enforce proper umask during signing
Jo-Philipp Wich [Tue, 26 Jul 2016 18:09:27 +0000 (20:09 +0200)]
phase1: enforce proper umask during signing

Ensure that the umask is set to 022 when signing tarballs since twistd invokes
the helper script with an umask of 077, preventing world access on the
resulting files.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: preserve umask
Jo-Philipp Wich [Tue, 26 Jul 2016 17:05:52 +0000 (19:05 +0200)]
phase2: preserve umask

Remove "sh -c" indirection to preserve proper umask in tar command.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: fixup find command
Jo-Philipp Wich [Tue, 26 Jul 2016 17:04:51 +0000 (19:04 +0200)]
phase1: fixup find command

Ensure that the find command prints all paths, also remove "sh -c" indirection
to preserve proper umask.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoImplement GPG signing support
Jo-Philipp Wich [Tue, 26 Jul 2016 16:07:25 +0000 (18:07 +0200)]
Implement GPG signing support

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: next attempt to get proper libc interpolation
Jo-Philipp Wich [Wed, 20 Jul 2016 16:26:36 +0000 (18:26 +0200)]
phase1: next attempt to get proper libc interpolation

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: fix libc format specifier
Jo-Philipp Wich [Wed, 20 Jul 2016 14:34:52 +0000 (16:34 +0200)]
phase1: fix libc format specifier

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: handle source directory with libc suffix
Jo-Philipp Wich [Wed, 20 Jul 2016 13:01:12 +0000 (15:01 +0200)]
phase1: handle source directory with libc suffix

The ARC architectures use "generic-uClibc" instead of the expected "generic"
as subtarget directory name, leading to errors in the rsync step because the
expected directory cannot be found.

Extract the libc flavor from the generated .config and decide whether to use
a directory suffix based on the value obtained in order to let ARC uploads
complete successfully.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoEnable graceful slave shutdown option in WebUI for authorized user
Ted Hess [Thu, 30 Jun 2016 20:06:38 +0000 (16:06 -0400)]
Enable graceful slave shutdown option in WebUI for authorized user

Signed-off-by: Ted Hess <thess@kitschensync.net>
8 years agophase2: only support lower case SDK archive names now
Jo-Philipp Wich [Tue, 28 Jun 2016 16:22:00 +0000 (18:22 +0200)]
phase2: only support lower case SDK archive names now

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: use pkgarch as unique partial directory name
Jo-Philipp Wich [Tue, 14 Jun 2016 13:24:33 +0000 (15:24 +0200)]
phase2: use pkgarch as unique partial directory name

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: use target~subtarget as unique partial directory name
Jo-Philipp Wich [Tue, 14 Jun 2016 13:22:48 +0000 (15:22 +0200)]
phase1: use target~subtarget as unique partial directory name

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoRevert "Use slave instead of buildername for partial dir to avoid slashes"
Jo-Philipp Wich [Tue, 14 Jun 2016 13:20:42 +0000 (15:20 +0200)]
Revert "Use slave instead of buildername for partial dir to avoid slashes"

This reverts commit b71f4a13b2986dff0a6ed551ea4123e99a58df4a.

8 years agoUse slave instead of buildername for partial dir to avoid slashes
Jo-Philipp Wich [Tue, 14 Jun 2016 11:27:46 +0000 (13:27 +0200)]
Use slave instead of buildername for partial dir to avoid slashes

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoUse unique rsync partial directories per builder
Jo-Philipp Wich [Tue, 14 Jun 2016 09:10:50 +0000 (11:10 +0200)]
Use unique rsync partial directories per builder

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: implement dl/ uploading
Jo-Philipp Wich [Mon, 13 Jun 2016 11:18:51 +0000 (13:18 +0200)]
phase2: implement dl/ uploading

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: implement dl/ uploading
Jo-Philipp Wich [Mon, 13 Jun 2016 11:18:44 +0000 (13:18 +0200)]
phase1: implement dl/ uploading

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoExclude sqlite working files as well
Jo-Philipp Wich [Fri, 27 May 2016 12:41:33 +0000 (14:41 +0200)]
Exclude sqlite working files as well

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase1: implement IRC notifications
Jo-Philipp Wich [Fri, 27 May 2016 12:39:24 +0000 (14:39 +0200)]
phase1: implement IRC notifications

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: prepare for lowercase SDK archive names
Jo-Philipp Wich [Thu, 26 May 2016 14:46:45 +0000 (16:46 +0200)]
phase2: prepare for lowercase SDK archive names

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agophase2: force regeneration of SDK .config
Jo-Philipp Wich [Tue, 24 May 2016 21:44:06 +0000 (23:44 +0200)]
phase2: force regeneration of SDK .config

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoAdd missing properties
Jo-Philipp Wich [Mon, 16 May 2016 10:43:35 +0000 (12:43 +0200)]
Add missing properties

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoDetermine number of cores dynamically
Jo-Philipp Wich [Mon, 16 May 2016 10:40:22 +0000 (12:40 +0200)]
Determine number of cores dynamically

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoAdd libc variant check to stop arc770 and archs38 early
Jo-Philipp Wich [Wed, 11 May 2016 11:52:49 +0000 (13:52 +0200)]
Add libc variant check to stop arc770 and archs38 early

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoFix key-build permissions
Jo-Philipp Wich [Wed, 11 May 2016 11:45:27 +0000 (13:45 +0200)]
Fix key-build permissions

Don't use readonly permissions for key-build and key-build.pub as subsequent
runs are unable to overwrite the files then.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoUse --delay-updates on rsync upload to minimize time of inconsistency in the repo
Felix Fietkau [Fri, 6 May 2016 09:14:04 +0000 (10:14 +0100)]
Use --delay-updates on rsync upload to minimize time of inconsistency in the repo

During the rsync operation, package lists could go out of sync with the
.ipk files, or a mirror might be pulling an incomplete list of files.
To reduce the chance of this happening, let rsync put all updated files
into place at the end of the transfer

Signed-off-by: Felix Fietkau <nbd@nbd.name>
8 years agoAdd .gitignore
Jo-Philipp Wich [Tue, 10 May 2016 14:12:40 +0000 (16:12 +0200)]
Add .gitignore

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoAdd relocatable buildbot.tac files
Jo-Philipp Wich [Tue, 10 May 2016 14:00:48 +0000 (16:00 +0200)]
Add relocatable buildbot.tac files

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoCanonicalize home_dir, download key-build.pub as well
Jo-Philipp Wich [Tue, 10 May 2016 13:58:40 +0000 (15:58 +0200)]
Canonicalize home_dir, download key-build.pub as well

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoMove config.ini to config.ini.example and make paths relative
Jo-Philipp Wich [Tue, 10 May 2016 13:49:00 +0000 (15:49 +0200)]
Move config.ini to config.ini.example and make paths relative

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoRemove intermediate master directory
Jo-Philipp Wich [Tue, 10 May 2016 13:47:28 +0000 (15:47 +0200)]
Remove intermediate master directory

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoFix typos in example configs
Jo-Philipp Wich [Wed, 4 May 2016 19:00:31 +0000 (20:00 +0100)]
Fix typos in example configs

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
8 years agoAdd buildmaster configurations
Jo-Philipp Wich [Wed, 4 May 2016 16:26:49 +0000 (17:26 +0100)]
Add buildmaster configurations

Signed-off-by: Jo-Philipp Wich <jo@mein.io>