aboutsummaryrefslogtreecommitdiff
path: root/Tools
Commit message (Collapse)AuthorAgeFilesLines
* Force cleaning of the environment from customized local settings.Erwin Lansing2009-06-051-0/+5
| | | | Notes: svn path=/head/; revision=235211
* Switch the 'skipped' column to point to duds.verbose instead of duds,Mark Linimon2009-06-031-1/+1
| | | | | | | now that it is available in all current builds. Notes: svn path=/head/; revision=235083
* Use zfs snapshots that are periodically updated on pointyhat for theErwin Lansing2009-05-281-27/+78
| | | | | | | | normal builds instead of a local cvs update. This speeds up the script several times and easies I/O on pointyhat. Notes: svn path=/head/; revision=234704
* - Update to reflect a reality on pointyhatPav Lucistnik2009-05-2117-716/+989
| | | | Notes: svn path=/head/; revision=234345
* - Separate ports tree updating to use cvsup from cvsup-master instead of cvsPav Lucistnik2009-05-212-6/+56
| | | | | | | update from NFS mount, which is just way too slow Notes: svn path=/head/; revision=234344
* note mark_safe.plPhilip M. Gollucci2009-05-171-0/+1
| | | | Notes: svn path=/head/; revision=234081
* o don't set anything in ports with NO_BUILDPhilip M. Gollucci2009-05-171-3/+3
| | | | | | | | o fix rubygem skipping o next, not die Notes: svn path=/head/; revision=234078
* Do not mark rubygem ports, its a NO-OPPhilip M. Gollucci2009-05-161-0/+2
| | | | | | | Discussed with: pav on irc (#bsdports) Notes: svn path=/head/; revision=233969
* Make previous commit work for both context and unified input patches.Ion-Mihai Tetcu2009-05-101-1/+1
| | | | Notes: svn path=/head/; revision=233587
* When naming the split patch files, replace the original path '/' with '__'Ion-Mihai Tetcu2009-05-101-1/+1
| | | | | | | instead of a single '_' to follow the current patch naming guidelines. Notes: svn path=/head/; revision=233586
* - fix --port supportPhilip M. Gollucci2009-04-291-1/+3
| | | | | | | - don't change the Makefile if MAKE_JOBS_(UN)SAFE is already present Notes: svn path=/head/; revision=232958
* - Start some docsPhilip M. Gollucci2009-04-291-11/+93
| | | | | | | - Add --ports -p to specify a list on the cli Notes: svn path=/head/; revision=232957
* Script to set MAKE_JOBS_(UN)SAFE= yes for a MAINTAINERs ports.Philip M. Gollucci2009-04-221-0/+191
| | | | | | | | | | Please be sure to completely test the results of running this _BEFORE_ committing several times and in a Tinderbox Feel free to commit to this file Notes: svn path=/head/; revision=232474
* Fix bug where number of error logs was incorrect.Mark Linimon2009-03-261-8/+9
| | | | Notes: svn path=/head/; revision=231055
* Rewrite of processlog by kris.Mark Linimon2009-03-261-84/+147
| | | | Notes: svn path=/head/; revision=231054
* - Add a new case 'nested-declaration' to catch errors on -current.Mark Linimon2009-03-251-2/+4
| | | | | | | | - Fix false positives for 'cluster'. - Add 'error: invalid pure specifier' case to badc++. Notes: svn path=/head/; revision=230943
* Use diff -p both when first creating and when updating a patch.Christian Weisgerber2009-02-151-1/+1
| | | | | | | | Previously a patch created by update-patches would be changed again by the next update-patches run. Notes: svn path=/head/; revision=228415
* Remove 6-exp2 builds, they haven't been used in a whilePav Lucistnik2009-01-212-6/+3
| | | | Notes: svn path=/head/; revision=226621
* Factor out the code that performs the PORTREVISION bump for one port.Gerald Pfeifer2009-01-071-42/+47
| | | | | | | Approved by: edwin Notes: svn path=/head/; revision=225408
* Reduce per branch code duplicationErwin Lansing2008-12-271-41/+16
| | | | | | | Submitted by: pgollucci Notes: svn path=/head/; revision=224862
* Make sure we don't add ${WRKDIR}.Maxim Sobolev2008-10-211-4/+4
| | | | Notes: svn path=/head/; revision=221883
* Handle the rest of the edge cases; drop 5 as obsolete; cleanup.Mark Linimon2008-09-181-23/+35
| | | | | | | | | Now re-enabled on production. Approved by: portmgr (self) Notes: svn path=/head/; revision=220428
* Bring this into the new portbuild world order. While here, try toMark Linimon2008-09-171-27/+94
| | | | | | | | | suppress hrefs for non-existant files (incomplete). Approved by: portmgr (self) Notes: svn path=/head/; revision=220421
* RELENG_5 is desupported.Mark Linimon2008-09-121-3/+3
| | | | | | | Approved by: portmgr (self) Notes: svn path=/head/; revision=220355
* Major overhaul:Mark Linimon2008-09-071-349/+172
| | | | | | | | | | | | - bring this closer to the default FreeBSD page style - remove unsupported releases - remove the date stamps, which no longer work - remove obsolete commented-out junk Discussed on: portmgr, some time ago Notes: svn path=/head/; revision=220059
* Add a new script for performing high-level administration of a build.Kris Kennaway2008-08-081-518/+496
| | | | | | | | | | | | | | | | | | list : lists available builds clone : creates a new build by cloning a previous one portsupdate : update a ports tree to the latest ZFS snapshot srcupdate : update a src tree to the latest ZFS snapshot cleanup : clean up or remove a build on the clients destroy : remove a build on the server There is some trickiness here in that various commands either expect to run as root, or expect to run as a ports-* user. For the latter case we can easily use su to proxy as the ports user when running as root; for the former we use the buildproxy to validate and re-execute the command as root. Notes: svn path=/head/; revision=218250
* Add an option to clean up old distfiles. We don't have enough space toKris Kennaway2008-08-081-0/+10
| | | | | | | keep dozens of full distfile sets lying around. Notes: svn path=/head/; revision=218249
* Update the list of openoffice.org ports: remove 1.x and add 3.xErwin Lansing2008-07-291-1/+1
| | | | Notes: svn path=/head/; revision=217823
* Oops, commit the right version of this (adds extra checks that onlyKris Kennaway2008-07-261-0/+5
| | | | | | | expected commands will be proxied) Notes: svn path=/head/; revision=217607
* * Simple client that passes on a build request to the buildproxyKris Kennaway2008-07-261-0/+32
| | | | Notes: svn path=/head/; revision=217606
* * Python daemon run as root that proxies privileged build commands forKris Kennaway2008-07-261-0/+75
| | | | | | | | | | | the ports-* users. Currently it is not possible to delegate management of ZFS filesystems to non-root users, so root privilege is required to manipulate them. We validate the command passed on a local domain socket and re-execute the build script with the requested parameters. Notes: svn path=/head/; revision=217605
* Script run from cron to regularly update the master ZFS copies of theKris Kennaway2008-07-261-0/+63
| | | | | | | | | | | | | | | | | ports and source trees. Since we have >=1 consumer of these trees that run frequently but do not insist on up-to-the-second trees, it makes sense to "pre-update" them regularly and then then re-use in all of the consumers, instead of potentially doing several updates simultaneously or on demand. Consumers can clone the ZFS snapshot into their local filesystem which takes a couple of seconds instead of minutes or tens of minutes for the CVS update. We update to a date stamp instead of "." because this avoids ambiguity of commits that happen while the tree update is in progress (unfortunately it's slower). Notes: svn path=/head/; revision=217604
* Script run from cron on the package clients to report metrics to ganglia.Kris Kennaway2008-07-261-0/+35
| | | | | | | | | | | Currently we collect: * The current and maximum number of vnodes in use * The number of packages built over the past hour Notes: svn path=/head/; revision=217603
* Simple script to expire ZFS snapshots older than a certain ageKris Kennaway2008-07-261-0/+47
| | | | Notes: svn path=/head/; revision=217602
* Python script for backing up ZFS filesystems on pointyhat. For eachKris Kennaway2008-07-261-0/+217
| | | | | | | | | listed filesystem we take a new snapshot each time it is run and if the last full backup was not too long ago, do a compressed incremental backup from the previous backup. Notes: svn path=/head/; revision=217601
* * Add comment that this is unusedKris Kennaway2008-07-261-0/+1
| | | | Notes: svn path=/head/; revision=217600
* * Add commentsKris Kennaway2008-07-261-1/+6
| | | | Notes: svn path=/head/; revision=217599
* * CleanupKris Kennaway2008-07-261-28/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Support a meta-hostname of 'all' for setting up all clients at once. This is better than the old way of running one copy of the script for each client by hand, since it is easier and involves less duplicated work. * We copy in the per-build ports, src, and bindist .tbz files and .md5 checksums, as well as refreshing the build scripts and bindist-$(hostname).tar customization tarball. * The -force switch forces copying of files and re-extraction of the tarballs on the client. This is necessary in order to propagate local changes to the tarballs after the initial client setup (e.g. if you need to change a file in the ports tree, it must be recompressed, redistributed, and re-extracted on the client). * The -queue switch will poll the client's job queue after completion of the setup. This is racy and should only be used when the machine is not currently accepting jobs. * For cleaning up a build the 'build cleanup' command should now be used instead. It calls back into this command but also allows full clenaup of build-local files on the client. TODO: "all" setups are hard on the server since they may spawn dozens of rsyncs at once. A better solution would be to have a worker pool of setup tasks to limit the maximum load. Notes: svn path=/head/; revision=217598
* * CleanupKris Kennaway2008-07-261-28/+23
| | | | | | | | | | | | | * Catch up to build ID directory changes * Make it easier to kill a build by not running dopackages in the background where it is detached from shell job control. Now, sending a termination signal to this process (e.g. ^C) will also kill off the dopackages script and in turn the processes created by it. Some background processes spawned by dopackages, pdispatch, etc, may still remain and need to be killed by hand. Notes: svn path=/head/; revision=217597
* * CleanupKris Kennaway2008-07-261-496/+581
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Improve usage() * Fix a variety of small bugs * Remove support for -ftp builds: we have not supported direct uploading for many years due to the desire to manually inspect build output for quality * All data associated to a build is now localized in its own directory named according to a build ID: /var/portbuild/${arch}/${branch}/builds/${buildid}, where ${buildid} is the creation time. These are actually ZFS filesystems. * Tasks such as cloning a new build, updating a ZFS snapshot, and cleaning up a build are exported to the "build" script, which can be used independently. * Creating a new build is done by ZFS cloning and takes a couple of seconds since it is copy-on-write (i.e. no data needs to be copied). * Ports and source trees are also cloned from pre-updated ZFS images (updated regularly from the "updatesnap" cron job). In most cases we do not care if we are building a ports tree that is an hour or so old since it will become outdated almost immediately anyway, so no matter what we do there will be times when a port has been fixed by the time the build error is generated by a client. * In case an up-to-the-second tree is desired, the -portscvs and -srccvs switches update the existing ports tree via CVS. * -noports and -nosrc can be used to prevent any automatic changes to the ports tree. This is useful for dealing with local modifications (e.g. for -exp builds), since the default when creating a new build is to replace the previous trees with fresh, pristine trees. If you forget to use this then any local changes that are not also present in other trees will be lost. * By default we keep two builds for each arch/branch pair. These build IDs also may be referred to via "latest" and "previous" symlinks. When creating a new build, the old "previous" build is destroyed by default, unless it was originally created using the -keep switch. This prevents the build from being destroyed automatically. * By default when a build finishes all of the clients are completely cleaned up (i.e. all build data such as ports trees, tarballs, client chroots, etc are deleted). This is needed to save space on the clients. If you expect to *immediately* perform further builds after this one completes, the -nocleanup switch prevents this step. Otherwise they will just be set up again if further builds are scheduled. * Try to parallelize build pre-processing as much as possible, by running jobs in the background wherever possible. In several places we operate on the same parts of the filesystem from multiple jobs, so we can make good use of caching to improve performance * Clients no longer need to be set up explicitly at the start of the build, they will be set up on-demand when the first job is dispatched to them. This allows fast clients or those that already have been set up to begin building ports as soon as possible, while slow clients are set up in the background. It also improves robustness of client recovery, e.g. if the client was offline at the time of build startup but later brought back online. * Optimize copying back in the previous set of restricted packages by hardlinking instead of copying. TODO: The record of failed ports is arch/branch-global still. This is the only thing preventing us from running concurrent builds of the same arch/branch (e.g. while one is stuck building openoffice, the next build can start to keep the cluster busy). The difficulty is that one build from a later ports tree may signal that a build was successful, then a phase 2 build from an earlier ports tree may indicate that it was broken. The solution is probably to migrate this to a real database instead of a flat file, and query it for the set of broken ports as of a certain ports tree date. Notes: svn path=/head/; revision=217596
* * Catch up to build ID directory changesKris Kennaway2008-07-261-42/+155
| | | | | | | | | | | | | | | | | | | | | * Clients no longer mount ports/src trees via NFS (even the FreeBSD.org local clients). This was putting too much load on the server and slowing down builds. * Instead ports and src .tbz files are pushed to the clients and unpacked. MD5 checksums are used to verify correctness * -force forces re-extraction of the tarballs even if they exist and appear to be checked out * Also unpack the compressed bindist TODO: When we are not using md or ZFS builds it would be even faster to keep an unpacked copy of the bindist on the scratch filesystem and hardlink the files into the target directory Notes: svn path=/head/; revision=217594
* * Catch up to build ID directory changesKris Kennaway2008-07-261-3/+24
| | | | | | | | * Optimize by copying old packages using cpio -dumpl (i.e. create hardlink instead of copying the file). Notes: svn path=/head/; revision=217592
* * Catch up to build ID directory changesKris Kennaway2008-07-262-9/+16
| | | | Notes: svn path=/head/; revision=217591
* * CleanupKris Kennaway2008-07-261-50/+51
| | | | | | | | * Catch up to build ID directory changes * Remove need for /etc/arch file Notes: svn path=/head/; revision=217590
* * CleanupKris Kennaway2008-07-261-71/+73
| | | | | | | | | * Catch up to build ID directory changes * Improved support for ZFS * Desupport X11BASE Notes: svn path=/head/; revision=217589
* * CleanupKris Kennaway2008-07-261-75/+120
| | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Improved support for ZFS builds * Improved robustness * Report status verbosely to the caller; whether we succeeded in claiming a chroot, whether the caller needs to first set up the client, or whether a setup is in progress. * If we discover that the client has not been set up either because it freshly booted and newfs'ed its filesystem, or because a particular build has not yet been encountered, atomically claim a cookie and report this to the caller to act on Notes: svn path=/head/; revision=217588
* * CleanupKris Kennaway2008-07-263-24/+50
| | | | | | | * Catch up to build ID directory changes Notes: svn path=/head/; revision=217587
* * CleanupKris Kennaway2008-07-261-19/+74
| | | | | | | | | | * Catch up to build ID directory changes * Add helper functions for resolving a build ID symlink and validating an arch/branch combination (centralize instead of doing it in many scripts) Notes: svn path=/head/; revision=217586
* * Desupport alpha and ia64Kris Kennaway2008-07-261-4/+3
| | | | | | | * Catch up to build ID directory changes Notes: svn path=/head/; revision=217585
* * Add commentsKris Kennaway2008-07-261-2/+2
| | | | Notes: svn path=/head/; revision=217584
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-07 16:48:48 +0000