aboutsummaryrefslogtreecommitdiff
path: root/Tools
Commit message (Collapse)AuthorAgeFilesLines
* Script to set MAKE_JOBS_(UN)SAFE= yes for a MAINTAINERs ports.Philip M. Gollucci2009-04-221-0/+191
| | | | | | | | | | Please be sure to completely test the results of running this _BEFORE_ committing several times and in a Tinderbox Feel free to commit to this file Notes: svn path=/head/; revision=232474
* Fix bug where number of error logs was incorrect.Mark Linimon2009-03-261-8/+9
| | | | Notes: svn path=/head/; revision=231055
* Rewrite of processlog by kris.Mark Linimon2009-03-261-84/+147
| | | | Notes: svn path=/head/; revision=231054
* - Add a new case 'nested-declaration' to catch errors on -current.Mark Linimon2009-03-251-2/+4
| | | | | | | | - Fix false positives for 'cluster'. - Add 'error: invalid pure specifier' case to badc++. Notes: svn path=/head/; revision=230943
* Use diff -p both when first creating and when updating a patch.Christian Weisgerber2009-02-151-1/+1
| | | | | | | | Previously a patch created by update-patches would be changed again by the next update-patches run. Notes: svn path=/head/; revision=228415
* Remove 6-exp2 builds, they haven't been used in a whilePav Lucistnik2009-01-212-6/+3
| | | | Notes: svn path=/head/; revision=226621
* Factor out the code that performs the PORTREVISION bump for one port.Gerald Pfeifer2009-01-071-42/+47
| | | | | | | Approved by: edwin Notes: svn path=/head/; revision=225408
* Reduce per branch code duplicationErwin Lansing2008-12-271-41/+16
| | | | | | | Submitted by: pgollucci Notes: svn path=/head/; revision=224862
* Make sure we don't add ${WRKDIR}.Maxim Sobolev2008-10-211-4/+4
| | | | Notes: svn path=/head/; revision=221883
* Handle the rest of the edge cases; drop 5 as obsolete; cleanup.Mark Linimon2008-09-181-23/+35
| | | | | | | | | Now re-enabled on production. Approved by: portmgr (self) Notes: svn path=/head/; revision=220428
* Bring this into the new portbuild world order. While here, try toMark Linimon2008-09-171-27/+94
| | | | | | | | | suppress hrefs for non-existant files (incomplete). Approved by: portmgr (self) Notes: svn path=/head/; revision=220421
* RELENG_5 is desupported.Mark Linimon2008-09-121-3/+3
| | | | | | | Approved by: portmgr (self) Notes: svn path=/head/; revision=220355
* Major overhaul:Mark Linimon2008-09-071-349/+172
| | | | | | | | | | | | - bring this closer to the default FreeBSD page style - remove unsupported releases - remove the date stamps, which no longer work - remove obsolete commented-out junk Discussed on: portmgr, some time ago Notes: svn path=/head/; revision=220059
* Add a new script for performing high-level administration of a build.Kris Kennaway2008-08-081-518/+496
| | | | | | | | | | | | | | | | | | list : lists available builds clone : creates a new build by cloning a previous one portsupdate : update a ports tree to the latest ZFS snapshot srcupdate : update a src tree to the latest ZFS snapshot cleanup : clean up or remove a build on the clients destroy : remove a build on the server There is some trickiness here in that various commands either expect to run as root, or expect to run as a ports-* user. For the latter case we can easily use su to proxy as the ports user when running as root; for the former we use the buildproxy to validate and re-execute the command as root. Notes: svn path=/head/; revision=218250
* Add an option to clean up old distfiles. We don't have enough space toKris Kennaway2008-08-081-0/+10
| | | | | | | keep dozens of full distfile sets lying around. Notes: svn path=/head/; revision=218249
* Update the list of openoffice.org ports: remove 1.x and add 3.xErwin Lansing2008-07-291-1/+1
| | | | Notes: svn path=/head/; revision=217823
* Oops, commit the right version of this (adds extra checks that onlyKris Kennaway2008-07-261-0/+5
| | | | | | | expected commands will be proxied) Notes: svn path=/head/; revision=217607
* * Simple client that passes on a build request to the buildproxyKris Kennaway2008-07-261-0/+32
| | | | Notes: svn path=/head/; revision=217606
* * Python daemon run as root that proxies privileged build commands forKris Kennaway2008-07-261-0/+75
| | | | | | | | | | | the ports-* users. Currently it is not possible to delegate management of ZFS filesystems to non-root users, so root privilege is required to manipulate them. We validate the command passed on a local domain socket and re-execute the build script with the requested parameters. Notes: svn path=/head/; revision=217605
* Script run from cron to regularly update the master ZFS copies of theKris Kennaway2008-07-261-0/+63
| | | | | | | | | | | | | | | | | ports and source trees. Since we have >=1 consumer of these trees that run frequently but do not insist on up-to-the-second trees, it makes sense to "pre-update" them regularly and then then re-use in all of the consumers, instead of potentially doing several updates simultaneously or on demand. Consumers can clone the ZFS snapshot into their local filesystem which takes a couple of seconds instead of minutes or tens of minutes for the CVS update. We update to a date stamp instead of "." because this avoids ambiguity of commits that happen while the tree update is in progress (unfortunately it's slower). Notes: svn path=/head/; revision=217604
* Script run from cron on the package clients to report metrics to ganglia.Kris Kennaway2008-07-261-0/+35
| | | | | | | | | | | Currently we collect: * The current and maximum number of vnodes in use * The number of packages built over the past hour Notes: svn path=/head/; revision=217603
* Simple script to expire ZFS snapshots older than a certain ageKris Kennaway2008-07-261-0/+47
| | | | Notes: svn path=/head/; revision=217602
* Python script for backing up ZFS filesystems on pointyhat. For eachKris Kennaway2008-07-261-0/+217
| | | | | | | | | listed filesystem we take a new snapshot each time it is run and if the last full backup was not too long ago, do a compressed incremental backup from the previous backup. Notes: svn path=/head/; revision=217601
* * Add comment that this is unusedKris Kennaway2008-07-261-0/+1
| | | | Notes: svn path=/head/; revision=217600
* * Add commentsKris Kennaway2008-07-261-1/+6
| | | | Notes: svn path=/head/; revision=217599
* * CleanupKris Kennaway2008-07-261-28/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Support a meta-hostname of 'all' for setting up all clients at once. This is better than the old way of running one copy of the script for each client by hand, since it is easier and involves less duplicated work. * We copy in the per-build ports, src, and bindist .tbz files and .md5 checksums, as well as refreshing the build scripts and bindist-$(hostname).tar customization tarball. * The -force switch forces copying of files and re-extraction of the tarballs on the client. This is necessary in order to propagate local changes to the tarballs after the initial client setup (e.g. if you need to change a file in the ports tree, it must be recompressed, redistributed, and re-extracted on the client). * The -queue switch will poll the client's job queue after completion of the setup. This is racy and should only be used when the machine is not currently accepting jobs. * For cleaning up a build the 'build cleanup' command should now be used instead. It calls back into this command but also allows full clenaup of build-local files on the client. TODO: "all" setups are hard on the server since they may spawn dozens of rsyncs at once. A better solution would be to have a worker pool of setup tasks to limit the maximum load. Notes: svn path=/head/; revision=217598
* * CleanupKris Kennaway2008-07-261-28/+23
| | | | | | | | | | | | | * Catch up to build ID directory changes * Make it easier to kill a build by not running dopackages in the background where it is detached from shell job control. Now, sending a termination signal to this process (e.g. ^C) will also kill off the dopackages script and in turn the processes created by it. Some background processes spawned by dopackages, pdispatch, etc, may still remain and need to be killed by hand. Notes: svn path=/head/; revision=217597
* * CleanupKris Kennaway2008-07-261-496/+581
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Improve usage() * Fix a variety of small bugs * Remove support for -ftp builds: we have not supported direct uploading for many years due to the desire to manually inspect build output for quality * All data associated to a build is now localized in its own directory named according to a build ID: /var/portbuild/${arch}/${branch}/builds/${buildid}, where ${buildid} is the creation time. These are actually ZFS filesystems. * Tasks such as cloning a new build, updating a ZFS snapshot, and cleaning up a build are exported to the "build" script, which can be used independently. * Creating a new build is done by ZFS cloning and takes a couple of seconds since it is copy-on-write (i.e. no data needs to be copied). * Ports and source trees are also cloned from pre-updated ZFS images (updated regularly from the "updatesnap" cron job). In most cases we do not care if we are building a ports tree that is an hour or so old since it will become outdated almost immediately anyway, so no matter what we do there will be times when a port has been fixed by the time the build error is generated by a client. * In case an up-to-the-second tree is desired, the -portscvs and -srccvs switches update the existing ports tree via CVS. * -noports and -nosrc can be used to prevent any automatic changes to the ports tree. This is useful for dealing with local modifications (e.g. for -exp builds), since the default when creating a new build is to replace the previous trees with fresh, pristine trees. If you forget to use this then any local changes that are not also present in other trees will be lost. * By default we keep two builds for each arch/branch pair. These build IDs also may be referred to via "latest" and "previous" symlinks. When creating a new build, the old "previous" build is destroyed by default, unless it was originally created using the -keep switch. This prevents the build from being destroyed automatically. * By default when a build finishes all of the clients are completely cleaned up (i.e. all build data such as ports trees, tarballs, client chroots, etc are deleted). This is needed to save space on the clients. If you expect to *immediately* perform further builds after this one completes, the -nocleanup switch prevents this step. Otherwise they will just be set up again if further builds are scheduled. * Try to parallelize build pre-processing as much as possible, by running jobs in the background wherever possible. In several places we operate on the same parts of the filesystem from multiple jobs, so we can make good use of caching to improve performance * Clients no longer need to be set up explicitly at the start of the build, they will be set up on-demand when the first job is dispatched to them. This allows fast clients or those that already have been set up to begin building ports as soon as possible, while slow clients are set up in the background. It also improves robustness of client recovery, e.g. if the client was offline at the time of build startup but later brought back online. * Optimize copying back in the previous set of restricted packages by hardlinking instead of copying. TODO: The record of failed ports is arch/branch-global still. This is the only thing preventing us from running concurrent builds of the same arch/branch (e.g. while one is stuck building openoffice, the next build can start to keep the cluster busy). The difficulty is that one build from a later ports tree may signal that a build was successful, then a phase 2 build from an earlier ports tree may indicate that it was broken. The solution is probably to migrate this to a real database instead of a flat file, and query it for the set of broken ports as of a certain ports tree date. Notes: svn path=/head/; revision=217596
* * Catch up to build ID directory changesKris Kennaway2008-07-261-42/+155
| | | | | | | | | | | | | | | | | | | | | * Clients no longer mount ports/src trees via NFS (even the FreeBSD.org local clients). This was putting too much load on the server and slowing down builds. * Instead ports and src .tbz files are pushed to the clients and unpacked. MD5 checksums are used to verify correctness * -force forces re-extraction of the tarballs even if they exist and appear to be checked out * Also unpack the compressed bindist TODO: When we are not using md or ZFS builds it would be even faster to keep an unpacked copy of the bindist on the scratch filesystem and hardlink the files into the target directory Notes: svn path=/head/; revision=217594
* * Catch up to build ID directory changesKris Kennaway2008-07-261-3/+24
| | | | | | | | * Optimize by copying old packages using cpio -dumpl (i.e. create hardlink instead of copying the file). Notes: svn path=/head/; revision=217592
* * Catch up to build ID directory changesKris Kennaway2008-07-262-9/+16
| | | | Notes: svn path=/head/; revision=217591
* * CleanupKris Kennaway2008-07-261-50/+51
| | | | | | | | * Catch up to build ID directory changes * Remove need for /etc/arch file Notes: svn path=/head/; revision=217590
* * CleanupKris Kennaway2008-07-261-71/+73
| | | | | | | | | * Catch up to build ID directory changes * Improved support for ZFS * Desupport X11BASE Notes: svn path=/head/; revision=217589
* * CleanupKris Kennaway2008-07-261-75/+120
| | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Improved support for ZFS builds * Improved robustness * Report status verbosely to the caller; whether we succeeded in claiming a chroot, whether the caller needs to first set up the client, or whether a setup is in progress. * If we discover that the client has not been set up either because it freshly booted and newfs'ed its filesystem, or because a particular build has not yet been encountered, atomically claim a cookie and report this to the caller to act on Notes: svn path=/head/; revision=217588
* * CleanupKris Kennaway2008-07-263-24/+50
| | | | | | | * Catch up to build ID directory changes Notes: svn path=/head/; revision=217587
* * CleanupKris Kennaway2008-07-261-19/+74
| | | | | | | | | | * Catch up to build ID directory changes * Add helper functions for resolving a build ID symlink and validating an arch/branch combination (centralize instead of doing it in many scripts) Notes: svn path=/head/; revision=217586
* * Desupport alpha and ia64Kris Kennaway2008-07-261-4/+3
| | | | | | | * Catch up to build ID directory changes Notes: svn path=/head/; revision=217585
* * Add commentsKris Kennaway2008-07-261-2/+2
| | | | Notes: svn path=/head/; revision=217584
* * Implement basename and dirname using shell builtinsKris Kennaway2008-07-261-0/+8
| | | | Notes: svn path=/head/; revision=217583
* * CleanupKris Kennaway2008-07-261-13/+15
| | | | | | | | * Catch up to build ID directory changes * Record package build completion for reporting to ganglia Notes: svn path=/head/; revision=217582
* * CleanupKris Kennaway2008-07-261-104/+160
| | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Add support for ssh_cmd and scp_cmd to allow using HPN-SSH with the none cipher where possible (for performance) * Lazy client setup; claim-chroot will report if the client needs to be set up with this buildid, and we initiate the setup and poll until it is complete. This allows fast clients to begin building before slow ones have finished setting up. TODO: a better solution would be to avoid trying to dispatch jobs onto clients that are in the process of setting up, since they often have low loads and are picked preferentially by the job scheduler. Notes: svn path=/head/; revision=217580
* * CleanupKris Kennaway2008-07-261-65/+41
| | | | | | | | | | | | | * Remove vestiges of archaic support for building bindists from FTP snapshots; we haven't used this for years and building a world is no longer a challenge * Revert half-baked bindist generation number and make it per-buildid instead. Compress and md5 it for distribution to the clients. TODO: Merge with makeworld? Notes: svn path=/head/; revision=217576
* * CleanupKris Kennaway2008-07-261-17/+24
| | | | | | | | | * Catch up to build ID directory changes * Optimize by using ECHO_MSG=true instead of /usr/bin/true * Try harder to avoid pollution from local host Notes: svn path=/head/; revision=217574
* * CleanupKris Kennaway2008-07-261-7/+18
| | | | | | | * Catch up to build ID directory layout Notes: svn path=/head/; revision=217573
* * CleanupKris Kennaway2008-07-261-51/+18
| | | | | | | | | * Catch up to build ID directory changes * Export the INDEX_PRISTINE and INDEX_QUIET variables (old bug) * Desupport X11BASE Notes: svn path=/head/; revision=217572
* * CleanupKris Kennaway2008-07-261-38/+39
| | | | | | | | * Catch up to build ID directory changes * Desupport 5.x Notes: svn path=/head/; revision=217570
* * CleanupKris Kennaway2008-07-261-15/+17
| | | | | | | | * Catch up to build ID directory changes * Optimize by using __MAKE_SHELL=/rescue/sh Notes: svn path=/head/; revision=217569
* Rewrite in python and combine the functions of the formerKris Kennaway2008-07-261-35/+263
| | | | | | | | | | | | checkmachines script. Polls build machines for their status either once-off or regularly as a daemon. Optionally it will update the queue entries but this remains subject to race conditions. TODO: Integrate with queue manager and forward machine status changes to it Notes: svn path=/head/; revision=217568
* More verbose status reporting using key=value format. We now alsoKris Kennaway2008-07-261-8/+18
| | | | | | | | report error status, architecture and OS version, and available build environments, as well as load and number of running jobs Notes: svn path=/head/; revision=217567
* Major optimizations for 'make index' and other recursive traversalKris Kennaway2008-07-191-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | targets. * Use /rescue/sh for index builds instead of /bin/sh, when it exists. The former is statically linked and faster to execute, which becomes significant when executing it tens of thousands of times. This trick can be used with other recursive targets by passing in __MAKE_SHELL. * Get rid of make variable assignments that use != command invocations in the critical path, using several methods: - rewriting logic to use shell or make builtins instead of external command executions - macroizing commands and executing them in the targets where they are needed instead of with every invocation of make - precomputing the results of invariant commands in bsd.port.subdir.mk and passing them in explicitly to child makes, and using this to avoid recalculation in all the children. NB: the commands are still run one per top-level subdirectory but this does not currently seem to be a major issue. They could be moved further up into the top-level Makefile at the cost of some cleanliness. - Committers are strongly discouraged from adding further "bare" != assignments to the ports tree, even in their own ports. One of the above strategies should be used to avoid future bloat. * Rewrite the core 'describe' target to work entirely within a single shell process using only builtin commands. The old version is retained as a backup for use on systems older than 603104, which does not have the make :u modifier. This cuts down the number of processes executed during the course of a 'make index' by an order of magnitude, and we are essentially now amortized to the minimum of a single make + sh instance per port, plus whatever commands the port makefile itself executes (which are usually unnecessary and bogus). * Less validation of the WWW: target is performed; this can become policed at a port level by portlint. Specifically we look at the second word of the first line beginning with "WWW:" in pkg-descr, and append "http://" to it unless it already begins with "http://", "https://" or "ftp://". Thanks to dougb for the idea of how to extract WWW: using shell builtins. * Use the "true" shell builtin instead of echo > /dev/null for a measurable decrease in CPU use. * Add a note about dubious escaping strategy in bsd.port.subdir.mk * Minor change in output of 'make describe': it no longer strips trailing CR characters from pkg-descr files with MSDOS CR/LF termination. Instead the makeindex perl script that post-processes make describe into the INDEX is tweaked to strip on input. The bottom line is that on my test hardware INDEX builds are now faster by more than a factor of 2 and with a reduction in system time by a factor of 4-8 depending on configuration. Notes: svn path=/head/; revision=217132
generated by cgit v1.2.3 (git 2.25.1) at 2024-10-14 02:43:52 +0000