aboutsummaryrefslogtreecommitdiff
path: root/sys/kern/kern_lock.c
Commit message (Collapse)AuthorAgeFilesLines
* locks: push lock_delay_arg_init calls downMateusz Guzik2020-11-241-1/+1
| | | | | | | | Minor cleanup to skip doing them when recursing on locks and so that they can act on found lock value if need be. Notes: svn path=/head/; revision=367978
* kern: clean up empty lines in .c and .h filesMateusz Guzik2020-09-011-3/+0
| | | | Notes: svn path=/head/; revision=365222
* lockmgr: add missing 'continue' to account for spuriously failed fcmpsetMateusz Guzik2020-07-241-0/+1
| | | | | | | | | | PR: 248245 Reported by: gbe Noted by: markj Fixes by: r363415 ("lockmgr: add adaptive spinning") Notes: svn path=/head/; revision=363480
* lockmgr: add adaptive spinningMateusz Guzik2020-07-221-18/+111
| | | | | | | | | | | | | | | | | | | | | | It is very conservative. Only spinning when LK_ADAPTIVE is passed, only on exclusive lock and never when any waiters are present. buffer cache is remains not spinning. This reduces total sleep times during buildworld etc., but it does not shorten total real time (culprits are contention in the vm subsystem along with slock + upgrade which is not covered). For microbenchmarks: open3_processes -t 52 (open/close of the same file for writing) ops/s: before: 258845 after: 801638 Reviewed by: kib Tested by: pho Differential Revision: https://reviews.freebsd.org/D25753 Notes: svn path=/head/; revision=363415
* lockmgr: denote recursion with a bit in lock valueMateusz Guzik2020-07-211-4/+6
| | | | | | | | | This reduces excessive reads from the lock. Tested by: pho Notes: svn path=/head/; revision=363394
* lockmgr: rewrite upgrade to stop always dropping the lockMateusz Guzik2020-07-211-36/+34
| | | | | | | This matches rw and sx locks. Notes: svn path=/head/; revision=363393
* lockmgr: add a helper for reading the lock valueMateusz Guzik2020-07-211-17/+17
| | | | Notes: svn path=/head/; revision=363392
* When running with a kernel compiled with DEBUG_LOCKS, beforeKirk McKusick2020-04-091-0/+1
| | | | | | | | panic'ing for recusing on a non-recursive lock, print out the kernel stack where the lock was originally acquired. Notes: svn path=/head/; revision=359759
* lockmgr: rename lock_fast_path to lock_flagsMateusz Guzik2020-02-141-2/+2
| | | | | | | | The routine is not much of a fast path and the flags name better describes its purpose. Notes: svn path=/head/; revision=357907
* lockmgr: retire the unused lockmgr_unlock_fast_path routineMateusz Guzik2020-02-141-40/+0
| | | | Notes: svn path=/head/; revision=357906
* lockmgr: don't touch the lock past unlockMateusz Guzik2020-01-241-14/+16
| | | | | | | | | | | | | This evens it up with other locking primitives. Note lock profiling still touches the lock, which again is in line with the rest. Reviewed by: jeff Differential Revision: https://reviews.freebsd.org/D23343 Notes: svn path=/head/; revision=357069
* Add KERNEL_PANICKED macro for use in place of direct panicstr testsMateusz Guzik2020-01-121-9/+9
| | | | Notes: svn path=/head/; revision=356655
* sleep(9), sleepqueue(9): const'ify wchan pointersConrad Meyer2019-12-241-1/+1
| | | | | | | | | | | | | | | | | | _sleep(9), wakeup(9), sleepqueue(9), et al do not dereference or modify the channel pointers provided in any way; they are merely used as intptrs into a dictionary structure to match waiters with wakers. Correctly annotate this such that _sleep() and wakeup() may be used on const pointers without invoking ugly patterns like __DECONST(). Plumb const through all of the underlying sleepqueue bits. No functional change. Reviewed by: rlibby Discussed with: kib, markj Differential Revision: https://reviews.freebsd.org/D22914 Notes: svn path=/head/; revision=356057
* vfs: locking primitives which elide ->v_vnlock and shared locking disablementMateusz Guzik2019-12-111-0/+82
| | | | | | | | | | | | | | | | | | | Both of these features are not needed by many consumers and result in avoidable reads which in turn puts them on profiles due to cache-line ping ponging. On top of that the current lockgmr entry point is slower than necessary single-threaded. As an attempted clean up preparing for other changes, provide new routines which don't support any of the aforementioned features. With these patches in place vop_stdlock and vop_stdunlock disappear from flamegraphs during -j 104 buildkernel. Reviewed by: jeff (previous version) Tested by: pho Differential Revision: https://reviews.freebsd.org/D22665 Notes: svn path=/head/; revision=355633
* lockmgr: remove more remnants of adaptive spinningMateusz Guzik2019-12-011-7/+1
| | | | | | | Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=355256
* Add lockmgr(9) probes to the lockstat DTrace provider.Mark Johnston2019-08-211-8/+43
| | | | | | | | | | | | | | | | | They follow the conventions set by rw and sx lock probes. There is an additional lockstat:::lockmgr-disown probe. Update lockstat(1) to report on contention and hold events for lockmgr locks. Document the new probes in dtrace_lockstat.4, and deduplicate some of the existing probe descriptions. Reviewed by: mjg MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21355 Notes: svn path=/head/; revision=351361
* Add flag LK_NEW for lockinit() that is converted to LO_NEW and passedGleb Smirnoff2019-01-151-0/+2
| | | | | | | | down to lock_init(). This allows for lockinit() on a not prezeroed memory. Notes: svn path=/head/; revision=343027
* Annotate Giant drop/pickup macros with __predict_falseMateusz Guzik2018-12-071-2/+2
| | | | | | | | | | They are used in important places of the kernel with the lock not being held majority of the time. Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=341683
* lockmgr: tidy up slock/sunlock similar to other locksMateusz Guzik2018-07-131-29/+22
| | | | Notes: svn path=/head/; revision=336267
* remove unused locked variable in lockmgr_unlock_fast_pathMatt Macy2018-05-191-4/+0
| | | | Notes: svn path=/head/; revision=333835
* lockmgr: avoid atomic on unlock in the slow pathMateusz Guzik2018-05-181-4/+6
| | | | | | | | | | | The code is pretty much guaranteed not to be able to unlock. This is a minor nit. The code still performs way too many reads. The altered exclusive-locked condition is supposed to be always true as well, to be cleaned up at a later date. Notes: svn path=/head/; revision=333816
* lockmgr: Add missed neutering during panicConrad Meyer2018-04-241-5/+9
| | | | | | | | | | | | | | | | r313683 introduced new lockmgr APIs that missed the panic-time neutering present in the rest of our locks. Correct that by adding the usual check. Additionally, move the __lockmgr_args neutering above the assertions at the top of the function. Drop the interlock unlock because we shouldn't have an unneutered interlock either. No point trying to unlock it. PR: 227749 Reported by: jtl Sponsored by: Dell EMC Isilon Notes: svn path=/head/; revision=332957
* At this point iwmesg isn't initialized yet, so print pointer to lockGleb Smirnoff2018-03-201-2/+2
| | | | | | | rather than panic before panicing. Notes: svn path=/head/; revision=331272
* lockmgr: save on sleepq when cmpset failsMateusz Guzik2018-03-051-14/+20
| | | | Notes: svn path=/head/; revision=330432
* lockmgr: whack unused lockmgr_note_exclusive_upgradeMateusz Guzik2018-03-041-12/+0
| | | | Notes: svn path=/head/; revision=330419
* lockmgr: start decomposing the main routineMateusz Guzik2018-03-041-396/+542
| | | | | | | | | | | | | The main routine takes 8 args, 3 of which are almost the same for most uses. This in particular pushes it above the limit of 6 arguments passable through registers on amd64 making it impossible to tail call. This is a prerequisite for further cleanups. Tested by: pho Notes: svn path=/head/; revision=330400
* sys/kern: adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326271
* lockmgr: remove the ADAPTIVE_LOCKMGRS optionMateusz Guzik2017-11-171-189/+0
| | | | | | | | | | | The code was never enabled and is very heavy weight. A revamped adaptive spinning may show up at a later time. Discussed with: kib Notes: svn path=/head/; revision=325958
* lockmgr: implement fast pathMateusz Guzik2017-02-121-61/+244
| | | | | | | | | | | | | | | | | | | The main lockmgr routine takes 8 arguments which makes it impossible to tail-call it by the intermediate vop_stdlock/unlock routines. The routine itself starts with an if-forest and reads from the lock itself several times. This slows things down both single- and multi-threaded. With the patch single-threaded fstats go 4% up and multithreaded up to ~27%. Note that there is still a lot of room for improvement. Reviewed by: kib Tested by: pho Notes: svn path=/head/; revision=313683
* Microoptimize locking primitives by avoiding unnecessary atomic ops.Mateusz Guzik2016-06-011-3/+9
| | | | | | | | | | | | | Inline version of primitives do an atomic op and if it fails they fallback to actual primitives, which immediately retry the atomic op. The obvious optimisation is to check if the lock is free and only then proceed to do an atomic op. Reviewed by: jhb, vangyzen Notes: svn path=/head/; revision=301157
* sys/kern: spelling fixes in comments.Pedro F. Giffuni2016-04-291-3/+3
| | | | | | | No functional change. Notes: svn path=/head/; revision=298819
* Don't modify curthread->td_locks unless INVARIANTS is enabled.Mark Johnston2015-08-021-5/+1
| | | | | | | | | | | | | This field is only used in a KASSERT that verifies that no locks are held when returning to user mode. Moreover, the td_locks accounting is only correct when LOCK_DEBUG > 0, which is implied by INVARIANTS. Reviewed by: jhb MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D3205 Notes: svn path=/head/; revision=286166
* Revert for r277213:Hans Petter Selasky2015-01-221-4/+2
| | | | | | | | | | | | | | FreeBSD developers need more time to review patches in the surrounding areas like the TCP stack which are using MPSAFE callouts to restore distribution of callouts on multiple CPUs. Bump the __FreeBSD_version instead of reverting it. Suggested by: kmacy, adrian, glebius and kib Differential Revision: https://reviews.freebsd.org/D1438 Notes: svn path=/head/; revision=277528
* Major callout subsystem cleanup and rewrite:Hans Petter Selasky2015-01-151-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Close a migration race where callout_reset() failed to set the CALLOUT_ACTIVE flag. - Callout callback functions are now allowed to be protected by spinlocks. - Switching the callout CPU number cannot always be done on a per-callout basis. See the updated timeout(9) manual page for more information. - The timeout(9) manual page has been updated to reflect how all the functions inside the callout API are working. The manual page has been made function oriented to make it easier to deduce how each of the functions making up the callout API are working without having to first read the whole manual page. Group all functions into a handful of sections which should give a quick top-level overview when the different functions should be used. - The CALLOUT_SHAREDLOCK flag and its functionality has been removed to reduce the complexity in the callout code and to avoid problems about atomically stopping callouts via callout_stop(). If someone needs it, it can be re-added. From my quick grep there are no CALLOUT_SHAREDLOCK clients in the kernel. - A new callout API function named "callout_drain_async()" has been added. See the updated timeout(9) manual page for a complete description. - Update the callout clients in the "kern/" folder to use the callout API properly, like cv_timedwait(). Previously there was some custom sleepqueue code in the callout subsystem, which has been removed, because we now allow callouts to be protected by spinlocks. This allows us to tear down the callout like done with regular mutexes, and a "td_slpmutex" has been added to "struct thread" to atomically teardown the "td_slpcallout". Further the "TDF_TIMOFAIL" and "SWT_SLEEPQTIMO" states can now be completely removed. Currently they are marked as available and will be cleaned up in a follow up commit. - Bump the __FreeBSD_version to indicate kernel modules need recompilation. - There has been several reports that this patch "seems to squash a serious bug leading to a callout timeout and panic". Kernel build testing: all architectures were built MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D1438 Sponsored by: Mellanox Technologies Reviewed by: jhb, adrian, sbruno and emaste Notes: svn path=/head/; revision=277213
* Do not try to dereference thread pointer when the value is not a pointer.Konstantin Belousov2014-11-131-3/+8
| | | | | | | | | Reported and tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=274474
* Add a new thread state "spinning" to schedgraph and add tracepoints at theJohn Baldwin2014-11-041-0/+20
| | | | | | | start and stop of spinning waits in lock primitives. Notes: svn path=/head/; revision=274092
* Followup to r273966. Fix the build with ADAPTIVE_LOCKMGRS kernel option.Konstantin Belousov2014-11-021-1/+1
| | | | | | | | | | | Note that the option is currently not used in any in-tree kernel configs, including LINTs. Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Notes: svn path=/head/; revision=273986
* Fix two issues with lockmgr(9) LK_CAN_SHARE() test, which determinesKonstantin Belousov2014-11-021-6/+7
| | | | | | | | | | | | | | | | | | | | | | | | | whether the shared request for already shared-locked lock could be granted. Both problems result in the exclusive locker starvation. The concurrent exclusive request is indicated by either LK_EXCLUSIVE_WAITERS or LK_EXCLUSIVE_SPINNERS flags. The reverse condition, i.e. no exclusive waiters, must check that both flags are cleared. Add a flag LK_NODDLKTREAT for shared lock request to indicate that current thread guarantees that it does not own the lock in shared mode. This turns back the exclusive lock starvation avoidance code; see man page update for detailed description. Use LK_NODDLKTREAT when doing lookup(9). Reported and tested by: pho No objections from: attilio Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Notes: svn path=/head/; revision=273966
* Add function and wrapper to switch lockmgr and vnode lock back toKonstantin Belousov2014-08-291-0/+8
| | | | | | | | | | | auto-promotion of shared to exclusive. Tested by: hrs, pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=270795
* - For kernel compiled only with KDTRACE_HOOKS and not any lock debuggingAttilio Rao2013-11-251-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | option, unbreak the lock tracing release semantic by embedding calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined version of the releasing functions for mutex, rwlock and sxlock. Failing to do so skips the lockstat_probe_func invokation for unlocking. - As part of the LOCKSTAT support is inlined in mutex operation, for kernel compiled without lock debugging options, potentially every consumer must be compiled including opt_kdtrace.h. Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES is linked there and it is only used as a compile-time stub [0]. [0] immediately shows some new bug as DTRACE-derived support for debug in sfxge is broken and it was never really tested. As it was not including correctly opt_kdtrace.h before it was never enabled so it was kept broken for a while. Fix this by using a protection stub, leaving sfxge driver authors the responsibility for fixing it appropriately [1]. Sponsored by: EMC / Isilon storage division Discussed with: rstone [0] Reported by: rstone [1] Discussed with: philip Notes: svn path=/head/; revision=258541
* Add LK_TRYUPGRADE operation for lockmgr(9), which attempts toKonstantin Belousov2013-09-291-0/+13
| | | | | | | | | | | | | | atomically upgrade shared lock to exclusive. On failure, error is returned and lock is not dropped in the process. Tested by: pho (previous version) No objections from: attilio Sponsored by: The FreeBSD Foundation MFC after: 1 week Approved by: re (glebius) Notes: svn path=/head/; revision=255940
* Fix lc_lock/lc_unlock() support for rmlocks held in shared mode. WithDavide Italiano2013-09-201-4/+4
| | | | | | | | | | | | | | | | | | current lock classes KPI it was really difficult because there was no way to pass an rmtracker object to the lock/unlock routines. In order to accomplish the task, modify the aforementioned functions so that they can return (or pass as argument) an uinptr_t, which is in the rm case used to hold a pointer to struct rm_priotracker for current thread. As an added bonus, this fixes rm_sleep() in the rm shared case, which right now can communicate priotracker structure between lc_unlock()/lc_lock(). Suggested by: jhb Reviewed by: jhb Approved by: re (delphij) Notes: svn path=/head/; revision=255745
* A few mostly cosmetic nits to aid in debugging:John Baldwin2013-06-251-3/+3
| | | | | | | | | | | - Call lock_init() first before setting any lock_object fields in lock init routines. This way if the machine panics due to a duplicate init the lock's original state is preserved. - Somewhat similarly, don't decrement td_locks and td_slocks until after an unlock operation has completed successfully. Notes: svn path=/head/; revision=252212
* - Fix a couple of inverted panic messages for shared/exclusive mismatchesJohn Baldwin2013-06-031-3/+5
| | | | | | | | | of a lock within a single thread. - Fix handling of interlocks in WITNESS by properly requiring the interlock to be held exactly once if it is specified. Notes: svn path=/head/; revision=251326
* Add option WITNESS_NO_VNODE to suppress printing LORs between VNODEMarcel Moolenaar2013-05-091-0/+2
| | | | | | | | | | | | locks. To support this, VNODE locks are created with the LK_IS_VNODE flag. This flag is propagated down using the LO_IS_VNODE flag. Note that WITNESS still records the LOR. Only the printing and the optional entering into the kernel debugger is bypassed with the WITNESS_NO_VNODE option. Notes: svn path=/head/; revision=250411
* lockmgr: unlock interlock (if requested) when dealing with upgrade/downgradeMateusz Guzik2013-01-061-0/+2
| | | | | | | | | | | requests for LK_NOSHARE locks, just like for shared locks. PR: kern/174969 Reviewed by: attilio MFC after: 1 week Notes: svn path=/head/; revision=245113
* Fixup r240424: On entering KDB backends, the hijacked thread to runAttilio Rao2012-12-221-1/+2
| | | | | | | | | | | | | | | | interrupt context can still be idlethread. At that point, without the panic condition, it can still happen that idlethread then will try to acquire some locks to carry on some operations. Skip the idlethread check on block/sleep lock operations when KDB is active. Reported by: jh Tested by: jh MFC after: 1 week Notes: svn path=/head/; revision=244582
* Check for lockmgr recursion in case of disown and downgrade and panicAttilio Rao2012-12-051-2/+19
| | | | | | | | | | also in !debugging kernel rather than having "undefined" behaviour. Tested by: avg MFC after: 1 week Notes: svn path=/head/; revision=243900
* Improve check coverage about idle threads.Attilio Rao2012-09-121-0/+3
| | | | | | | | | | | | | | | Idle threads are not allowed to acquire any lock but spinlocks. Deny any attempt to do so by panicing at the locking operation when INVARIANTS is on. Then, remove the check on blocking on a turnstile. The check in sleepqueues is left because they are not allowed to use tsleep() either which could happen still. Reviewed by: bde, jhb, kib MFC after: 1 week Notes: svn path=/head/; revision=240424
* Add software PMC support.Fabien Thomas2012-03-281-0/+15
| | | | | | | | | | | | | | | | New kernel events can be added at various location for sampling or counting. This will for example allow easy system profiling whatever the processor is with known tools like pmcstat(8). Simultaneous usage of software PMC and hardware PMC is possible, for example looking at the lock acquire failure, page fault while sampling on instructions. Sponsored by: NETASQ MFC after: 1 month Notes: svn path=/head/; revision=233628
generated by cgit v1.3 (git 2.53.0) at 2026-05-28 04:58:01 +0000