summaryrefslogtreecommitdiff
path: root/sys/kern/kern_mutex.c
Commit message (Collapse)AuthorAgeFilesLines
...
* Revert previous commit and add myself to the list of people who shouldPoul-Henning Kamp2009-09-081-6/+5
| | | | | | | know better than to commit with a cat in the area. Notes: svn path=/head/; revision=196970
* Add necessary include.Poul-Henning Kamp2009-09-081-5/+6
| | | | Notes: svn path=/head/; revision=196969
* * Change the scope of the ASSERT_ATOMIC_LOAD() from a generic check toAttilio Rao2009-08-171-2/+3
| | | | | | | | | | | | | | | | | | a pointer-fetching specific operation check. Consequently, rename the operation ASSERT_ATOMIC_LOAD_PTR(). * Fix the implementation of ASSERT_ATOMIC_LOAD_PTR() by checking directly alignment on the word boundry, for all the given specific architectures. That's a bit too strict for some common case, but it assures safety. * Add a comment explaining the scope of the macro * Add a new stub in the lockmgr specific implementation Tested by: marcel (initial version), marius Reviewed by: rwatson, jhb (comment specific review) Approved by: re (kib) Notes: svn path=/head/; revision=196334
* Add a new macro to test that a variable could be loaded atomically.Bjoern A. Zeeb2009-08-141-0/+2
| | | | | | | | | | | | | | | | | | | | | | Check that the given variable is at most uintptr_t in size and that it is aligned. Note: ASSERT_ATOMIC_LOAD() uses ALIGN() to check for adequate alignment -- however, the function of ALIGN() is to guarantee alignment, and therefore may lead to stronger alignment enforcement than necessary for types that are smaller than sizeof(uintptr_t). Add checks to mtx, rw and sx locks init functions to detect possible breakage. This was used during debugging of the problem fixed with r196118 where a pointer was on an un-aligned address in the dpcpu area. In collaboration with: rwatson Reviewed by: rwatson Approved by: re (kib) Notes: svn path=/head/; revision=196226
* Remove extra cpu_spinwait() invocations. This should really only be usedJohn Baldwin2009-05-291-3/+0
| | | | | | | | | | in tight spin loops, not in these edge cases where we restart a much larger loop only a few times. Reviewed by: attilio Notes: svn path=/head/; revision=193037
* Tweak a few comments on adaptive spinning.John Baldwin2009-05-291-2/+5
| | | | Notes: svn path=/head/; revision=193035
* Add the OpenSolaris dtrace lockstat provider. The lockstat providerStacey Son2009-05-261-9/+70
| | | | | | | | | | | | | adds probes for mutexes, reader/writer and shared/exclusive locks to gather contention statistics and other locking information for dtrace scripts, the lockstat(1M) command and other potential consumers. Reviewed by: attilio jhb jb Approved by: gnn (mentor) Notes: svn path=/head/; revision=192853
* Remove an obsolete assertion. We always wake up all waiters when unlockingJohn Baldwin2009-05-201-2/+0
| | | | | | | a mutex and never set the lock cookie == MTX_CONTESTED. Notes: svn path=/head/; revision=192456
* - Wrap lock profiling state variables in #ifdef LOCK_PROFILING blocks.Jeff Roberson2009-03-151-7/+17
| | | | Notes: svn path=/head/; revision=189846
* - When a mutex is destroyed while locked we need to inform lock profilingJeff Roberson2009-03-141-0/+1
| | | | | | | that it has been released. Notes: svn path=/head/; revision=189789
* Teach WITNESS about the interlocks used with lockmgr. This removes a bunchJohn Baldwin2008-09-101-3/+3
| | | | | | | | | | | of spurious witness warnings since lockmgr grew witness support. Before this, every time you passed an interlock to a lockmgr lock WITNESS treated it as a LOR. Reviewed by: attilio Notes: svn path=/head/; revision=182914
* Various whitespace fixes.John Baldwin2008-09-101-9/+9
| | | | Notes: svn path=/head/; revision=182909
* Add KASSERT()'s to catch attempts to recurse on spin mutexes that aren'tJohn Baldwin2008-02-131-1/+9
| | | | | | | | | marked recursable either via mtx_lock_spin() or thread_lock(). MFC after: 1 week Notes: svn path=/head/; revision=176260
* Add a couple of assertions and KTR logging to thread_lock_flags() toJohn Baldwin2008-02-131-1/+7
| | | | | | | | | match mtx_lock_spin_flags(). MFC after: 1 week Notes: svn path=/head/; revision=176257
* - Re-implement lock profiling in such a way that it no longer breaksJeff Roberson2007-12-151-20/+6
| | | | | | | | | | | | | | | | | | | | | | | | | the ABI when enabled. There is no longer an embedded lock_profile_object in each lock. Instead a list of lock_profile_objects is kept per-thread for each lock it may own. The cnt_hold statistic is now always 0 to facilitate this. - Support shared locking by tracking individual lock instances and statistics in the per-thread per-instance lock_profile_object. - Make the lock profiling hash table a per-cpu singly linked list with a per-cpu static lock_prof allocator. This removes the need for an array of spinlocks and reduces cache contention between cores. - Use a seperate hash for spinlocks and other locks so that only a critical_enter() is required and not a spinlock_enter() to modify the per-cpu tables. - Count time spent spinning in the lock statistics. - Remove the LOCK_PROFILE_SHARED option as it is always supported now. - Specifically drop and release the scheduler locks in both schedulers since we track owners now. In collaboration with: Kip Macy Sponsored by: Nokia Notes: svn path=/head/; revision=174629
* Make ADAPTIVE_GIANT as the default in the kernel and remove the option.Attilio Rao2007-11-281-8/+0
| | | | | | | | | | | | | Currently, Giant is not too much contented so that it is ok to treact it like any other mutexes. Please don't forget to update your own custom config kernel files. Approved by: cognet, marcel (maintainers of arches where option is not enabled at the moment) Notes: svn path=/head/; revision=174005
* Simplify the adaptive spinning algorithm in rwlock and mutex:Attilio Rao2007-11-261-29/+41
| | | | | | | | | | | | | | | | | | | | | | | | currently, before to spin the turnstile spinlock is acquired and the waiters flag is set. This is not strictly necessary, so just spin before to acquire the spinlock and to set the flags. This will simplify a lot other functions too, as now we have the waiters flag set only if there are actually waiters. This should make wakeup/sleeping couplet faster under intensive mutex workload. This also fixes a bug in rw_try_upgrade() in the adaptive case, where turnstile_lookup() will recurse on the ts_lock lock that will never be really released [1]. [1] Reported by: jeff with Nokia help Tested by: pho, kris (earlier, bugged version of rwlock part) Discussed with: jhb [2], jeff MFC after: 1 week [2] John had a similar patch about 6.x and/or 7.x about mutexes probabilly Notes: svn path=/head/; revision=173960
* Expand lock class with the "virtual" function lc_assert which will offerAttilio Rao2007-11-181-0/+10
| | | | | | | | | | | | an unified way for all the lock primitives to express lock assertions. Currenty, lockmgrs and rmlocks don't have assertions, so just panic in that case. This will be a base for more callout improvements. Ok'ed by: jhb, jeff Notes: svn path=/head/; revision=173733
* generally we are interested in what thread did something asJulian Elischer2007-11-141-1/+1
| | | | | | | | | opposed to what process. Since threads by default have teh name of the process unless over-written with more useful information, just print the thread name instead. Notes: svn path=/head/; revision=173600
* - Remove the global definition of sched_lock in mutex.h to breakJeff Roberson2007-07-181-2/+0
| | | | | | | | | | | | | | new code and third party modules which try to depend on it. - Initialize sched_lock in sched_4bsd.c. - Declare sched_lock in sparc64 pmap.c and assert that we're compiling with SCHED_4BSD to prevent accidental crashes from running ULE. This is the sole remaining file outside of the scheduler that uses the global sched_lock. Approved by: re Notes: svn path=/head/; revision=171488
* - Add the proper lock profiling calls to _thread_lock().Jeff Roberson2007-07-181-2/+8
| | | | | | | | Obtained from: kipmacy Approved by: re Notes: svn path=/head/; revision=171487
* Propagate volatile qualifier to make gcc4.2 happy.Matt Jacob2007-06-091-1/+1
| | | | Notes: svn path=/head/; revision=170465
* Remove the MUTEX_WAKE_ALL option and make it the default behaviour for ourAttilio Rao2007-06-081-37/+0
| | | | | | | | | mutexes. Currently we alredy force MUTEX_WAKE_ALL beacause of some problems with the !MUTEX_WAKE_ALL case (unavioidable priority inversion). Notes: svn path=/head/; revision=170441
* - Placing the 'volatile' on the right side of the * in the td_lockJeff Roberson2007-06-061-3/+3
| | | | | | | | | declaration removes the need for __DEVOLATILE(). Pointed out by: tegge Notes: svn path=/head/; revision=170358
* Fix a problem with not-preemptive kernels caming from mis-merging ofAttilio Rao2007-06-051-47/+0
| | | | | | | | | | existing code with the new thread_lock patch. This also cleans up a bit unlock operation for mutexes. Approved by: jhb, jeff(mentor) Notes: svn path=/head/; revision=170339
* Restore non-SMP build.Konstantin Belousov2007-06-051-1/+2
| | | | | | | Reviewed by: attilio Notes: svn path=/head/; revision=170327
* Commit 3/14 of sched_lock decomposition.Jeff Roberson2007-06-041-27/+122
| | | | | | | | | | | | | | | | | | | | - Add a per-turnstile spinlock to solve potential priority propagation deadlocks that are possible with thread_lock(). - The turnstile lock order is defined as the exact opposite of the lock order used with the sleep locks they represent. This allows us to walk in reverse order in priority_propagate and this is the only place we wish to multiply acquire turnstile locks. - Use the turnstile_chain lock to protect assigning mutexes to turnstiles. - Change the turnstile interface to pass back turnstile pointers to the consumers. This allows us to reduce some locking and makes it easier to cancel turnstile assignment while the turnstile chain lock is held. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each) Notes: svn path=/head/; revision=170295
* Move lock_profile_object_{init,destroy}() into lock_{init,destroy}().John Baldwin2007-05-181-2/+0
| | | | Notes: svn path=/head/; revision=169675
* Teach 'show lock' to properly handle a destroyed mutex.John Baldwin2007-05-081-1/+5
| | | | Notes: svn path=/head/; revision=169393
* move lock_profile calls out of the macros and into kern_mutex.cKip Macy2007-04-031-9/+17
| | | | | | | add check for mtx_recurse == 0 when releasing sleep lock Notes: svn path=/head/; revision=168329
* - Simplify the #ifdef's for adaptive mutexes and rwlocks by conditionallyJohn Baldwin2007-03-221-4/+8
| | | | | | | | defining a macro earlier in the file. - Add NO_ADAPTIVE_RWLOCKS option to disable adaptive spinning for rwlocks. Notes: svn path=/head/; revision=167801
* Rename the 'mtx_object', 'rw_object', and 'sx_object' members of mutexes,John Baldwin2007-03-211-68/+68
| | | | | | | rwlocks, and sx locks to 'lock_object'. Notes: svn path=/head/; revision=167787
* Add two new function pointers 'lc_lock' and 'lc_unlock' to lock classes.John Baldwin2007-03-091-0/+40
| | | | | | | | | | | | | | | | These functions are intended to be used to drop a lock and then reacquire it when doing an sleep such as msleep(9). Both functions accept a 'struct lock_object *' as their first parameter. The 'lc_unlock' function returns an integer that is then passed as the second paramter to the subsequent 'lc_lock' function. This can be used to communicate state. For example, sx locks and rwlocks use this to indicate if the lock was share/read locked vs exclusive/write locked. Currently, spin mutexes and lockmgr locks do not provide working lc_lock and lc_unlock functions. Notes: svn path=/head/; revision=167368
* Use C99-style struct member initialization for lock classes.John Baldwin2007-03-091-6/+6
| | | | Notes: svn path=/head/; revision=167365
* lock stats updates need to be protected by the lockKip Macy2007-03-021-20/+5
| | | | Notes: svn path=/head/; revision=167163
* Evidently I've overestimated gcc's ability to peak inside inline functionsKip Macy2007-03-011-4/+8
| | | | | | | | and optimize away unused stack values. The 48 bytes that the lock_profile_object adds to the stack evidently has a measurable performance impact on certain workloads. Notes: svn path=/head/; revision=167136
* Further improvements to LOCK_PROFILING:Kip Macy2007-02-271-3/+14
| | | | | | | | | | | | | | | - Fix missing initialization in kern_rwlock.c causing bogus times to be collected - Move updates to the lock hash to after the lock is released for spin mutexes, sleep mutexes, and sx locks - Add new kernel build option LOCK_PROFILE_FAST - only update lock profiling statistics when an acquisition is contended. This reduces the overhead of LOCK_PROFILING to increasing system time by 20%-25% which on "make -j8 kernel-toolchain" on a dual woodcrest is unmeasurable in terms of wall-clock time. Contrast this to enabling lock profiling without LOCK_PROFILE_FAST and I see a 5x-6x slowdown in wall-clock time. Notes: svn path=/head/; revision=167054
* general LOCK_PROFILING cleanupKip Macy2007-02-261-21/+8
| | | | | | | | | | | | | | | - only collect timestamps when a lock is contested - this reduces the overhead of collecting profiles from 20x to 5x - remove unused function from subr_lock.c - generalize cnt_hold and cnt_lock statistics to be kept for all locks - NOTE: rwlock profiling generates invalid statistics (and most likely always has) someone familiar with that should review Notes: svn path=/head/; revision=167012
* - Fix some gcc warnings in lock_profile.hKip Macy2006-12-161-6/+20
| | | | | | | | | | - add cnt_hold cnt_lock support for spin mutexes - make sure contested is initialized to zero to only bump contested when appropriate - move initialization function to kern_mutex.c to avoid cyclic dependency between mutex.h and lock_profile.h Notes: svn path=/head/; revision=165265
* track lock class name in a way that doesn't break WITNESSKip Macy2006-11-131-1/+1
| | | | Notes: svn path=/head/; revision=164246
* MUTEX_PROFILING has been generalized to LOCK_PROFILING. We now profileKip Macy2006-11-111-248/+30
| | | | | | | | | | | | | | wait (time waited to acquire) and hold times for *all* kernel locks. If the architecture has a system synchronized TSC, the profiling code will use that - thereby minimizing profiling overhead. Large chunks of profiling code have been moved out of line, the overhead measured on the T1 for when it is compiled in but not enabled is < 1%. Approved by: scottl (standing in for mentor rwatson) Reviewed by: des and jhb Notes: svn path=/head/; revision=164159
* - When spinning on a spin lock, if the debugger is active or we are in aJohn Baldwin2006-08-151-6/+12
| | | | | | | | | | panic, go ahead and do the longer DELAY(1) spin wait. - If we panic due to spinning too long, print out a few more details including the pointer to the mutex in question and the tid of the owning thread. Notes: svn path=/head/; revision=161336
* Adjust td_locks for non-spin mutexes, rwlocks, and sx locks so that it isJohn Baldwin2006-07-271-1/+7
| | | | | | | | | | | a count of all non-spin locks, not just lockmgr locks. This can give us a much cheaper way to see if we have any locks held (such as when returning to userland via userret()) without requiring WITNESS. MFC after: 1 week Notes: svn path=/head/; revision=160771
* Write a magic value into mtx_lock when destroying a mutex that will forceJohn Baldwin2006-07-271-0/+11
| | | | | | | | | | | | | all other mtx_lock() operations to block. Previously, when the mutex was destroyed, it would still have a valid value in mtx_lock(): either the unowned cookie, which would allow a subsequent mtx_lock() to succeed, or a pointer to the thread who destroyed the mutex if the mutex was locked when it was destroyed. MFC after: 3 days Notes: svn path=/head/; revision=160766
* Bah, fix fat finger in last. Invert the ~ on MTX_FLAGMASK as it'sJohn Baldwin2006-06-031-2/+2
| | | | | | | | | | | non-intuitive for the ~ to be built into the mask. All the users now explicitly ~ the mask. In addition, add MTX_UNOWNED to the mask even though it technically isn't a flag. This should unbreak mtx_owner(). Quickly spotted by: kris Notes: svn path=/head/; revision=159208
* Simplify mtx_owner() so it only reads m->mtx_lock once.John Baldwin2006-06-031-2/+1
| | | | Notes: svn path=/head/; revision=159204
* Style fix to be more like _mtx_lock_sleep(): use 'while (!foo) { ... }'John Baldwin2006-06-031-3/+1
| | | | | | | instead of 'for (;;) { if (foo) break; ... }'. Notes: svn path=/head/; revision=159203
* Since DELAY() was moved, most <machine/clock.h> #includes have beenPoul-Henning Kamp2006-05-161-1/+0
| | | | | | | unnecessary. Notes: svn path=/head/; revision=158651
* Remove various bits of conditional Alpha code and fixup a few comments.John Baldwin2006-05-121-6/+0
| | | | Notes: svn path=/head/; revision=158471
* Mark the thread pointer used during an adaptive spin volatile so that theJohn Baldwin2006-04-141-1/+1
| | | | | | | | | compiler doesn't decide to cache td_state. Cachine the state would cause the spinning thread to not notice when the owning thread stopped executing (if it was preempted for example) which could result in livelock. Notes: svn path=/head/; revision=157763
generated by cgit v1.2.3 (git 2.25.1) at 2025-08-24 21:28:25 +0000