summaryrefslogtreecommitdiff
path: root/sys/kern/kern_umtx.c
Commit message (Collapse)AuthorAgeFilesLines
* Make UMTX_OP_WAIT_UINT actually wait for an unsigned integer on 64-bitsDavid Xu2009-04-131-1/+1
| | | | | | | | | machine. MFC after: 1 week Notes: svn path=/head/; revision=190987
* 1) Check NULL pointer before calling umtx_pi_adjust_locked(), this avoidsDavid Xu2009-03-131-24/+26
| | | | | | | | | a PANIC. 2) Rework locking for POSIX priority-mutex, this fixes a race where a thread may wait there forever even if the mutex is unlocked. Notes: svn path=/head/; revision=189756
* Add two commands to _umtx_op system call to allow a simple mutex to beDavid Xu2008-06-241-35/+142
| | | | | | | | | | | | | | | | | locked and unlocked completely in userland. by locking and unlocking mutex in userland, it reduces the total time a mutex is locked by a thread, in some application code, a mutex only protects a small piece of code, the code's execution time is less than a simple system call, if a lock contention happens, however in current implemenation, the lock holder has to extend its locking time and enter kernel to unlock it, the change avoids this disadvantage, it first sets mutex to free state and then enters kernel and wake one waiter up. This improves performance dramatically in some sysbench mutex tests. Tested by: kris Sounds great: jeff Notes: svn path=/head/; revision=179970
* Use a seperated hash table for mutex and rwlock, avoid wasting some timeDavid Xu2008-05-301-17/+21
| | | | | | | on walking through idle threads sleeping on condition variables. Notes: svn path=/head/; revision=179421
* Introduce command UMTX_OP_WAIT_UINT_PRIVATE and UMTX_OP_WAKE_PRIVATEDavid Xu2008-04-291-12/+62
| | | | | | | | to allow userland to specify that an address is not shared by multiple processes. Notes: svn path=/head/; revision=178646
* let umtxq_busy() only spin on mp machine. make function nameDavid Xu2008-04-031-10/+14
| | | | | | | do_rwlock_unlock to be consistent with others. Notes: svn path=/head/; revision=177880
* Fix compiling problem for amd64.David Xu2008-04-021-2/+2
| | | | Notes: svn path=/head/; revision=177852
* Er, don't restart a timeout version.David Xu2008-04-021-2/+4
| | | | Notes: svn path=/head/; revision=177849
* Introduce kernel based userland rwlock. Each umtx chain now has two lists,David Xu2008-04-021-45/+487
| | | | | | | | | | one for readers and one for writers, other types of synchronization object just use first list. Asked by: jeff Notes: svn path=/head/; revision=177848
* Check NULL pointer.David Xu2007-12-171-1/+10
| | | | Notes: svn path=/head/; revision=174707
* Add missing changes for fixing LOR of umtx lock and thread lock, followDavid Xu2007-12-171-11/+24
| | | | | | | | | | the committing of files: kern_resource.c revision 1.181 sched_4bsd.c revision 1.111 sched_ule.c revision 1.218 Notes: svn path=/head/; revision=174701
* Add function UMTX_OP_WAIT_UINT, the function causes thread to wait forDavid Xu2007-11-211-2/+24
| | | | | | | an integer to be changed. Notes: svn path=/head/; revision=173800
* Backout experimental adaptive-spin umtx code.David Xu2007-06-061-67/+0
| | | | Notes: svn path=/head/; revision=170368
* Commit 8/14 of sched_lock decomposition.Jeff Roberson2007-06-041-33/+52
| | | | | | | | | | | | | - Use a global umtx spinlock to protect the sleep queues now that there is no global scheduler lock. - Use thread_lock() to protect thread state. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each) Notes: svn path=/head/; revision=170300
* Further system call comment cleanup:Robert Watson2007-03-051-1/+0
| | | | | | | | | | | | | - Remove also "MP SAFE" after prior "MPSAFE" pass. (suggested by bde) - Remove extra blank lines in some cases. - Add extra blank lines in some cases. - Remove no-op comments consisting solely of the function name, the word "syscall", or the system call name. - Add punctuation. - Re-wrap some comments. Notes: svn path=/head/; revision=167232
* Add a lwpid field into per-cpu structure, the lwpid represents currentDavid Xu2006-12-201-0/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | running thread's id on each cpu. This allow us to add in-kernel adaptive spin for user level mutex. While spinning in user space is possible, without correct thread running state exported from kernel, it hardly can be implemented efficiently without wasting cpu cycles, however exporting thread running state unlikely will be implemented soon as it has to design and stablize interfaces. This implementation is transparent to user space, it can be disabled dynamically. With this change, mutex ping-pong program's performance is improved massively on SMP machine. performance of mysql super-smack select benchmark is increased about 7% on Intel dual dual-core2 Xeon machine, it indicates on systems which have bunch of cpus and system-call overhead is low (athlon64, opteron, and core-2 are known to be fast), the adaptive spin does help performance. Added sysctls: kern.threads.umtx_dflt_spins if the sysctl value is non-zero, a zero umutex.m_spincount will cause the sysctl value to be used a spin cycle count. kern.threads.umtx_max_spins the sysctl sets upper limit of spin cycle count. Tested on: Athlon64 X2 3800+, Dual Xeon 5130 Notes: svn path=/head/; revision=165369
* Threading cleanup.. part 2 of several.Julian Elischer2006-12-061-6/+0
| | | | | | | | | | | | | | | | | | | | | | | | | Make part of John Birrell's KSE patch permanent.. Specifically, remove: Any reference of the ksegrp structure. This feature was never fully utilised and made things overly complicated. All code in the scheduler that tried to make threaded programs fair to unthreaded programs. Libpthread processes will already do this to some extent and libthr processes already disable it. Also: Since this makes such a big change to the scheduler(s), take the opportunity to rename some structures and elements that had to be moved anyhow. This makes the code a lot more readable. The ULE scheduler compiles again but I have no idea if it works. The 4bsd scheduler still reqires a little cleaning and some functions that now do ALMOST nothing will go away, but I thought I'd do that as a separate commit. Tested by David Xu, and Dan Eischen using libthr and libpthread. Notes: svn path=/head/; revision=164936
* if a thread blocked on userland condition variable isDavid Xu2006-12-041-5/+10
| | | | | | | | | | | | | pthread_cancel()ed, it is expected that the thread will not consume a pthread_cond_signal(), therefor, we use thr_wake() to mark a flag, the flag tells a thread calling do_cv_wait() in umtx code to not block on a condition variable. Thread library is expected that once a thread detected itself is in pthread_cond_wait, it will call the thr_wake() for itself in its SIGCANCEL handler. Notes: svn path=/head/; revision=164876
* Introduce userspace condition variable, since we have already POSIXDavid Xu2006-12-031-4/+200
| | | | | | | | | priority mutex implemented, it is the time to introduce this stuff, now we can use umutex and ucond together to implement pthread's condition wait/signal. Notes: svn path=/head/; revision=164839
* Sweep kernel replacing suser(9) calls with priv(9) calls, assigningRobert Watson2006-11-061-2/+3
| | | | | | | | | | | | | | | | specific privilege names to a broad range of privileges. These may require some future tweaking. Sponsored by: nCircle Network Security, Inc. Obtained from: TrustedBSD Project Discussed on: arch@ Reviewed (at least in part) by: mlaier, jmg, pjd, bde, ceri, Alex Lyashkov <umka at sevcity dot net>, Skip Ford <skip dot ford at verizon dot net>, Antoine Brodin <antoine dot brodin at laposte dot net> Notes: svn path=/head/; revision=164033
* Make KSE a kernel option, turned on by default in all GENERICJohn Birrell2006-10-261-0/+6
| | | | | | | | | | kernel configs except sun4v (which doesn't process signals properly with KSE). Reviewed by: davidxu@ Notes: svn path=/head/; revision=163709
* Optimize umtx_lock_pi() a bit by moving some heavy code out of the loop,David Xu2006-10-261-32/+27
| | | | | | | make a fast path when a umtx_pi can be allocated without being blocked. Notes: svn path=/head/; revision=163697
* In order to eliminate a branch, convert opcode to unsigned integer.David Xu2006-10-251-2/+2
| | | | Notes: svn path=/head/; revision=163678
* Eliminate an unnecessary `if' statement.David Xu2006-10-251-1/+2
| | | | Notes: svn path=/head/; revision=163677
* o Add keyword volatile for user mutex owner field.David Xu2006-10-171-29/+23
| | | | | | | | | o Fix type consistent problem by using type long for old umtx and wait channel. o Rename casuptr to casuword. Notes: svn path=/head/; revision=163449
* Implement 32bit umtx_lock and umtx_unlock system calls, these two systemDavid Xu2006-10-061-0/+14
| | | | | | | | | | calls are not used by libthr in RELENG_6 and HEAD, it is only used by the libthr in RELENG-5, the _umtx_op system call can do more incremental dirty works than these two system calls without having to introduce new system calls or throw away old system calls when things are going on. Notes: svn path=/head/; revision=163046
* Fix umtx command order error for freebsd 32bit.David Xu2006-09-221-1/+1
| | | | Notes: svn path=/head/; revision=162550
* Add umtx support for 32bit process on AMD64 machine.David Xu2006-09-221-82/+440
| | | | Notes: svn path=/head/; revision=162536
* Merge all code of do_lock_normal, do_lock_pi and do_lock_pp intoDavid Xu2006-09-051-120/+52
| | | | | | | function do_lock_umutex. Notes: svn path=/head/; revision=162030
* Check if it is root user in do_unlock_pp.David Xu2006-09-031-2/+4
| | | | Notes: svn path=/head/; revision=161926
* Make sure we get new m_owner value if we can not unlock it inDavid Xu2006-09-021-8/+12
| | | | | | | uncontested case. Reorder statements in do_unlock_umutex. Notes: svn path=/head/; revision=161855
* Reorder some statments. Fix typo and remove stale comments.David Xu2006-08-301-20/+16
| | | | Notes: svn path=/head/; revision=161742
* Update comments about interrupted mutex locking.David Xu2006-08-281-12/+5
| | | | Notes: svn path=/head/; revision=161684
* This is initial version of POSIX priority mutex support, a new userlandDavid Xu2006-08-281-187/+1710
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | mutex structure is added as following: struct umutex { __lwpid_t m_owner; uint32_t m_flags; uint32_t m_ceilings[2]; uint32_t m_spare[4]; }; The m_owner represents owner thread, it is a thread id, in non-contested case, userland can simply use atomic_cmpset_int to lock the mutex, if the mutex is contested, high order bit will be set, and userland should do locking and unlocking via kernel syscall. Flag UMUTEX_PRIO_INHERIT represents pthread's PTHREAD_PRIO_INHERIT mutex, which when contention happens, kernel should do priority propagating. Flag UMUTEX_PRIO_PROTECT indicates it is pthread's PTHREAD_PRIO_PROTECT mutex, userland should initialize m_owner to contested state UMUTEX_CONTESTED, then atomic_cmpset_int will be failure and kernel syscall should be invoked to do locking, this becauses for such a mutex, kernel should always boost the thread's priority before it can lock the mutex, m_ceilings is used by PTHREAD_PRIO_PROTECT mutex, the first element is used to boost thread's priority when it locked the mutex, second element is used when the mutex is unlocked, the PTHREAD_PRIO_PROTECT mutex's link list is kept in userland, the m_ceiling[1] is managed by thread library so kernel needn't allocate memory to keep the link list, when such a mutex is unlocked, kernel reset m_owner to UMUTEX_CONTESTED. Flag USYNC_PROCESS_SHARED indicate if the synchronization object is process shared, if the flag is not set, it saves a vm_map_lookup() call. The umtx chain is still used as a sleep queue, when a thread is blocked on PTHREAD_PRIO_INHERIT mutex, a umtx_pi is allocated to support priority propagating, it is dynamically allocated and reference count is used, it is not optimized but works well in my tests, while the umtx chain has its own locking protocol, the priority propagating protocol are all protected by sched_lock because priority propagating function is called with sched_lock held from scheduler. No visible performance degradation is found which these changes. Some parameter names in _umtx_op syscall are renamed. Notes: svn path=/head/; revision=161678
* Add user priority loaning code to support priority propagation forDavid Xu2006-08-251-0/+5
| | | | | | | | 1:1 threading's POSIX priority mutexes, the code is no-op unless priority-aware umtx code is committed. Notes: svn path=/head/; revision=161599
* Move flag TDF_UMTXQ into structure umtxq, this eliminates the requirementDavid Xu2006-05-181-15/+12
| | | | | | | of scheduler lock in some umtx code. Notes: svn path=/head/; revision=158718
* Use wakeup_one to avoid thundering herd.David Xu2006-05-091-6/+6
| | | | | | | Tested by: kris Notes: svn path=/head/; revision=158377
* Change msleep() and tsleep() to not alter the calling thread's priorityJohn Baldwin2006-04-171-6/+4
| | | | | | | | | | | | | | | if the specified priority is zero. This avoids a race where the calling thread could read a snapshot of it's current priority, then a different thread could change the first thread's priority, then the original thread would call sched_prio() inside msleep() undoing the change made by the second thread. I used a priority of zero as no thread that calls msleep() or tsleep() should be specifying a priority of zero anyway. The various places that passed 'curthread->td_priority' or some variant as the priority now pass 0. Notes: svn path=/head/; revision=157815
* Axe unused code.David Xu2006-02-041-96/+0
| | | | Notes: svn path=/head/; revision=155276
* do umtx_wake at userland thread exit address, so that others userlandDavid Xu2005-10-261-3/+3
| | | | | | | | | | | | | threads can wait for a thread to exit, and safely assume that the thread has left userland and is no longer using its userland stack, this is necessary for pthread_join when a thread is waiting for another thread to exit which has user customized stack, after pthread_join returns, the userland stack can be reused for other purposes, without this change, the joiner thread has to spin at the address to ensure the thread is really exited. Notes: svn path=/head/; revision=151692
* Allocate umtx_q from heap instead of stack, this avoidsDavid Xu2005-03-051-34/+46
| | | | | | | page fault panic in kernel under heavy swapping. Notes: svn path=/head/; revision=143149
* Revert my previous errno hack, that is certainly an issue,David Xu2005-01-181-2/+1
| | | | | | | | | | | and always has been, but the system call itself returns errno in a register so the problem is really a function of libc, not the system call. Discussed with : Matthew Dillion <dillon@apollo.backplane.com> Notes: svn path=/head/; revision=140421
* make umtx timeout relative so userland can select different clock type,David Xu2005-01-141-46/+51
| | | | | | | | e.g, CLOCK_REALTIME or CLOCK_MONOTONIC. merge umtx_wait and umtx_timedwait into single function. Notes: svn path=/head/; revision=140245
* Comment out debugging printf which doesn't compile on amd64.Poul-Henning Kamp2005-01-121-0/+2
| | | | Notes: svn path=/head/; revision=140110
* Let _umtx_op directly return error code rather than from errno becauseDavid Xu2005-01-121-12/+23
| | | | | | | | | errno can be tampered potentially by nested signal handle. Now all error codes are returned in negative value, positive value are reserved for future expansion. Notes: svn path=/head/; revision=140102
* Break out of loop earlier if it is not timeout.David Xu2005-01-081-1/+1
| | | | Notes: svn path=/head/; revision=139899
* /* -> /*- for copyright notices, minor format tweaks as necessaryWarner Losh2005-01-061-1/+1
| | | | Notes: svn path=/head/; revision=139804
* Return ETIMEDOUT when thread is timeouted since POSIX threadDavid Xu2005-01-061-5/+7
| | | | | | | | APIs expect ETIMEDOUT not EAGAIN, this simplifies userland code a bit. Notes: svn path=/head/; revision=139751
* Make umtx_wait and umtx_wake more like linux futex does, it isDavid Xu2004-12-301-41/+9
| | | | | | | | | | more general than previous. It also lets me implement cancelable point in thread library. Also in theory, umtx_lock and umtx_unlock can be implemented by using umtx_wait and umtx_wake, all atomic operations can be done in userland without kernel's casuptr() function. Notes: svn path=/head/; revision=139427
* Make _umtx_op() as more general interface, the final parameter needn't beDavid Xu2004-12-251-4/+4
| | | | | | | timespec pointer, every parameter will be interpreted by its opcode. Notes: svn path=/head/; revision=139292
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-07 02:31:11 +0000