summaryrefslogtreecommitdiff
path: root/sys/kern/subr_sleepqueue.c
Commit message (Collapse)AuthorAgeFilesLines
* Add new msleep(9) flag PBDY that shall be specified together withKonstantin Belousov2009-07-141-4/+8
| | | | | | | | | | | | | | | PCATCH, to indicate that thread shall not be stopped upon receipt of SIGSTOP until it reaches the kernel->usermode boundary. Also change thread_single(SINGLE_NO_EXIT) to only stop threads at the user boundary unconditionally. Tested by: pho Reviewed by: jhb Approved by: re (kensmith) Notes: svn path=/head/; revision=195702
* Revision 184199 had not been fully reverted, add missing piece.David Xu2008-12-011-0/+4
| | | | | | | Reported by: phk Notes: svn path=/head/; revision=185502
* Revert rev 184216 and 184199, due to the way the thread_lock works,David Xu2008-11-051-8/+8
| | | | | | | | | it may cause a lockup. Noticed by: peter, jhb Notes: svn path=/head/; revision=184667
* Don't bother calling setrunnable() and clearing the sleeping flag inJohn Baldwin2008-11-041-9/+12
| | | | | | | sleepq_resume_thread() if the thread isn't asleep. Notes: svn path=/head/; revision=184653
* partly revert revision 184199, because TDF_NEEDSIGCHK is persitentDavid Xu2008-10-241-10/+5
| | | | | | | | | | when thread is in kernel mode, it can cause dead loop, now unlock process lock after acquired sleep queue lock and thread lock to avoid the problem. This means TDF_NEEDSIGCHK and TDF_NEEDSUSPCHK must be set with process lock and thread lock being hold at same time. Notes: svn path=/head/; revision=184216
* Actually, for signal and thread suspension, extra process spin lock isDavid Xu2008-10-231-12/+13
| | | | | | | | | unnecessary, the normal process lock and thread lock are enough. The spin lock is still needed for process and thread exiting to mimic single sched_lock. Notes: svn path=/head/; revision=184199
* Make ddb command registration dynamic so modules can extendSam Leffler2008-09-151-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | the command set (only so long as the module is present): o add db_command_register and db_command_unregister to add and remove commands, respectively o replace linker sets with SYSINIT's (and SYSUINIT's) that register commands o expose 3 list heads: db_cmd_table, db_show_table, and db_show_all_table for registering top-level commands, show operands, and show all operands, respectively While here also: o sort command lists o add DB_ALIAS, DB_SHOW_ALIAS, and DB_SHOW_ALL_ALIAS to add aliases for existing commands o add "show all trace" as an alias for "show alltrace" o add "show all locks" as an alias for "show alllocks" Submitted by: Guillaume Ballet <gballet@gmail.com> (original version) Reviewed by: jhb MFC after: 1 month Notes: svn path=/head/; revision=183054
* Close a race in sleepq_broadcast() where the sleepq could be reused afterJohn Baldwin2008-09-081-3/+2
| | | | | | | | | | | | | | | it had been assigned to the last sleeping thread. That thread might have started running on another CPU and have reused that sleep queue. Fix it by just walking the thread queue using TAILQ_FOREACH_SAFE() rather than a while loop. PR: amd64/124200 Discovered by: tegge Tested by: benjsc MFC after: 1 week Notes: svn path=/head/; revision=182875
* If a thread that is swapped out is made runnable, then the setrunnable()John Baldwin2008-08-051-18/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | routine wakes up proc0 so that proc0 can swap the thread back in. Historically, this has been done by waking up proc0 directly from setrunnable() itself via a wakeup(). When waking up a sleeping thread that was swapped out (the usual case when waking proc0 since only sleeping threads are eligible to be swapped out), this resulted in a bit of recursion (e.g. wakeup() -> setrunnable() -> wakeup()). With sleep queues having separate locks in 6.x and later, this caused a spin lock LOR (sleepq lock -> sched_lock/thread lock -> sleepq lock). An attempt was made to fix this in 7.0 by making the proc0 wakeup use the ithread mechanism for doing the wakeup. However, this required grabbing proc0's thread lock to perform the wakeup. If proc0 was asleep elsewhere in the kernel (e.g. waiting for disk I/O), then this degenerated into the same LOR since the thread lock would be some other sleepq lock. Fix this by deferring the wakeup of the swapper until after the sleepq lock held by the upper layer has been locked. The setrunnable() routine now returns a boolean value to indicate whether or not proc0 needs to be woken up. The end result is that consumers of the sleepq API such as *sleep/wakeup, condition variables, sx locks, and lockmgr, have to wakeup proc0 if they get a non-zero return value from sleepq_abort(), sleepq_broadcast(), or sleepq_signal(). Discussed with: jeff Glanced at by: sam Tested by: Jurgen Weber jurgen - ish com au MFC after: 2 weeks Notes: svn path=/head/; revision=181334
* Really fix this.John Baldwin2008-07-281-2/+1
| | | | Notes: svn path=/head/; revision=180930
* Properly check if td_name is empty and if it is, print process name,Pawel Jakub Dawidek2008-07-281-2/+2
| | | | | | | | | instead of empty thread name. Reviewed by: jhb Notes: svn path=/head/; revision=180927
* - Make SCHED_STATS more generic by adding a wrapper to create theJeff Roberson2008-04-171-4/+2
| | | | | | | | | | | | | | | | | | | | | variables and sysctl nodes. - In reset walk the children of kern_sched_stats and reset the counters via the oid_arg1 pointer. This allows us to add arbitrary counters to the tree and still reset them properly. - Define a set of switch types to be passed with flags to mi_switch(). These types are named SWT_*. These types correspond to SCHED_STATS counters and are automatically handled in this way. - Make the new SWT_ types more specific than the older switch stats. There are now stats for idle switches, remote idle wakeups, remote preemption ithreads idling, etc. - Add switch statistics for ULE's pickcpu algorithm. These stats include how much migration there is, how often affinity was successful, how often threads were migrated to the local cpu on wakeup, etc. Sponsored by: Nokia Notes: svn path=/head/; revision=178272
* - Convert two timeout users to the new callout_reset_curcpu() api.Jeff Roberson2008-04-021-1/+1
| | | | | | | Sponsored by: Nokia Notes: svn path=/head/; revision=177860
* - Add a new td flag TDF_NEEDSUSPCHK that is set whenever a thread needsJeff Roberson2008-03-211-1/+1
| | | | | | | | | | | | | | | | to enter thread_suspend_check(). - Set TDF_ASTPENDING along with TDF_NEEDSUSPCHK so we can move the thread_suspend_check() to ast() rather than userret(). - Check TDF_NEEDSUSPCHK in the sleepq_catch_signals() optimization so that we don't miss a suspend request. If this is set use the expensive signal path. - Set NEEDSUSPCHK when creating a new thread in thr in case the creating thread is due to be suspended as well but has not yet. Reviewed by: davidxu (Authored original patch) Notes: svn path=/head/; revision=177471
* - At the top of sleepq_catch_signals() lock the thread and check TDF_NEEDSIGCHKJeff Roberson2008-03-191-4/+12
| | | | | | | | | | before doing the very expensive cursig() and related locking. NEEDSIGCHK is updated whenever our signal mask change or when a signal is delivered and should be sufficient to avoid the more expensive tests. This eliminates another source of PROC_LOCK contention in multithreaded programs. Notes: svn path=/head/; revision=177375
* - Add a facility similar to LOCK_PROFILING under SLEEPQUEUE_PROFILING. KeepJeff Roberson2008-03-191-1/+159
| | | | | | | | | | | | | | | a simple (wmesg, count) tuple in a hash to keep track of how many times we sleep at each wait message. We hash on message and not channel. No line number information is given as typically wait messages are not used in more than one place. Identical strings defined at different addresses will show up with seperate counters. - Use debug.sleepq.enable to enable, .reset to reset, and .stats dumps stats. - Do an unsynchronized check in sleepq_switch() prior to switching before calling sleepq_profile() which uses a global lock to synchronize the hash. Only sleeps which actually cause a context switch are counted. Notes: svn path=/head/; revision=177372
* PR 117603Jeff Roberson2008-03-131-2/+5
| | | | | | | | | | | | | - Close a sleepqueue signal race by interlocking with the per-process spinlock. This was mistakenly omitted from the thread_lock patch and has been a race since. MFC After: 1 week PR: bin/117603 Reported by: Danny Braniss <danny@cs.huji.ac.il> Notes: svn path=/head/; revision=177132
* Remove kernel support for M:N threading.Jeff Roberson2008-03-121-10/+2
| | | | | | | | | | | While the KSE project was quite successful in bringing threading to FreeBSD, the M:N approach taken by the kse library was never developed to its full potential. Backwards compatibility will be provided via libmap.conf for dynamically linked binaries and static binaries will be broken. Notes: svn path=/head/; revision=177091
* - Pass the priority argument from *sleep() into sleepq and down intoJeff Roberson2008-03-121-27/+23
| | | | | | | | | | | | | | | | | | | | sched_sleep(). This removes extra thread_lock() acquisition and allows the scheduler to decide what to do with the static boost. - Change the priority arguments to cv_* to match sleepq/msleep/etc. where 0 means no priority change. Catch -1 in cv_broadcastpri() and convert it to 0 for now. - Set a flag when sleeping in a way that is compatible with swapping since direct priority comparisons are meaningless now. - Add a sysctl to ule, kern.sched.static_boost, that defaults to on which controls the boost behavior. Turning it off gives better performance in some workloads but needs more investigation. - While we're modifying sleepq, change signal and broadcast to both return with the lock held as the lock was held on enter. Reviewed by: jhb, peter Notes: svn path=/head/; revision=177085
* Mark sleepqueue chain spin mutexes are recursable since the sleepq codeJohn Baldwin2008-02-131-1/+1
| | | | | | | | | | now recurses on them in sleepq_broadcast() and sleepq_signal() when resuming threads that are fully asleep. MFC after: 1 week Notes: svn path=/head/; revision=176258
* - Add THREAD_LOCKPTR_ASSERT() to assert that the thread's lock points atJeff Roberson2008-02-071-1/+1
| | | | | | | | | | | | | the provided lock or &blocked_lock. The thread may be temporarily assigned to the blocked_lock by the scheduler so a direct comparison can not always be made. - Use THREAD_LOCKPTR_ASSERT() in the primary consumers of the scheduling interfaces. The schedulers themselves still use more explicit asserts. Sponsored by: Nokia Notes: svn path=/head/; revision=176078
* Fix a bug where a thread that hit the race where the sleep timeout firesJohn Baldwin2008-01-251-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | while the thread does not hold the thread lock would stop blocking for subsequent interruptible sleeps and would always immediately fail the sleep with EWOULDBLOCK instead (even sleeps that didn't have a timeout). Some background: - KSE has a facility for allowing one thread to interrupt another thread. During this process, the target thread aborts any interruptible sleeps much as if the target thread had a pending signal. Once the target thread acknowledges the interrupt, normal sleep handling resumes. KSE manages this via the TDF_INTERRUPTED flag. Specifically, it sets the flag when it sends an interrupt to another thread and clears it when the interrupt is acknowledged. (Note that this is purely a software interrupt sort of thing and has no relation to hardware interrupts or kernel interrupt threads.) - The old code for handling the sleep timeout race handled the race by setting the TDF_INTERRUPT flag and faking a KSE-style thread interrupt to the thread in the process of going to sleep. It probably should have just checked the TDF_TIMEOUT flag in sleepq_catch_signals() instead. - The bug was that the sleepq code would set TDF_INTERRUPT but it was never cleared. The sleepq code couldn't safely clear it in case there actually was a real KSE thread interrupt pending for the target thread (in fact, the sleepq timeout actually stomped on said pending interrupt). Thus, any future interruptible sleeps (*sleep(.. PCATCH ..) or cv_*wait_sig()) would see the TDF_INTERRUPT flag set and immediately fail with EWOULDBLOCK. The flag could be cleared if the thread belonged to a KSE process and another thread posted an interrupt to the original thread. However, in the more common case of a non-KSE process, the thread would pretty much stop sleeping. - Fix the bug by just setting TDF_TIMEOUT in the sleepq timeout code and not messing with TDF_INTERRUPT and td_intrval. With yesterday's fix to fix sleepq_switch() to check TDF_TIMEOUT, this is now sufficient. MFC after: 3 days Notes: svn path=/head/; revision=175664
* Fix a race in the sleepqueue timeout code that resulted in sleeps notJohn Baldwin2008-01-251-4/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | being properly cancelled by a timeout. In general there is a race between a the sleepq timeout handler firing while the thread is still in the process of going to sleep. In 6.x with sched_lock, the race was largely protected by sched_lock. The only place it was "exposed" and had to be handled was while checking for any pending signals in sleepq_catch_signals(). With the thread lock changes, the thread lock is dropped in between sleepq_add() and sleepq_*wait*() opening up a new window for this race. Thus, if the timeout fired while the sleeping thread was in between sleepq_add() and sleepq_*wait*(), the thread would be marked as timed out, but the thread would not be dequeued and sleepq_switch() would still block the thread until it was awakened via some other means. In the case of pause(9) where there is no other wakeup, the thread would never be awakened. Fix this by teaching sleepq_switch() to check if the thread has had its sleep canceled before blocking by checking the TDF_TIMEOUT flag and aborting the sleep and dequeueing the thread if it is set. MFC after: 3 days Reported by: dwhite, peter Notes: svn path=/head/; revision=175654
* A bunch more files that should probably print out a thread nameJulian Elischer2007-11-141-1/+1
| | | | | | | instead of a process name. Notes: svn path=/head/; revision=173601
* generally we are interested in what thread did something asJulian Elischer2007-11-141-5/+5
| | | | | | | | | opposed to what process. Since threads by default have teh name of the process unless over-written with more useful information, just print the thread name instead. Notes: svn path=/head/; revision=173600
* subr_sleepqueue.c presents a thread lock missing which leads to dangerousAttilio Rao2007-09-131-0/+2
| | | | | | | | | | | | | | | | races for some struct thread members. More specifically, this bug seems responsible for some memory dumping problems people were experiencing. Fix this adding correct thread locking. Tested by: rwatson Submitted by: tegge Approved by: jeff Approved by: re Notes: svn path=/head/; revision=172155
* - Include opt_sched.h for SCHED_STATS.Jeff Roberson2007-06-121-0/+1
| | | | Notes: svn path=/head/; revision=170640
* Commit 2/14 of sched_lock decomposition.Jeff Roberson2007-06-041-106/+97
| | | | | | | | | | | | | | | | | | | | - Adapt sleepqueues to the new thread_lock() mechanism. - Delay assigning the sleep queue spinlock as the thread lock until after we've checked for signals. It is illegal for a thread to return in mi_switch() with any lock assigned to td_lock other than the scheduler locks. - Change sleepq_catch_signals() to do the switch if necessary to simplify the callers. - Simplify timeout handling now that locking a sleeping thread has the side-effect of locking the sleepqueue. Some previous races are no longer possible. Tested by: kris, current@ Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc. Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each) Notes: svn path=/head/; revision=170294
* - Convert turnstiles and sleepqueus to use UMA. This provides a modestJeff Roberson2007-05-181-17/+52
| | | | | | | | | | | | | speedup and will be more useful after each gains a spinlock in the impending thread_lock() commit. - Move initialization and asserts into init/fini routines. fini routines are only needed in the INVARIANTS case for now. Submitted by: Attilio Rao <attilio@FreeBSD.org> Tested by: kris, jeff Notes: svn path=/head/; revision=169666
* Cleaner fix for handling declaration of loop variable under INVARIANTSKip Macy2006-12-171-10/+7
| | | | | | | | | | - in trying to avoid nested brackets and #ifdef INVARIANTS around i at the top, I broke booting for INVARIANTS all together :-( - the cleanest fix is to simply assign to sq twice if INVARIANTS is enabled - tested both with and without INVARIANTS :-/ Notes: svn path=/head/; revision=165292
* Don't intermix assignments and variable declarations in prev. commitAndrey A. Chernov2006-12-161-4/+6
| | | | Notes: svn path=/head/; revision=165291
* Fix NULL pointer reference for INVARIANTS caseAndrey A. Chernov2006-12-161-1/+1
| | | | | | | Submitted by: Yuriy Tsibizov <Yuriy.Tsibizov@gfk.ru> Notes: svn path=/head/; revision=165290
* correct name of number of sleep queuesKip Macy2006-12-161-1/+1
| | | | Notes: svn path=/head/; revision=165275
* Add second sleep queue so that sx and lockmgr can have separate sleepKip Macy2006-12-161-26/+48
| | | | | | | | | | queues for shared and exclusive acquisitions Submitted by: Attilio Rao Approved by: jhb Notes: svn path=/head/; revision=165272
* Change sleepq_add(9) argument from 'struct mtx *' to 'struct lock_object *',Pawel Jakub Dawidek2006-11-161-3/+3
| | | | | | | | | | | which allows to use it with different kinds of locks. For example it allows to implement Solaris conditions variables which will be used in ZFS port on top of sx(9) locks. Reviewed by: jhb Notes: svn path=/head/; revision=164325
* Print td_name instead of p_comm if td_name is non-empty forJohn Baldwin2006-04-211-0/+1
| | | | | | | 'show turnstile' and 'show sleepq'. Notes: svn path=/head/; revision=157952
* Add a 'show sleepqueue' alias for 'show sleepq' in DDB.John Baldwin2006-04-171-0/+3
| | | | Notes: svn path=/head/; revision=157823
* Clear TDF_SINTR in sleepq_resume_thread, also sleepq_catch_signal doesDavid Xu2006-04-131-1/+1
| | | | | | | | | | | | not need to clear it now, this should fix panic when msleep is recursivly called. Patch is slightly adjusted after review. Reviewed by: jhb Tested by: Csaba Henk, csaba-ml at creo.hu MFC after: 3 days Notes: svn path=/head/; revision=157743
* Move comments to more accurate place.David Xu2006-02-231-4/+4
| | | | Notes: svn path=/head/; revision=155936
* Fix a sleep queue race for KSE thread.David Xu2006-02-231-11/+21
| | | | | | | Reviewed by: jhb Notes: svn path=/head/; revision=155932
* Fix a long standing race between sleep queue and threadDavid Xu2006-02-151-81/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | suspension code. When a thread A is going to sleep, it calls sleepq_catch_signals() to detect any pending signals or thread suspension request, if nothing happens, it returns without holding process lock or scheduler lock, this opens a race window which allows thread B to come in and do process suspension work, however since A is still at running state, thread B can do nothing to A, thread A continues, and puts itself into actually sleeping state, but B has never seen it, and it sits there forever until B is woken up by other threads sometimes later(this can be very long delay or never happen). Fix this bug by forcing sleepq_catch_signals to return with scheduler lock held. Fix sleepq_abort() by passing it an interrupted code, previously, it worked as wakeup_one(), and the interruption can not be identified correctly by sleep queue code when the sleeping thread is resumed. Let thread_suspend_check() returns EINTR or ERESTART, so sleep queue no longer has to use SIGSTOP as a hack to build a return value. Reviewed by: jhb MFC after: 1 week Notes: svn path=/head/; revision=155741
* lock unused when INVARIANTS not defined, so don't declare it thenWarner Losh2006-01-281-0/+2
| | | | Notes: svn path=/head/; revision=154944
* Add a new ddb command 'show sleepq'. It takes a wait channel as anJohn Baldwin2006-01-271-2/+64
| | | | | | | | | | | | argument and looks for a sleep queue associated with that wait channel. If it finds one it will display information such as the list of threads sleeping on that queue. If it can't find a sleep queue for that wait channel, then it will see if that address matches any of the active sleep queues. If so, it will display information about the sleepq at the specified address. Notes: svn path=/head/; revision=154936
* Clarify panic message, I parsed the old one 'trying to sleep while sleeping'Warner Losh2005-11-091-1/+1
| | | | Notes: svn path=/head/; revision=152221
* Normalize a significant number of kernel malloc type names:Robert Watson2005-10-311-1/+1
| | | | | | | | | | | | | | | | | | | | | | - Prefer '_' to ' ', as it results in more easily parsed results in memory monitoring tools such as vmstat. - Remove punctuation that is incompatible with using memory type names as file names, such as '/' characters. - Disambiguate some collisions by adding subsystem prefixes to some memory types. - Generally prefer lower case to upper case. - If the same type is defined in multiple architecture directories, attempt to use the same name in additional cases. Not all instances were caught in this change, so more work is required to finish this conversion. Similar changes are required for UMA zone names. Notes: svn path=/head/; revision=151897
* - Add a new simple facility for marking the current thread as being in aJohn Baldwin2005-09-151-0/+4
| | | | | | | | | | | | | | | | state where sleeping on a sleep queue is not allowed. The facility doesn't support recursion but uses a simple private per-thread flag (TDP_NOSLEEPING). The sleepq_add() function will panic if the flag is set and INVARIANTS is enabled. - Use this new facility to replace the g_xup and g_xdown mutexes that were (ab)used to achieve similar behavior. - Disallow sleeping in interrupt threads when invoking interrupt handlers. MFC after: 1 week Reviewed by: phk Notes: svn path=/head/; revision=150177
* Remove thread_upcall_check, it was used to avoid race bug in earlierDavid Xu2005-05-271-5/+1
| | | | | | | | day's sleep queue code, today the bug no longer exists. please see 04/25/2004 freebsd-threads@ mailing list archive. Notes: svn path=/head/; revision=146687
* Close a race between sleepq_broadcast() and sleepq_catch_signals().John Baldwin2005-04-141-47/+21
| | | | | | | | | | | | | | | | | | | | | | | | | Specifically, sleepq_broadcast() uses td_slpq for its private pending queue of threads that it is going to wake up after it takes them off the sleep queue. The problem is that if one of the threads is actually not asleep yet, then we can end up with td_slpq being corrupted and/or the thread being made runnable at the wrong time resulting in the td_sleepqueue == NULL assertion failures occasionally reported under heavy load. The fix is to stop being so fancy and ditch the whole pending queue bit. Instead, sleepq_remove_thread() and sleepq_resume_thread() were merged into one function that requires the caller to hold sched_lock. This fixes several places that unlocked sched_lock only to call a function that then locked sched_lock, so even though sched_lock is now held slightly longer, removing the extra lock acquires (1 pair instead of 3 in some cases) probably makes it an overall win if you don't include the fact that it closes a race. This is definitely a 5.4 candidate. PR: kern/79693 Submitted by: Steven Sears stevenjsears at yahoo dot com MFC after: 4 days Notes: svn path=/head/; revision=145056
* Make a bunch of malloc types static.Poul-Henning Kamp2005-02-101-1/+1
| | | | | | | Found by: src/tools/tools/kernxref Notes: svn path=/head/; revision=141616
* /* -> /*- for copyright notices, minor format tweaks as necessaryWarner Losh2005-01-061-1/+1
| | | | Notes: svn path=/head/; revision=139804
generated by cgit v1.2.3 (git 2.25.1) at 2025-11-29 01:24:21 +0000