summaryrefslogtreecommitdiff
path: root/sys/kern/subr_taskqueue.c
Commit message (Collapse)AuthorAgeFilesLines
* MFC r314553:Hans Petter Selasky2017-03-141-0/+17
| | | | | | | | | | | Implement taskqueue_poll_is_busy() for use by the LinuxKPI. Refer to comment above function for a detailed description. Discussed with: kib @ Sponsored by: Mellanox Technologies Notes: svn path=/stable/11/; revision=315267
* MFC r306441 and r306634:Hans Petter Selasky2016-10-101-1/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | While draining a timeout task prevent the taskqueue_enqueue_timeout() function from restarting the timer. Commonly taskqueue_enqueue_timeout() is called from within the task function itself without any checks for teardown. Then it can happen the timer stays active after the return of taskqueue_drain_timeout(), because the timeout and task is drained separately. This patch factors out the teardown flag into the timeout task itself, allowing existing code to stay as-is instead of applying a teardown flag to each and every of the timeout task consumers. Add assert to taskqueue_drain_timeout() which prevents parallel execution on the same timeout task. Update manual page documenting the return value of taskqueue_enqueue_timeout(). Differential Revision: https://reviews.freebsd.org/D8012 Reviewed by: kib, trasz Notes: svn path=/stable/11/; revision=306946
* MFC r304021: Update iflib to support more NIC designsStephen Hurd2016-08-231-360/+0
| | | | | | | | | | | | | | | | - Move group task queue into kern/subr_gtaskqueue.c - Change intr_enable to return an int so it can be detected if it's not implemented - Allow different TX/RX queues per set to be different sizes - Don't split up TX mbufs before transmit - Allow a completion queue for TX as well as RX - Pass the RX budget to isc_rxd_available() to allow an earlier return and avoid multiple calls Approved by: sbruno Notes: svn path=/stable/11/; revision=304704
* Replace a number of conflations of mp_ncpus and mp_maxid with eitherNathan Whitehorn2016-07-061-5/+18
| | | | | | | | | | | | | | | | | | | | | | mp_maxid or CPU_FOREACH() as appropriate. This fixes a number of places in the kernel that assumed CPU IDs are dense in [0, mp_ncpus) and would try, for example, to run tasks on CPUs that did not exist or to allocate too few buffers on systems with sparse CPU IDs in which there are holes in the range and mp_maxid > mp_ncpus. Such circumstances generally occur on systems with SMT, but on which SMT is disabled. This patch restores system operation at least on POWER8 systems configured in this way. There are a number of other places in the kernel with potential problems in these situations, but where sparse CPU IDs are not currently known to occur, mostly in the ARM machine-dependent code. These will be fixed in a follow-up commit after the stable/11 branch. PR: kern/210106 Reviewed by: jhb Approved by: re (glebius) Notes: svn path=/head/; revision=302372
* taskqueue: plug a leak in _taskqueue_createMateusz Guzik2016-06-021-4/+6
| | | | | | | | | | While here make some style fixes and postpone the sprintf so that it is only done when the function can no longer fail. CID: 1356041 Notes: svn path=/head/; revision=301208
* fix loss of taskqueue wakeups (introduced in r300113)Andriy Gapon2016-05-211-10/+3
| | | | | | | | Submitted by: kmacy Tested by: dchagin Notes: svn path=/head/; revision=300372
* Adjust the creation of tq_name so it can be freed correctlyScott Long2016-05-191-7/+8
| | | | | | | | Reviewed by: jhb, allanjude Differential Revision: D6454 Notes: svn path=/head/; revision=300219
* Import the 'iflib' API library for network drivers. From the author:Scott Long2016-05-181-9/+378
| | | | | | | | | | | | | | | | | | "iflib is a library to eliminate the need for frequently duplicated device independent logic propagated (poorly) across many network drivers." Participation is purely optional. The IFLIB kernel config option is provided for drivers that want to transition between legacy and iflib modes of operation. ixl and ixgbe driver conversions will be committed shortly. We hope to see participation from the Broadcom and maybe Chelsio drivers in the near future. Submitted by: mmacy@nextbsd.org Reviewed by: gallatin Differential Revision: D5211 Notes: svn path=/head/; revision=300113
* Remove taskqueue_enqueue_fast().John Baldwin2016-03-011-7/+0
| | | | | | | | | | | | | taskqueue_enqueue() was changed to support both fast and non-fast taskqueues 10 years ago in r154167. It has been a compat shim ever since. It's time for the compat shim to go. Submitted by: Howard Su <howard0su@gmail.com> Reviewed by: sephe Differential Revision: https://reviews.freebsd.org/D5131 Notes: svn path=/head/; revision=296272
* This fixes several places where callout_stops return is examined. TheRandall Stewart2015-11-131-1/+1
| | | | | | | | | | | | | new return codes of -1 were mistakenly being considered "true". Callout_stop now returns -1 to indicate the callout had either already completed or was not running and 0 to indicate it could not be stopped. Also update the manual page to make it more consistent no non-zero in the callout_stop or callout_reset descriptions. MFC after: 1 Month with associated callout change. Notes: svn path=/head/; revision=290805
* MFuser/delphij/zfs-arc-rebase@r281754:Xin LI2015-05-261-10/+9
| | | | | | | | | | | | | | | | | | | | In r256613, taskqueue_enqueue_locked() have been modified to release the task queue lock before returning. In r276665, taskqueue_drain_all() will call taskqueue_enqueue_locked() to insert the barrier task into the queue, but did not reacquire the lock after it but later code expects the lock still being held (e.g. TQ_SLEEP()). The barrier task is special and if we release then reacquire the lock, there would be a small race window where a high priority task could sneak into the queue. Looking more closely, the race seems to be tolerable but is undesirable from semantics standpoint. To solve this, in taskqueue_drain_tq_queue(), instead of directly calling taskqueue_enqueue_locked(), insert the barrier task directly without releasing the lock. Notes: svn path=/head/; revision=283551
* Remove taskqueue_start_threads_pinned(); there's noa generic cpuset version ↵Adrian Chadd2015-02-251-24/+0
| | | | | | | | | of this. Sponsored by: Norse Corp, Inc. Notes: svn path=/head/; revision=279300
* Implement taskqueue_start_threads_cpuset().Adrian Chadd2015-02-171-12/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | This is a more generic version of taskqueue_start_threads_pinned() which only supports a single cpuid. This originally came from John Baldwin <jhb@> who implemented it as part of a push towards NUMA awareness in drivers. I started implementing something similar for RSS and NUMA, then found he already did it. I'd like to axe taskqueue_start_threads_pinned() so it doesn't become part of a longer-term API. (Read: hps@ wants to MFC things, and if I don't do this soon, he'll MFC what's here. :-) I have a follow-up commit which converts the intel drivers over to using the cpuset version of this function, so we can eventually nuke the the pinned version. Tested: * igb, ixgbe Obtained from: jhbbsd Notes: svn path=/head/; revision=278879
* Prevent live-lock and access of destroyed data in taskqueue_drain_all().Justin T. Gibbs2015-01-041-16/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | Phabric: https://reviews.freebsd.org/D1247 Reviewed by: jhb, avg Sponsored by: Spectra Logic Corporation sys/kern_subr_taskqueue.c: Modify taskqueue_drain_all() processing to use a temporary "barrier task", rather than rely on a user task that may be destroyed during taskqueue_drain_all()'s execution. The barrier task is queued behind all previously queued tasks and then has its priority elevated so that future tasks cannot pass it in the queue. Use a similar barrier scheme to drain threads processing current tasks. This requires taskqueue_run_locked() to insert and remove the taskqueue_busy object for the running thread for every task processed. share/man/man9/taskqueue.9: Remove warning about live-lock issues with taskqueue_drain_all() and indicate that it does not wait for tasks queued after it begins processing. Notes: svn path=/head/; revision=276665
* Remove trailing whitespace.Justin T. Gibbs2014-11-301-2/+2
| | | | Notes: svn path=/head/; revision=275345
* Temporary revert r269661, it looks like the patch isn't complete.Andrey V. Elsukov2014-08-071-1/+1
| | | | Notes: svn path=/head/; revision=269666
* Use cpuset_setithread() to apply cpu mask to taskq threads.Andrey V. Elsukov2014-08-071-1/+1
| | | | | | | Sponsored by: Yandex LLC Notes: svn path=/head/; revision=269661
* Pin the right thread.Adrian Chadd2014-06-011-1/+1
| | | | | | | | | | | | | This _was_ right, a last minute suggestion and not enough testing makes Adrian a bad boy. Tested: * igb(4) with RSS patches, by hand verifying each igb(4) taskqueue tid from procstat -ka using cpuset -g -t <tid>. Notes: svn path=/head/; revision=266939
* Add a new taskqueue setup method that takes a cpuid to pin theAdrian Chadd2014-05-241-9/+56
| | | | | | | | | | | | | | | | | | | | | | | taskqueue worker thread(s) to. For now it isn't a taskqueue/taskthread error to fail to pin to the given cpuid. Thanks to rpaulo@, kib@ and jhb@ for feedback. Tested: * igb(4), with local RSS patches to pin taskqueues. TODO: * ask the doc team for help in documenting the new API call. * add a taskqueue_start_threads_cpuset() method which takes a cpuset_t - but this may require a bunch of surgery to bring cpuset_t into scope. Notes: svn path=/head/; revision=266629
* add taskqueue_drain_allAndriy Gapon2013-11-281-0/+30
| | | | | | | | | | | | | This API has semantics similar to that of taskqueue_drain but acts on all tasks that might be queued or running on a taskqueue. A caller must ensure that no new tasks are being enqueued otherwise this call would be totally meaningless. For example, if the tasks are enqueued by an interrupt filter then its interrupt must be disabled. MFC after: 10 days Notes: svn path=/head/; revision=258713
* taskqueue_cancel: garbage collect a write-only variableAndriy Gapon2013-11-191-2/+0
| | | | | | | MFC after: 3 days Notes: svn path=/head/; revision=258354
* Add comments that taskqueue_enqueue_locked() returns without the lock.Alexander Motin2013-10-211-0/+4
| | | | Notes: svn path=/head/; revision=256862
* Revert r256587.Gleb Smirnoff2013-10-181-3/+0
| | | | | | | Requested by: zec Notes: svn path=/head/; revision=256730
* MFprojects/camlock r254763:Alexander Motin2013-10-161-4/+17
| | | | | | | | | Move tq_enqueue() call out of the queue lock for known handlers (actually I have found no others in the base system). This reduces queue lock hold time and congestion spinning under active multithreaded enqueuing. Notes: svn path=/head/; revision=256613
* MFprojects/camlock r254685:Alexander Motin2013-10-161-6/+1
| | | | | | | Remove TQ_FLAGS_PENDING flag, softly duplicating queue emptiness status. Notes: svn path=/head/; revision=256612
* For VIMAGE kernels store vnet in the struct task, and set vnet contextGleb Smirnoff2013-10-161-0/+3
| | | | | | | | | during task processing. Reported & tested by: mm Notes: svn path=/head/; revision=256587
* MFprojects/camlock r254460:Alexander Motin2013-08-241-2/+0
| | | | | | | | | Remove locking from taskqueue_member(). The list of threads is static during the taskqueue life cycle, so there is no need to protect it, taking quite congested lock several more times for each ZFS I/O. Notes: svn path=/head/; revision=254787
* Extend taskqueue(9) to enable per-taskqueue callbacks.Will Andrews2013-03-231-3/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The scope of these callbacks is primarily to support actions that affect the taskqueue's thread environments. They are entirely optional, and consequently are introduced as a new API: taskqueue_set_callback(). This interface allows the caller to specify that a taskqueue requires a callback and optional context pointer for a given callback type. The callback types included in this commit can be used to register a constructor and destructor for thread-local storage using osd(9). This allows a particular taskqueue to define that its threads require a specific type of TLS, without the need for a specially-orchestrated task-based mechanism for startup and shutdown in order to accomplish it. Two callback types are supported at this point: - TASKQUEUE_CALLBACK_TYPE_INIT, called by every thread when it starts, prior to processing any tasks. - TASKQUEUE_CALLBACK_TYPE_SHUTDOWN, called by every thread when it exits, after it has processed its last task but before the taskqueue is reclaimed. While I'm here: - Add two new macros, TQ_ASSERT_LOCKED and TQ_ASSERT_UNLOCKED, and use them in appropriate locations. - Fix taskqueue.9 to mention taskqueue_start_threads(), which is a required interface for all consumers of taskqueue(9). Reviewed by: kib (all), eadler (taskqueue.9), brd (taskqueue.9) Approved by: ken (mentor) Sponsored by: Spectra Logic MFC after: 1 month Notes: svn path=/head/; revision=248649
* Add a special meaning to the negative ticks argument forKonstantin Belousov2012-11-201-2/+6
| | | | | | | | | | | | | | | | | | | taskqueue_enqueue_timeout(). Do not rearm the callout if it is already armed and the ticks is negative. Otherwise rearm it to fire in abs(ticks) ticks in the future. The intended use is to call taskqueue_enqueue_timeout() for the given timeout_task with the same negative ticks argument. As result, the task is scheduled to execute not further than abs(ticks) ticks in future, and the consequent enqueues are coalesced until the already scheduled task is finished. Reviewed by: rwatson Tested by: Markus Gebert <markus.gebert@hostpoint.ch> MFC after: 2 weeks Notes: svn path=/head/; revision=243341
* Shorten the name of the fast SWI taskqueue to "fast taskq" so thatJohn Baldwin2012-08-281-1/+1
| | | | | | | | | | it fits. Reported by: lev MFC after: 1 week Notes: svn path=/head/; revision=239779
* Ensure that ta_pending doesn't overflow u_short by capping its value at ↵Adrian Chadd2011-09-151-1/+3
| | | | | | | | | | | | | | | | USHRT_MAX. If it overflows before the taskqueue can run, the task will be re-added to the taskqueue and cause a loop in the task list. Reported by: Arnaud Lacombe <lacombar@gmail.com> Submitted by: Ryan Stone <rysto32@gmail.com> Reviewed by: jhb Approved by: re (kib) MFC after: 1 day Notes: svn path=/head/; revision=225570
* Implement the delayed task execution extension to the taskqueueKonstantin Belousov2011-04-261-13/+115
| | | | | | | | | | | | mechanism. The caller may specify a timeout in ticks after which the task will be scheduled. Sponsored by: The FreeBSD Foundation Reviewed by: jeff, jhb MFC after: 1 month Notes: svn path=/head/; revision=221059
* taskqueue: drop unused tq_name fieldAndriy Gapon2010-11-231-3/+1
| | | | | | | | | | | | | | | tq_name was used write-only and besides it was just a pointer, so it could point to some garbage in a temporary buffer that's gone. This change shouldn't change KPI/KBI as struct taskqueue is private to subr_taskqueue.c. If we find a need for tq_name it can be resurrected at any moment. taskqueue_create() interface is preserved for this purpose. Suggested by: jhb MFC after: 10 days Notes: svn path=/head/; revision=215750
* Use macros rather than inline functions to lock and unlock mutexes, so thatJuli Mallett2010-11-081-17/+15
| | | | | | | | | line number information is preserved in witness. Reviewed by: jhb Notes: svn path=/head/; revision=215021
* Add a taskqueue_cancel(9) to cancel a pending task without waiting forMatthew D Fleming2010-11-081-0/+18
| | | | | | | | | | | | it to run as taskqueue_drain(9) does. Requested by: hselasky Original code: jeff Reviewed by: jhb MFC after: 2 weeks Notes: svn path=/head/; revision=215011
* Use a safer mechanism for determining if a task is currently running,Matthew D Fleming2010-10-131-30/+45
| | | | | | | | | | | that does not rely on the lifetime of pointers being the same. This also restores the task KBI. Suggested by: jhb MFC after: 1 month Notes: svn path=/head/; revision=213813
* Re-expose and briefly document taskqueue_run(9). The function is usedMatthew D Fleming2010-10-121-3/+1
| | | | | | | | | in at least one 3rd party driver. Requested by: jhb Notes: svn path=/head/; revision=213739
* Run all tasks from a proper context, with proper priority, etc.Pawel Jakub Dawidek2010-08-281-1/+1
| | | | | | | | Reviewed by: jhb MFC after: 1 month Notes: svn path=/head/; revision=211928
* Simplify taskqueue_drain() by using proved macros.Pawel Jakub Dawidek2010-08-131-14/+7
| | | | Notes: svn path=/head/; revision=211284
* Remove unused variable that snuck in during development.Matthew D Fleming2010-07-221-2/+1
| | | | | | | Approved by: zml (mentor) Notes: svn path=/head/; revision=210380
* Fix taskqueue_drain(9) to not have false negatives. For threadedMatthew D Fleming2010-07-221-23/+38
| | | | | | | | | | | | | | | | | taskqueues, more than one task can be running simultaneously. Also make taskqueue_run(9) static to the file, since there are no consumers in the base kernel and the function signature needs to change with this fix. Remove mention of taskqueue_run(9) and taskqueue_run_fast(9) from the taskqueue(9) man page. Reviewed by: jhb Approved by: zml (mentor) Notes: svn path=/head/; revision=210377
* fix a few cases where a string is passed via format argument instead ofAndriy Gapon2010-06-111-1/+1
| | | | | | | | | | | | | via %s Most of the cases looked harmless, but this is done for the sake of correctness. In one case it even allowed to drop an intermediate buffer. Found by: clang MFC after: 2 week Notes: svn path=/head/; revision=209062
* Revert taskqueue(9) related commits until mdf@ is approved and canZachary Loafman2010-06-011-14/+6
| | | | | | | | | resolve issues. This reverts commits r207439, r208623, r208624 Notes: svn path=/head/; revision=208715
* Avoid a wakeup(9) if we can be sure no one is waiting on the task.Zachary Loafman2010-05-281-3/+11
| | | | | | | | Submitted by: Matthew Fleming <matthew.fleming@isilon.com> Reviewed by: zml, jhb Notes: svn path=/head/; revision=208624
* Revert r207439 and solve the problem differently. The task handlerZachary Loafman2010-05-281-6/+5
| | | | | | | | | | | | ta_func may free the task structure, so no references to its members are valid after the handler has been called. Using a per-queue member and having waits longer than strictly necessary was suggested by jhb. Submitted by: Matthew Fleming <matthew.fleming@isilon.com> Reviewed by: zml, jhb Notes: svn path=/head/; revision=208623
* Handle taskqueue_drain(9) correctly on a threaded taskqueue:Zachary Loafman2010-04-301-5/+6
| | | | | | | | | | | | | | | taskqueue_drain(9) will not correctly detect whether a task is currently running. The check is against a field in the taskqueue struct, but for a threaded queue with more than one thread, multiple threads can simultaneously be running a task, thus stomping over the tq_running field. Submitted by: Matthew Fleming <matthew.fleming@isilon.com> Reviewed by: jhb Approved by: dfr (mentor) Notes: svn path=/head/; revision=207439
* - Fix several off-by-one errors when using MAXCOMLEN. The p_comm[] andJohn Baldwin2009-10-231-2/+2
| | | | | | | | | | | | | | | | td_name[] arrays are actually MAXCOMLEN + 1 in size and a few places that created shadow copies of these arrays were just using MAXCOMLEN. - Prefer using sizeof() of an array type to explicit constants for the array length in a few places. - Ensure that all of p_comm[] and td_name[] is always zero'd during execve() to guard against any possible information leaks. Previously trailing garbage in p_comm[] could be leaked to userland in ktrace record headers via td_name[]. Reviewed by: bde Notes: svn path=/head/; revision=198411
* Remove unused taskqueue_find() function.Pawel Jakub Dawidek2009-08-181-43/+0
| | | | | | | | Reviewed by: dfr Approved by: re (kib) Notes: svn path=/head/; revision=196358
* Remove OpenSolaris taskq port (it performs very poorly in our kernel) andPawel Jakub Dawidek2009-08-171-0/+20
| | | | | | | | | | | replace it with wrappers around our taskqueue(9). To make it possible implement taskqueue_member() function which returns 1 if the given thread was created by the given taskqueue. Approved by: re (kib) Notes: svn path=/head/; revision=196295
* Because taskqueue_run() can drop tq_mutex, we need to check if thePawel Jakub Dawidek2009-08-171-0/+7
| | | | | | | | | | TQ_FLAGS_ACTIVE flag wasn't removed in the meantime, which means we missed a wakeup. Approved by: re (kib) Notes: svn path=/head/; revision=196293
generated by cgit v1.2.3 (git 2.25.1) at 2025-08-23 01:06:18 +0000