aboutsummaryrefslogtreecommitdiff
path: root/sys/kern/subr_epoch.c
Commit message (Collapse)AuthorAgeFilesLines
* epoch: support non-preemptible epochs checking in_epoch()Kyle Evans2020-11-071-5/+105
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, non-preemptible epochs could not check; in_epoch() would always fail, usually because non-preemptible epochs don't imply THREAD_NO_SLEEPING. For default epochs, it's easy enough to verify that we're in the given epoch: if we're in a critical section and our record for the given epoch is active, then we're in it. This patch also adds some additional INVARIANTS bookkeeping. Notably, we set and check the recorded thread in epoch_enter/epoch_exit to try and catch some edge-cases for the caller. It also checks upon freeing that none of the records had a thread in the epoch, which may make it a little easier to diagnose some improper use if epoch_free() took place while some other thread was inside. This version differs slightly from what was just previously reviewed by the below-listed, in that in_epoch() will assert that no CPU has this thread recorded even if it *is* currently in a critical section. This is intended to catch cases where the caller might have somehow messed up critical section nesting, we can catch both if they exited the critical section or if they exited, migrated, then re-entered (on the wrong CPU). Reviewed by: kib, markj (both previous version) MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D27098 Notes: svn path=/head/; revision=367440
* Add full support support for dynamic allocation and freeing of epoch's.Hans Petter Selasky2020-08-071-13/+58
| | | | | | | | | | | | | | | | | Make sure to reclaim epoch structures when they are freed to support dynamic allocation and freeing of epoch structures. While at it, move the 64 supported epoch control structures to the static memory domain. This overall simplifies the management and debugging of system epoch's. Reviewed by: kib, markj Differential Revision: https://reviews.freebsd.org/D25960 MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=364019
* Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)Pawel Biernacki2020-02-261-2/+4
| | | | | | | | | | | | | | | | | | | r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718 Notes: svn path=/head/; revision=358333
* epoch: convert zpcpu_get_cpua(.., curcpu) to zpcpu_getMateusz Guzik2020-02-121-1/+1
| | | | Notes: svn path=/head/; revision=357803
* Change argument order of epoch_call() to more natural, first function,Gleb Smirnoff2020-01-171-2/+2
| | | | | | | | | then its argument. Reviewed by: imp, cem, jhb Notes: svn path=/head/; revision=356826
* Code must not unlock a mutex while owning the thread lock.Konstantin Belousov2020-01-131-1/+1
| | | | | | | | | | Reviewed by: hselasky, markj Sponsored by: Mellanox Technologies MFC after: 1 week Differential revision: https://reviews.freebsd.org/D23150 Notes: svn path=/head/; revision=356682
* Restore important comment in RCU/EPOCH support in FreeBSD after r355784.Hans Petter Selasky2019-12-181-0/+7
| | | | | | | Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=355879
* schedlock 4/4Jeff Roberson2019-12-151-9/+1
| | | | | | | | | | | | | | | | | | | | | Don't hold the scheduler lock while doing context switches. Instead we unlock after selecting the new thread and switch within a spinlock section leaving interrupts and preemption disabled to prevent local concurrency. This means that mi_switch() is entered with the thread locked but returns without. This dramatically simplifies scheduler locking because we will not hold the schedlock while spinning on blocked lock in switch. This change has not been made to 4BSD but in principle it would be more straightforward. Discussed with: markj Reviewed by: kib Tested by: pho Differential Revision: https://reviews.freebsd.org/D22778 Notes: svn path=/head/; revision=355784
* Improve EPOCH_TRACEBjoern A. Zeeb2019-12-061-5/+12
| | | | | | | | | | | | | | | | | | | Two changes to EPOCH_TRACE: (1) add a sysctl to surpress the backtrace from epoch_trace_report(). Sometimes the log line for the recursion is enough and the backtrace massively spams the console. (2) In order to be able to go without the backtrace do not only print where the previous occurance happened, but also where the current one happens. That way we have file:line information for both and can look at them without the need for getting line numbers from backtrace and a debugging tool. Reviewed by: glebius Sponsored by: Netflix (originally) Differential Revision: https://reviews.freebsd.org/D22641 Notes: svn path=/head/; revision=355448
* Add explicit SI_SUB_EPOCHConrad Meyer2019-11-221-1/+1
| | | | | | | | | | | | | | | | Add explicit SI_SUB_EPOCH, after SI_SUB_TASKQ and before SI_SUB_SMP (EARLY_AP_STARTUP). Rename existing "SI_SUB_TASKQ + 1" to SI_SUB_EPOCH. epoch(9) consumers cannot epoch_alloc() before SI_SUB_EPOCH:SI_ORDER_SECOND, but likely should allocate before SI_SUB_SMP. Prior to this change, consumers (well, epoch itself, and net/if.c) just open-coded the SI_SUB_TASKQ + 1 order to match epoch.c, but this was fragile. Reviewed by: mmacy Differential Revision: https://reviews.freebsd.org/D22503 Notes: svn path=/head/; revision=355024
* Merge td_epochnest with td_no_sleeping.Gleb Smirnoff2019-10-291-11/+3
| | | | | | | | | | | | | | | | | | | Epoch itself doesn't rely on the counter and it is provided merely for sleeping subsystems to check it. - In functions that sleep use THREAD_CAN_SLEEP() to assert correctness. With EPOCH_TRACE compiled print epoch info. - _sleep() was a wrong place to put the assertion for epoch, right place is sleepq_add(), as there ways to call the latter bypassing _sleep(). - Do not increase td_no_sleeping in non-preemptible epochs. The critical section would trigger all possible safeguards, no sleeping counter is extraneous. Reviewed by: kib Notes: svn path=/head/; revision=354148
* Allow epoch tracker to use the very last byte of the stack. Not sureGleb Smirnoff2019-10-221-1/+1
| | | | | | | | | | this will help to avoid panic in this function, since it will also use some stack, but makes code more strict. Submitted by: hselasky Notes: svn path=/head/; revision=353896
* Assert that any epoch tracker belongs to the thread stack.Gleb Smirnoff2019-10-211-1/+5
| | | | | | | Reviewed by: kib Notes: svn path=/head/; revision=353876
* Remove epoch tracker from struct thread. It was an ugly crutch to emulateGleb Smirnoff2019-10-211-14/+0
| | | | | | | locking semantics for if_addr_rlock() and if_maddr_rlock(). Notes: svn path=/head/; revision=353869
* When assertion for a thread not being in an epoch fails also print allGleb Smirnoff2019-10-151-0/+11
| | | | | | | | | | entered epochs. Works with EPOCH_TRACE only. Reviewed by: hselasky Differential Revision: https://reviews.freebsd.org/D22017 Notes: svn path=/head/; revision=353596
* Since EPOCH_TRACE had been moved to opt_global.h, we don't need to wasteGleb Smirnoff2019-10-141-0/+2
| | | | | | | extra space in struct thread. Notes: svn path=/head/; revision=353485
* Add debugging facility EPOCH_TRACE that checks that epochs entered areGleb Smirnoff2019-09-251-18/+116
| | | | | | | | | | | | | | properly nested and warns about recursive entrances. Unlike with locks, there is nothing fundamentally wrong with such use, the intent of tracer is to help to review complex epoch-protected code paths, and we mean the network stack here. Reviewed by: hselasky Sponsored by: Netflix Pull Request: https://reviews.freebsd.org/D21610 Notes: svn path=/head/; revision=352707
* Fix the turnstile_lock() KPI.Mark Johnston2019-07-241-19/+13
| | | | | | | | | | | | | | | | | | | turnstile_{lock,unlock}() were added for use in epoch. turnstile_lock() returned NULL to indicate that the calling thread had lost a race and the turnstile was no longer associated with the given lock, or the lock owner. However, reader-writer locks may not have a designated owner, in which case turnstile_lock() would return NULL and epoch_block_handler_preempt() would leak spinlocks as a result. Apply a minimal fix: return the lock owner as a separate return value. Reviewed by: kib MFC after: 3 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21048 Notes: svn path=/head/; revision=350310
* Implement API for draining EPOCH(9) callbacks.Hans Petter Selasky2019-06-281-8/+89
| | | | | | | | | | | | | | | | | | The epoch_drain_callbacks() function is used to drain all pending callbacks which have been invoked by prior epoch_call() function calls on the same epoch. This function is useful when there are shared memory structure(s) referred to by the epoch callback(s) which are not refcounted and are rarely freed. The typical place for calling this function is right before freeing or invalidating the shared resource(s) used by the epoch callback(s). This function can sleep and is not optimized for performance. Differential Revision: https://reviews.freebsd.org/D20109 MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=349506
* Make taskqgroup_attach{,_cpu}(9) work across architecturesMarius Strobl2019-02-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | So far, intr_{g,s}etaffinity(9) take a single int for identifying a device interrupt. This approach doesn't work on all architectures supported, as a single int isn't sufficient to globally specify a device interrupt. In particular, with multiple interrupt controllers in one system as found on e. g. arm and arm64 machines, an interrupt number as returned by rman_get_start(9) may be only unique relative to the bus and, thus, interrupt controller, a certain device hangs off from. In turn, this makes taskqgroup_attach{,_cpu}(9) and - internal to the gtaskqueue implementation - taskqgroup_attach_deferred{,_cpu}() not work across architectures. Yet in turn, iflib(4) as gtaskqueue consumer so far doesn't fit architectures where interrupt numbers aren't globally unique. However, at least for intr_setaffinity(..., CPU_WHICH_IRQ, ...) as employed by the gtaskqueue implementation to bind an interrupt to a particular CPU, using bus_bind_intr(9) instead is equivalent from a functional point of view, with bus_bind_intr(9) taking the device and interrupt resource arguments required for uniquely specifying a device interrupt. Thus, change the gtaskqueue implementation to employ bus_bind_intr(9) instead and intr_{g,s}etaffinity(9) to take the device and interrupt resource arguments required respectively. This change also moves struct grouptask from <sys/_task.h> to <sys/gtaskqueue.h> and wraps struct gtask along with the gtask_fn_t typedef into #ifdef _KERNEL as userland likes to include <sys/_task.h> or indirectly drags it in - for better or worse also with _KERNEL defined -, which with device_t and struct resource dependencies otherwise is no longer as easily possible now. The userland inclusion problem probably can be improved a bit by introducing a _WANT_TASK (as well as a _WANT_MOUNT) akin to the existing _WANT_PRISON etc., which is orthogonal to this change, though, and likely needs an exp-run. While at it: - Change the gt_cpu member in the grouptask structure to be of type int as used elswhere for specifying CPUs (an int16_t may be too narrow sooner or later), - move the gtaskqueue_enqueue_fn typedef from <sys/gtaskqueue.h> to the gtaskqueue implementation as it's only used and needed there, - change the GTASK_INIT macro to use "gtask" rather than "task" as argument given that it actually operates on a struct gtask rather than a struct task, and - let subr_gtaskqueue.c consistently use __func__ to print functions names. Reported by: mmel Reviewed by: mmel Differential Revision: https://reviews.freebsd.org/D19139 Notes: svn path=/head/; revision=344062
* epoch(9) revert r340097 - no longer a need for multiple sections per cpuMatt Macy2018-11-141-11/+9
| | | | | | | | I spoke with Samy Bahra and recent changes to CK to make ck_epoch_call and ck_epoch_poll not modify the record have eliminated the need for this. Notes: svn path=/head/; revision=340420
* style(9), mostly adjusting overly long lines.Gleb Smirnoff2018-11-131-24/+30
| | | | Notes: svn path=/head/; revision=340419
* With epoch not inlined, there is no point in using _lite KPI. While here,Gleb Smirnoff2018-11-131-13/+13
| | | | | | | remove some unnecessary casts. Notes: svn path=/head/; revision=340417
* The dualism between epoch_tracker and epoch_thread is fragile andGleb Smirnoff2018-11-131-34/+19
| | | | | | | | | | unnecessary. So, expose CK types to kernel and use a single normal structure for epoch_tracker. Reviewed by: jtl, gallatin Notes: svn path=/head/; revision=340415
* For compatibility KPI functions like if_addr_rlock() that used to haveGleb Smirnoff2018-11-131-0/+14
| | | | | | | | | | | | | | | mutexes but now are converted to epoch(9) use thread-private epoch_tracker. Embedding tracker into ifnet(9) or ifnet derived structures creates a non reentrable function, that will fail miserably if called simultaneously from two different contexts. A thread private tracker will provide a single tracker that would allow to call these functions safely. It doesn't allow nested call, but this is not expected from compatibility KPIs. Reviewed by: markj Notes: svn path=/head/; revision=340413
* Uninline epoch(9) entrance and exit. There is no proof that modernGleb Smirnoff2018-11-131-16/+129
| | | | | | | | | | | | | | | | processors would benefit from avoiding a function call, but bloating code. In fact, clang created an uninlined real function for many object files in the network stack. - Move epoch_private.h into subr_epoch.c. Code copied exactly, avoiding any changes, including style(9). - Remove private copies of critical_enter/exit. Reviewed by: kib, jtl Differential Revision: https://reviews.freebsd.org/D17879 Notes: svn path=/head/; revision=340404
* Convert epoch to read / write records per cpuMatt Macy2018-11-031-4/+5
| | | | | | | | | | | | | | | | | | | In discussing D17503 "Run epoch calls sooner and more reliably" with sbahra@ we came to the conclusion that epoch is currently misusing the ck_epoch API. It isn't safe to do a "write side" operation (ck_epoch_call or ck_epoch_poll) in the middle of a "read side" section. Since, by definition, it's possible to be preempted during the middle of an EPOCH_PREEMPT epoch the GC task might call ck_epoch_poll or another thread might call ck_epoch_call on the same section. The right solution is ultimately to change the way that ck_epoch works for this use case. However, as a stopgap for 12 we agreed to simply have separate records for each use case. Tested by: pho@ MFC after: 3 days Notes: svn path=/head/; revision=340097
* epoch_block_wait: don't check TD_RUNNINGMatt Macy2018-08-091-12/+26
| | | | | | | | | struct epoch_thread is not type safe (stack allocated) and thus cannot be dereferenced from another CPU Reported by: novel@ Notes: svn path=/head/; revision=337525
* epoch(9): simplify initializationMatt Macy2018-07-061-70/+22
| | | | | | | replace manual NUMA aware allocation with a pcpu zone Notes: svn path=/head/; revision=336021
* epoch(9): make nesting assert in epoch_wait_preempt more specificMatt Macy2018-07-041-1/+3
| | | | | | | Reported by: markj Notes: svn path=/head/; revision=335970
* epoch(9): allow preemptible epochs to composeMatt Macy2018-07-041-135/+84
| | | | | | | | | | | | | | | | | | | | | | | - Add tracker argument to preemptible epochs - Inline epoch read path in kernel and tied modules - Change in_epoch to take an epoch as argument - Simplify tfb_tcp_do_segment to not take a ti_locked argument, there's no longer any benefit to dropping the pcbinfo lock and trying to do so just adds an error prone branchfest to these functions - Remove cases of same function recursion on the epoch as recursing is no longer free. - Remove the the TAILQ_ENTRY and epoch_section from struct thread as the tracker field is now stack or heap allocated as appropriate. Tested by: pho and Limelight Networks Reviewed by: kbowling at llnw dot com Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D16066 Notes: svn path=/head/; revision=335924
* fix assert and conditionally allow mutexes to be held across epoch_wait_preemptMatt Macy2018-06-241-13/+12
| | | | Notes: svn path=/head/; revision=335605
* epoch(9): Don't trigger taskq enqueue before the grouptaskqs are setupMatt Macy2018-06-231-1/+14
| | | | | | | | | | | | | | If EARLY_AP_STARTUP is not defined it is possible for an epoch to be allocated prior to it being possible to call epoch_call without issue. Based on patch by andrew@ PR: 229014 Reported by: andrew Notes: svn path=/head/; revision=335578
* epoch(9): make non-preemptible variant work early bootMatt Macy2018-06-221-0/+2
| | | | Notes: svn path=/head/; revision=335524
* epoch(9): make epoch closer to style(9)Matt Macy2018-05-301-38/+44
| | | | Notes: svn path=/head/; revision=334368
* Don't pass a section cookie to CK for non-preemptible epoch sections.Mark Johnston2018-05-211-12/+5
| | | | | | | | | | | They're only useful when multiple threads may share an epoch record, and that can't happen with non-preemptible sections. Reviewed by: mmacy Differential Revision: https://reviews.freebsd.org/D15507 Notes: svn path=/head/; revision=333983
* subr_epoch.c fix unused variable warningsMatt Macy2018-05-191-4/+2
| | | | Notes: svn path=/head/; revision=333827
* epoch(9): assert that epoch is allocated post-configureMatt Macy2018-05-181-1/+7
| | | | Notes: svn path=/head/; revision=333808
* epoch(9): Make epochs non-preemptible by defaultMatt Macy2018-05-181-17/+20
| | | | | | | | | | | There are risks associated with waiting on a preemptible epoch section. Change the name to make them not be the default and document the issue under CAVEATS. Reported by: markj Notes: svn path=/head/; revision=333802
* epoch: actually allocate the counters we've assigned sysctls tooMatt Macy2018-05-181-0/+2
| | | | | | | Approved by: sbruno Notes: svn path=/head/; revision=333776
* epoch: add non-preemptible "critical" variantMatt Macy2018-05-181-30/+61
| | | | | | | | | | | | | | | | | adds: - epoch_enter_critical() - can be called inside a different epoch, starts a section that will acquire any MTX_DEF mutexes or do anything that might sleep. - epoch_exit_critical() - corresponding exit call - epoch_wait_critical() - wait variant that is guaranteed that any threads in a section are running. - epoch_global_critical - an epoch_wait_critical safe epoch instance Requested by: markj Approved by: sbruno Notes: svn path=/head/; revision=333775
* epoch: skip poll function call in hardclock unless there are callbacks pendingMatt Macy2018-05-171-14/+6
| | | | | | | | Reported by: mjg Approved by: sbruno Notes: svn path=/head/; revision=333769
* epoch(9): schedule pcpu callback task in hardclock if there are callbacks ↵Matt Macy2018-05-171-61/+78
| | | | | | | | | pending Approved by: sbruno Notes: svn path=/head/; revision=333761
* epoch(9): eliminate the need to wait when polling for callbacks to runMatt Macy2018-05-171-27/+20
| | | | | | | | | | by using ck's own callback handling mechanism we can simply check which callbacks have had a grace period elapse Approved by: sbruno Notes: svn path=/head/; revision=333759
* epoch(9): fix potential deadlockMatt Macy2018-05-171-0/+4
| | | | | | | | | Don't acquire a waiting thread's lock while holding our own Approved by: sbruno Notes: svn path=/head/; revision=333758
* epoch(9): restore thread priority on exit if it was changed by a waiterMatt Macy2018-05-171-0/+6
| | | | | | | | Reported by: markj Approved by: sbruno Notes: svn path=/head/; revision=333755
* epoch(9): make recursion lighter weightMatt Macy2018-05-171-22/+9
| | | | | | | | | | There isn't any real work to do except bump td_epochnest when recursing. Skip the additional work in this case. Approved by: sbruno Notes: svn path=/head/; revision=333697
* epoch(9): Guarantee forward progress on busy sectionsMatt Macy2018-05-171-4/+13
| | | | | | | | | | | Add epoch section to struct thread. We can use this to ennable epoch counter to advance even if a section is perpetually occupied by a thread. Approved by: sbruno Notes: svn path=/head/; revision=333695
* epoch(9): allow sx locks to be held across epoch_wait()Matt Macy2018-05-141-2/+6
| | | | | | | | | | | | | The INVARIANTS checks in epoch_wait() were intended to prevent the block handler from returning with locks held. What it in fact did was preventing anything except Giant from being held across it. Check that the number of locks held has not changed instead. Approved by: sbruno@ Notes: svn path=/head/; revision=333595
* epoch(9): cleanups, additional debug checks, and add global_epochMatt Macy2018-05-131-52/+35
| | | | | | | | | | | | | | | | | | - GC the _nopreempt routines - to really benefit we'd need a separate routine - they're not currently in use - they complicate the API for no benefit at this time - check that we're actually in a epoch section at exit - handle epoch_call() early in boot - Fix copyright declaration language Approved by: sbruno@ Notes: svn path=/head/; revision=333591
generated by cgit v1.3 (git 2.53.0) at 2026-05-29 23:00:05 +0000