aboutsummaryrefslogtreecommitdiff
path: root/sys/kern/kern_rangelock.c
Commit message (Collapse)AuthorAgeFilesLines
* rangelock: Fix handling of trylocksMark Johnston2025-03-311-43/+79
| | | | | | | | | | | | | | | | | | | | | | When inserting a queue entry, i.e., locking a range, there are two points where a trylock operation may fail, one before the new entry is inserted, one after. In the latter case, rl_(r|w)_validate() would mark the entry and rangelock_lock_int() would free it. However, this is of course incorrect, since the entry is visible to other threads, which will eventually attempt to remove it and free it again. Factor out conflict handling in rl_(r|w)_validate() to a common function as they are functionally the same. Then, introduce a new result which indicates that a trylock failed but that the queue entry must not be cleaned up. While here, assert that a conflicting range isn't owned by the current thread, as that would indicate a bug in the consumer. Reviewed by: olce, kib Reported by: syzkaller Fixes: 5badbeeaf061 ("Re-implement rangelocks part 2") Differential Revision: https://reviews.freebsd.org/D49438
* rangelock: Use atomic_testandset_ptrJohn Baldwin2024-11-191-2/+2
| | | | | | | Reviewed by: imp, kib Obtained from: CheriBSD Sponsored by: AFRL, DARPA Differential Revision: https://reviews.freebsd.org/D47632
* rangelocks: stop caching per-thread rl_q_entryKonstantin Belousov2024-08-291-23/+3
| | | | | | | | | This should reduce the frequency of smr_synchronize() calls, that otherwise occur on almost each rangelock unlock. Reviewed by: markj Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D46482
* rangelocks: remove unneeded cast of the atomic_load_ptr() resultKonstantin Belousov2024-08-281-4/+4
| | | | | | Noted and reviewed by: markj Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D46465
* rangelocks: re-enable cheat modeKonstantin Belousov2024-08-281-1/+1
| | | | | | | Tested by: lwhsu Reviewed by: markj Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D46465
* Add rangelock_may_recurse(9)Konstantin Belousov2024-08-281-0/+41
| | | | | | | Reviewed by: markj Tested by: lwhsu Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D46465
* rangelocks: extract the cheat mode drain codeKonstantin Belousov2024-08-281-11/+19
| | | | | | | Reviewed by: markj Tested by: lwhsu Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D46465
* rangelock: Disable cheat mode by defaultMark Johnston2024-08-271-1/+1
| | | | | | | | | | | | Cheat mode is incompatible with code which locks multiple ranges in the same vnode, with at least one range being write-locked. This can arise in kern_copy_file_range(). Until that's handled somehow, avoid the problem to make the fusefs tests stable. PR: 281073 Fixes: 9ef425e560a9 ("rangelocks: add fast cheating mode") Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D46457
* rangelock: Fix an off-by-one errorMark Johnston2024-08-271-1/+1
| | | | | | | | | | A rangelock entry covers the range [start, end), so entries e1 and e2 with e1->end == e2->start do not overlap. PR: 281073 Fixes: 5badbeeaf061 ("Re-implement rangelocks part 2") Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D46458
* rangelocks: fix typo in rl_w_validateKonstantin Belousov2024-08-211-1/+1
| | | | | | | | The freed elements should be threaded using rl_q_free pointer. Reported by: dougm, markj Tested by: markj Sponsored by: The FreeBSD Foundation
* rangelocks: recheck that entry is not marked after sleepq is locked in ↵Konstantin Belousov2024-08-211-0/+6
| | | | | | | | | rl_w_validate() otherwise we might loose the wakeup. Reported and tested by: markj Sponsored by: The FreeBSD Foundation
* rangelock: if CAS for removal failed, restart list iterationKonstantin Belousov2024-08-211-6/+11
| | | | | | | Our next pointer is invalid and cannot be followed. Tested by: markj, pho Sponsored by: The FreeBSD Foundation
* rangelock: assert that we never insert or remove our entry after a logically ↵Konstantin Belousov2024-08-211-0/+2
| | | | | | | deleted one Tested by: markj, pho Sponsored by: The FreeBSD Foundation
* rangelock_destoy(): poison lock->head to trip fault on lock attemptKonstantin Belousov2024-08-211-0/+1
| | | | | Tested by: markj, pho Sponsored by: The FreeBSD Foundation
* ranglelock_destroy(): do not remove lock entries from under live lock acquirerKonstantin Belousov2024-08-211-9/+47
| | | | | Tested by: markj, pho Sponsored by: The FreeBSD Foundation
* rangelocks: add rangelock_free_free() helper to free free listKonstantin Belousov2024-08-211-14/+24
| | | | | Tested by: markj, pho Sponsored by: The FreeBSD Foundation
* rangelocks: re-check that the lock is still in the cheating mode after ↵Konstantin Belousov2024-08-081-9/+11
| | | | | | | | | | setting of DRAINING failed Noted by: markj Tested by: dougm, pho (previous version) Reviewed by: markj (previous version), dougm Fixes: 9ef425e560a97cabd1862e803eeb48468f89de18 Sponsored by: The FreeBSD Foundation
* kern_rangelock.c: add Foundation copyright, remove sys/cdefs.h includeKonstantin Belousov2024-08-061-1/+5
| | | | | | | | | Also drop 'All rights reserved' line from my copyright. Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr> Discussed with: emaste (whole series) Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D41787
* rangelocks: add fast cheating modeKonstantin Belousov2024-08-061-6/+233
| | | | | | | Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr> Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D41787
* rangelocks: restore caching of the single rl entry in the struct threadKonstantin Belousov2024-08-061-6/+27
| | | | | | | Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr> Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D41787
* Microoptimize rangelock_unlock_int()Konstantin Belousov2024-08-061-1/+5
| | | | | | | | | | Only broadcast if there are sleepers. Suggested by: markj Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr> Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D41787
* Re-implement rangelocks part 2Konstantin Belousov2024-08-061-63/+169
| | | | | | | | | Allow read locks to overlap. Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr> Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D41787
* kern_rangelock.c: add rl_e_is_rlock()Konstantin Belousov2024-08-061-0/+6
| | | | | | | Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr> Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D41787
* Re-implement rangelocks part 1Konstantin Belousov2024-08-061-212/+243
| | | | | | | | | | | Using the algorithms from https://doi.org/10.1145/3342195.3387533. For the first part, consider all range lock requests as exclusive. Reviewed by: markj, Olivier Certner <olce.freebsd@certner.fr> Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D41787
* sys: Automated cleanup of cdefs and other formattingWarner Losh2023-11-271-1/+0
| | | | | | | | | | | | | | | | Apply the following automated changes to try to eliminate no-longer-needed sys/cdefs.h includes as well as now-empty blank lines in a row. Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/ Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/ Remove /\n+#if.*\n#endif.*\n+/ Remove /^#if.*\n#endif.*\n/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/ Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/ Sponsored by: Netflix
* sys: Remove $FreeBSD$: one-line .c patternWarner Losh2023-08-161-2/+0
| | | | Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
* spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSDWarner Losh2023-05-121-1/+1
| | | | | | | | | The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch up to that fact and revert to their recommended match of BSD-2-Clause. Discussed with: pfg MFC After: 3 days Sponsored by: Netflix
* rangelock: add rangelock_cookie_assertKyle Evans2019-09-151-0/+32
| | | | | | | | | | | | | | | | | A future change to posixshm to add file sealing (in DIFF_21391[0] and child) will move locking out of shm_dotruncate as kern_shm_open() will require the lock to be held across the dotruncate until the seal is actually applied. For this, the cookie is passed into shm_dotruncate_locked which asserts RCA_WLOCKED. [0] Name changed to protect the innocent, hopefully, from getting autoclosed due to this reference... Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D21628 Notes: svn path=/head/; revision=352350
* Add non-blocking trylock variants for the rangelock functions.Rick Macklem2019-06-271-10/+61
| | | | | | | | | | | | | | | | | A future patch that will add a Linux compatible copy_file_range(2) syscall needs to be able to lock the byte ranges of two files concurrently. To do this without a risk of deadlock, a non-blocking variant of vn_rangelock_rlock() called vn_rangelock_tryrlock() was needed. This patch adds this, along with vn_rangelock_trywlock(), in order to do this. The patch also adds a couple of comments, that I hope clarify how the algorithm used in kern_rangelock.c works. Reviewed by: kib, asomers (previous version) Differential Revision: https://reviews.freebsd.org/D20645 Notes: svn path=/head/; revision=349476
* sys/kern: adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326271
* Change the queue of locks in kern_rangelock.c from holding lock requests inColin Percival2013-08-151-27/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | the order that they arrive, to holding (a) granted write lock requests, followed by (b) granted read lock requests, followed by (c) ungranted requests, in order of arrival. This changes the stopping condition for iterating through granted locks to see if a new request can be granted: When considering a read lock request, we can stop iterating as soon as we see a read lock request, since anything after that point is either a granted read lock request or a request which has not yet been granted. (For write lock requests, we must still compare against all granted lock requests.) For workloads with R parallel reads and W parallel writes, this improves the time spent from O((R+W)^2) to O(W*(R+W)); i.e., heavy parallel-read workloads become significantly more scalable. No statistically significant change in buildworld time has been measured, but synthetic tests of parallel 'dd > /dev/null' and 'openssl enc >/dev/null' with the input file cached yield dramatic (up to 10x) improvement with high (up to 128 processes) levels of parallelism. Reviewed by: kib Notes: svn path=/head/; revision=254380
* Add a rangelock implementation, intended to be used to range-lockingKonstantin Belousov2012-05-301-0/+246
the i/o regions of the vnode data space. The implementation is quite simple-minded, it uses the list of the lock requests, ordered by arrival time. Each request may be for read or for write. The implementation is fair FIFO. MFC after: 2 month Notes: svn path=/head/; revision=236317
generated by cgit v1.2.3 (git 2.25.1) at 2025-08-05 00:56:56 +0000