FreeBSD source tree https://cgit-dev.freebsd.org/src/atom?h=releng%2F14.4 2024-01-19T16:24:35Z 2024-01-19T16:24:35Z Alexander Motin mav@FreeBSD.org 2023-12-27T03:30:56Z urn:sha1:faf11e40b49aaa44a553df781d2716ae849bfc56 With fast taskqueues using direct callouts we can reduce number of CPU wakeups by scheduling callout on current CPU if taskqueue calls taskqueue_enqueue_timeout() on itself. The trick won't work for regular taskqueues, since the callout thread will occupy the CPU. It also may not work in case of multiple threads since we do not know which thread will pick the task, and we do not want excessive callout migrations. So we optimize only the other cases we can. In practice this allows iichid(4) taskqueue to stay on CPU where underlying ig4(4) interrupts are routed and to not kick CPU 0 with timer interrupts on each sampling period (every 2nd/3rd sleep). MFC after: 1 month (cherry picked from commit 7bbac6419d174c98cc6ea969b68fcfe0f9a9bab8) 2023-08-16T17:54:36Z Warner Losh imp@FreeBSD.org 2023-08-16T17:54:36Z urn:sha1:685dc743dc3b5645e34836464128e1c0558b404b Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/ 2023-05-12T16:44:03Z Warner Losh imp@FreeBSD.org 2023-05-10T15:40:58Z urn:sha1:4d846d260e2b9a3d4d0a701462568268cbfe7a5b The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch up to that fact and revert to their recommended match of BSD-2-Clause. Discussed with: pfg MFC After: 3 days Sponsored by: Netflix 2022-05-17T12:10:20Z Vladimir Kondratyev wulf@FreeBSD.org 2022-05-17T12:10:20Z urn:sha1:b6f87b78b5bb48e00f54b96ddea7ad5bf5e3aa1f Kthread worker is a single thread workqueue which can be used in cases where specific kthread association is necessary, for example, when it should have RT priority or be assigned to certain cgroup. This change implements Linux v4.9 interface which mostly hides kthread internals from users thus allowing to use ordinary taskqueue(9) KPI. As kthread worker prohibits enqueueing of already pending or canceling tasks some minimal changes to taskqueue(9) were done. taskqueue_enqueue_flags() was added to taskqueue KPI which accepts extra flags parameter. It contains one or more of the following flags: TASKQUEUE_FAIL_IF_PENDING - taskqueue_enqueue_flags() fails if the task is already scheduled to execution. EEXIST is returned and the ta_pending counter value remains unchanged. TASKQUEUE_FAIL_IF_CANCELING - taskqueue_enqueue_flags() fails if the task is in the canceling state and ECANCELED is returned. Required by: drm-kmod 5.10 MFC after: 1 week Reviewed by: hselasky, Pau Amma (docs) Differential Revision: https://reviews.freebsd.org/D35051 2021-09-03T01:16:46Z Alexander Motin mav@FreeBSD.org 2021-09-03T01:16:46Z urn:sha1:4730a8972b1f4b67bf9ffde8e63ca906ef4c9563 Implement lock_spin()/unlock_spin() lock class methods, moving the assertion to _sleep() instead. Change assertions in callout(9) to allow spin locks for both regular and C_DIRECT_EXEC cases. In case of C_DIRECT_EXEC callouts spin locks are the only locks allowed actually. As the first use case allow taskqueue_enqueue_timeout() use on fast task queues. It actually becomes more efficient due to avoided extra context switches in callout(9) thanks to C_DIRECT_EXEC. MFC after: 2 weeks Reviewed by: hselasky Differential Revision: https://reviews.freebsd.org/D31778 2021-09-01T03:50:35Z Alexander Motin mav@FreeBSD.org 2021-09-01T03:47:51Z urn:sha1:706b1a5724d668a8752ac89cd67113e4c6917d54 It is done for all other KPIs using HZ, but was missed here. MFC after: 2 weeks 2020-02-11T18:48:07Z Gleb Smirnoff glebius@FreeBSD.org 2020-02-11T18:48:07Z urn:sha1:4426b2e64bd6ff5b05997d985140323b3db67085 When processing a taskqueue and a task has associated epoch, then enter for duration of the task. If consecutive tasks belong to the same epoch, batch them. Now we are talking about the network epoch only. Shrink the ta_priority size to 8-bits. No current consumers use a priority that won't fit into 8 bits. Also complexity of taskqueue_enqueue() is a square of maximum value of priority, so we unlikely ever want to go over UCHAR_MAX here. Reviewed by: hselasky Differential Revision: https://reviews.freebsd.org/D23518 2019-12-15T21:11:15Z Jeff Roberson jeff@FreeBSD.org 2019-12-15T21:11:15Z urn:sha1:61a74c5ccd65d1a00a96779f16eda8c41ff3a426 Eliminate recursion from most thread_lock consumers. Return from sched_add() without the thread_lock held. This eliminates unnecessary atomics and lock word loads as well as reducing the hold time for scheduler locks. This will eventually allow for lockless remote adds. Discussed with: kib Reviewed by: jhb Tested by: pho Differential Revision: https://reviews.freebsd.org/D22626 2019-11-01T22:49:44Z Alexander Motin mav@FreeBSD.org 2019-11-01T22:49:44Z urn:sha1:3db35ffa2a1822ca92a7e7ca7a1cc2094a398cef - Optimize enqueue for two task priority values by adding new tq_hint field, pointing to the last task inserted into the middle of the list. In case of more then two priority values it should halve average search. - Move tq_active insert/remove out of the taskqueue_run_locked loop. Instead of dirtying few shared cache lines per task introduce different mechanism to drain active tasks, based on task sequence number counter, that uses only cache lines already present in cache. Since the new mechanism does not need ordering, switch tq_active from TAILQ to LIST. - Move static and dynamic struct taskqueue fields into different cache lines. Move lock into its own cache line, so that heavy lock spinning by multiple waiting threads would not affect the running thread. - While there, correct some TQ_SLEEP() wait messages. This change fixes certain ZFS write workloads, causing huge congestion on taskqueue lock. Those workloads combine some large block writes to saturate the pool and trigger allocation throttling, which uses higher priority tasks to requeue the delayed I/Os, with many small blocks to generate deep queue of small tasks for taskqueue to sort. MFC after: 1 week Sponsored by: iXsystems, Inc. 2019-10-17T06:32:34Z Andriy Gapon avg@FreeBSD.org 2019-10-17T06:32:34Z urn:sha1:5fdc2c044eae307f854affc29224c3c941d7eb43 This can be used to group all threads belonging to a single logical entity under a common kernel process. I am planning to use the new interface for ZFS threads. MFC after: 4 weeks