summaryrefslogtreecommitdiff
path: root/sys/vm/vm_page.h
Commit message (Collapse)AuthorAgeFilesLines
* vm_phys: Try to clean up NUMA KPIsMark Johnston2020-11-191-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | It can useful for code outside the VM system to look up the NUMA domain of a page backing a virtual or physical address, specifically when creating NUMA-aware data structures. We have _vm_phys_domain() for this, but the leading underscore implies that it's an internal function, and vm_phys.h has dependencies on a number of other headers. Rename vm_phys_domain() to vm_page_domain(), and _vm_phys_domain() to vm_phys_domain(). Make the latter an inline function. Add _vm_phys.h and define struct vm_phys_seg there so that it's easier to use in other headers. Include it from vm_page.h so that vm_page_domain() can be defined there. Include machine/vmparam.h from _vm_phys.h since it depends directly on some constants defined there. Reviewed by: alc Reviewed by: dougm, kib (earlier versions) Differential Revision: https://reviews.freebsd.org/D27207 Notes: svn path=/head/; revision=367828
* Avoid dump_avail[] redefinition.Konstantin Belousov2020-10-141-65/+0
| | | | | | | | | | | | | Move dump_avail[] extern declaration and inlines into a new header vm/vm_dumpset.h. This fixes default gcc build for mips. Reviewed by: alc, scottph Tested by: kevans (previous version) Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D26741 Notes: svn path=/head/; revision=366711
* Use unlocked page lookup for inmem() to avoid object lock contentionBryan Drewery2020-10-091-0/+1
| | | | | | | | | | Reviewed By: kib, markj Submitted by: mlaier Sponsored by: Dell EMC Differential Revision: https://reviews.freebsd.org/D26653 Notes: svn path=/head/; revision=366594
* vm_page_dump_index_to_pa(): Add braces to the expression involving + and &.Konstantin Belousov2020-10-081-1/+1
| | | | | | | | | | | | | | | | The precedence of the '&' operator is less than of '+'. Added braces do change the order of evaluation into the natural one, in my opinion. On the other hand, the value of the expression should not change since all elements should have page-aligned values. This fixes a gcc warning reported. Reported by: adrian Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=366552
* Sparsify the vm_page_dump bitmapD Scott Phillips2020-09-211-3/+46
| | | | | | | | | | | | | | | | | | | On Ampere Altra systems, the sparse population of RAM within the physical address space causes the vm_page_dump bitmap to be much larger than necessary, increasing the size from ~8 Mib to > 2 Gib (and overflowing `int` for the size). Changing the page dump bitmap also changes the minidump file format, so changes are also necessary in libkvm. Reviewed by: jhb Approved by: scottl (implicit) MFC after: 1 week Sponsored by: Ampere Computing, Inc. Differential Revision: https://reviews.freebsd.org/D26131 Notes: svn path=/head/; revision=365978
* Move vm_page_dump bitset array definition to MI codeD Scott Phillips2020-09-211-0/+22
| | | | | | | | | | | | | | | | | | | These definitions were repeated by all architectures, with small variations. Consolidate the common definitons in machine independent code and use bitset(9) macros for manipulation. Many opportunities for deduplication remain in the machine dependent minidump logic. The only intended functional change is increasing the bit index type to vm_pindex_t, allowing the indexing of pages with address of 8 TiB and greater. Reviewed by: kib, markj Approved by: scottl (implicit) MFC after: 1 week Sponsored by: Ampere Computing, Inc. Differential Revision: https://reviews.freebsd.org/D26129 Notes: svn path=/head/; revision=365977
* Add support for multithreading the inactive queue pageout within a domain.Conrad Meyer2020-08-111-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In very high throughput workloads, the inactive scan can become overwhelmed as you have many cores producing pages and a single core freeing. Since Mark's introduction of batched pagequeue operations, we can now run multiple inactive threads working on independent batches. To avoid confusing the pid and other control algorithms, I (Jeff) do this in a mpi-like fan out and collect model that is driven from the primary page daemon. It decides whether the shortfall can be overcome with a single thread and if not dispatches multiple threads and waits for their results. The heuristic is based on timing the pageout activity and averaging a pages-per-second variable which is exponentially decayed. This is visible in sysctl and may be interesting for other purposes. I (Jeff) have verified that this does indeed double our paging throughput when used with two threads. With four we tend to run into other contention problems. For now I would like to commit this infrastructure with only a single thread enabled. The number of worker threads per domain can be controlled with the 'vm.pageout_threads_per_domain' tunable. Submitted by: jeff (earlier version) Discussed with: markj Tested by: pho Sponsored by: probably Netflix (based on contemporary commits) Differential Revision: https://reviews.freebsd.org/D21629 Notes: svn path=/head/; revision=364129
* Remove most lingering references to the page lock in comments.Mark Johnston2020-08-041-19/+16
| | | | | | | | | | | | | Finish updating comments to reflect new locking protocols introduced over the past year. In particular, vm_page_lock is now effectively unused. Reviewed by: kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D25868 Notes: svn path=/head/; revision=363840
* Remove the volatile qualifier from busy_lock.Mark Johnston2020-07-291-9/+17
| | | | | | | | | | | | | Use atomic(9) to load the lock state. Some places were doing this already, so it was inconsistent. In initialization code, the lock state is still initialized with plain stores. Reviewed by: alc, kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D25861 Notes: svn path=/head/; revision=363671
* vm_page_xbusy_claim(): Use atomics to update busy lock state.Mark Johnston2020-07-281-1/+6
| | | | | | | | | | | | | | | | | | | vm_page_xbusy_claim() could clobber the waiter bit. For its original use, kernel memory pages, this was not a problem since nothing would ever block on the busy lock for such pages. r363607 introduced a new use where this could in principle be a problem. Fix the problem by using atomic_cmpset to update the lock owner. Since this macro is defined only for INVARIANTS kernels the extra overhead doesn't seem prohibitive. Reported by: vangyzen Reviewed by: alc, kib, vangyzen Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D25859 Notes: svn path=/head/; revision=363654
* Add a new function vm_page_free_invalid() for freeing invalid pagesChuck Silvers2020-07-171-0/+1
| | | | | | | | | | | | that might be wired. If the page is wired then it cannot be freed now, but the thread that eventually unwires it will free it at that point. Reviewed by: markj, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D25430 Notes: svn path=/head/; revision=363295
* Revert r362998, r326999 while a better compatibility strategy is devised.Scott Long2020-07-091-1/+1
| | | | Notes: svn path=/head/; revision=363060
* Migrate the feature of excluding RAM pages to use "excludelist"Scott Long2020-07-071-1/+1
| | | | | | | | | as its nomenclature. MFC after: 1 week Notes: svn path=/head/; revision=362998
* Re-check for wirings after busying the page in vm_page_release_locked().Mark Johnston2020-04-281-2/+2
| | | | | | | | | | | | | | | | | | | | A concurrent unlocked lookup can wire the page after vm_page_release_locked() releases the last wiring, in which case vm_page_release_locked() must not free the page. Once the xbusy lock is acquired, that, the object lock and the fact that the page is unmapped ensure that the wire count cannot increase, so re-check for new wirings after the page is xbusied. Update the comment above vm_page_wired() to reflect the new synchronization rules. Reported by: glebius Reviewed by: alc, jeff, kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D24592 Notes: svn path=/head/; revision=360436
* Provide a lock free alternative to resolve bogus pages. This is not likelyJeff Roberson2020-02-281-0/+1
| | | | | | | | | | to be much of a perf win, just a nice code simplification. Reviewed by: markj, kib Differential Revision: https://reviews.freebsd.org/D23866 Notes: svn path=/head/; revision=358451
* Add unlocked grab* function variants that use lockless radix code toJeff Roberson2020-02-271-2/+7
| | | | | | | | | | | lookup pages. These variants will fall back to their locked counterparts if the page is not present. Discussed with: kib, markj Differential Revision: https://reviews.freebsd.org/D23449 Notes: svn path=/head/; revision=358363
* Don't release xbusy on kmem pages. After lockless page lookup we will notJeff Roberson2020-02-191-0/+5
| | | | | | | | | | | be able to guarantee that they can be racquired without blocking. Reviewed by: kib Discussed with: markj Differential Revision: https://reviews.freebsd.org/D23506 Notes: svn path=/head/; revision=358098
* Refactor _vm_page_busy_sleep to reduce the delta between the variousJeff Roberson2020-02-171-0/+2
| | | | | | | | | | sleep routines and introduce a variant that supports lockless sleep. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D23612 Notes: svn path=/head/; revision=358011
* Add an explicit busy state for free pages. This improves behavior withJeff Roberson2020-02-041-3/+11
| | | | | | | | | | | potential bugs that access freed pages as well as providing a path towards lockless page lookup. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D23444 Notes: svn path=/head/; revision=357528
* Remove a couple of lingering usages of the page lock.Mark Johnston2020-02-011-1/+0
| | | | | | | | | | | | | | | | | | Update vm_page_scan_contig() and vm_page_reclaim_run() to stop using vm_page_change_lock(). It has no use after r356157. Remove vm_page_change_lock() now that it has no users. Remove an unncessary check for wirings in vm_page_scan_contig(), which was previously checking twice. The check is racy until vm_page_reclaim_run() ensures that the page is unmapped, so one check is sufficient. Reviewed by: jeff, kib (previous versions) Tested by: pho (previous version) Differential Revision: https://reviews.freebsd.org/D23279 Notes: svn path=/head/; revision=357374
* Remove some unused functions.Mark Johnston2019-12-281-27/+0
| | | | | | | | | | | | The previous series of patches orphaned some vm_page functions, so remove them. Reviewed by: dougm, kib Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D22886 Notes: svn path=/head/; revision=356159
* Update the vm_page.h block comment to reflect recent changes.Mark Johnston2019-12-281-28/+26
| | | | | | | | | | | Explain the new locking rules for per-page queue state updates. Reviewed by: jeff, kib Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D22884 Notes: svn path=/head/; revision=356158
* Remove page locking for queue operations.Mark Johnston2019-12-281-8/+0
| | | | | | | | | | | | | | | | With the previous reviews, the page lock is no longer required in order to perform queue operations on a page. It is also no longer needed in the page queue scans. This change effectively eliminates remaining uses of the page lock and also the false sharing caused by multiple pages sharing a page lock. Reviewed by: jeff Tested by: pho Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D22885 Notes: svn path=/head/; revision=356157
* Start implementing queue state updates using fcmpset loops.Mark Johnston2019-12-281-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is in preparation for eliminating the use of the vm_page lock for protecting queue state operations. Introduce the vm_page_pqstate_commit_*() functions. These functions act as helpers around vm_page_astate_fcmpset() and are specialized for specific types of operations. vm_page_pqstate_commit() wraps these functions. Convert a number of routines to use these new helpers. Use vm_page_release_toq() in vm_page_unwire() and vm_page_release() to atomically release a wiring reference and release the page into a queue. This has the side effect that vm_page_unwire() will leave the page in the active queue if it is already present there. Convert the page queue scans to use the new helpers. Simplify vm_pageout_reinsert_inactive(), which requeues pages that were found to be busy during an inactive queue scan, to avoid duplicating the work of vm_pqbatch_process_page(). In particular, if PGA_REQUEUE or PGA_REQUEUE_HEAD is set, let that be handled during batch processing. Reviewed by: jeff Tested by: pho Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D22770 Differential Revision: https://reviews.freebsd.org/D22771 Differential Revision: https://reviews.freebsd.org/D22772 Differential Revision: https://reviews.freebsd.org/D22773 Differential Revision: https://reviews.freebsd.org/D22776 Notes: svn path=/head/; revision=356155
* Make page busy state deterministic on free. Pages must be xbusy whenJeff Roberson2019-12-221-19/+3
| | | | | | | | | | | | | | | | | | | removed from objects including calls to free. Pages must not be xbusy when freed and not on an object. Strengthen assertions to match these expectations. In practice very little code had to change busy handling to meet these rules but we can now make stronger guarantees to busy holders and avoid conditionally dropping busy in free. Refine vm_page_remove() and vm_page_replace() semantics now that we have stronger guarantees about busy state. This removes redundant and potentially problematic code that has proliferated. Discussed with: markj Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D22822 Notes: svn path=/head/; revision=356002
* Fix the aflag shift on big-endian platforms after r355672.Mark Johnston2019-12-181-1/+5
| | | | | | | | | | The structure offset is zero regardless of endianness. Reported by: brooks Pointy hat: markj Notes: svn path=/head/; revision=355874
* Add a deferred free mechanism for freeing swap space that does not requireJeff Roberson2019-12-151-0/+7
| | | | | | | | | | | | | | | | | | | | | | an exclusive object lock. Previously swap space was freed on a best effort basis when a page that had valid swap was dirtied, thus invalidating the swap copy. This may be done inconsistently and requires the object lock which is not always convenient. Instead, track when swap space is present. The first dirty is responsible for deleting space or setting PGA_SWAP_FREE which will trigger background scans to free the swap space. Simplify the locking in vm_fault_dirty() now that we can reliably identify the first dirty. Discussed with: alc, kib, markj Differential Revision: https://reviews.freebsd.org/D22654 Notes: svn path=/head/; revision=355765
* Avoid relying on silent type casting in the native atomic_load_32.Mark Johnston2019-12-121-1/+1
| | | | | | | Reported by: np Notes: svn path=/head/; revision=355680
* Implement atomic state updates using the new vm_page_astate_t structure.Mark Johnston2019-12-121-43/+36
| | | | | | | | | | | | | | | | | Introduce primitives vm_page_astate_load() and vm_page_astate_fcmpset() to operate on the 32-bit per-page atomic state. Modify vm_page_pqstate_fcmpset() to use them. No functional change intended. Introduce PGA_QUEUE_OP_MASK, a subset of PGA_QUEUE_STATE_MASK that only includes queue operation flags. This will be used in subsequent patches. Reviewed by: alc, jeff, kib Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D22753 Notes: svn path=/head/; revision=355672
* Introduce vm_page_astate.Mark Johnston2019-12-101-13/+20
| | | | | | | | | | | | | | | | | This is a 32-bit structure embedded in each vm_page, consisting mostly of page queue state. The use of a structure makes it easy to store a snapshot of a page's queue state in a stack variable and use cmpset loops to update that state without requiring the page lock. This change merely adds the structure and updates references to atomic state fields. No functional change intended. Reviewed by: alc, jeff, kib Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D22650 Notes: svn path=/head/; revision=355586
* Reduce duplication in grab functions by providing allocflags based inlines.Jeff Roberson2019-12-081-1/+1
| | | | | | | | Reviewed by: kib, markj Differential Revision: https://reviews.freebsd.org/D22635 Notes: svn path=/head/; revision=355511
* Garbage collect the mostly unused us_keg field. Use appropriately namedJeff Roberson2019-11-281-1/+4
| | | | | | | | | | | union members in vm_page.h to store the zone and slab. Remove some nearby dead code. Reviewed by: markj Differential Revision: https://reviews.freebsd.org/D22564 Notes: svn path=/head/; revision=355169
* Record part of the owner struct thread pointer into busy_lock.Konstantin Belousov2019-11-241-5/+26
| | | | | | | | | | | | | | | | | | | | | Record as much bits from curthread into busy_lock as fits. Low bits for struct thread * representation are zero due to struct and zone alignment, and they leave space for busy flags (perhaps except statically allocated thread0). Upper bits are not very interesting for assert, and in most practical situations recorded value should allow to manually identify the owner with certainity. Assert that unbusy is performed by the owner, except few places where unbusy is done in io completion handler. For this case, add _unchecked variants of asserts and unbusy primitives. Reviewed by: markj (previous version) Tested by: pho Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D22298 Notes: svn path=/head/; revision=355062
* Remove unnecessary object locking from the vnode pager. Recent changes toJeff Roberson2019-11-191-0/+1
| | | | | | | | | | busy/valid/dirty locking make these acquires redundant. Reviewed by: kib, markj Differential Revision: https://reviews.freebsd.org/D22186 Notes: svn path=/head/; revision=354870
* Widen the vm_page aflags field to 16 bits.Mark Johnston2019-11-181-22/+22
| | | | | | | | | | | | | | | | | | | | We are now out of aflags bits, whereas the "flags" field only makes use of five of its sixteen bits, so narrow "flags" to eight bits. I have no intention of adding a new aflag in the near future, but would like to combine the aflags, queue and act_count fields into a single atomically updated word. This will allow vm_page_pqstate_cmpset() to become much simpler and is a step towards eliminating the use of the page lock array in updating per-page queue state. The change modifies the layout of struct vm_page, so bump __FreeBSD_version. Reviewed by: alc, dougm, jeff, kib Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D22397 Notes: svn path=/head/; revision=354820
* Remove page locking from pmap_mincore().Mark Johnston2019-10-161-1/+0
| | | | | | | | | | | | | | | | | | After r352110 the page lock no longer protects a page's identity, so there is no purpose in locking the page in pmap_mincore(). Instead, if vm.mincore_mapped is set to the non-default value of 0, re-lookup the page after acquiring its object lock, which holds the page's identity stable. The change removes the last callers of vm_page_pa_tryrelock(), so remove it. Reviewed by: kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21823 Notes: svn path=/head/; revision=353670
* (5/6) Move the VPO_NOSYNC to PGA_NOSYNC to eliminate the dependency on theJeff Roberson2019-10-151-1/+3
| | | | | | | | | | | | object lock in vm_page_set_validclean(). Reviewed by: kib, markj Tested by: pho Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D21595 Notes: svn path=/head/; revision=353540
* (4/6) Protect page valid with the busy lock.Jeff Roberson2019-10-151-39/+82
| | | | | | | | | | | | | | Atomics are used for page busy and valid state when the shared busy is held. The details of the locking protocol and valid and dirty synchronization are in the updated vm_page.h comments. Reviewed by: kib, markj Tested by: pho Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D21594 Notes: svn path=/head/; revision=353539
* (3/6) Add a shared object busy synchronization mechanism that blocks new pageJeff Roberson2019-10-151-8/+5
| | | | | | | | | | | | | | | | | busy acquires while held. This allows code that would need to acquire and release a very large number of page busy locks to use the old mechanism where busy is only checked and not held. This comes at the cost of false positives but never false negatives which the single consumer, vm_fault_soft_fast(), handles. Reviewed by: kib Tested by: pho Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D21592 Notes: svn path=/head/; revision=353538
* (1/6) Replace busy checks with acquires where it is trival to do so.Jeff Roberson2019-10-151-0/+1
| | | | | | | | | | | | | | This is the first in a series of patches that promotes the page busy field to a first class lock that no longer requires the object lock for consistency. Reviewed by: kib, markj Tested by: pho Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D21548 Notes: svn path=/head/; revision=353535
* Complete the removal of the "wire_count" field from struct vm_page.Mark Johnston2019-09-251-4/+4
| | | | | | | | | | | | Convert all remaining references to that field to "ref_count" and update comments accordingly. No functional change intended. Reviewed by: alc, kib Sponsored by: Intel, Netflix Differential Revision: https://reviews.freebsd.org/D21768 Notes: svn path=/head/; revision=352688
* Assert that the refcount value is not VPRC_BLOCKED in vm_page_drop().Mark Johnston2019-09-161-1/+5
| | | | | | | | | | | | | | | VPRC_BLOCKED is a refcount flag used to indicate that a thread is tearing down mappings of a page. When set, it causes attempts to wire a page via a pmap lookup to fail. It should never represent the last reference to a page, so assert this. Suggested by: kib Reviewed by: alc, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21639 Notes: svn path=/head/; revision=352411
* Fix a race in vm_page_dequeue_deferred_free() after r352110.Mark Johnston2019-09-161-2/+0
| | | | | | | | | | | | | | | | This function loaded the page's queue index before setting PGA_DEQUEUE. In this window the page daemon may have deactivated the page, updating its queue index. Make the operation atomic using vm_page_pqstate_cmpset(); the page daemon will not modify the page once it observes that PGA_DEQUEUE is set. Reported and tested by: pho Reviewed by: alc, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21639 Notes: svn path=/head/; revision=352410
* Revert r352406, which contained changes I didn't intend to commit.Mark Johnston2019-09-161-75/+82
| | | | Notes: svn path=/head/; revision=352407
* Fix a couple of nits in r352110.Mark Johnston2019-09-161-82/+75
| | | | | | | | | | | | | - Remove a dead variable from the amd64 pmap_extract_and_hold(). - Fix grammar in the vm_page_wire man page. Reported by: alc Reviewed by: alc, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21639 Notes: svn path=/head/; revision=352406
* Replace redundant code with a few new vm_page_grab facilities:Jeff Roberson2019-09-101-0/+4
| | | | | | | | | | | | | | - VM_ALLOC_NOCREAT will grab without creating a page. - vm_page_grab_valid() will grab and page in if necessary. - vm_page_busy_acquire() automates some busy acquire loops. Discussed with: alc, kib, markj Tested by: pho (part of larger branch) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21546 Notes: svn path=/head/; revision=352176
* Use the sleepq lock rather than the page lock to protect against wakeupJeff Roberson2019-09-101-2/+1
| | | | | | | | | | | | | races with page busy state. The object lock is still used as an interlock to ensure that the identity stays valid. Most callers should use vm_page_sleep_if_busy() to handle the locking particulars. Reviewed by: alc, kib, markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21255 Notes: svn path=/head/; revision=352174
* Change synchonization rules for vm_page reference counting.Mark Johnston2019-09-091-24/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are several mechanisms by which a vm_page reference is held, preventing the page from being freed back to the page allocator. In particular, holding the page's object lock is sufficient to prevent the page from being freed; holding the busy lock or a wiring is sufficent as well. These references are protected by the page lock, which must therefore be acquired for many per-page operations. This results in false sharing since the page locks are external to the vm_page structures themselves and each lock protects multiple structures. Transition to using an atomically updated per-page reference counter. The object's reference is counted using a flag bit in the counter. A second flag bit is used to atomically block new references via pmap_extract_and_hold() while removing managed mappings of a page. Thus, the reference count of a page is guaranteed not to increase if the page is unbusied, unmapped, and the object's write lock is held. As a consequence of this, the page lock no longer protects a page's identity; operations which move pages between objects are now synchronized solely by the objects' locks. The vm_page_wire() and vm_page_unwire() KPIs are changed. The former requires that either the object lock or the busy lock is held. The latter no longer has a return value and may free the page if it releases the last reference to that page. vm_page_unwire_noq() behaves the same as before; the caller is responsible for checking its return value and freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is introduced for use in pmap_extract_and_hold(). It fails if the page is concurrently being unmapped, typically triggering a fallback to the fault handler. vm_page_wire() no longer requires the page lock and vm_page_unwire() now internally acquires the page lock when releasing the last wiring of a page (since the page lock still protects a page's queue state). In particular, synchronization details are no longer leaked into the caller. The change excises the page lock from several frequently executed code paths. In particular, vm_object_terminate() no longer bounces between page locks as it releases an object's pages, and direct I/O and sendfile(SF_NOCACHE) completions no longer require the page lock. In these latter cases we now get linear scalability in the common scenario where different threads are operating on different files. __FreeBSD_version is bumped. The DRM ports have been updated to accomodate the KPI changes. Reviewed by: jeff (earlier version) Tested by: gallatin (earlier version), pho Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20486 Notes: svn path=/head/; revision=352110
* Add preliminary support for atomic updates of per-page queue state.Mark Johnston2019-09-031-33/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Queue operations on a page use the page lock when updating the page to reflect the desired queue state, and the page queue lock when physically enqueuing or dequeuing a page. Multiple pages share a given page lock, but queue state is per-page; this false sharing results in heavy lock contention. Take a small step towards the use of atomic_cmpset to synchronize updates to per-page queue state by introducing vm_page_pqstate_cmpset() and using it in the page daemon. In the longer term the plan is to stop using the page lock to protect page identity and rely only on the object and page busy locks. However, since the page daemon avoids acquiring the object lock except when necessary, some synchronization with a concurrent free of the page is required. vm_page_pqstate_cmpset() can be used to ensure that queue state updates are successful only if the page is not scheduled for a dequeue, which is sufficient for the page daemon. Add vm_page_swapqueue(), which moves a page from one queue to another using vm_page_pqstate_cmpset(). Use it in the active queue scan, which does not use the object lock. Modify vm_page_dequeue_deferred() to use vm_page_pqstate_cmpset() as well. Reviewed by: kib Discussed with: jeff Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21257 Notes: svn path=/head/; revision=351743
* Make vm_pqbatch_submit_page() externally visible.Mark Johnston2019-08-231-1/+2
| | | | | | | | | | | | | | | It will become useful for the page daemon to be able to directly create a batch queue entry for a page, and without modifying the page structure. Rename vm_pqbatch_submit_page() to vm_page_pqbatch_submit() to keep the namespace consistent. No functional change intended. Reviewed by: alc, kib MFC after: 1 week Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21369 Notes: svn path=/head/; revision=351436
generated by cgit v1.2.3 (git 2.25.1) at 2025-08-06 10:20:57 +0000