aboutsummaryrefslogtreecommitdiff
path: root/sys/vm
Commit message (Collapse)AuthorAgeFilesLines
* posixshm: Fix handling of sendfile() with largepage objectsMark Johnston2026-06-301-10/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | sendfile(2) can transmit POSIX shared memory objects. Typically it will look up and wire each page before sending it to a socket; once transmission is complete, the page is unwired and typically released back into the page queues. sendfile() has an advisory flag, SF_NOCACHE, which means, "try to free the page once transmission is complete." This is implemented in vm_page_release(), which expects to operate on managed pages. Pages belonging a largepage object are de-facto wired not explicitly so. Thus, vm_page_release() will unwire and, having found no additional references, free the page. Because mappings of largepage objects are unmanaged, userspace can still access the now freed page. Fix the problem by explicitly wiring largepage pages. Make the VM object destructor responsible for unwiring and freeing them. Add a regression test. Approved by: so Security: FreeBSD-SA-26:44.posixshm Security: CVE-2026-49427 Reported by: Chris Jarrett-Davies <chrisjd@openai.com> Reviewed by: kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D57832
* device_pager: Avoid double-insertion of pages into the pager listMark Johnston2026-06-303-19/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Unmanaged device pager objects maintain a linked list of pages that were returned from the fault handler. Initially, such pages are valid, but msync(MS_INVALIDATE) can mark them invalid (and clean). They are not removed from page tables (since they are unmanaged), but a subsequent mlock() call can trigger a page fault that is handled by the pager. dev_pager_getpages() then re-inserts the page into the linked list even though it's already present there. This patch fixes the problem by removing the linked list. OBJ_PG_DTOR is set, so vm_object_terminate_pages() does nothing, and dev_pager_dealloc() instead handles cleanup of the object. Add a regression test case which triggers a queue.h assertion failure in unpatched kernels. Note, in stable branches we should avoid changing the layout of struct vm_object. Approved by: so Security: FreeBSD-SA-26:37.vm Security: CVE-2026-49418 Reported by: slidybat Reviewed by: kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D57743
* witness: harden tunables for large settingsRyan Libby2026-06-291-2/+9
| | | | | | | | | | | | | | | | | | | | | | | Harden witness against reasonable tunable settings causing boot panics or scribblers. For now, don't bother to harden this completely. Panics are still achievable by setting the tunables to negative or very large values. The recent change to respect the setting of debug.witness.witness_count (08180f1b613b) exposed that there are no guard rails on that setting, setting it to large but plausible values can cause integer overflow or cause attempted use of memory beyond what is actually available. Add some guard rails, fall back to default configuration if the tuned values are too large, and fall back to a minimum memory allocation if even the defaults are too large. Also don't allow witness to use over half of the memory segment, as there are other early allocators after witness too. Reported by: asomers Reviewed by: kib, markj (previous version) Fixes: 08180f1b613b ("witness: actually set read-only tunables in time for witness_startup") Sponsored by: Dell Inc. Differential Revision: https://reviews.freebsd.org/D57793
* vfs: rename vop_mmapped() to vop_update_atime()Konstantin Belousov2026-06-231-1/+1
| | | | | | | | | | | | | This reflects the actual functionality of the VOP. While there, add the explicit struct timespec argument for the VOP allowing the caller to set specific atime, not just request an update for it. Requested by: rmacklem Reviewed by: rmacklem Discussed with: jah Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D57681
* sys: Add td_kstack_top inline helper functionJohn Baldwin2026-05-271-1/+1
| | | | | | | | This function returns a pointer to the top of the kstack. Reviewed by: kib, andrew (arm changes) Sponsored by: AFRL, DARPA Pull Request: https://ron-dev.freebsd.org/FreeBSD/src/pulls/23
* arm64: VM/PMAP changes for CCA guest supportSarah Walker2026-05-121-0/+8
| | | | | | | | | | | | | | When in a realm: - Mappings with mode VM_MEMATTR_DEVICE and VM_MEMATTR_DEVICE_NP are unprotected - Imported busdma buffers in protected memory are always bounced - If EARLY_PRINTK is in use, the UART physical address must be in the unprotected address space Reviewed by: andrew Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D56599
* vm: Add flags for unprotected allocationsSarah Walker2026-05-122-3/+21
| | | | | | | | | Unprotected allocations are intended to be accessible outside of the current VM on systems such as Arm CCA. Reviewed by: markj Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D56518
* vm_map_growstack(): use local for p->p_vmspaceKonstantin Belousov2026-05-071-2/+1
| | | | | | Noted by: alc Sponsored by: The FreeBSD Foundation MFC after: 1 week
* vm_map_growstack(): give a hint to user that stack was blown outKonstantin Belousov2026-05-071-2/+15
| | | | | | | Reviewed by: markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D56863
* vm_map_growstack(): consistently use local vars instead of curthread/procKonstantin Belousov2026-05-071-6/+8
| | | | | | | Reviewed by: markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D56863
* execve: Add guard pages around execve KVA buffersMark Johnston2026-05-044-10/+40
| | | | | | | | | | | | | | This helps ensure that overflows will trigger a panic instead of silently corrupting adjacent buffers, as happened in SA-26:13.exec. Extend kmap_alloc_wait() to support allocation of guard pages on both sides of a KVA allocation. Modify the exec_map setup accordingly. Add the "vm.exec_map_guard_pages" tunable to provide control over the guard page allocations. Reviewed by: kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D56711
* VM_PAGE_TO_DMAP: Wrapper macro to return direct map address for a pageJohn Baldwin2026-04-232-6/+3
| | | | | | | Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* DMAP_TO_VM_PAGE: Wrapper macro to map direct map address to a pageJohn Baldwin2026-04-231-0/+2
| | | | | | | | Effort: CHERI upstreaming Suggested by: kib Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* kmap_alloc_wait/kmap_free_wakeup: Use void * instead of vm_offset_tJohn Baldwin2026-04-232-6/+8
| | | | | | | Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* PHYS_TO_DMAP: Return a void * instead of a vm_offset_tJohn Baldwin2026-04-231-1/+1
| | | | | | | | | | Add a new PHYS_TO_DMAP_ADDR that still returns an address for use in places that only need an address and not a pointer. Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* sys: Permit passing pointers to VIRT_IN_DMAP and DMAP_TO_PHYSJohn Baldwin2026-04-232-2/+2
| | | | | | | | | | | | Add explicit uintptr_t casts to the arguments to these macros so that the work both with virtual addresses (e.g. vm_offset_t) and pointers. Drop no-longer-needed casts in various invocations of DMAP_TO_PHYS. Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* pmap_map: Use void * for the return value instead of vm_offset_tJohn Baldwin2026-04-234-7/+7
| | | | | | | Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* sys: Switch td_kstack from vm_offset_t to char *John Baldwin2026-04-231-6/+6
| | | | | | | Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* vm_kstack: Use void * instead of vm_offset_tJohn Baldwin2026-04-231-35/+34
| | | | | | | Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* kva_alloc/free: Use void * instead of vm_offset_tJohn Baldwin2026-04-237-24/+26
| | | | | | | Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* pmap_qenter/qremove: Use void * instead of vm_offset_tJohn Baldwin2026-04-236-20/+21
| | | | | | | Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* sf_buf_kva: Return a pointer instead of a vm_offset_tJohn Baldwin2026-04-231-1/+1
| | | | | | | | | This removes the need for several casts to pointer in callers. Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* pmap_quick_(enter|remove)_page: Use void * instead of vm_offset_tJohn Baldwin2026-04-231-2/+2
| | | | | | | Effort: CHERI upstreaming Reviewed by: kib Sponsored by: AFRL, DARPA Pull Request: https://github.com/freebsd/freebsd-src/pull/2068
* pkru: Fix handling of 1GB largepage mappingsMark Johnston2026-04-212-0/+33
| | | | | | | | | | | | | | pmap_pkru_update_range() did not handle the case where a PDPE has PG_PS set. More generally, the SET_PKRU and CLEAR_PKRU sysarch implementations did not check whether the request covers a "boundary" vm map entry. Fix this, add the missing PG_PS test, and add some tests. Approved by: so Security: FreeBSD-SA-26:11.amd64 Security: CVE-2026-6386 Reported by: Nicholas Carlini <npc@anthropic.com> Reviewed by: kib, alc Differential Revision: https://reviews.freebsd.org/D56184
* vm_swapout: Remove a special case from vm_swapout_map_deactivate_pages()Mark Johnston2026-04-151-13/+0
| | | | | | | | | | | | | | John points out that this probably should have been removed in commit 472888018ce, which removed a special case where we'd set desired=0 if the target process has P_INMEM clear. It's not obvious to me that the desired=0 case can't arise by setting an RSS limit to 0, but I'm not sure why we'd try to go the extra mile in that case anyway. Reported by: jhb Reviewed by: kib MFC after: 2 weeks Fixes: 472888018ce1 ("proc: Remove kernel stack swapping support, part 6") Differential Revision: https://reviews.freebsd.org/D56140
* vm_fault: Reset m_needs_zeroing properlyMark Johnston2026-04-081-4/+3
| | | | | | | | | | | | | | - When allocating a page, we should only consider the PG_ZERO flag when handling the top-level page. - Unconditionally reset the flag when restarting the fault handler. Previously, vm_fault_busy_sleep() would fail to reset it. PR: 294039 Reviewed by: kib Tested by: Peter Much <pmc@citylink.dinoex.sub.org> MFC after: 3 days Fixes: cff67bc43df1 ("vm_fault: only rely on PG_ZERO when the page was newly allocated") Differential Revision: https://reviews.freebsd.org/D56234
* vm_fault: Avoid creating clean, writeable superpage mappingsMark Johnston2026-03-271-2/+18
| | | | | | | | | | | | | | | | | | | | | | | | | The pmap layer requires writeable superpage mappings to be dirty. Otherwise, during demotion, we may miss a hw update of the PDE which sets the dirty bit. When creating a managed superpage mapping without promotion, i.e., with pmap_enter(psind == 1), we must therefore ensure that a writeable mapping is created with the dirty bit pre-set. To that end, vm_fault_soft_fast(), when handling a map entry with write permissions, checks whether all constituent pages are dirty, and if so, converts the fault to a write fault, so that pmap_enter() does the right thing. If one or more pages is not dirty, we simply create a 4K mapping. vm_fault_populate(), which may also create superpage mappings, did not do this, and thus could create mappings which violate the invariant described above. Modify it to instead check whether all constituent pages are already dirty, and if so, convert the fault to a write fault. Otherwise the mapping is downgraded to read-only. Reported by: ashafer Reviewed by: alc, kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D55536
* Provide more precise error explanations for mmap(2) EINVAL errors.Kirk McKusick2026-03-191-13/+23
| | | | | | | Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D55888 MFC-after: 1 week Sponsored by: Netflix
* Delete error-check code that can never happen.Kirk McKusick2026-03-191-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | Near the top of kern_mmap() that implements the mmap(2) system call, it sets prot = PROT_EXTRACT(prot); with So prot can only be the three PROT_ flags. The following test of the user's mmap(2) parameters (near line 275 in vm/vm_mmap.c): if (prot != PROT_NONE && (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC)) != 0) { return (EXTERROR(EINVAL, "invalid prot %#jx", prot)); } can never fail. This commit deletes it. No functional change intended. Reviewed by: kib MFC-after: 1 week Sponsored by: Netflix
* vm_fault: expand KASSERT message in vm_fault_populate_cleanupisoux2026-03-161-1/+4
| | | | | | | | | | | Include index values and object pointer in the diagnostic to improve debugging of pindex mismatches. No functional change. Signed-off-by: Isa Isoux <isa@isoux.org> Reviewed by: kib, pouria Pull Request: https://github.com/freebsd/freebsd-src/pull/2038
* vm_page.h: Materialize a spare bitOlivier Certner2026-02-191-0/+1
| | | | | | | No functional change (intended). MFC after: 2 weeks Sponsored by: The FreeBSD Foundation
* vm_phys: Check `RB_FIND()` return value in case it is NULLJean-Sébastien Pédron2026-02-031-1/+1
| | | | | | | | | | | | | | | | | | | | | When trying to unregister a fictitious range in `vm_phys_fictitious_unreg_range()`, the function checks the properties of the looked up segment, but it does not check if a segment was found in the first place. This can happen with the amdgpu DRM driver which could call `vm_phys_fictitious_unreg_range()` without a fictitious range registered if the initialisation of the driver failed (for example because firmwares are unavailable). The code in the DRM driver was improved to avoid that, but `vm_phys_fictitious_unreg_range()` should still check the return value of `RB_FIND()` before trying to dereference the segment pointer and panic with a page fault. Reviewed by: emaste Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D55076
* sys: do not allow entering vm_fault() on boot until VM is initializedKonstantin Belousov2026-01-191-0/+8
| | | | | | | | | | | | On amd64, a hack sets td_critnest to 1 in hammer_time(), and then clear it before returning from hammer_time(), which is too early. Instead, set TDP_NOFAULTING for thread0, and clear the flag after vm_init() finished. Noted by: adrian Reviewed by: adrian (previous version), markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54768
* vm/swap_pager.c: silence compiler warningKonstantin Belousov2026-01-171-1/+1
| | | | | | | | | | Initialize pindex in swap_pager_getpages_locked() before the loop actually calculating it by skipping bogus pages. Compiler is worried that loop might never assign to it, which actually cannot happen. Sponsored by: The FreeBSD Foundation Fixes: d198ad51ea73bbb162336923a387f52b0b1c1f1d MFC after: 1 week
* swap_pager_getpages(): some pages from ma[] might be bogusKonstantin Belousov2026-01-171-8/+29
| | | | | | | | | | | | | | | | | | | | | | | Same as vnode_pager_generic_getpages_async(), swap_pager_getpages() must handle a possibility of the provided page run to include bogus_page on some positions, when called from sendfile_swapin(). The swap pager is used for tmpfs vnodes. In particular, the bogus page must not be used for pindex calculation, we better not update the flags on it or wait for the flag clearing, and we must not call vm_page_valid() because the function expects busy page. This was bisected down to 72ddb6de1028426 (unix: increase net.local.(stream|seqpacket).(recv|send)space to 64 KiB), which is somewhat surprising, but apparently reasonable because it allowed the run of more than one page for page-in from the swap pager, which now might include valid pages replaced by bogus one. In collaboration with: pho Reviewed by: glebius, markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54713
* swap_pager_getpages(): assert that bp->b_pages[] is accessed in boundsKonstantin Belousov2026-01-171-0/+3
| | | | | | | | Reviewed by: glebius, markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54713
* vm_map_entry_delete(): fix the calculation of swap releaseKonstantin Belousov2026-01-131-1/+1
| | | | | | | Reported and tested by: andrew Fixes: d160447129fe060b28bcd6ba429d17afdf494ff2 Sponsored by: The FreeBSD Foundation MFC after: 1 week
* vm_object: remove the charge memberKonstantin Belousov2026-01-135-54/+54
| | | | | | | | | | | | | | | | | | | | | | | | State that the object charge is zero if object->cred == NULL, or equal to the ptoa(object->size) otherwise. Besides being much simpler, the transition to use object->size corrects the architectural issue with the use of object->charge. The split operations effectively carve the holes in the charged regions, but single counter cannot properly express it. As result, coalescing anonymous mappings cannot calculate correctly if the extended mapping already backed by the existing object is already accounted or not [1]. To properly solve the issue, either we need to start tracking exact charged regions in the anonymous objects, which has the significant overhead and complications. Or give up on the slight over-accounting and charge the whole object unconditionally, as it is done in the patch. Reported by: mmel, pho [1] Reviewed by: markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54572
* rfork(2): fix swap accounting in vmspace_unshare()Konstantin Belousov2026-01-133-2/+16
| | | | | | | | | | | | | | | When an attempt to increase the swap charge for the ucred failed, we must forcibly increase the charge to allow the vmspace_destroy() operation to correctly un-charge the accumulated objects. Add a swap_reserve_force_by_cred() helper and use it in vmspace_unshare(), same as it is done in normal fork operations. Reviewed by: markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54572
* swap_release_by_cred*(): give some additional info on panics due to underflowKonstantin Belousov2026-01-131-2/+4
| | | | | | | | Reviewed by: markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54572
* vm_object.h: tweak OBJ_ONEMAPPING comment even moreKonstantin Belousov2026-01-071-3/+2
| | | | | | | | | | | The formulation that pages are allowed to have at most one mapping is not quite clear. Nothing prevents pages from having more than one mapping, but the flag must be cleared then. Reviewed by: alc, markj Sponsored by: The FreeBSD Foundation MFC after: 3 days Differential revision: https://reviews.freebsd.org/D54570
* linker: Reset DMAP protections in link_elf_unload_file()Mark Johnston2026-01-061-8/+0
| | | | | | | | | | | | | | | | On x86, when a preloaded kernel module is unloaded, we free the backing (physically contiguous) pages. The ET_REL linker will have adjusted protections on segments of the preloaded file, which updates the direct map, so the original protections must be restored when unloading the module. Previously this was handled in kmem_bootstrap_free(), but there is no apparent reason not to handle this within the kernel linker. Moreover, we were not resetting permissions in the kernel map on arm64. Reviewed by: alc, kib MFC after: 3 weeks Differential Revision: https://reviews.freebsd.org/D54438
* vm/vm_object.h: clarify the OBJ_ONEMAPPING semanticKonstantin Belousov2026-01-041-2/+3
| | | | | | | Reviewed by: markj Sponsored by: The FreeBSD Foundation MFC after: 3 days Differential revision: https://reviews.freebsd.org/D54459
* vm_object_coalesce(): return swap reservation back if overchargedKonstantin Belousov2025-12-303-25/+68
| | | | | | | | | | | | | | | | | | | | | | It is possible for both vm_map_insert() and vm_object_coalesce() to charge both for the same region. The issue is that vm_map_insert() must charge in advance to ensure that the mapping would not exceed the swap limit, but then the coalesce might decide to extend the object, and already (partially) backs the mapped region. Handle this by passing to vm_object_coalesce() exact information about the charging mode of the extending range 'not charging', 'charged' using flags instead of simple boolean. In vm_object_coalesce(), detect overcharge and undo it if needed. Note that this relies on vm_object_coalesce() call being the last action in vm_map_insert() before extending the previous map entry. Reported and tested by: pho Reviewed by: markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54338
* vm_object_coalesce(): do not account holes twiceKonstantin Belousov2025-12-301-4/+7
| | | | | | | | | | | | | | | | | | | | alc wrote: Suppose that the object is OBJ_ONEMAPPING and that we, in fact, have a single mapping to it. Then, we punch a hole in that mapping. vm_map_entry_delete() only subtracts from the object's charge when we shrink the size of the object. Now, suppose that we perform mmap(MAP_ANON) to reallocate some of the hole. Aren't we going to add to the charge here, even though vm_map_entry_delete() never subtracted from the charge for the hole that was created? Only account the change in the charged object size that was added to it. Noted by: alc Reviewed by: alc, markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54263
* vm_object_coalesce(): simplify common expressionKonstantin Belousov2025-12-301-6/+7
| | | | | | | | | | | | Add the next_end variable to replace commonly occuring the next_pindex + next_size expression. Suggested by: alc Reviewed by: alc, markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54263
* vm_object_coalesce(): remove commented out codeKonstantin Belousov2025-12-301-12/+1
| | | | | | | Reviewed by: alc, markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54263
* vm/vm_mmap.c: inline erronous argument values for extended errorsKonstantin Belousov2025-12-291-18/+25
| | | | | | | Reviewed by: emaste, mckusick Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D54380
* vnode_pager: clean up undirty_pages()Doug Moore2025-12-261-39/+37
| | | | | | | | | | The first (second) loop of vnode_pager_undirty_pages() includes an if-test to test whether an iteration is the last (first). Move those if-tests out of their loops. That allows the compiler to unroll the loops. Reviewed by: alc, kib Differential Revision: https://reviews.freebsd.org/D54353
* vnode_pager: use ptoa(), atop()Doug Moore2025-12-231-10/+10
| | | | | | | | In vnode_pager.c, use ptoa() or atop() instead of arithmetic with PAGE_SIZE or PAGE_SHIFT. Reviewed by: alc, kib Differential Revision: https://reviews.freebsd.org/D49330
generated by cgit v1.3 (git 2.53.0) at 2026-09-04 06:32:59 +0000