aboutsummaryrefslogtreecommitdiff
path: root/sys/dev/drm2/ttm/ttm_bo_vm.c
Commit message (Collapse)AuthorAgeFilesLines
* drm2/ttm: Add missing includes to fix arm TEGRA124 buildJessica Clarke2025-05-311-1/+3
| | | | Fixes: bcd85e0150c9 ("vm_page: make iter_insert() public")
* vm_page: make iter_insert() publicDoug Moore2025-05-091-3/+6
| | | | | | | | | In places where vm_page_insert() is used after lookups, or for consecutive pages, use vm_page_iter_insert instead, to exploit locality. Reviewed by: kib, markj Differential Revision: https://reviews.freebsd.org/D50248
* device_pager: user iterators to free device pagesDoug Moore2024-11-211-17/+3
| | | | | | | | | | | Change cdev_mgtdev_page_free_page to take an iterator, rather than an object and page, so that removing the page from the object radix tree can take advantage of locality with iterators. Define a general-purpose function to free all pages, which can be used in several places. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D47692
* dev_pager: define free_page for mgt devicesDoug Moore2024-08-211-1/+1
| | | | | | | | | Callers of cdev_pager_free_page in the kernel always have object->type == OBJT_MGTDEVICE. Define a function for them to call that skips the runtime type check in cdev_pager_free. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D46389
* sys: Remove $FreeBSD$: one-line .c patternWarner Losh2023-08-161-2/+0
| | | | Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
* Make page busy state deterministic on free. Pages must be xbusy whenJeff Roberson2019-12-221-0/+2
| | | | | | | | | | | | | | | | | | | removed from objects including calls to free. Pages must not be xbusy when freed and not on an object. Strengthen assertions to match these expectations. In practice very little code had to change busy handling to meet these rules but we can now make stronger guarantees to busy holders and avoid conditionally dropping busy in free. Refine vm_page_remove() and vm_page_replace() semantics now that we have stronger guarantees about busy state. This removes redundant and potentially problematic code that has proliferated. Discussed with: markj Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D22822 Notes: svn path=/head/; revision=356002
* (4/6) Protect page valid with the busy lock.Jeff Roberson2019-10-151-1/+1
| | | | | | | | | | | | | | Atomics are used for page busy and valid state when the shared busy is held. The details of the locking protocol and valid and dirty synchronization are in the updated vm_page.h comments. Reviewed by: kib, markj Tested by: pho Sponsored by: Netflix, Intel Differential Revision: https://reviews.freebsd.org/D21594 Notes: svn path=/head/; revision=353539
* Replace redundant code with a few new vm_page_grab facilities:Jeff Roberson2019-09-101-4/+3
| | | | | | | | | | | | | | - VM_ALLOC_NOCREAT will grab without creating a page. - vm_page_grab_valid() will grab and page in if necessary. - vm_page_busy_acquire() automates some busy acquire loops. Discussed with: alc, kib, markj Tested by: pho (part of larger branch) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21546 Notes: svn path=/head/; revision=352176
* Use the sleepq lock rather than the page lock to protect against wakeupJeff Roberson2019-09-101-4/+1
| | | | | | | | | | | | | races with page busy state. The object lock is still used as an interlock to ensure that the identity stays valid. Most callers should use vm_page_sleep_if_busy() to handle the locking particulars. Reviewed by: alc, kib, markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21255 Notes: svn path=/head/; revision=352174
* Change synchonization rules for vm_page reference counting.Mark Johnston2019-09-091-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are several mechanisms by which a vm_page reference is held, preventing the page from being freed back to the page allocator. In particular, holding the page's object lock is sufficient to prevent the page from being freed; holding the busy lock or a wiring is sufficent as well. These references are protected by the page lock, which must therefore be acquired for many per-page operations. This results in false sharing since the page locks are external to the vm_page structures themselves and each lock protects multiple structures. Transition to using an atomically updated per-page reference counter. The object's reference is counted using a flag bit in the counter. A second flag bit is used to atomically block new references via pmap_extract_and_hold() while removing managed mappings of a page. Thus, the reference count of a page is guaranteed not to increase if the page is unbusied, unmapped, and the object's write lock is held. As a consequence of this, the page lock no longer protects a page's identity; operations which move pages between objects are now synchronized solely by the objects' locks. The vm_page_wire() and vm_page_unwire() KPIs are changed. The former requires that either the object lock or the busy lock is held. The latter no longer has a return value and may free the page if it releases the last reference to that page. vm_page_unwire_noq() behaves the same as before; the caller is responsible for checking its return value and freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is introduced for use in pmap_extract_and_hold(). It fails if the page is concurrently being unmapped, typically triggering a fallback to the fault handler. vm_page_wire() no longer requires the page lock and vm_page_unwire() now internally acquires the page lock when releasing the last wiring of a page (since the page lock still protects a page's queue state). In particular, synchronization details are no longer leaked into the caller. The change excises the page lock from several frequently executed code paths. In particular, vm_object_terminate() no longer bounces between page locks as it releases an object's pages, and direct I/O and sendfile(SF_NOCACHE) completions no longer require the page lock. In these latter cases we now get linear scalability in the common scenario where different threads are operating on different files. __FreeBSD_version is bumped. The DRM ports have been updated to accomodate the KPI changes. Reviewed by: jeff (earlier version) Tested by: gallatin (earlier version), pho Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20486 Notes: svn path=/head/; revision=352110
* Add a return value to vm_page_remove().Mark Johnston2019-06-261-1/+1
| | | | | | | | | | | | | | | | | | Use it to indicate whether the page may be safely freed following its removal from the object. Also change vm_page_remove() to assume that the page's object pointer is non-NULL, and have callers perform this check instead. This is a step towards an implementation of an atomic reference counter for each physical page structure. Reviewed by: alc, dougm, kib MFC after: 1 week Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20758 Notes: svn path=/head/; revision=349432
* Revert drm2 removal.Warner Losh2018-08-241-0/+563
| | | | | | | | | | | | | | | | | | | | | | | | | | | Revert r338177, r338176, r338175, r338174, r338172 After long consultations with re@, core members and mmacy, revert these changes. Followup changes will be made to mark them as deprecated and prent a message about where to find the up-to-date driver. Followup commits will be made to make this clear in the installer. Followup commits to reduce POLA in ways we're still exploring. It's anticipated that after the freeze, this will be removed in 13-current (with the residual of the drm2 code copied to sys/arm/dev/drm2 for the TEGRA port's use w/o the intel or radeon drivers). Due to the impending freeze, there was no formal core vote for this. I've been talking to different core members all day, as well as Matt Macey and Glen Barber. Nobody is completely happy, all are grudgingly going along with this. Work is in progress to mitigate the negative effects as much as possible. Requested by: re@ (gjb, rgrimes) Notes: svn path=/head/; revision=338285
* Remove legacy drm and drm2 from treeMatt Macy2018-08-221-563/+0
| | | | | | | | | | | | | | | | | As discussed on the MLs drm2 conflicts with the ports' version and there is no upstream for most if not all of drm. Both have been merged in to a single port. Users on powerpc, 32-bit hardware, or with GPUs predating Radeon and i915 will need to install the graphics/drm-legacy-kmod. All other users should be able to use one of the LinuxKPI-based ports: graphics/drm-stable-kmod, graphics/drm-next-kmod, graphics/drm-devel-kmod. MFC: never Approved by: core@ Notes: svn path=/head/; revision=338172
* vm_wait() rework.Konstantin Belousov2018-02-201-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Make vm_wait() take the vm_object argument which specifies the domain set to wait for the min condition pass. If there is no object associated with the wait, use curthread' policy domainset. The mechanics of the wait in vm_wait() and vm_wait_domain() is supplied by the new helper vm_wait_doms(), which directly takes the bitmask of the domains to wait for passing min condition. Eliminate pagedaemon_wait(). vm_domain_clear() handles the same operations. Eliminate VM_WAIT and VM_WAITPFAULT macros, the direct functions calls are enough. Eliminate several control state variables from vm_domain, unneeded after the vm_wait() conversion. Scetched and reviewed by: jeff Tested by: pho Sponsored by: The FreeBSD Foundation, Mellanox Technologies Differential revision: https://reviews.freebsd.org/D14384 Notes: svn path=/head/; revision=329636
* Fix a few uses of kern_yield() in the TTM and the LinuxKPI.Mark Johnston2017-05-181-2/+2
| | | | | | | | | | | | | | kern_yield(0) effectively causes the calling thread to be rescheduled immediately since it resets the thread's priority to the highest possible value. This can cause livelocks when the pattern "while (!trylock()) kern_yield(0);" is used since the thread holding the lock may linger on the runqueue for the CPU on which the looping thread is running. MFC after: 1 week Notes: svn path=/head/; revision=318478
* Fix a race in vm_page_busy_sleep(9).Konstantin Belousov2016-10-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Suppose that we have an exclusively busy page, and a thread which can accept shared-busy page. In this case, typical code waiting for the page xbusy state to pass is again: VM_OBJECT_WLOCK(object); ... if (vm_page_xbusied(m)) { vm_page_lock(m); VM_OBJECT_WUNLOCK(object); <---1 vm_page_busy_sleep(p, "vmopax"); goto again; } Suppose that the xbusy state owner locked the object, unbusied the page and unlocked the object after we are at the line [1], but before we executed the load of the busy_lock word in vm_page_busy_sleep(). If it happens that there is still no waiters recorded for the busy state, the xbusy owner did not acquired the page lock, so it proceeded. More, suppose that some other thread happen to share-busy the page after xbusy state was relinquished but before the m->busy_lock is read in vm_page_busy_sleep(). Again, that thread only needs vm_object lock to proceed. Then, vm_page_busy_sleep() reads busy_lock value equal to the VPB_SHARERS_WORD(1). In this case, all tests in vm_page_busy_sleep(9) pass and we are going to sleep, despite the page being share-busied. Update check for m->busy_lock == VPB_UNBUSIED in vm_page_busy_sleep(9) to also accept shared-busy state if we only wait for the xbusy state to pass. Merge sequential if()s with the same 'then' clause in vm_page_busy_sleep(). Note that the current code does not share-busy pages from parallel threads, the only way to have more that one sbusy owner is right now is to recurse. Reported and tested by: pho (previous version) Reviewed by: alc, markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D8196 Notes: svn path=/head/; revision=307218
* etc: minor spelling fixes.Pedro F. Giffuni2016-05-021-2/+2
| | | | | | | | | Mostly comments but also some user-visible strings. MFC after: 2 weeks Notes: svn path=/head/; revision=298931
* Handle the driver KPI change from r292373. Ensure that managed deviceKonstantin Belousov2016-03-241-15/+12
| | | | | | | | | | | | | | pagers fault routines always return with a result page, be it the proper and valid result page, or initially passed freshly allocated placeholder. Do not free the passed in page until we are able to provide the replacement, and do not assign NULL to *mres. Reported and tested by: dumbbell Reviewed by: royger (who also verified that Xen code is safe) Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=297231
* drm: Update the device-independent code to match Linux 3.8.13Jean-Sébastien Pédron2015-03-171-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | This update brings few features: o Support for the setmaster/dropmaster ioctls. For instance, they are used to run multiple X servers simultaneously. o Support for minor devices. The only user-visible change is a new entry in /dev/dri but it is useless at the moment. This is a first step to support render nodes [1]. The main benefit is to greatly reduce the diff with Linux (at the expense of an unreadable commit diff). Hopefully, next upgrades will be easier. No updates were made to the drivers, beside adapting them to API changes. [1] https://en.wikipedia.org/wiki/Direct_Rendering_Manager#Render_nodes Tested by: Many people MFC after: 1 month Relnotes: yes Notes: svn path=/head/; revision=280183
* drm: fix usage of vm_phys_fictitious_to_vm_pageRoger Pau Monné2014-08-061-2/+6
| | | | | | | | | | | | | | | vm_phys_fictitious_to_vm_page should not be called directly, even when operating on a range that has been registered using vm_phys_fictitious_reg_range. PHYS_TO_VM_PAGE should be used instead because on arches that use VM_PHYSSEG_DENSE the page might come directly from vm_page_array. Reported by: nwhitehorn Tested by: nwhitehorn, David Mackay <davidm.jx8p@gmail.com> Sponsored by: Citrix Systems R&D Notes: svn path=/head/; revision=269634
* ttm_bo_vm_lookup_rb: actually make use of the red-black treeAndriy Gapon2013-12-191-2/+5
| | | | | | | | | | | | | Previously the code would just iterate over the whole tree as if it were just a list. Without this change I would observe X server becoming more and more jerky over time. MFC after: 5 days Notes: svn path=/head/; revision=259612
* Partially revert r254880. The bitmap operations actually use long type now.Jung-uk Kim2013-08-291-2/+1
| | | | Notes: svn path=/head/; revision=255044
* drm: Use the new drm_atomic.h, following the merge of projects/atomic64Jean-Sébastien Pédron2013-08-251-1/+2
| | | | | | | Submitted by: jkim@ Notes: svn path=/head/; revision=254880
* drm/ttm: Fix style in ttm_bo_release_mmap()Jean-Sébastien Pédron2013-08-251-2/+1
| | | | Notes: svn path=/head/; revision=254877
* drm/ttm: Fix unmap of buffer objectJean-Sébastien Pédron2013-08-251-0/+27
| | | | | | | | | | | | | | | | | | Add a new ttm_bo_release_mmap() function to unmap pages in a vm_object_t. Pages are freed when the buffer object is later released. This function is called in ttm_bo_unmap_virtual_locked(), replacing Linux' unmap_mapping_range(). In particular this is called when a buffer object is about to be moved, so that its mapping is invalidated. However, we don't use this function in ttm_bo_vm_dtor(), because the vm_object_t is already marked as OBJ_DEAD and the pages will be unmapped. Approved by: kib@ Notes: svn path=/head/; revision=254876
* drm/ttm: Improve comment in ttm_bo_vm_ctor() about lack of ref acquisitionJean-Sébastien Pédron2013-08-251-2/+10
| | | | | | | Approved by: kib@ Notes: svn path=/head/; revision=254874
* drm/ttm: Make ttm_bo_wait() call uninterruptible in page fault handlerJean-Sébastien Pédron2013-08-251-1/+17
| | | | | | | | | | | This fixes a crash where a SIGLALRM, heavily used by X.Org, would interrupt the wait, causing the page fault to fail and the "Xorg" process to receive a SIGSEGV. Approved by: kib@ Notes: svn path=/head/; revision=254870
* On all the architectures, avoid to preallocate the physical memoryAttilio Rao2013-08-091-3/+11
| | | | | | | | | | | | | | | | | | | | | | | | for nodes used in vm_radix. On architectures supporting direct mapping, also avoid to pre-allocate the KVA for such nodes. In order to do so make the operations derived from vm_radix_insert() to fail and handle all the deriving failure of those. vm_radix-wise introduce a new function called vm_radix_replace(), which can replace a leaf node, already present, with a new one, and take into account the possibility, during vm_radix_insert() allocation, that the operations on the radix trie can recurse. This means that if operations in vm_radix_insert() recursed vm_radix_insert() will start from scratch again. Sponsored by: EMC / Isilon storage division Reviewed by: alc (older version) Reviewed by: jeff Tested by: pho, scottl Notes: svn path=/head/; revision=254141
* The soft and hard busy mechanism rely on the vm object lock to work.Attilio Rao2013-08-091-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Unify the 2 concept into a real, minimal, sxlock where the shared acquisition represent the soft busy and the exclusive acquisition represent the hard busy. The old VPO_WANTED mechanism becames the hard-path for this new lock and it becomes per-page rather than per-object. The vm_object lock becames an interlock for this functionality: it can be held in both read or write mode. However, if the vm_object lock is held in read mode while acquiring or releasing the busy state, the thread owner cannot make any assumption on the busy state unless it is also busying it. Also: - Add a new flag to directly shared busy pages while vm_page_alloc and vm_page_grab are being executed. This will be very helpful once these functions happen under a read object lock. - Move the swapping sleep into its own per-object flag The KPI is heavilly changed this is why the version is bumped. It is very likely that some VM ports users will need to change their own code. Sponsored by: EMC / Isilon storage division Discussed with: alc Reviewed by: jeff, kib Tested by: gavin, bapt (older version) Tested by: pho, scottl Notes: svn path=/head/; revision=254138
* MFgem r251960: re-check the mgt device object for the requested pageKonstantin Belousov2013-07-271-2/+9
| | | | | | | | | | | after the object was relocked. Tested by: dumbbell Sponsored by: The FreeBSD Foundation MFC after: 1 week Notes: svn path=/head/; revision=253710
* Remove unneeded page lock around vm_page_insert().Konstantin Belousov2013-07-061-2/+0
| | | | | | | Submitted by: alc Notes: svn path=/head/; revision=252864
* drm/ttm: Explain why we don't need to acquire a ref in ttm_bo_vm_ctor()Jean-Sébastien Pédron2013-03-231-0/+5
| | | | Notes: svn path=/head/; revision=248661
* drm/ttm: Fix TTM buffer object refcountJean-Sébastien Pédron2013-03-231-5/+1
| | | | | | | | | | This fixes memory leaks in the radeonkms driver. Reviewed by: Konstantin Belousov (kib@) Tested by: J.R. Oldroyd <jr@opal.com> Notes: svn path=/head/; revision=248657
* Switch the vm_object mutex to be a rwlock. This will enable in theAttilio Rao2013-03-091-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | future further optimizations where the vm_object lock will be held in read mode most of the time the page cache resident pool of pages are accessed for reading purposes. The change is mostly mechanical but few notes are reported: * The KPI changes as follow: - VM_OBJECT_LOCK() -> VM_OBJECT_WLOCK() - VM_OBJECT_TRYLOCK() -> VM_OBJECT_TRYWLOCK() - VM_OBJECT_UNLOCK() -> VM_OBJECT_WUNLOCK() - VM_OBJECT_LOCK_ASSERT(MA_OWNED) -> VM_OBJECT_ASSERT_WLOCKED() (in order to avoid visibility of implementation details) - The read-mode operations are added: VM_OBJECT_RLOCK(), VM_OBJECT_TRYRLOCK(), VM_OBJECT_RUNLOCK(), VM_OBJECT_ASSERT_RLOCKED(), VM_OBJECT_ASSERT_LOCKED() * The vm/vm_pager.h namespace pollution avoidance (forcing requiring sys/mutex.h in consumers directly to cater its inlining functions using VM_OBJECT_LOCK()) imposes that all the vm/vm_pager.h consumers now must include also sys/rwlock.h. * zfs requires a quite convoluted fix to include FreeBSD rwlocks into the compat layer because the name clash between FreeBSD and solaris versions must be avoided. At this purpose zfs redefines the vm_object locking functions directly, isolating the FreeBSD components in specific compat stubs. The KPI results heavilly broken by this commit. Thirdy part ports must be updated accordingly (I can think off-hand of VirtualBox, for example). Sponsored by: EMC / Isilon storage division Reviewed by: jeff Reviewed by: pjd (ZFS specific review) Discussed with: alc Tested by: pho Notes: svn path=/head/; revision=248084
* Import the preliminary port of the TTM.Konstantin Belousov2013-03-051-0/+492
The early commit is done to facilitate the off-tree work on the porting of the Radeon driver. Sponsored by: The FreeBSD Foundation Debugged and tested by: dumbbell MFC after: 1 month Notes: svn path=/head/; revision=247835
generated by cgit v1.2.3 (git 2.25.1) at 2025-10-13 05:59:10 +0000