aboutsummaryrefslogtreecommitdiff
path: root/sys/amd64/include
Commit message (Collapse)AuthorAgeFilesLines
...
* amd64/vmm: Rename vm_get_vmspace() to vm_vmspace()Mark Johnston2025-01-071-1/+1
| | | | | | | | | For consistency with other vm accessors. No functional change intended. Reviewed by: corvink Differential Revision: https://reviews.freebsd.org/D48268
* x86 atomics: Remove unused WANT_FUNCTIONSOlivier Certner2024-12-171-3/+0
| | | | | | | | | | This macro has not been in use since commit "inline atomics and allow tied modules to inline locks" (r335873, f4b3640475cec929). Reviewed by: markj, kib, emaste, imp MFC after: 5 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D48061
* atomics: Constify loadsOlivier Certner2024-12-161-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | In order to match reality, allow using these functions with pointers on const objects, and bring us closer to C11. Remove the '+' modifier in the atomic_load_acq_64_i586()'s inline asm statement's constraint for '*p' (the value to load). CMPXCHG8B always writes back some value, even when the value exchange does not happen in which case what was read is written back. atomic_load_acq_64_i586() further takes care of the operation atomically writing back the same value that was read in any case. All in all, this makes the inline asm's write back undetectable by any other code, whether executing on other CPUs or code on the same CPU before and after the call to atomic_load_acq_64_i586(), except for the fact that CMPXCHG8B will trigger a #GP(0) if the memory address is part of a read-only mapping. This unfortunate property is however out of scope of the C abstract machine, and in particular independent of whether the 'uint64_t' pointed to is declared 'const' or not. Approved by: markj (mentor) MFC after: 5 days Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D46887
* amd64: Add wrappers for XRSTORS and XSAVESBojan Novković2024-12-151-0/+23
| | | | | Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D46984
* vmm: Correctly suspend and resume the vmm driver.Joshua Rogers2024-11-292-1/+4
| | | | | | | | | | | | | | | Previously, VMXON would be executed on a resume, contrary to proper initalization. The contents of MSR_IA32_FEATURE_CONTROL may be lost on suspension, therefore must be restored. Likewise, the VMX Enable bit may be cleared upon suspend, requiring it to be re-set. Concretely disable VMX on suspend, and re-enable it on resume. Note: any IOMMU context will remain lost for any enabled vmm devices. Signed-off-by: Joshua Rogers <Joshua@Joshua.Hu> Reviewed by: jhb,imp Pull Request: https://github.com/freebsd/freebsd-src/pull/1419
* amd64: add machine/pte.hKonstantin Belousov2024-11-262-57/+105
| | | | | | | | | | | Following arm64 and risc-v, move definitions that describe hardware-enforced layout of PTEs and #PF error bits, into a dedicated header. Reviewed by: markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D47749
* atomic(9): Implement atomic_testand(clear|set)_ptrJohn Baldwin2024-11-191-0/+2
| | | | | | | | | | For current architectures, these are just aliases for the existing operation on the relevant scalar integer. Reviewed by: imp, kib Obtained from: CheriBSD Sponsored by: AFRL, DARPA Differential Revision: https://reviews.freebsd.org/D47631
* Remove stray whitespaces from sys/amd64/Joshua Rogers2024-09-2110-12/+12
| | | | | | Signed-off-by: Joshua Rogers <Joshua@Joshua.Hu> Reviewed by: imp Pull Request: https://github.com/freebsd/freebsd-src/pull/1418
* vmm: Make vmm_dev.h more self-containedMark Johnston2024-09-011-1/+2
| | | | | | | | | | | vmm.h is required for VM_MAX_SUFFIXLEN. vmm_snapshot.h is required for struct vm_snapshot_meta. This is a prerequisite for including vmm_dev.h in the headers parsed by libsysdecode. Reviewed by: jhb Differential Revision: https://reviews.freebsd.org/D46485
* vmm: Move compat ioctl definitions to vmm_dev.cMark Johnston2024-08-261-11/+0
| | | | | | | | There is no reason to keep them in vmm_dev.h. No functional change intended. Reviewed by: corvink, jhb Differential Revision: https://reviews.freebsd.org/D46432
* vmm: Merge vmm_dev.cMark Johnston2024-08-261-5/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | This file contains the vmm device file implementation. Most of this code is not machine-dependent and so shouldn't be duplicated this way. Move most of it into a generic dev/vmm/vmm_dev.c. This will make it easier to introduce a cdev-based interface for VM creation, which in turn makes it possible to implement support for running bhyve as an unprivileged user. Machine-dependent ioctls continue to be handled in machine-dependent code. To make the split a bit easier to handle, introduce a pair of tables which define MI and MD ioctls. Each table entry can set flags which determine which locks need to be held in order to execute the handler. vmmdev_ioctl() now looks up the ioctl in one of the tables, acquires locks and either handles the ioctl directly or calls vmmdev_machdep_ioctl() to handle it. No functional change intended. There is a lot of churn in this change but the underlying logic in the ioctl handlers is the same. For now, vmm_dev.h is still mostly separate, even though some parts could be merged in principle. This would involve changing include paths for userspace, though. Reviewed by: corvink, jhb Differential Revision: https://reviews.freebsd.org/D46431
* amd64: add variables indicating INVLPGB worksKonstantin Belousov2024-08-211-0/+2
| | | | | | | | Reviewed by: alc, markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D45191
* amd64: add convenience wrappers for INVLPGB and TBLSYNCKonstantin Belousov2024-08-211-0/+23
| | | | | | | | Reviewed by: alc, markj Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D45191
* Retire non-NEW_PCIB code and remove config optionJessica Clarke2024-07-181-2/+0
| | | | | | | | | | | All architectures enable NEW_PCIB in DEFAULTS (arm being the most recent to do so in 121be555997b (arm: Set NEW_PCIB in DEFAULTS rather than a subset of kernel configs")), so it's time we removed the legacy code that no longer sees much testing and has a significant maintenance burden. Reviewed by: jhb, andrew, emaste Differential Revision: https://reviews.freebsd.org/D32954
* sdt: Use a multibyte nop for tracepoints on amd64Mark Johnston2024-07-081-1/+1
| | | | Differential Revision: https://reviews.freebsd.org/D45666
* amd64 pcpu: fix clobbers, suppress warnings, and clean upRyan Libby2024-07-032-31/+24
| | | | | | | | | | | | | | | | | These changes mostly apply to the !__SEG_GS section, which is no longer the normal compilation path. They're made to be consistent with changes to i386. - Add missing cc clobber to __PCPU_ADD (which is currently unused). - Allow the compiler the opportunity to marginally improve code generation from __PCPU_PTR by letting it figure out how to do the add (also removing the addition fixes a missing cc clobber). - Quiet gcc -Warray-bounds by using constant operands instead of bogus memory references. - Remove the struct __s __s temporaries, just cast through the type. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D45827
* amd64 kernel __storeload_barrier: quiet gcc -Warray-boundsRyan Libby2024-06-231-2/+2
| | | | | | | | | | | Use a constant input operand instead of an output operand to tell the compiler about OFFSETOF_MONITORBUF. If we tell it we are writing to *(u_int *)OFFSETOF_MONITORBUF, it rightly complains, but we aren't. The memory clobber already covers the necessary semantics for the compiler. Reviewed by: kib Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D45694
* sdt: Implement SDT probes using hot-patchingMark Johnston2024-06-191-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The idea here is to avoid a memory access and conditional branch per probe site. Instead, the probe is represented by an "unreachable" unconditional function call. asm goto is used to store the address of the probe site (represented by a no-op sled) and the address of the function call into a tracepoint record. Each SDT probe carries a list of tracepoints. When the probe is enabled, the no-op sled corresponding to each tracepoint is overwritten with a jmp to the corresponding label. The implementation uses smp_rendezvous() to park all other CPUs while the instruction is being overwritten, as this can't be done atomically in general. The compiler moves argument marshalling code and the sdt_probe() function call out-of-line, i.e., to the end of the function. Per gallatin@ in D43504, this approach has less overhead when probes are disabled. To make the implementation a bit simpler, I removed support for probes with 7 arguments; nothing makes use of this except a regression test case. It could be re-added later if need be. The approach taken in this patch enables some more improvements: 1. We can now automatically fill out the "function" field of SDT probe names. The SDT macros let the programmer specify the function and module names, but this is really a bug and shouldn't have been allowed. The intent was to be able to have the same probe in multiple functions and to let the user restrict which probes actually get enabled by specifying a function name or glob. 2. We can avoid branching on SDT_PROBES_ENABLED() by adding the ability to include blocks of code in the out-of-line path. For example: if (SDT_PROBES_ENABLED()) { int reason = CLD_EXITED; if (WCOREDUMP(signo)) reason = CLD_DUMPED; else if (WIFSIGNALED(signo)) reason = CLD_KILLED; SDT_PROBE1(proc, , , exit, reason); } could be written SDT_PROBE1_EXT(proc, , , exit, reason, int reason; reason = CLD_EXITED; if (WCOREDUMP(signo)) reason = CLD_DUMPED; else if (WIFSIGNALED(signo)) reason = CLD_KILLED; ); In the future I would like to use this mechanism more generally, e.g., to remove branches and marshalling code used by hwpmc, and generally to make it easier to add new tracepoint consumers without having to add more conditional branches to hot code paths. Reviewed by: Domagoj Stolfa, avg MFC after: 2 months Differential Revision: https://reviews.freebsd.org/D44483
* vm_phys: Fix a typoMark Johnston2024-06-161-1/+1
| | | | | Fixes: b16b4c22d2d1 ("vm_page: Implement lazy page initialization") Reported by: Steffen Nurpmeso <steffen@sdaoden.eu>
* amd64/vmm: Make vmm.h more self-containedMark Johnston2024-06-141-0/+2
| | | | | | | CTASSERT is defined in kassert.h, so include that here. No functional change intended. MFC after: 1 week
* vm_page: Implement lazy page initializationMark Johnston2024-06-141-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | FreeBSD's boot times have decreased to the point where vm_page array initialization represents a significant fraction of the total boot time. For example, when booting FreeBSD in Firecracker (a VMM designed to support lightweight VMs) with 128MB and 1GB of RAM, vm_page initialization consumes 9% (3ms) and 37% (21.5ms) of the kernel boot time, respectively. This is generally relevant in cloud environments, where one wants to be able to spin up VMs as quickly as possible. This patch implements lazy initialization of (most) page structures, following a suggestion from cperciva@. The idea is to introduce a new free pool, VM_FREEPOOL_LAZYINIT, into which all vm_page structures are initially placed. For this to work, we need only initialize the first free page of each chunk placed into the buddy allocator. Then, early page allocations draw from the lazy init pool and initialize vm_page chunks (up to 16MB, 4096 pages) on demand. Once APs are started, an idle-priority thread drains the lazy init pool in the background to avoid introducing extra latency in the allocator. With this scheme, almost all of the initialization work is moved out of the critical path. A couple of vm_phys operations require the pool to be drained before they can run: vm_phys_find_range() and vm_phys_unfree_page(). However, these are rare operations. I believe that vm_phys_find_freelist_contig() does not require any special treatment, as it only ever accesses the first page in a power-of-2-sized free page chunk, which is always initialized. For now the new pool is only used on amd64 and arm64, since that's where I can easily test and those platforms would get the most benefit. Reviewed by: alc, kib Differential Revision: https://reviews.freebsd.org/D40403
* pmap: move the smp_targeted_tlb_shutdown pointer stuff to amd64 pmap.hKonstantin Belousov2024-06-061-0/+27
| | | | | Fixes: bec000c9c1ef409989685bb03ff0532907befb4aESC Sponsored by: The FreeBSD Foundation
* libkern: add ilog2 macroDoug Moore2024-06-031-4/+0
| | | | | | | | | | | | | | | The kernel source contains several definitions of an ilog2 function; some are slower than necessary, and one of them is incorrect. Elimininate them all and define an ilog2 macro in libkern to replace them, in a way that is fast, correct for all argument types, and, in a GENERIC kernel, includes a check for an invalid zero parameter. Folks at Microsoft have verified that having a correct ilog2 definition for their MANA driver doesn't break it. Reviewed by: alc, markj, mhorne (older version), jhibbits (older version) Differential Revision: https://reviews.freebsd.org/D45170 Differential Revision: https://reviews.freebsd.org/D45235
* vm: Simplify startup page dumping conditionalBojan Novković2024-05-251-1/+2
| | | | | | | | | | This commit introduces the MINIDUMP_STARTUP_PAGE_TRACKING symbol and uses it to simplify several instances of a complex preprocessor conditional for adding pages allocated when bootstraping the kernel to minidumps. Reviewed by: markj, mhorne Approved by: markj (mentor) Differential Revision: https://reviews.freebsd.org/D45085
* uma: Deduplicate uma_small_allocBojan Novković2024-05-251-3/+3
| | | | | | | | | | | | | | | | | This commit refactors the UMA small alloc code and removes most UMA machine-dependent code. The existing machine-dependent uma_small_alloc code is almost identical across all architectures, except for powerpc where using the direct map addresses involved extra steps in some cases. The MI/MD split was replaced by a default uma_small_alloc implementation that can be overridden by architecture-specific code by defining the UMA_MD_SMALL_ALLOC symbol. Furthermore, UMA_USE_DMAP was introduced to replace most UMA_MD_SMALL_ALLOC uses. Reviewed by: markj, kib Approved by: markj (mentor) Differential Revision: https://reviews.freebsd.org/D45084
* vmm: Fix typoElyes Haouas2024-04-111-1/+1
| | | | | | Signed-off-by: Elyes Haouas <ehaouas@noos.fr> Reviewed by: imp Pull Request: https://github.com/freebsd/freebsd-src/pull/885
* atomic: Fix typoElyes Haouas2024-04-111-1/+1
| | | | | | Signed-off-by: Elyes Haouas <ehaouas@noos.fr> Reviewed by: imp Pull Request: https://github.com/freebsd/freebsd-src/pull/885
* msan: Fix typoElyes Haouas2024-04-111-1/+1
| | | | | | Signed-off-by: Elyes Haouas <ehaouas@noos.fr> Reviewed by: imp Pull Request: https://github.com/freebsd/freebsd-src/pull/885
* amd64 pcb.h: use 4 hex digits for pcb flagsKonstantin Belousov2024-02-111-8/+8
| | | | | Sponsored by: The FreeBSD Foundation MFC after: 3 days
* vmm: Expose more registers to VM_GET_REGISTERMark Johnston2024-02-071-0/+4
| | | | | | | | | | | In a follow-up revision the gdb stub will support sending an XML target description to gdb, which lets us send additional registers, including the ones added in this patch. Reviewed by: jhb MFC after: 1 month Sponsored by: Innovate UK Differential Revision: https://reviews.freebsd.org/D43665
* vmm: Fix compiling error with BHYVE_SNAPSHOTVitaliy Gusev2024-02-061-2/+5
| | | | | | | | | | | The return values of copyin() and copyout() must be checked. vm_snapshot_buf_cmp() is unused by the kernel and was incorrectly implemented, so just remove it. Reviewed by: markj Sponsored by: vStack MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D43754
* pmap: Convert boolean_t to bool.John Baldwin2024-01-311-3/+3
| | | | | Reviewed by: kib (older version) Differential Revision: https://reviews.freebsd.org/D39921
* arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directlyMark Johnston2024-01-261-1/+0
| | | | | | | | | | | | | | | pmap_san_bootstrap() doesn't really do much, and it was hard-coding the the bootstrap stack size defined in locore.S. Moreover, the name is a bit confusing given the existence of pmap_bootstrap_san(). Just remove it and call kasan_init_early() directly like we do on amd64. It will not be used by KMSAN in a forthcoming patch series. No functional change intended. MFC after: 1 week Sponsored by: Klara, Inc. Sponsored by: Juniper Networks, Inc. Differential Revision: https://reviews.freebsd.org/D43403
* vmm.h: remove dup declarationKonstantin Belousov2023-12-201-2/+0
| | | | | | | Reviewed by: markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D43139
* xen: add SPDX license tags to Xen headersElliott Mitchell2023-12-151-0/+2
| | | | | | | | | These are in fact GPLv2 when distributed with the Linux kernel, but the license also allows MIT if distributed separately. Add the markers to avoid interference by automated tools. Differential Revision: https://reviews.freebsd.org/D32796 Reviewed by: royger
* vmm: implement single-stepping for AMD CPUsBojan Novković2023-12-071-0/+8
| | | | | | | | | | | | | | | | | | This patch implements single-stepping for AMD CPUs using the RFLAGS.TF single-stepping mechanism. The GDB stub requests single-stepping using the VM_CAP_RFLAGS_TF capability. Setting this capability will set the RFLAGS.TF bit on the selected vCPU, activate DB exception intercepts, and activate POPF/PUSH instruction intercepts. The resulting DB exception is then caught by the IDT_DB vmexit handler and bounced to userland where it is processed by the GDB stub. This patch also makes sure that the value of the TF bit is correctly updated and that it is not erroneously propagated into memory. Stepping over PUSHF will cause the vm_handle_db function to correct the pushed RFLAGS value and stepping over POPF will update the shadowed TF bit copy. Reviewed by: jhb Sponsored by: Google, Inc. (GSoC 2022) Differential Revision: https://reviews.freebsd.org/D42296
* xen: correct spacing in hypercall.h headersElliott Mitchell2023-11-281-28/+28
| | | | | | | | A precursor to merging them. The spacing differs quite a bit between the i386 and amd64 hypercall headers, despite very similar content. Consistently use tabs instead of spaces. Reviewed by: royger
* sys: Remove ancient SCCS tags.Warner Losh2023-11-2717-34/+0
| | | | | | | | Remove ancient SCCS tags from the tree, automated scripting, with two minor fixup to keep things compiling. All the common forms in the tree were removed with a perl script. Sponsored by: Netflix
* _bus.h: Use standard licnese textWarner Losh2023-11-131-24/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All of these used the 'immediately at beginning' variation of the BSD-2-Clause license. This wasn't intentional, just what I copied from from a random file in the tree back in 2005. It was not an intentional decision. The different arch bus.h files are a mix of BSD-2-Clause and BSD-4-Clause that have various copyright holders (Charles M. Hannum, Christopher G. Demetriou, The NetBSD Foundation and KATO Takenori), and some of the content of these files were likely copied from there. However, apart from the uncopyrightable interface lines, there are very few comments. It's unclear if these comments are 'original material' here to copyright, but to the extent that there is, license it under the standard BSD-2-Clause copyright that's the norm for the project today. In any event, the standard BSD-2-Clause is also closer to those originals. In addition, FreeBSD uses different type definitions than the original NetBSD code in part. The comments that were copied have been copied a lot, but appear in NetBSD's bus.h files in NetBSD 1.3. While I'm here, assign the copyright, to the extent any exists from me, to the FreeBSD Foundation. I just cut and pasted these into _bus.h from the different machine files and those files have a rich history of modification from the original imports from NetBSD over more than 25 years so it's tricky to say who, exactly, wrote each bit. Given the size of the files, this seems like the best compromise. Also add an acknowledgement to the NetBSD 1.3 bus.h files and their authors (there were no additional FreeBSD authors listed in the various sys/*/include/bus.h files). Finally, use the SPDX identifier instead of multiple copies of the text. Differential Revision: https://reviews.freebsd.org/D42532 Sponsored by: Netflix
* sys: Remove $FreeBSD$: one-line .c comment patternWarner Losh2023-08-1636-36/+0
| | | | Remove /^/[*/]\s*\$FreeBSD\$.*\n/
* sys: Remove $FreeBSD$: one-line .h patternWarner Losh2023-08-1617-17/+0
| | | | Remove /^\s*\*+\s*\$FreeBSD\$.*$\n/
* sys: Remove $FreeBSD$: two-line .h patternWarner Losh2023-08-1634-68/+0
| | | | Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/
* amd64: Bump MAXCPU to 1024 (from 256)Ed Maste2023-08-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Hardware with more than 256 CPU cores is currently available and will become increasingly common over FreeBSD 14's lifetime. Increase MAXCPU in the amd64 GENERIC kernel configuration to 1024. Earlier commits increased some related limits. These prerequisite commits include at least: - d7ed40243769 Increase MAX_APIC_ID safeguard to 0x800 - d1639e43c589 cpuset: increase userland maximum size to 1024 Global and allocated arrays sized by MAXCPU result in excessive bloat on systems with lower core counts. In addition, some code used u_char (8 bits) to hold a CPU index, which is not valid if MAXCPU is greater than 256. A number of recent commits addressed these sorts of issues, including at least: - 133935d26f20 pf: atomically increment state ids - 74ac712f72cf vmm: Dynamically allocate a couple of per-CPU state save areas - 78cfa762ebf2 callout: Move per-CPU callout state into the dpcpu region - 42f722e721cd amd64: store pcids pmap data in pcpu zone - 9801e7c275f6 smp_topo: dynamically allocate group array - 9fb6718d1b18 smp: Dynamically allocate the stoppcbs array - 2bb16c635249 x86: retire use of intr_bind There are some additional allocations still to be converted and more scalability work is required to make effective use of very high core count systems, but this change allows us to boot on these systems and provides a Kernel Binary Interface (KBI) for the FreeBSD 14 release that supports these configurations. Special thanks to AMD for providing hardware to test these changes. PR: 269572 Reviewed by: des Relnotes: Yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D36838
* kmsan: Add a comment explaining why KMSAN doesn't shadow above KERNBASEMark Johnston2023-07-271-2/+9
| | | | Sponsored by: The FreeBSD Foundation
* Fix MINIMAL build on amd64Kristof Provost2023-07-141-0/+1
| | | | | amd64/include/counter.h uses KASSERT, but failed to include the kassert.h header.
* amd64 pcpu.h: Add missing 'do' from do-while loop around __PCPU_SET.John Baldwin2023-07-081-1/+1
| | | | | Reported by: mjg Diagnosed by: jrtc27
* amd64: Use __seg_gs to implement per-CPU data accesses.John Baldwin2023-07-071-6/+63
| | | | | | | | | | | | | | | | | This makes use of the alternate address space support in both GCC and clang to access per-CPU data as accesses relative to GS:. The original motivation for this is that it quiets verbose warnings from GCC 12. However, this version is also much easier to read and allows the compiler to generate better code (e.g. the compiler can use a GS: memory operand directly in other instructions such as IMUL and CMP rather than always MOVing to a temporary register). The one caveat is that the current approach is very inefficient at -O0 since the compiler expects to load the 0 base offset from a global variable instead of assuming it is 0 (even with the const). Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D40647
* Consistently provide ffs/fls using builtinsMitchell Horne2023-07-061-37/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Use of compiler builtin ffs/ctz functions will result in optimized instruction sequences when possible, and fall back to calling a function provided by the compiler run-time library. We have slowly shifted our platforms to take advantage of these builtins in 60645781d613 (arm64), 1c76d3a9fbef (arm), 9e319462a03a (powerpc, partial). Some platforms still rely on the libkern implementations of these functions provided by libkern, namely riscv, powerpc (ffs*, flsll), and i386 (ffsll and flsll). These routines are slow, as they perform a linear search for the bit in question. Even on platforms lacking dedicated bit-search instructions, such as riscv, the compiler library will provide better-optimized routines, e.g. by using binary search. Consolidate all definitions of these functions (whether currently using builtins or not) to libkern.h. This should result in equivalent or better performing routines in all cases. One wart in all of this is the existing HAVE_INLINE_F*** macros, which we use in a few places to conditionally avoid the slow libkern routines. These aren't easily removed in one commit. For now, provide these defines unconditionally, but marked for removal after subsequent cleanup. Removal of the now unused libkern routines will follow in the next commit. Reviewed by: dougm, imp (previous version) Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D40698
* amd64/arm64 pmap: Stop requiring the accessed bit for superpage promotionAlan Cox2023-06-121-1/+1
| | | | | | | | | | | | | | | | | | Stop requiring all of the PTEs to have the accessed bit set for superpage promotion to occur. Given that change, add support for promotion to pmap_enter_quick(), which does not set the accessed bit in the PTE that it creates. Since the final mapping within a superpage-aligned and sized region of a memory-mapped file is typically created by a call to pmap_enter_quick(), we now achieve promotions in circumstances where they did not occur before, for example, the X server's read-only mapping of libLLVM-15.so. See also https://www.usenix.org/system/files/atc20-zhu-weixi_0.pdf Reviewed by: kib, markj MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D40478
* smp: Dynamically allocate the stoppcbs arrayMark Johnston2023-05-251-2/+0
| | | | | | | | | | | | | | This avoids bloating the kernel image when MAXCPU is large. A follow-up patch for kgdb and other kernel debuggers is needed since the stoppcbs symbol is now a pointer. Bump __FreeBSD_version so that debuggers can use osreldate to figure out how to handle stoppcbs. PR: 269572 MFC after: never Reviewed by: mjg, emaste Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D39806
generated by cgit v1.3 (git 2.53.0) at 2026-03-29 11:41:01 +0000