aboutsummaryrefslogtreecommitdiff
path: root/sys/i386/include/pcpu.h
Commit message (Collapse)AuthorAgeFilesLines
* Centralize __pcpu definitions.Konstantin Belousov2019-08-291-30/+0
| | | | | | | | | | | | | | | | | | | | | | | | | Many extern struct pcpu <something>__pcpu declarations were copied/pasted in sources. The issue is that the definition is MD, but it cannot be provided by machine/pcpu.h due to actual struct pcpu defined in sys/pcpu.h later than the inclusion of machine/pcpu.h. This forced the copying when other code needed direct access to __pcpu. There is no way around it, due to machine/pcpu.h supplying part of struct pcpu fields. To work around the problem, add a new machine/pcpu_aux.h header, which should fill any needed MD definitions after struct pcpu definition is completed. This allows to remove copies of __pcpu spread around the source. Also on x86 it makes it possible to remove work arounds like OFFSETOF_CURTHREAD or clang specific warnings supressions. Reported and tested by: lwhsu, bcran Reviewed by: imp, markj (previous version) Discussed with: jhb Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D21418 Notes: svn path=/head/; revision=351594
* Mitigations for Microarchitectural Data Sampling.Konstantin Belousov2019-05-141-2/+6
| | | | | | | | | | | | | | | | | | Microarchitectural buffers on some Intel processors utilizing speculative execution may allow a local process to obtain a memory disclosure. An attacker may be able to read secret data from the kernel or from a process when executing untrusted code (for example, in a web browser). Reference: https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00233.html Security: CVE-2018-12126, CVE-2018-12127, CVE-2018-12130, CVE-2019-11091 Security: FreeBSD-SA-19:07.mds Reviewed by: jhb Tested by: emaste, lwhsu Approved by: so (gtetlow) Notes: svn path=/head/; revision=347566
* x86: store pending bitmapped IPIs in per-cpu areasMateusz Guzik2019-05-121-1/+2
| | | | | | | | | | | | This gets rid of the global cpu_ipi_pending array. While replace cmpset with fcmpset in the delivery code and opportunistically check if given IPI is already pending. Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=347502
* x86: Implement MWAIT support for stopping a CPUConrad Meyer2019-05-041-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | IPI_STOP is used after panic or when ddb is entered manually. MONITOR/ MWAIT allows CPUs that support the feature to sleep in a low power way instead of spinning. Something similar is already used at idle. It is perhaps especially useful in oversubscribed VM environments, and is safe to use even if the panic/ddb thread is not the BSP. (Except in the presence of MWAIT errata, which are detected automatically on platforms with known wakeup problems.) It can be tuned/sysctled with "machdep.stop_mwait," which defaults to 0 (off). This commit also introduces the tunable "machdep.mwait_cpustop_broken," which defaults to 0, unless the CPU has known errata, but may be set to "1" in loader.conf to signal that mwait wakeup is broken on CPUs FreeBSD does not yet know about. Unfortunately, Bhyve doesn't yet support MONITOR extensions, so this doesn't help bhyve hypervisors running FreeBSD guests. Submitted by: Anton Rang <rang AT acm.org> (earlier version) Reviewed by: kib Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D20135 Notes: svn path=/head/; revision=347134
* x86: Define pc_monitorbuf as a logical structureConrad Meyer2019-05-041-1/+7
| | | | | | | | | | | | | Rather than just accessing it via pointer cast. No functional change intended. Discussed with: kib (earlier version) Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D20135 Notes: svn path=/head/; revision=347129
* Expand x86 struct pcpus to UMA_PCPU_ALLOC_SIZE AKA PAGE_SIZE.Konstantin Belousov2018-07-061-1/+1
| | | | | | | | | | | | This restores counters(9) operation. Revert r336024. Improve assert of pcpu size on x86. Reviewed by: mmacy Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D16163 Notes: svn path=/head/; revision=336047
* Revert to recommit with the proper message.Konstantin Belousov2018-07-061-1/+1
| | | | Notes: svn path=/head/; revision=336046
* Save a call to pmap_remove() if entry cannot have any pages mapped.Konstantin Belousov2018-07-061-1/+1
| | | | | | | | | | | | | | | | Due to the way rtld creates mappings for the shared objects, each dso causes unmap of at least three guard map entries. For instance, in the buildworld load, this change reduces the amount of pmap_remove() calls by 1/5. Profiled by: alc Reviewed by: alc, markj Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D16148 Notes: svn path=/head/; revision=336045
* Optimize i386 pmap_extract_and_hold().Konstantin Belousov2018-05-251-1/+3
| | | | | | | | | | | | | | | | | | | | | In particular, stop using pmap_pte() to read non-promoted pte while walking the page table. pmap_pte() needs to shoot down the kernel mapping globally which causes IPI broadcast. Since pmap_extract_and_hold() is used for slow copyin(9), it is very significant hit for the 4/4 kernels. Instead, create single purpose per-processor page frame and use it to locally map page table page inside the critical section, to avoid reuse of the frame by other thread if context switched. Measurement demostrated very significant improvements in any load that utilizes copyin/copyout. Found and benchmarked by: bde Sponsored by: The FreeBSD Foundation Notes: svn path=/head/; revision=334211
* fix i386 builds after r334005 and r334009Matt Macy2018-05-221-1/+2
| | | | | | | | | | | r334005: add pc_ibpb_set as it is now referenced by common code (although presumably not needed on i386 since it has been there since the first spectre mitigation work on amd64) r334009: there is no amd64 rflags -> i386 eflags Notes: svn path=/head/; revision=334018
* Fix i386 build after r332970 by adding IS_BSP() definition.Li-Wen Hsu2018-04-251-0/+2
| | | | | | | Approved by: kib Notes: svn path=/head/; revision=332989
* i386 4/4G split.Konstantin Belousov2018-04-131-7/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The change makes the user and kernel address spaces on i386 independent, giving each almost the full 4G of usable virtual addresses except for one PDE at top used for trampoline and per-CPU trampoline stacks, and system structures that must be always mapped, namely IDT, GDT, common TSS and LDT, and process-private TSS and LDT if allocated. By using 1:1 mapping for the kernel text and data, it appeared possible to eliminate assembler part of the locore.S which bootstraps initial page table and KPTmap. The code is rewritten in C and moved into the pmap_cold(). The comment in vmparam.h explains the KVA layout. There is no PCID mechanism available in protected mode, so each kernel/user switch forth and back completely flushes the TLB, except for the trampoline PTD region. The TLB invalidations for userspace becomes trivial, because IPI handlers switch page tables. On the other hand, context switches no longer need to reload %cr3. copyout(9) was rewritten to use vm_fault_quick_hold(). An issue for new copyout(9) is compatibility with wiring user buffers around sysctl handlers. This explains two kind of locks for copyout ptes and accounting of the vslock() calls. The vm_fault_quick_hold() AKA slow path, is only tried after the 'fast path' failed, which temporary changes mapping to the userspace and copies the data to/from small per-cpu buffer in the trampoline. If a page fault occurs during the copy, it is short-circuit by exception.s to not even reach C code. The change was motivated by the need to implement the Meltdown mitigation, but instead of KPTI the full split is done. The i386 architecture already shows the sizing problems, in particular, it is impossible to link clang and lld with debugging. I expect that the issues due to the virtual address space limits would only exaggerate and the split gives more liveness to the platform. Tested by: pho Discussed with: bde Sponsored by: The FreeBSD Foundation MFC after: 1 month Differential revision: https://reviews.freebsd.org/D14633 Notes: svn path=/head/; revision=332489
* sys/i386: further adoption of SPDX licensing ID tags.Pedro F. Giffuni2017-11-271-0/+2
| | | | | | | | | | | | | | | Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Notes: svn path=/head/; revision=326260
* Remove lint support from system headers and MD x86 headers.Konstantin Belousov2017-11-231-14/+3
| | | | | | | | | | Reviewed by: dim, jhb Discussed with: imp Sponsored by: The FreeBSD Foundation Differential revision: https://reviews.freebsd.org/D13156 Notes: svn path=/head/; revision=326123
* - Remove 'struct vmmeter' from 'struct pcpu', leaving only global vmmeterGleb Smirnoff2017-04-171-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | in place. To do per-cpu stats, convert all fields that previously were maintained in the vmmeters that sit in pcpus to counter(9). - Since some vmmeter stats may be touched at very early stages of boot, before we have set up UMA and we can do counter_u64_alloc(), provide an early counter mechanism: o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter. o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter, so that at early stages of boot, before counters are allocated we already point to a counter that can be safely written to. o For sparc64 that required a whole dummy pcpu[MAXCPU] array. Further related changes: - Don't include vmmeter.h into pcpu.h. - vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit, to match kernel representation. - struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion. This is based on benno@'s 4-year old patch: https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html Reviewed by: kib, gallatin, marius, lidl Differential Revision: https://reviews.freebsd.org/D10156 Notes: svn path=/head/; revision=317061
* Implement get_pcpu() for i386 and use it to replace pcpu_find(curcpu)Jason A. Harmening2017-01-291-0/+10
| | | | | | | | | | | | | | | | | | in the i386 pmap. The curcpu macro loads the per-cpu data pointer as its first step, so the remaining steps of pcpu_find(curcpu) are circular. get_pcpu() is already implemented for arm, arm64, and risc-v. My plan is to implement it for the remaining architectures and use it to replace several instances of pcpu_find(curcpu) in MI code. Reviewed by: kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D9370 Notes: svn path=/head/; revision=312952
* Move the objects used to create temporary mappings for i386 pmap zero and copyJason A. Harmening2016-12-231-1/+9
| | | | | | | | | | | | | | | | | operations to the MD PCPU region. Change sysmap initialization to only allocate KVA pages for CPUs that are actually present. As a minor optimization, this also prevents false sharing between adjacent sysmap objects since the pcpu struct is already cacheline-aligned. While here, move pc_qmap_addr initialization for the BSP into pmap_bootstrap(), which allows use of pmap_quick* functions during early boot. Reviewed by: kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D8833 Notes: svn path=/head/; revision=310481
* Re-apply r306516 (by cem):Konstantin Belousov2016-10-041-1/+2
| | | | | | | | | | | | | | | | | | | | | | Reduce the cost of TLB invalidation on x86 by using per-CPU completion flags Reduce contention during TLB invalidation operations by using a per-CPU completion flag, rather than a single atomically-updated variable. On a Westmere system (2 sockets x 4 cores x 1 threads), dtrace measurements show that smp_tlb_shootdown is about 50% faster with this patch; observations with VTune show that the percentage of time spent in invlrng_single_page on an interrupt (actually doing invalidation, rather than synchronization) increases from 31% with the old mechanism to 71% with the new one. (Running a basic file server workload.) Submitted by: Anton Rang <rang at acm.org> Reviewed by: cem (earlier version) Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D8041 Notes: svn path=/head/; revision=306680
* Add two new pmap functions:Jason A. Harmening2015-08-041-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | vm_offset_t pmap_quick_enter_page(vm_page_t m) void pmap_quick_remove_page(vm_offset_t kva) These will create and destroy a temporary, CPU-local KVA mapping of a specified page. Guarantees: --Will not sleep and will not fail. --Safe to call under a non-sleepable lock or from an ithread Restrictions: --Not guaranteed to be safe to call from an interrupt filter or under a spin mutex on all platforms --Current implementation does not guarantee more than one page of mapping space across all platforms. MI code should not make nested calls to pmap_quick_enter_page. --MI code should not perform locking while holding onto a mapping created by pmap_quick_enter_page The idea is to use this in busdma, for bounce buffer copies as well as virtually-indexed cache maintenance on mips and arm. NOTE: the non-i386, non-amd64 implementations of these functions still need review and testing. Reviewed by: kib Approved by: kib (mentor) Differential Revision: http://reviews.freebsd.org/D3013 Notes: svn path=/head/; revision=286296
* Remove support for Xen PV domU kernels. Support for HVM domU kernelsJohn Baldwin2015-04-301-30/+2
| | | | | | | | | | | | | | | | | | | | | | | | remains. Xen is planning to phase out support for PV upstream since it is harder to maintain and has more overhead. Modern x86 CPUs include virtualization extensions that support HVM guests instead of PV guests. In addition, the PV code was i386 only and not as well maintained recently as the HVM code. - Remove the i386-only NATIVE option that was used to disable certain components for PV kernels. These components are now standard as they are on amd64. - Remove !XENHVM bits from PV drivers. - Remove various shims required for XEN (e.g. PT_UPDATES_FLUSH, LOAD_CR3, etc.) - Remove duplicate copy of <xen/features.h>. - Remove unused, i386-only xenstored.h. Differential Revision: https://reviews.freebsd.org/D2362 Reviewed by: royger Tested by: royger (i386/amd64 HVM domU and amd64 PVH dom0) Relnotes: yes Notes: svn path=/head/; revision=282274
* Formalize the concept of virtual CPU ids by adding a per-cpu vcpu_idJustin T. Gibbs2013-10-051-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | field. Perform vcpu enumeration for Xen PV and HVM environments and convert all Xen drivers to use vcpu_id instead of a hard coded assumption of the mapping algorithm (acpi or apic ID) in use. Submitted by: Roger Pau Monné Sponsored by: Citrix Systems R&D Reviewed by: gibbs Approved by: re (blanket Xen) amd64/include/pcpu.h: i386/include/pcpu.h: Add vcpu_id to the amd64 and i386 pcpu structures. dev/xen/timer/timer.c x86/xen/xen_intr.c Use new vcpu_id instead of assuming acpi_id == vcpu_id. i386/xen/mp_machdep.c: i386/xen/mptable.c x86/xen/hvm.c: Perform Xen HVM and Xen full PV vcpu_id mapping. x86/xen/hvm.c: x86/acpica/madt.c Change SYSINIT ordering of acpi CPU enumeration so that it is guaranteed to be available at the time of Xen HVM vcpu id mapping. Notes: svn path=/head/; revision=256073
* Implement PV IPIs for PVHVM guests and further converge PV and HVMJustin T. Gibbs2013-09-061-9/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | IPI implmementations. Submitted by: Roger Pau Monné Sponsored by: Citrix Systems R&D Submitted by: gibbs (misc cleanup, table driven config) Reviewed by: gibbs MFC after: 2 weeks sys/amd64/include/cpufunc.h: sys/amd64/amd64/pmap.c: Move invltlb_globpcid() into cpufunc.h so that it can be used by the Xen HVM version of tlb shootdown IPI handlers. sys/x86/xen/xen_intr.c: sys/xen/xen_intr.h: Rename xen_intr_bind_ipi() to xen_intr_alloc_and_bind_ipi(), and remove the ipi vector parameter. This api allocates an event channel port that can be used for ipi services, but knows nothing of the actual ipi for which that port will be used. Removing the unused argument and cleaning up the comments surrounding its declaration helps clarify its actual role. sys/amd64/amd64/mp_machdep.c: sys/amd64/include/cpu.h: sys/i386/i386/mp_machdep.c: sys/i386/include/cpu.h: Implement a generic framework for amd64 and i386 that allows the implementation of certain CPU management functions to be selected at runtime. Currently this is only used for the ipi send function, which we optimize for Xen when running on a Xen hypervisor, but can easily be expanded to support more operations. sys/x86/xen/hvm.c: Implement Xen PV IPI handlers and operations, replacing native send IPI. sys/amd64/include/pcpu.h: sys/i386/include/pcpu.h: sys/i386/include/smp.h: Remove NR_VIRQS and NR_IPIS from FreeBSD headers. NR_VIRQS is defined already for us in the xen interface files. NR_IPIS is only needed in one file per Xen platform and is easily inferred by the IPI vector table that is defined in those files. sys/i386/xen/mp_machdep.c: Restructure to more closely match the HVM implementation by performing table driven IPI setup. Notes: svn path=/head/; revision=255331
* Implement vector callback for PVHVM and unify event channel implementationsJustin T. Gibbs2013-08-291-14/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Re-structure Xen HVM support so that: - Xen is detected and hypercalls can be performed very early in system startup. - Xen interrupt services are implemented using FreeBSD's native interrupt delivery infrastructure. - the Xen interrupt service implementation is shared between PV and HVM guests. - Xen interrupt handlers can optionally use a filter handler in order to avoid the overhead of dispatch to an interrupt thread. - interrupt load can be distributed among all available CPUs. - the overhead of accessing the emulated local and I/O apics on HVM is removed for event channel port events. - a similar optimization can eventually, and fairly easily, be used to optimize MSI. Early Xen detection, HVM refactoring, PVHVM interrupt infrastructure, and misc Xen cleanups: Sponsored by: Spectra Logic Corporation Unification of PV & HVM interrupt infrastructure, bug fixes, and misc Xen cleanups: Submitted by: Roger Pau Monné Sponsored by: Citrix Systems R&D sys/x86/x86/local_apic.c: sys/amd64/include/apicvar.h: sys/i386/include/apicvar.h: sys/amd64/amd64/apic_vector.S: sys/i386/i386/apic_vector.s: sys/amd64/amd64/machdep.c: sys/i386/i386/machdep.c: sys/i386/xen/exception.s: sys/x86/include/segments.h: Reserve IDT vector 0x93 for the Xen event channel upcall interrupt handler. On Hypervisors that support the direct vector callback feature, we can request that this vector be called directly by an injected HVM interrupt event, instead of a simulated PCI interrupt on the Xen platform PCI device. This avoids all of the overhead of dealing with the emulated I/O APIC and local APIC. It also means that the Hypervisor can inject these events on any CPU, allowing upcalls for different ports to be handled in parallel. sys/amd64/amd64/mp_machdep.c: sys/i386/i386/mp_machdep.c: Map Xen per-vcpu area during AP startup. sys/amd64/include/intr_machdep.h: sys/i386/include/intr_machdep.h: Increase the FreeBSD IRQ vector table to include space for event channel interrupt sources. sys/amd64/include/pcpu.h: sys/i386/include/pcpu.h: Remove Xen HVM per-cpu variable data. These fields are now allocated via the dynamic per-cpu scheme. See xen_intr.c for details. sys/amd64/include/xen/hypercall.h: sys/dev/xen/blkback/blkback.c: sys/i386/include/xen/xenvar.h: sys/i386/xen/clock.c: sys/i386/xen/xen_machdep.c: sys/xen/gnttab.c: Prefer FreeBSD primatives to Linux ones in Xen support code. sys/amd64/include/xen/xen-os.h: sys/i386/include/xen/xen-os.h: sys/xen/xen-os.h: sys/dev/xen/balloon/balloon.c: sys/dev/xen/blkback/blkback.c: sys/dev/xen/blkfront/blkfront.c: sys/dev/xen/console/xencons_ring.c: sys/dev/xen/control/control.c: sys/dev/xen/netback/netback.c: sys/dev/xen/netfront/netfront.c: sys/dev/xen/xenpci/xenpci.c: sys/i386/i386/machdep.c: sys/i386/include/pmap.h: sys/i386/include/xen/xenfunc.h: sys/i386/isa/npx.c: sys/i386/xen/clock.c: sys/i386/xen/mp_machdep.c: sys/i386/xen/mptable.c: sys/i386/xen/xen_clock_util.c: sys/i386/xen/xen_machdep.c: sys/i386/xen/xen_rtc.c: sys/xen/evtchn/evtchn_dev.c: sys/xen/features.c: sys/xen/gnttab.c: sys/xen/gnttab.h: sys/xen/hvm.h: sys/xen/xenbus/xenbus.c: sys/xen/xenbus/xenbus_if.m: sys/xen/xenbus/xenbusb_front.c: sys/xen/xenbus/xenbusvar.h: sys/xen/xenstore/xenstore.c: sys/xen/xenstore/xenstore_dev.c: sys/xen/xenstore/xenstorevar.h: Pull common Xen OS support functions/settings into xen/xen-os.h. sys/amd64/include/xen/xen-os.h: sys/i386/include/xen/xen-os.h: sys/xen/xen-os.h: Remove constants, macros, and functions unused in FreeBSD's Xen support. sys/xen/xen-os.h: sys/i386/xen/xen_machdep.c: sys/x86/xen/hvm.c: Introduce new functions xen_domain(), xen_pv_domain(), and xen_hvm_domain(). These are used in favor of #ifdefs so that FreeBSD can dynamically detect and adapt to the presence of a hypervisor. The goal is to have an HVM optimized GENERIC, but more is necessary before this is possible. sys/amd64/amd64/machdep.c: sys/dev/xen/xenpci/xenpcivar.h: sys/dev/xen/xenpci/xenpci.c: sys/x86/xen/hvm.c: sys/sys/kernel.h: Refactor magic ioport, Hypercall table and Hypervisor shared information page setup, and move it to a dedicated HVM support module. HVM mode initialization is now triggered during the SI_SUB_HYPERVISOR phase of system startup. This currently occurs just after the kernel VM is fully setup which is just enough infrastructure to allow the hypercall table and shared info page to be properly mapped. sys/xen/hvm.h: sys/x86/xen/hvm.c: Add definitions and a method for configuring Hypervisor event delievery via a direct vector callback. sys/amd64/include/xen/xen-os.h: sys/x86/xen/hvm.c: sys/conf/files: sys/conf/files.amd64: sys/conf/files.i386: Adjust kernel build to reflect the refactoring of early Xen startup code and Xen interrupt services. sys/dev/xen/blkback/blkback.c: sys/dev/xen/blkfront/blkfront.c: sys/dev/xen/blkfront/block.h: sys/dev/xen/control/control.c: sys/dev/xen/evtchn/evtchn_dev.c: sys/dev/xen/netback/netback.c: sys/dev/xen/netfront/netfront.c: sys/xen/xenstore/xenstore.c: sys/xen/evtchn/evtchn_dev.c: sys/dev/xen/console/console.c: sys/dev/xen/console/xencons_ring.c Adjust drivers to use new xen_intr_*() API. sys/dev/xen/blkback/blkback.c: Since blkback defers all event handling to a taskqueue, convert this task queue to a "fast" taskqueue, and schedule it via an interrupt filter. This avoids an unnecessary ithread context switch. sys/xen/xenstore/xenstore.c: The xenstore driver is MPSAFE. Indicate as much when registering its interrupt handler. sys/xen/xenbus/xenbus.c: sys/xen/xenbus/xenbusvar.h: Remove unused event channel APIs. sys/xen/evtchn.h: Remove all kernel Xen interrupt service API definitions from this file. It is now only used for structure and ioctl definitions related to the event channel userland device driver. Update the definitions in this file to match those from NetBSD. Implementing this interface will be necessary for Dom0 support. sys/xen/evtchn/evtchnvar.h: Add a header file for implemenation internal APIs related to managing event channels event delivery. This is used to allow, for example, the event channel userland device driver to access low-level routines that typical kernel consumers of event channel services should never access. sys/xen/interface/event_channel.h: sys/xen/xen_intr.h: Standardize on the evtchn_port_t type for referring to an event channel port id. In order to prevent low-level event channel APIs from leaking to kernel consumers who should not have access to this data, the type is defined twice: Once in the Xen provided event_channel.h, and again in xen/xen_intr.h. The double declaration is protected by __XEN_EVTCHN_PORT_DEFINED__ to ensure it is never declared twice within a given compilation unit. sys/xen/xen_intr.h: sys/xen/evtchn/evtchn.c: sys/x86/xen/xen_intr.c: sys/dev/xen/xenpci/evtchn.c: sys/dev/xen/xenpci/xenpcivar.h: New implementation of Xen interrupt services. This is similar in many respects to the i386 PV implementation with the exception that events for bound to event channel ports (i.e. not IPI, virtual IRQ, or physical IRQ) are further optimized to avoid mask/unmask operations that aren't necessary for these edge triggered events. Stubs exist for supporting physical IRQ binding, but will need additional work before this implementation can be fully shared between PV and HVM. sys/amd64/amd64/mp_machdep.c: sys/i386/i386/mp_machdep.c: sys/i386/xen/mp_machdep.c sys/x86/xen/hvm.c: Add support for placing vcpu_info into an arbritary memory page instead of using HYPERVISOR_shared_info->vcpu_info. This allows the creation of domains with more than 32 vcpus. sys/i386/i386/machdep.c: sys/i386/xen/clock.c: sys/i386/xen/xen_machdep.c: sys/i386/xen/exception.s: Add support for new event channle implementation. Notes: svn path=/head/; revision=255040
* Merge from projects/counters:Gleb Smirnoff2013-04-081-3/+7
| | | | | | | | | | Pad struct pcpu so that its size is denominator of PAGE_SIZE. This is done to reduce memory waste in UMA_PCPU_ZONE zones. Sponsored by: Nginx, Inc. Notes: svn path=/head/; revision=249265
* MFamd64 r238623:Konstantin Belousov2012-07-261-1/+21
| | | | | | | | | | Introduce curpcb magic variable. Requested and reviewed by: bde MFC after: 3 weeks Notes: svn path=/head/; revision=238792
* Make "options XENHVM" compile for i386, not just amd64 -- a largelyRobert Watson2011-01-041-2/+15
| | | | | | | | | | | | mechanical change. This opens the door for using PV device drivers under Xen HVM on i386, as well as more general harmonisation of i386 and amd64 Xen support in FreeBSD. Reviewed by: cperciva MFC after: 3 weeks Notes: svn path=/head/; revision=216956
* Mark the __curthread() functions as __pure2 and remove the volatile keywordJohn Baldwin2010-07-291-2/+2
| | | | | | | | | | | from the inline assembly. This allows the compiler to cache invocations of curthread since it's value does not change within a thread context. Submitted by: zec (i386) MFC after: 1 week Notes: svn path=/head/; revision=210623
* MFamd64: Add a new macro PCPU_XEN_FIELDS to hold XEN-specific per-CPUJohn Baldwin2010-06-021-28/+11
| | | | | | | | | | | | | fields that is always included in PCPU_MD_FIELDS. The macro is empty for non-XEN kernels. This avoids duplicating non-XEN per-CPU fields in two places. While here, remove several unused fields from the XEN-specific structure. Reviewed by: kmacy, gibbs MFC after: 1 month Notes: svn path=/head/; revision=208742
* Add support for corrected machine check interrupts. CMCI is a new localJohn Baldwin2010-05-241-1/+3
| | | | | | | | | | | | | | | | | | | APIC interrupt that fires when a threshold of corrected machine check events is reached. CMCI also includes a count of events when reporting corrected errors in the bank's status register. Note that individual banks may or may not support CMCI. If they do, each bank includes its own threshold register that determines when the interrupt fires. Currently the code uses a very simple strategy where it doubles the threshold on each interrupt until it succeeds in throttling the interrupt to occur only once a minute (this interval can be tuned via sysctl). The threshold is also adjusted on each hourly poll which will lower the threshold once events stop occurring. Tested by: Sailaja Bangaru sbappana at yahoo com MFC after: 1 month Notes: svn path=/head/; revision=208507
* whitespace commitJulian Elischer2009-09-041-2/+2
| | | | | | | Submitted by: bde@ Notes: svn path=/head/; revision=196816
* Bring i386 up to date with amd64 and others.Julian Elischer2009-09-041-5/+5
| | | | | | | | | | The macros for PCPU can be slightly simplified, which makes the resulting tangle qa lot easier to understand when trying to read them. MFC after: 4 weeks Notes: svn path=/head/; revision=196811
* Fix IPI supportKip Macy2008-10-231-2/+27
| | | | Notes: svn path=/head/; revision=184198
* Export 'struct pcpu' to userland w/o requiring _KERNEL. A few portsJohn Baldwin2008-08-191-2/+3
| | | | | | | | | | already define _KERNEL to get to this and I'm about to add hooks to libkvm to access per-CPU data. MFC after: 1 week Notes: svn path=/head/; revision=181875
* Integrate support for xen in to i386 common code.Kip Macy2008-08-151-0/+24
| | | | | | | MFC after: 1 month Notes: svn path=/head/; revision=181775
* - Add an integer argument to idle to indicate how likely we are to wakeJeff Roberson2008-04-251-0/+1
| | | | | | | | | | | | | | | | | | from idle over the next tick. - Add a new MD routine, cpu_wake_idle() to wakeup idle threads who are suspended in cpu specific states. This function can fail and cause the scheduler to fall back to another mechanism (ipi). - Implement support for mwait in cpu_idle() on i386/amd64 machines that support it. mwait is a higher performance way to synchronize cpus as compared to hlt & ipis. - Allow selecting the idle routine by name via sysctl machdep.idle. This replaces machdep.cpu_idle_hlt. Only idle routines supported by the current machine are permitted. Sponsored by: Nokia Notes: svn path=/head/; revision=178471
* Rework the PCPU_* (MD) interface:Attilio Rao2007-06-041-3/+26
| | | | | | | | | | | | | | | - Rename PCPU_LAZY_INC into PCPU_INC - Add the PCPU_ADD interface which just does an add on the pcpu member given a specific value. Note that for most architectures PCPU_INC and PCPU_ADD are not safe. This is a point that needs some discussions/work in the next days. Reviewed by: alc, bde Approved by: jeff (mentor) Notes: svn path=/head/; revision=170291
* Push down the implementation of PCPU_LAZY_INC() into the machine-dependentAlan Cox2007-03-111-0/+25
| | | | | | | | | | header file. Reimplement PCPU_LAZY_INC() on amd64 and i386 making it atomic with respect to interrupts. Reviewed by: bde, jhb Notes: svn path=/head/; revision=167429
* Fixed some style bugs. Routine except:Bruce Evans2007-02-061-34/+35
| | | | | | | | - don't use __GNUCLIKE___OFFSETOF, since __offsetof() is a standard FreeBSD implementaion detail which has nothing to do with GNUC. Notes: svn path=/head/; revision=166540
* Simplified PCPU_GET() and PCPU_SET(). We must copy through a temporaryBruce Evans2007-02-061-36/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | variable to avoid invalid constraints in dead code. Use an array of u_char's (inside a struct) instead of a char/short/int/long variable so that the variable and its accesses can be spelled in the same way in all cases and code doesn't need to be cloned just to hold the spelling differences. Fixed strict-aliasing errors in PCPU_SET() and in the amd64 PCPU_GET(). Cast to (void *) as in rev.1.37 of the i386 version where the errors were fixed for the i386 PCPU_GET() only. It would be more correct to copy to and from the temp. variable using memcpy(), but then an ifdef tangle would be required to ensure using the builtin memcpy(). We depend on fairly aggressive optimization to put the temp. variable only in a register despite it being copied using *(type *)(void *)&anothertype and could depend on this when using memcpy() too. This seems to work right even for -O0, but the -O0 case has not been completely tested. This change gives identical object code for all object files in LINT on amd64 (except for one file with a __TIME__ stamp). For LINT on i386 it gives unimportant differences in instruction order and padding in a few object files. This was only tested for -O. This change (actually a previous version of it) gives the following reductions in the number of object files in LINT that fail to compile with -O2 but without the -fno-strict-aliasing kludge: - amd64: 29 (down from 211) - i386: 36 (down from 47) gcc-3.4.6 actually allows the invalid constraints that result from not using the temp. variable, at least with -O[1-2], but gcc-3.3.3 crashes on them and I don't want to depend on compiler bugs. Notes: svn path=/head/; revision=166536
* Test commit after repoman upgrade. Remove one of my many email addressesPeter Wemm2006-05-121-1/+1
| | | | | | | from a copyright message. Notes: svn path=/head/; revision=158502
* Move global variable private_tss into per-cpu area.David Xu2005-12-261-1/+2
| | | | | | | Reviewed by: jhb Notes: svn path=/head/; revision=153726
* Change the segment limits to 4GB, we set the user accessible bit on allPeter Wemm2005-04-131-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | of the kernel address space already. Intel recommend this anyway, because using a non-4GB limit adds an additional clock cycle to address generation. We were able to install 4GB segments into the LDT, so any limits we imposed on %cs and %ds were academic anyway. More importantly, this allows us to make a page in the kernel readable to user applications, for holding things like the signal trampoline and other fun things. Move the user %cs/%ds segments from the LDT to the GDT. There was no good reason for them to be there anyway. The old LDT entries are still there but we can now relax the restriction that prevented users from emptying the default LDT entries. Putting user and kernel %cs and %ds together allows us to access the fast sysenter/sysexit/syscall/sysret instructions. syscall/sysret in particular require that the user/kernel segments be laid out this way. Reserve a slot specifically for NDIS while here. Create two user controllable slots in the GDT that are context switched with the (kernel) thread. This allows user applications to set two user privilige selectors to arbitary values. Create i386_set_fsbase(void *base) and friends. (get/set, fs/gs). For i386, %gs is used by tls and the thread libraries and this means that user processes no longer have to have the cost of having a custom LDT, and we will no longer to do a ldt switch when activating a kthread/ithread in the usual case any more. In other words, we can now set the base address for %fs and %gs to arbitary addresses without the pain of messing with ldt segments. Notes: svn path=/head/; revision=145034
* netchild's mega-patch to isolate compiler dependencies into a centralJoerg Wunsch2005-03-021-2/+7
| | | | | | | | | | | | | | | | | | | place. This moves the dependency on GCC's and other compiler's features into the central sys/cdefs.h file, while the individual source files can then refer to #ifdef __COMPILER_FEATURE_FOO where they by now used to refer to #if __GNUC__ > 3.1415 && __BARC__ <= 42. By now, GCC and ICC (the Intel compiler) have been actively tested on IA32 platforms by netchild. Extension to other compilers is supposed to be possible, of course. Submitted by: netchild Reviewed by: various developers on arch@, some time ago Notes: svn path=/head/; revision=143063
* These are changes to allow to use the Intel C/C++ compiler (lang/icc)Tom Rhodes2004-03-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | to build the kernel. It doesn't affect the operation if gcc. Most of the changes are just adding __INTEL_COMPILER to #ifdef's, as icc v8 may define __GNUC__ some parts may look strange but are necessary. Additional changes: - in_cksum.[ch]: * use a generic C version instead of the assembly version in the !gcc case (ASM code breaks with the optimizations icc does) -> no bad checksums with an icc compiled kernel Help from: andre, grehan, das Stolen from: alpha version via ppc version The entire checksum code should IMHO be replaced with the DragonFly version (because it isn't guaranteed future revisions of gcc will include similar optimizations) as in: ---snip--- Revision Changes Path 1.12 +1 -0 src/sys/conf/files.i386 1.4 +142 -558 src/sys/i386/i386/in_cksum.c 1.5 +33 -69 src/sys/i386/include/in_cksum.h 1.5 +2 -0 src/sys/netinet/igmp.c 1.6 +0 -1 src/sys/netinet/in.h 1.6 +2 -0 src/sys/netinet/ip_icmp.c 1.4 +3 -4 src/contrib/ipfilter/ip_compat.h 1.3 +1 -2 src/sbin/natd/icmp.c 1.4 +0 -1 src/sbin/natd/natd.c 1.48 +1 -0 src/sys/conf/files 1.2 +0 -1 src/sys/conf/files.amd64 1.13 +0 -1 src/sys/conf/files.i386 1.5 +0 -1 src/sys/conf/files.pc98 1.7 +1 -1 src/sys/contrib/ipfilter/netinet/fil.c 1.10 +2 -3 src/sys/contrib/ipfilter/netinet/ip_compat.h 1.10 +1 -1 src/sys/contrib/ipfilter/netinet/ip_fil.c 1.7 +1 -1 src/sys/dev/netif/txp/if_txp.c 1.7 +1 -1 src/sys/net/ip_mroute/ip_mroute.c 1.7 +1 -2 src/sys/net/ipfw/ip_fw2.c 1.6 +1 -2 src/sys/netinet/igmp.c 1.4 +158 -116 src/sys/netinet/in_cksum.c 1.6 +1 -1 src/sys/netinet/ip_gre.c 1.7 +1 -2 src/sys/netinet/ip_icmp.c 1.10 +1 -1 src/sys/netinet/ip_input.c 1.10 +1 -2 src/sys/netinet/ip_output.c 1.13 +1 -2 src/sys/netinet/tcp_input.c 1.9 +1 -2 src/sys/netinet/tcp_output.c 1.10 +1 -1 src/sys/netinet/tcp_subr.c 1.10 +1 -1 src/sys/netinet/tcp_syncache.c 1.9 +1 -2 src/sys/netinet/udp_usrreq.c 1.5 +1 -2 src/sys/netinet6/ipsec.c 1.5 +1 -2 src/sys/netproto/ipsec/ipsec.c 1.5 +1 -1 src/sys/netproto/ipsec/ipsec_input.c 1.4 +1 -2 src/sys/netproto/ipsec/ipsec_output.c and finally remove sys/i386/i386 in_cksum.c sys/i386/include in_cksum.h ---snip--- - endian.h: * DTRT in C++ mode - quad.h: * we don't use gcc v1 anymore, remove support for it Suggested by: bde (long ago) - assym.h: * avoid zero-length arrays (remove dependency on a gcc specific feature) This change changes the contents of the object file, but as it's only used to generate some values for a header, and the generator knows how to handle this, there's no impact in the gcc case. Explained by: bde Submitted by: Marius Strobl <marius@alchemy.franken.de> - aicasm.c: * minor change to teach it about the way icc spells "-nostdinc" Not approved by: gibbs (no reply to my mail) - bump __FreeBSD_version (lang/icc needs to know about the changes) Incarnations of this patch survive gcc compiles since a loooong time, I use it on my desktop. An icc compiled kernel works since Nov. 2003 (exceptions: snd_* if used as modules), it survives a build of the entire ports collection with icc. Parts of this commit contains suggestions or submissions from Marius Strobl <marius@alchemy.franken.de>. Reviewed by: -arch Submitted by: netchild Notes: svn path=/head/; revision=126891
* MFamd64: use a less compiler-intensive MD implementation of 'curthread'Peter Wemm2003-11-201-0/+10
| | | | | | | | | so that the compiler doesn't have to do so much work. Approved by: re (jhb) Notes: svn path=/head/; revision=122931
* Fixed pedantic warnings for statement-expressions using __extension__Bruce Evans2003-11-171-4/+4
| | | | | | | | and by not using a statement-expression for the non-expression __PCPU_SET(). Notes: svn path=/head/; revision=122833
* Fixed a pedantic syntax error (a stray semicolon at the end ofBruce Evans2003-11-171-1/+1
| | | | | | | PCPU_MD_FIELDS). Notes: svn path=/head/; revision=122829
* New APIC support code:John Baldwin2003-11-031-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - The apic interrupt entry points have been rewritten so that each entry point can serve 32 different vectors. When the entry is executed, it uses one of the 32-bit ISR registers to determine which vector in its assigned range was triggered. Thus, the apic code can support 159 different interrupt vectors with only 5 entry points. - We now always to disable the local APIC to work around an errata in certain PPros and then re-enable it again if we decide to use the APICs to route interrupts. - We no longer map IO APICs or local APICs using special page table entries. Instead, we just use pmap_mapdev(). We also no longer export the virtual address of the local APIC as a global symbol to the rest of the system, but only in local_apic.c. To aid this, the APIC ID of each CPU is exported as a per-CPU variable. - Interrupt sources are provided for each intpin on each IO APIC. Currently, each source is given a unique interrupt vector meaning that PCI interrupts are not shared on most machines with an I/O APIC. That mapping for interrupt sources to interrupt vectors is up to the APIC enumerator driver however. - We no longer probe to see if we need to use mixed mode to route IRQ 0, instead we always use mixed mode to route IRQ 0 for now. This can be disabled via the 'NO_MIXED_MODE' kernel option. - The npx(4) driver now always probes to see if a built-in FPU is present since this test can now be performed with the new APIC code. However, an SMP kernel will panic if there is more than one CPU and a built-in FPU is not found. - PCI interrupts are now properly routed when using APICs to route interrupts, so remove the hack to psuedo-route interrupts when the intpin register was read. - The apic.h header was moved to apicreg.h and a new apicvar.h header that declares the APIs used by the new APIC code was added. Notes: svn path=/head/; revision=121986
* Detour via (void *) to defeat gcc's strict-aliasing warnings when usingPeter Wemm2003-07-301-3/+3
| | | | | | | | | | -O2 or -Os (such as 'make release'). This commit brought to you by the warning: dereferencing type-punned pointer will break strict-aliasing rules Notes: svn path=/head/; revision=118188
* Fix the false IPIs on smp when using LAZY_SWITCH caused by pmap_activate()Peter Wemm2003-06-271-0/+1
| | | | | | | not releasing the pm_active bit in the old pmap. Notes: svn path=/head/; revision=116926
generated by cgit v1.3 (git 2.53.0) at 2026-04-10 17:17:31 +0000