src/sys/xen/evtchn, branch main

x86/xen: fix out of bounds access to the event channel masks on resume

2024-02-22T10:08:03Z

When resuming from migration or suspension all regular event channels ports are reset to the INVALID_EVTCHN value, and drivers should re-initialize them according to the new value provided by the other end of the connection. However, the driver would first attempt to unbind the event channel handler before attempting to bind it using the newly provided port. This unbind uses the stale event channel port that has been set to INVALID_EVTCHN for some operations (notably as a result of the handler removal the interrupt subsystem ends up calling disable intr and source PIC hooks). This was fine when INVALID_EVTCHN was 0, as then the operation would just result in pointless setting of the 0 bit in the different event channel related control arrays (evtchn_{pending,mask} for example). However with the change to define INVALID_EVTCHN as ~0 the write is no longer pointless, and we end up triggering a page-fault, or corrupting random data that happens to be mapped at the array position + ~0 bits. In hindsight the change of INVALID_EVTCHN from 0 to ~0 was way more risky than initially assessed, and I believe has end up resulting in more fragile code for no real benefit. Fix the disable intr and source wrappers to check whether the event channel is valid before attempting to use it. Also introduce some extra KASSERTs in several array accesses in order to avoid out of bounds accesses if INVALID_EVTCHN ever reaches those functions. Fixes: 1797ff962769 ('xen/intr: cleanup event channel number use') MFC after: 1 week Sponsored by: Cloud Software Group Reviewed by: markj Differential revision: https://reviews.freebsd.org/D43928

xen: add atomic #defines to accomodate differing xen_ulong_t sizes

2023-12-15T13:59:26Z

Alas, ARM declared xen_ulong_t to be 64-bits long, unlike i386 where it matches the word size. As a result, compatibility wrappers are needed for Xen atomic operations. Reviewed by: royger

xen: remove declaration of evtchn_device_upcall()

2023-11-28T12:25:35Z

This function was removed at 5779d8ad577. Long past time to remove the declaration to ensure people aren't confused. Differential Revision: https://reviews.freebsd.org/D30865 Reviewed by: royger

sys: Remove $FreeBSD$: two-line .h pattern

2023-08-16T17:54:11Z

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

xen/intr: move evtchn_type to intr-internal.h

2023-04-14T13:58:53Z

The evtchn_type enum is only touched by the Xen interrupt code. Other event channel uses no longer need the value, so that has been moved to restrict its use. Copyright note. The current evtchn_type was introduced at 76acc41fb7c7 by Justin T. Gibbs. This in turn appears to have been heavily inspired by 30d1eefe3937 done by Kip Macy. Reviewed by: royger

xen: switch to using core atomics for synchronization

2023-03-29T07:51:42Z

Now that the atomic macros are always genuinely atomic on x86, they can be used for synchronization with Xen. A single core VM isn't too unusual, but actual single core hardware is uncommon. Replace an open-coding of evtchn_clear_port() with the inline. Substantially inspired by work done by Julien Grall , 2014-01-13 17:40:58. Reviewed by: royger MFC after: 1 week

xen: switch to use headers in contrib

2022-02-07T09:11:56Z

These headers originate with the Xen project and shouldn't be mixed with the main portion of the FreeBSD kernel. Notably they shouldn't be the target of clean-up commits. Switch to use the headers in sys/contrib/xen. Reviewed by: royger

x86/xen: remove PVHv1 code

2021-05-17T09:41:21Z

PVHv1 was officially removed from Xen in 4.9, so just axe the related code from FreeBSD. Note FreeBSD supports PVHv2, which is the replacement for PVHv1. Sponsored by: Citrix Systems R&D Reviewed by: kib, Elliott Mitchell Differential Revision: https://reviews.freebsd.org/D30228

xen: import a proper event channel user-space device

2014-10-22T16:57:11Z

The user-space event channel device is used by applications to receive and send event channel interrupts. This device is based on the Linux evtchn device. Sponsored by: Citrix Systems R&D xen/evtchn/evtchn_dev.c: - Remove the old event channel device, which was already disabled in the build system. dev/xen/evtchn/evtchn_dev.c: - Import a new event channel device based on the one present in Linux. - This device allows the following operations: - Bind VIRQ event channels (ioctl). - Bind regular event channels (ioctl). - Create and bind new event channels (ioctl). - Unbind event channels (ioctl). - Send notifications to event channels (ioctl). - Reset the device shared memory ring (ioctl). - Unmask event channels (write). - Receive event channel upcalls (read). - The new code is MP safe, and can be used concurrently. conf/files: - Add the new device to the build system.

Implement vector callback for PVHVM and unify event channel implementations

2013-08-29T19:52:18Z

Re-structure Xen HVM support so that: - Xen is detected and hypercalls can be performed very early in system startup. - Xen interrupt services are implemented using FreeBSD's native interrupt delivery infrastructure. - the Xen interrupt service implementation is shared between PV and HVM guests. - Xen interrupt handlers can optionally use a filter handler in order to avoid the overhead of dispatch to an interrupt thread. - interrupt load can be distributed among all available CPUs. - the overhead of accessing the emulated local and I/O apics on HVM is removed for event channel port events. - a similar optimization can eventually, and fairly easily, be used to optimize MSI. Early Xen detection, HVM refactoring, PVHVM interrupt infrastructure, and misc Xen cleanups: Sponsored by: Spectra Logic Corporation Unification of PV & HVM interrupt infrastructure, bug fixes, and misc Xen cleanups: Submitted by: Roger Pau Monné Sponsored by: Citrix Systems R&D sys/x86/x86/local_apic.c: sys/amd64/include/apicvar.h: sys/i386/include/apicvar.h: sys/amd64/amd64/apic_vector.S: sys/i386/i386/apic_vector.s: sys/amd64/amd64/machdep.c: sys/i386/i386/machdep.c: sys/i386/xen/exception.s: sys/x86/include/segments.h: Reserve IDT vector 0x93 for the Xen event channel upcall interrupt handler. On Hypervisors that support the direct vector callback feature, we can request that this vector be called directly by an injected HVM interrupt event, instead of a simulated PCI interrupt on the Xen platform PCI device. This avoids all of the overhead of dealing with the emulated I/O APIC and local APIC. It also means that the Hypervisor can inject these events on any CPU, allowing upcalls for different ports to be handled in parallel. sys/amd64/amd64/mp_machdep.c: sys/i386/i386/mp_machdep.c: Map Xen per-vcpu area during AP startup. sys/amd64/include/intr_machdep.h: sys/i386/include/intr_machdep.h: Increase the FreeBSD IRQ vector table to include space for event channel interrupt sources. sys/amd64/include/pcpu.h: sys/i386/include/pcpu.h: Remove Xen HVM per-cpu variable data. These fields are now allocated via the dynamic per-cpu scheme. See xen_intr.c for details. sys/amd64/include/xen/hypercall.h: sys/dev/xen/blkback/blkback.c: sys/i386/include/xen/xenvar.h: sys/i386/xen/clock.c: sys/i386/xen/xen_machdep.c: sys/xen/gnttab.c: Prefer FreeBSD primatives to Linux ones in Xen support code. sys/amd64/include/xen/xen-os.h: sys/i386/include/xen/xen-os.h: sys/xen/xen-os.h: sys/dev/xen/balloon/balloon.c: sys/dev/xen/blkback/blkback.c: sys/dev/xen/blkfront/blkfront.c: sys/dev/xen/console/xencons_ring.c: sys/dev/xen/control/control.c: sys/dev/xen/netback/netback.c: sys/dev/xen/netfront/netfront.c: sys/dev/xen/xenpci/xenpci.c: sys/i386/i386/machdep.c: sys/i386/include/pmap.h: sys/i386/include/xen/xenfunc.h: sys/i386/isa/npx.c: sys/i386/xen/clock.c: sys/i386/xen/mp_machdep.c: sys/i386/xen/mptable.c: sys/i386/xen/xen_clock_util.c: sys/i386/xen/xen_machdep.c: sys/i386/xen/xen_rtc.c: sys/xen/evtchn/evtchn_dev.c: sys/xen/features.c: sys/xen/gnttab.c: sys/xen/gnttab.h: sys/xen/hvm.h: sys/xen/xenbus/xenbus.c: sys/xen/xenbus/xenbus_if.m: sys/xen/xenbus/xenbusb_front.c: sys/xen/xenbus/xenbusvar.h: sys/xen/xenstore/xenstore.c: sys/xen/xenstore/xenstore_dev.c: sys/xen/xenstore/xenstorevar.h: Pull common Xen OS support functions/settings into xen/xen-os.h. sys/amd64/include/xen/xen-os.h: sys/i386/include/xen/xen-os.h: sys/xen/xen-os.h: Remove constants, macros, and functions unused in FreeBSD's Xen support. sys/xen/xen-os.h: sys/i386/xen/xen_machdep.c: sys/x86/xen/hvm.c: Introduce new functions xen_domain(), xen_pv_domain(), and xen_hvm_domain(). These are used in favor of #ifdefs so that FreeBSD can dynamically detect and adapt to the presence of a hypervisor. The goal is to have an HVM optimized GENERIC, but more is necessary before this is possible. sys/amd64/amd64/machdep.c: sys/dev/xen/xenpci/xenpcivar.h: sys/dev/xen/xenpci/xenpci.c: sys/x86/xen/hvm.c: sys/sys/kernel.h: Refactor magic ioport, Hypercall table and Hypervisor shared information page setup, and move it to a dedicated HVM support module. HVM mode initialization is now triggered during the SI_SUB_HYPERVISOR phase of system startup. This currently occurs just after the kernel VM is fully setup which is just enough infrastructure to allow the hypercall table and shared info page to be properly mapped. sys/xen/hvm.h: sys/x86/xen/hvm.c: Add definitions and a method for configuring Hypervisor event delievery via a direct vector callback. sys/amd64/include/xen/xen-os.h: sys/x86/xen/hvm.c: sys/conf/files: sys/conf/files.amd64: sys/conf/files.i386: Adjust kernel build to reflect the refactoring of early Xen startup code and Xen interrupt services. sys/dev/xen/blkback/blkback.c: sys/dev/xen/blkfront/blkfront.c: sys/dev/xen/blkfront/block.h: sys/dev/xen/control/control.c: sys/dev/xen/evtchn/evtchn_dev.c: sys/dev/xen/netback/netback.c: sys/dev/xen/netfront/netfront.c: sys/xen/xenstore/xenstore.c: sys/xen/evtchn/evtchn_dev.c: sys/dev/xen/console/console.c: sys/dev/xen/console/xencons_ring.c Adjust drivers to use new xen_intr_*() API. sys/dev/xen/blkback/blkback.c: Since blkback defers all event handling to a taskqueue, convert this task queue to a "fast" taskqueue, and schedule it via an interrupt filter. This avoids an unnecessary ithread context switch. sys/xen/xenstore/xenstore.c: The xenstore driver is MPSAFE. Indicate as much when registering its interrupt handler. sys/xen/xenbus/xenbus.c: sys/xen/xenbus/xenbusvar.h: Remove unused event channel APIs. sys/xen/evtchn.h: Remove all kernel Xen interrupt service API definitions from this file. It is now only used for structure and ioctl definitions related to the event channel userland device driver. Update the definitions in this file to match those from NetBSD. Implementing this interface will be necessary for Dom0 support. sys/xen/evtchn/evtchnvar.h: Add a header file for implemenation internal APIs related to managing event channels event delivery. This is used to allow, for example, the event channel userland device driver to access low-level routines that typical kernel consumers of event channel services should never access. sys/xen/interface/event_channel.h: sys/xen/xen_intr.h: Standardize on the evtchn_port_t type for referring to an event channel port id. In order to prevent low-level event channel APIs from leaking to kernel consumers who should not have access to this data, the type is defined twice: Once in the Xen provided event_channel.h, and again in xen/xen_intr.h. The double declaration is protected by __XEN_EVTCHN_PORT_DEFINED__ to ensure it is never declared twice within a given compilation unit. sys/xen/xen_intr.h: sys/xen/evtchn/evtchn.c: sys/x86/xen/xen_intr.c: sys/dev/xen/xenpci/evtchn.c: sys/dev/xen/xenpci/xenpcivar.h: New implementation of Xen interrupt services. This is similar in many respects to the i386 PV implementation with the exception that events for bound to event channel ports (i.e. not IPI, virtual IRQ, or physical IRQ) are further optimized to avoid mask/unmask operations that aren't necessary for these edge triggered events. Stubs exist for supporting physical IRQ binding, but will need additional work before this implementation can be fully shared between PV and HVM. sys/amd64/amd64/mp_machdep.c: sys/i386/i386/mp_machdep.c: sys/i386/xen/mp_machdep.c sys/x86/xen/hvm.c: Add support for placing vcpu_info into an arbritary memory page instead of using HYPERVISOR_shared_info->vcpu_info. This allows the creation of domains with more than 32 vcpus. sys/i386/i386/machdep.c: sys/i386/xen/clock.c: sys/i386/xen/xen_machdep.c: sys/i386/xen/exception.s: Add support for new event channle implementation.