src/sys/modules/ena, branch release/13.2.0

Upgrade ENA to v2.4.1

2021-10-07T16:10:32Z

ena: Remove redundant declaration of ena_log_level. GCC6 raises a -Wredundant-decl error due to duplicate declarations in ena_fbsd_log.h and ena_plat.h. Sponsored by: Chelsio Communications (cherry picked from commit 8843787aa1bdbd10de6ba47a04489179ec2d2d3c) ena: Avoid unnecessary mbuf collapses for LLQ condition In case of Low-latency Queue, one small enough descriptor can be pushed directly to the ENA hw, thus saving one fragment. Check for this condition before performing collapse. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit c81f8c26115a64b9a97ecdb2a64e824dd839ee73) ena: Trigger reset on ena_com_prepare_tx failure All ena_com_prepare_tx errors other than ENA_COM_NO_MEM are fatal and require device reset. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 36130d2979d695dd439bc607feb00dcdb9a1937b) ena: Prevent reset after device destruction Check for ENA_FLAG_TRIGGER_RESET inside a locked context in order to avoid potential race conditions with ena_destroy_device. This aligns the reset task logic with the Linux driver. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 433ab9b6987b42b3e5b25b8b5dc7e5178c7ef9bb) ena: Add extra log messages Stay aligned with the Linux driver by adding the following logs: * inform the user about retrying queue creation * warn on non-empty ena_tx_buffer.mbuf prior to ena_tx_map_mbuf Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 77160654a162b5faa8ad7a02e18d2bef2589f868) ena: Add locking assertions ENA silently assumed that ena_up, ena_down and ena_start_xmit routines should be called within locked context. Driver's logic heavily assumes on concurrent access to those routines, so for safety and better documentation about this assumption, the locking assertions were added to the above functions. The assertion was added only for the main steps (skipping the helper functions) which can be called from multiple places including the kernel and the driver itself. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit cb98c439d66c303353a9f4abbbe9ddb51559c638) ena: Move RSS logic into its own source files Delegate RSS related functionality into separate .c/.h files in preparation for the full RSS support. While at it, reorder functions and remove prototypes for ones with internal linkage. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 986e7b9227668caf9620f207e3c1d708c87b634d) ena: Disable meta descriptor caching for netmap If LLQ is being used, `ena_tx_ctx.meta_valid` must stay enabled. This fixes netmap support on latest generation ENA HW and aligns it with the core driver behavior. As netmap doesn't support any csum offloads, the `adapter->disable_meta_caching` value can be simply passed to the HW. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit a831466830de6ab55fc03170290b313157196e81) ena: Share ena_global_lock between driver instances In order to use `ena_global_lock` in sysctl context, it must be kept outside the driver instance's software context, as sysctls can be called before attach and after detach, leading to lock use before sx_init and after sx_destroy otherwise. Solve this issue by turning `ena_global_lock` into a file scope variable, shared between all instances of the driver and associated sysctl context, and in turn initialized/destroyed in dedicated SYSINIT/SYSUNINIT functions. As a side effect, this change also fixes existing race in the reset routine, when simultaneously accessing sysctl exposed properties. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 07aff471c0de2de9a1dc5c7749c46b525bdd0201) ena: Add missing statistics Provide the following sysctl statistics in order to stay aligned with the Linux driver: * rx_ring.csum_good * tx_ring.unmask_interrupt_num Also rename the 'bad_csum' statistic name to 'csum_bad' for alignment. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 223c8cb12e951c63807300a0cbdc4a1569520b4b) ena: Implement full RSS reconfiguration Bind RX/TX queues and MSI-X vectors to matching CPUs based on the RSS bucket entries. Introduce sysctls for the following RSS functionality: - rss.indir_table: indirection table mapping - rss.indir_table_size: indirection table size - rss.key: RSS hash key (if Toeplitz used) Said sysctls are only available when compiled without `option RSS`, as kernel-side RSS support currently doesn't offer RSS reconfiguration. Migrate the hash algorithm from CRC32 to Toeplitz and change the initial hash value to 0x0 in order to match the standard Toeplitz implementation. Provide helpers for hash key inversion required for HW operations. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 6d1ef2abd330fac4057f092abbbdc28a568b4327) ena: fix building in-kernel driver When building ENA as compiled into the kernel, the driver would fail to build. Resolve the problem by introducing the following changes: 1. Add missing `ena_rss.c` entry in `sys/conf/files`. 2. Prevent SYSCTL_ADD_INT from throwing an assert due to an extra CTLTYPE_INT flag. Fixes: 986e7b92276 ("ena: Move RSS logic into its own source files") Fixes: 6d1ef2abd33 ("ena: Implement full RSS reconfiguration") Obtained from: Semihalf Sponsored by: Amazon, Inc. MFC after: 1 week (cherry picked from commit a3f0d18237bdcf272461d3b4b682de384c572144) ena: Update driver version to v2.4.1 Some of the changes in this release: * Hardware RSS hash key reconfiguration and indirection table reconfiguration support. * Full kernel RSS support. * Extra statistic counters. * Netmap support for ENAv3. * Locking assertions. * Extra log messages. * Reset handling fixes. Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 42c7760be3ea420668f625f2064ae347aa7e818e)

Add SPDX license tag to the ENA driver files

2020-11-18T15:07:34Z

Refering to guide: https://wiki.freebsd.org/SPDX the SPDX tag should not replace the standard license text, however it should be added over the standard license text to make the automation easier. Because of that, the old license was kept, but the SPDX tag was added on top of every ENA driver file. Submited by: Michal Krawczyk Obtained from: Semihalf Sponsored by: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27117

ena: fix module build after r362530

2020-06-26T09:32:57Z

Somehow I missed the makefile when moving the change from phabricator to svn. MFC after: 1 week X-MFC with: r362530

Introduce NETMAP support in ENA

2019-10-31T15:51:18Z

Mock implementation of NETMAP routines is located in ena_netmap.c/.h files. All code is protected under the DEV_NETMAP macro. Makefile was updated with files and flag. As ENA driver provide own implementations of (un)likely it must be undefined before including NETMAP headers. ena_netmap_attach function is called on the end of NIC attach. It fills structure with NIC configuration and callbacks. Then provides it to netmap_attach. Similarly netmap_detach is called during ena_detach. Three callbacks are used. nm_register is implemented by ena_netmap_reg. It is called when user space application open or close NIC in NETMAP mode. Current action is recognized based on onoff parameter: true means on and false off. As NICs rings need to be reconfigured ena_down and ena_up are reused. When user space application wants to receive new packets from NIC nm_rxsync is called, and when there are new packets ready for Tx nm_txsync is called. Differential Revision: https://reviews.freebsd.org/D21934 Submitted by: Rafal Kozik Michal Krawczyk Obtained from: Semihalf Sponsored by: Amazon, Inc.

Split Rx/Tx from initialization code in ENA driver

2019-10-31T15:44:26Z

Move Rx/Tx routines to separate file. Some functions: * ena_restore_device, * ena_destroy_device, * ena_up, * ena_down, * ena_refill_rx_bufs could be reused in upcoming netmap code in the driver. To make it possible, they were moved to ena.h header. Differential Revision: https://reviews.freebsd.org/D21933 Submitted by: Rafal Kozik Obtained from: Semihalf Sponsored by: Amazon, Inc.

Update ENA version to v2.0.0

2019-05-30T13:52:32Z

ENAv2 introduces many new features, bug fixes and improvements. Main new features are LLQ (Low Latency Queues) and independent queues reconfiguration using sysctl commands. The year in copyright notice was updated to 2019. Submitted by: Michal Krawczyk Obtained from: Semihalf Sponsored by: Amazon, Inc.

Add support for Amazon Elastic Network Adapter (ENA) NIC

2017-05-22T14:46:13Z

ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk Jakub Palider Jan Medala Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427