src/sys/dev/ena/ena_netmap.c, branch release/12.3.0

Upgrade ENA to v2.4.1

2021-10-11T14:32:14Z

Approved by: re ena: Remove redundant declaration of ena_log_level. GCC6 raises a -Wredundant-decl error due to duplicate declarations in ena_fbsd_log.h and ena_plat.h. Submitted by: jhb Sponsored by: Chelsio Communications (cherry picked from commit 8843787aa1bdbd10de6ba47a04489179ec2d2d3c) ena: Avoid unnecessary mbuf collapses for LLQ condition In case of Low-latency Queue, one small enough descriptor can be pushed directly to the ENA hw, thus saving one fragment. Check for this condition before performing collapse. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit c81f8c26115a64b9a97ecdb2a64e824dd839ee73) ena: Trigger reset on ena_com_prepare_tx failure All ena_com_prepare_tx errors other than ENA_COM_NO_MEM are fatal and require device reset. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 36130d2979d695dd439bc607feb00dcdb9a1937b) ena: Prevent reset after device destruction Check for ENA_FLAG_TRIGGER_RESET inside a locked context in order to avoid potential race conditions with ena_destroy_device. This aligns the reset task logic with the Linux driver. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 433ab9b6987b42b3e5b25b8b5dc7e5178c7ef9bb) ena: Add extra log messages Stay aligned with the Linux driver by adding the following logs: * inform the user about retrying queue creation * warn on non-empty ena_tx_buffer.mbuf prior to ena_tx_map_mbuf Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 77160654a162b5faa8ad7a02e18d2bef2589f868) ena: Add locking assertions ENA silently assumed that ena_up, ena_down and ena_start_xmit routines should be called within locked context. Driver's logic heavily assumes on concurrent access to those routines, so for safety and better documentation about this assumption, the locking assertions were added to the above functions. The assertion was added only for the main steps (skipping the helper functions) which can be called from multiple places including the kernel and the driver itself. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit cb98c439d66c303353a9f4abbbe9ddb51559c638) ena: Move RSS logic into its own source files Delegate RSS related functionality into separate .c/.h files in preparation for the full RSS support. While at it, reorder functions and remove prototypes for ones with internal linkage. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 986e7b9227668caf9620f207e3c1d708c87b634d) ena: Disable meta descriptor caching for netmap If LLQ is being used, `ena_tx_ctx.meta_valid` must stay enabled. This fixes netmap support on latest generation ENA HW and aligns it with the core driver behavior. As netmap doesn't support any csum offloads, the `adapter->disable_meta_caching` value can be simply passed to the HW. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit a831466830de6ab55fc03170290b313157196e81) ena: Share ena_global_lock between driver instances In order to use `ena_global_lock` in sysctl context, it must be kept outside the driver instance's software context, as sysctls can be called before attach and after detach, leading to lock use before sx_init and after sx_destroy otherwise. Solve this issue by turning `ena_global_lock` into a file scope variable, shared between all instances of the driver and associated sysctl context, and in turn initialized/destroyed in dedicated SYSINIT/SYSUNINIT functions. As a side effect, this change also fixes existing race in the reset routine, when simultaneously accessing sysctl exposed properties. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 07aff471c0de2de9a1dc5c7749c46b525bdd0201) ena: Add missing statistics Provide the following sysctl statistics in order to stay aligned with the Linux driver: * rx_ring.csum_good * tx_ring.unmask_interrupt_num Also rename the 'bad_csum' statistic name to 'csum_bad' for alignment. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 223c8cb12e951c63807300a0cbdc4a1569520b4b) ena: Implement full RSS reconfiguration Bind RX/TX queues and MSI-X vectors to matching CPUs based on the RSS bucket entries. Introduce sysctls for the following RSS functionality: - rss.indir_table: indirection table mapping - rss.indir_table_size: indirection table size - rss.key: RSS hash key (if Toeplitz used) Said sysctls are only available when compiled without `option RSS`, as kernel-side RSS support currently doesn't offer RSS reconfiguration. Migrate the hash algorithm from CRC32 to Toeplitz and change the initial hash value to 0x0 in order to match the standard Toeplitz implementation. Provide helpers for hash key inversion required for HW operations. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 6d1ef2abd330fac4057f092abbbdc28a568b4327) ena: fix building in-kernel driver When building ENA as compiled into the kernel, the driver would fail to build. Resolve the problem by introducing the following changes: 1. Add missing `ena_rss.c` entry in `sys/conf/files`. 2. Prevent SYSCTL_ADD_INT from throwing an assert due to an extra CTLTYPE_INT flag. Fixes: 986e7b92276 ("ena: Move RSS logic into its own source files") Fixes: 6d1ef2abd33 ("ena: Implement full RSS reconfiguration") Submitted by: Artur Rojek Obtained from: Semihalf Sponsored by: Amazon, Inc. MFC after: 1 week (cherry picked from commit a3f0d18237bdcf272461d3b4b682de384c572144) ena: Update driver version to v2.4.1 Some of the changes in this release: * Hardware RSS hash key reconfiguration and indirection table reconfiguration support. * Full kernel RSS support. * Extra statistic counters. * Netmap support for ENAv3. * Locking assertions. * Extra log messages. * Reset handling fixes. Submitted by: Michal Krawczyk Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 42c7760be3ea420668f625f2064ae347aa7e818e)

Upgrade ENA to v2.4.0

2021-10-11T14:31:50Z

Approved by: re ena: change ENA C++-style comment into C-style According to man style(9), only C-style comments should be used. Submitted by: Michal Krawczyk Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 438c9e3cf89403628ec237cfecdd0538f208087b) ena: add support for the large LLQ headers in ENA Default LLQ (Low-latency queue) maximum header size is 96 bytes and can be too small for some types of packets - like IPv6 packets with multiple extension. This can be fixed, by using large LLQ headers. If the device supports larger LLQ headers, the user can activate this feature by setting sysctl tunable 'hw.ena.force_large_llq_header' to '1' in the /boot/loader.conf file. In case the device isn't supporting this feature, the default value (96B) will be used. Submitted by: Michal Krawczyk Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit beaadec9eaec8e6b266faff3b0880a141728fcef) ena: remove surplus NULL checks when freeing ENA resources Calling free on a NULL pointer is valid, as appropriate check is already done internally: /* free(NULL, ...) does nothing */ if (addr == NULL) return; Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit ddec69e6a796283497ebada0569f34bd41bbdf37) ena: hide sysctl nodes for unused ENA queues IO queue related attributes are registered statically at driver attach with the rest of the ENA specific sysctl nodes. However, the number of queues can be changed at runtime via the `ena_sysctl_io_queues_nb` request, leading to a potential exposure of attributes for non-existing queues. Introduce a new `ena_sysctl_update_queue_node_nb` function, which updates the sysctl nodes after the number of queues is altered. This happens by either registering or unregistering node specific oids, based on a delta between the previous and current queue count. NOTE: All unregistered oids must be registered again before the driver detach, e.g. by another call to this function. Submitted by: Artur Rojek Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 0e7d31f63b9db869c91228d8ed1e984bdee2b931) Merge tag 'vendor/ena-com/2.4.0' Update the driver in order not to break its compilation and make use of the new ENA logging system Migrate platform code to the new logging system provided by ena_com layer. Make ENA_INFO the new default log level. Remove all explicit use of `device_printf`, all new logs requiring one of the log macros to be used. (cherry picked from commit 3fc5d816f8831d6fc2816ac97bd78dc486cd080c) Update ENA driver man page Bring the obsolete man page up to date: * update diagnostic error messages * add documentation of loader tunables * document netmap support * add a driver history section * update the contact information Submitted by: Artur Rojek Submitted by: Michal Krawczyk Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit e34856a2c44a45512463aed0d1794f34258c66ee) Update ENA version to v2.4.0 Some of the changes in this release: * Large LLQ headers, * Bug/stability fixes, * Change of the README/Documentation. Submitted by: Michal Krawczyk Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. (cherry picked from commit 93f0df457bf1d0e5f71839ab969c94d1f95813fb)

MFC: Merge ENA v2.3.0 driver

2020-11-25T03:24:43Z

r367805 Update ENA driver version to v2.3.0 r367803 Rename descriptions of the supported ENA devices r367802 Add ENI metrics for the ENA driver r367801 Add SPDX license tag to the ENA driver files r367800 Add Rx offsets support for the ENA driver r367799 Adjust ENA driver files to latest ena-com changes r367795 Fix completion descriptors alignment for the ENA Obtained from: Semihalf Sponsored by: Amazon, Inc

MFC: Merge ENA v2.2.0 driver

2020-09-06T14:41:35Z

r361530 Update ENA driver version to v2.2.0 r361529 Refactor ena_tx_map_mbuf() function r361528 Fix double-free bug within ena_detach() r361527 Allow disabling meta caching for ENA Tx path r361526 Create ENA IO queues with optional backoff r361525 Add sysctl node for ENA IO queues number adjustment r361524 Fix assumptions about number of IO queues in the ENA r361523 Rework ENA Tx buffer ring size reconfiguration r361521 Rework ENA Rx queue size configuration r361518 Improve indentation in ena_up() and ena_down() r361517 Expose argument names for non static ENA driver functions r361516 Use single global lock in the ENA driver r361515 Add trigger reset function in the ENA driver r361514 Provide ENA driver version in a sysctl node r361513 Remove unused argument from static function in ena.c r361512 Enable Tx drops reporting in the ENA driver r361511 Adjust ENA driver to the new HAL r358289 Rework and simplify Tx DMA mapping in ENA Obtained from: Semihalf Sponsored by: Amazon, Inc.

MFC: ENA netmap support and bug fixes

2020-09-06T14:23:31Z

r363638 Fix ENA build when integrated into kernel r354242 Make valdiate_rx_req_id static inline because it uses other static r354225 Update ENA version to v2.1.0 r354224 Add support for ENA NETMAP partial initialization r354223 Add support for ENA NETMAP Tx r354222 Add support for ENA NETMAP Rx r354221 Introduce NETMAP support in ENA r354220 Split Rx/Tx from initialization code in ENA driver r354219 Fix ENA keep-alive timeout due to prolonged reset r354218 Add WC support for arm64 in the ENA driver Obtained from: Semihalf Sponsored by: Amazon, Inc.