aboutsummaryrefslogtreecommitdiff
path: root/sys/dev/ena
Commit message (Collapse)AuthorAgeFilesLines
* Update ENA driver version to v2.3.0Marcin Wojtas2020-11-181-1/+1
| | | | | | | | | | | | | | The v2.3.0 introduces new ena_com layer, ENI metrics updates and SPDX license tags. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27120 Notes: svn path=/head/; revision=367805
* Rename descriptions of the supported ENA devicesMarcin Wojtas2020-11-182-6/+6
| | | | | | | | | | | | | | | | Some of the PCI ID were described as ENA with LLQ support - it's not fully accurate and because of that, their names were changed. Instead of LLQ, use RSERV0 for the description of those devices. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27119 Notes: svn path=/head/; revision=367803
* Add ENI metrics for the ENA driverMarcin Wojtas2020-11-183-1/+171
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new HAL allows the driver to read extra ENI stats. Exact meaning of each of them can be found in base/ena_defs/ena_admin_defs.h file and structure ena_admin_eni_stats. Those stats are being updated inside of the timer service, which is executed every second. ENI metrics are turned off by default. They can be enabled, using the sysctl node: dev.ena.X.eni_metrics.update_delay 0 value in this node means that the update is turned off. Other values determine how many seconds must pass, before ENI metrics will be updated. They can be acquired, using sysctl: sysctl dev.ena.X.eni_metrics Where X stands for the interface number. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27118 Notes: svn path=/head/; revision=367802
* Add SPDX license tag to the ENA driver filesMarcin Wojtas2020-11-188-8/+8
| | | | | | | | | | | | | | | | | | Refering to guide: https://wiki.freebsd.org/SPDX the SPDX tag should not replace the standard license text, however it should be added over the standard license text to make the automation easier. Because of that, the old license was kept, but the SPDX tag was added on top of every ENA driver file. Submited by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27117 Notes: svn path=/head/; revision=367801
* Add Rx offsets support for the ENA driverMarcin Wojtas2020-11-182-0/+8
| | | | | | | | | | | | | | | | | | For the first descriptor in a chain the data may start at an offset. It is optional feature of some devices, so the driver must ack that it supports it. The data pointer of the mbuf is simply shifted by the given value. Submitted by: Maciej Bielski <mba@semihalf.com> Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27116 Notes: svn path=/head/; revision=367800
* Adjust ENA driver files to latest ena-com changesMarcin Wojtas2020-11-184-139/+114
| | | | | | | | | | | | | | | | | * Use the new API of ena_trace_* * Fix typo syndrom --> syndrome * Remove validation of the Rx req ID (already performed in the ena-com) * Remove usage of deprecated ENA_ASSERT macro Submitted by: Ido Segev <idose@amazon.com> Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27115 Notes: svn path=/head/; revision=367799
* Fix completion descriptors alignment for the ENAMarcin Wojtas2020-11-181-2/+2
| | | | | | | | | | | | | | | | | | | | The latest generation hardware requires IO CQ (completion queue) descriptors memory to be aligned to a 4K. It needs that feature for the best performance. Allocating unaligned descriptors will have a big performance impact as the packet processing in a HW won't be optimized properly. For that purpose adjust ena_dma_alloc() to support it. It's a critical fix, especially for the arm64 EC2 instances. Submitted by: Ido Segev <idose@amazon.com> Obtained from: Amazon, Inc MFC after: 1 week Differential revision: https://reviews.freebsd.org/D27114 Notes: svn path=/head/; revision=367795
* teach ena driver about RSS kernel optionAndriy Gapon2020-06-232-0/+29
| | | | | | | | | | | | | | | | | | | | Networking is broken if the driver configures its (virtual) hardware to use a hash algorithm (or a key) different from the one that the network stack (software RSS) uses. This can be seen with connections initiated from the host. The PCB will be placed into the hash table based on the hash value calculated by the software. The hardware-calculated hash value in reponse packets will be different, so the PCB won't be found. Tested with a kernel compiled with 'options RSS' on an instance with ena driver. Reviewed by: mw, adrian MFC after: 2 weeks Sponsored by: Panzura Differential Revision: https://reviews.freebsd.org/D24733 Notes: svn path=/head/; revision=362530
* Update ENA driver version to v2.2.0Marcin Wojtas2020-05-268-10/+10
| | | | | | | | | | | | | | | | | | | | | | | Driver version upgrade is connected with support for the new device fetures, like Tx drops reporting or disabling meta caching. Moreover, the driver configuration from the sysctl was reworked to provide safer and better flow for configuring: * number of IO queues (new feature), * drbr size on Tx, * Rx queue size. Moreover, a lot of minor bug fixes and improvements were added. Copyright date in the license of the modified files in this release was updated to 2020. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361530
* Refactor ena_tx_map_mbuf() functionMarcin Wojtas2020-05-261-61/+28
| | | | | | | | | | | | | There is no guarantee from bus_dmamap_load_mbuf_sg() for matching mbuf chain segments to dma physical segments. This patch ensure correctly mapping to LLQ header and DMA segments. Submitted by: Ido Segev <idose@amazon.com> Obtained from: Amazon, Inc. Notes: svn path=/head/; revision=361529
* Fix double-free bug within ena_detach()Marcin Wojtas2020-05-261-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | There is ena_free_all_io_rings_resources() called twice on device detach: ena_detach(): ena_destroy_device(): /* First call */ ena_free_all_io_rings_resources() /* Second call */ ena_free_all_io_rings_resources() The double-free causes panic() on kldunload, for example. As the ena_destroy_device() is also called by ena_reset_task() it is better to stay unchanged. Thus, remove the "Second call" of the function. Submitted by: Maciej Bielski <mba@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361528
* Allow disabling meta caching for ENA Tx pathMarcin Wojtas2020-05-263-4/+16
| | | | | | | | | | | | Determined by a flag passed from the device. No metadata is set within ena_tx_csum when caching is disabled. Submitted by: Maciej Bielski <mba@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361527
* Create ENA IO queues with optional backoffMarcin Wojtas2020-05-264-51/+129
| | | | | | | | | | | | If requested size of IO queues is not supported try to decrease it until finding the highest value that can be satisfied. Submitted by: Maciej Bielski <mba@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361526
* Add sysctl node for ENA IO queues number adjustmentMarcin Wojtas2020-05-263-1/+130
| | | | | | | | | | | | | | | | | | | | | | | | | | By default, in ena_attach() the driver attempts to acquire ena_adapter::max_num_io_queues MSI-X vectors for the purpose of IO queues, however this is not guaranteed. The number of vectors acquired depends also on system resources availability. Regardless of that, enable the number of effectively used IO queues to be further limited through the sysctl node. Example: Assumming that there are 8 IO queues configured by default, the command $ sysctl dev.ena.0.io_queues_nb=4 will reduce the number of available IO queues to 4. Similarly, the value can be also increased up to maximum supported value. A value higher than maximum supported number of IO queues is ignored. Zero is ignored too. Submitted by: Maciej Bielski <mba@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361525
* Fix assumptions about number of IO queues in the ENAMarcin Wojtas2020-05-261-5/+35
| | | | | | | | | | | | | | | | | | | | | | | | Make the ena_adapter::num_io_queues a number of effectively used IO queues. While the ena_adapter::max_num_io_queues is an upper-bound specified by the HW, the ena_adapter::num_io_queues may be lower than that, depending on runtime system resources availability. On reset, there are called ena_destroy_device() and then ena_restore_device(). The latter calls, in turn, ena_enable_msix(), which will attempt to re-acquire ena_adapter::max_num_io_queues of MSIX vectors again. Thus, the value of ena_adapter::num_io_queues may be different before and after reset. For this reason, free the IO rings structures (drbr, counters) in ena_destroy_device() and allocate again in ena_restore_device(). Submitted by: Maciej Bielski <mba@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361524
* Rework ENA Tx buffer ring size reconfigurationMarcin Wojtas2020-05-263-14/+73
| | | | | | | | | | | | | | | | This method has been aligned with the way how the Rx queue size is being updated - so it's now done synchronously instead of resetting the device. Moreover, the input parameter is now being validated if it's a power of 2. Without this, it can cause kernel panic. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361523
* Rework ENA Rx queue size configurationMarcin Wojtas2020-05-265-216/+247
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch reworks how the Rx queue size is being reconfigured and how the information from the device is being processed. Reconfiguration of the queues and reset of the device in order to make the changes alive isn't the best approach. It can be done synchronously and it will let to pass information if the reconfiguration was successful to the user. It now is done in the ena_update_queue_size() function. To avoid reallocation of the ring buffer, statistic counters and the reinitialization of the mutexes when only new size has to be assigned, the io queues initialization function has been split into 2 stages: basic, which is just copying appropriate fields and the advanced, which allocates and inits more advanced structures for the IO rings. Moreover, now the max allowed Rx and Tx ring size is being kept statically in the adapter and the size of the variables holding those values has been changed to uint32_t everywhere. Information about IO queues size is now being logged in the up routine instead of the attach. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361521
* Mark the ENA driver as epoch readyMarcin Wojtas2020-05-261-1/+2
| | | | | | | | | | | | | | | | Recent changes to the epoch requires driver to notify that they knows epoch in order to prevent input packet function to enter epoch each time the packet is received. ENA is using NET_TASK for handling Rx, so it's entering epoch automatically whenever this task is being executed. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361519
* Improve indentation in ena_up() and ena_down()Marcin Wojtas2020-05-261-73/+75
| | | | | | | | | | | | If the conditional check for ENA_FLAG_DEV_UP is negated, the body of the function can have smaller indentation and it makes the code cleaner. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361518
* Expose argument names for non static ENA driver functionsMarcin Wojtas2020-05-264-20/+20
| | | | | | | | | | | | | | As functions which are declared in the header files are intended to be the interface and are going to be used by other files, it's better to include argument names in the definition, so the caller won't have to check the .c file in order to check their meaning and order. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361517
* Use single global lock in the ENA driverMarcin Wojtas2020-05-263-28/+38
| | | | | | | | | | | | | | | | | | | | | | | | Currently, the driver had 2 global locks - one was sx lock used for up/down synchronization and the second one was mutex, which was used for link configuration and timer service callout. It is better to have single lock for that. We cannot use mutex, as it can sleep and cause witness errors in up/down configuration, so sx lock seems to be the only choice. Callout cannot use sx lock, but the timer service is MP safe, so we just need to avoid race between ena_down() and ena_detach(). It can be avoided by acquiring sx lock. Simple macros were added that are encapsulating implementation of the lock and makes the code cleaner. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361516
* Add trigger reset function in the ENA driverMarcin Wojtas2020-05-264-37/+22
| | | | | | | | | | | | | | As the reset triggering is no longer a simple macro that was just setting appropriate flag, the new function for triggering reset was added. It improves code readability a lot, as we are avoiding additional indentation. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361515
* Provide ENA driver version in a sysctl nodeMarcin Wojtas2020-05-261-0/+3
| | | | | | | | | | | Usage example: $ sysctl hw.ena.driver_version Submitted by: Maciej Bielski <mba@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361514
* Remove unused argument from static function in ena.cMarcin Wojtas2020-05-261-7/+4
| | | | | | | | | | | | | The function ena_enable_msix_and_set_admin_interrupts takes two arguments while the second is not used and so can be spared. This is a static function, only ena.c is affected. Submitted by: Maciej Bielski <mba@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361513
* Enable Tx drops reporting in the ENA driverMarcin Wojtas2020-05-263-0/+9
| | | | | | | | | | | | | | Tx drops statistics are fetched from HW every ena_keepalive_wd() call and are observable using one of the commands: * sysctl dev.ena.0.hw_stats.tx_drops * netstat -I ena0 -d Submitted by: Maciej Bielski <mba@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361512
* Adjust ENA driver to the new HALMarcin Wojtas2020-05-265-27/+31
| | | | | | | | | | | | | | | | | | * Removed adaptive interrupt moderation (not suported on FreeBSD). * Use ena_com_free_q_entries instead of ena_com_free_desc. * Don't use ENA_MEM_FREE outside of the ena_com. * Don't use barriers before calling doorbells as it's already done in the HAL. * Add function that generates random RSS key, common for all driver's interfaces. * Change admin stats sysctls to U64. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=361511
* Optimize ENA Rx refill for low memory conditionsMarcin Wojtas2020-05-074-7/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Sometimes, especially when there is not much memory in the system left, allocating mbuf jumbo clusters (like 9KB or 16KB) can take a lot of time and it is not guaranteed that it'll succeed. In that situation, the fallback will work, but if the refill needs to take a place for a lot of descriptors at once, the time spent in m_getjcl looking for memory can cause system unresponsiveness due to high priority of the Rx task. This can also lead to driver reset, because Tx cleanup routine is being blocked and timer service could detect that Tx packets aren't cleaned up. The reset routine can further create another unresponsiveness - Rx rings are being refilled there, so m_getjcl will again burn the CPU. This was causing NVMe driver timeouts and resets, because network driver is having higher priority. Instead of 16KB jumbo clusters for the Rx buffers, 9KB clusters are enough - ENA MTU is being set to 9K anyway, so it's very unlikely that more space than 9KB will be needed. However, 9KB jumbo clusters can still cause issues, so by default the page size mbuf cluster will be used for the Rx descriptors. This can have a small (~2%) impact on the throughput of the device, so to restore original behavior, one must change sysctl "hw.ena.enable_9k_mbufs" to "1" in "/boot/loader.conf" file. As a part of this patch (important fix), the version of the driver was updated to v2.1.2. Submitted by: cperciva Reviewed by: Michal Krawczyk <mk@semihalf.com> Reviewed by: Ido Segev <idose@amazon.com> Reviewed by: Guy Tzalik <gtzalik@amazon.com> MFC after: 3 days PR: 225791, 234838, 235856, 236989, 243531 Differential Revision: https://reviews.freebsd.org/D24546 Notes: svn path=/head/; revision=360777
* Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)Pawel Biernacki2020-02-261-13/+14
| | | | | | | | | | | | | | | | | | | r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718 Notes: svn path=/head/; revision=358333
* Rework and simplify Tx DMA mapping in ENAMarcin Wojtas2020-02-243-148/+54
| | | | | | | | | | | | | | | | | | | | | | | | Driver working in LLQ mode in some cases can send only few last segments of the mbuf using DMA engine, and the rest of them are sent to the device using direct PCI transaction. To map the only necessary data, two DMA maps were used. That solution was very rough and was causing a bug - if both maps were used (head_map and seg_map), there was a race in between two flows on two queues and the device was receiving corrupted data which could be further received on the other host if the Tx cksum offload was enabled. As it's ok to map whole mbuf and then send to the device only needed segments, the design was simplified to use only single DMA map. The driver version was updated to v2.1.1 as it's important bug fix. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=358289
* Use NET_TASK_INIT() and NET_GROUPTASK_INIT() for drivers that processGleb Smirnoff2020-02-111-1/+1
| | | | | | | | | | incoming packets in taskqueue context. Reviewed by: hselasky Differential Revision: https://reviews.freebsd.org/D23518 Notes: svn path=/head/; revision=357772
* Make valdiate_rx_req_id static inline because it uses other staticWarner Losh2019-11-021-2/+1
| | | | | | | | | inline functions. gcc complains about this, most likely due to the subtle differences between inline and static inline functions defined in headers. Notes: svn path=/head/; revision=354242
* Update ENA version to v2.1.0Marcin Wojtas2019-10-311-1/+1
| | | | | | | | | | | | | | | In this release the netmap support was introduced. Moreover, it is also now possible to use the LLQ mode of the driver on the arm64 AWS instances (A1 type). Differential Revision: https://reviews.freebsd.org/D21938 Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=354225
* Add support for ENA NETMAP partial initializationMarcin Wojtas2019-10-313-6/+64
| | | | | | | | | | | | | | | | | In NETMAP mode not all queues need to be allocated to NETMAP. Some of them could be left to the kernel. Configuration is managed by the flags nr_mode and nr_pending_mode provided per each NETMAP kring. ENA driver checks those flags and perform proper rings initialization. Differential Revision: https://reviews.freebsd.org/D21937 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=354224
* Add support for ENA NETMAP TxMarcin Wojtas2019-10-315-15/+706
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=354223
* Add support for ENA NETMAP RxMarcin Wojtas2019-10-315-24/+395
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Most of code used for Rx ring initialization could be reused in NETMAP. Reset of NETMAP ring and new alloc method was added. Driver decides if use kernels mbufs or NETMAPs slots based on IFCAP_NETMAP flag. It allows to reuse ena_refill_rx_bufs, which provides proper handling of Rx out of order completion. ena_netmap_alloc_rx_slot takes exactly the same arguments as ena_alloc_rx_mbuf, but instead of allocating one mbuf it takes one slot from NETMAP ring. Based on queue id proper netmap_ring is found. As NETMAP provides the "partial opening" feature not all of the rings are avaiable. Not used points to invalid ring. If there is available slot, it is taken from the ring. Its buffer is mapped to DMA and its index is stored in ena_rx_buffer field in ena_rx_buffer structure. Then ena_buf is filled with addresses and ring state is updated. Cleanup is handled by ena_netmap_free_rx_slot. It unmaps DMA and returns buffer to ring. As we could not return more bufs than we have taken and we should not override occupied slots, buf_index should be 0. It is being checked by assertion. ena_netmap_rxsync callback puts received packets back to NETMAP ring and passes them to user space by updating ring pointers. First it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_rx_frames moves received frames from NIC to NETMAP ring, * ena_netmap_rx_cleanup fills NIC ring with slots released by userspace app. In case of Rx error that could be handled by NIC driver (for example by performing reset) rx sync should return 0. ena_netmap_rx_frames first checks if NETMAP ring is in consistent state and then in the loop receives new frames. When all available frames are taken nr_hwtail is updated. Receiving one frame is handled by ena_netmap_rx_frame. If no error occurrs, each Descriptor is loaded by ena_netmap_rx_load_desc function. If packets take more than one segments NS_MOREFRAG flag must be set in all, but not last slot. In case of wrong req_id packet is removed from NETMAP ring. If packet is successful received counters are updated. Refiling of NIC ring is performed by ena_netmap_rx_cleanup function. It calculates number of available slots and call ena_refill_rx_bufs with proper number. Differential Revision: https://reviews.freebsd.org/D21935 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=354222
* Introduce NETMAP support in ENAMarcin Wojtas2019-10-313-0/+185
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Mock implementation of NETMAP routines is located in ena_netmap.c/.h files. All code is protected under the DEV_NETMAP macro. Makefile was updated with files and flag. As ENA driver provide own implementations of (un)likely it must be undefined before including NETMAP headers. ena_netmap_attach function is called on the end of NIC attach. It fills structure with NIC configuration and callbacks. Then provides it to netmap_attach. Similarly netmap_detach is called during ena_detach. Three callbacks are used. nm_register is implemented by ena_netmap_reg. It is called when user space application open or close NIC in NETMAP mode. Current action is recognized based on onoff parameter: true means on and false off. As NICs rings need to be reconfigured ena_down and ena_up are reused. When user space application wants to receive new packets from NIC nm_rxsync is called, and when there are new packets ready for Tx nm_txsync is called. Differential Revision: https://reviews.freebsd.org/D21934 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=354221
* Split Rx/Tx from initialization code in ENA driverMarcin Wojtas2019-10-314-1158/+1247
| | | | | | | | | | | | | | | | | | | | Move Rx/Tx routines to separate file. Some functions: * ena_restore_device, * ena_destroy_device, * ena_up, * ena_down, * ena_refill_rx_bufs could be reused in upcoming netmap code in the driver. To make it possible, they were moved to ena.h header. Differential Revision: https://reviews.freebsd.org/D21933 Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=354220
* Fix ENA keep-alive timeout due to prolonged resetMarcin Wojtas2019-10-311-4/+21
| | | | | | | | | | | | | | | | | | When the ENA_FLAG_DEVICE_RUNNING flag is disabled, the AENQ handlers aren't executed. To fix that, the watchdog timestamp should be updated just before enabling the watchdog. Timer service was always being enabled, even if the device wasn't up before the reset. That shouldn't happen, as the timer service is being executed only for working interface. Differential Revision: https://reviews.freebsd.org/D21932 Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=354219
* Add WC support for arm64 in the ENA driverMarcin Wojtas2019-10-311-10/+2
| | | | | | | | | | | | | | | | | As the pmamp_change_attr() is public on arm64 since r351131, it can be used on the arm64 to map memory range as with the write combined attribute. It requires the driver to use generic VM_MEMATTR_WRITE_COMBINING flag instead of the x86 specific PAT_WRITE_COMBINING. Differential Revision: https://reviews.freebsd.org/D21931 Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=354218
* Update ENA version to v2.0.0Marcin Wojtas2019-05-304-7/+7
| | | | | | | | | | | | | | | | ENAv2 introduces many new features, bug fixes and improvements. Main new features are LLQ (Low Latency Queues) and independent queues reconfiguration using sysctl commands. The year in copyright notice was updated to 2019. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348416
* Improve ENA reset handlingMarcin Wojtas2019-05-301-15/+36
| | | | | | | | | | | | | | | | For easier debugging, the reset is being triggered and the reset reason is being set only in case it is done for the first time. Such approach will ensure that the first reset reason is not going to be overwritten and will make it easier for debugging. Also, add a reset trigger upon invalid Tx requested ID. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348413
* Fix NULL pointer dereference in ena_up()Marcin Wojtas2019-05-301-5/+14
| | | | | | | | | | | | | | If the call to ena_up() in ena_restore_device() fails, next usage of `ifconfig up` will cause NULL pointer dereference. This patch adds additional checks to prevent that. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348412
* Unify new line characters in the ENA driverMarcin Wojtas2019-05-301-28/+28
| | | | | | | | | | | | | | | Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348411
* Fix Tx offloads for fragmented pkt headers in ENAMarcin Wojtas2019-05-301-2/+7
| | | | | | | | | | | | | | | | | If the headers of the packets are split into multiple segments of the mbuf chain, the previous version of ena_tx_csum which was assuming, that all segments will lay in the first mbuf, will eventually fail to map the headers properties to meta descriptor. That will cause Tx checksum offload to do not work and was leading to memory corruption. It could even cause the crash of the system. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348410
* Split ENA reset routine into restore and destroy stagesMarcin Wojtas2019-05-302-60/+128
| | | | | | | | | | | | | | | | | | | | | For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348409
* Use bitfield for storing global ENA device statesMarcin Wojtas2019-05-303-53/+83
| | | | | | | | | | | | | | | | | | | As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348408
* Fix error handling when ENA reset failsMarcin Wojtas2019-05-301-0/+4
| | | | | | | | | | | | | | Before the patch, error handling was not releasing all resources and was not issuing device reset if the reset task failed. That could cause memory leak and fault of the device. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348407
* Fill bdf field of the host_info structure in ENAMarcin Wojtas2019-05-301-3/+6
| | | | | | | | | | | | | | | The host info bdf field is the abbreviation for the bus, device, function of the PCI on which the device is being attached to. Now the driver is filling information about that using FreeBSD RID resource. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348406
* Add additional doorbells on ENA Tx pathMarcin Wojtas2019-05-302-12/+19
| | | | | | | | | | | | | | | | The new ENA HAL is introducing API, which can determine on Tx path if the doorbell is needed. That way, it can tell the driver, that it should call an doorbell. The old threshold value wasn't removed, as not all HW is supporting this feature - so it was reworked to also work with the new API. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348405
* Limit maximum size of Rx refill threshold in ENAMarcin Wojtas2019-05-302-1/+8
| | | | | | | | | | | | The Rx ring size can be as high as 8k. Because of that we want to limit the cleanup threshold by maximum value of 256. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Notes: svn path=/head/; revision=348404
generated by cgit v1.3 (git 2.53.0) at 2026-05-23 14:30:05 +0000