aboutsummaryrefslogtreecommitdiff
path: root/sys/dev/mlx5/mlx5_en
Commit message (Collapse)AuthorAgeFilesLines
* Remove erradic assert after SVN r367149 in mlx5en(4).Hans Petter Selasky2020-11-241-1/+0
| | | | | | | | | | | | The ratelimit tags may be shared, especially for unlimited TLS traffic, and then the refcount is allowed to be greater than one when freeing the send tag. MFC after: 1 week Sponsored by: Mellanox Technologies // NVIDIA Networking Notes: svn path=/head/; revision=367981
* Report EQE data upon CQ completion in mlx5core.Hans Petter Selasky2020-11-165-11/+11
| | | | | | | | | | | | | Report EQE data upon CQ completion to let upper layers use this data. Linux commit: 4e0e2ea1886afe8c001971ff767f6670312a9b04 MFC after: 1 week Sponsored by: Mellanox Technologies // NVIDIA Networking Notes: svn path=/head/; revision=367718
* Enhance the mlx5_core_create_cq() function in mlx5core.Hans Petter Selasky2020-11-161-1/+2
| | | | | | | | | | | | | | Enhance mlx5_core_create_cq() to get the command out buffer from the callers to let them use the output. Linux commit: 38164b771947be9baf06e78ffdfb650f8f3e908e MFC after: 1 week Sponsored by: Mellanox Technologies // NVIDIA Networking Notes: svn path=/head/; revision=367717
* mlx5en: Set ifmr_current same as ifmr_active.Konstantin Belousov2020-11-121-1/+1
| | | | | | | | | | | | | | | | This both: - makes ifconfig media line similar to that of other drivers. - fixes ENXIO in case when paradoxical current media word is not registered. Now e.g. ifconfig mce0 -mediaopt txpause,rxpause works by disabling pauses if enabled. Sponsored by: Mellanox Technologies/NVidia Networking MFC after: 1 week Notes: svn path=/head/; revision=367612
* mlx5en: stop ignoring pauses and flow in the media reqs.Konstantin Belousov2020-11-121-3/+2
| | | | | | | | Sponsored by: Mellanox Technologies/NVidia Networking MFC after: 1 week Notes: svn path=/head/; revision=367611
* mlx5en: Register all combinations of FDX/RXPAUSE/TXPAUSE as valid media types.Konstantin Belousov2020-11-121-0/+9
| | | | | | | | Sponsored by: Mellanox Technologies/NVidia Networking MFC after: 1 week Notes: svn path=/head/; revision=367610
* mlx5en: Refactor repeated code to register media type to mlx5e_ifm_add().Konstantin Belousov2020-11-121-12/+11
| | | | | | | | Sponsored by: Mellanox Technologies/NVidia Networking MFC after: 1 week Notes: svn path=/head/; revision=367609
* Remove IF_SND_TAG_TYPE_TLS_RATE_LIMIT conditionals.John Baldwin2020-10-303-10/+10
| | | | | | | | | | | | Support for TLS rate limit tags is now in the tree, so this macro is always defined. Reviewed by: hselasky Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D27020 Notes: svn path=/head/; revision=367187
* Fix a couple of silly bugs in r367149.John Baldwin2020-10-301-3/+1
| | | | | | | | | | | | - Assign the TLS rate limit value to the correct member of the rl_params for the nested rate limit tag. - Remove a dead condition. Pointy hat to: jhb Notes: svn path=/head/; revision=367153
* Add m_snd_tag_alloc() as a wrapper around if_snd_tag_alloc().John Baldwin2020-10-291-1/+1
| | | | | | | | | | | This gives a more uniform API for send tag life cycle management. Reviewed by: gallatin, hselasky Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D27000 Notes: svn path=/head/; revision=367151
* Use public interfaces to manage the nested rate limit send tag.John Baldwin2020-10-291-34/+15
| | | | | | | | | | | | | | | | Each TLS send tag in mlx5 contains a nested rate limit send tag. Previously, the driver was calling internal functions to manage the nested tag. Calling free methods directly instead of m_snd_tag_rele() leaked send tag references and references on the ifp. Changes to use the ifp methods for the nested tag for other methods are more cosmetic but do simplify the code. Reviewed by: gallatin, hselasky Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26996 Notes: svn path=/head/; revision=367149
* Support hardware rate limiting (pacing) with TLS offload.John Baldwin2020-10-291-1/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Add a new send tag type for a send tag that supports both rate limiting (packet pacing) and TLS offload (mostly similar to D22669 but adds a separate structure when allocating the new tag type). - When allocating a send tag for TLS offload, check to see if the connection already has a pacing rate. If so, allocate a tag that supports both rate limiting and TLS offload rather than a plain TLS offload tag. - When setting an initial rate on an existing ifnet KTLS connection, set the rate in the TCP control block inp and then reset the TLS send tag (via ktls_output_eagain) to reallocate a TLS + ratelimit send tag. This allocates the TLS send tag asynchronously from a task queue, so the TLS rate limit tag alloc is always sleepable. - When modifying a rate on a connection using KTLS, look for a TLS send tag. If the send tag is only a plain TLS send tag, assume we failed to allocate a TLS ratelimit tag (either during the TCP_TXTLS_ENABLE socket option, or during the send tag reset triggered by ktls_output_eagain) and ignore the new rate. If the send tag is a ratelimit TLS send tag, change the rate on the TLS tag and leave the inp tag alone. - Lock the inp lock when setting sb_tls_info for a socket send buffer so that the routines in tcp_ratelimit can safely dereference the pointer without needing to grab the socket buffer lock. - Add an IFCAP_TXTLS_RTLMT capability flag and associated administrative controls in ifconfig(8). TLS rate limit tags are only allocated if this capability is enabled. Note that TLS offload (whether unlimited or rate limited) always requires IFCAP_TXTLS[46]. Reviewed by: gallatin, hselasky Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26691 Notes: svn path=/head/; revision=367123
* Store the send tag type in the common send tag header.John Baldwin2020-10-067-55/+38
| | | | | | | | | | | | | | | | | | | | Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with their own identical headers that stored the type that the type-specific tag structures inherited from, so in practice it seems drivers need this in the tag anyway. This permits removing these extra header indirections (struct cxgbe_snd_tag and struct mlx5e_snd_tag). In addition, this permits driver-independent code to query the type of a tag, e.g. to know what type of tag is being queried via if_snd_query. Reviewed by: gallatin, hselasky, np, kib Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D26689 Notes: svn path=/head/; revision=366491
* Poll statistics more frequently in mlx5en(4).Hans Petter Selasky2020-09-141-1/+1
| | | | | | | | | | | This makes traffic steering algorithms more accurate. MFC after: 1 week Submitted by: gallatin @ Sponsored by: Mellanox Technologies // NVIDIA Networking Notes: svn path=/head/; revision=365719
* Make hardware TLS send tag allocation synchronous in mlx5en(4).Hans Petter Selasky2020-09-011-10/+9
| | | | | | | | | | | | | | | Previously the send tag was setup in the background, and all packets for the given send tag were dropped until ready. Change this to be blocking behaviour so that once the setsocketopt() for enabling TLS completes, the socket is ready to send packets. Do this by simply flushing the work request which does the needed firmware programming during send tag allocation. MFC after: 1 week Sponsored by: Mellanox Technologies // Nvidia Notes: svn path=/head/; revision=365036
* mlx5en: Implement SIOCGIFDOWNREASON.Konstantin Belousov2020-08-311-0/+11
| | | | | | | | Sponsored by: Mellanox Technologies - Nvidia MFC after: 1 week Notes: svn path=/head/; revision=365001
* Infiniband clients must be attached and detached in a specific order in ibcore.Hans Petter Selasky2020-07-061-2/+2
| | | | | | | | | | | | | | | | | | | Currently the linking order of the infiniband, IB, modules decide in which order the clients are attached and detached. For example one IB client may use resources from another IB client. This can lead to a potential deadlock at shutdown. For example if the ipoib is unregistered after the ib_multicast client is detached, then if ipoib is using multicast addresses a deadlock may happen, because ib_multicast will wait for all its resources to be freed before returning from the remove method. Fix this by using module_xxx_order() instead of module_xxx(). Differential Revision: https://reviews.freebsd.org/D23973 MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=362953
* Fix HW TLS offload regression issue after r359919, in mlx5en(4).Hans Petter Selasky2020-06-171-4/+12
| | | | | | | | | | | | Changes in the mbuf layout regarding HW TLS, resulted in wrong detection of starting mbuf. Use a boolean variable to handle this and pass m_adj() the top mbuf, so that the packet header is adjusted correctly. MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=362272
* Avoid trying to toggle TSO twiceRyan Moeller2020-06-151-0/+2
| | | | | | | | | | | | | | Remove TSO from the toggle mask when automatically disabled by TXCKSUM* in various NIC drivers. Reviewed by: hselasky, np, gallatin, jpaetzel Approved by: mav (mentor) MFC after: 1 week Sponsored by: iXsystems, Inc. Differential Revision: https://reviews.freebsd.org/D25120 Notes: svn path=/head/; revision=362201
* Make sure packets generated by raw IP code is let through by mlx5en(4).Hans Petter Selasky2020-06-111-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | Allow the TCP header to reside in the mbuf following the IP header. Else such packets will get dropped. Backtrace: mlx5e_sq_xmit() mlx5e_xmit() ether_output_frame() ether_output() ip_output_send() ip_output() rip_output() sosend_generic() sosend() kern_sendit() sendit() sys_sendto() amd64_syscall() fast_syscall_common() MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=362045
* Extend use of unlikely() in the fast path, in mlx5en(4).Hans Petter Selasky2020-06-111-6/+6
| | | | | | | | | | | | | Typically the TCP/IP headers fit within the first mbuf and should not trigger any of the error cases. Use unlikely() for these cases. No functional change. MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=362044
* Use const keyword when parsing the TCP/IP header in the fast path in mlx5en(4).Hans Petter Selasky2020-06-113-11/+11
| | | | | | | | | | | | | | When parsing the TCP/IP header in the fast path, make it clear by using the const keyword, no fields are to be modified inside the transmitted packet. No functional change. MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=362043
* Improve set progress parameters, SET PSV for HW TLS in mlx5en(4).Hans Petter Selasky2020-05-251-7/+1
| | | | | | | | | | There is no need for a fence and there is no need to provide the TCP sequence number. Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=361447
* Correctly set the initial vector for TLS v1.3 for mlx5en(4).Hans Petter Selasky2020-05-251-11/+7
| | | | | | | | | | | | | For TLS v1.3 the 12 bytes of the initial vector, IV, should just be copied as-is from the kernel to the gcm_iv field, which hold the first 4 bytes, and the remaining 8 bytes go to the subsequent implicit_iv field. There is no need to consider the byte order on the 12 bytes of IV like initially done. Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=361446
* Update the TLS capability bit after recent PRM changes in mlx5en(4).Hans Petter Selasky2020-05-251-2/+2
| | | | | | | | | | A CX6-DX firmware version equal to or newer than 12.27.0372 is now required. Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=361445
* Step 4.1: mechanically rename M_NOMAP to M_EXTPGGleb Smirnoff2020-05-031-2/+2
| | | | | | | | Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598 Notes: svn path=/head/; revision=360581
* Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbufGleb Smirnoff2020-05-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598 Notes: svn path=/head/; revision=360579
* Step 2.4: Stop using 'struct mbuf_ext_pgs' in drivers.Gleb Smirnoff2020-05-021-3/+1
| | | | | | | | Reviewed by: gallatin, hselasky Differential Revision: https://reviews.freebsd.org/D24598 Notes: svn path=/head/; revision=360576
* Add support for reading temperature in mlx5en(4).Hans Petter Selasky2020-04-273-0/+79
| | | | | | | | MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=360377
* KTLS: Re-work unmapped mbufs to carry ext_pgs in the mbuf itself.Andrew Gallatin2020-04-141-11/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While the original implementation of unmapped mbufs was a large step forward in terms of reducing cache misses by enabling mbufs to carry more than a single page for sendfile, they are rather cache unfriendly when accessing the ext_pgs metadata and data. This is because the ext_pgs part of the mbuf is allocated separately, and almost guaranteed to be cold in cache. This change takes advantage of the fact that unmapped mbufs are never used at the same time as pkthdr mbufs. Given this fact, we can overlap the ext_pgs metadata with the mbuf pkthdr, and carry the ext_pgs meta directly in the mbuf itself. Similarly, we can carry the ext_pgs data (TLS hdr/trailer/array of pages) directly after the existing m_ext. In order to be able to carry 5 pages (which is the minimum required for a 16K TLS record which is not perfectly aligned) on LP64, I've had to steal ext_arg2. The only user of this in the xmit path is sendfile, and I've adjusted it to use arg1 when using unmapped mbufs. This change is almost entirely mechanical, except that we change mb_alloc_ext_pgs() to no longer allow allocating pkthdrs, the change to avoid ext_arg2 as mentioned above, and the removal of the ext_pgs zone, This change saves roughly 2% "raw" CPU (~59% -> 57%), or over 3% "scaled" CPU on a Netflix 100% software kTLS workload at 90+ Gb/s on Broadwell Xeons. In a follow-on commit, I plan to remove some hacks to avoid access ext_pgs fields of mbufs, since they will now be in cache. Many thanks to glebius for helping to make this better in the Netflix tree. Reviewed by: hselasky, jhb, rrs, glebius (early version) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24213 Notes: svn path=/head/; revision=359919
* Account out of buffer as dropped packets in mlx5en(4).Hans Petter Selasky2020-04-081-0/+1
| | | | | | | | MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=359724
* Remove obsolete bufring stats in mlx5en(4).Hans Petter Selasky2020-04-082-5/+0
| | | | | | | | | | Leftover from when DRBR was removed. MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=359723
* Don't drop packets having too many TCP option headers in mlx5en(4).Hans Petter Selasky2020-04-061-7/+15
| | | | | | | | | | | | When using SACK it can happen there are multiple option headers. Don't drop these packets, but instead limit the amount of inlining to the maximum supported. MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=359655
* Ensure a minimum inline size of 16 bytes in mlx5en(4).Hans Petter Selasky2020-04-061-8/+12
| | | | | | | | | | | | | | | This includes 14 bytes of ethernet header and 2 bytes of VLAN header. This allows for making assumptions about the inline size limit in the fast transmit path later on. Use a signed integer variable to catch underflow. MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=359654
* Count number of times transmit ring is out of buffers in mlx5en(4).Hans Petter Selasky2020-04-062-2/+8
| | | | | | | | | Differential Revision: https://reviews.freebsd.org/D24273 MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=359653
* mlx5: Integrate eswitch and mpfs management code.Konstantin Belousov2020-03-181-1/+2
| | | | | | | | | Reviewed by: hselasky Sponsored by: Mellanox Technologies MFC after: 2 weeks Notes: svn path=/head/; revision=359101
* mlx5en: Support 50GBase-KR4 media type in mlx5en driver.Konstantin Belousov2020-03-041-0/+8
| | | | | | | | | | Submitted by: Adam Peace <adam.e.peace@gmail.com> Reviewed by: hselasky Sponsored by: Mellanox Technologies MFC after: 1 week Notes: svn path=/head/; revision=358624
* Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)Pawel Biernacki2020-02-264-15/+20
| | | | | | | | | | | | | | | | | | | r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718 Notes: svn path=/head/; revision=358333
* Although most of the NIC drivers are epoch ready, due to peer pressureGleb Smirnoff2020-02-241-1/+2
| | | | | | | | | | | | | | | | | | | switch over to opt-in instead of opt-out for epoch. Instead of IFF_NEEDSEPOCH, provide IFF_KNOWSEPOCH. If driver marks itself with IFF_KNOWSEPOCH, then ether_input() would not enter epoch when processing its packets. Now this will create recursive entrance in epoch in >90% network drivers, but will guarantee safeness of the transition. Mark several tested drivers as IFF_KNOWSEPOCH. Reviewed by: hselasky, jeff, bz, gallatin Differential Revision: https://reviews.freebsd.org/D23674 Notes: svn path=/head/; revision=358301
* Don't check the auth algorithm for GCM.John Baldwin2020-02-131-8/+0
| | | | | | | | | | | | | | | | The upstream OpenSSL changes only set the cipher for GCM since the authentication is redundant, and changes to OCF will soon remove the GCM authentication algorithm constants entirely for the same reason. In addition, ktls_create_session() already validates these fields and wouldn't pass down an invalid auth_algorithm value to any drivers or ktls backends. Reviewed by: hselasky Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D23671 Notes: svn path=/head/; revision=357891
* Add support for debugnet in mlx5en(4).Hans Petter Selasky2020-02-121-0/+72
| | | | | | | | MFC after: 1 week Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=357802
* Widen EPOCH(9) usage in mlx5en(4).Hans Petter Selasky2020-01-301-0/+9
| | | | | | | | | | | | | | | | | | Make completion event path mostly lockless using EPOCH(9). Implement a mechanism using EPOCH(9) which allows us to make the callback path for completion events mostly lockless. Simplify draining callback events using epoch_wait(). While at it make sure all receive completion callbacks are covered by the network EPOCH(9), because this is required when calling if_input() and ether_input() after r357012. Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=357294
* Implement hardware TLS via send tags for mlx5en(4), which is supported byHans Petter Selasky2019-12-068-87/+1366
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ConnectX-6 DX. Currently TLS v1.2 and v1.3 with AES 128/256 crypto over TCP/IP (v4 and v6) is supported. A per PCI device UMA zone is used to manage the memory of the send tags. To optimize performance some crypto contexts may be cached by the UMA zone, until the UMA zone finishes the memory of the given send tag. An asynchronous task is used manage setup of the send tags towards the firmware. Most importantly setting the AES 128/256 bit pre-shared keys for the crypto context. Updating the state of the AES crypto engine and encrypting data, is all done in the fast path. Each send tag tracks the TCP sequence number in order to detect non-contiguous blocks of data, which may require a dump of prior unencrypted data, to restore the crypto state prior to wire transmission. Statistics counters have been added to count the amount of TLS data transmitted in total, and the amount of TLS data which has been dumped prior to transmission. When non-contiguous TCP sequence numbers are detected, the software needs to dump the beginning of the current TLS record up until the point of retransmission. All TLS counters utilize the counter(9) API. In order to enable hardware TLS offload the following sysctls must be set: kern.ipc.mb_use_ext_pgs=1 kern.ipc.tls.ifnet.permitted=1 kern.ipc.tls.enable=1 Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=355446
* mlx5: Do not poke hardware for statistic after teardown is started.Konstantin Belousov2019-12-051-2/+3
| | | | | | | | Sponsored by: Mellanox Technologies MFC after: 1 week Notes: svn path=/head/; revision=355422
* Add basic support for TCP/IP based hardware TLS offload to mlx5core.Hans Petter Selasky2019-12-051-0/+11
| | | | | | | | | | | The hardware offload is primarily targeted for TLS v1.2 and v1.3, using AES 128/256 bit pre-shared keys. This patch adds all the needed hardware structures, capabilites and firmware commands. Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=355421
* Convert to if_foreach_llmaddr() KPI.Gleb Smirnoff2019-10-141-50/+35
| | | | | | | Reviewed by: hselasky Notes: svn path=/head/; revision=353505
* Fix regression issue after r352989:Hans Petter Selasky2019-10-081-2/+2
| | | | | | | | | | | | | | As noted by the commit message, callouts are now persistant and should not be in the auto-zero section of the RQ's and SQ's. This fixes an assert when using the TX completion event factor feature with mlx5en(4). Found by: gallatin@ MFC after: 3 days Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=353321
* Fix build failure for gcc after r352983, due toHans Petter Selasky2019-10-021-1/+0
| | | | | | | | | | not using static variable declared by net/sff8472.h . MFC after: 3 days Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=353001
* Bump driver version for mlx5core, mlx5en(4) and mlx5ib(4).Hans Petter Selasky2019-10-021-2/+2
| | | | | | | | MFC after: 3 days Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=352998
* Add print to show user a reason for rejecting buffer size change in mlx5en(4).Hans Petter Selasky2019-10-021-10/+16
| | | | | | | | | Submitted by: kib@ MFC after: 3 days Sponsored by: Mellanox Technologies Notes: svn path=/head/; revision=352996
generated by cgit v1.3 (git 2.53.0) at 2026-05-14 13:00:43 +0000