1 files changed, 102 insertions, 7 deletions
diff --git a/share/man/man4/gve.4 b/share/man/man4/gve.4
index 54e59b86108b..c5627e929044 100644
--- a/share/man/man4/gve.4
+++ b/share/man/man4/gve.4
@@ -1,6 +1,6 @@
 .\" SPDX-License-Identifier: BSD-3-Clause
 .\"
-.\" Copyright (c) 2023 Google LLC
+.\" Copyright (c) 2023-2024 Google LLC
 .\"
 .\" Redistribution and use in source and binary forms, with or without modification,
 .\" are permitted provided that the following conditions are met:
@@ -26,7 +26,7 @@
 .\" ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 .\" SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-.Dd April 26, 2023
+.Dd May 20, 2025
 .Dt GVE 4
 .Os
 .Sh NAME
@@ -79,6 +79,19 @@ binds to a single PCI device ID presented by gVNIC:
 .It
 0x1AE0:0x0042
 .El
+.Sh EXAMPLES
+.Pp
+Change the TX queue count to 4 for the gve0 interface:
+.D1 sysctl dev.gve.0.num_tx_queues=4
+.Pp
+Change the RX queue count to 4 for the gve0 interface:
+.D1 sysctl dev.gve.0.num_rx_queues=4
+.Pp
+Change the TX ring size to 512 for the gve0 interface:
+.D1 sysctl dev.gve.0.tx_ring_size=512
+.Pp
+Change the RX ring size to 512 for the gve0 interface:
+.D1 sysctl dev.gve.0.rx_ring_size=512
 .Sh DIAGNOSTICS
 The following messages are recorded during driver initialization:
 .Bl -diag
@@ -127,7 +140,6 @@ Tx queue-specific allocation failures:
 .It "Failed to alloc desc ring for tx ring %d"
 .It "Failed to vmap fifo, qpl_id = %d"
 .El
-.El
 .Pp
 The following messages are recorded when the interface detach fails:
 .Bl -diag
@@ -168,6 +180,12 @@ These messages are seen if any admin queue command fails:
 .It "Unknown AQ command opcode %d"
 .El
 .Pp
+These messages appear if a TX timeout is detected:
+.Bl -diag
+.It "Found %d timed out packet(s) on txq%d, kicking it for completions"
+.It "Found %d timed out packet(s) on txq%d with its last kick %ld sec ago which is less than the cooldown period %d. Resetting device"
+.El
+.Pp
 These messages are recorded when the device is being reset due to an error:
 .Bl -diag
 .It "Scheduling reset task!"
@@ -193,22 +211,99 @@ These two messages correspond to the NIC alerting the driver to link state chang
 .Pp
 Apart from these messages, the driver exposes per-queue packet and error counters as sysctl nodes.
 Global (across queues) counters can be read using
-.Xr netstat 8 .
+.Xr netstat 1 .
+.Sh SYSCTL VARIABLES
+.Nm
+exposes the following
+.Xr sysctl 8
+variables:
+.Bl -tag -width indent
+.It Va hw.gve.driver_version
+The driver version.
+This is read-only.
+.It Va hw.gve.queue_format
+The queue format in use.
+This is read-only.
+.It Va hw.gve.disable_hw_lro
+Setting this boot-time tunable to 1 disables Large Receive Offload (LRO) in the NIC.
+The default value is 0, which means hardware LRO is enabled by default.
+The software LRO stack in the kernel is always used.
+This sysctl variable needs to be set before loading the driver, using
+.Xr loader.conf 5 .
+.It Va hw.gve.allow_4k_rx_buffers
+Setting this boot-time tunable to 1 enables support for 4K RX Buffers.
+The default value is 0, which means 2K RX Buffers will be used.
+4K RX Buffers are only supported on DQO_RDA and DQO_QPL queue formats.
+When enabled, 4K RX Buffers will be used either when HW LRO is enabled
+or mtu is greated than 2048.
+This sysctl variable needs to be set before loading the driver, using
+.Xr loader.conf 5 .
+.It Va dev.gve.X.num_rx_queues and dev.gve.X.num_tx_queues
+Run-time tunables that represent the number of currently used RX/TX queues.
+The default value is the max number of RX/TX queues the device can support.
+.Pp
+This call turns down the interface while setting up the new queues,
+which may potentially cause any new packets to be dropped.
+This call can fail if the system is not able to provide the driver with enough resources.
+In that situation, the driver will revert to the previous number of RX/TX queues.
+If this also fails, a device reset will be triggered.
+.Pp
+Note: sysctl nodes for queue stats remain available even if a queue is removed.
+.Pp
+.It Va dev.gve.X.rx_ring_size and dev.gve.X.tx_ring_size
+Run-time tunables that represent the current ring size for RX/TX queues.
+The default value is set to device defaults for ring size.
+.Pp
+This call turns down the interface while setting up the queues with the new ring size,
+which may potentially cause any new packets to be dropped.
+This call can fail if the system is not able to provide the driver with enough resources.
+In that situation, the driver will try to revert to the previous ring size for RX/TX queues.
+If this also fails, the device will be in an unhealthy state and will need to be reloaded.
+This value must be a power of 2 and within the defined range.
+.Pp
+.El
 .Sh LIMITATIONS
 .Nm
 does not support the transmission of VLAN-tagged packets.
 All VLAN-tagged traffic is dropped.
+.Sh QUEUE FORMATS
+.Nm
+features different datapath modes called queue formats:
+.Pp
+.Bl -bullet -compact
+.It
+GQI_QPL: "QPL" stands for "Queue Page List" and refers to the fact that
+hardware expects a fixed bounce buffer and cannot access arbitrary memory.
+GQI is the older descriptor format.
+The G in "GQI" refers to an older generation of hardware, and the "QI"
+stands for "Queue In-order" referring to the fact that the NIC sends
+Tx and Rx completions in the same order as the one in which the corresponding
+descriptors were posted by the driver.
+.It
+DQO_RDA: DQO is the descriptor format required to take full advantage of
+next generation VM shapes.
+"RDA" stands for "Raw DMA Addressing" and refers to the fact that hardware
+can work with DMA-ed packets and does not expect them to be copied into or
+out of a fixed bounce buffer.
+The D in "DQO" refers to a newer generation of hardware, and the "QO"
+stands for "Queue Out-of-order" referring to the fact that the NIC might
+send Tx and Rx completions in an order different from the one in which
+the corresponding descriptors were posted by the driver.
+.It
+DQO_QPL: The next generation descriptor format in the "QPL" mode.
+.El
 .Sh SUPPORT
 Please email gvnic-drivers@google.com with the specifics of the issue encountered.
-.El
 .Sh SEE ALSO
+.Xr netstat 1 ,
+.Xr loader.conf 5 ,
 .Xr ifconfig 8 ,
-.Xr netstat 8
+.Xr sysctl 8
 .Sh HISTORY
 The
 .Nm
 device driver first appeared in
-.Fx 14.0 .
+.Fx 13.3 .
 .Sh AUTHORS
 The
 .Nm