Tweaking your NIC for lower latency (and stuff)

whats your strategy?

Post Reply
User avatar
rip
Posts: 1010
Joined: Sat Oct 18, 2008 9:51 am

Tweaking your NIC for lower latency (and stuff)

Post by rip » Mon May 04, 2015 4:19 pm

This applies to wired GIGE NICs, and assumes you have a moderately fast 4 core or so PC. RSS and Interrupt moderation may actually decrease performance in a variety of scenarios, especially on mono or 2 core systems, or otherwise limited resources. HT cores don't count, a dual with HT is not 4 cores, sorry.

Also, leave IPv6 support on even if you don't use IPv6. Some of the features are dependent on the condition that it is enabled.

Your NIC will not necessarily have all these options. For instance, a lot of boards have REALTEK 8111 series chips and only support one RSS Queue, so it's not even an option on how many to use.

Adaptive Frame Spacing: Disable
This is a left over from half duplex 10/100 Ethernet. gigE is full duplex and shouldn't need this. It leaves space (time) between packets to receive any collision detection that should never happen on modern switched networks.

PME: Enable

Power Management event, allows the NIC to wake the PC, related the WAKE parameters below.

Energy Efficient Ethernet: Disable
No, we don't want you to snooze when we're trying to kill people.

Flow Control, Enabled TX and RX
Required, your NIC is probably faster than your internet.

Gige Master/Slave Mode: Auto
If you have to force this for some reason, you are not doing it right.

Interrupt Moderation: DISABLE
Every packet generates an interrupt, unless you have moderation on. This tells the CPU to stop what it's doing and pick up a packet to process.
Turning it on allows the NIC to bundle up a bunch of packets and generate an interrupt for the CPU to come pick up the mail, which is more
efficient. But this comes at the cost of latency. Turning it off reduces latency and increases CPU load.


Interrupt Moderation Rate: Low (doesn't matter if Disabled above)
Determines how many packets to bundle before generating an interrupt. If you run interrupt moderation, lower is better for latency (what most gamers would opt for.). Higher is better for efficiency.

IPv4 Checksum Offload: Enable
IPv6 Checksum Offload: Enable

Allows the NIC to do the work of calculating Checksums. No real upside to having the CPU do that job.

Jumbo Packet: Disable
Allows the NIC to stuff a bigger payload into a packet. This requires that all network gear end to end supports Jumbo packets (almost never will happen in the wild) and would help with large file transfers. gaming wouldn't benefit much if at all as it's mostly tons if small packets.
This is useful in an internal setup (like moving backups or other huge files sets around in a data center.)

Large Send Offload IPv4:Enable
Large Send Offload IPv6:Enable

Old school thought was, turn this off. It had poor performance when initially introduced but helped with large files. However it is needed for RSS and RSS Queues. Doesn't do anything for gaming, but is a dependency for other features.

Locally Administered Address (leave blank)
Shouldn't need this in normal use. Could be handy in specific instances bonding or failing over NICs.

Log Link State Event: Enable
Leave a note in the logs if the link goes away, and when it comes back, or changes speed.

Priority and Vlan: Enabled (Both)

ARP Offload: Enable
NS Offload: Enabled

Let the NIC handle ARP and Nameserver tasks instead of the CPU.

Receive Buffers: Enabled (Maximum, 2048 for most intel)
Buffer for received packets, More is generally better and shouldn't be an issue of modern systems to max them out.

Receive Side Scaling: Enabled
Without RSS, you PC uses CPU0 to process ALL the packets. With RSS, it spreads the loads dynamically across the CPUs.

Receive Side Scaling Queue(s): Enabled (or 1-4, depending on NIC, try 4)
Queues for the RSS, you may find that 2 may actually work better than 4 in some cases (They get serviced in order, if your traffic is low, you have lonely bits waiting to be picked up, so it could be better to have more bits in fewer queues getting picked up faster.)


Reduce Speed on Power Down: Enable
OK, you can snooze when we're effectively off. (Wake on lan settings generally only take place at 10MB/s anyway.)

Speed and Duplex: Auto
You shouldn't need to force your link. If you do, somethings wrong.

Idle Power Saver: Disabled
Pffft, KILL NOW!

TCP Checksum Offload IPv4: Enabled
TCP Checksum Offload IPv6: Enabled

Let the NIC do the checksums for TCP packets instead of the CPU.

Wait for link: Auto
Generally, leave this alone. Disabling it means the PC will initialize the link and try to use it immediately, instead of waiting for it to negotiate speed and duplex. That can work on fiber for instance, but Copper, per specs, has 3 seconds to finish that job. If you disable it and use something like iSCSI, you can get link errors because the resource is not available yet.


Wake on link: Disabled (Unless you really use it)
Wake on Magic Packet: Disabled (Unless you really use it)
Wake on Pattern Match: Disabled (Unless you really use it)


Unless you are in a corp environment or doing remote stuff that utilizes WAKE commands, you don't need this. But probably doesn't hurt leaving it on either, so meh.

Other thoughts:

Intel 825xx or newer series Preferred!
Server oriented chipsets preferred (Larger buffers, RSS queues, etc.)

Realtek
Smaller receive and TX buffers, less or single RSS queue.

Broadcom
Shouldn't be allowed to make NICs. Ugh, get off of this if you have it.

Special considerations when you are running 10G and 4 or more processor sockets,
shouldn't apply to mortals.

If you are running 3rd party firmware on your router (Tomato, WRT, etc., that has QOS, you are bottle-necked unless you tweak the kernel and don't use QOS.)

Your mileage will vary.
rip

Build a fire for a man, you keep him warm that night.
Set a man on fire, and he'll be warm the rest of his life.

User avatar
MarioBros
Posts: 3405
Joined: Fri Jul 06, 2012 12:40 am
Location: Tijuana, MX.

Re: Tweaking your NIC for lower latency (and stuff)

Post by MarioBros » Mon May 04, 2015 9:33 pm

That's a lot of words :P
"Text translated via Google Translate"

User avatar
rip
Posts: 1010
Joined: Sat Oct 18, 2008 9:51 am

Re: Tweaking your NIC for lower latency (and stuff)

Post by rip » Tue May 05, 2015 1:08 pm

MarioBros wrote:That's a lot of words :P
Ya, and they mean things.
rip

Build a fire for a man, you keep him warm that night.
Set a man on fire, and he'll be warm the rest of his life.

n1xon
Posts: 1
Joined: Tue Nov 24, 2015 3:45 pm

Re: Tweaking your NIC for lower latency (and stuff)

Post by n1xon » Tue Nov 24, 2015 3:48 pm

rip wrote: If you are running 3rd party firmware on your router (Tomato, WRT, etc., that has QOS, you are bottle-necked unless you tweak the kernel and don't use QOS.)
what do you mean by tweak the kernel? examples?

curious because I thought router QOS could be used to help improve performance or consistency when playing games.

User avatar
rip
Posts: 1010
Joined: Sat Oct 18, 2008 9:51 am

Re: Tweaking your NIC for lower latency (and stuff)

Post by rip » Wed Sep 07, 2016 10:16 pm

n1xon wrote:
rip wrote: If you are running 3rd party firmware on your router (Tomato, WRT, etc., that has QOS, you are bottle-necked unless you tweak the kernel and don't use QOS.)
what do you mean by tweak the kernel? examples?

curious because I thought router QOS could be used to help improve performance or consistency when playing games.
WOW, this is old, but I saw it looking for something else. :)

It varies by the distro you are using. But there are known issues pretty much across the board on WRT, Tomato Shibby, and other distros and mainly Broadcom chipsets.

Depending on what you are using, older routers start to hit a wall with the stock builds. The problem is a lot of these distros are on old linux versions and don't enable hardware nat by default. For instance, untweaked my old Asus RT-N16 would hit the wall around 150-180MB/s.

In this case, you'd be interested in bcm-nat/ctf/fastnat. These do different things on different hardware, but basically when usage is very high, it allows a hardware bypass of packet inspection for established connections.

i.e FastNAT says "We're moving packets faster than we can look at them with the CPU, it's an already established stream, just move it from point a to point b without any other processing."

This works OK for an established and (non-malicious) connection. Streaming a movie, no big deal, we know where it's coming from and we know where it's going, we don't have to look up every source and destination on these packets between these two....usually including firewall rules on consumer items. :/
rip

Build a fire for a man, you keep him warm that night.
Set a man on fire, and he'll be warm the rest of his life.

Holliday
Posts: 206
Joined: Mon Nov 03, 2014 12:41 pm

Re: Tweaking your NIC for lower latency (and stuff)

Post by Holliday » Sat Sep 10, 2016 2:32 am

"This applies to" <- This is the only part I understood rip :(

Post Reply

Return to “CS tips and helpful hints!”

Who is online

Users browsing this forum: No registered users and 126 guests