You are here
Dropped Packets on MariaDB/MySQL Linux Servers
Table of Contents
ifconfigipethtool --statisticsnstatnetstat --statisticsss- Reasons for dropped packets
- Features of Network card
- Features of Network driver
- Network driver ring buffer
- TCP socket received and send buffer
- Flow Control
- Check over time:
sar - Dropwatch
perftcpdump/proc/net/netstat- System Tap (
stap) - Literature
When we do MariaDB and MySQL Database server analysis we see from time to time systems with a lot of dropped packets on a network interface:
ifconfig
# export DEV=enp0s25
# ifconfig ${DEV}
eth0 Link encap:Ethernet HWaddr d0:50:99:70:f8:8e
inet addr:192.168.1.135 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4005096 errors:0 dropped:365956 overruns:0 frame:0
TX packets:3293230 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2897493579 (2.8 GB) TX bytes:461542567 (461.5 MB)
As we all learned ifconfig is old and thus bad so we should use the new command ip instead. See also our new hint: Replacement for ifconfig statistics.
ip
# ip --statistics [--human-readable] address show ${DEV}
2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether d0:50:99:70:f8:8e brd ff:ff:ff:ff:ff:ff
inet 192.168.1.135/24 brd 192.168.1.255 scope global eth0
valid_lft forever preferred_lft forever
RX: bytes packets errors dropped overrun mcast
2897493579 4005096 0 365956 0 19703
TX: bytes packets errors dropped carrier collsns
461542567 3293230 0 0 0 0
ethtool --statistics
If you want to see some more statistics about your packets you can run:
# ethtool --statistics ${DEV}
NIC statistics:
rx_packets: 4134
tx_packets: 3507
rx_bytes: 2791271
tx_bytes: 607408
rx_broadcast: 59
tx_broadcast: 82
rx_multicast: 89
tx_multicast: 85
rx_errors: 0
tx_errors: 0
tx_dropped: 0
multicast: 89
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 53
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_csum_offload_good: 3983
rx_csum_offload_errors: 0
rx_header_split: 0
alloc_rx_buff_failed: 0
tx_smbus: 0
rx_smbus: 53
dropped_smbus: 0
rx_dma_failed: 0
tx_dma_failed: 0
rx_hwtstamp_cleared: 0
uncorr_ecc_errors: 0
corr_ecc_errors: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0
But unfortunately RX dropped is not listed but only TX dropped.
nstat
Detailed network statistics should be checked with nstat today:
# nstat --ignore --noupdate --zero #kernel SctpCurrEstab 0 0.0 SctpActiveEstabs 0 0.0 SctpPassiveEstabs 0 0.0 SctpAborteds 0 0.0 SctpShutdowns 0 0.0 SctpOutOfBlues 0 0.0 SctpChecksumErrors 0 0.0 SctpOutCtrlChunks 0 0.0 SctpOutOrderChunks 0 0.0 SctpOutUnorderChunks 0 0.0 SctpInCtrlChunks 0 0.0 SctpInOrderChunks 0 0.0 SctpInUnorderChunks 0 0.0 SctpFragUsrMsgs 0 0.0 SctpReasmUsrMsgs 0 0.0 SctpOutSCTPPacks 0 0.0 SctpInSCTPPacks 0 0.0 SctpT1InitExpireds 0 0.0 SctpT1CookieExpireds 0 0.0 SctpT2ShutdownExpireds 0 0.0 SctpT3RtxExpireds 0 0.0 SctpT4RtoExpireds 0 0.0 SctpT5ShutdownGuardExpireds 0 0.0 SctpDelaySackExpireds 0 0.0 SctpAutocloseExpireds 0 0.0 SctpT3Retransmits 0 0.0 SctpPmtudRetransmits 0 0.0 SctpFastRetransmits 0 0.0 SctpInPktSoftirq 0 0.0 SctpInPktBacklog 0 0.0 SctpInPktDiscards 0 0.0 SctpInDataChunkDiscards 0 0.0 IpInReceives 2146596 0.0 IpInHdrErrors 0 0.0 IpInAddrErrors 2 0.0 IpForwDatagrams 0 0.0 IpInUnknownProtos 0 0.0 IpInDiscards 0 0.0 IpInDelivers 2146589 0.0 IpOutRequests 1821062 0.0 IpOutDiscards 20 0.0 IpOutNoRoutes 3 0.0 IpReasmTimeout 0 0.0 IpReasmReqds 6 0.0 IpReasmOKs 3 0.0 IpReasmFails 0 0.0 IpFragOKs 0 0.0 IpFragFails 0 0.0 IpFragCreates 0 0.0 IcmpInMsgs 7434 0.0 IcmpInErrors 14 0.0 IcmpInCsumErrors 0 0.0 IcmpInDestUnreachs 3703 0.0 IcmpInTimeExcds 0 0.0 IcmpInParmProbs 0 0.0 IcmpInSrcQuenchs 0 0.0 IcmpInRedirects 0 0.0 IcmpInEchos 610 0.0 IcmpInEchoReps 3121 0.0 IcmpInTimestamps 0 0.0 IcmpInTimestampReps 0 0.0 IcmpInAddrMasks 0 0.0 IcmpInAddrMaskReps 0 0.0 IcmpOutMsgs 12373 0.0 IcmpOutErrors 0 0.0 IcmpOutDestUnreachs 3701 0.0 IcmpOutTimeExcds 0 0.0 IcmpOutParmProbs 0 0.0 IcmpOutSrcQuenchs 0 0.0 IcmpOutRedirects 0 0.0 IcmpOutEchos 8062 0.0 IcmpOutEchoReps 610 0.0 IcmpOutTimestamps 0 0.0 IcmpOutTimestampReps 0 0.0 IcmpOutAddrMasks 0 0.0 IcmpOutAddrMaskReps 0 0.0 IcmpMsgInType0 3121 0.0 IcmpMsgInType3 3703 0.0 IcmpMsgInType8 610 0.0 IcmpMsgOutType0 610 0.0 IcmpMsgOutType3 3701 0.0 IcmpMsgOutType8 8062 0.0 TcpActiveOpens 31370 0.0 TcpPassiveOpens 18080 0.0 TcpAttemptFails 2997 0.0 TcpEstabResets 56 0.0 TcpInSegs 2166708 0.0 TcpOutSegs 1842768 0.0 TcpRetransSegs 4093 0.0 TcpInErrs 5 0.0 TcpOutRsts 2905 0.0 TcpInCsumErrors 0 0.0 UdpInDatagrams 40549 0.0 UdpNoPorts 41 0.0 UdpInErrors 0 0.0 UdpOutDatagrams 38161 0.0 UdpRcvbufErrors 0 0.0 UdpSndbufErrors 0 0.0 UdpInCsumErrors 0 0.0 UdpIgnoredMulti 181 0.0 UdpLiteInDatagrams 0 0.0 UdpLiteNoPorts 0 0.0 UdpLiteInErrors 0 0.0 UdpLiteOutDatagrams 0 0.0 UdpLiteRcvbufErrors 0 0.0 UdpLiteSndbufErrors 0 0.0 UdpLiteInCsumErrors 0 0.0 UdpLiteIgnoredMulti 0 0.0 Ip6InReceives 71921 0.0 Ip6InHdrErrors 0 0.0 Ip6InTooBigErrors 0 0.0 Ip6InNoRoutes 1 0.0 Ip6InAddrErrors 0 0.0 Ip6InUnknownProtos 0 0.0 Ip6InTruncatedPkts 0 0.0 Ip6InDiscards 0 0.0 Ip6InDelivers 71920 0.0 Ip6OutForwDatagrams 0 0.0 Ip6OutRequests 68903 0.0 Ip6OutDiscards 0 0.0 Ip6OutNoRoutes 47 0.0 Ip6ReasmTimeout 0 0.0 Ip6ReasmReqds 0 0.0 Ip6ReasmOKs 0 0.0 Ip6ReasmFails 0 0.0 Ip6FragOKs 0 0.0 Ip6FragFails 0 0.0 Ip6FragCreates 0 0.0 Ip6InMcastPkts 2060 0.0 Ip6OutMcastPkts 94 0.0 Ip6InOctets 80749746 0.0 Ip6OutOctets 9844802 0.0 Ip6InMcastOctets 247993 0.0 Ip6OutMcastOctets 12725 0.0 Ip6InBcastOctets 0 0.0 Ip6OutBcastOctets 0 0.0 Ip6InNoECTPkts 103468 0.0 Ip6InECT1Pkts 1 0.0 Ip6InECT0Pkts 0 0.0 Ip6InCEPkts 1 0.0 Icmp6InMsgs 3054 0.0 Icmp6InErrors 0 0.0 Icmp6OutMsgs 1948 0.0 Icmp6OutErrors 0 0.0 Icmp6InCsumErrors 0 0.0 Icmp6InDestUnreachs 0 0.0 Icmp6InPktTooBigs 0 0.0 Icmp6InTimeExcds 0 0.0 Icmp6InParmProblems 0 0.0 Icmp6InEchos 776 0.0 Icmp6InEchoReplies 0 0.0 Icmp6InGroupMembQueries 294 0.0 Icmp6InGroupMembResponses 0 0.0 Icmp6InGroupMembReductions 0 0.0 Icmp6InRouterSolicits 0 0.0 Icmp6InRouterAdvertisements 836 0.0 Icmp6InNeighborSolicits 684 0.0 Icmp6InNeighborAdvertisements 462 0.0 Icmp6InRedirects 0 0.0 Icmp6InMLDv2Reports 2 0.0 Icmp6OutDestUnreachs 0 0.0 Icmp6OutPktTooBigs 0 0.0 Icmp6OutTimeExcds 0 0.0 Icmp6OutParmProblems 0 0.0 Icmp6OutEchos 0 0.0 Icmp6OutEchoReplies 776 0.0 Icmp6OutGroupMembQueries 0 0.0 Icmp6OutGroupMembResponses 0 0.0 Icmp6OutGroupMembReductions 0 0.0 Icmp6OutRouterSolicits 1 0.0 Icmp6OutRouterAdvertisements 0 0.0 Icmp6OutNeighborSolicits 464 0.0 Icmp6OutNeighborAdvertisements 684 0.0 Icmp6OutRedirects 0 0.0 Icmp6OutMLDv2Reports 23 0.0 Icmp6InType128 776 0.0 Icmp6InType130 294 0.0 Icmp6InType134 836 0.0 Icmp6InType135 684 0.0 Icmp6InType136 462 0.0 Icmp6InType143 2 0.0 Icmp6OutType129 776 0.0 Icmp6OutType133 1 0.0 Icmp6OutType135 464 0.0 Icmp6OutType136 684 0.0 Icmp6OutType143 23 0.0 Udp6InDatagrams 248 0.0 Udp6NoPorts 0 0.0 Udp6InErrors 0 0.0 Udp6OutDatagrams 153 0.0 Udp6RcvbufErrors 0 0.0 Udp6SndbufErrors 0 0.0 Udp6InCsumErrors 0 0.0 Udp6IgnoredMulti 0 0.0 UdpLite6InDatagrams 0 0.0 UdpLite6NoPorts 0 0.0 UdpLite6InErrors 0 0.0 UdpLite6OutDatagrams 0 0.0 UdpLite6RcvbufErrors 0 0.0 UdpLite6SndbufErrors 0 0.0 UdpLite6InCsumErrors 0 0.0 TcpExtSyncookiesSent 0 0.0 TcpExtSyncookiesRecv 0 0.0 TcpExtSyncookiesFailed 0 0.0 TcpExtEmbryonicRsts 0 0.0 TcpExtPruneCalled 0 0.0 TcpExtRcvPruned 0 0.0 TcpExtOfoPruned 0 0.0 TcpExtOutOfWindowIcmps 0 0.0 TcpExtLockDroppedIcmps 0 0.0 TcpExtArpFilter 0 0.0 TcpExtTW 20345 0.0 TcpExtTWRecycled 0 0.0 TcpExtTWKilled 0 0.0 TcpExtPAWSActive 0 0.0 TcpExtPAWSEstab 2 0.0 TcpExtDelayedACKs 15602 0.0 TcpExtDelayedACKLocked 9 0.0 TcpExtDelayedACKLost 114 0.0 TcpExtListenOverflows 0 0.0 TcpExtListenDrops 0 0.0 TcpExtTCPHPHits 1591674 0.0 TcpExtTCPPureAcks 96024 0.0 TcpExtTCPHPAcks 92591 0.0 TcpExtTCPRenoRecovery 0 0.0 TcpExtTCPSackRecovery 3 0.0 TcpExtTCPSACKReneging 0 0.0 TcpExtTCPSACKReorder 84 0.0 TcpExtTCPRenoReorder 0 0.0 TcpExtTCPTSReorder 0 0.0 TcpExtTCPFullUndo 0 0.0 TcpExtTCPPartialUndo 0 0.0 TcpExtTCPDSACKUndo 0 0.0 TcpExtTCPLossUndo 7 0.0 TcpExtTCPLostRetransmit 1 0.0 TcpExtTCPRenoFailures 0 0.0 TcpExtTCPSackFailures 0 0.0 TcpExtTCPLossFailures 0 0.0 TcpExtTCPFastRetrans 31 0.0 TcpExtTCPSlowStartRetrans 0 0.0 TcpExtTCPTimeouts 3774 0.0 TcpExtTCPLossProbes 69 0.0 TcpExtTCPLossProbeRecovery 3 0.0 TcpExtTCPRenoRecoveryFail 0 0.0 TcpExtTCPSackRecoveryFail 0 0.0 TcpExtTCPRcvCollapsed 0 0.0 TcpExtTCPDSACKOldSent 109 0.0 TcpExtTCPDSACKOfoSent 1 0.0 TcpExtTCPDSACKRecv 25 0.0 TcpExtTCPDSACKOfoRecv 0 0.0 TcpExtTCPAbortOnData 548 0.0 TcpExtTCPAbortOnClose 29 0.0 TcpExtTCPAbortOnMemory 0 0.0 TcpExtTCPAbortOnTimeout 27 0.0 TcpExtTCPAbortOnLinger 0 0.0 TcpExtTCPAbortFailed 0 0.0 TcpExtTCPMemoryPressures 0 0.0 TcpExtTCPMemoryPressuresChrono 0 0.0 TcpExtTCPSACKDiscard 0 0.0 TcpExtTCPDSACKIgnoredOld 0 0.0 TcpExtTCPDSACKIgnoredNoUndo 18 0.0 TcpExtTCPSpuriousRTOs 0 0.0 TcpExtTCPMD5NotFound 0 0.0 TcpExtTCPMD5Unexpected 0 0.0 TcpExtTCPMD5Failure 0 0.0 TcpExtTCPSackShifted 28 0.0 TcpExtTCPSackMerged 20 0.0 TcpExtTCPSackShiftFallback 95 0.0 TcpExtTCPBacklogDrop 0 0.0 TcpExtPFMemallocDrop 0 0.0 TcpExtTCPMinTTLDrop 0 0.0 TcpExtTCPDeferAcceptDrop 1 0.0 TcpExtIPReversePathFilter 0 0.0 TcpExtTCPTimeWaitOverflow 0 0.0 TcpExtTCPReqQFullDoCookies 0 0.0 TcpExtTCPReqQFullDrop 0 0.0 TcpExtTCPRetransFail 0 0.0 TcpExtTCPRcvCoalesce 200321 0.0 TcpExtTCPOFOQueue 61316 0.0 TcpExtTCPOFODrop 0 0.0 TcpExtTCPOFOMerge 2 0.0 TcpExtTCPChallengeACK 5 0.0 TcpExtTCPSYNChallenge 5 0.0 TcpExtTCPFastOpenActive 0 0.0 TcpExtTCPFastOpenActiveFail 0 0.0 TcpExtTCPFastOpenPassive 0 0.0 TcpExtTCPFastOpenPassiveFail 0 0.0 TcpExtTCPFastOpenListenOverflow 0 0.0 TcpExtTCPFastOpenCookieReqd 0 0.0 TcpExtTCPFastOpenBlackhole 0 0.0 TcpExtTCPSpuriousRtxHostQueues 12 0.0 TcpExtBusyPollRxPackets 0 0.0 TcpExtTCPAutoCorking 4012 0.0 TcpExtTCPFromZeroWindowAdv 0 0.0 TcpExtTCPToZeroWindowAdv 0 0.0 TcpExtTCPWantZeroWindowAdv 188 0.0 TcpExtTCPSynRetrans 3737 0.0 TcpExtTCPOrigDataSent 437079 0.0 TcpExtTCPHystartTrainDetect 35 0.0 TcpExtTCPHystartTrainCwnd 1153 0.0 TcpExtTCPHystartDelayDetect 0 0.0 TcpExtTCPHystartDelayCwnd 0 0.0 TcpExtTCPACKSkippedSynRecv 0 0.0 TcpExtTCPACKSkippedPAWS 0 0.0 TcpExtTCPACKSkippedSeq 10 0.0 TcpExtTCPACKSkippedFinWait2 0 0.0 TcpExtTCPACKSkippedTimeWait 0 0.0 TcpExtTCPACKSkippedChallenge 0 0.0 TcpExtTCPWinProbe 0 0.0 TcpExtTCPKeepAlive 5134 0.0 TcpExtTCPMTUPFail 0 0.0 TcpExtTCPMTUPSuccess 0 0.0 TcpExtTCPWqueueTooBig 0 0.0 IpExtInNoRoutes 0 0.0 IpExtInTruncatedPkts 0 0.0 IpExtInMcastPkts 2893 0.0 IpExtOutMcastPkts 466 0.0 IpExtInBcastPkts 190 0.0 IpExtOutBcastPkts 15 0.0 IpExtInOctets 4837352803 0.0 IpExtOutOctets 1199138713 0.0 IpExtInMcastOctets 2244078 0.0 IpExtOutMcastOctets 40977 0.0 IpExtInBcastOctets 45209 0.0 IpExtOutBcastOctets 4035 0.0 IpExtInCsumErrors 0 0.0 IpExtInNoECTPkts 3140592 0.0 IpExtInECT1Pkts 0 0.0 IpExtInECT0Pkts 0 0.0 IpExtInCEPkts 0 0.0 IpExtReasmOverlaps 0 0.0
netstat --statistics
Old command netstat but still widely used an known (use the previous nstat instead):
# netstat --statistics ${DEV}
TcpExt:
65 invalid SYN cookies received
155 resets received for embryonic SYN_RECV sockets
89 packets pruned from receive queue because of socket buffer overrun
ArpFilter: 0
14341 TCP sockets finished time wait in fast timer
10 time wait sockets recycled by time stamp
713178 delayed acks sent
316 delayed acks further delayed because of locked socket
Quick ack mode was activated 3908 times
29091413 packets directly queued to recvmsg prequeue.
19881778 packets directly received from backlog
1892094685 packets directly received from prequeue
31118010 packets header predicted
1272468 packets header predicted and directly queued to user
TCPPureAcks: 1118011
TCPHPAcks: 30188520
TCPRenoRecovery: 2524
TCPSackRecovery: 1
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 277
TCPTSReorder: 128
TCPFullUndo: 1038
TCPPartialUndo: 640
TCPDSACKUndo: 0
TCPLossUndo: 1
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 117
TCPSackFailures: 0
TCPLossFailures: 42
TCPFastRetrans: 1991
TCPForwardRetrans: 0
TCPSlowStartRetrans: 668
TCPTimeouts: 3276
TCPRenoRecoveryFail: 674
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 2
TCPRcvCollapsed: 5439
TCPDSACKOldSent: 4
TCPDSACKOfoSent: 0
TCPDSACKRecv: 2
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 14
TCPAbortOnClose: 56
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 32
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
ss
Current replacement of netstat is ss:
# ss --summary Total: 1449 (kernel 3546) TCP: 79 (estab 19, closed 33, orphaned 0, synrecv 0, timewait 31/0), ports 0 Transport Total IP IPv6 * 3546 - - RAW 0 0 0 UDP 9 6 3 TCP 46 16 30 INET 55 22 33 FRAG 0 0 0
Reasons for dropped packets
- NIC ring buffers getting full and unable to cope-up with incoming bursts of traffic
- CPU receiving NIC interrupts is very busy and unable to process
- Some cable/hardware/duplex issues
- Some bug in NIC driver
- Softnet backlog full (Measured from
/proc/net/softnet_stat) - Bad / Unintended VLAN tags
- Unknown / Unregistered protocols
- IPv6 frames when the server is not configured for IPv6
- Possibly some miss-configuration in Hypervisor (VMware)?
- Missing flow control?
Features of Network card
# ethtool ${DEV}
Settings for enp0s25:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Features of Network driver
# ethtool --show-features ${DEV}
Features for enp0s25:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
Network driver ring buffer
# ethtool --show-ring ${DEV}
Ring parameters for enp0s25:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 256
# ethtool --set-ring ${DEV} rx 4096
# ethtool --show-ring ${DEV}
Ring parameters for enp0s25:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 256
This did not help in my case.
TCP socket received and send buffer
# /sbin/sysctl -a --pattern 'net.ipv4' | grep mem net.ipv4.igmp_max_memberships = 20 net.ipv4.tcp_mem = 186753 249007 373506 net.ipv4.tcp_rmem = 4096 87380 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.udp_mem = 373509 498014 747018 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096
Flow Control
If flow control is activated can be checked as follows:
# ethtool -a ${DEV}
Pause parameters for enp0s25:
Autonegotiate: on
RX: on
TX: on
Check over time: sar
To check the behaviour over time sar is a good tool:
# sar -n EDEV 1 | grep -v lo Linux 4.15.0-70-generic (chef) 28/11/19 _x86_64_ (4 CPU) 11:56:51 IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s txcarr/s rxfram/s rxfifo/s txfifo/s 11:56:52 enp0s25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:56:53 enp0s25 0.00 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 11:56:54 enp0s25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:56:55 enp0s25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Dropwatch
# git clone https://github.com/pavel-odintsov/drop_watch # cd drop_watch/src # make gcc -I/usr/include/libnl3/ -c -g -D_GNU_SOURCE -Wall -Werror main.c gcc -I/usr/include/libnl3/ -c -g -D_GNU_SOURCE -Wall -Werror lookup.c gcc -I/usr/include/libnl3/ -c -g -D_GNU_SOURCE -Wall -Werror lookup_bfd.c gcc -I/usr/include/libnl3/ -c -g -D_GNU_SOURCE -Wall -Werror lookup_kas.c gcc -I/usr/include/libnl3/ -g -o dropwatch main.o lookup.o lookup_bfd.o lookup_kas.o -lbfd -lreadline -lnl-3 -lnl-genl-3
# ./dropwatch -l kas Initalizing kallsyms db dropwatch> start Enabling monitoring... Kernel monitoring activated. Issue Ctrl-C to stop monitoring 4 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d) 1 drops at inet_sock_destruct+4d (0xffffffff8c0f9e8d) 2 drops at arp_error_report+41 (0xffffffff8c0f00f1) 4 drops at unix_stream_connect+2e5 (0xffffffff8c12bbf5) 1 drops at neigh_probe+61 (0xffffffff8c076e61) 1 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 2 drops at neigh_probe+61 (0xffffffff8c076e61) 4 drops at unix_stream_connect+2e5 (0xffffffff8c12bbf5) 3 drops at sk_stream_kill_queues+4d (0xffffffff8c05c22d) 2 drops at arp_error_report+41 (0xffffffff8c0f00f1) 1 drops at icmpv6_rcv+1c1 (0xffffffff8c159081) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 2 drops at __netif_receive_skb_core+6b8 (0xffffffff8c06a498) 2 drops at __netif_receive_skb_core+6b8 (0xffffffff8c06a498) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) 4 drops at icmpv6_rcv+1c1 (0xffffffff8c159081) 9 drops at skb_queue_purge+17 (0xffffffff8c0550f7) Got a stop message dropwatch> Terminating dropwatch...
Some probing led to the function: __netif_receive_skb_core+6b8. But no clue what it is.
perf
perf record -g -a -e skb:kfree_skb
perf script
...
swapper 0 [003] 3161.113873: skb:kfree_skb: skbaddr=0xffff9247df562000 protocol=34939 location=0xffffffffb586a498
7fffb5856ec6 kfree_skb ([kernel.kallsyms])
7fffb586c498 __netif_receive_skb_core ([kernel.kallsyms])
7fffb586c998 __netif_receive_skb ([kernel.kallsyms])
7fffb586ed25 netif_receive_skb_internal ([kernel.kallsyms])
7fffb586faa5 napi_gro_receive ([kernel.kallsyms])
7fffc01fa9c6 e1000_receive_skb ([kernel.kallsyms])
7fffc01fc58e e1000_clean_rx_irq ([kernel.kallsyms])
7fffc0203fee e1000e_poll ([kernel.kallsyms])
7fffb586f180 net_rx_action ([kernel.kallsyms])
7fffb5c020e4 __softirqentry_text_start ([kernel.kallsyms])
7fffb5098bd5 irq_exit ([kernel.kallsyms])
7fffb5a04736 do_IRQ ([kernel.kallsyms])
7fffb5a0290c ret_from_intr ([kernel.kallsyms])
7fffb5806337 cpuidle_enter_state ([kernel.kallsyms])
7fffb58065b7 cpuidle_enter ([kernel.kallsyms])
7fffb50d96c3 call_cpuidle ([kernel.kallsyms])
7fffb50d993c do_idle ([kernel.kallsyms])
7fffb50d9b83 cpu_startup_entry ([kernel.kallsyms])
7fffb505c61b start_secondary ([kernel.kallsyms])
7fffb50020d5 secondary_startup_64 ([kernel.kallsyms])
...
tcpdump
# tcpdump --interface=${DEV} -w trace.pcap
tcpdump: listening on enp0s25, link-type EN10MB (Ethernet), capture size 262144 bytes
642 packets captured
642 packets received by filter
0 packets dropped by kernel
5 packets dropped by interface
# tcpdump -n -t -i ${DEV}
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s25, link-type EN10MB (Ethernet), capture size 262144 bytes
STP 802.1d, Config, Flags [none], bridge-id 8000.cc:5d:4e:2e:37:94.8002, length 47
IP chef > static.211.20.9.176.clients.your-server.de: ICMP echo request, id 7432, seq 1, length 64
IP chef.45753 > internetbox.home.domain: 51306+ [1au] PTR? 211.20.9.176.in-addr.arpa. (54)
IP static.211.20.9.176.clients.your-server.de > chef: ICMP echo reply, id 7432, seq 1, length 64
IP chef.51876 > static.211.20.9.176.clients.your-server.de.2222: Flags [S], seq 2727621364, win 29200, options [mss 1460,sackOK,TS val 1819633171 ecr 0,nop,wscale 7], length 0
IP internetbox.home.domain > chef.45753: 51306 1/0/1 PTR static.211.20.9.176.clients.your-server.de. (110)
IP chef.58204 > internetbox.home.domain: 37314+ [1au] PTR? 1.1.168.192.in-addr.arpa. (53)
IP internetbox.home.domain > chef.58204: 37314* 1/0/1 PTR internetbox.home. (83)
IP static.211.20.9.176.clients.your-server.de.2222 > chef.51876: Flags [R.], seq 0, ack 2727621365, win 0, length 0
IP chef > static.173.10.47.78.clients.your-server.de: ICMP echo request, id 7436, seq 1, length 64
IP chef.40629 > internetbox.home.domain: 32783+ [1au] PTR? 173.10.47.78.in-addr.arpa. (54)
IP static.173.10.47.78.clients.your-server.de > chef: ICMP echo reply, id 7436, seq 1, length 64
IP chef.34262 > static.173.10.47.78.clients.your-server.de.2222: Flags [S], seq 2923371146, win 29200, options [mss 1460,sackOK,TS val 1741234319 ecr 0,nop,wscale 7], length 0
IP internetbox.home.domain > chef.40629: 32783 1/0/1 PTR static.173.10.47.78.clients.your-server.de. (110)
e8:d1:1b:ec:98:30 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60:
0x0000: 0120 6000 0000 0000 0000 0000 0000 0000 ..`.............
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
e8:d1:1b:ec:98:31 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 64:
0x0000: 0120 6000 0000 0000 0000 0000 0000 0000 ..`.............
0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x0020: 0000 0000 0000 0000 0000 0000 0000 ..............
IP chef.34262 > static.173.10.47.78.clients.your-server.de.2222: Flags [S], seq 2923371146, win 29200, options [mss 1460,sackOK,TS val 1741235343 ecr 0,nop,wscale 7], length 0
IP internetbox.home.mdns > 224.0.0.251.mdns: 0 [25a] [13q] PTR (QM)? _services._dns-sd._udp.local. PTR (QM)? _printer._tcp.local. PTR (QM)? _pdl-datastream._tcp.local. PTR (QM)? _ipp._tcp.local. PTR (QM)? _http._tcp.local. PTR (QM)? _scanner._tcp.local. PTR (QM)? _http-alt._tcp.local. PTR (QM)? _uscan._tcp.local. PTR (QM)? _ipps._tcp.local. PTR (QM)? _privet._tcp.local. PTR (QM)? _uscans._tcp.local. PTR (QM)? _mqtt._tcp.local. PTR (QM)? _ssw._tcp.local. (968)
IP chef.43103 > internetbox.home.domain: 55791+ [1au] PTR? 251.0.0.224.in-addr.arpa. (53)
IP internetbox.home.domain > chef.43103: 55791 NXDomain 0/1/1 (110)
IP chef.43103 > internetbox.home.domain: 55791+ PTR? 251.0.0.224.in-addr.arpa. (42)
IP internetbox.home.domain > chef.43103: 55791 NXDomain 0/0/0 (42)
STP 802.1d, Config, Flags [none], bridge-id 8000.cc:5d:4e:2e:37:94.8002, length 47
23 packets captured
23 packets received by filter
0 packets dropped by kernel
1 packet dropped by interface
/proc/net/netstat
# ./myenv/utl/netstat.php --skip-zeroj | grep -i drop LockDroppedIcmps 0 ListenDrops 0 TCPBacklogDrop 0 PFMemallocDrop 0 TCPMinTTLDrop 0 TCPDeferAcceptDrop 0 TCPReqQFullDrop 0 TCPOFODrop 0
System Tap (stap)
# stap --all-modules dropwatch.stp Monitoring for dropped packets 18 packets dropped at skb_queue_purge 8 packets dropped at unix_stream_connect 5 packets dropped at sk_stream_kill_queues 2 packets dropped at icmp_rcv 2 packets dropped at __netif_receive_skb_core 1 packets dropped at tcp_v4_do_rcv 1 packets dropped at tcp_v4_rcv Stopping dropped packet monitor
# stap --all-modules drop.stp Timestamp Src IP Dst IP IP Proto TCP Src Port TCP Dst Port Execname Wed Nov 27 21:55:51 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:55:51 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:56:06 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:56:06 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:56:36 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3 Wed Nov 27 21:56:36 2019 0.0.0.0 0.128.239.209 0 0 0 swapper/3
#! /usr/bin/env stap
#
# drop.stp
#
probe begin {
printf("Timestamp Src IP Dst IP IP Proto TCP Src Port TCP Dst Port Execname\n")
}
probe end {
printf("Stopping dropped packet monitor\n")
}
probe kernel.trace("kfree_skb") {
if (symname($location) == "__netif_receive_skb_core") {
printf("%s %s %s %5d %5d %5d %s\n", ctime(gettimeofday_s()),
ip_ntop(__ip_skb_saddr($skb)),
ip_ntop(__ip_skb_daddr($skb)),
__ip_skb_proto($skb),
__tcp_skb_dport($skb),
__tcp_skb_sport($skb),
execname() )
}
}
If interpretation of IP protocol is correctly:
| 0x00 | 0 | HOPOPT | IPv6 | Hop-by-Hop Option | RFC 8200 |
No clue, what it finally is. Giving up...
Literature
- What exactly is an ifconfig dropped RX packet?
- ifconfig reports packet drop
Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. Before, dropped packets was most likely due to an error. Now, the rx_dropped counter shows statistics for dropped frames because of:
- Softnet backlog full
- Bad / Unintended VLAN tags
- Unknown / Unregistered protocols
- IPv6 frames when the server is not configured for IPv6
- Finding out if/why a server is dropping packets
- ifconfig shows dropped rx packets
- Using netstat and dropwatch to observe packet loss on Linux servers
- Monitoring and Tuning the Linux Networking Stack: Receiving Data
- Monitoring and Tuning the Linux Networking Stack: Sending Data
- vSphere 6 experiencing high packet loss
- Network Troubleshooting Guide
dropwatch- Linux man page- Brendan Gregg:
perfExamples - Manpage of
tcpdump - List of IP protocol numbers
- Monitoring Network Packets Drops in Kernel
- SystemTap
Comments
WMware snapshots or Veeam backups
Good explanation for bond interfaces
802.3ad bond interface have show high RX dropped packets
I found a god explanation for dropped packages on bonded interfaces: [ 1 ]
This is related to the bonding mode and _not_ a bug. The bonding module will drop duplicate frames received on inactive ports, which is normal behaviour. Overall the packets should be getting into the machine without problems since they are received on the active slave. To confirm this do the following
1) Check dropped packets from all interfaces. So if eth0/eth1 are connected to bond0, we may see dropped packets for bond0 and eth0, but not for eth1. This depends on which interface is the active interface. This can be checked using the following:
So if the
active_slaveisn't dropping packets, and the inactive slave is dropping packets this is normal in 'active-backup' mode (or any mode where there is an inactive slave).2) If we want both interfaces to not drop packets we can use '
all_slaves_active' bonding module parameter.Check:
It should default to 0 which means drop frames on the inactive slave.
If we set this to 1, we will no longer drop frames:
Kernel Documentation