ASA 5515-X overrun

Problemy z zakresu security (VPN, firewall, IDS/IPS itp.)
Wiadomość
Autor
max239
wannabe
wannabe
Posty: 270
Rejestracja: 04 paź 2007, 09:22

ASA 5515-X overrun

#1

#1 Post autor: max239 »

Witajcie

jest problem
z ASA5515

Kod: Zaznacz cały

6586 input errors, 0 CRC, 0 frame, 6586 overrun, 0 ignored, 0 abort
przy czym asa jest wogole nie oversubscribed bo :

Kod: Zaznacz cały

 5 minute input rate 7999 pkts/sec,  5605190 bytes/sec
      5 minute output rate 8932 pkts/sec,  4449910 bytes/sec
      5 minute drop rate, 2 pkts/sec

co poczac :(

lacze do do netu z ktorego czytam te overruny ma 100m ... wiec wogole nie moze tutaj sie stac nic takiego ... a jednak sie dzieje.....

Awatar użytkownika
peper
CCIE / Site Admin
CCIE / Site Admin
Posty: 5005
Rejestracja: 13 sie 2004, 12:19
Lokalizacja: Warsaw, PL
Kontakt:

Re: ASA 5515-X overrun

#2

#2 Post autor: peper »

Strzelam, że jeżeli nie jest to jakiś efekt wady fizycznej sprzętu to jest to efekt software-u. Jakie procesy obciążają Ci pudełko, czy nie skacze Ci CPU (show proc cpu-hog), NetFlow nie powoduje przeciążenia (choć przy takim ruchu nie powinien). Zobacz też na wszelki wypadek czy kolejka na wejściu się nie wysyca, może ktoś jakieś dziwne parametry na QoS/FlowControl ustawił.
Szkoła DevNet: https://szkoladevnet.pl


Facebook: https://www.facebook.com/Piotr.Wojciechowski.CCIE
LinkedIn: https://www.linkedin.com/in/peper
Twitter: https://www.twitter.com/PiotrW_CCIE

"Zapomniałem że od kilku lat wszyscy giną jakby nigdy ich nie miało być
w stu tysiącach jednakowych miast giną jak psy"

max239
wannabe
wannabe
Posty: 270
Rejestracja: 04 paź 2007, 09:22

Re: ASA 5515-X overrun

#3

#3 Post autor: max239 »

Kod: Zaznacz cały

 sh processes cpu-hog

Process:      tmatch compile thread, NUMHOG: 2, MAXHOG: 2, LASTHOG: 2
LASTHOG At:   12:43:20 CEDT May 23 2017
PC:           0x00000000006e0ccb (suspend)
Call stack:   0x00000000006c953b  0x00000000006c9600  0x00000000006c9600
              0x00000000006c9600  0x00000000006c9600  0x00000000006c9600
              0x00000000006c9c08  0x00000000006c9c08  0x00000000006c9600
              0x00000000006d4d6b  0x00000000006f046e  0x00000000006f07e3
              0x00000000006f07e3  0x00000000006e41fd


Process:      tmatch compile thread, NUMHOG: 3, MAXHOG: 2, LASTHOG: 2
LASTHOG At:   12:43:22 CEDT May 23 2017
PC:           0x00000000006e0ccb (suspend)
Call stack:   0x00000000006d499d  0x00000000006d4cc1  0x00000000006f046e
              0x00000000006f07e3  0x00000000006f07e3  0x00000000006e41fd
              0x000000000044fdeb


Process:      tmatch compile thread, NUMHOG: 2, MAXHOG: 2, LASTHOG: 2
LASTHOG At:   12:43:22 CEDT May 23 2017
PC:           0x00000000006e0ccb (suspend)
Call stack:   0x00000000006d499d  0x00000000006d4cc1  0x00000000006f08c6
              0x00000000006f07e3  0x00000000006f07e3  0x00000000006e41fd
              0x000000000044fdeb


Process:      tmatch compile thread, NUMHOG: 1, MAXHOG: 2, LASTHOG: 2
LASTHOG At:   12:43:22 CEDT May 23 2017
PC:           0x00000000006e0ccb (suspend)
Call stack:   0x00000000006c953b  0x00000000006c9600  0x00000000006c9c08
              0x00000000006c9c08  0x00000000006c9600  0x00000000006d4d6b
              0x00000000006f08c6  0x00000000006f07e3  0x00000000006f07e3
              0x00000000006e41fd  0x000000000044fdeb


Process:      tmatch compile thread, NUMHOG: 2, MAXHOG: 2, LASTHOG: 2
LASTHOG At:   12:43:27 CEDT May 23 2017
PC:           0x00000000006e0ccb (suspend)
Call stack:   0x00000000006c8980  0x00000000006c821f  0x00000000006c7079
              0x00000000006d2ca7  0x00000000006c9600  0x00000000006c821f
              0x00000000006c821f  0x00000000006c7079  0x00000000006d2ca7
              0x00000000006c9600  0x00000000006c821f  0x00000000006c821f
              0x00000000006c7079  0x00000000006d2ca7


Process:      tmatch compile thread, NUMHOG: 12, MAXHOG: 2, LASTHOG: 2
LASTHOG At:   12:43:31 CEDT May 23 2017
PC:           0x00000000006e0ccb (suspend)
Call stack:   0x00000000006c953b  0x00000000006d4d6b  0x00000000006f046e
              0x00000000006f07e3  0x00000000006f07e3  0x00000000006e41fd
              0x000000000044fdeb


Process:      tmatch compile thread, PROC_PC_TOTAL: 159, MAXHOG: 28, LASTHOG: 2
LASTHOG At:   12:43:31 CEDT May 23 2017
PC:           0x00000000006e0ccb (suspend)

Process:      tmatch compile thread, NUMHOG: 26, MAXHOG: 28, LASTHOG: 2
LASTHOG At:   12:43:31 CEDT May 23 2017
PC:           0x00000000006e0ccb (suspend)
Call stack:   0x00000000006c953b  0x00000000006d4d6b  0x00000000006f08c6
              0x00000000006e41fd  0x000000000044fdeb


Process:      telnet/ci, PROC_PC_TOTAL: 1, MAXHOG: 28, LASTHOG: 28
LASTHOG At:   12:44:53 CEDT May 23 2017
PC:           0x00000000019e9116 (suspend)

Process:      telnet/ci, NUMHOG: 1, MAXHOG: 28, LASTHOG: 28
LASTHOG At:   12:44:53 CEDT May 23 2017
PC:           0x00000000019e9116 (suspend)
Call stack:   0x00000000019e9116  0x00000000019e96dd  0x0000000001a0f962
              0x0000000001a0f9cb  0x000000000146e047  0x00000000005027a0
              0x0000000000502e5f  0x00000000004f0800  0x00000000004f18fd
              0x000000000044fdeb


Process:      NIC status poll, PROC_PC_TOTAL: 1, MAXHOG: 55, LASTHOG: 55
LASTHOG At:   13:08:32 CEDT May 23 2017
PC:           0x00000000015276fb (suspend)

Process:      aaa_shim_thread, PROC_PC_TOTAL: 1, MAXHOG: 1, LASTHOG: 1
LASTHOG At:   13:11:25 CEDT May 23 2017
PC:           0x0000000001a5f0df (suspend)

Process:      aaa_shim_thread, NUMHOG: 1, MAXHOG: 1, LASTHOG: 1
LASTHOG At:   13:11:25 CEDT May 23 2017
PC:           0x0000000001a5f0df (suspend)
Call stack:   0x0000000000450fd6  0x0000000001a5f0df  0x0000000001a34ff0
              0x0000000001a3b337  0x0000000001a3b428


Process:      arp_timer, PROC_PC_TOTAL: 1, MAXHOG: 1, LASTHOG: 1
LASTHOG At:   13:11:54 CEDT May 23 2017
PC:           0x0000000000bff8ef (suspend)

Process:      tmatch compile thread, PROC_PC_TOTAL: 50, MAXHOG: 1, LASTHOG: 1
LASTHOG At:   13:24:19 CEDT May 23 2017
PC:           0x00000000006e4f9c (suspend)

Process:      tmatch compile thread, NUMHOG: 50, MAXHOG: 1, LASTHOG: 1
LASTHOG At:   13:24:19 CEDT May 23 2017
PC:           0x00000000006e4f9c (suspend)
Call stack:   0x00000000006e4f9c  0x000000000044fdeb


Process:      EIGRP-IPv4 Hello, PROC_PC_TOTAL: 5, MAXHOG: 10, LASTHOG: 2
LASTHOG At:   13:24:22 CEDT May 23 2017
PC:           0x00000000013d3095 (suspend)

Process:      EIGRP-IPv4 Hello, NUMHOG: 5, MAXHOG: 10, LASTHOG: 2
LASTHOG At:   13:24:22 CEDT May 23 2017
PC:           0x00000000013d3095 (suspend)
Call stack:   0x00000000013d3095  0x000000000044fdeb


Process:      CP HA Processing, PROC_PC_TOTAL: 17, MAXHOG: 55, LASTHOG: 55
LASTHOG At:   13:24:40 CEDT May 23 2017
PC:           0x000000000083f0bf (suspend)

Process:      CP Crypto Result Processing, PROC_PC_TOTAL: 17, MAXHOG: 55, LASTHO                                                                                                                                                             G: 22
LASTHOG At:   13:25:55 CEDT May 23 2017
PC:           0x000000000083f319 (suspend)

Process:      CP Threat-Detection Processing, PROC_PC_TOTAL: 18, MAXHOG: 55, LAS                                                                                                                                                             THOG: 20
LASTHOG At:   13:26:03 CEDT May 23 2017
PC:           0x000000000083eef7 (suspend)

Process:      CP ARP Processing, PROC_PC_TOTAL: 13, MAXHOG: 42, LASTHOG: 9
LASTHOG At:   13:26:27 CEDT May 23 2017
PC:           0x000000000083ee11 (suspend)

Process:      CP DP CXSC Event Processing, PROC_PC_TOTAL: 19, MAXHOG: 55, LASTHO                                                                                                                                                             G: 3
LASTHOG At:   13:26:59 CEDT May 23 2017
PC:           0x000000000083efdf (suspend)

Process:      CP Midpath Processing, PROC_PC_TOTAL: 21, MAXHOG: 55, LASTHOG: 25
LASTHOG At:   13:27:23 CEDT May 23 2017
PC:           0x000000000083ebec (suspend)

Process:      CP Processing, PROC_PC_TOTAL: 133, MAXHOG: 55, LASTHOG: 13
LASTHOG At:   13:27:47 CEDT May 23 2017
PC:           0x000000000083f2b6 (suspend)

Process:      CP Midpath Processing, NUMHOG: 80953, MAXHOG: 55, LASTHOG: 13
LASTHOG At:   13:27:47 CEDT May 23 2017
PC:           0x000000000083ebec (suspend)
Call stack:   0x000000000044fdeb


Process:      Environment Monitor Process, PROC_PC_TOTAL: 98982, MAXHOG: 55, LAS                                                                                                                                                             THOG: 1
LASTHOG At:   13:28:04 CEDT May 23 2017
PC:           0x0000000000c13a5e (suspend)

Process:      Environment Monitor Process, NUMHOG: 95960, MAXHOG: 55, LASTHOG: 1
LASTHOG At:   13:28:04 CEDT May 23 2017
PC:           0x0000000000c13a5e (suspend)
Call stack:   0x0000000000c13a5e  0x0000000000c1499b  0x00000000007cb440
              0x000000000044fdeb


Process:      EIGRP-IPv4, PROC_PC_TOTAL: 492794, MAXHOG: 111, LASTHOG: 2
LASTHOG At:   13:28:14 CEDT May 23 2017
PC:           0x00000000013d86a0 (suspend)

Process:      EIGRP-IPv4, NUMHOG: 484164, MAXHOG: 103, LASTHOG: 2
LASTHOG At:   13:28:14 CEDT May 23 2017
PC:           0x00000000013d86a0 (suspend)
Call stack:   0x00000000013d86a0  0x000000000044fdeb


Process:      DATAPATH-0-1792, PROC_PC_TOTAL: 134208, MAXHOG: 56, LASTHOG: 33
LASTHOG At:   13:28:15 CEDT May 23 2017
PC:           0x0000000000000000 (suspend)

Process:      DATAPATH-0-1792, NUMHOG: 134179, MAXHOG: 56, LASTHOG: 33
LASTHOG At:   13:28:15 CEDT May 23 2017
PC:           0x0000000000000000 (suspend)
Call stack:   0x00000000004392ea  0x000000000072bd7d  0x000000000172fa90
              0x00000000017391fc  0x0000003e82808201


CPU hog threshold (msec):  1.542
Last cleared: None
ASA5515#

max239
wannabe
wannabe
Posty: 270
Rejestracja: 04 paź 2007, 09:22

Re: ASA 5515-X overrun

#4

#4 Post autor: max239 »

netflow dopiero chce wlaczyc zeby popodgladac ten ruch

tdewille
member
member
Posty: 26
Rejestracja: 01 gru 2016, 17:45

Re: ASA 5515-X overrun

#5

#5 Post autor: tdewille »

ja bym zobaczył czy nie za dużo logujesz. zmniejsz poziom logowania i obserwuj. no i flow control tak jak już peper mówił.
http://www.cisco.com/c/en/us/support/do ... te-00.html

używasz tej asy do terminowania vpna? robisz SNMP?

i jeszcze to:
https://s3.amazonaws.com/tacsecuritypod ... ode_20.mp3
(pierwsze pare minut)
https://bst.cloudapps.cisco.com/bugsear ... kviewredir

max239
wannabe
wannabe
Posty: 270
Rejestracja: 04 paź 2007, 09:22

Re: ASA 5515-X overrun

#6

#6 Post autor: max239 »

ok "solve" dla wszystkich - chociaż trudno to nazwac solve..

downgrade z

Kod: Zaznacz cały

asa944-5-smp-k8.bin
do

Kod: Zaznacz cały

asa912-smp-k8.bin
rozwiazal problem

max239
wannabe
wannabe
Posty: 270
Rejestracja: 04 paź 2007, 09:22

Re: ASA 5515-X overrun

#7

#7 Post autor: max239 »

Niestety downgrade nie rozwiązał problemu, więc postanowiliśmy postawić CSR 1000v tylko do routowania Publicznych adresów.

I tu mamy taki problem (config ocenzurowany :D):

Kod: Zaznacz cały

interface GigabitEthernet1
 description WAN1
 ip tcp adjust-mss 1460
 negotiation auto
!
interface GigabitEthernet2
 description WAN2
 ip tcp adjust-mss 1460
 negotiation auto
!
interface GigabitEthernet3
 description LACZE
 ip tcp adjust-mss 1460
 negotiation auto
 hold-queue 1000 in

GigabitEthernet1 is up, line protocol is up
  Hardware is CSR vNIC, address is ea0a.7690.f661 (bia ea0a.7690.f661)
  Description: WAN1
  Internet address is 
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 248/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1000Mbps, link type is auto, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:05, output hang never
  Last clearing of "show interface" counters 00:04:14
  Input queue: 0/375/151374/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 432000 bits/sec, 77 packets/sec
  5 minute output rate 363000 bits/sec, 62 packets/sec
     25410 packets input, 15269384 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     518 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     20490 packets output, 12196922 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     64 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet2 is up, line protocol is up
  Hardware is CSR vNIC, address is 7ec3.3388.449d (bia 7ec3.3388.449d)
  Description: WAN2
  Internet address is 
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 8/255, rxload 10/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1000Mbps, link type is auto, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:03, output hang never
  Last clearing of "show interface" counters 00:04:14
  Input queue: 0/375/4111482/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 42370000 bits/sec, 7082 packets/sec
  5 minute output rate 33996000 bits/sec, 6091 packets/sec
     1962426 packets input, 1520768113 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     8984 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     1705636 packets output, 1224090214 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     355 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet3 is up, line protocol is up
  Hardware is CSR vNIC, address is 8e9a.6c65.7914 (bia 8e9a.6c65.7914)
  Description: LACZE
  Internet address is 
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 10/255, rxload 9/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1000Mbps, link type is auto, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:01, output 01:00:49, output hang never
  Last clearing of "show interface" counters 00:04:14
  Input queue: 0/1000/9156026/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 35601000 bits/sec, 6286 packets/sec
  5 minute output rate 42121000 bits/sec, 7115 packets/sec
     1758534 packets input, 1280110754 bytes, 0 no buffer
     Received 0 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     16058 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     1969346 packets output, 1509755358 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     520 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
I niestety poprawy nie ma, natomiast na tym routerze widać trochę więcej, a mianowicie:
Input queue: 0/1000/9156026/0 (size/max/drops/flushes); Total output drops: 0 na interfejscie lacza internetowego ma niesamowitą ilość dropów na kolejce input.

Czy wie ktoś skąd taki problem ??
Nie chce mi się wierzyć, że łącze 100Mb/100Mb niszczy ASE 5515-X i CSRa 1000v :(

Wygląda to jak by SPD zadzialalo jak przy zapchanej sieci, ale to jest raczej nie mozliwe przy takich prędkościach :(

Awatar użytkownika
peper
CCIE / Site Admin
CCIE / Site Admin
Posty: 5005
Rejestracja: 13 sie 2004, 12:19
Lokalizacja: Warsaw, PL
Kontakt:

Re: ASA 5515-X overrun

#8

#8 Post autor: peper »

A rozmawiałeś z operatorem jak wyglądają statystyki interfejsu po jego stronie?
Szkoła DevNet: https://szkoladevnet.pl


Facebook: https://www.facebook.com/Piotr.Wojciechowski.CCIE
LinkedIn: https://www.linkedin.com/in/peper
Twitter: https://www.twitter.com/PiotrW_CCIE

"Zapomniałem że od kilku lat wszyscy giną jakby nigdy ich nie miało być
w stu tysiącach jednakowych miast giną jak psy"

tdewille
member
member
Posty: 26
Rejestracja: 01 gru 2016, 17:45

Re: ASA 5515-X overrun

#9

#9 Post autor: tdewille »

to skoro SPD to może zwiększ thresholdy dla spd?

ODPOWIEDZ