Zyxel T-56 ping spikes

  • 4 September 2023
  • 43 reacties
  • 1517 Bekeken

Reputatie 1

Hi all, pardon the engels, I am seeing the weirdest latency spikes to my new Zyxel T-56 modem, from devices directly attached to it. 

64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=0.555 ms
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.533 ms
64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=516 ms
64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=0.578 ms
64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=210 ms
64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=0.568 ms

 

If I run a tcpping to google.com , so that is now LAN to WAN, and not just LAN to router. 

 

255  ams16s37-in-f14.1e100.net (172.217.23.206)  6.677 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  7.382 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  554.650 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  7.165 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  6.706 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  6.652 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  6.536 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  7.220 ms

 

I see the same thing?

This is weird how it gets these spikes, on LAN being directly attached I would expect to see max maybe 2ms. Not 516ms. Is this router getting overloaded, or is the hardware just kinda cheap? 

Tommie van Odido 10 maanden geleden

Hey @ChaosMonkey, welcome to our Community!

Good that you are sounding the alarm for this. I immediately went to check for you and I see that you have your own router/switch connected. Could you run these two commands if you have unplugged everything and are only connected directly wired to your laptop/desktop? Ping -n 50 1.1.1.1 & Ping -n 50 8.8.8.8 in CMD. Do this with only the Zyxel modem connected to your network with a wired connection. Suppose you have a VPN, disable it too.

Bekijk origineel

43 reacties

Reputatie 1

Seems it is based on the EX5601/EX5600-T SERIES 

Reputatie 7
Badge +14

Hi @ChaosMonkey 

Did you use a wired connection between your test device and the router? Wifi is unreliable for such test.

Reputatie 1

Hi @ChaosMonkey 

Did you use a wired connection between your test device and the router? Wifi is unreliable for such test.

Yeah, you can see that from the first ping, the sub ms ping is normally not achievable on wifi. 
I am wired in with all my devices except my phone. 

Reputatie 7
Badge +9

Hey @ChaosMonkey, welcome to our Community!

Good that you are sounding the alarm for this. I immediately went to check for you and I see that you have your own router/switch connected. Could you run these two commands if you have unplugged everything and are only connected directly wired to your laptop/desktop? Ping -n 50 1.1.1.1 & Ping -n 50 8.8.8.8 in CMD. Do this with only the Zyxel modem connected to your network with a wired connection. Suppose you have a VPN, disable it too.

Reputatie 1

Hey @ChaosMonkey, welcome to our Community!

Good that you are sounding the alarm for this. I immediately went to check for you and I see that you have your own router/switch connected. Could you run these two commands if you have unplugged everything and are only connected directly wired to your laptop/desktop? Ping -n 50 1.1.1.1 & Ping -n 50 8.8.8.8 in CMD. Do this with only the Zyxel modem connected to your network with a wired connection. Suppose you have a VPN, disable it too.

What do you mean you can see I have my own router/switch connected? The tests I showed was done from a linux machine directly connected to the Zyxel router. Even with just my PC connected it still shows odd spikes, and I was initially talking about ping spikes to the Zyxel router itself. That is why my concern is sitting 
 

 

Reputatie 7
Badge +9

Hey @ChaosMonkey, can you take a screenshot of the results instead of the hops? Also, I would like to ask you to make a sketch of your home network. What we suggest is ONT/Media Converter -> Zyxel -> Wired connection laptop/other device. I am very curious to know what else is in between.

Reputatie 1

Hey @ChaosMonkey, can you take a screenshot of the results instead of the hops? Also, I would like to ask you to make a sketch of your home network. What we suggest is ONT/Media Converter -> Zyxel -> Wired connection laptop/other device. I am very curious to know what else is in between.

I do have it as ONT → Zyxel → Wired connection.
There is nothing between the ONT and the Zyxel. 

 

I have made a diagram of my network. I have done the tests from both my PC and the orange pi (see diagram, that is directly connected to the Zyxel) and it doesn’t change. I am questioning this Zyxel because of the spikes to it. I haven’t seen for example at a previous ISP my Mikrotik (ISP supplied router) do this. 

Even if I unplug everything like I did in the previous results and have just my PC connected directly to the Zyxel do I see the exact same behaviour. 
 


From the PC
TCP ping to google.com


From the orange pi 
 

I have an openwrt router I can also test with , I just need to know what IPoE settings to use? 

Reputatie 7
Badge +9

Hi @ChaosMonkey, thanks for all the screenshots. I would like to ask if @TMTV and @louisL would like to assist you with the settings. I am also curious if they can see anything else in the screenshots with their expert eye.  

Reputatie 1

Hi @ChaosMonkey, thanks for all the screenshots. I would like to ask if @TMTV and @louisL would like to assist you with the settings. I am also curious if they can see anything else in the screenshots with their expert eye.  

Hi, one thing that seems to stand out to me, is that the average response time of the modem seems to be an issue. Which could explain the odd avg’s seen in the pings to the google and cloudflare. If ICMP was being throttled that could explain it, but the network isn’t busy and a TCP ping rules that out. Seems the modem or the custom firmware T-Mobile (Odido) puts on this Zyxel might have an issue?

Reputatie 7
Badge +9

@ChaosMonkey, at the moment, I am not leaving or ruling anything out. The modem could possibly be a cause. I just want to hold out for another look from our Superusers! 

Reputatie 1

For comparison, I used my own router, so just swapped out the Zyxel for a mikrotik hex (and it’s not even as powerful.) 

Fyi I connect to the Zyxel over a 2.5Gbps connection. 
 

Here is what it looks like using the mikrotik

 

 

So it looks like the custom firmware that is on the Zyxel from Odido(T-Mobile) could be the cause. 

  1.  Can I ask for a replacement router?
  2. Can I beta test new firmware?

 

Reputatie 7
Badge +9

Hey @ChaosMonkey, that's an interesting comparison. I have sent you a new modem just to be sure. Could you test this one out and provide me with feedback?

Reputatie 1

Hey @ChaosMonkey, that's an interesting comparison. I have sent you a new modem just to be sure. Could you test this one out and provide me with feedback?

Will do. 

 

Also there seems to be something unstable happening in the routing. 
 

hop 2 and 3 show route changes, so I am getting unstable latencies 

Reputatie 1

Hey @ChaosMonkey, that's an interesting comparison. I have sent you a new modem just to be sure. Could you test this one out and provide me with feedback?

Hi Tommie, 

Got the new modem, just in time as well, the old one actually stopped getting an IP address on it’s WAN port, meaning I had no internet. Even after resetting it and a call with support it just stopped working. 

 

At least the new modem immediately got an IP address. So far so good. Even my internet speeds are better. 
I did pick up times where I was getting ping spikes, however the logs of the new modem revealed this 
102.611585] SYN_FLOODING ATTACK:IN=eth1.3 OUT= MAC=10:71:b3:a1:dc:bd:00:0e:00:00:00:04:08:00 SRC=185.191.225.130 DST=<IP> LEN=60 TOS=0x00 PREC=0x00 TTL=58 ID=1016 DF PROTO=TCP SPT=35387 DPT=8000 WINDOW=64240 RES=0x00 SYN URGP=0 

This actually caused the modem to drop my internet connection. 

Reputatie 7
Badge +9

Hey @ChaosMonkey , this seems to be a possible denial of service attempt. More people seem to have reported the IP-address 185.191.225.130 as suspect. See for example https://www.abuseipdb.com/check/185.191.225.130 abuseipdb.comabuseipdb.com 185.191.225.130 | Probe Networks | AbuseIPDB 185.191.225.130 has been reported 142 times.

I am not sure how to tackle this issue. This may be an hacker attacking you (do you run a game where bad actors want to attack you?). But as you have a new modem (and a new IP-address) it may be directed to the previous user that had that IP-address. The destination port 8000 could also be the result of a service you have or had had open to the internet (forwarded to an internal server) that people want to disable (see  https://www.speedguide.net/port.php?port=8000 for some examples) SpeedGuideSpeedGuide

Port 8000 (tcp/udp) Port 8000 tcp/udp information, assignments, application use and known security risks. Was this the only message or did you receive more?

Reputatie 1

Hey @ChaosMonkey , this seems to be a possible denial of service attempt. More people seem to have reported the IP-address 185.191.225.130 as suspect. See for example https://www.abuseipdb.com/check/185.191.225.130 abuseipdb.comabuseipdb.com 185.191.225.130 | Probe Networks | AbuseIPDB 185.191.225.130 has been reported 142 times.

I am not sure how to tackle this issue. This may be an hacker attacking you (do you run a game where bad actors want to attack you?). But as you have a new modem (and a new IP-address) it may be directed to the previous user that had that IP-address. The destination port 8000 could also be the result of a service you have or had had open to the internet (forwarded to an internal server) that people want to disable (see  https://www.speedguide.net/port.php?port=8000 for some examples) SpeedGuideSpeedGuide

Port 8000 (tcp/udp) Port 8000 tcp/udp information, assignments, application use and known security risks. Was this the only message or did you receive more?

Hi @Tommie van Odido , so first off, I don’t have an ports mapped on port 8000. I was getting ping of death logs, and tons of these SYN flood attack logs, even with a new IP. So it’s definitely not targeted at me specifically. Odd thing is, this type of attack is normally not an issue on routers where it will cause a drop in the connection or a spike in CPU usage.


Now I cannot adjust the firewall rules in any meaningful manner, and that is what normally is done to stop this kind of thing. Which would also explain why this wasn’t an issue when I was using the mikrotik, since it would just drop this traffic and be done with it. 


As to why port 8000 shows up, that is something for Odido to address, since that isn’t something I configured nor can see that it is open. The games I play that might open a port with uPnP, but they don’t use port 8000, they either use a port in the 300-400 range, or in the 27k range. 

It is puzzling why it would lead to the device dropping the WAN connection.

Reputatie 1

Hey @ChaosMonkey, that's an interesting comparison. I have sent you a new modem just to be sure. Could you test this one out and provide me with feedback?

Hi Tommie, so I did some testing, seems the zyxel does indeed take a knock with the attacks. I can help beta firmware and see if that helps. 
For completeness sake, see the screenshot below. This was taken from my Mikrotik CCR2004 (there is no ways this connection will ever over load this device. 
 

What is a concern and what I would like to draw your attention to, is the fact that I am getting a route change between two IP’s at hope 2 and depending on the path hop 3 as well. now this shouldn’t be happening this frequently, this immediately make me wonder what exactly is going on?

Reputatie 7
Badge +9

Hey @ChaosMonkey, hmm could you run a couple of speedtests for me? Then I will forward this to our tech guys.

The instructions for this can be found on our website. 

Please follow all the steps carefully, please share the screenshots in the topic instead of to our e-mail. We cannot process an application with missing steps.

Reputatie 1

Hey @ChaosMonkey, hmm could you run a couple of speedtests for me? Then I will forward this to our tech guys.

The instructions for this can be found on our website. 

Please follow all the steps carefully, please share the screenshots in the topic instead of to our e-mail. We cannot process an application with missing steps.

Would this still be behind the Zyxel or behind my mikrotik as the edge router?

Curious why you have no guide for linux? That would be my go to, to make sure nothing is running the background.

Reputatie 7
Badge +9

@ChaosMonkey, could you run the tests with only the Zyxel? 

We do not have a lot of customers using Linux haha. Therefore we do not have a Linus manual. 

Reputatie 6
Badge

@ChaosMonkey, could you run the tests with only the Zyxel? 

We do not have a lot of customers using Linux haha. Therefore we do not have a Linus manual. 

hi @ChaosMonkey : for Linux these test are not so dififcult  if you know how to use the command line.

Download the appropiate CLI client from https://www.speedtest.net/nl/apps/cli install it and run the test. I don’t know if they have a CLI client for non x86_64 hardware. When I run this client on my box I get something like:

[louis@lair ~]$ speedtest

   Speedtest by Ookla

      Server: ColoCenter bv - Zoetermeer (id: 3242)
         ISP: T-Mobile Thuis
Idle Latency:     4.72 ms   (jitter: 0.25ms, low: 4.46ms, high: 5.11ms)
    Download:   275.14 Mbps (data used: 311.8 MB)                                                   
                  4.93 ms   (jitter: 0.49ms, low: 3.92ms, high: 18.98ms)
      Upload:   389.91 Mbps (data used: 502.5 MB)                                                   
                  4.89 ms   (jitter: 7.02ms, low: 3.87ms, high: 222.81ms)
 Packet Loss:     0.6%
Re-running that test gives me better results (from another remote server)

I don’t feel however that this will give you much: your observation that there is a route flapping in your path indeed could be related. When the route (in the Odido network) flaps seems to be the point where packet loss occurs (the next hop after the first flapping router seems to show some packet loss when that happens. @Tommie van Odido: please ask the network guys to have a look at that. It may however be related to the second issue below when a DOS attack occurs?

The router going down when you are DOS’ed is another issue however.  The Zyxel should not be affected by it. You could try to disable the DOS protection under “beveiliging → firewall to see if that helps. It may be a case where the firewal is to pro-active and reporting on all threats (even on closed ports!) takes so much resources that the router hangs or reboots. @Tommie van Odido you may want to check with network people if this needs to be reported to Zyxel

The DOS attack may be triggered on the IP-address as soon as you get identified and you IP-address gets found out, so you cannot rule out that the attack is directed to you.

I may be wrong, but to me it all looks like being related to a DOS attack on you.

 

 

 

Reputatie 1

Hey @ChaosMonkey, hmm could you run a couple of speedtests for me? Then I will forward this to our tech guys.

The instructions for this can be found on our website. 

Please follow all the steps carefully, please share the screenshots in the topic instead of to our e-mail. We cannot process an application with missing steps.

Would this still be behind the Zyxel or behind my mikrotik as the edge router?

Curious why you have no guide for linux? That would be my go to, to make sure nothing is running the background.

Hi @Tommie van Odido , 

Here are the screenshots 
 

 

Reputatie 1

@ChaosMonkey, could you run the tests with only the Zyxel? 

We do not have a lot of customers using Linux haha. Therefore we do not have a Linus manual. 

hi @ChaosMonkey : for Linux these test are not so dififcult  if you know how to use the command line.

Download the appropiate CLI client from https://www.speedtest.net/nl/apps/cli install it and run the test. I don’t know if they have a CLI client for non x86_64 hardware. When I run this client on my box I get something like:

[louis@lair ~]$ speedtest

   Speedtest by Ookla

      Server: ColoCenter bv - Zoetermeer (id: 3242)
         ISP: T-Mobile Thuis
Idle Latency:     4.72 ms   (jitter: 0.25ms, low: 4.46ms, high: 5.11ms)
    Download:   275.14 Mbps (data used: 311.8 MB)                                                   
                  4.93 ms   (jitter: 0.49ms, low: 3.92ms, high: 18.98ms)
      Upload:   389.91 Mbps (data used: 502.5 MB)                                                   
                  4.89 ms   (jitter: 7.02ms, low: 3.87ms, high: 222.81ms)
 Packet Loss:     0.6%
Re-running that test gives me better results (from another remote server)

I don’t feel however that this will give you much: your observation that there is a route flapping in your path indeed could be related. When the route (in the Odido network) flaps seems to be the point where packet loss occurs (the next hop after the first flapping router seems to show some packet loss when that happens. @Tommie van Odido: please ask the network guys to have a look at that. It may however be related to the second issue below when a DOS attack occurs?

The router going down when you are DOS’ed is another issue however.  The Zyxel should not be affected by it. You could try to disable the DOS protection under “beveiliging → firewall to see if that helps. It may be a case where the firewal is to pro-active and reporting on all threats (even on closed ports!) takes so much resources that the router hangs or reboots. @Tommie van Odido you may want to check with network people if this needs to be reported to Zyxel

The DOS attack may be triggered on the IP-address as soon as you get identified and you IP-address gets found out, so you cannot rule out that the attack is directed to you.

I may be wrong, but to me it all looks like being related to a DOS attack on you.

 

 

 

That is a good hunch, however I have now had multiple IP addresses, and when I connected with my mikrotik this wasn’t an issue. So between that and the logs showing a scan happening, I think this is automated, and going after certain devices on a block of IP addresses. Turning off the DoS protection made no difference. 

I have been told that this type of attack shouldn’t be an issue in any linux based devices as they patched the kernel for it, it should just drop the packets. 

Reputatie 6
Badge

@ChaosMonkey, could you run the tests with only the Zyxel? 

We do not have a lot of customers using Linux haha. Therefore we do not have a Linus manual. 

hi @ChaosMonkey : for Linux these test are not so dififcult  if you know how to use the command line.

Download the appropiate CLI client from https://www.speedtest.net/nl/apps/cli install it and run the test. I don’t know if they have a CLI client for non x86_64 hardware. When I run this client on my box I get something like:

[louis@lair ~]$ speedtest

   Speedtest by Ookla

      Server: ColoCenter bv - Zoetermeer (id: 3242)
         ISP: T-Mobile Thuis
Idle Latency:     4.72 ms   (jitter: 0.25ms, low: 4.46ms, high: 5.11ms)
    Download:   275.14 Mbps (data used: 311.8 MB)                                                   
                  4.93 ms   (jitter: 0.49ms, low: 3.92ms, high: 18.98ms)
      Upload:   389.91 Mbps (data used: 502.5 MB)                                                   
                  4.89 ms   (jitter: 7.02ms, low: 3.87ms, high: 222.81ms)
 Packet Loss:     0.6%
Re-running that test gives me better results (from another remote server)

I don’t feel however that this will give you much: your observation that there is a route flapping in your path indeed could be related. When the route (in the Odido network) flaps seems to be the point where packet loss occurs (the next hop after the first flapping router seems to show some packet loss when that happens. @Tommie van Odido: please ask the network guys to have a look at that. It may however be related to the second issue below when a DOS attack occurs?

The router going down when you are DOS’ed is another issue however.  The Zyxel should not be affected by it. You could try to disable the DOS protection under “beveiliging → firewall to see if that helps. It may be a case where the firewal is to pro-active and reporting on all threats (even on closed ports!) takes so much resources that the router hangs or reboots. @Tommie van Odido you may want to check with network people if this needs to be reported to Zyxel

The DOS attack may be triggered on the IP-address as soon as you get identified and you IP-address gets found out, so you cannot rule out that the attack is directed to you.

I may be wrong, but to me it all looks like being related to a DOS attack on you.

 

 

 

That is a good hunch, however I have now had multiple IP addresses, and when I connected with my mikrotik this wasn’t an issue. So between that and the logs showing a scan happening, I think this is automated, and going after certain devices on a block of IP addresses. Turning off the DoS protection made no difference. 

I have been told that this type of attack shouldn’t be an issue in any linux based devices as they patched the kernel for it, it should just drop the packets. 

I guess that the Zyxel (as most routers like your Mikrotik) is Linux based, hence my  hunch that turning DOS protection off might stop it from trying to analyse incoming traffic. Too bad that it did not work out. The fact that the Mikrotik is not affected (at least not as much, the route flapping is still there) as the Zyxel is indeed important. One more thing to try might be to turn UPNP off (hamburger menu → thuisnewerken → UPNP). This will probably break some games. UPNP might still allow something in your network out on some ports, including port 8000.  Check if  UPNP is enabled on the Mikrotik ip/upnp/print or the ip → upnp menu on Webfig or Winbox). To the best of my knowledge it is by default off

I still can not exlude the possibility that something you do invites this DOS attack (but the Zyxel should not be affected by it!!!), nor can I exclude the IP-scanning theory… 

I hope that @Tommie van Odido can find some help from the Odido network guys

Reputatie 1

@ChaosMonkey, could you run the tests with only the Zyxel? 

We do not have a lot of customers using Linux haha. Therefore we do not have a Linus manual. 

hi @ChaosMonkey : for Linux these test are not so dififcult  if you know how to use the command line.

Download the appropiate CLI client from https://www.speedtest.net/nl/apps/cli install it and run the test. I don’t know if they have a CLI client for non x86_64 hardware. When I run this client on my box I get something like:

[louis@lair ~]$ speedtest

   Speedtest by Ookla

      Server: ColoCenter bv - Zoetermeer (id: 3242)
         ISP: T-Mobile Thuis
Idle Latency:     4.72 ms   (jitter: 0.25ms, low: 4.46ms, high: 5.11ms)
    Download:   275.14 Mbps (data used: 311.8 MB)                                                   
                  4.93 ms   (jitter: 0.49ms, low: 3.92ms, high: 18.98ms)
      Upload:   389.91 Mbps (data used: 502.5 MB)                                                   
                  4.89 ms   (jitter: 7.02ms, low: 3.87ms, high: 222.81ms)
 Packet Loss:     0.6%
Re-running that test gives me better results (from another remote server)

I don’t feel however that this will give you much: your observation that there is a route flapping in your path indeed could be related. When the route (in the Odido network) flaps seems to be the point where packet loss occurs (the next hop after the first flapping router seems to show some packet loss when that happens. @Tommie van Odido: please ask the network guys to have a look at that. It may however be related to the second issue below when a DOS attack occurs?

The router going down when you are DOS’ed is another issue however.  The Zyxel should not be affected by it. You could try to disable the DOS protection under “beveiliging → firewall to see if that helps. It may be a case where the firewal is to pro-active and reporting on all threats (even on closed ports!) takes so much resources that the router hangs or reboots. @Tommie van Odido you may want to check with network people if this needs to be reported to Zyxel

The DOS attack may be triggered on the IP-address as soon as you get identified and you IP-address gets found out, so you cannot rule out that the attack is directed to you.

I may be wrong, but to me it all looks like being related to a DOS attack on you.

 

 

 

That is a good hunch, however I have now had multiple IP addresses, and when I connected with my mikrotik this wasn’t an issue. So between that and the logs showing a scan happening, I think this is automated, and going after certain devices on a block of IP addresses. Turning off the DoS protection made no difference. 

I have been told that this type of attack shouldn’t be an issue in any linux based devices as they patched the kernel for it, it should just drop the packets. 

I guess that the Zyxel (as most routers like your Mikrotik) is Linux based, hence my  hunch that turning DOS protection off might stop it from trying to analyse incoming traffic. Too bad that it did not work out. The fact that the Mikrotik is not affected (at least not as much, the route flapping is still there) as the Zyxel is indeed important. One more thing to try might be to turn UPNP off (hamburger menu → thuisnewerken → UPNP). This will probably break some games. UPNP might still allow something in your network out on some ports, including port 8000.  Check if  UPNP is enabled on the Mikrotik ip/upnp/print or the ip → upnp menu on Webfig or Winbox). To the best of my knowledge it is by default off

I still can not exlude the possibility that something you do invites this DOS attack (but the Zyxel should not be affected by it!!!), nor can I exclude the IP-scanning theory… 

I hope that @Tommie van Odido can find some help from the Odido network guys

I can test the UPnP on the Zyxel, I do actually have it on the mikrotik. I had a look at the games I play and naturally I can see it on the mikrotik. Nothing around port 8000, I did list the ports that do get used earlier. 
Also have a look what my mikrotik is picking up  connection-state:new src-mac 00:0e:00:00:00:04, proto TCP (SYN), 162.142.125.138:33690->5.132.1xx.xx:57223, len 44

I see it going for random ports , port 23, then 81, 89, 3393, 10258, these look like common open ports. 

Reageer