website statistics

Today at 9:47 UTC CloudFlare went down due to a wrong protocol rule in all their edge router, made by Juniper. Which took down all of their client's site. Here we have detail story.
1 Star2 Stars3 Stars4 Stars5 Stars (Rate This)
Loading...

Popular DNS service provider CouldFlare went down today due to an issue with Flowspec protocol rule update in their edge Juniper router. Which took their whole system down for about an hour and almost all of their 785,000 customer (Including 4chan, Wikileaks, Metallica.com & our TheTechJournal) went down with it. CloudFlare tried to act quick, now everything is operational. CloudFlare CEO Matthew Prince explained the whole thing, keep reading to understand more about the issue.


CloudFlare

What Happened With CloudFlare Today?

CloudFlare is not only a DNS service, but it also provide DNS level security, caching service, which helps website to fight with spam to DDoS attack, as well as serving content faster. Here at TTJ we use it as DNS service in some lever of our cloud system. So we are entirely depending on them. But the whole CloudFlare system went down today at around 9:47 UTC (1:47 AM in California). And all of the site its serving went down with it. The number of website use their service in some level is quite high. About 785,000 websites went down use to this issue. Due to the network-wide router fails in their edge network all of their 23 data center around the world was disconnected from internet. Some of the router came back online within few min but as all network traffic were directed to those available few, they also went down immediately after reboot. CloudFlare operation team tried to act fast and as their 23 data center is distributed in 14 countries it took them more 30 minute to get some one in all data center to manually hard reboot all router. Then they figured out what went wrong and which rule caused the issue and reverted the rule change. Within an hour whole network started to act in full power.

How All This Happened?

According to CloudFlare CEO Matthew Prince, it all started from a DDoS attack on one of their client’s site. They have a automated attack profiling system, which detected a very unusual size of data packets of between 99,971 and 99,985 bytes long, but the largest packets sent across the Internet are typically in the 1,500-byte range. And they had a limit of maximum packet size set to 4,470 on their entire network. So when the profiler distinguish the attack patter, their operation team wrote a rule like this below
CloudFlare Rule

Flowspec accepted the rule and relayed it to their edge network, and caused all Juniper router consume all their RAM until they crashed.

Overall:

All site seems running smoothly now. Most of the site came back online within an hour. As far in our experience we seen a downtime of around 15 minute for TheTechJournal. CloudFlare promised to look more deep into this matter so they could avoid this in future. They will be giving credit back to all their paid customer.

cloudflare_outage

Now it all seems good, and they way CloudFlare reacted is nice. But still I personally feel DNS is a very important for internet, it should be more redundant. Most of the big names relies on service like CloudFlare, maybe we should have some sort of backup or DNS lever load balancing to avoid this type of incident.

Buy Cheapest Related Product From Amazon.com


Texas Declares War On Drones

Rumor Says T-Mobile May Move To No-Contact Model
You can also press the left/right arrow key on your keyboard to go to previous/next post
  On March 4, 2013(3 years, 2 months ago.)

You May Also Like:

What Do You Think?

1 Response

  1. Jim Says:

    It’s physically impossible to have IPv4 packets between 99,971 and 99,985 bytes, in fact anything over the 16-bit word limit of 65535 bytes is impossible. Their software that generates FlowSpec rules malfunctioned and they’re blaming the vendor.

    Posted on March 5th, 2013 at 3:31 AM

Leave a Reply




Loading Facebook Comments ...

FTC Disclosure: Some of the links of this website are "affiliate links." This means if you click on the link and purchase the item, we will receive an affiliate commission.


Recent Search

Recent Tutorials

Even if iPhone is muted, still Siri's voice is loudly chime. Now you can change the setting to silence Siri using your iPhone's mute switch.
Check out how to teach Siri about nicknames and relationships of the important people in your life in order to make your life easier.
Has your iPhone suddenly turned black & white? Check out this tutorial to know possible reasons & how to fix iPhone screen with few taps.
Check out this tutorial to download FREE movies and TV Shows streaming Showbox app and install it on your Android smartphone or tablet.
Parental Controls allows you to put restrictions on apps or content that can or can not be used by anyone else on your Apple iPhone or iPad.
If you want to print from iPhone without using any computer, check this out to connect and print wirelessly right from the Apple device.
Using Siri, anyone can bypass the Passcode even if iPhone or iPad is locked. Check out this to know how you can disable Siri on lock screen.
Floatify will let you quick reply to the messages or emails through notification bar using your Android Lollipop or Marshmallow.
If you do not know to do face swap, check out this tutorial to learn how the new Snapchat face swapping feature actually works.
Facebook has released the Live video streaming feature and we will show you how to go live using the Facebook app for Android or Apple.
Close You Have To Login
User:
Pass:
Login With »Login With TwitterLogin With Facebook