website statistics

Today at 9:47 UTC CloudFlare went down due to a wrong protocol rule in all their edge router, made by Juniper. Which took down all of their client's site. Here we have detail story.
1 Star2 Stars3 Stars4 Stars5 Stars (Rate This)
Loading...

Popular DNS service provider CouldFlare went down today due to an issue with Flowspec protocol rule update in their edge Juniper router. Which took their whole system down for about an hour and almost all of their 785,000 customer (Including 4chan, Wikileaks, Metallica.com & our TheTechJournal) went down with it. CloudFlare tried to act quick, now everything is operational. CloudFlare CEO Matthew Prince explained the whole thing, keep reading to understand more about the issue.


CloudFlare

What Happened With CloudFlare Today?

CloudFlare is not only a DNS service, but it also provide DNS level security, caching service, which helps website to fight with spam to DDoS attack, as well as serving content faster. Here at TTJ we use it as DNS service in some lever of our cloud system. So we are entirely depending on them. But the whole CloudFlare system went down today at around 9:47 UTC (1:47 AM in California). And all of the site its serving went down with it. The number of website use their service in some level is quite high. About 785,000 websites went down use to this issue. Due to the network-wide router fails in their edge network all of their 23 data center around the world was disconnected from internet. Some of the router came back online within few min but as all network traffic were directed to those available few, they also went down immediately after reboot. CloudFlare operation team tried to act fast and as their 23 data center is distributed in 14 countries it took them more 30 minute to get some one in all data center to manually hard reboot all router. Then they figured out what went wrong and which rule caused the issue and reverted the rule change. Within an hour whole network started to act in full power.

How All This Happened?

According to CloudFlare CEO Matthew Prince, it all started from a DDoS attack on one of their client’s site. They have a automated attack profiling system, which detected a very unusual size of data packets of between 99,971 and 99,985 bytes long, but the largest packets sent across the Internet are typically in the 1,500-byte range. And they had a limit of maximum packet size set to 4,470 on their entire network. So when the profiler distinguish the attack patter, their operation team wrote a rule like this below
CloudFlare Rule

Flowspec accepted the rule and relayed it to their edge network, and caused all Juniper router consume all their RAM until they crashed.

Overall:

All site seems running smoothly now. Most of the site came back online within an hour. As far in our experience we seen a downtime of around 15 minute for TheTechJournal. CloudFlare promised to look more deep into this matter so they could avoid this in future. They will be giving credit back to all their paid customer.

cloudflare_outage

Now it all seems good, and they way CloudFlare reacted is nice. But still I personally feel DNS is a very important for internet, it should be more redundant. Most of the big names relies on service like CloudFlare, maybe we should have some sort of backup or DNS lever load balancing to avoid this type of incident.

Buy Cheapest Related Product From Amazon.com


Texas Declares War On Drones

Rumor Says T-Mobile May Move To No-Contact Model
You can also press the left/right arrow key on your keyboard to go to previous/next post
  On March 4, 2013(3 years, 6 months ago.)

You May Also Like:

What Do You Think?

1 Response

  1. Jim Says:

    It’s physically impossible to have IPv4 packets between 99,971 and 99,985 bytes, in fact anything over the 16-bit word limit of 65535 bytes is impossible. Their software that generates FlowSpec rules malfunctioned and they’re blaming the vendor.

    Posted on March 5th, 2013 at 3:31 AM

Leave a Reply




Loading Facebook Comments ...

FTC Disclosure: Some of the links of this website are "affiliate links." This means if you click on the link and purchase the item, we will receive an affiliate commission.


Recent Search

Recent Tutorials

Check out this tutorial to know how to install Apple watchOS 3 beta certificate on your Apple Watch and start enjoying the new version.
If you are trying to jailbreak iPhone, iPad or iPod on iOS 9.2 - 9.3.3 without using a computer or Apple ID, then check this video tutorial.
Pokemon Go users are complaining about the crashing and server issues. Check out the tutorial to solve error problems and thanks us later.
Turning off Wi-Fi Assist is a great way to save mobile data since it automatically starts using cellular data when Wi-Fi signal is poor .
If you want to secure your SIM card from others using it, then check out this tutorial to know how to set up the SIM Pin code on your iPhone.
CiderTV is a great alternative to control Apple TV from the Notification Center. Check out this tutorial to set up CiderTV on your iPhone.
Are you annoyed by the split screen mode on the iPhone 6 Plus or 6s Plus? Check out this quick tutorial to turn off split screen feature.
If you could not wait to installed the iOS 10 beta version on you iPhone and now struggling for the errors, then this tutorial is for you.
Siri might not understand the question you asked. But you can use Siri by editing the text that you asked & it will give an updated answer.
Perhaps, you are new Apple user and you might have no idea how to change name of your iPhone. Check out this tutorial to change the name.
Close You Have To Login
User:
Pass:
Login With »Login With TwitterLogin With Facebook