What the ABS can learn about online humility
Last night's #Censusfail debacle provided many lessons for organisations in how to manage large scale online events, says Paul Wallbank, not the least of which is to never promise what you can't deliver.
There’s a reason why airlines don’t make marketing claims about safety and that’s a lesson the Australian Bureau of Statistics has probably learned after last night’s Census embarrassment.
Over the past two months the ABS had been fighting widespread claims the Census data being retaining created security concerns and moving the nationwide survey online risked an ‘internet meltdown’ on the night.
In response, the agency and the Federal Government ran aggressive and at times, haughty, communications campaigns dismissing privacy concerns through both traditional and social media services.
As it turned out, reliability of the site on the night turned out to be the big communications challenge for the ABS because it turned out it wasn’t particularly well prepared as hubris and overconfidence took over from reality and caution.
The ABS told Fairfax last week it was confident there would not be an internet meltdown. When asked about the ability of the online Census database to cope with such high traffic numbers, an ABS spokesman said online could handle “1,000,000 form submissions every hour. That’s twice the capacity we expect to need.”
Catering for a million submissions an hour capacity wasn’t enough and the ABS’ communications team was caught unprepared with its responses as people complained on Twitter about the website being unavailable, and for responses being a template “online form and website are operating smoothly as expected. Please try again.”
Adding insult to injury was the ABS’ Twitter account auto-tweeting responses which encouraged people to a site that was clearly down. The auto-response was mercifully put down late in the evening after the agency admitted defeat and stated the site wouldn’t be back online that night.
This morning the chief statistician was blaming a denial of service attack, as he told ABC Radio’s AM program, “at this stage the information that we’ve received is that it came from an international source but we don’t know any more… The Australian Signals Directorate is investigating these issues.”
Sadly, that response isn’t going to help David Kalisch’s credibility as the ‘denial of service attack’ is the cyber equivalent of ‘the dog ate my homework’ with such an attack on a high-profile website like the Census almost a given in today’s online world.
To compound the problems of Mr Kalisch and his organisation, referring to denial of service attacks as ‘hacking’ is only going to further irritate the technologists and security professionals whose concerns about the census have been fobbed off by the ABS and government for months.
It’s hard also not to get the feeling the ABS was genuinely unprepared for things going wrong. As one seasoned public relations and corporation communications veteran said late last night, “feeling for the PR team, which will be pushed no doubt to the front of the telephone queue, having spent a number of years saying something along the lines of “what’s the plan if it doesn’t work?”
No doubt there were some inside the ABS’s communications team who asked that question, and it seems likely they were fobbed off with the same haughtiness with which the organisation’s management dismissed its external critics.
For the communications folk looking at the ruins of the ABS’s credibility this morning, the lessons are clear – social media auto-responders are necessary when there’s expected to be a high level of traffic but need to be turned off at the first sign of trouble. Of course that assumes the comms people will be told by operations when there is a problem.
Equally, all hands in the communications team need to be on deck and briefed fully about an event’s status. If websites are down, queues are long or call centres collapsing under the weight of traffic, then that has to be allowed for in the live updates.
The big lesson, though, is about hubris – both the ABS’s management and the government dismissed its critics and other industry experts who raised valid concerns about the reliability of the service. Today both look foolish and those security claims will now be subject of even more scrutiny.
Probably the greatest lesson from last night’s #Censusfail is to engage with the community earlier and not be so dammed arrogant to publicly dismiss the views of informed critics. Sadly, though, arrogant management will always trump the best efforts of any good PR team or communications strategy.
Paul Wallbank is a freelance writer and the publisher of Networked Globe
The big questions:
Was there really a DDos attack or is that just arse covering?
If there was, is the data entered before the attack now out in the wild?
User ID not verified.
This is so badly written – do you have an editor or did they fail last night as well?
User ID not verified.
An even bigger problem was the failure of ABS this morning to describe what was happening.
No-one was available to comment on breakfast TV so the networks just ran with their own commentators, most of whom spoke of “hacking” and idly speculated that “maybe the scare campaign was right” with no alternative perspective. This just reinforced the community concerns about security of information and will probably compound non-participation once the site is back up and running.
Surely ABS should have had a team of co-ordinated spokespeople hitting every media outlet from 5am, not just saying “We’ll give a press conference at 9.”
User ID not verified.
Why do bureaucrats (and politicians, for that matter) never learn that they should NOT dabble or meddle in areas requiring expertise that is way beyond their own experience and qualifications? Major fails like this one just push their credibility through the floor.
User ID not verified.
Unfortunately, they’ve been getting lessons in government from Donald Trump.
What do they care? Just fine people if they don’t complete them (which they have broadcasted by loudspeaker, although now I notice they are saying no fines due to their website being down, so I’ll give them that), punish wilfully, no need to learn, you’ve got the power, give me your details about your life, we’ll keep them, all ours, gotta give it to us etc etc etc.
Learn? Who’s going to make them!
User ID not verified.
DDos attacks are just greatly increasing the amount of queries to load the website, overwhelming the server hosting the website with traffic, causing the website to crash. Doesn’t steal data, I believe (not an IT expert, could be wrong).
User ID not verified.
#DoItAgainOnAugust10
User ID not verified.
Come on anonymous. A DDOS attack isn’t a compromised site.
It literally refers to an attack that prevents a site from talking to genuine users by flooding it with fake requests. It’s like a thousand people standing in the bank and yelling at the teller to stop them working. it’s loud and nothing gets done, but the vault is entirely secure.
User ID not verified.
How would data be exposed by a DDoS attack? Sounds like you have no idea what a DDoS attack is…
User ID not verified.
@AJ – A DDoS doesn’t even need to crash the server to take out the website. It might just saturate one or more internet links which lead to the server, thereby taking the site offline either for everyone or a subset of visitors.
User ID not verified.
The bureaucrats didn’t dabble in it, they paid IBM $9m+ create this catastrophe.
User ID not verified.
If we assume there really was an overseas DDoS attack (and there is DDoS attack traffic monitoring evidence on Twitter to suggest there wasn’t), considering this is a survey on Australian households I don’t understand why they didn’t scrub overseas traffic before it even hit Australia. That seems like a no-brainer.
Perhaps I should tender for the $9m+ contract next time!
User ID not verified.
It really IS an exciting time to be an Australian.
User ID not verified.
Well… that escalated quickly ?
User ID not verified.
Visnja,
badly written?? You response displays a striking similarity to the failed, arrogant traits that Paul has alluded to. You need to go to Specsavers…. on your way to your job at ABS HQ.
User ID not verified.
They claim that they thought they could handle “1,000,000 form submissions every hour. That’s twice the capacity we expect to need.” says it all. Some pretty basic analysis suggests maybe 7 million households on the East coast would try to complete between 7pm and 9pm – that’s an average load of over three times their estimated load, and it would be likely to spike to at least twice that again through normal traffic fluctuations – and any competent web traffic analyst should have been able to tell them that.
The DDOS explanation sounds dodgy to me – The normal load you would have expected would have been 6-10 times what they planned for.
User ID not verified.
BBC’s digitisation project went wrong. Boss got fired.
BA’s move into Heathrow Terminal 5 went wrong. Boss got fired.
Abs #censusfail…
You know that old phrase nobody can get fired for using IBM?
It’s going to be interesting to find out if that’s true.
User ID not verified.
Thanks for the insights on dos attacks, I based my second question on a quote from the ABS Chief:
“But the site was shut down after a “gap” in the system’s security measures was found during a fourth attack, Mr Kalisch said.”
Perhaps the teller left the key in the outside of the door when she unlocked the bank?
User ID not verified.
Denial of service (DDOS) attacks have been known to be used as a way of masking other activities. While IT staff are focused on scrambling to keep their service safe and reliable from one side, the real intention is going on where they are not focused.
Although I doubt a large offshore DDOS attack was the culprit as statistics for the period show low level activity towards the domain from anywhere outside Australian IP ranges, there was definitely some sort site penetration that led to whole service being taken offline by the ABS and IBM manually, at very short notice.
User ID not verified.
This is one of the better articles on the census fail i have read.
User ID not verified.
It’s called the Dunning-Kruger effect. Look for it on your favourite Wikipedia site.
Essentially, it’s when you don’t know enough to even realise that you’re out of your depth.
AKA “talking out of the wrong orifice”
User ID not verified.
I’m amused by the many ‘official’ statements that the DDos “didn’t succeed” (assuming it really *was* a DDoS, not merely an overloaded server struggling to cope with a load far greater than it was designed for).
Yes, it DID SUCCEED. A DDoS is intended to do nothing more than stop people accessing a web site. Due to the combined efforts of the overload AND the on-site staff, people were unable to reach the census web site.
Clearly, the DDoS succeeded.
Whether there were other factors at play (such as a misdirection) we don’t yet know. But I remain very nervous about the statements that no data was taken… seems very ‘Dunning Kruger” to me (see comment above).
User ID not verified.
Chris, I agree completely. They pretty quickly added to the story that Defence Signals Directorate weren’t able to determine the source of these four attacks. This story started to change as the morning went on, though. I can imagine that DSD weren’t very happy about being made to look incompetent. There would’ve been some hastily arranged phone conversations between those two organisations.
User ID not verified.
How is data compromised ?
Because a DDOS attack overwhelms the system and allows a routine to create a way into the system while it tries to cope with the attack.
This is in fact the purpose of a DDOS attack.
The system didn’t break down from the attack though, it was closed down in response to a large volume of packets from an apparently similar source. It’s quite likely that the packets where legitimate and simply routed through a foreign server when the local ones became saturated.
IBM’s software is junk in other words.
User ID not verified.
The problem is it’s written by just another blogger who thinks he knows better.
The Census failed we get it. Government stuffs up; what else is new? The megaphone in an echo chamber giving us just another headache.
User ID not verified.