There are good bots and there are bad bots. Should somebody tell you that the percentage of bad bots is on the decline, they won’t be lying. The fact is, though, that the number of good bots filling cyberspace are on a rocket-like trajectory. So yes, bad bots account for a smaller share of traffic on the Web, but there are still lots more of them this year than than were last year.
Many more interesting bits of robot reality emerge from the “2015 Bot Landscape Report” published by Distil Networks, which makes its money isolating the good from the bad bots for its blue-chip publisher and advertiser clientele. First a definition: A good bot is one that adds value to a publisher or advertiser and abides by the robot.txt industry opt-in standard. A bad bot is just a bad guy masquerading as a person and scraping data, content, and eventually cash from Internet players. An ad targeting tool is a good bot; fake publishers loading up a hundred screens behind that of an actual publisher and stealing ads is a bad bot.
Distil Networks’ analysis, which covers hundreds of customers,17 data centers, and some 23 billion bad bot threats finds that good bots nearly doubled their Web presence over the past year. Bad bots’ share of traffic dipped a single point to 23%, but good bots added 15 points to rise to 36% of traffic. The sheer magnitude of this shift is best borne out by the fact that most of that share transfer came from human traffic, which declined from 55% to 41%.
“The reason that good bots increased so much is that social medial companies have doubled the number of bots they used in the past year,” says Distil Networks CEO Rami Essaid. “Google’s rate of crawling went up 30% in the past two weeks because of the change to its algorithm.”
As for bad bots, more of them are working their way into the Web through cable and wireless networks. Comcast and Time Warner Cable are both among the Top 10 companies serving bad bot traffic, and together account for more than 11% of it. T-Mobile became the first wireless provider to crack the Top 20 at 1.22%. “We’ve been noticing a trend over the past three years where people are putting one and one together that, if it’s coming from a data center, it might be fraudulent,” Essaid says. “So, the bad guys are reacting and that’s why we’re seeing more bad bots from cable networks. On wireless, they’re pretending to be mobile browsers.”
Amazon replaced Verizon Business as the leading disseminator of bad bots in 2014. Its Amazon Technologies unit was number one at 15% and Amazon.com finished second with more than 3%. Bad bots made up 78% of the traffic on Amazon’s cloud business and 70% on the e-commerce site. But Essaid says it’s to be expected. “Amazon has done an amazing job of getting servers and leveraging them for what they need,” he observes. “Bad bots are part of their ecosystem. A lot of different people and companies interact with it. They hit a share button and call out to something else. It’s so easy to spin up servers, so bad guys use Amazon to spin up bad bots and do what they want to do.”
No matter how much cover the good bots may provide them, bad bots are in for the long haul. Networks that erect no barriers at all to bots are easy pickings. In a test with the Online Trust Alliance, Distil Networks sent out bots identified as being bad bots and still gained entry to half of networks, so even unsophisticated hackers have easy entree to the Web. The top-level bot syndicates that use real Web browsers and mimic people—even going so far as to move around computer mouses—can break into even the most guarded networks. “They account for a quarter of the bot traffic,” Essaid says.
