News Corp has for what appears to be the first time prevented a search engine from crawling a publicly accessible site in a move that was threatened by boss Rupert Murdoch late last year.
The company has targeted a specialised search player – the UK-based news aggregation site NewsNow – for the move, which has seen it institute the robots.txt protocol on its British Times Online site. The piece of code tells search engines not to crawl a site. In this case, it specifies that the block is for NewsNow’s web crawler.
The move was revealed in a statement from NewsNow.
In November, Murdoch threatened to use robots.txt to make News Ltd sites invisible to search giants such as Google.
However, as Duncan Riley argues in The Inquisitr, NewsNow makes an easier target than Google.