A crawler that sent a DELETE request to every resource it encountered

RESTful Web APIs_2013

The crawler simulates a very curious but not very picky human. Give it a URL to startwith, and it will fetch a representation. Then it will follow all the links it can find to getmore representations. It will do this recursively, until there are no more representations to be had.The Mapmaker client from earlier in this chapter is a kind of crawler for Maze+XML documents. The spiders used by search engines are crawlers for HTML documents.It's quite difficult to write a crawler for an API that doesn't use hypermedia. But you can write a crawler for a hypermedia-based API without even understanding that API's linkrelations.Generally speaking, a crawler will only trigger state transitions that are safe. Otherwise,there's  no  telling  what  will  happen  to  resource  state.  A  crawler  that  sent  a  DELETE request to every resource it encountered, just to see what happened, would be a terrible client.