RESTful Web APIs_2013
The crawler simulates a very curious but not very picky human. Give it a URL to startwith, and it will fetch a representation. Then it will follow all the links it can find to getmore representations. It will do this recursively, until there are no more representations to be had.The Mapmaker client from earlier in this chapter is a kind of crawler for Maze+XML documents. The spiders used by search engines are crawlers for HTML documents.It‘s quite difficult to write a crawler for an API that doesn‘t use hypermedia. But you can write a crawler for a hypermedia-based API without even understanding that API‘s linkrelations.Generally speaking, a crawler will only trigger state transitions that are safe. Otherwise,there‘s no telling what will happen to resource state. A crawler that sent a DELETE request to every resource it encountered, just to see what happened, would be a terrible client.