采集器的前世今生

爬出和反爬出是矛与盾的关系,

进化史

1,java原生自带url类:url.getContent();

2,httpclient,

3,jsoup,htmlclean.

4,htmlunit,

5,se.

1-2只是原生http链接,

3,做了解析层面的支持,比如页面html清理,xpath支持;

4添加了js支持,这是很大的改进,不过由于各种衍生js库,支持不是很好.

5,se,这就是走了捷径.

好的反爬出机制,能识别在电脑另一头的是虫子还是人

,eg.阿里系网站机制 anti spider.:供应商,搜索食品,

http://s.1688.com/company/company_search.htm?keywords=%CA%B3%C6%B7&sug=2_0&n=y&spm=a260k.635.1998096057.d1

短时间内连续翻页超过n页,n属于(8,20),自动让你登陆.

登陆后,继续翻超过n页,触发阿里反爬虫机制:http://sec.1688.com/query.htm?smApp=searchweb2&smPolicy=searchweb2-company-anti_Spider-html-checkcode&smCharset=GBK&smTag=MTIxLjIwNC4xOTUuMTkzLDE5MzIzNDI3NSxjNjEzZDJhOTQyMTM0ZjJhOTg4OTM5OWMyMTdlY2MyZg%3D%3D&smReturn=http%3A%2F%2Fs.1688.com%2Fcompany%2Fcompany_search.htm%3Fkeywords%3D%25CA%25B3%25C6%25B7%26button_click%3Dtop%26earseDirect%3Dfalse%26n%3Dy%26sortType%3Dpop%26pageSize%3D30%26beginPage%3D15&smSign=qxdnSrcI%2B5VTtjVJumzY7w%3D%3D

其实,阿里系也是有漏洞的.