1、nutch2.3-snapshot中batchid为null引发的.NullPointerException异常:
Exception in thread "main" java.lang.NullPointerException at org.apache.nutch.parse.ParserJob.getBatchIdFilter(ParserJob.java:265) at org.apache.nutch.parse.ParserJob.run(ParserJob.java:253) at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:69) at org.apache.nutch.crawl.Crawler.run(Crawler.java:174) at org.apache.nutch.crawl.Crawler.run(Crawler.java:253) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.nutch.crawl.Crawler.main(Crawler.java:260)
这里有点没搞懂,batchid为null是没有fetch的url,怎么会出现在parserjob中,暂时不管,我们直接忽略掉为null的batchid即可,在ParserJob.java的getBatchIdFilter函数中增加代码遇到null即return即可,如下所示:
private MapFieldValueFilter<String, WebPage> getBatchIdFilter(String batchId) { if (batchId==null||batchId.equals(REPARSE.toString()) || batchId.equals(Nutch.ALL_CRAWL_ID.toString())) { return null; }
时间: 2024-11-06 08:25:46