CRF++词性标注
2016-02-28 分类：NLP 阅读(5558) 评论(19)
训练和测试的语料都是人民日报98年标注语料，训练和测试比例是10：1，直接通过CRF++标注词性的准确率:0.933882。特征有一千多万个，训练时间比较长。机器cpu是48核，通过crf++，指定并线数量 -p为40，训练了大概七个小时才结束。
语料库、生成训练数据的python脚本、训练日志、模型、计算准确率脚本都上传到网盘，可以直接下载：戳我下载 CRF++词性标注，程序在centos6.5+python2.7下面运行通过，如果在win下或者ubuntu下可能会有异常，通常都是编码、路径规范等小问题，通过逐行debug脚本应该很容易找到问题，同时要确定crf++在自己机器本身编译没有问题，下面说一下每一步的过程。
文章目录 [展开]
生成训练和测试数据
生成训练和测试数据脚本：get_post_train_test_data.py，执行过程中会打印出来一些调试信息。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
#coding=utf8 import sys #home_dir = "D:/source/NLP/people_daily//" home_dir = "./" def saveDataFile(trainobj,testobj,isTest,word,handle): if isTest: saveTrainFile(testobj,word,handle) else: saveTrainFile(trainobj,word,handle) def saveTrainFile(fiobj,word,handle): if len(word) > 0 and word != "。" and word != "，": fiobj.write(word + ‘\t‘ + handle + ‘\n‘) else: fiobj.write(‘\n‘) def convertTag(): fiobj = open( home_dir + ‘people-daily.txt‘,‘r‘) trainobj = open( home_dir +‘train.data‘,‘w‘ ) testobj = open( home_dir +‘test.data‘,‘w‘) arr = fiobj.readlines() i = 0 for a in sys.stdin: i += 1 a = a.strip(‘\r\n\t ‘) if a=="":continue words = a.split(" ") test = False if i % 10 == 0: test = True for word in words[1:]: print "---->", word word = word.strip(‘\t ‘) if len(word) > 0: i1 = word.find(‘[‘) if i1 >= 0: word = word[i1+1:] i2 = word.find(‘]‘) if i2 > 0: w = word[:i2] word_hand = word.split(‘/‘) print "----",word w,h = word_hand #print w,h if h == ‘nr‘: #ren min #print ‘NR‘,w if w.find(‘·‘) >= 0: tmpArr = w.split(‘·‘) for tmp in tmpArr: saveDataFile(trainobj,testobj,test,tmp,h) continue saveDataFile(trainobj,testobj,test,w,h) saveDataFile(trainobj, testobj, test,"","") trainobj.flush() testobj.flush() if __name__ == ‘__main__‘: convertTag() |
执行训练和测试
设置模板为：
|
1 2 3 4 5 6 7 8 |
# Unigram U00:%x[-2,0] U01:%x[-1,0] U02:%x[0,0] U03:%x[1,0] U04:%x[2,0] U05:%x[-1,0]/%x[0,0] U06:%x[0,0]/%x[1,0] |
训练的时候的-p参数根据自己机器情况设置
|
1 2 |
crf_learn -f 3 -p 4 -c 4.0 template train.data model > train.rst crf_test -m model test.data > test.rst |
计算准确率
通过命令：python clc_f.py test.rst 执行python脚本，clc_f.py中的具体程序：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
#!/usr/bin/python # -*- coding: utf-8 -*- import sys if __name__=="__main__": try: file = open(sys.argv[1], "r") except: print "result file is not specified, or open failed!" sys.exit() wc = 0 wc_of_test = 0 wc_of_gold = 0 wc_of_correct = 0 flag = True for l in file: if l==‘\n‘: continue _, g, r = l.strip().split() if r != g: flag = False wc += 1 if flag: wc_of_correct +=1 flag = True print "WordCount from result:", wc print "WordCount of correct post :", wc_of_correct #准确率 P = wc_of_correct/float(wc) print "准确率:%f" % (P) |