自然语言19_Lemmatisation

https://en.wikipedia.org/wiki/Lemmatisation

Lemmatisation (or lemmatization) in linguistics is the process of grouping together the inflected forms of a word so they can be analysed as a single item, identified by the word‘s lemma, or dictionary form.[1]

In computational linguistics, lemmatisation is the algorithmic process of determining the lemma of a word based on its intended meaning. Unlike stemming, lemmatisation depends on correctly identifying the intended part of speech and meaning of a word in a sentence, as well as within the larger context surrounding that sentence, such as neighboring sentences or even an entire document. As a result, developing efficient lemmatisation algorithms is an open area of research.[2][3]

Contents

Description

In many languages, words appear in several inflected forms. For example, in English, the verb ‘to walk‘ may appear as ‘walk‘, ‘walked‘, ‘walks‘, ‘walking‘. The base form, ‘walk‘, that one might look up in a dictionary, is called the lemma for the word. The association of the base form with a part of speech is often called a lexeme of the word.

Lemmatisation is closely related to stemming. The difference is that a stemmer operates on a single word without knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech. However, stemmers are typically easier to implement and run faster. The reduced "accuracy" may not matter for some applications. In fact, when used within information retrieval systems, stemming improves query recall accuracy, or true positive rate, when compared to lemmatisation. Nonetheless, stemming reduces precision, or true negative rate, for such systems.[4]

For instance:

  1. The word "better" has "good" as its lemma. This link is missed by stemming, as it requires a dictionary look-up.
  2. The word "walk" is the base form for word "walking", and hence this is matched in both stemming and lemmatisation.
  3. The word "meeting" can be either the base form of a noun or a form of a verb ("to meet") depending on the context; e.g., "in our last meeting" or "We are meeting again tomorrow". Unlike stemming, lemmatisation attempts to select the correct lemma depending on the context.

Document indexing software like Lucene[5] can store the base stemmed format of the word without the knowledge of meaning, but only considering word formation grammar rules. The stemmed word itself might not be a valid word: ‘lazy‘, as seen in the example below, is stemmed by many stemmers to ‘lazi‘. This is because the purpose of stemming is not to produce the appropriate lemma – that is a more challenging task that requires knowledge of context. The main purpose of stemming is to map different forms of a word to a single form.[6] As a rules-based algorithm, dependent only upon the spelling of a word, it sacrifices accuracy to ensure that, for example, when ‘laziness‘ is stemmed to ‘lazi‘, it has the same stem as ‘lazy‘.

Use in biomedicine

Morphological analysis of published biomedical literature can yield useful results. Morphological processing of biomedical text can be more effective by a specialised lemmatisation program for biomedicine, and may improve the accuracy of practical information extraction tasks.[7]

时间: 2024-09-27 03:02:58

自然语言19_Lemmatisation的相关文章

深度学习与自然语言处理之四:卷积神经网络模型(CNN)

/* 版权声明:可以任意转载,转载时请标明文章原始出处和作者信息 .*/ author: 张俊林 大纲如下: 1.CNN基础模型 2.单CNN模型的改进    2.1对输入层的改进    2.2Convolution层的改进    2.3Sub-Sampling层的改进    2.4全连接层的改进 3.多CNN模型的改进 4.探讨与思考 扫一扫关注微信号:"布洛卡区" ,深度学习在自然语言处理等智能应用的技术研讨与科普公众号.

Python自然语言处理工具小结

Python自然语言处理工具小结 作者:白宁超 2016年11月21日21:45:26 1 Python 的几个自然语言处理工具 NLTK:NLTK 在用 Python 处理自然语言的工具中处于领先的地位.它提供了 WordNet 这种方便处理词汇资源的借口,还有分类.分词.除茎.标注.语法分析.语义推理等类库. Pattern:Pattern 的自然语言处理工具有词性标注工具(Part-Of-Speech Tagger),N元搜索(n-gram search),情感分析(sentiment a

大数据文本分析:灵玖自然语言中文语义分词系统

自然语言通常是指一种自然地随文化演化的语言.英语.汉语.日语为自然语言的例子,而世界语则为人造语言,即是一种为某些特定目的而创造的语言. 自然语言具备两个属性:语言属性与自然属性."语言"属性表现为公认的某些约定俗成的内在规律性;"自然"属性是说并不存在某个人为制造的.严格的语法规则体系来约定人们的语言表达方式,这是和程序设计语言大相径庭的.自然语言需要遵循一定的内在规律,但更大程度上是"存在即合理". 一个自然语言处理系统必须考虑许多语言自身与

【行业聚焦】畅捷通用自然语言技术颠覆企业应用人机交互模式!

2016年7月30日,由崔牛会主办的主题为"小技术·大场景"的2016中国企业互联网技术论坛圆满落幕.活动现场20多位顶级技术专家,近千名行业技术大咖一起交流学习,商讨技术对未来商业模式的影响 ,洞察TO B领域的趋势和变革.畅捷通深度参与了此次大会,技术委员会主席李鲲带来了主题分享:自然语言处理在企业应用领域的实践,与行业专家一起深度讨论了企业SaaS未来的发展趋势. 颠覆企业级人机交互自然语言技术 自然语言处理技术正在日常生活中迅速普及,并给大家带来了很多方便.畅捷通经过三年多的技

深度学习与自然语言处理(2)_斯坦福cs224d Lecture 2

原文作者:Rohit Mundra, Richard Socher 原文翻译:@熊杰([email protected]) && @王昱森 内容调整与校对:寒小阳 && 龙心尘 特别鸣谢:@面包包包包包同学的帮助 时间:2016年6月 出处:http://blog.csdn.net/han_xiaoyang/article/details/51648483 http://blog.csdn.net/longxinchen_ml/article/details/5164853

Python自然语言工具包(NLTK)入门

在本期文章中,小生向您介绍了自然语言工具包(Natural Language Toolkit),它是一个将学术语言技术应用于文本数据集的 Python 库.称为“文本处理”的程序设计是其基本功能:更深入的是专门用于研究自然语言的语法以及语义分析的能力. 鄙人并非见多识广, 语言处理(linguistic processing) 是一个相对新奇的领域.如果在对意义非凡的自然语言工具包(NLTK)的说明中出现了错误,请您谅解.NLTK 是使用 Python 教学以及实践计算语言学的极好工具.此外,计

自然语言处理怎么最快入门?

http://www.zhihu.com/question/19895141 stanford NLP课程网址 https://class.coursera.org/nlp/lecture 14 个回答 赞同337反对,不会显示你的姓名 吴俣,自然语言处理民工 杨泽.林斌.王汪汪 等人赞同 推荐<数学之美>,这个书写得特别科普且生动形象,我相信你不会觉得枯燥.这个我极力推荐,我相信科研的真正原因是因为兴趣,而不是因为功利的一些东西. 接下来说,<统计自然语言处理基础>这本书,这书实

python自然语言处理1——从网络抓取数据

python自然语言处理1--从网络抓取数据 写在前面 本节学习python2.7 BeautifulSoup库从网络抽取数据的技术,检验之简而言之就是爬虫技术.网络编程是一门复杂的技术,在需要基础的地方,文中给出的链接地址,都是很好的教程,可以参考,我在这里不在重复发明轮子.本节的主旨在于: 帮助快速掌握基本爬虫技术,形成一条主线,能为自己的实验构造基础数据.掌握爬虫技术后,可以从网络抓取符合特定需求的数据供分析,这里学习的爬虫技术适用于数据挖掘.自然语言处理等需要从外部挖掘数据的学科. 1.

自然语言处理(3)之条件频率分布

自然语言处理(3)之条件频率分布 条件频率分布式频率分布的集合,每个频率分布有一个不同的条件. 从下面的例子就可以看出,cfd就是两个条件(news,romance)的频率分布集合 1 >>> cfd=nltk.ConditionalFreqDist( 2 ... (genre,word) 3 ... for genre in ['news','romance'] 4 ... for word in brown.words(categories=genre)) 5 >>>