利用BLEU进行机器翻译检测(Python-NLTK-BLEU评分方法)

双语评估替换分数(简称BLEU)是一种对生成语句进行评估的指标。完美匹配的得分为1.0,而完全不匹配则得分为0.0。这种评分标准是为了评估自动机器翻译系统的预测结果而开发的,具备了以下一些优点:

  1. 计算速度快,计算成本低。
  2. 容易理解。
  3. 与具体语言无关。
  4. 已被广泛采用。

BLEU评分是由Kishore Papineni等人在他们2002年的论文BLEU a Method for Automatic Evaluation of Machine Translation中提出的。BLEU计算的原理是计算待评价译文和一个或多个参考译文间的距离。距离是文本间n元相似度的平均,n=1,2,3(更高的值似乎无关紧要)。也就是说,如果待选译文和参考译文的2元(连续词对)或3元相似度较高,那么该译文的得分就较高。

我们是翻译众包业务,对于我们的应用场景,如何得知译员是否有参考机器翻译引擎就成了一个比较重要的问题。我提出的基本思路是:

  1. 在多个翻译网站上翻译原文,得到一组机器翻译评测集,以下的例子中就是一段原文通过百度、有道翻译之后,组织了一个机器翻译评测集
  2. 将译员翻译出来的译文,作为待评测数据,计算其与机器翻译评测集的BLEU值(使用NLTK中提供的BLEU评分方法)
  3. 值越高,表明匹配度越高,则译员参考机器翻译或者直接拷贝机器翻译的可能性就越高,此时需要项目经理介入。

以下是示例:

  1、原文

新译星将代表四达时代集团在展览会上闪亮登场,届时我们将从新译星所开展的业务、具备的优势、成功案例等多个维度进行介绍,让您更加全面的了解新译星。我们拥有稳定的全职国际化团队,能够确保守时、高效的完成翻译和配音,并通过至臻完善的质量控制和项目管理体系进行全方位把控,提供翻译、配音、字幕制作、后期制作、播出以及收视率调查等一条龙服务。

  2、人工翻译

New Transtar will present itself at the Exhibition on behalf of StarTimes, and we will give a comprehensive introduction of ourselves, including the current services we offer, the advantages we hold, and the projects we have completed, to help you understand us more. New Transtar boasts of an international team of professionals and is capable of providing fast and quality-guaranteed services including translating, dubbing, subtitle making, post-production, broadcasting and collecting of viewership ratings, thanks to our strict, streamlined and developed quality control and project management system.

  3、百度翻译

The new translator will stand on the exhibition on behalf of the four times group at the exhibition. We will introduce the new star‘s business, the advantages and the successful cases, so that you can understand the new translator more comprehensively. We have a stable full-time international team that ensures punctual, efficient translation and dubbing, and provides a full range of control through the perfect quality control and project management system, providing a one-stop service for translation, dubbing, subtitle production, post production, broadcasting, and ratings surveys.

  4、有道翻译

The new translator star will represent sida times group in the exhibition, when we will introduce the new translator star‘s business, advantages, successful cases and other dimensions, so that you can have a more comprehensive understanding of the new translator star. We have a stable full-time international team, which can ensure timely and efficient translation and dubbing. Through perfect quality control and project management system, we provide translation, dubbing, subtitle production, post-production, broadcasting and rating survey.

  5、用百度翻译和有道翻译组织机器翻译评测集

[[‘The‘, ‘new‘, ‘translator‘, ‘will‘, ‘stand‘, ‘on‘, ‘the‘, ‘exhibition‘, ‘on‘, ‘behalf‘, ‘of‘, ‘the‘, ‘four‘, ‘times‘, ‘group‘, ‘at‘, ‘the‘, ‘exhibition‘, ‘We‘, ‘will‘, ‘introduce‘, ‘the‘, ‘new‘, ‘star`s‘, ‘business‘, ‘the‘, ‘advantages‘, ‘and‘, ‘the‘, ‘successful‘, ‘cases‘, ‘so‘, ‘that‘, ‘you‘, ‘can‘, ‘understand‘, ‘the‘, ‘new‘, ‘translator‘, ‘more‘, ‘comprehensively‘, ‘We‘, ‘have‘, ‘a‘, ‘stable‘, ‘full-time‘, ‘international‘, ‘team‘, ‘that‘, ‘ensures‘, ‘punctual‘, ‘efficient‘, ‘translation‘, ‘and‘, ‘dubbing‘, ‘and‘, ‘provides‘, ‘a‘, ‘full‘, ‘range‘, ‘of‘, ‘control‘, ‘through‘, ‘the‘, ‘perfect‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘, ‘providing‘, ‘a‘, ‘one-stop‘, ‘service‘, ‘for‘, ‘translation‘, ‘dubbing‘, ‘subtitle‘, ‘production‘, ‘post‘, ‘production‘, ‘broadcasting‘, ‘and‘, ‘ratings‘, ‘surveys‘],[‘The‘, ‘new‘, ‘translator‘, ‘star‘, ‘will‘, ‘represent‘, ‘sida‘, ‘times‘, ‘group‘, ‘in‘, ‘the‘, ‘exhibition‘, ‘when‘, ‘we‘, ‘will‘, ‘introduce‘, ‘the‘, ‘new‘, ‘translator‘, ‘star`s‘, ‘business‘, ‘advantages‘, ‘successful‘, ‘cases‘, ‘and‘, ‘other‘, ‘dimensions‘, ‘so‘, ‘that‘, ‘you‘, ‘can‘, ‘have‘, ‘a‘, ‘more‘, ‘comprehensive‘, ‘understanding‘, ‘of‘, ‘the‘, ‘new‘, ‘translator‘, ‘star‘, ‘We‘, ‘have‘, ‘a‘, ‘stable‘, ‘full-time‘, ‘international‘, ‘team‘, ‘which‘, ‘can‘, ‘ensure‘, ‘timely‘, ‘and‘, ‘efficient‘, ‘translation‘, ‘and‘, ‘dubbing‘, ‘Through‘, ‘perfect‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘, ‘we‘, ‘provide‘, ‘translation‘, ‘dubbing‘, ‘subtitle‘, ‘production‘, ‘post-production‘, ‘broadcasting‘, ‘and‘, ‘rating‘, ‘survey‘]]

  6、用人工翻译组织待检测数据

[‘New‘, ‘Transtar‘, ‘will‘, ‘present‘, ‘itself‘, ‘at‘, ‘the‘, ‘Exhibition‘, ‘on‘, ‘behalf‘, ‘of‘, ‘StarTimes‘, ‘and‘, ‘we‘, ‘will‘, ‘give‘, ‘a‘, ‘comprehensive‘, ‘introduction‘, ‘of‘, ‘ourselves‘, ‘including‘, ‘the‘, ‘current‘, ‘services‘, ‘we‘, ‘offer‘, ‘the‘, ‘advantages‘, ‘we‘, ‘hold‘, ‘and‘, ‘the‘, ‘projects‘, ‘we‘, ‘have‘, ‘completed‘, ‘to‘, ‘help‘, ‘you‘, ‘understand‘, ‘us‘, ‘more‘, ‘New‘, ‘Transtar‘, ‘boasts‘, ‘of‘, ‘an‘, ‘international‘, ‘team‘, ‘of‘, ‘professionals‘, ‘and‘, ‘is‘, ‘capable‘, ‘of‘, ‘providing‘, ‘fast‘, ‘and‘, ‘quality-guaranteed‘, ‘services‘, ‘including‘, ‘translating‘, ‘dubbing‘, ‘subtitle‘, ‘making‘, ‘post-production‘, ‘broadcasting‘, ‘and‘, ‘collecting‘, ‘of‘, ‘viewership‘, ‘ratings‘, ‘thanks‘, ‘to‘, ‘our‘, ‘strict‘, ‘streamlined‘, ‘and‘, ‘developed‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘]

  7、首先测试人工翻译产出的译文与机器翻译评测集之间的BLEU值,得到结果为0.119115465241,如下

[[email protected]10-0-251-156 ~]# python
Python 2.7.5 (default, Apr 11 2018, 07:36:10)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from nltk.translate.bleu_score import sentence_bleu
>>>
>>> reference=[[‘The‘, ‘new‘, ‘translator‘, ‘will‘, ‘stand‘, ‘on‘, ‘the‘, ‘exhibition‘, ‘on‘, ‘behalf‘, ‘of‘, ‘the‘, ‘four‘, ‘times‘, ‘group‘, ‘at‘, ‘the‘, ‘exhibition‘, ‘We‘, ‘will‘, ‘introduce‘, ‘the‘, ‘new‘, ‘star`s‘, ‘business‘, ‘the‘, ‘advantages‘, ‘and‘, ‘the‘, ‘successful‘, ‘cases‘, ‘so‘, ‘that‘, ‘you‘, ‘can‘, ‘understand‘, ‘the‘, ‘new‘, ‘translator‘, ‘more‘, ‘comprehensively‘, ‘We‘, ‘have‘, ‘a‘, ‘stable‘, ‘full-time‘, ‘international‘, ‘team‘, ‘that‘, ‘ensures‘, ‘punctual‘, ‘efficient‘, ‘translation‘, ‘and‘, ‘dubbing‘, ‘and‘, ‘provides‘, ‘a‘, ‘full‘, ‘range‘, ‘of‘, ‘control‘, ‘through‘, ‘the‘, ‘perfect‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘, ‘providing‘, ‘a‘, ‘one-stop‘, ‘service‘, ‘for‘, ‘translation‘, ‘dubbing‘, ‘subtitle‘, ‘production‘, ‘post‘, ‘production‘, ‘broadcasting‘, ‘and‘, ‘ratings‘, ‘surveys‘],[‘The‘, ‘new‘, ‘translator‘, ‘star‘, ‘will‘, ‘represent‘, ‘sida‘, ‘times‘, ‘group‘, ‘in‘, ‘the‘, ‘exhibition‘, ‘when‘, ‘we‘, ‘will‘, ‘introduce‘, ‘the‘, ‘new‘, ‘translator‘, ‘star`s‘, ‘business‘, ‘advantages‘, ‘successful‘, ‘cases‘, ‘and‘, ‘other‘, ‘dimensions‘, ‘so‘, ‘that‘, ‘you‘, ‘can‘, ‘have‘, ‘a‘, ‘more‘, ‘comprehensive‘, ‘understanding‘, ‘of‘, ‘the‘, ‘new‘, ‘translator‘, ‘star‘, ‘We‘, ‘have‘, ‘a‘, ‘stable‘, ‘full-time‘, ‘international‘, ‘team‘, ‘which‘, ‘can‘, ‘ensure‘, ‘timely‘, ‘and‘, ‘efficient‘, ‘translation‘, ‘and‘, ‘dubbing‘, ‘Through‘, ‘perfect‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘, ‘we‘, ‘provide‘, ‘translation‘, ‘dubbing‘, ‘subtitle‘, ‘production‘, ‘post-production‘, ‘broadcasting‘, ‘and‘, ‘rating‘, ‘survey‘]]
>>>
>>> candidate=[‘New‘, ‘Transtar‘, ‘will‘, ‘present‘, ‘itself‘, ‘at‘, ‘the‘, ‘Exhibition‘, ‘on‘, ‘behalf‘, ‘of‘, ‘StarTimes‘, ‘and‘, ‘we‘, ‘will‘, ‘give‘, ‘a‘, ‘comprehensive‘, ‘introduction‘, ‘of‘, ‘ourselves‘, ‘including‘, ‘the‘, ‘current‘, ‘services‘, ‘we‘, ‘offer‘, ‘the‘, ‘advantages‘, ‘we‘, ‘hold‘, ‘and‘, ‘the‘, ‘projects‘, ‘we‘, ‘have‘, ‘completed‘, ‘to‘, ‘help‘, ‘you‘, ‘understand‘, ‘us‘, ‘more‘, ‘New‘, ‘Transtar‘, ‘boasts‘, ‘of‘, ‘an‘, ‘international‘, ‘team‘, ‘of‘, ‘professionals‘, ‘and‘, ‘is‘, ‘capable‘, ‘of‘, ‘providing‘, ‘fast‘, ‘and‘, ‘quality-guaranteed‘, ‘services‘, ‘including‘, ‘translating‘, ‘dubbing‘, ‘subtitle‘, ‘making‘, ‘post-production‘, ‘broadcasting‘, ‘and‘, ‘collecting‘, ‘of‘, ‘viewership‘, ‘ratings‘, ‘thanks‘, ‘to‘, ‘our‘, ‘strict‘, ‘streamlined‘, ‘and‘, ‘developed‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘]
>>>
>>> score = sentence_bleu(reference, candidate)
>>> print score
0.119115465241
>>>

  8、其次我们稍微改动以下百度翻译出来的译文,并测试其与机器翻译评测集之间的BLEU值,得到结果0.875629670466,如下:

    8.1稍微改动之后的百度翻译

New Transtar will stand on the exhibition on behalf of the four times group at the exhibition. We will introduce the new star‘s business, the advantages and the successful cases, so that you can understand the new translator more comprehensively. We have a stable full-time international team that ensures punctual, efficient translation and dubbing, and provides a full range of control through the perfect quality control and project management system, providing a one-stop service for translation, dubbing, subtitle production, streamlined and developed quality control and project management system.

    8.2用改动之后的百度翻译作为待评测数据

[‘New‘, ‘Transtar‘, ‘will‘, ‘stand‘, ‘on‘, ‘the‘, ‘exhibition‘, ‘on‘, ‘behalf‘, ‘of‘, ‘the‘, ‘four‘, ‘times‘, ‘group‘, ‘at‘, ‘the‘, ‘exhibition‘, ‘We‘, ‘will‘, ‘introduce‘, ‘the‘, ‘new‘, ‘star`s‘, ‘business‘, ‘the‘, ‘advantages‘, ‘and‘, ‘the‘, ‘successful‘, ‘cases‘, ‘so‘, ‘that‘, ‘you‘, ‘can‘, ‘understand‘, ‘the‘, ‘new‘, ‘translator‘, ‘more‘, ‘comprehensively‘, ‘We‘, ‘have‘, ‘a‘, ‘stable‘, ‘full-time‘, ‘international‘, ‘team‘, ‘that‘, ‘ensures‘, ‘punctual‘, ‘efficient‘, ‘translation‘, ‘and‘, ‘dubbing‘, ‘and‘, ‘provides‘, ‘a‘, ‘full‘, ‘range‘, ‘of‘, ‘control‘, ‘through‘, ‘the‘, ‘perfect‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘, ‘providing‘, ‘a‘, ‘one-stop‘, ‘service‘, ‘for‘, ‘translation‘, ‘dubbing‘, ‘subtitle‘, ‘production‘, ‘streamlined‘, ‘and‘, ‘developed‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘]

    8.3BLEU计算

>>> candidate_baidu=[‘New‘, ‘Transtar‘, ‘will‘, ‘stand‘, ‘on‘, ‘the‘, ‘exhibition‘, ‘on‘, ‘behalf‘, ‘of‘, ‘the‘, ‘four‘, ‘times‘, ‘group‘, ‘at‘, ‘the‘, ‘exhibition‘, ‘We‘, ‘will‘, ‘introduce‘, ‘the‘, ‘new‘, ‘star`s‘, ‘business‘, ‘the‘, ‘advantages‘, ‘and‘, ‘the‘, ‘successful‘, ‘cases‘, ‘so‘, ‘that‘, ‘you‘, ‘can‘, ‘understand‘, ‘the‘, ‘new‘, ‘translator‘, ‘more‘, ‘comprehensively‘, ‘We‘, ‘have‘, ‘a‘, ‘stable‘, ‘full-time‘, ‘international‘, ‘team‘, ‘that‘, ‘ensures‘, ‘punctual‘, ‘efficient‘, ‘translation‘, ‘and‘, ‘dubbing‘, ‘and‘, ‘provides‘, ‘a‘, ‘full‘, ‘range‘, ‘of‘, ‘control‘, ‘through‘, ‘the‘, ‘perfect‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘, ‘providing‘, ‘a‘, ‘one-stop‘, ‘service‘, ‘for‘, ‘translation‘, ‘dubbing‘, ‘subtitle‘, ‘production‘, ‘streamlined‘, ‘and‘, ‘developed‘, ‘quality‘, ‘control‘, ‘and‘, ‘project‘, ‘management‘, ‘system‘]
>>> score_baidu = sentence_bleu(reference, candidate_baidu)
>>> print score_baidu
0.875629670466
>>>

  9、由上面示例可看到,当待评测译文非常接近(也就是说该译员参考了机器翻译或直接进行的拷贝)机器翻译评测集中的数据时,BLEU值会升高。不过至于高到什么程度才需要项目经理介入,这就需要在实际项目中不断的摸索了。

原文地址:https://www.cnblogs.com/zhenyuyaodidiao/p/9413166.html

时间: 2024-10-13 15:54:46

利用BLEU进行机器翻译检测(Python-NLTK-BLEU评分方法)的相关文章

【NLP】干货!Python NLTK结合stanford NLP工具包进行文本处理

干货!详述Python NLTK下如何使用stanford NLP工具包 作者:白宁超 2016年11月6日19:28:43 摘要:NLTK是由宾夕法尼亚大学计算机和信息科学使用python语言实现的一种自然语言工具包,其收集的大量公开数据集.模型上提供了全面.易用的接口,涵盖了分词.词性标注(Part-Of-Speech tag, POS-tag).命名实体识别(Named Entity Recognition, NER).句法分析(Syntactic Parse)等各项 NLP 领域的功能.

python实现文件下载的方法总结

前端时间遇到一个通过url下载文件的需求,只需要简单的编写一个py脚本即可.从网上搜了下python实现文件下载的方法,总结如下,备查. 以下方法均已测试,环境win8.1  python2.6/2.7 待补充.-- 20150507 方法一: 使用 urllib 模块提供的 urlretrieve() 函数.urlretrieve() 方法直接将远程数据下载到本地. urlretrieve(url, [filename=None, [reporthook=None, [data=None]]]

Python中字典get方法的使用

Python中字典get方法的使用 说起来,这个功能是否需要学习还真是有待于讨论.其实,知道了字典这种数据结构以后,通过最基本的Python功能也能够实现一个类似的功能.不过,既然内置了这个功能那么我们就没有必要重复造车轮. 在字典内置的方法中,想说的方法为get.这个方法是通过键来获取相应的值,但是如果相应的键不存在则返回None.其实,None只是一个默认的返回值但是并不是一个不能修改的返回值.其实,如果查询失败,我们可以指定一个返回值. 上面提到的所有功能汇总在一个示范代码,具体如下: #

Python中的sort()方法的使用

本文和大家分享的主要是python中的sort()方法相关知识,一起来看看吧,希望对大家学习python有所帮助. 一.基本形式 sorted(iterable[, cmp[, key[, reverse]]]) iterable.sort(cmp[, key[, reverse]]) 参数解释: (1)iterable指定要排序的list或者iterable,不用多说: (2)cmp为函数,指定排序时进行比较的函数,可以指定一个函数或者lambda函数,如: students为类对象的list

Windows系统下Python与NumPy安装方法

Windows系统下Python与NumPy安装方法 Windows下Python的某些第三方包安装步骤实在是太麻烦了(这里主要以NumPy为例,目前只有遇到安装它的时候出现了很多问题),晚上花了好几个小时才把NumPy科学计算包安装好,在这里描述下安装过程,避免大家走没有必要的弯路. 1,安装Python 首先,运行下载的MSI安装包,选择安装组件时,确保勾上了所有的组件. 特别要注意选上pip和Add python.exe to Path,然后多次点击Next即可完成安装. Python解释

Python中的__new__()方法与实例化

@Python中的__new__()方法与实例化 __new__()是在新式类中新出现的方法,它作用在构造方法建造实例之前,可以这么理解,在Python 中 存在于类里面的构造方法__init__()负责将类的实例化,而在__init__()启动之前,__new__()决定是否 要使用该__init__()方法,因为__new__()可以调用其他类的构造方法或者直接返回别的对象来作为本类 的实例. 如果将类比喻为工厂,那么__init__()方法则是该工厂的生产工人,__init__()方法接受

Python字典高级使用方法汇总

Python字典高级使用方法汇总 字典(dictionary)是python中的一种非常灵活和强大的数据结构,可以完成很多操作.本文总结了一些除了基本的初始化.赋值.取值之外的常用的字典使用方法. 字典基础参考: [1]:http://www.w3cschool.cc/python/python-dictionary.html [2]:http://www.111cn.net/phper/python/56355.htm [3]:http://skyfen.iteye.com/blog/5675

利用反射机制,获取类的字段、方法、并实现简单调用

这篇文章是为之后要介绍Android的ICO框架做预备的,所以,如果想最近学习Android的ICO框架的同学,可以稍微看一下. 首先,简单介绍一下Java里面的反射. JAVA反射机制是在运行状态中,对于任意一个类,都能够知道这个类的所有属性和方法:对于任意一个对象,都能够调用它的任意一个方法和属性:这种动态获取的信息以及动态调用对象的方法的功能称为java语言的反射机制. 不知道这样的一段解释,你能否看懂.如果更简单的说,反射就是能够根据你给出类名实例化出一个实实在在的对象.所以,对象的实例

可以进行自动补全的python的eclipse配置方法

/*********************************************************************  * Author  : Samson  * Date    : 05/21/2014  * Test platform:  *              Mint 15-3.8.13.13  *              GNU bash, version 4.2.45  * ***************************************