52nlp网站上有个很详细的帖子介绍如何在Ubuntu下搭建Moses, 可参见
http://www.52nlp.cn/ubuntu-moses-platform-build-process-record
以下记录本人在按上述流程操作时,碰到的一些问题及解决方案.
1. The command to download Moses via SVN (you have to install svn in advance):
svn co https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk moses
2. While comping Moses with the command "./regnerate-makefiles.sh" under the directory 'moses', it meets:
name@name:~/smt/moses$ ./regenerate-makefiles.sh
Calling /usr/bin/aclocal…
Calling /usr/bin/autoconf…
configure.in:12: error: possibly undefined macro: AC_DISABLE_SHARED
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.in:13: error: possibly undefined macro: AC_PROG_LIBTOOL
autoconf failed
run the command again, it meets:
Calling /usr/bin/aclocal...
Calling /usr/bin/autoconf...
Calling /usr/bin/automake...
moses/src/Makefile.am:1: Libtool library used but `LIBTOOL' is undefined
moses/src/Makefile.am:1: The usual way to define `LIBTOOL' is to add
`AC_PROG_LIBTOOL'
moses/src/Makefile.am:1: to `configure.in' and run `aclocal' and
`autoconf' again.
moses/src/Makefile.am:1: If `AC_PROG_LIBTOOL' is in `configure.in', make
sure
moses/src/Makefile.am:1: its definition is in aclocal's search path.
automake failed
Check that you have libtool installed, and that you're using recent versions
of the autoconf/automake tools (The version of autoconf/automake should be 1.9 or higher).
3. While comping Moses Support Scripts with the command "make release" under the directory 'moses/scripts', it meets the following errors:
phrasetable.h:17:27: error: boost/bimap.hpp: No such file or directory
Make sure the boost have installed (download boost at http://www.boost.org).
(While installing boost, use "sudo ./bjam install" command and ignor such errors as "libs/iostreams/src/bzip2.cpp....")
Then, the following error happens:
Please specify a BINDIR.
The BINDIR directory must contain GIZA++, snt2cooc.out and mkcls executables.
These are available from http://www.fjoch.com/GIZA++.html and
http://www-i6.informatik.rwth-aachen.de/Colleagues/och/software/mkcls.html .
Copy GIZA++ and snt2cooc.out from the directory GIZA++-v2, and mkcls from the diectory mkcls-v2 into the directory "$SMT/bin". Then run the command "make release".
OK, done. It creates a directory "scripts-yyyymmdd-hhmm".
其他问题:
安装srilm时, 提示tcl.h找不到时, 需要修改common/Makefile.machine.i686文件, 找到Tcl support部分, 修改为
# Tcl support (standard in Linux)
TCL_INCLUDE = -I/usr/include/tcl8.5
TCL_LIBRARY = -L/usr/lib/tcl8.5
安装irstlm (irstlm-5.60.02)时, 运行./regenerate-makefiles.sh, autoconf出错:
configure.in:17: error: possibly undefined macro: AC_DISABLE_SHARED
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.in:18: error: possibly undefined macro: AC_PROG_LIBTOOL
autoconf failed
再次运行./regenerate-makefiles.sh, automake出错
src/Makefile.am:1: Libtool library used but `LIBTOOL' is undefined
src/Makefile.am:1: The usual way to define `LIBTOOL' is to add `AC_PROG_LIBTOOL'
src/Makefile.am:1: to `configure.in' and run `aclocal' and `autoconf' again.
src/Makefile.am:1: If `AC_PROG_LIBTOOL' is in `configure.in', make sure
src/Makefile.am:1: its definition is in aclocal's search path.
src/Makefile.am: installing `./depcomp'
src/Makefile.am: C source seen but `CC' is undefined
src/Makefile.am: The usual way to define `CC' is to add `AC_PROG_CC'
src/Makefile.am: to `configure.in' and run `autoconf' again.
automake failed
确保安装libtool (sudo apt-get install libtool) 和 autoconf/automake版本在1.9或以上 (sudo apt-get install automake1.10).
再次运行./regenerate-makefiles.sh, OK!!
运行./configure --prefix=/home/..../irstlm --enable-caching
运行make, 提示gzfilebuf.h:8:18: error: zlib.h: No such file or directory, 安装zlib (sudo apt-get install zlib1g-dev), 重新运行make, OK!!
运行make install, OK!!
2010年8月17日星期二
2010年5月9日星期日
往latex里插入word图片
一直只会用word工具画图, 可以通过以下的方式将word图片转化为eps格式文件, 然后添加到latex代码中
其流程可以简化为: word图片 --> wmf格式图片 --> eps格式图片
1). Word图片-->wmf格式图片
可以将word图片直接复制到visio中, 另存为wmf格式的图片
2). wmf格式图片 --> eps格式图片
可参考http://www.softbunny.net/post/191.shtml
其具体步骤为
2.1) 下载wmf2eps.exe工具 http://www.softbunny.net/upload/wmf2eps.rar
2.2) 打开wmf2eps.exe对话框, 导入wmf格式图片, 执行菜单命令EPS-Conversion/Conversion Setup..., 在弹出的对话框中, 找到EPS-processing/PS-Printer项, 点击Change按钮, 选择pdf打印机, 同时设置pdf打印机的两个属性, 分别是: 一, 首选项/Adobe PDF Settings, 将Do not sent fonts to "Adobe PDF"为非选中状态. 二, 首选项/纸张质量/高级/文档选项/PostScript选项/PostScript输出选项, 先择内嵌的PostScript
2.3) 运行菜单命令EPS-Conversion/Current, 将生成与wmf格式文件同名的eps文件
***************************************
以上方法太麻烦.
1) word画图, 页面设置为图片大小.
2) 转成pdf文件, 两种情况都可能发生: a. 生成的pdf文件页面大小为图片的大小; b. 生成的pdf文件页面大小很大, 如普通A1/letter大小. 按步骤3处理
3) 打印生成pdf文件, 在打印选项时, 在属性选项里自定义打印的页面大小为图片的大小
这样得到pdf文件, 记figure1.pdf
在latex插入图片时, 用latex **.tex命令编译会出错, 找不到figure1. 两种解决方案:
方案1) 用pdflatex **.tex命令
方案2) pdf2ps figure1.pdf生成ps文件, 发现latex **.tex编译通过, 但最后生成的pdf中figure1显示的有问题; 需要进一步将ps文件生成eps, ps2eps figure.ps, 得到figure1.eps文件后, latex **.tex编译通过, 并且pdf文件中的图片显示正常.
其流程可以简化为: word图片 --> wmf格式图片 --> eps格式图片
1). Word图片-->wmf格式图片
可以将word图片直接复制到visio中, 另存为wmf格式的图片
2). wmf格式图片 --> eps格式图片
可参考http://www.softbunny.net/post/191.shtml
其具体步骤为
2.1) 下载wmf2eps.exe工具 http://www.softbunny.net/upload/wmf2eps.rar
2.2) 打开wmf2eps.exe对话框, 导入wmf格式图片, 执行菜单命令EPS-Conversion/Conversion Setup..., 在弹出的对话框中, 找到EPS-processing/PS-Printer项, 点击Change按钮, 选择pdf打印机, 同时设置pdf打印机的两个属性, 分别是: 一, 首选项/Adobe PDF Settings, 将Do not sent fonts to "Adobe PDF"为非选中状态. 二, 首选项/纸张质量/高级/文档选项/PostScript选项/PostScript输出选项, 先择内嵌的PostScript
2.3) 运行菜单命令EPS-Conversion/Current, 将生成与wmf格式文件同名的eps文件
***************************************
以上方法太麻烦.
1) word画图, 页面设置为图片大小.
2) 转成pdf文件, 两种情况都可能发生: a. 生成的pdf文件页面大小为图片的大小; b. 生成的pdf文件页面大小很大, 如普通A1/letter大小. 按步骤3处理
3) 打印生成pdf文件, 在打印选项时, 在属性选项里自定义打印的页面大小为图片的大小
这样得到pdf文件, 记figure1.pdf
在latex插入图片时, 用latex **.tex命令编译会出错, 找不到figure1. 两种解决方案:
方案1) 用pdflatex **.tex命令
方案2) pdf2ps figure1.pdf生成ps文件, 发现latex **.tex编译通过, 但最后生成的pdf中figure1显示的有问题; 需要进一步将ps文件生成eps, ps2eps figure.ps, 得到figure1.eps文件后, latex **.tex编译通过, 并且pdf文件中的图片显示正常.
2010年4月29日星期四
2009年5月23日星期六
不知道哪些人抽风, 弄得都不能访问blogger的.
百度暗吧暗帖,极其暗含搞笑的回复
楼主 寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏?
2 回复:吾有上将陈腊肠,可斩武藤兰! 作者: 孤独の灵 2008-12-9 09:42
3 回复:准了 作者: 里特普林斯 2008-12-9 09:42
4 回复:草民虽一小女子,但壮志雄心胜男儿,愿以此贱躯犒劳前方将士 作者: 221.4.181.* 2008-12-9 09:45
5 回复:4L,忠烈可嘉,但恐伤我军元气,且等凯旋之日再议 作者: 神手6870 2008-12-9 09:49 回复此发言--------------------------------------------------------------------------------
6 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 吾有上将马加爵,可斩小泉! 作者: 121.229.53.* 2008-12-9 09:51 回复此发言--------------------------------------------------------------------------------
7 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 吾有上将 谢亚龙,可段其足球经济 吾乃还有史玉柱,史家第58代 其曰 游戏乃其所最好 可派史家军冲其游戏行业,扰乱其电子文化。 作者: 222.210.237.* 2008-12-9 09:58 回复此发言--------------------------------------------------------------------------------
8 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 某愿往!吾使得一对八十斤铜锤,对不死生物造成500%伤害,原为征靖国神社先锋! 作者: 暴走的马加爵 2008-12-9 10:04 回复此发言--------------------------------------------------------------------------------
9 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 看片不看武藤兰,看遍A片也枉然。 作者: 60.166.155.* 2008-12-9 10:08 回复此发言--------------------------------------------------------------------------------
10 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 大将出现在地表 作者: Vipermagi 2008-12-9 10:08 回复此发言--------------------------------------------------------------------------------
11 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 待凯旋班师之日,勿忘广收小太阳的姿色小妞,归来犒赏儿郎!!! 作者: lovershome 2008-12-9 10:08 回复此发言--------------------------------------------------------------------------------
12 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 某虽不才,愿身先士卒,杀敌立功,马革裹尸,以报陛下知遇之恩。 作者: 220.232.82.* 2008-12-9 10:09 回复此发言--------------------------------------------------------------------------------
13 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 专乱阵型者在也!野蛮旋风狂搅浑水! 作者:铿锵妞妞 2008-12-11 19:39 回复此发言--------------------------------------------------------------------------------2 百度暗吧看到的帖子,回复有点意思 作者: 60.166.155.* 2008-12-9 10:10 回复此发言--------------------------------------------------------------------------------
14 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 男的统统杀掉,女的统统俘虏 作者: 222.240.83.* 2008-12-9 10:14 回复此发言--------------------------------------------------------------------------------
15 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 东征行,我看行! 作者: 219.128.130.* 2008-12-9 10:42 回复此发言--------------------------------------------------------------------------------
16 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 陛下,微臣保举一人,四川成都人士,姓李名宇春字水泥,此人身高丈二,膀阔十围,诚为中国第一纯爷们也。 作者: 屠龙的史莱姆 2008-12-9 10:48 回复此发言--------------------------------------------------------------------------------
17 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 16楼真贤臣也,内举不避亲! 作者: 疯狂的骷髅 2008-12-9 10:51 回复此发言--------------------------------------------------------------------------------
18 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 臣愿做右骑大将军,必生擒武藤兰,进奉楼主! 作者: 暗黑新手上路 2008-12-9 11:11 回复此发言--------------------------------------------------------------------------------
19 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 传朕旨意,召李水泥、芙蓉觐见,即日领兵讨伐此二将乃我国之栋梁,朕得此二人分忧,国之大幸矣 作者: 神手6870 2008-12-9 11:19 回复此发言--------------------------------------------------------------------------------
20 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 吾举神将米山!一式老汉推车!天下难寻敌手! 作者: 60.9.71.* 2008-12-9 11:20 回复此发言--------------------------------------------------------------------------------
21 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 米山固然勇猛,却年事已高,不可胜任,寡人思得安姐毁罗格、占墓穴,勇不可挡,兼美貌、性感响彻全国,可委以重任,众卿有异议否? 作者: 神手6870 2008-12-9 11:34 回复此发言--------------------------------------------------------------------------------
23 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 吾谓大谬不然。安姐虽才貌双全,终非吾国之人,借他国之将而攻彼国,恐小太阳本不服也。 作者: 风叶千奈 2008-12-9 11:38 回复此发言--------------------------------------------------------------------------------
24 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 臣保举一人:花果山的孙悟空,此人生性好动,刀枪不入,且精通易形,想当年大闹天宫将天庭搅的天翻地覆!此次任他出征,想那东赢不死也得被折腾死! 作者: 暗黑新手上路 2008-12-9 11:57 回复此发言--------------------------------------------------------------------------------
25 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 某愿为先锋,率部嫖妓营,长鸟直入,奸灭敌众 作者: 见缝插茎 2008-12-9 11:57 回复此发言--------------------------------------------------------------------------------
26 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 何必劳师动众?JP KR ,弹丸小国,遣一黄昏松之士,片纸即可致彼肉袒来降。 作者: 222.220.236.* 2008-12-9 11:58 回复此发言--------------------------------------------------------------------------------
27 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 能不能斩掉深圳猥亵小女孩的林大员外先? 作者: 116.30.157.* 2008-12-9 12:01 回复此发言 作者:铿锵妞妞 2008-12-11 19:39 回复此发言--------------------------------------------------------------------------------3 回复:百度暗吧看到的帖子,回复有点意思
28 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 草民只是路过打酱油的,近来传说要开战,羊城内物价飞涨,人人自危…… 作者: 水迭澜 2008-12-9 12:02 回复此发言--------------------------------------------------------------------------------
29 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 倭奴不灭,汝休要回来 作者: 58.213.248.* 2008-12-9 12:08 回复此发言--------------------------------------------------------------------------------
30 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 吾举荐一不世猛将,名谓林JX,交通部人,生性王八,好以猥亵制人。若配其城管精锐,何愁夷蛮不灭! 作者: 亚瑞克督军 2008-12-9 12:09 回复此发言--------------------------------------------------------------------------------
31 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 吾年事已高,特派遣楼下我儿,起兵东征,倭国不灭,誓不回潮 作者: 125.77.17.* 2008-12-9 12:14 回复此发言--------------------------------------------------------------------------------
32 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 王师东定东亚日,家祭无忘告乃翁~顶了~ 作者: 218.28.54.* 2008-12-9 13:07 回复此发言--------------------------------------------------------------------------------
33 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 先帝创业未半而中道崩殂;今天下三分,临表涕零,不知所言。作者: zzzzz_why 2008-12-9 13:14 回复此发言 作者:铿锵妞妞 2008-12-11 19:40 回复此发言--------------------------------------------------------------------------------4 回复:百度暗吧看到的帖子,回复有点意思
34 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 洒家愿誓死追随陛下,某拿得87斤冷艳钜,开得300斤射雕弓,有万夫不当之勇,且先拿些酒饭与吾吃了,待吾生擒萨科齐来我中原,粪尿淋之,方解吾心头之恨 作者: 黏土石魔宝宝 2008-12-9 13:31 回复此发言--------------------------------------------------------------------------------
35 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 哀家准奏. 作者: 低俗ID 2008-12-9 13:42 回复此发言--------------------------------------------------------------------------------
36 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 林子大了什么鸟都有,我自己去了,你们别拦我 作者: nalirenshi 2008-12-9 14:50 回复此发言--------------------------------------------------------------------------------
37 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 属下已在东瀛隐匿多日,每西望故土,以思为国捐躯之时几至。今闻陛下在朝议麾师东征,与在倭游子语之,皆慷慨涕零,誓效犬马之力以报家国。愿举李大鞭,商丘人士,膂力过人,弓马娴熟,更腰缠老二,粗若象拔,比垂过膝,有万夫不当之勇,若以为先锋,克东瀛自慰队乃摧枯拉朽耳;又有索钻头,洛阳人,饱读兵书,阴险狡诈天怒人怨,可为幕僚,定可设计使东瀛男丁一夜之间尽皆去势也!更有小钻风,机智过人,投给钻营之术堪可夺鬼神之力,三旬之内可使东洋利市尽皆枯槁,我天朝神兵借机发兵,岂非事半功倍哉?愿陛下早日降诏,臣等翘首以待天师。谨顿首再拜 作者: 路过_993 2008-12-9 17:49 回复此发言--------------------------------------------------------------------------------
38 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 33a88d 作者: 218.65.255.* 2008-12-9 17:51 回复此发言--------------------------------------------------------------------------------
39 回复:寡人欲挥师东征,踏平东瀛、高丽,众位爱卿有何本奏? 可派国足英豪到东瀛几条河流的上流洗脚。
2009年2月2日星期一
记录下09年的火车运转
日期 车次 乘车区间 票价 备注
01.17 K467 苏州-向塘 61.0 (学) 春节前回家
01.18 L7549 向塘-抚州北 5.5 春节前回家
02.01 L7550 抚州北-南昌 3.5(学) 春节后回校
02.01 2188 南昌-杭州 45 春节后回校 02.01 L374 杭州东-嘉兴 7.0 春节后回校
05.15 K469 苏州-向塘 120 这车客流不错, 一直到向塘, 都有人站着无座位.
05.16 2817 向塘-南昌 4.5 这车真和谐, 早点到向塘20分钟左右, 到南昌车补4.5元, 可惜马上要升空调车底了
05.17 K288 南昌-嘉兴 99
07.21 T178 苏州-济南 207 硬卧中铺. 苏州晚点一小时, 到济南只晚点20分钟, 夕发朝致.
07.22 5003 济南-烟台 80 绿皮硬卧, 超级便宜. 但睡得很不舒服
07.27 2584 烟台-泰山 73 硬座
07.28 T177 泰山-苏州 197 还是这个车的硬卧舒服啊, 凌晨3.40到苏州太早了点
南昌八一广场


01.17 K467 苏州-向塘 61.0 (学) 春节前回家
01.18 L7549 向塘-抚州北 5.5 春节前回家
02.01 L7550 抚州北-南昌 3.5(学) 春节后回校
02.01 2188 南昌-杭州 45 春节后回校 02.01 L374 杭州东-嘉兴 7.0 春节后回校
05.15 K469 苏州-向塘 120 这车客流不错, 一直到向塘, 都有人站着无座位.
05.16 2817 向塘-南昌 4.5 这车真和谐, 早点到向塘20分钟左右, 到南昌车补4.5元, 可惜马上要升空调车底了
05.17 K288 南昌-嘉兴 99
07.21 T178 苏州-济南 207 硬卧中铺. 苏州晚点一小时, 到济南只晚点20分钟, 夕发朝致.
07.22 5003 济南-烟台 80 绿皮硬卧, 超级便宜. 但睡得很不舒服
07.27 2584 烟台-泰山 73 硬座
07.28 T177 泰山-苏州 197 还是这个车的硬卧舒服啊, 凌晨3.40到苏州太早了点
南昌八一广场
2008年11月24日星期一
Java中常用的代码
对Java很菜鸟, 收录一些java中经常需要使用的代码
1). JAVA操作文本文件
public class TestFile2
{
public static void main(String[] args) throws IOException
{
BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream("ming.txt")));
String data = null;
while((data = br.readLine())!=null)
{
System.out.println(data);
}
OutputStreamWriter osw = new OutputStreamWriter(new FileOutputStream("hello2.txt"));
osw.write(s,0,s.length());
osw.flush();
PrintWriter pw = new PrintWriter(new OutputStreamWriter(new FileOutputStream("hello3.txt")),true);
pw.println(s);
br.close();
osw.close();
pw.close();
}
}
2008年11月10日星期一
Berkeley Parser的使用
Berkeley Parser的使用
Berkeley Parser可以从http://code.google.com/p/berkeleyparser/下载。下载后,在eclipse下运行其代码,发现实验得到的parsing性能比其文章中的要略低,大概低于1%多点。以下是实验过程:
一、训练模型
1). 解压Berkeley Parser提供的jar包后,将源代码添加到eclipse或jbuilder中,会发现编译不通过,需要将edu.berkeley.nlp.PCFGLA.HierarchicalFullyConnectedAdaptiveLexiconWithFeatures.java的第9行注释掉.
2). 在使用ctb2.0或5.1训练时,需要注意2个地方:
I. edu.berkeley.nlp.PCFGLA.Corpus.java的loadChinese方法中,对中文训练语料集文件的选定.
II. edu.berkeley.nlp.PennTreebanReader.java按PTB的格式从文件中读取树,因此,需要整理中文树库文件,即将新闻编号,句子编号等多余的内容删除(每个句子前的“( (”仍需保留)。使得文件中只保留bracketed格式的数据.
III. edu.berkeley.nlp.PennTreebanReader.java的TreeCollection方法(第136行),是默认为文件名为.mrg,需要与CTB的.fid相一致.
3). 开始训练,在eclipse下,设置edu.berkeley.nlp.PCFGLA.GrammarTrain的参数为-path ../Corpus/xma/ctb2.0/data2 -out ./ctb2.0/ctb2.0_sm5.gr -SMcycles 5 -treebank CHINESE
ctb2.0训练集/开发集/测试集分别包含3485/353/348个句子, 当SMcycles设置为5时,训练耗时15分钟. 当SMcycles设置为6时,训练耗时45分钟. SMcycles值每增加1,所需训练时间呈指数级增长
ctb5.1训练集/开发集/测试集分别包含18103/352/348个句子, 当SMcycles设置为5时,训练耗时15分钟.
二、测试模型
java -jar berkeleyParsem5.gr <> ctb2.0/ctb2.0_test.sm5.txt
三、性能评测
采用标准的评测程序
SMcycles=5的情况下在ctb2.0下的评测结果, 按标准训练/开发/测试划分
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 75.23
Bracketing Precision = 78.11
Bracketing FMeasure = 76.64
Complete match = 24.71
Average crossing = 2.61
No crossing = 45.69
2 or less crossing = 67.82
Tagging accuracy = 91.93
-- len<=40 --
Number of sentence = 300
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 300
Bracketing Recall = 77.59
Bracketing Precision = 80.65
Bracketing FMeasure = 79.09
Complete match = 28.67
Average crossing = 1.68
No crossing = 52.33
2 or less crossing = 75.33
Tagging accuracy = 92.38
SMcycles=6的情况下在ctb2.0下的评测结果, 按标准训练/开发/测试划分
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 74.79
Bracketing Precision = 77.02
Bracketing FMeasure = 75.89
Complete match = 24.43
Average crossing = 2.78
No crossing = 44.83
2 or less crossing = 64.94
Tagging accuracy = 92.17
-- len<=40 --
Number of sentence = 300
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 300
Bracketing Recall = 77.55
Bracketing Precision = 80.15
Bracketing FMeasure = 78.83
Complete match = 28.00
Average crossing = 1.80
No crossing = 51.67
2 or less crossing = 73.67
Tagging accuracy = 92.70
在NAACL07_Improved Inference for Unlexicalized Parsing文中,作者在ctb2.0上得到的性能为<=40words(LP/80.8, LR/80.7), all(LP/78.8, LR/78.5),但其训练集/开发集/测试集分别为chtb026.fid-chtb270.fid/chtb001-chtb025.fid/chtb271.fid-chtb300.fid. 此划分的训练集/开发集/测试集分别包含有3178/307/348句子。
按此划分,SMcycles取值为5,得到的结果为
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 75.65
Bracketing Precision = 78.91
Bracketing FMeasure = 77.25
Complete match = 23.85
Average crossing = 2.49
No crossing = 45.11
2 or less crossing = 70.11
Tagging accuracy = 92.54
-- len<=40 --
Number of sentence = 300
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 300
Bracketing Recall = 78.23
Bracketing Precision = 81.75
Bracketing FMeasure = 79.95
Complete match = 27.67
Average crossing = 1.53
No crossing = 51.67
2 or less crossing = 79.00
Tagging accuracy = 93.10
按此划分, SMcycles取值为6, 得到的结果为
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 75.30
Bracketing Precision = 77.29
Bracketing FMeasure = 76.28
Complete match = 25.00
Average crossing = 2.62
No crossing = 44.25
2 or less crossing = 68.97
Tagging accuracy = 91.84
-- len<=40 --
Number of sentence = 300
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 300
Bracketing Recall = 78.53
Bracketing Precision = 80.74
Bracketing FMeasure = 79.62
Complete match = 28.67
Average crossing = 1.59
No crossing = 51.00
2 or less crossing = 78.00
Tagging accuracy = 92.51
在NAACL07_Improved Inference for Unlexicalized Parsing文中,作者在ctb2.0上得到的性能为<=40words(LP/86.9, LR/85.7), all(LP/84.8, LR/81.9)
采用CTB5.1, 训练集采用chtb001.fid-chtb270.fid + chtb400.fid-chtb1021.fid + chtb1030-chtb1151.fid (当训练集包括文件chtb1022.fid或chtb1129.fid时,对很多测试句子找不到句法分析树,得到的是(())结果).
SMcycles=5的情况下在ctb5.1下的评测结果
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 81.01
Bracketing Precision = 83.63
Bracketing FMeasure = 82.30
Complete match = 30.46
Average crossing = 1.99
No crossing = 53.74
2 or less crossing = 72.41
Tagging accuracy = 95.08
-- len<=40 --
Number of sentence = 299
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 299
Bracketing Precision = 86.65
Bracketing FMeasure = 85.38
Complete match = 35.45
Average crossing = 1.16
No crossing = 60.87
2 or less crossing = 80.94
Tagging accuracy = 95.58
Berkeley Parser可以从http://code.google.com/p/berkeleyparser/下载。下载后,在eclipse下运行其代码,发现实验得到的parsing性能比其文章中的要略低,大概低于1%多点。以下是实验过程:
一、训练模型
1). 解压Berkeley Parser提供的jar包后,将源代码添加到eclipse或jbuilder中,会发现编译不通过,需要将edu.berkeley.nlp.PCFGLA.HierarchicalFullyConnectedAdaptiveLexiconWithFeatures.java的第9行注释掉.
2). 在使用ctb2.0或5.1训练时,需要注意2个地方:
I. edu.berkeley.nlp.PCFGLA.Corpus.java的loadChinese方法中,对中文训练语料集文件的选定.
II. edu.berkeley.nlp.PennTreebanReader.java按PTB的格式从文件中读取树,因此,需要整理中文树库文件,即将新闻编号,句子编号等多余的内容删除(每个句子前的“( (”仍需保留)。使得文件中只保留bracketed格式的数据.
III. edu.berkeley.nlp.PennTreebanReader.java的TreeCollection方法(第136行),是默认为文件名为.mrg,需要与CTB的.fid相一致.
3). 开始训练,在eclipse下,设置edu.berkeley.nlp.PCFGLA.GrammarTrain的参数为-path ../Corpus/xma/ctb2.0/data2 -out ./ctb2.0/ctb2.0_sm5.gr -SMcycles 5 -treebank CHINESE
ctb2.0训练集/开发集/测试集分别包含3485/353/348个句子, 当SMcycles设置为5时,训练耗时15分钟. 当SMcycles设置为6时,训练耗时45分钟. SMcycles值每增加1,所需训练时间呈指数级增长
ctb5.1训练集/开发集/测试集分别包含18103/352/348个句子, 当SMcycles设置为5时,训练耗时15分钟.
二、测试模型
java -jar berkeleyParsem5.gr <> ctb2.0/ctb2.0_test.sm5.txt
三、性能评测
采用标准的评测程序
SMcycles=5的情况下在ctb2.0下的评测结果, 按标准训练/开发/测试划分
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 75.23
Bracketing Precision = 78.11
Bracketing FMeasure = 76.64
Complete match = 24.71
Average crossing = 2.61
No crossing = 45.69
2 or less crossing = 67.82
Tagging accuracy = 91.93
-- len<=40 --
Number of sentence = 300
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 300
Bracketing Recall = 77.59
Bracketing Precision = 80.65
Bracketing FMeasure = 79.09
Complete match = 28.67
Average crossing = 1.68
No crossing = 52.33
2 or less crossing = 75.33
Tagging accuracy = 92.38
SMcycles=6的情况下在ctb2.0下的评测结果, 按标准训练/开发/测试划分
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 74.79
Bracketing Precision = 77.02
Bracketing FMeasure = 75.89
Complete match = 24.43
Average crossing = 2.78
No crossing = 44.83
2 or less crossing = 64.94
Tagging accuracy = 92.17
-- len<=40 --
Number of sentence = 300
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 300
Bracketing Recall = 77.55
Bracketing Precision = 80.15
Bracketing FMeasure = 78.83
Complete match = 28.00
Average crossing = 1.80
No crossing = 51.67
2 or less crossing = 73.67
Tagging accuracy = 92.70
在NAACL07_Improved Inference for Unlexicalized Parsing文中,作者在ctb2.0上得到的性能为<=40words(LP/80.8, LR/80.7), all(LP/78.8, LR/78.5),但其训练集/开发集/测试集分别为chtb026.fid-chtb270.fid/chtb001-chtb025.fid/chtb271.fid-chtb300.fid. 此划分的训练集/开发集/测试集分别包含有3178/307/348句子。
按此划分,SMcycles取值为5,得到的结果为
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 75.65
Bracketing Precision = 78.91
Bracketing FMeasure = 77.25
Complete match = 23.85
Average crossing = 2.49
No crossing = 45.11
2 or less crossing = 70.11
Tagging accuracy = 92.54
-- len<=40 --
Number of sentence = 300
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 300
Bracketing Recall = 78.23
Bracketing Precision = 81.75
Bracketing FMeasure = 79.95
Complete match = 27.67
Average crossing = 1.53
No crossing = 51.67
2 or less crossing = 79.00
Tagging accuracy = 93.10
按此划分, SMcycles取值为6, 得到的结果为
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 75.30
Bracketing Precision = 77.29
Bracketing FMeasure = 76.28
Complete match = 25.00
Average crossing = 2.62
No crossing = 44.25
2 or less crossing = 68.97
Tagging accuracy = 91.84
-- len<=40 --
Number of sentence = 300
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 300
Bracketing Recall = 78.53
Bracketing Precision = 80.74
Bracketing FMeasure = 79.62
Complete match = 28.67
Average crossing = 1.59
No crossing = 51.00
2 or less crossing = 78.00
Tagging accuracy = 92.51
在NAACL07_Improved Inference for Unlexicalized Parsing文中,作者在ctb2.0上得到的性能为<=40words(LP/86.9, LR/85.7), all(LP/84.8, LR/81.9)
采用CTB5.1, 训练集采用chtb001.fid-chtb270.fid + chtb400.fid-chtb1021.fid + chtb1030-chtb1151.fid (当训练集包括文件chtb1022.fid或chtb1129.fid时,对很多测试句子找不到句法分析树,得到的是(())结果).
SMcycles=5的情况下在ctb5.1下的评测结果
-- All --
Number of sentence = 348
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 348
Bracketing Recall = 81.01
Bracketing Precision = 83.63
Bracketing FMeasure = 82.30
Complete match = 30.46
Average crossing = 1.99
No crossing = 53.74
2 or less crossing = 72.41
Tagging accuracy = 95.08
-- len<=40 --
Number of sentence = 299
Number of Error sentence = 0
Number of Skip sentence = 0
Number of Valid sentence = 299
Bracketing Precision = 86.65
Bracketing FMeasure = 85.38
Complete match = 35.45
Average crossing = 1.16
No crossing = 60.87
2 or less crossing = 80.94
Tagging accuracy = 95.58
订阅:
博文 (Atom)