Mr.Mira

1) 下载

git clone https://github.com/kho/mr-mira.git

2) 安装
2.1) autoreconf -ifv
错误: 提示autoconf版本出错需要2.65以上版本, 而当前autoconf为2.63.
方法: 修改configure.ac文件第一行, 将AC_PREREQ(2.65)改为AC_PREREQ(2.63)

2.2）./configure --with-boost=/..../boost_1_53_0
错误: 提示一个boost::program_options...的错误
方法: 重新安装boost: ./bootstrap.sh -prefix=/.../boost_1_53_0
                                ./bjam install

错误: checking for glog... configure: error: Package requirements (libglog >= 0.3) were not met:

No package 'libglog' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables glog_CFLAGS
and glog_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
方法: 安装glog, 设置glog_CFLAGS和glog_LIBS环境变量
         export glog_CFLAGS=-I/../glog/include
         export glog_LIBS=/../glog/lib/libglog.a
2.3) make


3) 使用
由于没有make install, 所以在使用的时候需要做如下修改scripts/run-mira-qsub.bash:

SUBMIT=`which qsub-mr.bash`

MAPPER=`which kbest_mirav5`

REDUCER=`which mr_mira_reduce`

-->

SUBMIT=/.../mr-mira/scripts/qsub-mr.bash

MAPPER=/.../mr-mira/mira/kbest_mirav5

REDUCER=/.../mr-mira/mira/mr_mira_reduce

[ ! -e "$OUTPUT" ] || { log "$OUTPUT already exists"; exit 1; }

-->

#[ ! -e "$OUTPUT" ] || { log "$OUTPUT already exists"; exit 1; }

注释掉此行. 通常我更喜欢将stderr, stdout重定向输出放入到$OUTPUT目录下. 如果$OUTPUT已存在, 该行代码将出错; 如果$OUTPUT不存在, 又无法重定向stderr, stdout. 因此注释掉此行代码.

QSUB_OPTS=(-q wide -l pmem=4g,walltime=2:00:00)

-->

if [ "x$PMEM" = x ]; then

    PMEM=4g

fi

QSUB_OPTS=(-q wide -l pmem=$PMEM,walltime=2:00:00)

添加一个PMEM参数用来控制申请内存大小.

run-mira-qsub.bash将多个作业提交到集群上跑.
run-mira-serial.bash是在单机上只开一个进程跑, 暂不支持多线程跑..

关于input文件的格式, 每个句子格式如下:

source ref rest
其中source指 source
      ref 如果有多个, 用 ||| 隔开
      rest暂可以不提供
见http://aclweb.org/anthology//P/P13/P13-4034.pdf

cdec 获取top-best translation

viterbi.cc

prob_t ViterbiESentence(const Hypergraph& hg, vector* result) {
  return Viterbi(hg, result);
}

viterbi.h
template
typename WeightFunction::Weight Viterbi(const Hypergraph& hg,
                   typename Traversal::Result* result,
                   const Traversal& traverse,
                   const WeightFunction& weight) {
  typedef typename Traversal::Result T;
  typedef typename WeightFunction::Weight WeightType;
  const int num_nodes = hg.nodes_.size();
  std::vector vit_result(num_nodes);
  std::vector vit_weight(num_nodes, WeightType());

  for (int i = 0; i < num_nodes; ++i) {
    const Hypergraph::Node& cur_node = hg.nodes_[i];
    WeightType* const cur_node_best_weight = &vit_weight[i];
    T*          const cur_node_best_result = &vit_result[i];

    const unsigned num_in_edges = cur_node.in_edges_.size();
    if (num_in_edges == 0) {
      *cur_node_best_weight = WeightType(1);
      continue;
    }
    HG::Edge const* edge_best=0;
    for (unsigned j = 0; j < num_in_edges; ++j) {
      const HG::Edge& edge = hg.edges_[cur_node.in_edges_[j]];
      WeightType score = weight(edge);
      for (unsigned k = 0; k < edge.tail_nodes_.size(); ++k)
        score *= vit_weight[edge.tail_nodes_[k]];
      if (!edge_best || *cur_node_best_weight < score) {
        *cur_node_best_weight = score;
        edge_best=&edge;
      }
    }
    assert(edge_best);
    HG::Edge const& edgeb=*edge_best;
    std::vector antsb(edgeb.tail_nodes_.size());
    for (unsigned k = 0; k < edgeb.tail_nodes_.size(); ++k)
      antsb[k] = &vit_result[edgeb.tail_nodes_[k]];
    traverse(edgeb, antsb, cur_node_best_result);
  }
  if (vit_result.empty())
    return WeightType(0);
  std::swap(*result, vit_result.back());
  return vit_weight.back();
}

其中traverse调用ESentenceTraversal() --> TRule::ESubstitute()

Perl一些代码

1) 得到当前目录

use Cwd            qw( abs_path );
use File::Basename qw( dirname );

my $current_dir = dirname(abs_path($0));


2) 使用regex进行部分lowercase替换

$line =~ s/( [^\(\)]+\))/lc($1)/ge;

((S (NP (DT The)(NN luxury)(NN auto)(NN maker))(NP (JJ last)(NN year))(VP (VBD sold)(NP (CD 1,214)(NNS cars))(PP (IN in)(NP (DT the)(NNP U.S.))))))

替换为
((S (NP (DT the)(NN luxury)(NN auto)(NN maker))(NP (JJ last)(NN year))(VP (VBD sold)(NP (CD 1,214)(NNS cars))(PP (IN in)(NP (DT the)(NNP u.s.))))))

3) 使用regex提取字符数组
my @terms = ($line =~ /\([^\(\)]+\)/g);

((S (NP (DT The)(NN luxury)(NN auto)(NN maker))(NP (JJ last)(NN year))(VP (VBD sold)(NP (CD 1,214)(NNS cars))(PP (IN in)(NP (DT the)(NNP U.S.))))))
提取出(DT The)  (NN luxury)  (NN auto)  (NN maker)  (JJ last)  (NN year) ....  (NNP U.S.)

使用maxent zhang le

1) 下载
git clone https://github.com/lzhang10/maxent.git

2) 安装
按照INSTALL说明, 依次安装
$ brew install gfortran  (需要先安装howbrew (ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"), 见http://mxcl.github.io/homebrew/)

$./configure

$make

$make install

3) 使用, 在eclipse中,

#include "maxent/maxentmodel.hpp"

MaxentModel *pModel = new MaxentModel();

pModel->begin_add_event();

.....

pModel->add_event(vecContext, p, 1);

.....

pModel->end_add_event();

pModel->train(100, "lbfgs"); // train the model with 30 iterations of L-BFGS method

编译时在link这一步会提示关于gfortran的错误

需要在Properties属性中添加linker libraries (-l): gfortan, 以及linker libraries path: /usr/local/Cellar/gfortran/4.8.1/gfortran/lib

cdec中的cube pruning

cdec的cube pruning代码对应于apple_models.cc的CubePruningRescorer类.


在传统的cky解码过程中, 我们会为每一个chart定义一个heap, 如heap[i, j]用来存放所以souce side span为[i, j]的hypothese. 该heap的大小通常限置为某个常数(如200). 并且要求每个hypothesis的概率不能小于heap[i,j][0]的某个倍数(如0.05).
在cube pruning的时候, 通过设置pop_limit来限制最多生成多少个hypotheses. 
heap中各项的排列通常是按照score + heuristic_lm_score (即未计算在score中的前面几个单词的lm值)

在cdec的void KBestFast(const int vert_index, const bool is_goal) 方法中, 并没有为每个span[i, j]设置一个heap. 在cube pruning的时候, 设置pop_limit来限制最多生成多少个hypotheses. 虽然没有为每个span[i, j]设置一个heap,  cdec用candidate heap(没有大小限制)来保存:

      D_v.resize(state2node.size());
      int c = 0;
      for (State2Node::iterator i = state2node.begin(); i != state2node.end(); ++i){
          D_v[c++] = i->second;
          // cerr << "MERGED: " << *i->second << endl;
      }
      //cerr <<"Node id: "<< vert_index<< endl;
      //#ifdef MEASURE_CA
      // cerr << "countInProcess (pop/tot): node id: " << vert_index << " (" << count_in_process_pop << "/" << count_in_process_tot << ")"<      // cerr << "countAtEnd (pop/tot): node id: " << vert_index << " (" << count_at_end_pop << "/" << count_at_end_tot << ")"<      //#endif
      sort(D_v.begin(), D_v.end(), EstProbSorter());

cdec默认设置pop_limit为200, 为了增加搜索空间, 似乎只能放大pop_limit的值.

cdec是这样做cube pruning,

假设source side span为[i, j], 其source端看以表达为  a X1 b X2 c, ..., a X1 c,..., 各个source端对应的目标端翻译, 加起来共有m种, t1, t2, ..., tm; X1可能的hypotheses有n种, X11, ..., X1n; X2可能的hypotheses有k种, X21, ..., X2k.

1:  生成一个candidate list = {(0, 0, 0), (1, 0, 0), ..., (m, 0, 0)}. 第1/2/3维分别为上面三个变量的下标. 
2:  pop = 0;
3:  while (pop < pop_limit && candidate_list not empty) do
4:     对candidate list排序;
5:     从candidate_list中取出最大项(i, j, k);
6      生成新的hypothesis;
7      对(i, j, k)扩展, 往candidate_list中添加(i, j+1, k)和(i, j, k+1);
8      pop ++; 


X11, ..., X1n和X21, ..., X2k是按大小排序的. 不确定t1, t2, ..., tm是否也是按大小排序, 但这应该没有关系.

Fixfor is already running

Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system.

http://www.mattcutts.com/blog/how-to-fix-firefox-is-already-running-error/

1) 通过 “ps auxwww | grep firefox” 找到firefox进程号, 然后“kill [processnumber]”.

2) 如果以上还不能解决, Step 1. 找到firefox的profile文件, 通常是 ~/.mozilla/firefox/profiles.ini 该文件里面会说明Path, 比如Path=src6ip5x.default Step 2. 在Path的路径下, 删除lock files. 需要删除“lock”和“.parentlock” .

cdec forced decoding

cdec支持forced decoding, 数据准备格式为:

<seg grammar="grammar/gra.0.gz" id="0"> <s> 今年 前 两 月 广东 高新 技术 产品 出口 ３７．６亿 美元 </s> ||| <s> export of high-tech products in guangdong in first two months this year ３７．６亿 us dollars </s> </seg>

有时候一些可以可以手工根据翻译规则拼成的translation, 在forced decoding的时候并没有找到, 可以关闭language model feature试试, 可能是由于language model的原因, 使得提供的translation并不在hyper-graph中.