linux下替换行末的^M

在Windows下编辑的文本在linux下查看时可能会出现^M字符. 可按下面命令在vim下替换掉。

%s/^M//g "Press and hold control then press V then M" 

关于cdec运行出现connection timeout错误

查看~/cdec/training/utils/parallelize.pl,　将其host修改为下面形式.

#my $host = check_output("hostname");

my $host = check_output("ifconfig | grep -Eo 'inet (addr:)?([0-9]*\\.){3}[0-9]*' | grep -Eo '([0-9]*\\.){3}[0-9]*' | grep -v '127.0.0.1' | head -n 1");

cdec/python 关于运行 python setup.py build 的错误

g++ -pthread -shared build/temp.linux-x86_64-2.7/cdec/_cdec.o -L../decoder -L../utils -L../mteval -L../training/utils -L../klm/lm -L../klm/util -L../klm/util/double-conversion -L../klm/search -lcdec -lutils -lmteval -ltraining_utils -lklm -lklm_util -lklm_util_double -lksearch -ldl -lrt -lboost_program_options -lboost_regex-mt -lboost_serialization-mt -lboost_system -lboost_filesystem-mt -lz -lbz2 -llzma -o build/lib.linux-x86_64-2.7/cdec/_cdec.so -L/usr/local/lib -Wl,-R,/usr/local/lib -L/usr/local/lib -Wl,-R,/usr/local/lib -L/usr/local/lib -Wl,-R,/usr/local/lib -L/usr/local/lib -Wl,-R,/usr/local/lib -L/usr/local/lib -Wl,-R,/usr/local/lib /usr/bin/ld: /usr/local/lib/libbz2.a(bzlib.o): relocation R_X86_64_32S against `BZ2_crc32Table' can not be used when making a shared object; recompile with -fPIC /usr/local/lib/libbz2.a: could not read symbols: Bad value collect2: error: ld returned 1 exit status error: command 'g++' failed with exit status 1 

解决方法:　用一个较新版本cdec/python/setup.py替换掉原版本(cdec-2014-06-15)的cdec-2014-06-15/python/setup.py cdec-2014-06-15/python/setup.py的内容是:

from distutils.core import setup
from distutils.extension import Extension
import re

INC = ['..', 'cdec/', '../decoder', '../utils', '../mteval']
LIB = ['../decoder', '../utils', '../mteval', '../training/utils', '../klm/lm', '../klm/util', '../klm/util/double-conversion', '../klm/search']

# Set automatically by configure
LIBS = re.findall('-l([^\s]+)', '-ldl -lrt  -lboost_program_options -lboost_regex-mt -lboost_serialization-mt -lboost_system -lboost_filesystem-mt  -lz -lbz2 -llzma')
CPPFLAGS = re.findall('-[^\s]+', '-DPIC   -pthread -DHAVE_CONFIG_H -DKENLM_MAX_ORDER=6 -std=gnu++11  -fPIC -g -O3')
LDFLAGS = re.findall('-[^\s]+', ' -L/usr/local/lib -Wl,-R,/usr/local/lib -L/usr/local/lib -Wl,-R,/usr/local/lib -L/usr/local/lib -Wl,-R,/usr/local/lib -L/usr/local/lib -Wl,-R,/usr/local/lib -L/usr/local/lib -Wl,-R,/usr/local/lib')

ext_modules = [
    Extension(name='cdec._cdec',
        sources=['cdec/_cdec.cpp'],
        include_dirs=INC,
        library_dirs=LIB,
        libraries=['cdec', 'utils', 'mteval', 'training_utils', 'klm', 'klm_util', 'klm_util_double', 'ksearch'] + LIBS,
        extra_compile_args=CPPFLAGS,
        extra_link_args=LDFLAGS),
    Extension(name='cdec.sa._sa',
        sources=['cdec/sa/_sa.cpp', 'cdec/sa/strmap.cc'],
        extra_compile_args=CPPFLAGS)
]

setup(
    name='cdec',
    ext_modules=ext_modules,
    packages=['cdec', 'cdec.sa']
)

较新版本的setup.py的内容

 
from distutils.core import setup
from distutils.extension import Extension
import re

INC = ['..', 'cdec/', '../decoder', '../utils', '../mteval']
LIB = ['../decoder', '../utils', '../mteval', '../training/utils', '../klm/lm', '../klm/util', '../klm/util/double-conversion', '../klm/search']

# Set automatically by configure
LIBS = re.findall('-l([^\s]+)', '-ldl -lrt  -lboost_program_options -lboost_regex-mt -lboost_serialization-mt -lboost_system -lboost_filesystem-mt  -lz -lbz2 -llzma')
CPPFLAGS = re.findall('-[^\s]+', '-DPIC  -I/home/jhli/local/include  -DHAVE_CONFIG_H -DKENLM_MAX_ORDER=6 -std=gnu++11  -fPIC -g -O3')
LDFLAGS = re.findall('-[^\s]+', ' -L/home/jhli/local/lib -Wl,-R,/home/jhli/local/lib -L/home/jhli/local/lib -Wl,-R,/home/jhli/local/lib -L/home/jhli/local/lib -Wl,-R,/home/jhli/local/lib -L/home/jhli/local/lib -Wl,-R,/home/jhli/local/lib -L/home/jhli/local/lib -Wl,-R,/home/jhli/local/lib')

ext_modules = [
    Extension(name='cdec._cdec',
        sources=['cdec/_cdec.cpp'],
        include_dirs=INC,
        library_dirs=LIB,
        libraries=['cdec', 'utils', 'mteval', 'training_utils', 'klm', 'klm_util', 'klm_util_double', 'ksearch'] + LIBS,
        extra_compile_args=CPPFLAGS,
        extra_link_args=LDFLAGS),
    Extension(name='cdec.sa._sa',
        sources=['cdec/sa/_sa.cpp', 'cdec/sa/strmap.cc'],
        extra_compile_args=CPPFLAGS)
]

setup(
    name='cdec',
    ext_modules=ext_modules,
    packages=['cdec', 'cdec.sa']
)

linux下安装swipl

1) 下载 swipl http://www.swi-prolog.org/download/stable 2) 安装 swipl 根据INSTALL文件中的提示, 进行 cp build.templ build build ./build 出现以下错误: pl-arith.c文件中关于mp_bitcnt_t一系列的错误信息. 解决办法: 1) 安装gmp, https://gmplib.org/#DOWNLOAD 下载gmp-6.0.0a.tar.bz2 (使用较早版本gmp-4.3.2仍然会出现以上错误). 安装(到自定义路径./configure --prefix=/home/***/local) 2) 回到swipl, 修改build文件, 设置, 这样可以找到上一步安装的gmp export CIFLAGS="-I/home/***/local/include" export LDFLAGS="-O2 -g -L/home/***/local/lib"

nplm代码学习与使用 (3)

模型的训练. model类 (model.h)中定义了如下的成员变量

    Input_word_embeddings input_layer;
    Linear_layer first_hidden_linear;
    Activation_function first_hidden_activation;
    Linear_layer second_hidden_linear;
    Activation_function second_hidden_activation;
    Output_word_embeddings output_layer;
    Matrix<double,Dynamic,Dynamic,Eigen::RowMajor> output_embedding_matrix,
      input_embedding_matrix,
      input_and_output_embedding_matrix;
    
    activation_function_type activation_function;
    int ngram_size, input_vocab_size, output_vocab_size, input_embedding_dimension, num_hidden, output_embedding_dimension;
    bool premultiplied;

model定义了两个构造方法,在trainNeuralNetWork.cpp中调用的是这个:

    model() : ngram_size(1), 
            premultiplied(false),
            activation_function(Rectifier),
            output_embedding_matrix(Matrix<double,Dynamic,Dynamic,Eigen::RowMajor>()),
            input_embedding_matrix(Matrix<double,Dynamic,Dynamic,Eigen::RowMajor>())
        {
          output_layer.set_W(&output_embedding_matrix);
          input_layer.set_W(&input_embedding_matrix);
        }

接着会调用下面代码对其中一些成员变量初始化.

    nn.resize(myParam.ngram_size, myParam.input_vocab_size,
        myParam.output_vocab_size, myParam.input_embedding_dimension,
        myParam.num_hidden, myParam.output_embedding_dimension);

    nn.initialize(rng, myParam.init_normal, myParam.init_range,
        -log(myParam.output_vocab_size), myParam.parameter_update,
        myParam.adagrad_epsilon);
    nn.set_activation_function(
        string_to_activation_function(myParam.activation_function));

其中resize方法

nplm代码学习与使用 (2)

TrainNeuralNetWork main() 运行流程
参数:
--train_file /home/li***/workspace/nplm/example/work/train.ngrams --validation_file /home/li***/workspace/nplm/example/work/validation.ngrams --num_epochs 10 --words_file /home/li***/workspace/nplm/example/work/words --model_prefix /home/li***/workspace/nplm/example/work/inferno.nnlm --learning_rate 1 --minibatch_size 8

1) 读取并获取各参数

2) 读取训练数据

  // Read training data

  vector<int> training_data_flat;
  vec * training_data_flat_mmap;
  data_size_t training_data_size; //num_tokens;
  ip::managed_mapped_file mmap_file;
  if (use_mmap_file == false) {
    cerr << "Reading data from regular text file " << endl;
    readDataFile(myParam.train_file, myParam.ngram_size, training_data_flat,
        myParam.minibatch_size);
    training_data_size = training_data_flat.size() / myParam.ngram_size;
  }

readDataFile()方法会读取train.ngrams中所有的数据, 并放入到vector training_data_flat中, 即training_data_flat[0 ~ 2]为第一个ngram (这里ngram = 3-gram), training_data_flat[3~ 5]为第二个ngram, 依此类推.

  Matrix<int, Dynamic, Dynamic> training_data;
  //(training_data_flat.data(), myParam.ngram_size, training_data_size);

#ifdef MAP
  cerr<<"Setting up eigen map"<<endl;
  if (use_mmap_file == false) {
    training_data = Map< Matrix<int,Dynamic,Dynamic> >(training_data_flat.data(), myParam.ngram_size, training_data_size);
  } else {
    training_data = Map< Matrix<int,Dynamic,Dynamic> >(training_data_flat_mmap->data().get(), myParam.ngram_size, training_data_size);
  }
  cerr<<"Created eigen map"<<endl;
#else
  if (use_mmap_file == false) {
    training_data = Map<Matrix<int, Dynamic, Dynamic> >(
        training_data_flat.data(), myParam.ngram_size, training_data_size);
  }
#endif

由于MAP没有预定义,运行training_data = Map< Matrix<int dynamic=""> >(training_data_flat.data(), myParam.ngram_size, training_data_size);得到的结果是,生成一个int类型的二维矩陈,其中行数为myParam.ngram_size (即3), 列数为training_data_size (即ngram的样例数).　二维矩阵的值可能根据vector training_data_flat进行初始化. 接着,

  
if (use_mmap_file == false && randomize == true) {
    cerr << "Randomly shuffling data..." << endl;
    // Randomly shuffle training data to improve learning
    for (data_size_t i = training_data_size - 1; i > 0; i--) {
      data_size_t j = uniform_int_distribution<data_size_t>(0, i - 1)(rng);
      training_data.col(i).swap(training_data.col(j));
    }
  }

对training_data中的一些进行替换, 即对数据进行重新洗牌.

 3) 读取校正数据

  // Read validation data
  vector<int> validation_data_flat;
  int validation_data_size = 0;

  if (myParam.validation_file != "") {
    readDataFile(myParam.validation_file, myParam.ngram_size,
        validation_data_flat);
    validation_data_size = validation_data_flat.size() / myParam.ngram_size;
    cerr << "Number of validation instances: " << validation_data_size << endl;
  }

  Map<Matrix<int, Dynamic, Dynamic> > validation_data(
      validation_data_flat.data(), myParam.ngram_size, validation_data_size);

与训练数据处理的类似,　得到校正数据的二维矩阵validation_data

 4) 读取输入单词/和输出单词文件

  vector<string> input_words;
  if (myParam.input_words_file != "") {
    readWordsFile(myParam.input_words_file, input_words);
    if (myParam.input_vocab_size == 0)
      myParam.input_vocab_size = input_words.size();
  }



  vector<string> output_words;
  if (myParam.output_words_file != "") {
    readWordsFile(myParam.output_words_file, output_words);
    if (myParam.output_vocab_size == 0)
      myParam.output_vocab_size = output_words.size();
  }

对每个ngram,　前n-1项数字对应的单词可根据输入单词文件找到原型. 而第n项数字对应的单词可根据输出单词文件找到原型.

 5)

  vector<data_size_t> unigram_counts(myParam.output_vocab_size);
  for (data_size_t train_id = 0; train_id < training_data_size; train_id++) {
    int output_word;
    if (use_mmap_file == false) {
      output_word = training_data(myParam.ngram_size - 1, train_id);
    } else {
      //cerr<<"mmap word is "<<training_data_flat_mmap->at((train_id+1)*myParam.ngram_size - 1)<<endl;
      output_word = training_data_flat_mmap->at(
          (train_id + 1) * myParam.ngram_size - 1);
    }
    //cerr<<"output word is "<<output_word<<endl;
    unigram_counts[output_word] += 1;
  }
  multinomial<data_size_t> unigram(unigram_counts);

unigram_counts统计每个输出单词出现的次数, unigram_counts[i]表示编号为i的单词出现在ngram最后一项的次数. unigram为一个多项式分布变量,以unigram_counts初始化每个输出单词的概率.   5) 模型训练

  model nn;
  // IF THE MODEL FILE HAS BEEN DEFINED, THEN
  // LOAD THE NEURAL NETWORK MODEL
  if (myParam.model_file != "") {
    nn.read(myParam.model_file);
    cerr << "reading the model" << endl;
  } else {
    nn.resize(myParam.ngram_size, myParam.input_vocab_size,
        myParam.output_vocab_size, myParam.input_embedding_dimension,
        myParam.num_hidden, myParam.output_embedding_dimension);

    nn.initialize(rng, myParam.init_normal, myParam.init_range,
        -log(myParam.output_vocab_size), myParam.parameter_update,
        myParam.adagrad_epsilon);
    nn.set_activation_function(
        string_to_activation_function(myParam.activation_function));
  }
  loss_function_type loss_function = string_to_loss_function(
      myParam.loss_function);

在model类(model.h)中, 定义了:

    Matrix<double,Dynamic,Dynamic,Eigen::RowMajor> output_embedding_matrix,
      input_embedding_matrix,
      input_and_output_embedding_matrix;

    Input_word_embeddings input_layer;
    Output_word_embeddings output_layer;

其中input_embedding_matrix和output_embedding_matrix都是动态维数的double类型矩阵.

nplm代码学习与使用 (1)

一. 下载/编译/示例运行
1) 从http://nlg.isi.edu/software/nplm/ 下载nplm-0.3.tar.gz
2) 修改src/Makefile (根据其他软件在本机安装的路径)

CXX=/usr/bin/g++
BOOST=/usr/local
EIGEN=/usr/local/include/eigen3
PYTHON_ROOT=/usr
BOOST_LIB_SUFFIX=

3) 切换至src目录下, 编译: make all

4) 切换至example目录下, 运行: make
根据屏幕上打印信息可以看到运行了以下命令:
a.
sed -n 10339,15586p pg8800.txt | ./preprocess.pl > inferno.txt
准备数据, 生成文本文件inferno.txt, 即该文件为训练语言模型的文件.

b.
./train_ngram.sh inferno.txt inferno.nnlm work
该命令进一步分别执行以下命令:
I:
/home/li***/toolkit/NEURAL_LANGUAGE_MODEL/example/../src/prepareNeuralLM --train_text inferno.txt --ngram_size 3 --vocab_size 5000 --validation_size 500 --write_words_file work/words --train_file work/train.ngrams --validation_file work/validation.ngrams
数据准备
以上读取文本文件inferno.txt, 生成词汇文件work/words,　并对单词数字化,　数字化后生成3-gram的训练文件work/train.ngrams和校正文件work/validation.ngrams.　

II:
/home/li***/toolkit/NEURAL_LANGUAGE_MODEL/example/../src/trainNeuralNetwork --train_file work/train.ngrams --validation_file work/validation.ngrams --num_epochs 10 --words_file work/words --model_prefix work/inferno.nnlm --learning_rate 1 --minibatch_size 8
训练模型

III:
/home/li***/toolkit/NEURAL_LANGUAGE_MODEL/example/../src/testNeuralNetwork --test_file work/train.ngrams --model_file inferno.nnlm
测试

二. 加载到eclipse项目中
1) 新建eclipse项目nplm, 并将以上src文件夹中的代码(包括Makefile)复制到nplm文件夹中. 为了使用已有的makefile文件, 需要告诉eclipse不自动生成makefile文件.　
Project Properties (right click on project and select properties) -> C/C++ Build -> in that window uncheck "Generate Makefiles Automatically."
再将src同一目录夹下的3rdparty文件夹复制到nplm中.
2) 修改Makefile文件 (在以上修改Makefile的基础之上):
TCLAP=./3rdparty/tclap/include
CFLAGS=-g
2) 为每个main创建一个target

这样可以在eclipse下调试各个main方法.