notebook/DL_nlp_text_similarity/text_similarity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自然语言处理实战 —— 文本相似度\n",
    "\n",
    "在自然语言处理（NLP）过程中，经常会涉及到如何度量两个文本之间的相似性。文本相似度（Semantic Text Similarity）计算就是判断两段文本之间是否相似。相似度有多种粒度的表示方式，可以使用（1,2,3,4,5）中的一个数字表示相似度，值越大表示越相似；也可以使用（0,1）中的一个数字表示相似度，1表示相似，0表示不相似。在本案例中，我们使用（0,1）表示相似度。文本相似度技术可以应用到信息检索、自动问答、机器翻译和自动文摘等NLP任务中。\n",
    "\n",
    "度量文本相似度包括如下三种方法：\n",
    "\n",
    "1. 基于关键词匹配的传统方法，如N-gram相似度；\n",
    "\n",
    "2. 将文本映射到向量空间，再利用余弦相似度等方法；\n",
    "\n",
    "3. 深度学习的方法，如基于用户点击数据的深度学习语义匹配模型DSSM，基于卷积神经网络的ConvNet等方法。 \n",
    "\n",
    "本案例中将使用深度学习的 **BERT** 模型进行文本相似度计算。\n",
    "\n",
    "中文相似度按照长度可以有字与字的相似度、单词与单词的相似度、句子与句子的相似度、段落与段落的相似度和文章与文章的相似度。\n",
    "\n",
    "本案例主要介绍一种基于词嵌入的中文短句文本相似度计算方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 进入ModelArts\n",
    "\n",
    "点击如下链接：https://www.huaweicloud.com/product/modelarts.html ， 进入ModelArts主页。点击“立即使用”按钮，输入用户名和密码登录，进入ModelArts使用页面。\n",
    "\n",
    "### 创建ModelArts notebook\n",
    "\n",
    "下面，我们在ModelArts中创建一个notebook开发环境，ModelArts notebook提供网页版的Python开发环境，可以方便的编写、运行代码，并查看运行结果。\n",
    "\n",
    "第一步：在ModelArts服务主界面依次点击“开发环境”、“创建”\n",
    "\n",
    "![create_nb_create_button](./img/create_nb_create_button.png)\n",
    "\n",
    "第二步：填写notebook所需的参数：\n",
    "\n",
    "| 参数 | 说明 |\n",
    "| - - - - - | - - - - - |\n",
    "| 计费方式 | 按需计费  |\n",
    "| 名称 | Notebook实例名称，如 text_sentiment_analysis |\n",
    "| 工作环境 | Python3 |\n",
    "| 资源池 | 选择\"公共资源池\"即可 |\n",
    "| 类型 | 本案例使用较为复杂的深度神经网络模型，需要较高算力，选择\"GPU\" |\n",
    "| 规格 | 选择\"8核 &#124; 64GiB &#124; 1*p100\" |\n",
    "| 存储配置 | 选择EVS，磁盘规格5GB |\n",
    "\n",
    "第三步：配置好notebook参数后，点击下一步，进入notebook信息预览。确认无误后，点击“立即创建”\n",
    "\n",
    "![create_nb_creation_summary](./img/create_nb_creation_summary.png)\n",
    "\n",
    "第四步：创建完成后，返回开发环境主界面，等待Notebook创建完毕后，打开Notebook，进行下一步操作。\n",
    "![modelarts_notebook_index](./img/modelarts_notebook_index.png)\n",
    "\n",
    "### 在ModelArts中创建开发环境\n",
    "\n",
    "接下来，我们创建一个实际的开发环境，用于后续的实验步骤。\n",
    "\n",
    "第一步：点击下图所示的“打开”按钮，进入刚刚创建的Notebook\n",
    "![inter_dev_env](img/enter_dev_env.png)\n",
    "\n",
    "第二步：创建一个Python3环境的的Notebook。点击右上角的\"New\"，然后创建TensorFlow 1.13.1开发环境。\n",
    "\n",
    "第三步：点击左上方的文件名\"Untitled\"，并输入一个与本实验相关的名称\n",
    "![notebook_untitled_filename](./img/notebook_untitled_filename.png)\n",
    "![notebook_name_the_ipynb](./img/notebook_name_the_ipynb.png)\n",
    "\n",
    "\n",
    "### 在Notebook中编写并执行代码\n",
    "\n",
    "在Notebook中，我们输入一个简单的打印语句，然后点击上方的运行按钮，可以查看语句执行的结果：\n",
    "![run_helloworld](./img/run_helloworld.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文本相似度计算\n",
    "\n",
    "\n",
    "\n",
    "### 数据集\n",
    "\n",
    "本案例采用西安科技大学提供的中文文本相似度语料库。相似度值：（0,1），0表示不相似，1表示相似。\n",
    "\n",
    "数据格式：\n",
    "\n",
    "| 字段 | Quality | #1 ID      | #2 ID        | #1 String  | #2 String  |\n",
    "| ---- | ------- | ---------- | ------------ | ---------- | ---------- |\n",
    "| 含义 | 相似度  | 第一句编号 | 第二句的编号 | 第一句文本 | 第二句文本 |\n",
    "\n",
    "\n",
    "### BERT 模型\n",
    "\n",
    "本实践使用 NLP 领域最新最强大的 **BERT** 模型。\n",
    "\n",
    "中文**BERT-Base,Chinese**预训练模型，可以从链接[BERT-Base, Chinese](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)下载使用。\n",
    "\n",
    "#### 准备源代码和数据\n",
    "\n",
    "准备案例所需的源代码和数据，相关资源已经保存在 OBS 中，我们通过 ModelArts SDK 将资源下载到本地。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully download file modelarts-labs/notebook/DL_nlp_text_similarity/text_similarity.tar.gz from OBS to local ./text_similarity.tar.gz\n",
      "total 375896\r\n",
      "drwxrwxrwx  4 ma-user ma-group      4096 Sep 29 10:12 .\r\n",
      "drwsrwsr-x 22 ma-user ma-group      4096 Sep 29 10:10 ..\r\n",
      "drwxr-x---  2 ma-user ma-group      4096 Sep 29 10:00 .ipynb_checkpoints\r\n",
      "-rw-r-----  1 ma-user ma-group   1855715 Sep 29 10:10 text_similarity.ipynb\r\n",
      "-rw-r-----  1 ma-user ma-group 383037805 Sep 29 10:12 text_similarity.tar.gz\r\n",
      "drwx------  2 ma-user ma-group      4096 Sep 29 10:12 .Trash-1000\r\n"
     ]
    }
   ],
   "source": [
    "from modelarts.session import Session\n",
    "session = Session()\n",
    "\n",
    "if session.region_name == 'cn-north-1':\n",
    "    bucket_path = 'modelarts-labs/notebook/DL_nlp_text_similarity/text_similarity.tar.gz'\n",
    "    \n",
    "elif session.region_name == 'cn-north-4':\n",
    "    bucket_path = 'modelarts-labs-bj4/notebook/DL_nlp_text_similarity/text_similarity.tar.gz'\n",
    "else:\n",
    "    print(\"请更换地区到北京一或北京四\")\n",
    "    \n",
    "session.download_data(bucket_path=bucket_path, path='./text_similarity.tar.gz')\n",
    "\n",
    "!ls -la"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解压从obs下载的压缩包，解压后删除压缩包。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 1836\r\n",
      "drwxrwxrwx  5 ma-user ma-group    4096 Sep 29 10:12 .\r\n",
      "drwsrwsr-x 22 ma-user ma-group    4096 Sep 29 10:10 ..\r\n",
      "drwxr-x---  2 ma-user ma-group    4096 Sep 29 10:00 .ipynb_checkpoints\r\n",
      "drwxr-x---  6 ma-user ma-group    4096 Sep 24 18:12 text_similarity\r\n",
      "-rw-r-----  1 ma-user ma-group 1855715 Sep 29 10:10 text_similarity.ipynb\r\n",
      "drwx------  2 ma-user ma-group    4096 Sep 29 10:12 .Trash-1000\r\n"
     ]
    }
   ],
   "source": [
    "!tar xf ./text_similarity.tar.gz\n",
    "\n",
    "!rm ./text_similarity.tar.gz\n",
    "\n",
    "!ls -la"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 导入依赖包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import os\n",
    "import csv\n",
    "import collections\n",
    "from text_similarity.bert import modeling, optimization, tokenization\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 定义数据和模型路径"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# BERT模型配置文件\n",
    "bert_config_file = 'text_similarity/chinese_L-12_H-768_A-12/bert_config.json'\n",
    "vocab_file = 'text_similarity/chinese_L-12_H-768_A-12/vocab.txt'\n",
    "init_checkpoint = 'text_similarity/chinese_L-12_H-768_A-12/bert_model.ckpt'\n",
    "\n",
    "# 数据集路径\n",
    "data_dir = 'text_similarity/data/'\n",
    "\n",
    "# 模型训练输出位置\n",
    "output_dir = 'text_similarity/output/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 设置TensorFlow运行相关参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "label_list = [\"0\", \"1\"]\n",
    "do_lower_case = False\n",
    "is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2\n",
    "use_tpu = False\n",
    "tpu_cluster_resolver = None\n",
    "master = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 设置模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_batch_size=32\n",
    "eval_batch_size=8 \n",
    "predict_batch_size=8\n",
    "num_epochs = 5.0 \n",
    "warmup_proportion = 0.1 \n",
    "learning_rate = 2e-5 \n",
    "max_seq_length = 128 \n",
    "save_checkpoints_steps = 1000 \n",
    "iterations_per_loop = 1000 \n",
    "num_gpu_cores = 1 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取BERT预训练模型中文字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['今', '天', '的', '天', '气', '真', '好', '！']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = tokenization.FullTokenizer(vocab_file=vocab_file, do_lower_case=do_lower_case)\n",
    "\n",
    "tokenizer.tokenize(\"今天的天气真好！\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 创建数据输入类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class InputExample(object):\n",
    "\n",
    "  def __init__(self, guid, text_a, text_b=None, label=None):\n",
    "    self.guid = guid\n",
    "    self.text_a = text_a\n",
    "    self.text_b = text_b\n",
    "    self.label = label\n",
    "\n",
    "class InputFeatures(object):\n",
    "\n",
    "  def __init__(self,\n",
    "               input_ids,\n",
    "               input_mask,\n",
    "               segment_ids,\n",
    "               label_id,\n",
    "               is_real_example=True):\n",
    "    self.input_ids = input_ids\n",
    "    self.input_mask = input_mask\n",
    "    self.segment_ids = segment_ids\n",
    "    self.label_id = label_id\n",
    "    self.is_real_example = is_real_example\n",
    "    \n",
    "    \n",
    "class PaddingInputExample(object):\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取训练数据集\n",
    "\n",
    "数据集每行的格式为：相似度（Quality），第一句编号（1 ID），第二句的编号（2 ID），第一句文本（1 String），第二句文本（2 String）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "line 0 : ['Quality', '#1 ID', '#2 ID', '#1 String', '#2 String']\n",
      "line 1 : ['0', '636', '3175303', '老太太的情绪不稳定。', '这个模式对很多领域都很实用，尤其是数位版权管理方面，因为它的最小描述单元层次，可利用来指定识别码进行个别的筛选与授权使用。']\n",
      "line 2 : ['0', '922', '4606174', '事情几乎没办成。', '作为拜仁慕尼黑的青训产物，穆勒在2009至10赛季被时任主教练路易斯·范加尔提拔至一线队，他在该赛季几乎参加了队内的所有比赛，为球队赢得了联赛及杯赛双冠王，并且晋身欧洲冠军联赛决赛。']\n",
      "line 3 : ['0', '387', '1930624', '小明在家务上给妈妈帮了不少忙。', '中华民国的绥远省政府至此彻底消亡。']\n",
      "line 4 : ['1', '930', '3125', '好不难受。', '很不难受。']\n",
      "line 5 : ['0', '327', '1633734', '他下午也许来不了。', '从20世界90年代开始，哮喘的得病率在发达国家趋于平稳，而在发展中国家快速增长。']\n",
      "line 6 : ['1', '832', '2617', '假设真的没有文明和文化，那么这个世界就像个未成品。', '如果真的没有文明和文化，这个世界便像个未成品。']\n",
      "line 7 : ['1', '804', '2464', '当官不为民做主，不如回家卖红薯。', '假如当官不为民做主，还不如回家卖红薯。']\n",
      "line 8 : ['0', '801', '4000941', '必须尽快改变现状，否则我真的没有出路了。', '另外，由于每位选手必须要最少投球一次（捕手除外），故参加者多是全能球员。']\n",
      "line 9 : ['0', '574', '2866598', '路上有许多人在赶路。', '后来，站台上部玻璃板开始松动，考虑到玻璃板可能断裂车站立即紧急疏散。']\n",
      "line 10 : ['1', '788', '2089', '这一次，投资者仅仅是“有望收回成本”，换句话说，很可能赔本！', '这一次，投资者仅仅是“有望收回成本”，就是说，很可能赔本！']\n",
      "line 11 : ['0', '569', '2841282', '挑剔的母亲又逼着裁缝把做好的衣服修改了两次。', '银菊露在一般凉茶舖都有卖，功用有解毒解暑、清热明目，可以防止肝火眼痛，有保健预防的作用。']\n",
      "line 12 : ['0', '945', '4723074', '小心别打碎杯子。', '与前任法老荷尔-阿哈相似，哲尔葬于当时的圣地阿拜多斯。']\n",
      "line 13 : ['0', '342', '1705159', '他们家平常不外出旅游。', '）欧马利的远见也值得加分，在1958年以前，位于密苏里的城市通常就是大联盟球队最西方的边境，而如今的30个球队中，有12个将他们的根据地设在更西边。']\n",
      "line 14 : ['1', '313', '3363', '到了广州，不能不夜游珠江。', '到了广州，一定要夜游珠江。']\n",
      "line 15 : ['1', '1029', '3309', '他没努力去解决这个问题，这个问题对他本来就很容易。', '他没努力去解决这个问题，这个问题对他来说很容易。']\n",
      "line 16 : ['0', '553', '2760878', '墙角站着一个人。', '火花由通过一个安装在引擎前方凸轮轴端部的低电压计时器分配到火星塞产生，这种计时器就是现代的分电器的前身，它将电流导引安装在驾驶室前部一个箱子内的4个振动线圈，点火的计时调节是通过借助转向柱上的杠杆装置人工转动计时器来实现的。']\n",
      "line 17 : ['1', '49', '818', '我喝酒喝了半个月的工资。', '我用半个月的工资来喝酒。']\n",
      "line 18 : ['0', '49', '243572', '我做题做了三个小时。', '第三插部是全曲中唯一小调性的主题，它运用一个回音音型连续不断地出现于高音部和低音部，相互对比，加强乐曲的戏剧性，c小调、4/4拍子。']\n",
      "line 19 : ['0', '135', '671565', '有苦没处诉。', '一般的，从阿斯旺到瓦迪哈勒法之间的地区被称为下努比亚，从瓦迪蛤勒法到库赖迈之间的地区则被称为上努比亚。']\n",
      "line 20 : ['0', '281', '1404681', '准是瞎指挥，没个不出事儿。', '在推出双叉路口之前，她担任豪记唱片夜市走唱系列那卡西专辑与伴唱带的主唱2年。']\n",
      "line 21 : ['0', '552', '2756968', '笼屉里冒着热气。', '在这段时期，很多人都穿着中世纪华丽的服装，并且有大量的活动举行：音乐会，小丑表演，戏剧，中世纪市场，马术比赛等等。']\n",
      "line 22 : ['1', '278', '1323', '哪儿他都去过。', '他去过很多地方。']\n",
      "line 23 : ['0', '378', '1888954', '她用那把刀切菜。', '李渊募集兵士，以刘弘基和长孙顺德统领。']\n",
      "line 24 : ['0', '995', '4973071', '老李不是老师。', '倘若杀不死白兔子，就表示自己不是真正的爱丽丝。']\n",
      "line 25 : ['0', '366', '1826443', '古人管眼睛叫目。', '声明选秀资格的球员并不会自动失去大学比赛资格，除非在之前的年度声明了选秀资格并反悔。']\n",
      "line 26 : ['1', '146', '1235', '颗颗都挂了果。', '每课树上都有果子。']\n",
      "line 27 : ['1', '794', '2453', '在困难面前，或者当个懦弱的逃兵，或者做个勇猛的战士。', '在困难面前，不是做个勇猛的战士，就是当个懦弱的逃兵。']\n",
      "line 28 : ['0', '946', '4729776', '小心别擦破手。', '画中有个十分显眼的黑人在挥动手巾，这个名叫让·查理的非洲水手想通过这一方式吸引路过船只注意，而他右下面亦有人同他一齐求救。']\n",
      "line 29 : ['0', '124', '617149', '他这个人没个正经的。', '承上文，曹氏解丂字之说显然是错误的，而其分析形声别于转注之说，亦也就是错误。']\n",
      "line 30 : ['0', '377', '1882369', '他朝大山的方向走去。', '而实验发现屏上形成了几条清晰的黑斑，表明银原子的磁矩只能取几个特定的方向，从而验证了原子角动量的投影是量子化的。']\n",
      "line 31 : ['1', '834', '2643', '明天下雨的话，活动就会泡汤。', '若明天下雨的话，活动便会泡汤。']\n",
      "line 32 : ['0', '366', '1827660', '古人管眼睛叫目。', '全县境内有乡道12条：东至进贤北山，长40公里；']\n",
      "line 33 : ['1', '692', '1910', '吃了这剂药，过两天就会好的。', '把这剂药吃了，过两天就会好的。']\n",
      "line 34 : ['1', '428', '1198', '他难道会介意这些事情吗？', '这些事情难道他会介意？']\n",
      "line 35 : ['0', '409', '2040261', '他是个心胸宽广的人，哪里会介意这点小事。', '当初二次大战时英美是盟军，英国对于美军的物品取得也方便，当初不知起头者是谁，为了骑车或修车时，不让污渍沾染里面昂贵的西装，而穿上美军大衣，之后便争相模仿，也造就了Mods最明显的身上行头。']\n",
      "line 36 : ['0', '856', '4275363', '不管我们如何好说歹说，他却仍然是无动于衷。', '小行星2209']\n",
      "line 37 : ['0', '314', '1566505', '大家互相在纪念册上签名留念。', '小孩子则用自制的松明火药，手拿秸秆火把互相攻击（白族话为取）。']\n",
      "line 38 : ['0', '824', '4118000', '因前一段时间连降暴雨，以致工期一再拖延。', '英国音乐评论家形容这股浪潮的特色是结合欧式即兴重复段和喉音歌声。']\n",
      "line 39 : ['0', '587', '2930926', '你们拿了人家不少好处。', '香港的玄学家苏民峰曾在电视节目内指出：观塘区的山势本来是很差的，不过由于当地的公共房屋把原来难看的山势阻挡了，使新填海区再看不到原来的山势，反而只看到整齐的楼景。']\n",
      "line 40 : ['0', '152', '757274', '她都去了三次了。', '，但是也长久遭外界质疑其部分作为违背学术伦理。']\n",
      "line 41 : ['0', '805', '4022479', '任凭风吹浪打，都毫不动摇。', '卢贝新城人口变化图示']\n",
      "line 42 : ['0', '98', '485664', '那人看着都乏味。', '语言和文学。']\n",
      "line 43 : ['0', '766', '3826282', '我让她坐进我的三七炮位里，给她扣上我那沉重的钢盔，告诉她这炮火力相当猛烈。', '图图杯的抽签于2008年4月21日进行，结果如下：']\n",
      "line 44 : ['0', '657', '3284498', '在下次董事会上，一定得拿出个让大家都满意的新方案出来。', '高而深的豆、圈足碗、盘、杯为最重要的器型。']\n",
      "line 45 : ['0', '467', '2333015', '他一次会议都没参加。', '至于其他首相，如墨尔本勋爵和帕尔姆斯顿子爵，都只以唐宁街10号用作办公室和内阁会议场所。']\n",
      "line 46 : ['0', '467', '2333062', '他一次会议都没参加。', '踏入1950年代，唐宁街10号的楼宇结构安全渐渐成为一个极待解决的问题，建筑物沉降、墙壁倒塌和门框扭曲已经是司空见惯，而在内阁会议室内，有达200年历史，用作支撑楼房的房柱，更被发现只剩下了外表的清漆和油漆，内里的实心原木却早已经腐朽，几近灰尘。']\n",
      "line 47 : ['0', '834', '4168006', '全组人员正在马不停蹄地准备材料填写各种表格，以求能够按时向有关部门递交项目申请书。', '，但实际上，除了使用国际武联的标志替换国际奥委会的五环标志外，其余各项元素（祥云标志、奖牌、领奖服、升旗仪式、志愿者等）与奥运会正式项目并无区别，运动员也可入住奥运村内的专门区域，官方网站甚至可以查询到比赛成绩和运动员资料（但和正式项目所处的位置不同）']\n",
      "line 48 : ['1', '846', '2723', '我不知道该信哪一种说法，因此也就无法判断，。', '我无法判断，因为不知道该信哪一种说法。']\n",
      "line 49 : ['1', '865', '2838', '这段描写固然有些夸张，但我国古代宝刀宝剑之锋利，却非虚传。', '这段描写虽说有些夸张，但我国古代宝刀宝剑之锋利绝对不是虚传。']\n"
     ]
    }
   ],
   "source": [
    "def read_tsv(input_file, quotechar=None):\n",
    "    with tf.gfile.Open(input_file, \"r\") as f:\n",
    "        reader = csv.reader(f, delimiter=\"\\t\", quotechar=quotechar)\n",
    "        lines = []\n",
    "        i = 0\n",
    "        for line in reader:\n",
    "            lines.append(line)\n",
    "            if i < 50:\n",
    "                print('line', i, ':', line)\n",
    "                i += 1\n",
    "        return lines\n",
    "    \n",
    "def create_examples(lines, set_type):\n",
    "    examples = []\n",
    "    for (i, line) in enumerate(lines):\n",
    "      if i == 0:\n",
    "        continue\n",
    "      guid = \"%s-%s\" % (set_type, i)\n",
    "      text_a = tokenization.convert_to_unicode(line[3])\n",
    "      text_b = tokenization.convert_to_unicode(line[4])\n",
    "      if set_type == \"test\":\n",
    "        label = \"0\"\n",
    "      else:\n",
    "        label = tokenization.convert_to_unicode(line[0])\n",
    "      examples.append(\n",
    "          InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))\n",
    "    return examples\n",
    "\n",
    "def get_train_examples(data_dir):\n",
    "    return create_examples(read_tsv(os.path.join(data_dir, \"train.tsv\")), \"train\")\n",
    "\n",
    "train_examples = get_train_examples(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 转换为 BERT 输入向量\n",
    "\n",
    "打印前5个样例文本，及其字向量、文本向量、位置向量和标签。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 0 of 9416\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-1\n",
      "INFO:tensorflow:tokens: [CLS] 老 太 太 的 情 绪 不 稳 定 。 [SEP] 这 个 模 式 对 很 多 领 域 都 很 实 用 ， 尤 其 是 数 位 版 权 管 理 方 面 ， 因 为 它 的 最 小 描 述 单 元 层 次 ， 可 利 用 来 指 定 识 别 码 进 行 个 别 的 筛 选 与 授 权 使 用 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 5439 1922 1922 4638 2658 5328 679 4937 2137 511 102 6821 702 3563 2466 2190 2523 1914 7566 1818 6963 2523 2141 4500 8024 2215 1071 3221 3144 855 4276 3326 5052 4415 3175 7481 8024 1728 711 2124 4638 3297 2207 2989 6835 1296 1039 2231 3613 8024 1377 1164 4500 3341 2900 2137 6399 1166 4772 6822 6121 702 1166 4638 5033 6848 680 2956 3326 886 4500 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-2\n",
      "INFO:tensorflow:tokens: [CLS] 事 情 几 乎 没 办 成 。 [SEP] 作 为 拜 仁 慕 尼 黑 的 青 训 产 物 ， 穆 勒 在 2009 至 10 赛 季 被 时 任 主 教 练 路 易 斯 · 范 加 尔 提 拔 至 一 线 队 ， 他 在 该 赛 季 几 乎 参 加 了 队 内 的 所 有 比 赛 ， 为 球 队 赢 得 了 联 赛 及 杯 赛 双 冠 王 ， 并 且 晋 身 欧 洲 冠 军 联 赛 决 赛 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 752 2658 1126 725 3766 1215 2768 511 102 868 711 2876 785 2710 2225 7946 4638 7471 6378 772 4289 8024 4946 1239 1762 8170 5635 8108 6612 2108 6158 3198 818 712 3136 5298 6662 3211 3172 185 5745 1217 2209 2990 2869 5635 671 5296 7339 8024 800 1762 6421 6612 2108 1126 725 1346 1217 749 7339 1079 4638 2792 3300 3683 6612 8024 711 4413 7339 6617 2533 749 5468 6612 1350 3344 6612 1352 1094 4374 8024 2400 684 3232 6716 3616 3828 1094 1092 5468 6612 1104 6612 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-3\n",
      "INFO:tensorflow:tokens: [CLS] 小 明 在 家 务 上 给 妈 妈 帮 了 不 少 忙 。 [SEP] 中 华 民 国 的 绥 远 省 政 府 至 此 彻 底 消 亡 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 2207 3209 1762 2157 1218 677 5314 1968 1968 2376 749 679 2208 2564 511 102 704 1290 3696 1744 4638 5324 6823 4689 3124 2424 5635 3634 2515 2419 3867 767 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-4\n",
      "INFO:tensorflow:tokens: [CLS] 好 不 难 受 。 [SEP] 很 不 难 受 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 1962 679 7410 1358 511 102 2523 679 7410 1358 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 1 (id = 1)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: train-5\n",
      "INFO:tensorflow:tokens: [CLS] 他 下 午 也 许 来 不 了 。 [SEP] 从 20 世 界 90 年 代 开 始 ， 哮 喘 的 得 病 率 在 发 达 国 家 趋 于 平 稳 ， 而 在 发 展 中 国 家 快 速 增 长 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 800 678 1286 738 6387 3341 679 749 511 102 794 8113 686 4518 8192 2399 807 2458 1993 8024 1527 1596 4638 2533 4567 4372 1762 1355 6809 1744 2157 6633 754 2398 4937 8024 5445 1762 1355 2245 704 1744 2157 2571 6862 1872 7270 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    }
   ],
   "source": [
    "def truncate_seq_pair(tokens_a, tokens_b, max_length):\n",
    "    while True:\n",
    "        total_length = len(tokens_a) + len(tokens_b)\n",
    "        if total_length <= max_length:\n",
    "            break\n",
    "        if len(tokens_a) > len(tokens_b):\n",
    "            tokens_a.pop()\n",
    "        else:\n",
    "            tokens_b.pop()\n",
    "\n",
    "\n",
    "def convert_single_example(ex_index, example, label_list, max_seq_length,\n",
    "                           tokenizer):\n",
    "\n",
    "    if isinstance(example, PaddingInputExample):\n",
    "        return InputFeatures(\n",
    "            input_ids=[0] * max_seq_length,\n",
    "            input_mask=[0] * max_seq_length,\n",
    "            segment_ids=[0] * max_seq_length,\n",
    "            label_id=0,\n",
    "            is_real_example=False)\n",
    "    \n",
    "    label_map = {}\n",
    "    for (i, label) in enumerate(label_list):\n",
    "        label_map[label] = i\n",
    "\n",
    "    tokens_a = tokenizer.tokenize(example.text_a)\n",
    "    tokens_b = None\n",
    "    if example.text_b:\n",
    "        tokens_b = tokenizer.tokenize(example.text_b)\n",
    "\n",
    "    if tokens_b:\n",
    "        truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)\n",
    "    else:\n",
    "        if len(tokens_a) > max_seq_length - 2:\n",
    "            tokens_a = tokens_a[0:(max_seq_length - 2)]\n",
    "\n",
    "    tokens = []\n",
    "    segment_ids = []\n",
    "    tokens.append(\"[CLS]\") # 句头添加 [CLS] 标志\n",
    "    segment_ids.append(0)\n",
    "    for token in tokens_a:\n",
    "        tokens.append(token)\n",
    "        segment_ids.append(0)\n",
    "    tokens.append(\"[SEP]\") # 句尾添加[SEP] 标志\n",
    "    segment_ids.append(0)\n",
    "\n",
    "    if tokens_b:\n",
    "        for token in tokens_b:\n",
    "            tokens.append(token)\n",
    "            segment_ids.append(1)\n",
    "        tokens.append(\"[SEP]\")\n",
    "        segment_ids.append(1)\n",
    "\n",
    "    input_ids = tokenizer.convert_tokens_to_ids(tokens)  \n",
    "    input_mask = [1] * len(input_ids)\n",
    "\n",
    "    while len(input_ids) < max_seq_length:\n",
    "        input_ids.append(0)\n",
    "        input_mask.append(0)\n",
    "        segment_ids.append(0)\n",
    "\n",
    "    assert len(input_ids) == max_seq_length\n",
    "    assert len(input_mask) == max_seq_length\n",
    "    assert len(segment_ids) == max_seq_length\n",
    "\n",
    "    label_id = label_map[example.label]\n",
    "    \n",
    "    if ex_index < 5:\n",
    "        tf.logging.info(\"*** Example ***\")\n",
    "        tf.logging.info(\"guid: %s\" % (example.guid)) \n",
    "        tf.logging.info(\"tokens: %s\" % \" \".join([tokenization.printable_text(x) for x in tokens])) \n",
    "        tf.logging.info(\"input_ids: %s\" % \" \".join([str(x) for x in input_ids]))  \n",
    "        tf.logging.info(\"input_mask: %s\" % \" \".join([str(x) for x in input_mask])) \n",
    "        tf.logging.info(\"segment_ids: %s\" % \" \".join([str(x) for x in segment_ids])) \n",
    "        tf.logging.info(\"label: %s (id = %d)\" % (example.label, label_id)) \n",
    "\n",
    "    feature = InputFeatures(\n",
    "        input_ids=input_ids,\n",
    "        input_mask=input_mask,\n",
    "        segment_ids=segment_ids,\n",
    "        label_id=label_id,\n",
    "        is_real_example=True)\n",
    "    return feature\n",
    "\n",
    "\n",
    "def file_based_convert_examples_to_features(examples, label_list, max_seq_length, tokenizer, output_file):\n",
    "    writer = tf.python_io.TFRecordWriter(output_file)\n",
    "\n",
    "    for (ex_index, example) in enumerate(examples):\n",
    "        if ex_index % 10000 == 0:\n",
    "            tf.logging.info(\"Writing example %d of %d\" % (ex_index, len(examples)))\n",
    "\n",
    "        feature = convert_single_example(ex_index, example, label_list, max_seq_length, tokenizer)\n",
    "        def create_int_feature(values):\n",
    "            f = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values)))\n",
    "            return f\n",
    "\n",
    "        features = collections.OrderedDict()\n",
    "        features[\"input_ids\"] = create_int_feature(feature.input_ids)\n",
    "        features[\"input_mask\"] = create_int_feature(feature.input_mask)\n",
    "        features[\"segment_ids\"] = create_int_feature(feature.segment_ids)\n",
    "        features[\"label_ids\"] = create_int_feature([feature.label_id])\n",
    "        features[\"is_real_example\"] = create_int_feature([int(feature.is_real_example)])\n",
    "\n",
    "        tf_example = tf.train.Example(features=tf.train.Features(feature=features))\n",
    "        writer.write(tf_example.SerializeToString())\n",
    "    writer.close()\n",
    "\n",
    "train_file = os.path.join(output_dir, \"train.tf_record\")\n",
    "\n",
    "file_based_convert_examples_to_features(train_examples, label_list, max_seq_length, tokenizer, train_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 加载模型参数，构造模型结构"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "bert_config = modeling.BertConfig.from_json_file(bert_config_file)\n",
    "\n",
    "def create_model(bert_config, is_training, input_ids, input_mask, segment_ids,\n",
    "                 labels, num_labels, use_one_hot_embeddings):\n",
    "    \n",
    "    model = modeling.BertModel(\n",
    "        config=bert_config,\n",
    "        is_training=is_training,\n",
    "        input_ids=input_ids,\n",
    "        input_mask=input_mask,\n",
    "        token_type_ids=segment_ids,\n",
    "        use_one_hot_embeddings=use_one_hot_embeddings)\n",
    "\n",
    "    output_layer = model.get_pooled_output()\n",
    "    hidden_size = output_layer.shape[-1].value\n",
    "\n",
    "    output_weights = tf.get_variable(\n",
    "        \"output_weights\", [num_labels, hidden_size],\n",
    "        initializer=tf.truncated_normal_initializer(stddev=0.02))\n",
    "\n",
    "    output_bias = tf.get_variable(\"output_bias\", [num_labels], initializer=tf.zeros_initializer())\n",
    "\n",
    "    with tf.variable_scope(\"loss\"):\n",
    "        if is_training:\n",
    "            output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)\n",
    "\n",
    "        logits = tf.matmul(output_layer, output_weights, transpose_b=True)\n",
    "        logits = tf.nn.bias_add(logits, output_bias)\n",
    "        probabilities = tf.nn.softmax(logits, axis=-1)\n",
    "        log_probs = tf.nn.log_softmax(logits, axis=-1)\n",
    "\n",
    "        one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)\n",
    "\n",
    "        per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)\n",
    "        loss = tf.reduce_mean(per_example_loss)\n",
    "\n",
    "        return (loss, per_example_loss, logits, probabilities)\n",
    "\n",
    "def model_fn_builder(bert_config, num_labels, init_checkpoint, learning_rate,\n",
    "                     num_train_steps, num_warmup_steps, use_tpu,\n",
    "                     use_one_hot_embeddings):\n",
    "\n",
    "  def model_fn(features, labels, mode, params):\n",
    "\n",
    "    tf.logging.info(\"*** Features ***\")\n",
    "    for name in sorted(features.keys()):\n",
    "      tf.logging.info(\"  name = %s, shape = %s\" % (name, features[name].shape))\n",
    "\n",
    "    input_ids = features[\"input_ids\"]\n",
    "    input_mask = features[\"input_mask\"]\n",
    "    segment_ids = features[\"segment_ids\"]\n",
    "    label_ids = features[\"label_ids\"]\n",
    "    is_real_example = None\n",
    "    if \"is_real_example\" in features:\n",
    "      is_real_example = tf.cast(features[\"is_real_example\"], dtype=tf.float32)\n",
    "    else:\n",
    "      is_real_example = tf.ones(tf.shape(label_ids), dtype=tf.float32)\n",
    "\n",
    "    is_training = (mode == tf.estimator.ModeKeys.TRAIN)\n",
    "\n",
    "    (total_loss, per_example_loss, logits, probabilities) = create_model(\n",
    "        bert_config, is_training, input_ids, input_mask, segment_ids, label_ids,\n",
    "        num_labels, use_one_hot_embeddings)\n",
    "\n",
    "    tvars = tf.trainable_variables()\n",
    "    initialized_variable_names = {}\n",
    "    scaffold_fn = None\n",
    "    if init_checkpoint:\n",
    "      (assignment_map, initialized_variable_names) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)\n",
    "      tf.train.init_from_checkpoint(init_checkpoint, assignment_map)\n",
    "\n",
    "    tf.logging.info(\"**** Trainable Variables ****\")\n",
    "    for var in tvars:\n",
    "      init_string = \"\"\n",
    "      if var.name in initialized_variable_names:\n",
    "        init_string = \", *INIT_FROM_CKPT*\"\n",
    "      tf.logging.info(\"  name = %s, shape = %s%s\", var.name, var.shape,\n",
    "                      init_string)\n",
    "\n",
    "    output_spec = None\n",
    "\n",
    "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "\n",
    "      train_op = optimization.create_optimizer(\n",
    "          total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)\n",
    "\n",
    "      output_spec = tf.contrib.tpu.TPUEstimatorSpec(\n",
    "          mode=mode,\n",
    "          loss=total_loss,\n",
    "          train_op=train_op,\n",
    "          scaffold_fn=scaffold_fn)\n",
    "\n",
    "    elif mode == tf.estimator.ModeKeys.EVAL:\n",
    "\n",
    "      def metric_fn(per_example_loss, label_ids, logits, is_real_example):\n",
    "        predictions = tf.argmax(logits, axis=-1, output_type=tf.int32)\n",
    "        accuracy = tf.metrics.accuracy(\n",
    "            labels=label_ids, predictions=predictions, weights=is_real_example)\n",
    "        loss = tf.metrics.mean(values=per_example_loss, weights=is_real_example)\n",
    "        return {\n",
    "            \"eval_accuracy\": accuracy,\n",
    "            \"eval_loss\": loss,\n",
    "        }\n",
    "\n",
    "      eval_metrics = (metric_fn, [per_example_loss, label_ids, logits, is_real_example])\n",
    "      output_spec = tf.contrib.tpu.TPUEstimatorSpec(\n",
    "          mode=mode,\n",
    "          loss=total_loss,\n",
    "          eval_metrics=eval_metrics,\n",
    "          scaffold_fn=scaffold_fn)\n",
    "\n",
    "    else:\n",
    "      output_spec = tf.contrib.tpu.TPUEstimatorSpec(\n",
    "          mode=mode,\n",
    "          predictions={\"probabilities\": probabilities},\n",
    "          scaffold_fn=scaffold_fn)\n",
    "    return output_spec\n",
    "  return model_fn\n",
    "\n",
    "\n",
    "num_train_steps = int(len(train_examples) / train_batch_size * num_epochs)\n",
    "num_warmup_steps = int(num_train_steps * warmup_proportion)\n",
    "\n",
    "model_fn = model_fn_builder(\n",
    "    bert_config=bert_config,\n",
    "    num_labels=len(label_list),\n",
    "    init_checkpoint=init_checkpoint,\n",
    "    learning_rate=learning_rate,\n",
    "    num_train_steps=num_train_steps,\n",
    "    num_warmup_steps=num_warmup_steps,\n",
    "    use_tpu=use_tpu,\n",
    "    use_one_hot_embeddings=use_tpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7f42c22e6bf8>) includes params argument, but params are not passed to Estimator.\n",
      "INFO:tensorflow:Using config: {'_model_dir': 'text_similarity/output/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f42c4092240>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=1, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}\n",
      "WARNING:tensorflow:Setting TPUConfig.num_shards==1 is an unsupported behavior. Please fix as soon as possible (leaving num_shards as None.)\n",
      "INFO:tensorflow:_TPUContext: eval_on_tpu True\n",
      "WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.\n",
      "INFO:tensorflow:***** Running training *****\n",
      "INFO:tensorflow:  Num examples = 9416\n",
      "INFO:tensorflow:  Batch size = 32\n",
      "INFO:tensorflow:  Num steps = 1471\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From <ipython-input-12-ab1fa162cd77>:33: map_and_batch (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.experimental.map_and_batch(...)`.\n",
      "WARNING:tensorflow:From <ipython-input-12-ab1fa162cd77>:17: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Running train on CPU\n",
      "INFO:tensorflow:*** Features ***\n",
      "INFO:tensorflow:  name = input_ids, shape = (?, 128)\n",
      "INFO:tensorflow:  name = input_mask, shape = (?, 128)\n",
      "INFO:tensorflow:  name = is_real_example, shape = (?,)\n",
      "INFO:tensorflow:  name = label_ids, shape = (?,)\n",
      "INFO:tensorflow:  name = segment_ids, shape = (?, 128)\n",
      "WARNING:tensorflow:From /home/ma-user/work/text_similarity/bert/modeling.py:358: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From /home/ma-user/work/text_similarity/bert/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n",
      "INFO:tensorflow:**** Trainable Variables ****\n",
      "INFO:tensorflow:  name = bert/embeddings/word_embeddings:0, shape = (21128, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = output_weights:0, shape = (2, 768)\n",
      "INFO:tensorflow:  name = output_bias:0, shape = (2,)\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Saving checkpoints for 0 into text_similarity/output/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.85518\n",
      "INFO:tensorflow:examples/sec: 59.3657\n",
      "INFO:tensorflow:global_step/sec: 2.08442\n",
      "INFO:tensorflow:examples/sec: 66.7015\n",
      "INFO:tensorflow:global_step/sec: 2.08459\n",
      "INFO:tensorflow:examples/sec: 66.7069\n",
      "INFO:tensorflow:global_step/sec: 2.08437\n",
      "INFO:tensorflow:examples/sec: 66.6999\n",
      "INFO:tensorflow:global_step/sec: 2.0858\n",
      "INFO:tensorflow:examples/sec: 66.7456\n",
      "INFO:tensorflow:global_step/sec: 2.08543\n",
      "INFO:tensorflow:examples/sec: 66.7339\n",
      "INFO:tensorflow:global_step/sec: 2.08572\n",
      "INFO:tensorflow:examples/sec: 66.7429\n",
      "INFO:tensorflow:global_step/sec: 2.08546\n",
      "INFO:tensorflow:examples/sec: 66.7347\n",
      "INFO:tensorflow:global_step/sec: 2.0856\n",
      "INFO:tensorflow:examples/sec: 66.7391\n",
      "INFO:tensorflow:Saving checkpoints for 1000 into text_similarity/output/model.ckpt.\n",
      "INFO:tensorflow:global_step/sec: 1.98411\n",
      "INFO:tensorflow:examples/sec: 63.4916\n",
      "INFO:tensorflow:global_step/sec: 2.08471\n",
      "INFO:tensorflow:examples/sec: 66.7107\n",
      "INFO:tensorflow:global_step/sec: 2.08407\n",
      "INFO:tensorflow:examples/sec: 66.6901\n",
      "INFO:tensorflow:global_step/sec: 2.08452\n",
      "INFO:tensorflow:examples/sec: 66.7046\n",
      "INFO:tensorflow:global_step/sec: 2.0857\n",
      "INFO:tensorflow:examples/sec: 66.7424\n",
      "INFO:tensorflow:Saving checkpoints for 1471 into text_similarity/output/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.0009783262.\n",
      "INFO:tensorflow:training_loop marked as finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.contrib.tpu.python.tpu.tpu_estimator.TPUEstimator at 0x7f42c406bf98>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def file_based_input_fn_builder(input_file, seq_length, is_training, drop_remainder):\n",
    "\n",
    "    name_to_features = {\n",
    "        \"input_ids\": tf.FixedLenFeature([seq_length], tf.int64),\n",
    "        \"input_mask\": tf.FixedLenFeature([seq_length], tf.int64),\n",
    "        \"segment_ids\": tf.FixedLenFeature([seq_length], tf.int64),\n",
    "        \"label_ids\": tf.FixedLenFeature([], tf.int64),\n",
    "        \"is_real_example\": tf.FixedLenFeature([], tf.int64),\n",
    "    }\n",
    "\n",
    "    def _decode_record(record, name_to_features):\n",
    "        example = tf.parse_single_example(record, name_to_features)\n",
    "\n",
    "        for name in list(example.keys()):\n",
    "            t = example[name]\n",
    "            if t.dtype == tf.int64:\n",
    "                t = tf.to_int32(t)\n",
    "            example[name] = t\n",
    "        return example\n",
    "\n",
    "    def input_fn(params):\n",
    "        batch_size = params[\"batch_size\"]\n",
    "\n",
    "        d = tf.data.TFRecordDataset(input_file)        \n",
    "        if is_training:\n",
    "            d = d.repeat()\n",
    "            d = d.shuffle(buffer_size=100)\n",
    "\n",
    "        d = d.apply(\n",
    "            tf.contrib.data.map_and_batch(\n",
    "                lambda record: _decode_record(record, name_to_features),\n",
    "                batch_size=batch_size,\n",
    "                drop_remainder=drop_remainder))\n",
    "\n",
    "        return d\n",
    "    return input_fn\n",
    "\n",
    "\n",
    "run_config = tf.contrib.tpu.RunConfig(\n",
    "    cluster=tpu_cluster_resolver,\n",
    "    master=master,\n",
    "    model_dir=output_dir,\n",
    "    save_checkpoints_steps=save_checkpoints_steps,\n",
    "    tpu_config=tf.contrib.tpu.TPUConfig(\n",
    "        iterations_per_loop=iterations_per_loop,\n",
    "        num_shards=num_gpu_cores,\n",
    "        per_host_input_for_training=is_per_host))\n",
    "\n",
    "train_input_fn = file_based_input_fn_builder(\n",
    "    input_file=train_file,\n",
    "    seq_length=max_seq_length,\n",
    "    is_training=True,\n",
    "    drop_remainder=False) \n",
    "\n",
    "\n",
    "estimator = tf.contrib.tpu.TPUEstimator(\n",
    "    use_tpu=use_tpu,\n",
    "    model_fn=model_fn,\n",
    "    config=run_config,\n",
    "    train_batch_size=train_batch_size,\n",
    "    eval_batch_size=eval_batch_size,\n",
    "    predict_batch_size=predict_batch_size)\n",
    "\n",
    "tf.logging.info(\"***** Running training *****\")\n",
    "tf.logging.info(\"  Num examples = %d\", len(train_examples))\n",
    "tf.logging.info(\"  Batch size = %d\", train_batch_size)\n",
    "tf.logging.info(\"  Num steps = %d\", num_train_steps)\n",
    "\n",
    "estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 读取验证集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 0 of 2000\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: dev-1\n",
      "INFO:tensorflow:tokens: [CLS] 他 花 光 了 钱 。 [SEP] 崇 祯 十 四 年 （ 164 ##1 年 ） ， 李 自 成 数 次 围 攻 开 封 ， 丁 启 睿 督 催 左 良 玉 、 虎 大 威 、 杨 德 政 、 方 国 安 、 傅 宗 龙 等 人 率 兵 解 围 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 800 5709 1045 749 7178 511 102 2300 4875 1282 1724 2399 8020 10048 8148 2399 8021 8024 3330 5632 2768 3144 3613 1741 3122 2458 2196 8024 672 1423 4729 4719 998 2340 5679 4373 510 5988 1920 2014 510 3342 2548 3124 510 3175 1744 2128 510 987 2134 7987 5023 782 4372 1070 6237 1741 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: dev-2\n",
      "INFO:tensorflow:tokens: [CLS] 他 花 光 了 钱 。 [SEP] 这 种 抗 生 素 的 临 床 实 验 开 始 于 1960 年 代 ， 并 成 功 的 治 疗 急 性 白 血 病 和 淋 巴 瘤 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 800 5709 1045 749 7178 511 102 6821 4905 2834 4495 5162 4638 707 2414 2141 7741 2458 1993 754 8779 2399 807 8024 2400 2768 1216 4638 3780 4545 2593 2595 4635 6117 4567 1469 3900 2349 4606 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: dev-3\n",
      "INFO:tensorflow:tokens: [CLS] 有 我 杨 某 在 ， 你 就 别 想 翻 天 ！ [SEP] 只 要 有 我 杨 某 在 ， 你 就 别 想 翻 天 ！ [SEP]\n",
      "INFO:tensorflow:input_ids: 101 3300 2769 3342 3378 1762 8024 872 2218 1166 2682 5436 1921 8013 102 1372 6206 3300 2769 3342 3378 1762 8024 872 2218 1166 2682 5436 1921 8013 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 1 (id = 1)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: dev-4\n",
      "INFO:tensorflow:tokens: [CLS] 余 德 利 抬 头 发 现 李 东 宝 的 目 光 很 慌 。 [SEP] 后 母 回 家 后 ， 发 现 叶 限 抱 树 而 睡 ， 便 没 有 追 究 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 865 2548 1164 2848 1928 1355 4385 3330 691 2140 4638 4680 1045 2523 2707 511 102 1400 3678 1726 2157 1400 8024 1355 4385 1383 7361 2849 3409 5445 4717 8024 912 3766 3300 6841 4955 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n",
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: dev-5\n",
      "INFO:tensorflow:tokens: [CLS] 我 不 常 逛 街 ， 因 为 我 老 没 时 间 。 [SEP] 齐 格 飞 （ [UNK] ， 齐 格 鲁 德 的 德 语 写 法 ， 为 同 一 人 ） 在 杀 掉 法 夫 纳 时 就 全 身 浴 血 ， 但 因 为 有 一 片 树 叶 黏 在 背 后 ， 所 以 造 成 有 一 小 块 皮 肤 没 有 沾 到 血 ， 而 成 为 他 唯 一 的 弱 点 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 2769 679 2382 6859 6125 8024 1728 711 2769 5439 3766 3198 7313 511 102 7970 3419 7607 8020 100 8024 7970 3419 7826 2548 4638 2548 6427 1091 3791 8024 711 1398 671 782 8021 1762 3324 2957 3791 1923 5287 3198 2218 1059 6716 3861 6117 8024 852 1728 711 3300 671 4275 3409 1383 7945 1762 5520 1400 8024 2792 809 6863 2768 3300 671 2207 1779 4649 5502 3766 3300 3783 1168 6117 8024 5445 2768 711 800 1546 671 4638 2483 4157 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "line 0 : ['Quality', '#1 ID', '#2 ID', '#1 String', '#2 String']\n",
      "line 1 : ['0', '662', '3307482', '他花光了钱。', '崇祯十四年（1641年），李自成数次围攻开封，丁启睿督催左良玉、虎大威、杨德政、方国安、傅宗龙等人率兵解围。']\n",
      "line 2 : ['0', '662', '3306833', '他花光了钱。', '这种抗生素的临床实验开始于1960年代，并成功的治疗急性白血病和淋巴瘤。']\n",
      "line 3 : ['1', '818', '2516', '有我杨某在，你就别想翻天！', '只要有我杨某在，你就别想翻天！']\n",
      "line 4 : ['0', '559', '2791731', '余德利抬头发现李东宝的目光很慌。', '后母回家后，发现叶限抱树而睡，便没有追究。']\n",
      "line 5 : ['0', '828', '4138697', '我不常逛街，因为我老没时间。', '齐格飞（Siegfried，齐格鲁德的德语写法，为同一人）在杀掉法夫纳时就全身浴血，但因为有一片树叶黏在背后，所以造成有一小块皮肤没有沾到血，而成为他唯一的弱点。']\n",
      "line 6 : ['0', '78', '387909', '嘴干死了。', '此文先后对诗，赋，碑，诔，铭，箴，颂，论，奏，说十种进行分析。']\n",
      "line 7 : ['1', '881', '2386', '塑料不腐烂分解是一大长处，因为当塑料垃圾被深埋时，他永远不会变成任何有毒的化学物质污染人类生存的环境，而且即便被焚烧，大部分塑料也不会释放出有毒气体。', '塑料垃圾被深埋时，他永远不会变成任何有毒的化学物质污染人类生存的环境，而且即便被焚烧，大部分塑料也不会释放出有毒气体，故塑料不腐烂分解是一大长处。']\n",
      "line 8 : ['0', '421', '2101313', '静静地坐着。', '这个发现令许多人想进一步了解海马区在记忆及学习机制的作用，因而成为一种流行，无论在神经解剖学、生理学、行为学等等各种不同领域，都对海马区做了相当丰富的研究。']\n",
      "line 9 : ['0', '846', '4228944', '阳春四月，平原地区的桃花早就凋谢了，可是这里却仍然是一片绯红，桃花含苞欲放，艳丽多姿。', '在大陆地区共有2500多名员工，研发中心位于苏州和广州。']\n",
      "line 10 : ['1', '240', '1297', '来客人的家庭。', '家里来了客人。']\n",
      "line 11 : ['0', '897', '4484472', '你不觉得我们的战士很可爱吗？', '斑马马蹄一直梦想著要去他梦寐以求的大草原，因为牠已经十岁了而觉得在动物园生活总是一成不变，后来四只企鹅企图逃离动物园到野外而挖地洞却挖到马蹄那里，牠觉得企鹅们一心想逃离动物园内心受到动摇，虽然生长在动物园完全不愁吃和住，却仍然一心想要体验自然狂野生活，晚上三个好朋友帮马蹄庆生，却觉得他想去野外的愿望感觉不妥，马蹄对牠们询问一生都没见识过动物园以外的世界难道不会感到难过吗？']\n",
      "line 12 : ['0', '706', '3527149', '你还讲不讲理了？', '佐佐木健介与妻子北斗晶于1995年10月1日结婚，现在两人育有两个儿子。']\n",
      "line 13 : ['0', '278', '1385879', '你喝了不到两盅酒，就叨叨叨，叨叨叨，你有个够没有？', '这个技术或许可以追溯到修建卡夫拉金字塔的时候，虽然还没有明确的证据。']\n",
      "line 14 : ['0', '384', '1919266', '她在图书馆里从书架上拿了一本书。', '人类女性的子宫位于骨盆腔中央，呈倒置的梨形。']\n",
      "line 15 : ['0', '779', '3891683', '这次活动与其说是在培养孩子们吃苦精神，还不如说是给他们一次体验集体生活的机会。', '他们不同意斯坦加入该组织，但会长告诉他们：我们不是要他‘加入’我们；']\n",
      "line 16 : ['0', '368', '1838451', '桌子叫人抬走了。', '奋身斗，后骑皆进，手击杀数十人，身中四矢三刃，遂仆。']\n",
      "line 17 : ['1', '207', '1033', '树上飞来一只鸟。', '一只鸟飞到树上。']\n",
      "line 18 : ['0', '796', '3978150', '一旦有机会，就要全力以赴。', '由于在2005年时把头发染成金色，外表和卡通片樱桃小丸子中花轮同学一角相似，因而被网友冠以花轮同学的绰号。']\n",
      "line 19 : ['0', '656', '3277494', '他们厂上个月从欧洲购进了一批先进的设备。', '使用者必须注册才能使用Facebook，注册后他们可以创建个人档案、将其他使用者加为好友、传递讯息，并在其他使用者更新个人档案时获得自动通知。']\n",
      "line 20 : ['1', '368', '1067', '他刚才暧昧地瞅了小红一眼。', '他刚才暧昧地看了小红一眼。']\n",
      "line 21 : ['0', '431', '2151282', '他不看电影。', '墙基用石块砌筑，墙身用青砖，山墙承重，木构架，瓦顶。']\n",
      "line 22 : ['1', '25', '83', '她洗净了一件衣服。', '一件衣服被她洗净了。']\n",
      "line 23 : ['0', '114', '567576', '缺不了你的钱花。', '对web服务和。']\n",
      "line 24 : ['1', '977', '3100', '一个人难免犯错误。', '人不犯错误是不可能的。']\n",
      "line 25 : ['0', '962', '4807479', '这件事非你去不成。', '门廊上除了题词还有关于描述圣彼得从大海中拯救的故事。']\n",
      "line 26 : ['0', '289', '1442523', '整个小区的业主没有他不认识的。', '参加1998年在希腊雅典世界篮球锦标赛的美国男篮并不能算作梦之队，因为其中没有一名NBA现役球员。']\n",
      "line 27 : ['0', '884', '4415850', '我们无论评价什么样的历史人物，都必须全面的看待，不但要看到他们的历史功绩和贡献，而且要看到他们的过失和负面影响，否则，就不可能做出全面、客观的评价。', '中华民国最高法院101年度第11次刑事庭会议：共同正犯在主观上须有共同犯罪之意思，客观上须为共同犯罪行为之实行。']\n",
      "line 28 : ['0', '26', '126137', '他修好了电脑。', '边疆非汉族地方产生动荡和骚乱，清朝政权出现分崩离析的危险，尤其在尚未建省的蒙古和西藏等地纷纷出现分离倾向。']\n",
      "line 29 : ['1', '1000', '3170', '我会是坏人吗？', '我难道会是坏人吗？']\n",
      "line 30 : ['1', '584', '1777', '别揣着明白装糊涂啦。', '不要明白还装糊涂啦。']\n",
      "line 31 : ['0', '898', '4488190', '你不以我们的祖国有着这样的英雄而自豪吗？', '另外，不以此种艺术活动作为本职职业棒球选手或是相扑力士等名人推出唱片之时，乃是有某种程度的歌唱实力，情感歌谣的曲调时有变化。']\n",
      "line 32 : ['0', '593', '2964112', '师傅比我更懂得如何做人。', '中国大陆。']\n",
      "line 33 : ['0', '950', '4747111', '我没有说清楚，难免被人误会。', '研究型大学与其他学院没有差别。']\n",
      "line 34 : ['0', '175', '871598', '他送了汪老师一张贺卡。', '最新的理论通过对河流和地名的研究认为日耳曼人的产生地在今天德国中部山区的北部。']\n",
      "line 35 : ['0', '870', '4348744', '不成也值。', '用水限制通常在英格兰及南威尔斯地区实施，而影响气候较湿的苏格兰的情况则十分旱见。']\n",
      "line 36 : ['0', '307', '1530363', '也弄个事儿管管。', '史蒂夫·琼斯在一家商店打周末零工。']\n",
      "line 37 : ['0', '235', '1170119', '我跟这个人通过信。', '该剧描述一个普通银行出纳员突然心血来潮，席卷了大批钞票潜逃后12小时内发生的故事和心理轨迹，最终表达金钱是这个世界上所有卑鄙龌龊的诈骗中最卑鄙的骗局。']\n",
      "line 38 : ['0', '63', '311366', '一个女孩家，只管拿着诗作正经事讲起来，叫有学问的人听了，反笑话说不守本分。', '在一个典型的三权分立国家中，创造和解释法律的核心机构为政府的三大部门：公正不倚的司法、民主的立法和负责的行政。']\n",
      "line 39 : ['1', '502', '1574', '她离开广州已经好几年了。', '好几年前她已经离开广州。']\n",
      "line 40 : ['0', '724', '3615267', '真不像话！', '这部由山姆·雷米导演的影片也是邓斯特当时商业上最成功的作品。']\n",
      "line 41 : ['0', '309', '1543514', '老陈换了个姿势继续说。', '史诗所见的最早版本是用楔形文字刻在之上。']\n",
      "line 42 : ['0', '473', '2360303', '洗衣机把衣服洗坏了。', '克罗斯比从不抱怨她的残疾。']\n",
      "line 43 : ['0', '710', '3547959', '千万别大意！', '正方时间飞船的六边形T字标记的设定，是黑底白字。']\n",
      "line 44 : ['0', '908', '4537848', '好不伤心。', '分布于中国大陆的四川等地，生长于海拔2，600米的地区，一般生长在河边干旱灌丛中，目前尚未由人工引种栽培。']\n",
      "line 45 : ['1', '234', '3398', '小王没看完书。', '书没有被小王看完。']\n",
      "line 46 : ['0', '927', '4630471', '这件事办了很长时间，还没有办成。', '虽然球证没有对史基泰尔作出任何惩罚（可能在球证的视线范围看不清史基泰尔的动作），但该场赛事后不久英足总决定对史基泰尔秋后算账，作出停赛三场的惩罚。']\n",
      "line 47 : ['0', '674', '3369681', '想走就走呗，又不是没你不行。', '目前已发现数个未成年鹦鹉嘴龙化石。']\n",
      "line 48 : ['1', '381', '1457', '为这件事，我们都很高兴。', '我们很高兴因为这件事。']\n",
      "line 49 : ['0', '323', '1614635', '我们就去帮他的忙。', '面对职责，杰里科说一针见血的指出：我们不能在一次舰队决战中留下任何碰运气的事，因为我们的舰队对英国的存在是一个，也是唯一一个至关重要的因素。']\n"
     ]
    }
   ],
   "source": [
    "def get_dev_examples(data_dir):\n",
    "    return create_examples(read_tsv(os.path.join(data_dir, \"dev.tsv\")), \"dev\")\n",
    "\n",
    "eval_examples = get_dev_examples(data_dir)\n",
    "\n",
    "eval_file = os.path.join(output_dir, \"eval.tf_record\")\n",
    "\n",
    "file_based_convert_examples_to_features(eval_examples, label_list, max_seq_length, tokenizer, eval_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 在验证集上验证模型，评估结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:***** Running evaluation *****\n",
      "INFO:tensorflow:  Num examples = 2000 (2000 actual, 0 padding)\n",
      "INFO:tensorflow:  Batch size = 8\n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Running eval on CPU\n",
      "INFO:tensorflow:*** Features ***\n",
      "INFO:tensorflow:  name = input_ids, shape = (?, 128)\n",
      "INFO:tensorflow:  name = input_mask, shape = (?, 128)\n",
      "INFO:tensorflow:  name = is_real_example, shape = (?,)\n",
      "INFO:tensorflow:  name = label_ids, shape = (?,)\n",
      "INFO:tensorflow:  name = segment_ids, shape = (?, 128)\n",
      "INFO:tensorflow:**** Trainable Variables ****\n",
      "INFO:tensorflow:  name = bert/embeddings/word_embeddings:0, shape = (21128, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
      "INFO:tensorflow:  name = output_weights:0, shape = (2, 768)\n",
      "INFO:tensorflow:  name = output_bias:0, shape = (2,)\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Starting evaluation at 2019-09-29T02:25:25Z\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from text_similarity/output/model.ckpt-1471\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n",
      "INFO:tensorflow:Finished evaluation at 2019-09-29-02:25:38\n",
      "INFO:tensorflow:Saving dict for global step 1471: eval_accuracy = 0.9895, eval_loss = 0.037643015, global_step = 1471, loss = 0.037643015\n",
      "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1471: text_similarity/output/model.ckpt-1471\n",
      "INFO:tensorflow:evaluation_loop marked as finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "打印文本相似度评估指标\n",
      "eval_accuracy : 0.9895\n",
      "eval_loss : 0.037643015\n",
      "loss : 0.037643015\n",
      "global_step : 1471\n"
     ]
    }
   ],
   "source": [
    "num_actual_eval_examples = len(eval_examples)\n",
    "\n",
    "tf.logging.info(\"***** Running evaluation *****\")\n",
    "tf.logging.info(\"  Num examples = %d (%d actual, %d padding)\",\n",
    "                len(eval_examples), num_actual_eval_examples,\n",
    "                len(eval_examples) - num_actual_eval_examples)\n",
    "tf.logging.info(\"  Batch size = %d\", eval_batch_size)\n",
    "\n",
    "\n",
    "eval_input_fn = file_based_input_fn_builder(\n",
    "    input_file=eval_file,\n",
    "    seq_length=max_seq_length,\n",
    "    is_training=False,\n",
    "    drop_remainder=False)\n",
    "\n",
    "result = estimator.evaluate(input_fn=eval_input_fn, steps=None)\n",
    "\n",
    "print(\"\\n打印文本相似度评估指标\")\n",
    "for key in result:\n",
    "    print(key+' : '+str(result[key]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 在线测试\n",
    "\n",
    "由以上训练得到模型进行在线测试，可以任意输入两个句子，进行相似度分析。\n",
    "\n",
    "任意一个句子未输入，则结束在线文本相似度分析。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_model_dir': './text_similarity/output/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 0.9\n",
      "  allow_growth: true\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f42bbb764e0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "在线测试\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py:429: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "tf.py_func is deprecated in TF V2. Instead, use\n",
      "    tf.py_function, which takes a python function which manipulates tf eager\n",
      "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
      "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
      "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
      "    being differentiable using a gradient tape.\n",
      "    \n",
      "INFO:tensorflow:Calling model_fn.\n",
      "INFO:tensorflow:Done calling model_fn.\n",
      "INFO:tensorflow:Graph was finalized.\n",
      "INFO:tensorflow:Restoring parameters from ./text_similarity/output/model.ckpt-1471\n",
      "INFO:tensorflow:Running local_init_op.\n",
      "INFO:tensorflow:Done running local_init_op.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "输入句子1: 我曾经帮这位教授整理过稿子。\n",
      "\n",
      "输入句子2: 这位教授的稿子我帮着整理过。\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 我 曾 经 帮 这 位 教 授 整 理 过 稿 子 。 [SEP] 这 位 教 授 的 稿 子 我 帮 着 整 理 过 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 2769 3295 5307 2376 6821 855 3136 2956 3146 4415 6814 4943 2094 511 102 6821 855 3136 2956 4638 4943 2094 2769 2376 4708 3146 4415 6814 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "相似度是：0.9986897110939026\n",
      "\n",
      "输入句子1: 我习惯喝咖啡不放糖。\n",
      "\n",
      "输入句子2: 他边打工挣学费边上学。\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n",
      "INFO:tensorflow:guid: test-0\n",
      "INFO:tensorflow:tokens: [CLS] 我 习 惯 喝 咖 啡 不 放 糖 。 [SEP] 他 边 打 工 挣 学 费 边 上 学 。 [SEP]\n",
      "INFO:tensorflow:input_ids: 101 2769 739 2679 1600 1476 1565 679 3123 5131 511 102 800 6804 2802 2339 2914 2110 6589 6804 677 2110 511 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "相似度是：0.0003190733550582081\n",
      "\n",
      "输入句子1: \n",
      "\n",
      "输入句子2: \n",
      "\n",
      "再见\n"
     ]
    }
   ],
   "source": [
    "from text_similarity.bert import similarity\n",
    "sim = similarity.BertSim()\n",
    "\n",
    "print(\"在线测试\\n\")\n",
    "sim.set_mode(tf.estimator.ModeKeys.PREDICT)\n",
    "predict = 1\n",
    "while predict is not None:\n",
    "    sentence1 = input('\\n输入句子1: ')\n",
    "    sentence2 = input('\\n输入句子2: ')\n",
    "    predict = sim.predict(sentence1, sentence2)\n",
    "    if predict is not None:\n",
    "        print('\\n相似度是：{}'.format(predict[0][1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}