diff --git a/extras/sentiment_analysis/README.md b/extras/sentiment_analysis/README.md new file mode 100644 index 0000000..905e08d --- /dev/null +++ b/extras/sentiment_analysis/README.md @@ -0,0 +1,271 @@ +# Sentiment Analysis Tutorial + +This is the code and data used for the Sentiment Analysis Tutorial available +at [`tutorial.md`](tutorial.md) + +In this tutorial we're going to show how to build a recurrent neural network +(RNN) that learns how to classify movie reviews as positive or negative using +TensorFlow high level APIs. Here, the focus is on introducing the high level APIs, +rather than on building a high-accuracy model. + +## How to run this? + +### Install TensorFlow + +Go [here](https://www.tensorflow.org/install/) for instructions. +**Make sure you have installed TensorFlow v1.2 or higher** + +### Train your model + +The first time you run the script can take a while to the model actually +starts training, since it will first download the files available at +[LSTM Sentiment Analysis](https://github.com/adeshpande3/LSTM-Sentiment-Analysis) +where you can find a preprocessed version of the +[Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/) +and also a +[great sentiment analysis tutorial using low-level TensorFlow by O’Reilly](https://www.oreilly.com/learning/perform-sentiment-analysis-with-lstms-using-tensorflow). + +To start you can run the `sentiment_analysis.py` file. There are +a number of different arguments you can try in order to produce different models +so we encourage you to have a look on the code to see all you can do. + +Below are some examples of what you can change in the model just using arguments. + +```shell + +# Train the model, then evaluate and predict using the default model, +# will save the output at "sentiment_analysis_output". +# The default model is a single LSTM layer with 128 cells +# and a dense softmax layer on top of the last cell. +# This was the \same command used to train the the model mentioned in +# the tutorial. +$ python sentiment_analysis.py + +# Use the average of the hidden states as the final output for the RNN layers +# as suggested by "Sentiment Analysis with Deeply Learned Distributed +# Representations of Variable Length Texts" from James Hong and Michael Fang +# (2015) http://cs224d.stanford.edu/reports/HongJames.pdf. +# It will not run an experiment, instead it will use the estimator's +# interface to first train and then evaluate the model +# (this is the biggest difference of using/not using experiments in local +# settings). +$ python sentiment_analysis.py --use_hidden_states=average --dont_run_experiment + +# Change the model to have 3 LSTM layers with dropout of 0.75, 0.5 and 0.5 +# and add two DNN layers on top of the RNNs. +# It will train for 1 epoch +$ python sentiment_analysis.py --rnn_cell_sizes 128 128 64 \ + --dropout_keep_probabilities 0.75 0.5 0.5 \ + --dnn_layer_sizes 64 64 \ + --model_dir "my_deep_rnn" \ + --num_epochs 1 + +# There are more command line flags to play with; check sentiment_analysis.py +# for details. +``` + +### Using TensorBoard + +Use this command during, or after training to visualize metrics, +the model graph, and more. +```shell +# Set --log_dir to point to the model_dir of the previous step +# by default the model_dir is "sentiment_analysis_output" +$ tensorboard --log_dir="sentiment_analysis_output" +``` + +![](../../images/sentiment_analysis_tensorboard.png) + +You can also check your TensorFlow graph to debug your graph if needed. + +![](../../images/sentiment_analysis_tensorboard_graph.png) + +### Training example output + +```shell +INFO:tensorflow:loss = 0.691962, step = 101 (36.537 sec) +INFO:tensorflow:global_step/sec: 3.79198 +INFO:tensorflow:loss = 0.637554, step = 201 (26.371 sec) +INFO:tensorflow:global_step/sec: 4.12 +INFO:tensorflow:loss = 0.461921, step = 301 (24.272 sec) +INFO:tensorflow:global_step/sec: 4.23288 +INFO:tensorflow:loss = 0.456651, step = 401 (23.625 sec) +INFO:tensorflow:global_step/sec: 4.18946 +INFO:tensorflow:loss = 0.603483, step = 501 (23.869 sec) +INFO:tensorflow:global_step/sec: 4.07666 +INFO:tensorflow:loss = 0.617782, step = 601 (24.530 sec) +INFO:tensorflow:global_step/sec: 4.19543 +INFO:tensorflow:loss = 0.565088, step = 701 (23.835 sec) +INFO:tensorflow:global_step/sec: 3.94001 +INFO:tensorflow:loss = 0.509592, step = 801 (25.381 sec) +INFO:tensorflow:global_step/sec: 4.19204 +INFO:tensorflow:loss = 0.652886, step = 901 (23.855 sec) +INFO:tensorflow:global_step/sec: 4.06573 +INFO:tensorflow:loss = 0.696719, step = 1001 (24.596 sec) +INFO:tensorflow:global_step/sec: 4.03502 +INFO:tensorflow:loss = 0.519887, step = 1101 (24.783 sec) +INFO:tensorflow:global_step/sec: 3.93356 +INFO:tensorflow:loss = 0.579439, step = 1201 (25.422 sec) +INFO:tensorflow:global_step/sec: 3.87702 +``` + +### Exaluation example output + +```shell +INFO:tensorflow:Starting evaluation at 2017-07-20-22:01:39 +INFO:tensorflow:Restoring parameters from pre/model.ckpt-6262 +INFO:tensorflow:Evaluation [1/100] +INFO:tensorflow:Evaluation [2/100] +INFO:tensorflow:Evaluation [3/100] +INFO:tensorflow:Evaluation [4/100] +INFO:tensorflow:Evaluation [5/100] +INFO:tensorflow:Evaluation [6/100] +INFO:tensorflow:Evaluation [7/100] +INFO:tensorflow:Evaluation [8/100] +INFO:tensorflow:Evaluation [9/100] +INFO:tensorflow:Evaluation [10/100] +... +INFO:tensorflow:Evaluation [100/100] +INFO:tensorflow:Finished evaluation at 2017-07-24-20:39:32 +INFO:tensorflow:Saving dict for global step 6262: accuracy = 0.856875, global_step = 6262, loss = 0.374715 +``` + +### Prediction sample output + +In the sentences below *unk* stands for a word that isn't in the word +embedding. The *0 0 ... 0* in some lines means that the line was padded +with zeros in the end so it could have length equals to 250 (more details +about this in the tutorial). + +The model outputs a 2-dim tensor as result containing the probability +of a review be negative (index 0) or positive (index 1) + +```shell +if this is the best commander hamilton movie i have no curiosity about the others a movie actors greatest tools are his eyes but when peter stormare wants to show great emotion he closes his so for five or six seconds we get to admire his eyelids while his feelings remain unknown behind them lousy acting technique stormare also flinches sometimes when he fires a gun turning his head away and clamping his eyes shut watch carefully james bond can rest easy with competition like this there are some interesting supporting performances from other actors but not enough to hang a whole movie on the cinematography is unk doing a fine job of capturing the nordic cold even the sahara winds up looking cold perhaps hamilton carries his own climate with him there are some individual good action sequences here unfortunately the only sense of humor on screen belongs to the villain which turns the hero into a big pill james bonds jokes may not be particularly good but at least he doesnt look constipated all the time one positive point in the movies favor is that the psychotic contorted vicious hatred of israel in unk books has been left out what has been kept in is worship of a noble heroic plo that he shows us functioning in libya without the dictator unk knowledge or supervision this fantasy is hard to believe since unk actually threw the plo out of libya for four years at a time and at +Prediction: [ 0.1009393 0.89906073] +Label: 0 +this is the most stupid movie ever made the story is laughable his wife and kid think hes insane then they dont then it turns out he is and i think they knew it all along there is a dog named ned that causes some problems and i think its all his unk does jim carey god only knows why virginia madsen took this unk is a career sinker i think the target audience for this is 11 and 12 year olds and that adds up to 23 or maybe its for 8 and 10 years olds which also adds up to 23 or maybe its for really dumb 23 year olds or maybe really dumb 32 year olds because thats 23 in reverse or maybe 46 year olds would enjoy it because half of that is 23 i think looking up things on the internet about the number 23 would be more entertaining than this movie unless you wanted to see a comedy 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 +Prediction: [ 0.74981982 0.25018018] +Label: 0 +there isnt much that comes close to the unk storytelling and suspenseful unk levels as goldeneye when it came out it was the greatest game of alltime and even today it stays strong i will admit that this game did get boring after a few months of playing and by not playing it again until two years later i was thrust back into its greatest almost as if i was playing it for the first time again there are 20 unk levels which is probably the most of any james bond game to date probably the most unforgettable one is the tank level which was likely the most explosive video game sequence at that time and the unk shooting as well as usage of q gadgets is what james bond fans are always dying to use frankly as a james bond fan i look for aspects of a true james bond experience which are now showing up in the ps2 games so this game while it has some great action and usable gadgets i was somewhat expecting a little more even back in 1997 i also disliked that this game didnt have q or m or moneypenny or anyone from mi6 while watching the movies bond interacts with these characters at least a few times throughout each movie but they are nowhere to be seen in this game and vocal dialogue would have made the game more lively rather than the text dialogue they wound up using they had the +Prediction: [ 0.12032966 0.87967038] +Label: 1 +as a true canadian i always avoid canadian movies however now and then i get trapped into watching one this one is better than most which is to say mediocre it has many of the usual flaws of canadian unk unk excess of cinematic gimmicks and above all the unk canadian habit of using canadian cities as unk for american ones i mean using the historic metropolis of montreal as a stand in for harrisburg pennsylvania is just short of obscene i was in a generous mood i gave it a 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 +Prediction: [ 0.71495974 0.28504029] +Label: 0 +if youre interested in learning about the real side of spying this movie is for you unlike 007 movies this shows how things really go down in the world of espionage timothy hutton and sean penn both give outstanding performances in this unk film certainly worth watching 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 +Prediction: [ 0.25497162 0.74502844] +Label: 1 +``` + +## Done training? Play with your model! + +Make sure you pass the same arguments to *sentiment_analysis.py* +and *sentiment_analysis.py --mode=classify* since in the *classify mode* +the script will load the model trained in the `model_dir` so make sure +you're running the same model. + +**Only lower case letters and numbers are accepted as input.** + +```shell +# The script will load your model and you can use it to classify new sentences +$ python sentiment_analysis.py --mode=classify --model_dir="sentiment_analysis_output" +``` + +### Output example + +``` +Write your review (or type to exit): it was fine i guess +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.527832 +Positive: 0.472169 +Write your review (or type to exit): it was good +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.357005 +Positive: 0.642995 +Write your review (or type to exit): it wasnt good +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.60162 +Positive: 0.39838 +Write your review (or type to exit): it was not good +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.411701 +Positive: 0.588299 +Write your review (or type to exit): i thought the movie was incredible and inspiring +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.0676128 +Positive: 0.932387 +Write your review (or type to exit): this is a great movie +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.130054 +Positive: 0.869946 +Write your review (or type to exit): this is a good movie but isnt the best +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.224676 +Positive: 0.775324 +Write your review (or type to exit): this is a good movie +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.189163 +Positive: 0.810837 +Write your review (or type to exit): this is a good movie it is the best +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.0859528 +Positive: 0.914047 +Write your review (or type to exit): it was definitely bad +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.894781 +Positive: 0.105219 +Write your review (or type to exit): its not that bad +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.740116 +Positive: 0.259884 +Write your review (or type to exit): it is bad +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.906015 +Positive: 0.0939852 +Write your review (or type to exit): its not that bad i think its a good movie +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.427492 +Positive: 0.572508 +Write your review (or type to exit): its not bad i think its a good movie +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.489538 +Positive: 0.510462 +Write your review (or type to exit): its not good i think its a bad movie +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.538815 +Positive: 0.461185 +``` + +## Whats next? + +In this tutorial we showed how to implement a Recurrent Neural Network for +binary sentiment analysis using TensorFlow high level APIs. + +* We encourage you to run the code and see how the model performs for yourself. + The model parameters were not tuned, so a good exercise is just play with + the parameters and in order to have better results. + Try changing the learning rate, optimizer, hidden state size, + number of RNN cells, number of DNN layers, and so on. + +* Finally, the model presented above can be easily changed to be used on + different data and even perform different classification + or prediction tasks. A great example of how to change this implementation to + perform different tasks is [colorbot](https://github.com/random-forests/tensorflow-workshop/blob/master/extras/colorbot/) + a deep RNN model that receives a word (sequence of characters) as input + and learns to predict a rgb value that better represents this word. + As a result we have a color generator! diff --git a/extras/sentiment_analysis/imdb.py b/extras/sentiment_analysis/imdb.py new file mode 100644 index 0000000..dbddbc4 --- /dev/null +++ b/extras/sentiment_analysis/imdb.py @@ -0,0 +1,129 @@ +# Copyright 2017 The TensorFlow Authors. All Rights Reserved. +# # +# # Licensed under the Apache License, Version 2.0 (the "License"); +# # you may not use this file except in compliance with the License. +# # You may obtain a copy of the License at +# # +# # http://www.apache.org/licenses/LICENSE-2.0 +# # +# # Unless required by applicable law or agreed to in writing, software +# # distributed under the License is distributed on an "AS IS" BASIS, +# # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# # See the License for the specific language governing permissions and +# # limitations under the License. +# # +# ============================================================================== +"""A helper class for fetching and importing the IMDB dataset. + +This helper will download the data available at +https://github.com/adeshpande3/LSTM-Sentiment-Analysis that is a preprocessed +version of the Large Movie Review Dataset available at +http://ai.stanford.edu/~amaas/data/sentiment/. Here you'll also +find functions to access this data once it's available. +""" + +import os +import tarfile + +import numpy as np +from six.moves import urllib + + +class IMDB(object): + """A helper class for fetching and importing the IMDB dataset. + + The three `get` methods each import an component of data + from the downloaded files. + """ + + def __init__(self, data_path, percentage_train=0.9): + """Create an IMDB data loader. + Args: + data_path: Where to store the downloaded files. + percentage_train: The fraction of the dataset set to use for training. + """ + # path where the data will be stored + self.data_path = data_path + # postive reviews will have label 1, and negative reviews label 0 + self._POS = 1 + self._NEG = 0 + # path to where data is hosted + self._DATA_URL = 'https://github.com/adeshpande3/LSTM-Sentiment-Analysis/blob/master/training_data.tar.gz?raw=true' + # perecentage of data used for training + self._PERCENTAGE_TRAIN = percentage_train + # if data is not in data_path download it from _DATA_URL + self._maybe_download() + + def _get_word_list(self): + """Returns list with words available in the word embedding.""" + return list(np.load(os.path.join(self.data_path, 'wordsList.npy'))) + + def get_word_to_index(self): + """Returns dict mapping a word to an index in the word embedding.""" + word_list = self._get_word_list() + word_dict = {word_list[i]: i for i in range(len(word_list))} + return word_dict + + def get_index_to_word(self): + """Returns dict mapping an index to a word in the word embedding.""" + word_list = self._get_word_list() + word_dict = {i: word_list[i] for i in range(len(word_list))} + return word_dict + + def get_word_vector(self): + """Returns the pretrained word embedding.""" + return np.load(os.path.join(self.data_path, 'wordVectors.npy')) + + def get_data(self): + """Returns the preprocessed IMDB dataset for training and evaluation. + + The data contain 25000 reviews where the first half is positive and the + second half is negative. This function by default will return 90% of the + data as training data and 10% as evaluation data. + """ + + data = np.load(os.path.join(self.data_path, 'idsMatrix.npy')) + # the first half of the data length are positive reviews + # the other half are negative reviews + data_len = data.shape[0] + label = np.array( + [self._POS if i < data_len/2 else self._NEG for i in range(data_len)] + ) + + # shuffle the data + p = np.random.permutation(data_len) + shuffled_data = data[p] + shuffled_label = label[p] + + # separate training and evaluation + train_limit = int(data_len * self._PERCENTAGE_TRAIN) + + train_data = shuffled_data[:train_limit] + train_label = shuffled_label[:train_limit] + eval_data = shuffled_data[train_limit:] + eval_label = shuffled_label[train_limit:] + + return train_data, train_label, eval_data, eval_label + + def _maybe_download(self): + """Maybe downloads data available at https://github.com/adeshpande3/LSTM-Sentiment-Analysis.""" + try: + self.get_word_to_index() + self.get_word_vector() + self.get_data() + except IOError: + print('Data is not available at %s, Downloading it...' % self.data_path) + # if the data_path does not exist we'll create it + if not os.path.exists(self.data_path): + os.makedirs(self.data_path) + + # download data + tar_path = os.path.join(self.data_path, 'data.tar.gz') + urllib.request.urlretrieve(self._DATA_URL, tar_path) + # extract data and save at self.data_path + tar = tarfile.open(tar_path) + tar.extractall(self.data_path) + tar.close() + + print('Download complete!') + diff --git a/extras/sentiment_analysis/input_function_lib.py b/extras/sentiment_analysis/input_function_lib.py new file mode 100644 index 0000000..259c697 --- /dev/null +++ b/extras/sentiment_analysis/input_function_lib.py @@ -0,0 +1,160 @@ +# Copyright 2017 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== +"""Input functions implementations used by sentiment_analysis.py. + +You'll find 2 input function implementations: + +* build_input_fn: expects preprocessed numpy data as input + (more details in the tutorial) and will be used to train and evaluate the + model. + +* build_classify_input_fn: expects a string as input and will be used + to classify new reviews in real time. + +""" +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function + +import numpy as np +import tensorflow as tf +from tensorflow.contrib.learn.python.learn.estimators import rnn_common + + +def build_input_fn(x_in, y_in, batch_size, + shuffle=True, epochs=1, + batch_by_seq_len=False, + max_length=250): + """Returns an input function created from word and class index arrays. + + + + Args: + x_in: A numpy array of word indexes with shape (num_examples, + max_sequence_length). The array is padded on the right with zeros. + y_in: A numpy array of class indexes with shape (num_examples) + batch_size: Batch size for the input_fn to return + shuffle: A bool, indicating whether to shuffle the data or not. + epochs: Number of epochs for the input fun to generate. + batch_by_seq_len: A bool to activate sequence length batching. + max_length: Truncate sequences longer than max_length. + + Returns: + An `input_fn`. + """ + def _length_bin(length, max_seq_len, length_step=10): + """Sets the sequence length bin.""" + bin_id = (length // length_step + 1) * length_step + return tf.cast(tf.minimum(bin_id, max_seq_len), tf.int64) + + def _make_batch(key, ds): + """Removes extra padding and batchs the bin.""" + # eliminate the extra padding + key = tf.cast(key, tf.int32) + ds = ds.map(lambda x, x_len, y: (x[:key], x_len, y)) + + # convert the entire contents of the bin to a batch + ds = ds.batch(batch_size) + return ds + + def input_fn(): + """Input function used for train and eval; usually not called directly. + """ + # calculates the length of the sequences + # since the inputs are already padded with zeros in the end + # the length will be the last index that is non zero + 1 + x_len = np.array( + [np.nonzero(seq)[0][-1] + 1 for seq in x_in]).astype('int32') + + # creates the dataset from in memory data + # x_in: sequence of indexes that map a word to an embedding + # x_len: sequence lengths + # y_in: 1 if positive review, 0 if negative review + ds = tf.contrib.data.Dataset.from_tensor_slices((x_in, x_len, y_in)) + + # repeats the dataset `epochs` times + ds = ds.repeat(epochs) + + if shuffle: + # make sure the buffer is big enough for your data + ds = ds.shuffle(buffer_size=25000 * 2) + + if batch_by_seq_len: + # implement a simple `Dataset` version of `bucket_by_sequence_length` + # https://goo.gl/y67FQm + ds = ds.group_by_window( + key_func=lambda x, x_len, y: _length_bin(x_len, max_length), + reduce_func=_make_batch, + window_size=batch_size) + else: + ds = ds.batch(batch_size) + + # creates iterator + x, x_len, y = ds.make_one_shot_iterator().get_next() + + # feature must be a dictionary + dict_x = {'x': x, rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY: x_len} + return dict_x, y + + return input_fn + + +def build_classify_input_fn(review, word_to_id): + """Returns an Input function from a string review, and a word_to_id mapping. + The input_fn only yields a single batch before throwing an end of + sequence error. + The input_fn does not yield labels, so it cannot be used for training or + evaluation. + + Args: + review(str): A string review sentence. + word_to_id(dict): A dict mapping words to embedding indexes. + """ + def _word_to_index(sequence): + """Convert a sequence of words into a sequence of indexes that map each + word to a row in the embedding. + """ + id_sequence = [] + UNK = 399999 # index for unknown words + for word in sequence: + try: + id_sequence.append(word_to_id[word]) + except KeyError: + id_sequence.append(UNK) # if not in the word_to_id list set to UNK + return np.array(id_sequence) + + def input_fn(): + """Input function used to classify new reviews manually inserted.""" + # make review a sequence of words + review_split = review.split(' ') + # converting words to indexes + review_id = _word_to_index(review_split) + # calculates the length of the sequence + x_len = len(review_split) + # creates the dataset from in memory data + ds = tf.contrib.data.Dataset.from_tensors(review_id) + # the model expects a batch + ds = ds.batch(1) + + # creates iterator + x = ds.make_one_shot_iterator().get_next() + + dict_x = {'x': x, rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY: [x_len]} + # no label needed since we're only using this input function for prediction + # if training make sure to return a label + return dict_x, None + + return input_fn + diff --git a/extras/sentiment_analysis/model_fn_lib.py b/extras/sentiment_analysis/model_fn_lib.py new file mode 100644 index 0000000..484b9f5 --- /dev/null +++ b/extras/sentiment_analysis/model_fn_lib.py @@ -0,0 +1,166 @@ +# Copyright 2017 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== +"""RNN Model implementation using a model function for an estimator.""" +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function + +import tensorflow as tf +from tensorflow.contrib.learn.python.learn.estimators import rnn_common + + +def model_fn(features, labels, mode, params): + """Returns an EstimatorSpec. + + Args: + features(dict): a dictionary with the following keys and values. + { + 'x': tensor, + rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY: list + } Where, tensor.shape == [BATCH_SIZE, SEQUENCE_LENGTH] + and len(list) == BATCH_SIZE. + labels: tensor of shape [BATCH_SIZE] containing integers. The value is + 0 for a negative review or 1 if it's a positive review. + mode(str): equals to tf.estimator.ModeKeys.TRAIN, + tf.estimator.ModeKeys.EVAL or tf.estimator.ModeKeys.PREDICT. + params(dict): this contains hyperparameters allowing a more flexible + implementation. + """ + + # list containing the size of each RNN (int), they will be stacked in + # the order specified in the list + rnn_cell_sizes = params['rnn_cell_sizes'] + + # list containg the probability of applying dropout on each RNN cell + # each value should be a float from 0 to 1 + dropout_keep_probabilities = params['dropout_keep_probabilities'] + + # list containg the size of each dense layer in the model (int), they will + # be stacked on top of the last RNN in the order specified in the list + dnn_layer_sizes = params['dnn_layer_sizes'] + + # final label dimension, since this is a classification problem this + # is the number of classes + label_dimension = params['label_dimension'] + + # pretrained word embedding + pretrained_embeddings = params['word_vector'] + + # string, class or optimizer instance. String should be name of optimizer, + # like 'SGD', 'Adam', 'Adagrad', ... + # More details at: https://www.tensorflow.org/api_docs/python/tf/contrib/layers/optimize_loss + optimizer = params['optimizer'] + + # step size used by the optimizer + learning_rate = params['learning_rate'] + + # If true the final output from the RNNs will be the average of the hidden + # states, otherwise the output from the last cell will be used + average_hidden_states = (params['use_hidden_states'] == 'average') + + is_training = mode == tf.estimator.ModeKeys.TRAIN + is_eval = mode == tf.estimator.ModeKeys.EVAL + is_predict = mode == tf.estimator.ModeKeys.PREDICT + + review = features['x'] + sequence_length = tf.cast( + features[rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY], + tf.int32) + + # applying pre-trained embedding + W = tf.constant(pretrained_embeddings, name='W') + data = tf.nn.embedding_lookup(W, review) + + if dropout_keep_probabilities: + # if we're not training we want to keep all RNN cells + if is_training: + probabilities = dropout_keep_probabilities + else: + probabilities = [1] * len(dropout_keep_probabilities) + + # creating the LSTMCells and adding dropout + # check https://www.tensorflow.org/api_docs/python/tf/contrib/rnn for more + rnn_layers = [ + tf.nn.rnn_cell.DropoutWrapper(tf.nn.rnn_cell.LSTMCell(size), + output_keep_prob=keep_prob, + state_keep_prob=keep_prob) + for size, keep_prob in zip(rnn_cell_sizes, probabilities) + ] + + else: + # if not using dropout each RNN layer will consist of a regular LSTM cell + rnn_layers = [tf.nn.rnn_cell.LSTMCell(size) for size in rnn_cell_sizes] + + # stack the layers + multi_rnn_cell = tf.nn.rnn_cell.MultiRNNCell(rnn_layers) + + # runs the RNN dynamically + # more about it at https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn + # the output consists of a tuple with two values: + # outputs: a tensor with shape [BATCH_SIZE, SEQUENCE_LENGTH, STATE_SIZE] + # final state: tuple where the for each RNN layer (cell) there's a + # tf.contrib.rnn.LSTMStateTuple where: + # c is the hidden state and h is the output of a given cell + # https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/LSTMStateTuple + outputs, final_state = tf.nn.dynamic_rnn(cell=multi_rnn_cell, + inputs=data, + sequence_length=sequence_length, + dtype=tf.float32) + + if average_hidden_states: + dense_layer_input = tf.div( + tf.reduce_sum(outputs, axis=1), + tf.cast(sequence_length[:, tf.newaxis], dtype=tf.float32)) + else: + # slice to keep only the last cell of the RNN + # each value at final state is a LSTMStateTuple + dense_layer_input = final_state[-1].h + + # construct dense layers using tf.layers + for units in dnn_layer_sizes: + dense_layer_input = tf.layers.dense( + dense_layer_input, units, activation=tf.nn.relu) + + # final dense layer for prediction + predictions = tf.layers.dense(dense_layer_input, label_dimension) + predictions_softmax = tf.nn.softmax(predictions) + + # define model operations + loss = None + train_op = None + eval_op = None + + if not is_predict: + loss = tf.losses.sparse_softmax_cross_entropy(labels, predictions) + + if is_eval: + eval_op = { + 'accuracy': tf.metrics.accuracy( + tf.argmax(input=predictions_softmax, axis=1), + labels) + } + + if is_training: + train_op = tf.contrib.layers.optimize_loss( + loss, + tf.contrib.framework.get_global_step(), + optimizer=optimizer, + learning_rate=learning_rate) + + return tf.estimator.EstimatorSpec(mode, + predictions=predictions_softmax, + loss=loss, + train_op=train_op, + eval_metric_ops=eval_op) diff --git a/extras/sentiment_analysis/sentiment_analysis.py b/extras/sentiment_analysis/sentiment_analysis.py new file mode 100644 index 0000000..f03f591 --- /dev/null +++ b/extras/sentiment_analysis/sentiment_analysis.py @@ -0,0 +1,287 @@ +# Copyright 2017 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== +"""Train and evaluate a RNN Model used for Sentiment Analysis.""" +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function + +import argparse +import re + +from imdb import IMDB +from input_function_lib import build_classify_input_fn +from input_function_lib import build_input_fn +from model_fn_lib import model_fn +import numpy as np +import tensorflow as tf +from tensorflow.contrib.learn.python.learn import learn_runner # run an experiment +print('TensorFlow version', tf.__version__) + + +parser = argparse.ArgumentParser() + + +# script related args +parser.add_argument( + '--mode', type=str, choices=['train', 'classify'], default='train', + help='This defines how you want to execute this script.\n' + 'train: train and eval a new model and save it on model_dir\n' + 'classify: predict new reviews with a pretrained model using' + ' the standard IO') + +parser.add_argument( + '--data_path', type=str, default='data', + help='Path to where the data should be downloaded.') + +# tensorflow related args +parser.add_argument( + '--model_dir', type=str, default='sentiment_analysis_output', + help='The directory where the model outputs should be stored.') + +parser.add_argument('--run_experiment', dest='run_experiment', + action='store_true') +parser.add_argument('--dont_run_experiment', dest='run_experiment', + action='store_false') +parser.set_defaults(run_experiment=True) + +# IO related args +parser.add_argument('--batch_by_seq_len', dest='batch_by_seq_len', + action='store_true') +parser.add_argument('--dont_batch_by_seq_len', dest='batch_by_seq_len', + action='store_false') +parser.set_defaults(batch_by_seq_len=True) + +parser.add_argument( + '--train_batch_size', type=int, default=32, + help='Batch size used for training.') + +parser.add_argument( + '--eval_batch_size', type=int, default=32, + help='Batch size used for evaluation.') + +parser.add_argument( + '--sample_input_size', type=int, default=5, + help='Number of examples to be used for prediction.' + 'Those will be randomly chosen from the evaluation dataset.') + +# training related args +parser.add_argument( + '--num_epochs', type=int, default=8, + help='Num epochs used for training (for evaluation is always 1).') + +# tunning model +parser.add_argument( + '--optimizer', type=str, default='Adam', + help='Optimizer used for training.') + +parser.add_argument( + '--learning_rate', type=int, default=0.001, + help='Learning rate.') + +parser.add_argument('--use_hidden_states', type=str, + choices=['last', 'average'], default='last', + help='By default it will average the hidden states' + ' as describe in the paper linked in the tutorial.' + ' Otherwise will just consider the last output.') + +parser.add_argument( + '--rnn_cell_sizes', nargs='+', type=int, default=[128], + help='Size of the hidden state for each RNN cell.') + +parser.add_argument( + '--dnn_layer_sizes', nargs='+', type=int, default=[], + help='Size of the hidden state for each RNN cell.') + +parser.add_argument( + '--dropout_keep_probabilities', nargs='+', type=float, + default=[], + help='Dropout probabilities to keep the cell. ' + 'If provided should have the same length ' + 'as rnn_cell_sizes.') + +# model specific args +parser.add_argument( + '--num_classes', type=int, default=2, + help='Number of output classes. ' + 'For sentiment analysis is 2 (positive and negative)') + + +def ids_to_sentence(sequence, id_to_word): + """Given a sequence of numbers returns a string that represents it.""" + return ' '.join(id_to_word[index] for index in sequence) + + +def get_sample_data(x_dataset, y_dataset, sample_size): + """Randomly chooses sample_size elements from x_dataset and y_dataset.""" + indexes = np.random.randint(x_dataset.shape[0], + size=sample_size) + + return x_dataset[indexes], y_dataset[indexes] + + +def build_experiment_fn(estimator, train_input, eval_input): + """Return an Experiment function.""" + def _experiment_fn(run_config, hparams): + """Create experiment. + + Experiments perform training on several workers in parallel. In other + words Experiments know how to invoke train and eval in a sensible + fashion for distributed training. + + We first prepare an estimator, and bundle it together with input functions + for training and evaluation then collect all that in an Experiment object + that will train and evaluate our model. + """ + del run_config, hparams # unused args + return tf.contrib.learn.Experiment( + estimator, + train_input_fn=train_input, + eval_input_fn=eval_input + ) + return _experiment_fn + + +def validate_args(FLAGS): + """Validate arguments.""" + len_keep_prob = len(FLAGS.dropout_keep_probabilities) + if len_keep_prob > 0 and len_keep_prob != len(FLAGS.rnn_cell_sizes): + raise ValueError('If using dropout dropout_keep_probabilites ' + 'must have the same length as FLAGS.rnn_cell_sizes') + + +def print_instructions(): + """Print input instructions for classifaction mode.""" + print('INSTRUCTIONS') + print('=' * 40) + print('The model expects lower case letters or numbers as input' + ' (no special puntuaction will be removed and upper case letters will' + ' be converted to lower case.') + print('Review Example: this is a good movie i cant believe') + print('=' * 40) + print('If you see this warning message: "Input graph does not contain a' + ' Queue Runner..." ignore it; it is not fatal.') + + +def format_input(review): + """Remove non-alphanumeric chars and turns the chars to lower case.""" + pattern = re.compile(r'([^\s\w]|_)+') + return pattern.sub('', review).lower() + + +def main(unused_argv): + # validate args + FLAGS = parser.parse_args() + validate_args(FLAGS) + print(FLAGS) + + # get the data from https://github.com/adeshpande3/LSTM-Sentiment-Analysis. + print('Getting data...') + + imdb = IMDB(FLAGS.data_path) + x_train, y_train, x_eval, y_eval = imdb.get_data() + + print('Size of the train dataset:', x_train.shape[0]) + print('Size of the eval dataset:', x_eval.shape[0]) + + # creating sample dataset from the evaluation data + # used only for visualization + x_sample, y_sample = get_sample_data(x_eval, y_eval, FLAGS.sample_input_size) + + # creating run config + run_config = tf.contrib.learn.RunConfig(model_dir=FLAGS.model_dir) + # define model parameters + model_params = { + 'rnn_cell_sizes': FLAGS.rnn_cell_sizes, + 'label_dimension': FLAGS.num_classes, + 'word_vector': imdb.get_word_vector(), + 'dnn_layer_sizes': FLAGS.dnn_layer_sizes, + 'optimizer': FLAGS.optimizer, + 'learning_rate': FLAGS.learning_rate, + 'dropout_keep_probabilities': FLAGS.dropout_keep_probabilities, + 'use_hidden_states': FLAGS.use_hidden_states + } + + # creating estimator + estimator = tf.estimator.Estimator(model_fn=model_fn, + config=run_config, + params=model_params) + + if FLAGS.mode == 'train': + # defining input functions + # train input function + train_input_fn = build_input_fn(x_train, y_train, FLAGS.train_batch_size, + epochs=FLAGS.num_epochs, + batch_by_seq_len=FLAGS.batch_by_seq_len) + + # eval input function + eval_input_fn = build_input_fn(x_eval, y_eval, FLAGS.eval_batch_size, + epochs=1) + + # input function used to classify samples + sample_input_fn = build_input_fn(x_sample, y_sample, 1, epochs=1, + shuffle=False) + + if FLAGS.run_experiment: + # run training and evaluation + learn_runner.run( + build_experiment_fn(estimator, train_input_fn, eval_input_fn), + run_config=run_config + ) + pass + else: + # training + estimator.train(input_fn=train_input_fn) + + # evalutaion + estimator.evaluate(input_fn=eval_input_fn) + + # since we have a small number of predictions we're converting it to a + # list, when running `predict` on a big dataset is better to iterate in + # the predictions instead + predictions = list(estimator.predict(input_fn=sample_input_fn)) + + # loading map from index to word + index_to_word = imdb.get_index_to_word() + + # printing movie review, prediction and label + # for visualization + for i in range(FLAGS.sample_input_size): + print(ids_to_sentence(x_sample[i], index_to_word)) + print('Prediction:', predictions[i]) + print('Label:', y_sample[i]) + + elif FLAGS.mode == 'classify': + # loading map from word to index + word_to_index = imdb.get_word_to_index() + + print_instructions() + while True: + try: + review = format_input(raw_input('Write your review: ')) + except EOFError: + break + + print('Your review:', review) + print('Generating prediction...') + preds = estimator.predict(input_fn=build_classify_input_fn(review, + word_to_index)) + for p in preds: + print('Negative:', p[0]) + print('Positive:', p[1]) + + +if __name__ == '__main__': + tf.logging.set_verbosity(tf.logging.INFO) # enable TensorFlow logs + tf.app.run() diff --git a/extras/sentiment_analysis/sentiment_analysis_test.py b/extras/sentiment_analysis/sentiment_analysis_test.py new file mode 100644 index 0000000..7c20153 --- /dev/null +++ b/extras/sentiment_analysis/sentiment_analysis_test.py @@ -0,0 +1,130 @@ +# Copyright 2017 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== +"""Basic test for the Sentiment Analysis tutorial related files. +""" +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function + +from imdb import IMDB +from input_function_lib import build_input_fn +from model_fn_lib import model_fn +import numpy as np +import sentiment_analysis +import tensorflow as tf +from tensorflow.contrib.learn.python.learn.estimators import rnn_common + + +class BaseTest(tf.test.TestCase): + + def input_fn(self): + """Provides valid random features and labels.""" + # sequences of indexes (considering that all the sequences have length=250 + # [BATCH_SIZE, SEQUENCE_LENGTH] + features = { + 'x': tf.random_uniform([32, 250], 0, 400000, dtype=tf.int32), + rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY: [250] * 32 + } + # 0: negative review, 1: positive review + labels = tf.random_uniform([32], maxval=2, dtype=tf.int32) + return features, labels + + def get_word_vector(self): + """Provides a random word embedding.""" + return np.array(np.random.uniform(size=[40000, 50]), dtype=np.float32) + + def get_default_model_params(self): + """Returns model params for default model.""" + params = { + 'rnn_cell_sizes': FLAGS.rnn_cell_sizes, + 'label_dimension': FLAGS.num_classes, + 'word_vector': self.get_word_vector(), + 'dnn_layer_sizes': FLAGS.dnn_layer_sizes, + 'optimizer': FLAGS.optimizer, + 'learning_rate': FLAGS.learning_rate, + 'dropout_keep_probabilities': FLAGS.dropout_keep_probabilities, + 'use_hidden_states': FLAGS.use_hidden_states + } + return params + + def model_fn_helper(self, mode, params): + """Basic test for model_function.""" + + features, labels = self.input_fn() + + spec = model_fn(features, labels, mode, params) + + predictions = spec.predictions + self.assertAllEqual(predictions.shape[1], 2) + self.assertEqual(predictions.dtype, tf.float32) + + if mode != tf.estimator.ModeKeys.PREDICT: + loss = spec.loss + self.assertAllEqual(loss.shape, ()) + self.assertEqual(loss.dtype, tf.float32) + + if mode == tf.estimator.ModeKeys.TRAIN: + train_op = spec.train_op + self.assertAllEqual(train_op.shape, ()) + self.assertEqual(train_op.dtype, tf.float32) + + if mode == tf.estimator.ModeKeys.EVAL: + eval_metric_ops = spec.eval_metric_ops + self.assertAllEqual(eval_metric_ops['accuracy'][0].shape, ()) + self.assertAllEqual(eval_metric_ops['accuracy'][1].shape, ()) + self.assertEqual(eval_metric_ops['accuracy'][0].dtype, tf.float32) + self.assertEqual(eval_metric_ops['accuracy'][1].dtype, tf.float32) + + def test_model_fn_train_mode(self): + """Basic test for train mode.""" + params = self.get_default_model_params() + self.model_fn_helper(tf.estimator.ModeKeys.TRAIN, params) + + def test_model_fn_eval_mode(self): + """Basic test for eval mode.""" + params = self.get_default_model_params() + self.model_fn_helper(tf.estimator.ModeKeys.EVAL, params) + + def test_model_fn_predict_mode(self): + """Basic test for predict mode.""" + params = self.get_default_model_params() + self.model_fn_helper(tf.estimator.ModeKeys.PREDICT, params) + + def test_input_fn(self): + """Basic test for input function.""" + imdb = IMDB('data') + x_train, y_train, _, _ = imdb.get_data() + input_fn = build_input_fn(x_train, y_train, 32, epochs=1) + + features, labels = input_fn() + # shape + self.assertAllEqual(features['x'][0].shape, (250,)) + self.assertAllEqual( + features[rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY][0].shape, + () + ) + # type + self.assertAllEqual(features['x'][0].dtype, tf.int32) + self.assertAllEqual( + features[rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY][0].dtype, + tf.int32 + ) + self.assertAllEqual(labels.dtype, tf.int64) + + +if __name__ == '__main__': + FLAGS = sentiment_analysis.parser.parse_args() + tf.logging.set_verbosity(tf.logging.ERROR) # enable TensorFlow logs + tf.test.main() diff --git a/extras/sentiment_analysis/tutorial.md b/extras/sentiment_analysis/tutorial.md new file mode 100644 index 0000000..ae7124d --- /dev/null +++ b/extras/sentiment_analysis/tutorial.md @@ -0,0 +1,835 @@ +# Building a RNN model for Sentiment Analysis using TensorFlow’s high level APIs + +In this tutorial we're going to learn how to build a +[Recurrent Neural Network (RNN)](https://en.wikipedia.org/wiki/Recurrent_neural_network) +to classify movie reviews as positive or negative using TensorFlow high level APIs +([Estimators](https://www.tensorflow.org/extend/estimators), +[Datasets](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/docs_src/programmers_guide/datasets.md), +[tf.layers](https://www.tensorflow.org/api_docs/python/tf/layers)), these APIs +make it easier to build scalable and maintainable models that you can efficiently +train on a large amount of data. + +You may be thinking… “Why should I read another sentiment analysis tutorial using +TensorFlow if there are already many of those?” + +Sentiment analysis is a well know and easily understable problem that can be approached +with a RNN model. Also, it’s a real application and an interesting problem! Check +[this paper](https://arxiv.org/pdf/1708.00524.pdf) about detecting sentiment on text +using emojis occurrences. + +Our goal is not implementing the greatest sentiment analysis model ever, +but mainly to give a practical starting point to write your own Estimators models +using the new APIs and at the same time learn more about how to build RNN models +on TensorFlow. If you’re disappointed that we’re not getting state of the art accuracy, +check [this tutorial](https://www.tensorflow.org/tutorials/recurrent). + +## Introduction + +For this tutorial we understand you're already familiar with basic RNN +concepts and have implemented a basic TensorFlow model (for example, perhaps you’ve worked through the +[MNIST For ML Beginners tutorial](https://www.tensorflow.org/get_started/mnist/beginners)). +If you want to learn more about Estimators, and how they look like check: + * [Estimators](https://www.tensorflow.org/extend/estimators) + * [Effective TensorFlow for Non-Experts (Google I/O '17)](https://www.youtube.com/watch?v=5DknTFbcGVM&t=1217s) + +## Tutorial Files + +This tutorial references the following files at this folder. + +File | Purpose +--- | --- +`sentiment_analysis.py` | The code that does training, evaluation and prediction. +`model_fn_lib.py` | The model function implementation. +`input_function_lib.py` | The input function implementation. +`imdb.py` | The code to read the dataset. +`sentiment_analysis_test.py` | Basic test to `model_fn_lib.py` and `input_function_lib.py`. + +## About the Data + +We'll use the +[Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/), +which is a popular dataset for binary sentiment classification containing +25,000 reviews for training, and 25,000 reviews for testing with an +even number of positive and negative reviews. +``` +@InProceedings{maas-EtAl:2011:ACL-HLT2011, + author = {Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher}, + title = {Learning Word Vectors for Sentiment Analysis}, + booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies}, + month = {June}, + year = {2011}, + address = {Portland, Oregon, USA}, + publisher = {Association for Computational Linguistics}, + pages = {142--150}, + url = {http://www.aclweb.org/anthology/P11-1015} +} +``` + +Examples of reviews in the dataset: + +* **Negative review**: I was very curious to see this film, after having heard that +it was clever and witty. I had to stop halfway because of the unbearable boredom I +felt... First of all, the film was so down-to-earth that it looked as if, by +describing the problems that a couple must solve on a day-to-day basis, it became +itself ordinary and dull. Secondly, the overall sloppiness of the production, with +dialogues that were barely understandable. Too bad. + +* **Positive Review**: Cinematically, this film stinks. So does a lot of the acting. +But I don't care. If there is a strong representation of what the 80's were like... +This is the best of all the hip-hop/break dancing movies that came out around that +period. Of course the 80's are considered a joke now with all the bad tv shows and +movies, but those of us who lived through it will always remember it fondly for a time +when music, dancing, and graffiti were fresh, yo! + +### Prepare the Data + +When dealing with +[NLP](https://en.wikipedia.org/wiki/Natural_language_processing) +tasks it's very important to preprocess your data and to choose a good +way to represent it, we're not going into details about how doing it in +the best way possible (this could be another complete tutorial), instead +we're just going to describe how we did for this particular problem and +other known popular approaches. + +#### Preprocess the Data + +In this example the reviews were padded with zeros or truncated to have +length equal 250. This was done mainly in order to avoid very long sequences. + +Padding is a common practice when working with RNNs but is not mandatory when +working with RNNs on TensorFlow and is not the most efficient approach to pad +all the sequences to the same length, we're going into more details about why +this is not efficient and what are other more efficient approaches later in this +tutorial. + +All characters were converted to lowercase and punctuation was removed +for simplicity. + +> Note: punctuation and special characters can say a lot about emotions, + in order to have more expressive results maybe keeping and treating those + correctly can be a good idea. + +#### Numeric Representation + +Neural Networks expect numeric inputs, this means we need to represent +text as a numeric value. There are many possible approaches, two classical +ways to do this are to: + * Segment the text into words, representing each word as a vector; + * Segment the text into characters, representing each character as a vector. + +Once you segmented the text, another question is how to represent the words +or characters as vectors? There are also two popular ways to do so: + * [One hot representation](https://www.tensorflow.org/api_docs/python/tf/one_hot): + sparse and high-dimensional vectors (not so efficient when dealing with + sequences in a word level, but can be really useful when dealing + with sequences in a character level); + * [Word Embedding](https://www.tensorflow.org/tutorials/word2vec): + also called word vectors, dense and low-dimensional vectors. + You can train your own embedding along with your model, or you can + use a pre-trained embedding. + +In this tutorial we'll segment the reviews into words and use a pre-trained +word embedding to convert the words to a vector representation. This word +embedding was trained using the +[GloVe algorithm](https://nlp.stanford.edu/projects/glove/) +and contains 400000 words as 50 dim vectors. As a result we have a matrix +of shape [400000, 50] where each row is a word representation. + +> Note: Thanks to [@adeshpande3](https://github.com/adeshpande3/LSTM-Sentiment-Analysis) + for providing this word embedding and this great tutorial + [Sentiment Analysis tutorial using low-level TensorFlow by O'Reilly](https://preview.oreilly.com/learning/perform-sentiment-analysis-with-lstms-using-tensorflow). + +![](../../images/sentiment_analysis_embedding.png) + +Each word in the review will be converted to an index that points to a row in the +embedding. Each row has a 50 dim vector that better represents a particular +word. + +## About the Model + +Our model will consist of a +[LSTM cell](https://www.tensorflow.org/versions/master/api_docs/python/tf/contrib/rnn/LSTMCell) +with a dense softmax layer on top of it. The final output is the probability of a +review to be a positive (index 1) or negative review (index 0). + +![](../../images/sentiment_analysis_model.png) + +Before going into more details about the model itself, let's discuss what is needed +in order to implement an Estimator on TensorFlow. + +### Estimators + +Estimators are a high-level abstraction that support all the basic +operations you need on a Machine Learning model. They encode best +practices, are ready for deployment with [tensorflow/serving](https://www.tensorflow.org/serving/) +and are distributed and scalable by design. + +![](../../images/estimator.png) +*Image from Effective TensorFlow for Non-Experts (Google I/O '17)* + +In order to implement our own Estimator we basically need: + * An [input function](https://www.tensorflow.org/get_started/input_fn): + the input pipeline implementation, where you're going to + process your data and return the features and labels that will be used + for training, evaluation and prediction using the Estimator interface. + * A [model function](https://www.tensorflow.org/extend/estimators#constructing_the_model_fn): + where will actually define our model, and the training, evaluation and + prediction operations. + +Now let's have a look at the code! + +### Input function + +If your data fits in memory (and you're okay about loading it in memory) +there are "prebuilt" input functions for +[numpy](https://www.tensorflow.org/versions/master/api_docs/python/tf/estimator/inputs/numpy_input_fn) +and [pandas](https://www.tensorflow.org/versions/master/api_docs/python/tf/estimator/inputs/pandas_input_fn). + +But if you need to manipulate the data with more complex operations or if +the data doesn't fit in memory, an efficient and scalable way to implement +your own input function is to use the +[Dataset API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/docs_src/programmers_guide/datasets.md). + +"*The Dataset API enables you to build complex input pipelines from simple, +reusable pieces, making it easy to deal with large amounts of data, different +data formats, and complicated transformations.*" + +Here's an input function implementation using the Dataset API. + +```python +def build_input_fn(x_in, y_in, batch_size, + shuffle=True, epochs=1, + max_length=250): + """Returns an input function created from word and class index arrays. + Args: + x_in: A numpy array of word indexes with shape (num_examples, + max_sequence_length). The array is padded on the right with zeros. + y_in: A numpy array of class indexes with shape (num_examples) + batch_size: Batch size for the input_fn to return + shuffle: A bool, indicating whether to shuffle the data or not. + epochs: Number of epochs for the input fun to generate. + max_length: Truncate sequences longer than max_length. + Returns: + An `input_fn`. + """ + def input_fn(): + """Input function used for train and eval; usually not called directly. + """ + # calculates the length of the sequences + # since the inputs are already padded with zeros in the end + # the length will be the last index that is non zero + 1 + x_len = np.array( + [np.nonzero(seq)[0][-1] + 1 for seq in x_in]).astype('int32') + + # creates the dataset from in memory data + # x_in: sequence of indexes that map a word to an embedding + # x_len: sequence lengths + # y_in: 1 if positive review, 0 if negative review + ds = tf.contrib.data.Dataset.from_tensor_slices((x_in, x_len, y_in)) + + # repeats the dataset `epochs` times + ds = ds.repeat(epochs) + + if shuffle: + # make sure the buffer is big enough for your data + ds = ds.shuffle(buffer_size=25000 * 2) + + ds = ds.batch(batch_size) + + # creates iterator + x, x_len, y = ds.make_one_shot_iterator().get_next() + + # feature must be a dictionary + dict_x = {'x': x, rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY: x_len} + return dict_x, y + + return input_fn +``` + +The Dataset API introduces two new abstractions to TensorFlow: **datasets** +and **iterators**. + +* A Dataset can either be a source or a transformation: + * Creating a source (e.g. Dataset.from_tensor_slices()) constructs a dataset + from one or more tf.Tensor objects. + * Applying a transformation constructs a dataset from one or more + tf.contrib.data.Dataset objects. + * Repeat: produce multiple epochs; + * Shuffle: it maintains a fixed-size buffer and chooses the next element + uniformly at random from that buffer; + * Batch: constructs a dataset by stacking consecutive elements of another + dataset into a single element. + +* An Iterator provides the main way to extract elements from a dataset. + The Iterator.get_next() operation yields the next element of a Dataset, and + typically acts as the interface between input pipeline code and your model. + +Most of this content is from the [Dataset API documentation](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/docs_src/programmers_guide/datasets.md), +where you can learn more about this API. + +The implementation above is not the most efficient way to batch the data for +a RNN, we're wasting time and space by padding all the batches to have length +equals to 250. + +![](../../images/regular_batch.png) +*Image from Sequence Models and the [RNN API (TensorFlow Dev Summit 2017)](https://www.youtube.com/watch?v=RIR_-Xlbp7s)* + +One possible approach to make it more efficient is to put sequences with +similar length in the same batch. We implement this in the code below. + +```python +def build_input_fn(x_in, y_in, batch_size, + shuffle=True, epochs=1, + batch_by_seq_len=False, + max_length=250): + """Returns an input function created from word and class index arrays. + Args: + x_in: A numpy array of word indexes with shape (num_examples, + max_sequence_length). The array is padded on the right with zeros. + y_in: A numpy array of class indexes with shape (num_examples) + batch_size: Batch size for the input_fn to return + shuffle: A bool, indicating whether to shuffle the data or not. + epochs: Number of epochs for the input fun to generate. + batch_by_seq_len: A bool to activate sequence length batching. + max_length: Truncate sequences longer than max_length. + Returns: + An `input_fn`. + """ + def _length_bin(length, max_seq_len, length_step=10): + """Sets the sequence length bin.""" + bin_id = (length // length_step + 1) * length_step + return tf.cast(tf.minimum(bin_id, max_seq_len), tf.int64) + + def _make_batch(key, ds): + """Removes extra padding and batchs the bin.""" + # eliminate the extra padding + key = tf.cast(key, tf.int32) + ds = ds.map(lambda x, x_len, y: (x[:key], x_len, y)) + + # convert the entire contents of the bin to a batch + ds = ds.batch(batch_size) + return ds + + def input_fn(): + """Input function used for train and eval; usually not called directly. + """ + # calculates the length of the sequences + # since the inputs are already padded with zeros in the end + # the length will be the last index that is non zero + 1 + x_len = np.array( + [np.nonzero(seq)[0][-1] + 1 for seq in x_in]).astype('int32') + + # creates the dataset from in memory data + # x_in: sequence of indexes that map a word to an embedding + # x_len: sequence lengths + # y_in: 1 if positive review, 0 if negative review + ds = tf.contrib.data.Dataset.from_tensor_slices((x_in, x_len, y_in)) + + # repeats the dataset `epochs` times + ds = ds.repeat(epochs) + + if shuffle: + # make sure the buffer is big enough for your data + ds = ds.shuffle(buffer_size=25000 * 2) + + if batch_by_seq_len: + # implement a simple `Dataset` version of `bucket_by_sequence_length` + # https://goo.gl/y67FQm + ds = ds.group_by_window( + key_func=lambda x, x_len, y: _length_bin(x_len, max_length), + reduce_func=_make_batch, + window_size=batch_size) + else: + ds = ds.batch(batch_size) + + # creates iterator + x, x_len, y = ds.make_one_shot_iterator().get_next() + + # feature must be a dictionary + dict_x = {'x': x, rnn_common.RNNKeys.SEQUENCE_LENGTH_KEY: x_len} + return dict_x, y + + return input_fn +``` + +We're using a transformation called `group_by_window` that maps each +consecutive element in this dataset to a key using a `key_func` and then +groups the elements by key. It then applies `reduce_func` to at most +`window_size` elements matching the same key. + +We're using the `group_by_window` transformation to batch reviews that +have similar length together, since the batches are created based on the +sequence length the reviews in the same batch will have approximately the +same length which means we'll have less padding saving space and computation +time. + +Using this more complex implementation, that we called `batch_by_seq_len`, +on this specific dataset I can see an improvement of 2 global_step/sec +running it on my local machine, in other words if we take ~16 seconds to +process 100 batches using the usual batch implementation, now we take ~12 +seconds to process 100 batches. + +![](../../images/batch_by_length.png) +*Image from Sequence Models and the [RNN API (TensorFlow Dev Summit 2017)](https://www.youtube.com/watch?v=RIR_-Xlbp7s)* + +For more details about padding and batching with RNNs watch +this great talk: +[Sequence Models and the RNN API (TensorFlow Dev Summit 2017)](https://youtu.be/RIR_-Xlbp7s?t=4m14s). + +You can see the all the input function implementations used in this +tutorial at [`input_function_lib.py`](input_function_lib.py). + +We can create different input functions calling `build_input_fn`. + +```python +# defining input functions +# train input function +train_input_fn = build_input_fn(x_train, y_train, FLAGS.train_batch_size, + epochs=FLAGS.num_epochs, + batch_by_seq_len=FLAGS.batch_by_seq_len) + +# eval input function +eval_input_fn = build_input_fn(x_eval, y_eval, FLAGS.eval_batch_size, + epochs=1) + +# input function used to classify samples +sample_input_fn = build_input_fn(x_sample, y_sample, 1, epochs=1, + shuffle=False) +``` + +### Model Definition + +We'll define our model implementing a model function, where we'll also define +the operations used for training, evaluation and prediction. In this tutorial +we'll focus on the model itself, and we'll comment briefly about the +operations chosen, since you can easily learn more about them in the TensorFlow +documentation and other online materials. + +Our model function definition looks like: + +```python +def model_fn(features, labels, mode, params): + # model and operations definition + ... + # estimator definition + return EstimatorSpec(...) +``` + +Where the `features` and `labels` are returned by the input function we just +defined, the `mode` is a string value indicating the context in which the +`model_fn` was invoked (TRAIN, EVAL, PREDICT) and `params` is an optional +argument containing a dict of hyperparameters used for training. More details +[here](https://www.tensorflow.org/extend/estimators#constructing_the_model_fn). + +The complete model implementation can be found at [`model_fn_lib.py`](model_fn_lib.py). + +Let's have look at the model function implementation. + +#### Embedding + +First, we need to represent the words as vectors, the `features['x']` +is a tensor with indexes mapping the word to a row in the `pretrained_embeddings` +matrix (the pre-trained word embedding). + +This is simplest code to load the embedding and convert the indexes to +vectors, there's a discussion about how to do this in a more efficient way +[here](https://stackoverflow.com/questions/35687678/using-a-pre-trained-word-embedding-word2vec-or-glove-in-tensorflow). + +```python + # get the sequences from the features dict + review = features['x'] + + # applying pre-trained embedding + W = tf.constant(pretrained_embeddings, name='W') + data = tf.nn.embedding_lookup(W, review) +``` + +Once we converted the indexes to actual vectors, the `data` variable defined +above will be a 3-dim vector with shape [BATCH_SIZE, MAX_LENGTH, 50] + +![](../../images/sentiment_analysis_input_shape.png) + +#### RNN + +Now that we have our input in the expected format we can implement the model +itself. + +Our model consists of a Recurrent Neural Network (RNN). Neural networks +like densely-connected networks and Convolutional Neural Networks +have no memory, which means that each input is processed independently. +RNNs are neural networks that have memory, in other words they have +an internal state that is updated based on the seen inputs and on +it's own previous state (memory). + +![](../../images/sentiment_analysis_RNN.jpg) +*Image from: http://colah.github.io/posts/2015-08-Understanding-LSTMs/* + +RNNs can be seen as multiple copies (cells) of the same network, +where each copy shares information about what it has seen to the next cell. + +![](../../images/sentiment_analysis_RNN_unfold.jpg) +*Image from: http://colah.github.io/posts/2015-08-Understanding-LSTMs/* + +The Pseudo-code to run and update the state of a basic RNN cell +would be something similar to: + +```python +for t in range(len(inputs)): + # updates internal state + hidden_state = activation_fn(W_x * inputs[t] + W_h * hidden_state + bias) +``` + +To learn more about RNNs check: + * [Understanding LSTM Networks at colah's blog](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) + * [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) + +In this tutorial we'll actually implement a special type of RNN cell called +Long Sort Term Memory Cell (LSTM), which is capable of learning +long-term dependencies. + +TensorFlow allow us to implement complex cell types and operations in a few +lines of code. In the code below we're creating multiple LSTMCells, +then adding dropout in the output and hidden state of each of them if running in +training mode. + +```python +if dropout_keep_probabilities:0 + # if we're not training we want to keep all RNN cells + if is_training: + probabilities = dropout_keep_probabilities + else: + probabilities = [1] * len(dropout_keep_probabilities) + + # creating the LSTMCells and adding dropout + # check https://www.tensorflow.org/api_docs/python/tf/contrib/rnn for more + rnn_layers = [ + tf.nn.rnn_cell.DropoutWrapper(tf.nn.rnn_cell.LSTMCell(size), + output_keep_prob=keep_prob, + state_keep_prob=keep_prob) + for size, keep_prob in zip(rnn_cell_sizes, probabilities) + ] +``` + +Once we created the cells, we can stack them. + +```python +multi_rnn_cell = tf.nn.rnn_cell.MultiRNNCell(rnn_layers) +``` + +All the RNN code above just created the RNN cells (you can see all the RNN +cell types available +[here](https://www.tensorflow.org/versions/master/api_docs/python/tf/nn/rnn_cell)) +in order to "unroll" the cells we can use +[tf.nn.dynamic_rnn](https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn) +that will actually implement the RNN "for loop", returning all the outputs +over time and the final state. + +```python +# outputs: a tensor with shape [BATCH_SIZE, SEQUENCE_LENGTH, STATE_SIZE] +# final state: tuple where the for each RNN layer (cell) there's a +# tf.contrib.rnn.LSTMStateTuple where: +# c is the hidden state and h is the output of a given cell +# https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/LSTMStateTuple +outputs, final_state = tf.nn.dynamic_rnn(cell=multi_rnn_cell, + inputs=data, + sequence_length=sequence_length, + dtype=tf.float32) +``` + +On top of the RNN we can add other neural network layers, for this example +we'll add a dense layer, in order to stack these two layers we need to define +what will be the final output from the RNN layers. + +In this implementation we can get the output from the last step (usual +implementation) or get the average from the hidden states as suggested by +["Sentiment Analysis with Deeply Learned Distributed Representations of Variable Length Texts" from James Hong and Michael Fang (2015)](http://cs224d.stanford.edu/reports/HongJames.pdf). + +```python +if average_hidden_states: + dense_layer_input = tf.div( + tf.reduce_sum(outputs, axis=1), + tf.cast(sequence_length[:, tf.newaxis], dtype=tf.float32)) +else: + # slice to keep only the last cell of the RNN + # each value at final state is a LSTMStateTuple + dense_layer_input = final_state[-1].h + +``` + +#### Dense Softmax Layer + +Adding dense layers to the model is very straight forward with the +[tf.layers API](https://www.tensorflow.org/api_docs/python/tf/layers). + +```python +# final dense layer for prediction +predictions = tf.layers.dense(dense_layer_input, label_dimension) +predictions_softmax = tf.nn.softmax(predictions) +``` + +#### Defining operations + +After defining our model we can just specify which operations to run on each +execution mode. We are using accuracy as evaluation metric, calculating +the loss using Softmax Cross Entropy, specifying how to optimize the loss, +and defining the predict operation. + +```python +# define model operations +loss = None +train_op = None +eval_op = None + +if not is_predict: + loss = tf.losses.sparse_softmax_cross_entropy(labels, predictions) + +if is_eval: + eval_op = { + 'accuracy': tf.metrics.accuracy( + tf.argmax(input=predictions_softmax, axis=1), + labels) + } + +if is_training: + train_op = tf.contrib.layers.optimize_loss( + loss, + tf.contrib.framework.get_global_step(), + optimizer=optimizer, + learning_rate=learning_rate) + +return tf.estimator.EstimatorSpec(mode, + predictions=predictions_softmax, + loss=loss, + train_op=train_op, + eval_metric_ops=eval_op) +``` + +## Training + +Now we can just create the Estimator using the model function above, +and call the methods available on the Estimator interface. + +```python +estimator.train(input_fn=train_input_fn) +``` + +``` +INFO:tensorflow:loss = 0.691962, step = 101 (36.537 sec) +INFO:tensorflow:global_step/sec: 3.79198 +INFO:tensorflow:loss = 0.637554, step = 201 (26.371 sec) +INFO:tensorflow:global_step/sec: 4.12 +INFO:tensorflow:loss = 0.461921, step = 301 (24.272 sec) +INFO:tensorflow:global_step/sec: 4.23288 +INFO:tensorflow:loss = 0.456651, step = 401 (23.625 sec) +INFO:tensorflow:global_step/sec: 4.18946 +INFO:tensorflow:loss = 0.603483, step = 501 (23.869 sec) +INFO:tensorflow:global_step/sec: 4.07666 +INFO:tensorflow:loss = 0.617782, step = 601 (24.530 sec) +.... +INFO:tensorflow:loss = 0.696719, step = 1001 (24.596 sec) +INFO:tensorflow:global_step/sec: 4.03502 +INFO:tensorflow:loss = 0.519887, step = 1101 (24.783 sec) +INFO:tensorflow:global_step/sec: 3.93356 +INFO:tensorflow:loss = 0.579439, step = 1201 (25.422 sec) +INFO:tensorflow:global_step/sec: 3.87702 +``` + +## Evaluation + +```python +estimator.evaluate(input_fn=eval_input_fn) +``` + +``` +INFO:tensorflow:Evaluation [1/100] +INFO:tensorflow:Evaluation [2/100] +INFO:tensorflow:Evaluation [3/100] +INFO:tensorflow:Evaluation [4/100] +INFO:tensorflow:Evaluation [5/100] +INFO:tensorflow:Evaluation [6/100] +INFO:tensorflow:Evaluation [7/100] +INFO:tensorflow:Evaluation [8/100] +INFO:tensorflow:Evaluation [9/100] +INFO:tensorflow:Evaluation [10/100] +... +INFO:tensorflow:Evaluation [100/100] +INFO:tensorflow:Finished evaluation at 2017-07-24-20:39:32 +INFO:tensorflow:Saving dict for global step 6262: accuracy = 0.856875, global_step = 6262, loss = 0.374715 +``` + +## Training and Evaluation in a Distributed Environment + +As mentioned before a great thing about Estimators is that they are +distributed and scalable by design. In order to run the model in a distributed +way using data-parallelism you just need to create an +[Experiment](https://www.tensorflow.org/api_docs/python/tf/contrib/learn/Experiment). +Experiments know how to invoke train and eval in a sensible fashion for +distributed training. + +Below is the code used to create and run an Experiment. + +```python +def build_experiment_fn(estimator, train_input, eval_input): + """Return an Experiment function.""" + def _experiment_fn(run_config, hparams): + """Create experiment. + Experiments perform training on several workers in parallel. In other + words Experiments know how to invoke train and eval in a sensible + fashion for distributed training. + We first prepare an estimator, and bundle it together with input functions + for training and evaluation then collect all that in an Experiment object + that will train and evaluate our model. + """ + del run_config, hparams # unused args + return tf.contrib.learn.Experiment( + estimator, + train_input_fn=train_input, + eval_input_fn=eval_input + ) + return _experiment_fn +``` + +## Predicting + +Once we're done training we can check how well the model classify new reviews. + +In this case we're just classifying 5 new sentences randomly chosen from the +eval dataset. In the sentences below *unk* stands for a word that isn't +represented in the word embedding. + +```python +predictions = estimator.predict(input_fn=sample_input) +``` + +``` +if this is the best commander hamilton movie i have no curiosity about the others a movie actors greatest tools are his eyes but when peter stormare wants to show great emotion he closes his so for five or six seconds we get to admire his eyelids while his feelings remain unknown behind them lousy acting technique stormare also flinches sometimes when he fires a gun turning his head away and clamping his eyes shut watch carefully james bond can rest easy with competition like this there are some interesting supporting performances from other actors but not enough to hang a whole movie on the cinematography is unk doing a fine job of capturing the nordic cold even the sahara winds up looking cold perhaps hamilton carries his own climate with him there are some individual good action sequences here unfortunately the only sense of humor on screen belongs to the villain which turns the hero into a big pill james bonds jokes may not be particularly good but at least he doesnt look constipated all the time one positive point in the movies favor is that the psychotic contorted vicious hatred of israel in unk books has been left out what has been kept in is worship of a noble heroic plo that he shows us functioning in libya without the dictator unk knowledge or supervision this fantasy is hard to believe since unk actually threw the plo out of libya for four years at a time and at +Prediction: [ 0.1009393 0.89906073] +Label: 0 +this is the most stupid movie ever made the story is laughable his wife and kid think hes insane then they dont then it turns out he is and i think they knew it all along there is a dog named ned that causes some problems and i think its all his unk does jim carey god only knows why virginia madsen took this unk is a career sinker i think the target audience for this is 11 and 12 year olds and that adds up to 23 or maybe its for 8 and 10 years olds which also adds up to 23 or maybe its for really dumb 23 year olds or maybe really dumb 32 year olds because thats 23 in reverse or maybe 46 year olds would enjoy it because half of that is 23 i think looking up things on the internet about the number 23 would be more entertaining than this movie unless you wanted to see a comedy 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 +Prediction: [ 0.74981982 0.25018018] +Label: 0 +there isnt much that comes close to the unk storytelling and suspenseful unk levels as goldeneye when it came out it was the greatest game of alltime and even today it stays strong i will admit that this game did get boring after a few months of playing and by not playing it again until two years later i was thrust back into its greatest almost as if i was playing it for the first time again there are 20 unk levels which is probably the most of any james bond game to date probably the most unforgettable one is the tank level which was likely the most explosive video game sequence at that time and the unk shooting as well as usage of q gadgets is what james bond fans are always dying to use frankly as a james bond fan i look for aspects of a true james bond experience which are now showing up in the ps2 games so this game while it has some great action and usable gadgets i was somewhat expecting a little more even back in 1997 i also disliked that this game didnt have q or m or moneypenny or anyone from mi6 while watching the movies bond interacts with these characters at least a few times throughout each movie but they are nowhere to be seen in this game and vocal dialogue would have made the game more lively rather than the text dialogue they wound up using they had the +Prediction: [ 0.12032966 0.87967038] +Label: 1 +as a true canadian i always avoid canadian movies however now and then i get trapped into watching one this one is better than most which is to say mediocre it has many of the usual flaws of canadian unk unk excess of cinematic gimmicks and above all the unk canadian habit of using canadian cities as unk for american ones i mean using the historic metropolis of montreal as a stand in for harrisburg pennsylvania is just short of obscene i was in a generous mood i gave it a 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 +Prediction: [ 0.71495974 0.28504029] +Label: 0 +if youre interested in learning about the real side of spying this movie is for you unlike 007 movies this shows how things really go down in the world of espionage timothy hutton and sean penn both give outstanding performances in this unk film certainly worth watching 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 +Prediction: [ 0.25497162 0.74502844] +Label: 1 +``` + +### Classifying New Data + +You can also try to classify new sentences with this model. + +```shell +# The script will load your model and you can use it to classify new sentences +# by default the model_dir is "sentiment_analysis_output" +$ python sentiment_analysis.py --mode=classify --model_dir="sentiment_analysis_output" +``` +Make sure you pass the same arguments to `sentiment_analysis.py` and +`sentiment_analysis.py --mode=classify`, since it will load the same model you +just trained make sure you're building the same model running in both modes. + +Here are some sentences we tried with a model that got 82% accuracy, we can get +about 86% accuracy with the model training it for ~6000 steps. + +``` +Write your review (or type to exit): it was fine i guess +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.527832 +Positive: 0.472169 +Write your review (or type to exit): it was good +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.357005 +Positive: 0.642995 +Write your review (or type to exit): it wasnt good +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.60162 +Positive: 0.39838 +Write your review (or type to exit): this is a great movie +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.130054 +Positive: 0.869946 +Write your review (or type to exit): its not that bad +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.740116 +Positive: 0.259884 +Write your review (or type to exit): it is bad +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.906015 +Positive: 0.0939852 +Write your review (or type to exit): its not bad i think its a good +movie +Generating prediction... +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.489538 +Positive: 0.510462 +Write your review (or type to exit): its not good i think its a bad +movie +INFO:tensorflow:Restoring parameters from pretrained_model/model.ckpt-4380 +Negative: 0.538815 +Positive: 0.461185 +``` + +We can see that the model learned some interesting relations, +but is definitely not perfect and can be improved. + +## Visualizing your Model with TensorBoard + +When using estimators you can also visualize your data in +[TensorBoard](https://www.tensorflow.org/get_started/summaries_and_tensorboard), +with no changes in your code. You can use TensorBoard to visualize your TensorFlow graph, +plot quantitative metrics about the execution of your graph, +and show additional data like images that pass through it. + +Here's what you see if you run TensorBoard in the `model_dir` you used for your model. + +```shell +# Check TensorBoard during training or after it. +# Just point TensorBoard to the model_dir you chose on the previous step +# by default the model_dir is "sentiment_analysis_output" +$ tensorboard --log_dir="sentiment_analysis_output" +``` + +![](../../images/sentiment_analysis_tensorboard.png) + +You can also visualize your TensorFlow graph, which is very useful for debugging purposes. + +![](../../images/sentiment_analysis_tensorboard_graph.png) + +## What's next? + +In this tutorial we showed how to implement a recurrent neural network for +binary sentiment analysis using TensorFlow high level APIs. + +* We encourage you to run the code and see how the model performs for yourself. + The model parameters were not tuned, so a good exercise is just play with + the parameters and in order to have better results. + Try changing the learning rate, optimizer, hidden state size, + number of RNN cells, number of DNN layers, and so on. + +* Finally, the model presented above can be easily changed to be used on + different data and even perform different classification or prediction tasks. + More details can be seen in the code presented here. + A great example is + [colorbot](../colorbot/) + a deep RNN model that receives a word (sequence of characters) as input and + learns to predict a rgb value that better represents this word. As a result + we have a color generator! + +![](../../images/colorbot_prediction_sample.png) + +* Learn more about: + * [RNNs](https://www.tensorflow.org/tutorials/recurrent) + * [Estimators](https://www.tensorflow.org/versions/master/api_docs/python/tf/estimator/Estimator) + * [Dataset API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/docs_src/programmers_guide/datasets.md) + * [Word Embeddings](https://www.tensorflow.org/tutorials/word2vec) + diff --git a/images/colorbot_prediction_sample.png b/images/colorbot_prediction_sample.png new file mode 100644 index 0000000..a023416 Binary files /dev/null and b/images/colorbot_prediction_sample.png differ diff --git a/images/estimator.png b/images/estimator.png new file mode 100644 index 0000000..e103264 Binary files /dev/null and b/images/estimator.png differ diff --git a/images/regular_batch.png b/images/regular_batch.png new file mode 100644 index 0000000..199d027 Binary files /dev/null and b/images/regular_batch.png differ diff --git a/images/sentiment_analysis_RNN.jpg b/images/sentiment_analysis_RNN.jpg new file mode 100644 index 0000000..23656ab Binary files /dev/null and b/images/sentiment_analysis_RNN.jpg differ diff --git a/images/sentiment_analysis_RNN_unfold.jpg b/images/sentiment_analysis_RNN_unfold.jpg new file mode 100644 index 0000000..75fac91 Binary files /dev/null and b/images/sentiment_analysis_RNN_unfold.jpg differ diff --git a/images/sentiment_analysis_embedding.png b/images/sentiment_analysis_embedding.png new file mode 100644 index 0000000..b31468c Binary files /dev/null and b/images/sentiment_analysis_embedding.png differ diff --git a/images/sentiment_analysis_input_shape.png b/images/sentiment_analysis_input_shape.png new file mode 100644 index 0000000..634b74d Binary files /dev/null and b/images/sentiment_analysis_input_shape.png differ diff --git a/images/sentiment_analysis_model.png b/images/sentiment_analysis_model.png new file mode 100644 index 0000000..83721f6 Binary files /dev/null and b/images/sentiment_analysis_model.png differ diff --git a/images/sentiment_analysis_tensorboard.png b/images/sentiment_analysis_tensorboard.png new file mode 100644 index 0000000..c4dd4ee Binary files /dev/null and b/images/sentiment_analysis_tensorboard.png differ diff --git a/images/sentiment_analysis_tensorboard_accuracy.png b/images/sentiment_analysis_tensorboard_accuracy.png new file mode 100644 index 0000000..0a2fe74 Binary files /dev/null and b/images/sentiment_analysis_tensorboard_accuracy.png differ diff --git a/images/sentiment_analysis_tensorboard_graph.png b/images/sentiment_analysis_tensorboard_graph.png new file mode 100644 index 0000000..b857db8 Binary files /dev/null and b/images/sentiment_analysis_tensorboard_graph.png differ