POCASUM : Policy Categorizer and Summarizer Based on Text Mining and Machine Learning

Abstract

Having control over your data is a right and a duty that every citizen has in our digital society. It is often that users skip entire policies of applications or websites to save time and energy without realizing the potential sticky points in these policies. Due to obscure language and verbose explanations majority of users of hypermedia do not bother to read them. Further, sometimes digital media companies do not spend enough effort in stating their policies clearly which often time can also be incomplete. A summarized version of these privacy policies that can be categorized into the useful information can help the users. To solve this problem, in this work we propose to use machine learning based models for policy categorizer that classifies the policy paragraphs under the attributes proposed like security, contact etc. By benchmarking different machine learning based classifier models, we show that artificial neural network model performs with higher accuracy on a challenging dataset of textual privacy policies. We thus show that machine learning can help summarize the relevant paragraphs under the various attributes so that the user can get the gist of that topic within a few lines.

1. Introduction

Controlling and protecting your personal data is a prerogative unless you allow someone to have control over it. Privacy policies mentions the regulation which the particular application promises to follow regarding the user’s data. It usually mentions what kind of data they collect from you, how they use the data, under which circumstances is the data shared, who has the access of your data, the security regarding the storage of data, the update policies and a lot more. It intends to mention beforehand of all the risk the user might undertake by agreeing to the policies of the application. The real problem lies in the inability of the user to comprehend the policy and understand the risks associated with it due to a plethora of reasons. With increasing technology almost everyone has a smartphone today which allows even illiterate people to gain access to these applications. In general, due to their lack of techinical knowledge they have to accept the policies without reading and understanding them. However, that is a relatively small percentage, the remaining majority of users of these applications and websites in spite of having the required education do simply click to agree to the policies without even reading in whole since it requires an arduous effort in understanding the verbose and technical terms utilized in these privacy policies.

The formal language of the policies along with its confusing semantics makes it irksome for the users which then accept these policies without understanding. The companies also hold the users responsible for any kind of misuse of data as the users had already agreed to the privacy policy which mentions about it. There are some applications which make it optional for the user to go through the policy, which leads to even fewer users referring to the policy. The enormous power of data can offer insights but can also cause obstacles in terms of the huge volumes. To facilitate the users to comprehend the policies, we present a policy summarizer which shows the various attributes that have been covered by the policy and the ones which they have omitted. The attributes include the various measures or the topics that the privacy policy should include based on directives, regulations and practices that an ideal policy should incorporate. After showing the topics the policy covers, the users also get the facility to view the passage which contains the particular topic in the policy in a summarized form. Hence this saves the user plenty of time by only reading the short summarized points present in the policy of the topics the user wishes to view. Users will more likely go through the policy if they are encountered with short summarized points related to the topics that do matter to them rather than reading huge blocks of information containing all the points in them without categorization. Reading the part of the policy which mentions about the attributes in one or two lines is much more beneficial than searching through the whole text. Users do not always go through the content of privacy policies while installing apps. The problem lies in the length of the content. To address the problems faced by the user there have been advancements in text mining that are considered to facilitate users of webpages, and apps to see summarized textual data (Izumi, Matsui, & Matsuo, 2007; J. Li, Fong, Zhuang, & Khoury, 2016). (Costante, Sun, Petković, & den Hartog, 2012) have proposed a method to resolve such issues by providing a completeness analyzer. They check the completeness and give a grade to privacy policies. This helps the users in understanding policies based on the grades given by the analyzer. They have classified each of the paragraphs present in a policy and checked if all classes are present and gives a completeness grade for the policies. Harkous et al. (2018) focused on creating a full fledged framework so users can easily use that framework to automate the entire process. They have not only focused on low level features but have also focused on high-level, more complex classes. Privacy policies also have options for users to opt-out of some policies and take control of their data. Efforts have been made to find these particular options present in a privacy policy (Sathyendra, Wilson, Schaub, Zimmeck, & Sadeh, 2017).

Privacy policies comprises a lot of text and a lot of attributes, thus summarizing it directly would miss out salient text. Thus to tackle that problem, classification of the text is needed based on the attributes it covers. Multi class text-classification is a classification problem in which text is classified in more than two classes (Cherfi, Napoli, & Toussaint, 2006). The existing solutions to the problem of multi-class classification is focused on using machine learning techniques like naive Bayes, support vector machines (SVM) (Rennie & Rifkin, 2001; Silva & Ribeiro, 2007). Another method is to counter the problem of multiple classification as n binary classification and solve these problems one by one and combining the result of these binary classifications. There have been exciting progress in utilizing using neural networks and deep learning based techniques for classifying text (Minaee et al., 2021; Satapathy, Li, Cavallari, & Cambria, 2019), document modeling (Majumder, Poria, Gelbukh, & Cambria, 2017) and for various natural language processing (NLP) tasks (Young, Hazarika, Poria, & Cambria, 2018). Among these techniques convolutional and recurrent neural networks based architectures for labeling multiple categories (Chen, Ye, Xing, Chen, & Cambria, 2017), generative models for synthetically generating text for training and classifications (Y. Li, Pan, Wang, Yang, & Cambria, 2018; Russell, Li, & Tian, 2019) are important. In text summarization, deep recurrent belief networks can be effectively used to model word dependencies (Chaturvedi, Ong, Tsang, Welsch, & Cambria, 2016). Recent improvements in deep learning involves the usage of capsule networks (Zhao, Peng, Eger, Cambria, & Yang, 2019) for challenging NLP applications such as the multi-label text classification and question answering, and long short-term (LSTM) model (Ma, Peng, Khan, Cambria, & Hussain, 2018) for aspect-based sentiment analysis.

Text summarization models have been considered before the deep learning era as well. For example, Nomoto and Matsumoto (2001) present a novel approach to unsupervised text summarization. The novelty lies in exploiting the diversity of concepts in text for summarization, which has not received much attention in the summarization literature. A diversity-based approach here is a principled generalization of maximal marginal relevance (MMR) criterion by Carbonell and Goldstein (1998). They have presented a new summarization scheme where evaluation does not rely on matching extracts against human made summaries but measuring the loss. There are many text summarization techniques presented using classical machine learning techniques like hidden Markov model (HMM), SVM, and Bayes model. Another way of text summarization can be done using lexical chains (Barzilay & Elhadad, 1999) formed with help of lexical cohesion. AutoEncoders are also being used for text summarization (Yousefi-Azar & Hamey, 2017) which used sentence ranking to extract sentences in their summary. For training the model on more specific kind of policies we can use a web-based information retrieval framework (Vijayarajan, Dinakaran, Tejaswin, & Lohani, 2016) to ingest peculiar policies.

In this work, we consider automatic text mining and machine learning base summarization of policies. Our policy categorizer and summarizer (POCASUM) allows us to test various plug-n-play machine learning classifiers. We test various classical machine learning models such as the K-nearest neighbors (KNN) (Abu Alfeilat et al., 2019), support vector classifier (SVC) (Suykens & Vandewalle, 1999), random forests (RF) (Breiman, 2001), stochastic gradient descent (SGD) (Kabir, Siddique, Kotwal, & Huda, 2015), along with deep learning artificial neural networks (ANNs). Experimental results on the APP-350 dataset (Zimmeck et al., 2019) indicate that ANN driven model obtains the highest accuracy among other KNN, SVC, SGD, RF based approaches.

The rest of the work is organized as follows. Section 2 introduces our machine learning driven approach to policy categorization and summarization. Section 3 provides experimental results with different machine learning models. Section 4 concludes the paper.

2. Policy Categorizer and Summarization with Machine Learning

2.1. Aim and Methodology

The goal of our proposed policy categorizer and summarizer (POCASUM) is to make it easy for users to comprehend the policies by reading short paragraphs and be clearly aware about how complete the policy is in terms of attributes that need to be mentioned in an ideal privacy policy. Hence, this includes two main tasks namely categorization and summarization. The policy paragraphs are classified under the proposed attributes so that a completeness score can be assigned to it. After checking its completeness the user might want to see the policy under a certain topic (say security). Users can then read the summarized text of the security attribute in about 2 to 3 lines which makes it tremendously easy for the user to comprehend and understand it properly.

The proposed flow of our POCASUM approach is given in Figure 1, and consists of the following steps.

Figure 1:

Overall flow of our proposed policy categorizer and summarizer (POCASUM) approach. Here we can utilize plug-n-play machine learning classifiers at the fitting models stage.

1. Data Annotation - The dataset has been created manually wherein each paragraph of every policy has been labelled according to the categories mentioned above.

2. Cleaning - Includes tokenization, stemming, lemmatization, removal of stop words

3. Feature Extraction - Extracting features from text using tf-idf vectorizer or word-2-vec.

4. Data Split - The reduced vectors are then split into testing and training data.

5. Dimensionality Reduction - Number of features are reduced using LSI which uses SVD for getting the number of features to a bare minimum.

6. Fitting Models - Training various machine learning models to find the best potential model.

7. Testing - Checking the accuracy by various testing parameters.

8. Sentence Embedding - Creating the sentences as vector that represents the sentence semantically and syntactically.

9. Clustering - Clustering of the embeddings in proximity gives us the most relevant sentences.

10. Extractive Summarization - Picking n most relevant lines and putting it up as a paragraph.

We note that the plug-and-play methodology of our POCASUM, at the fitting models stage one can utilize various machine learning models from classical models such as KNN, SVC, RF, SGD to deep learning ANN. In what follows, we benchmark various machine learning models and test their accuracy on text summarization in privacy policies.

2.2. Dataset

The dataset was manually created taking into consideration 60 privacy policies of applications from the APP-350 dataset (Zimmeck et al., 2019) (https://usableprivacy.org/data). The policies were divided into discrete paragraphs and each paragraph was labelled manually by the annotator. There are 5 attributes or categories that have been chosen to classify the paragraphs of the policy. The speculated important topics in a privacy policy that would affect the user’s have been taken as the attributes of the analyzer. These attributes are:

• Collection - The type of personal data that the company collects from the user

• Info Usage - How is the data being utilized by the company

• Location - How is the location information being used

• Share - Under what circumstances will the users data be disclosed or shared

• Contact - Do they provide any contact information about the company

All 60 policies were labelled manually under these categories by 3 annotators in-order to create the dataset. Each policy had been segmented into clusters of 2 or 3 sentences depending on the context and were then annotated within these attributes.

Algorithm 1: Text Preprocessing

$$\begin{array}{cc}\hfill & \mathbf{Input}:\text{Raw text input}{R}_t\hfill \\ \hfill & \mathrm{Output}:\text{Vectors}V\hfill \\ \hfill & \mathbf{foreach}\mathcal{S}\in R_t\mathbf{do}\hfill \\ \hfill & \lfloor \begin{array}{c}T=\text{Tokenize}(s)\hfill \\ \mathbf{foreach}t\in T\mathbf{do}\hfill \\ \lfloor \begin{array}{c}\mathbf{if}tinstop\_words\mathbf{then}\hfill \\ \mid \text{remove}t;\hfill \\ \mathbf{else}\hfill \\ \lfloor \text{continue};\hfill \\ \mathbf{if}checktwithREforcompoundwords\hfill \\ \mathbf{then}\hfill \\ \mid \text{continue};\hfill \\ \mathbf{else}\hfill \\ \lfloor \text{remove}t;\hfill \\ l=\text{Lemmatize}(t)\hfill \\ v=\text{tf-idf}(l)\hfill \end{array}\phantom{\rfloor }\hfill \\ \text{Add}v\text{to}V\hfill \end{array}\phantom{\rfloor }\hfill \end{array}$$

2.3. Text Preprocessing

Text preprocessing plays a vital role in the classification of the policy statements as it provides the attributes that are to be fed into the model (Uysal & Gunal, 2014). There is no perfect way of performing text preprocessing as it is subjective to the problem and the dataset available. In case of privacy policies it was important to include compound words and headings which may not be necessary for another problem. The step-by-step process has been stated in Algorithm 1. The raw text was initially converted into tokens and the stop words were removed. The stop word removal is the process of removal of very frequently occurring words that do not contribute to the prominence of the text and was performed by a list of stop words in the english language. Then certain regular expression patterns were compiled to include certain kinds of words like email or compound words which might contribute to the prediction process.

The tokens were further normalized using a Lemmatizer and then were marshalled into a list of tokens which created our cleaned tokens. These tokens were then converted into a tf-idf vector (Fautsch & Savoy, 2010) instead of Count Vectorizer as it suppresses the importance of the word occurrence and takes an egalitarian weightage of the words. This allows us to take into consideration those words that are important to the topic rather than those that appear very frequently like “an” or “the”. Figure 2 shows the flowchart of the text preprocessing steps. Due to the large number of features the vectors were further reduced by applying dimensionality reduction using latent semantic indexing (LSI) approach. Again the number of dimensions to be reduced is based on the dataset at hand, and the optimal number of dimensions can only be found through experimentation.

Figure 2:

Text preprocessing steps employed here for our POCASUM approach.

2.4. Modeling

After the text has been cleaned and been converted to vectors they are then taken for the model training. Constructing a multi class text classifier involves assigning the class label to the particular vector of text that represents the features of the raw text. Supervised learning techniques were used for this purpose as the dataset had been annotated and gives it a better chance to improve the model’s accuracy. We have used machine learning (Kotthoff, Gent, & Miguel, 2011) and deep learning techniques in-order to get the best accuracy on multi label classification. Under classical machine learning models we trained four different models namely K-nearest neighbors (KNN), support vector classifier (SVC), stochastic gradient descent (SGD) and random forest (RF) classifier.

The KNN (Abu Alfeilat et al., 2019; Yang & Liu, 1999) is the go-to algorithm for ubiquitous clustering problems due to it’s simple approach without a mathematical model and is very effective in many cases. The number of clusters is an important parameter to be set which is found out by empirical method. The elbow method is used to find out the ideal number of neighbors but in case of supervised learning, the number of clusters are already been provided in the dataset. The SVC (Burges, 1998) is a novel technique and has proved to be immensely accurate in classification tasks due to the less number of parameters to tune. The SGD (Kabir et al., 2015) is the most commonly used linear model and due to good reasons due to it is a good fit to the data. It aims at reducing the error by choosing the number of parameters to be changed at a given descent or step. The ensemble random forest (RF) (Breiman, 2001) classifier was also used due to the multitude of decision trees giving a holistic prediction rather than a specific one leading to more chances of a higher accuracy and better prediction. We also tried out different architectures of deep artificial neural network (ANN) (Ghiassi, Olschimke, Moon, & Arnaudo, 2012) for the multi class classification problems due to its success in the past. The one drawback of ANN is that there are a lot of things to decide on before the model can start training. The first and foremost is the architecture of the network which is again to be decided by empirical process. After building the architecture other parameters like batch size, number of epochs, cross validation, metrics, loss function and a lot others are to be tuned as per your requirement which is again optimally found by experimentation.

2.5. Testing

After the models have been trained on the training data, we need to test the model by giving it data that the model has never seen before and validate the results for those. There are a ton of evaluation metrics available out there. For an efficient classifier the precision and recall are two ubiquitous evaluation metrics that one relies on. Apart from these two metrics, the F1 score and classification accuracy will also be used to judge the model. F1 Score is a measure concocted by precision and recall, however one might want to look at the specific metrics according to the type of problem.

K-fold cross validation (Fushiki, 2011) was also used for increasing the robustness of the model wherein the data is divided into k equal parts, and each part is given one iteration where k-1 parts are utilized for training and the remaining part is utilized for testing. We have used a k value as 5 for training the model. All experiments were performed in the same conditions. K-fold cross validation improves the adaptability of the model to a great extent by increasing the variance in the test data provided to the model.

2.6. Sentence Embedding

After the data has been correctly classified by the model, we need to combine the texts of all attributes in order to summarize the text of the topics. The raw combined topic wise text is to be summarized so that it can be very easy for the users to comprehend the meaning of it. Just as we represent each word as a vector in a tf-idf or any other vectorizer, in order to carry out extractive summarization each sentence needs to be represented as a vector (Ahamad, 2019). That vector should be made by taking the syntactic, semantic and all the important properties of the sentence it represents. Sentence embedding are those vectors that represent the sentence taking its various properties so that we can get to know the proximity between sentences. Hence sentences that share similar semantic and syntactic properties lead to similar vector representation (Barzilay & Elhadad, 1999).

The sentence embedding was facilitated using skip-thought vectors which uses an encoder-decoder model for encoding sentences into vectors. The encode maps the words to a sentence vector and the decoder generates the sentences surrounding the current sentence. ConvNet and RNN were specifically used to create encoder and decoder models which generate sentence embeddings. Now these sentences can also be represented on a graph as a vector which will enable us to see the characteristics or the surroundings of the vector. Since we only require the sentence embeddings and not the surrounding sentences we will only utilize the encoder part of the model as explained using Algorithm 2.

Algorithm 2: Sentence Embedding

$$\begin{array}{cc}\hfill & \mathbf{Input}:\text{Words}\{w_i:W\in s\},\text{word}\hfill \\ \hfill & \text{embedding}(x_i{)}^t\text{denote its word}\hfill \\ \hfill & \text{embeddings,a set of sentences S}\hfill \\ \hfill & \mathrm{Output}:\text{Sentence Embeddings}\{v_s:s\in \mathcal{S}\}\hfill \\ \hfill & \mathbf{foreach}s\in \mathcal{S}\mathbf{do}\hfill \\ \hfill & \lfloor \begin{array}{c}\mathbf{foreach}W\in s\mathbf{do}\hfill \\ \lfloor \begin{array}{c}{r}^t=\sigma (W_r{x}^t+U_r{h}^{t-1})\hfill \\ z^t=\sigma (W_z{x}^t+U_z{h}^{t-1}\hfill \\ {\overline{h}}^t=\tanh (W{x}^t+U(r^t\odot h^{t-1}))\hfill \\ h^t=(1-z)\odot h^{t-1}+z^t\odot {\overline{h}}^t\hfill \\ \hfill \end{array}\phantom{\rfloor }\hfill \\ \hfill \end{array}\phantom{\rfloor }\hfill \\ \hfill & Thehiddenstate(h_i{)}^{t}thusrepresentsthefull\hfill \\ \hfill & encodedsentenceembedding\hfill \end{array}$$

2.7. Clustering

Once the sentences have been converted into vectors, now we need to take n most relevant sentences from the sample space by using an unsupervised clustering algorithm (Bennani-Smires, Musat, Hossmann, Baeriswyl, & Jaggi, 2018; Valdivia et al., 2020). These embeddings are clustered in highdimensional vector space into a pre-defined number of clusters according to the number of sentences required in the summary of the text. K-nearest neighbors (KNN) (Abu Alfeilat et al., 2019) was used to cluster the vectors and the top 3 vectors were chosen. The measure used to cluster was euclidean distance however many other metrics are also present. These vectors were then mapped to their text and their corresponding sentences are extracted and displayed as a paragraph. This gives us the extractive summary of the particular attribute the user wishes to read.

3. Experimental Results

We tested different machine learning classifiers in our POCASUM approach, namely, the K-nearest neighbors classifier (KNN), support vector classifier (SVC), stochastic gradient descent (SGD), random forest (RF) and artificial neural network (ANN) models. There have been a plethora of empirical methods that have been used throughout the process. Starting from what kind of words to keep, trying out various cleaning techniques, number of dimensions, and the features that must be reduced to all experiments, have been tried out to get the best possible results. Different dimensions were for each classification model as shown in Figure 3. Even the kind of machine learning model to choose, the parameters on which the model is being tuned to and the architecture of the neural network are all subjective things that do not provide ideal answers for the input data. These choices vary from problem to problem and carrying out empirical methods is the only way to get an idealized value. As can be seen, the performance of KNN, RF, and SGD are lower, in general than the SVC and ANN.

Figure 3:

Accuracy of different machine learning classifiers with respect to the number of dimensions. Here, we tested K-nearest neighbors (KNN), support vector classifier (SVC), stochastic gradient descent (SGD) and random forest (RF), and artificial neural network (ANN) classifiers.

In terms of the performance of various machine learning classifiers, our experimental testing was done with classical - KNN, SVC, RF, SGD, as well as various architectures of ANN. Among the obtained F1 scores of KNN (68%), SVC (70%), SGD (71%) and RF (67%), SGD proved out to be the best machine learning model in our sample space with an F1 score of 71%. The deep learning ANN however performed very well with an F1 score of 78% and an accuracy score of 75.62 thus outperforming the rest of the models. Figure 4 shows the performance of various models with F1 score and accuracy. To assess the summarizer we took a random policy on which the model had not been trained on. The policy was then segmented into chunks of 2 to 3 lines and were fed into the ANN model for classification as shown in the input text Figure 5. For testing purposes we took up the first category which was the type of information the particular company collected. All the sentences from the policy that belong to this class were combined and appended. Then these sentences were then fed into the summarizer to check if the results semantically made sense. The results shown in Figure 5 turned out to be comprehensible which covered the main points of the text paragraph and hence making it convenient for the user.

Figure 4:

Comparison of different classical (KNN, SVC, RF, SGD), and deep ANN machine learning models in terms of (a) F1 score, and (b) accuracy.

Figure 5:

Examples of POCASUM with ANN result from the APP-350 data set. (a) Input, and (b) output from the system.

We have tried and tested different architectures for the ANN model for the classification task, and ANN with architecture (64+32+6) as in Table 1 proves out to be the superior of them all which obtained 78% F1 score. The performance of this best ANN model in terms of training and validation is shown in Figure 6. Out of all available machine learning models SGD turned out to be the accurate classifier among classical machine learning models as shown in Table 2. As can be seen, the SGD model obtained reduced F1 score as the number of classes increased. Overall, the ANN model within POCSAUM obtained the best performance indicating promise in obtaining comprehensible text summarization. However, our tested ANN architecture as shown in Table 1 are limited to 3 or 4 layers and unlike typical ‘deeper’ architectures considered in other works in text mining and visual computing. The reason for this is the limitation of the dataset we considered here, deeper layers tend to overfit and designing an optimal ANN remains a challenge.

Table 1:

Different architectures of ANN tested in our POCASUM approach for text summarization.

Architecture	F1 Score	Precision
64+32+6	0.78	76%
64+32+32+6	0.73	72.4%
32+32+6	0.72	72.35%
128+64+32+6	0.69	70.1%
128+32+6	0.70	70.9%

Figure 6:

Performance of the best ANN model (64+32+6 architecture) with epoch steps in terms of training and validation (a) loss, and (b) accuracy.

Table 2:

Classification with stochastic gradient descent (SGD) for different classes.

Classes	F1 score	Precision	Recall	Support
0	0.79	0.78	0.814	340
1	0.44	0.41	0.483	60
2	0.39	0.41	0.37	58
3	0	0	0	4
4	0.52	0.58	0.47	44
5	0.52	0.65	0.43	30

In summary, in this work we have provided a framework to analyze the authenticity of a privacy policy by carrying out multi-class classification and summarization techniques. Further, we have combined the text that belongs to a particular class or attribute so that the paragraphs can be summarized and only the necessary information is presented to the user which makes it easy for them to comprehend the policy. Despite the positive results obtained with ANN combined POCASUM, however there is still improvements that can be done to make this work more convenient for users, (i) the tested architectures represent our experimental heuristics and model reduction within the ANN context remains to be tested further, and (ii) an expanded dataset that includes other textual policies are required to study the feasibility of machine learning driven POCASUM approach. Moreover, semantic analysis of sentences (Nguyen, Duong, & Cambria, 2019; Tur, Deng, Hakkani-Tür, & He, 2012) is one area which we might be lacking in NLP based text summarization considered here. Due to the changing dynamics of the language and several ways to write the same sentence it is very difficult to make the algorithm understand the meaning behind the sentence. If semantic analysis is facilitated within our POCASUM approach, it will make it very convenient for users as we can directly tell them if a certain topic abides by the rules mentioned in an ideal privacy policy. We can directly assign a score which will tell them how safe the policy is, and the users will not even have to see the content of the policy and will get to know how safe the policy is automatically.

4. Conclusion

In this work, we used text mining and machine learning models for policy notices categorization and summarization to aid users with relevant information in a succinct way. Our policy categorizer and summarizer (POCASUM) approach can aid the users of hypermedia to be more careful about their rights and data and makes it easy for the users to understand the digital media company’s motives. We benchmarked different plug-n-play machine learning classifiers including KNN, SVM, RF, SGD, and deep ANN models. Our experimental on a dataset of policies indicate that ANN based model performs well with good accuracy. Our initial experiments indicate that we can reduce the time taken to analyze a policy from 15-20 minutes down per user to a few seconds. This indicates that deeper networks can be used in our POCASUM approach thereby improving the summarization capabilities, however this requires a bigger dataset for training the learning model. Further, benchmarking this reduction of reading time requires a deeper user-based feedback study. Another area of improvement can be where sentence embedding is executed. Apart from skip thought vectors there are many other embedding techniques that can be used to better represent the sentences as vectors like skip gram vectors and quick thought vectors. Quick thought vectors (Russell et al., 2019) is a recent development in skip thought vectors in which the task of forecasting the next sentence given the previous one is handled as a classification problem.