Author(s): SHARON ZACHARIA Originally published on Towards AI. Can machines understand human language? What if we could interact with them the same way we do with other humans? These questions are addressed by the field of Natural Language processing, which allows machines to mimic human comprehension and usage of natural language. Early foundations of NLP were established by statistical and rule-based models like the Bag of Words (BoW). Even while traditional methods are straightforward, they are nevertheless widely used today and are often contrasted with more sophisticated models based on Transformers. In this article, we will discuss what BoW is and how Transformers revolutionized the field of NLP over time. We will then implement BoW and a Transformer based text summarizer model using T5 model. Extractive Summarization extracts key sentences, phrases, or sections directly from the original text to create a summary. It doesn’t generate new sentences but identifies the most important pieces of information based on factors like sentence importance, relevance, and position in the text. Abstractive Summarization: generates a summary by understanding the original content and then rephrasing or rewriting it in a more concise form. The model generates new sentences that may not appear in the original text but capture the main ideas. It is typically more flexible and capable of producing more human-like summaries Bag of Words BoW is one of the foundational models in NLP for text representation. It represents text in numerical format, considering each document as a collection of words. BoW mainly focuses on word frequency, ignoring grammar and word order. It is one of the widely used technique in NLP despite its simplicity. Bag of Words Representation mainly involves a vocabulary of words and Measure of presence of each word. Consider two sentences as follows: Sentence 1 : “I Love Cats” Sentence 2 : “I Love Dogs” After tokenisation process the sentence is split into individual words(tokens) this will be as follows. Sentence 1 : [”I”, ”Love”, ”Cats”] Sentence 2 : [”I”, ”Love”, ”Dogs”] The resulting vocabulary would be as follows: [”I” , ”Love” , ”Cats” , ”Dogs”] The resulting vectors will be as follows : Sentence 1 : [1, 1, 1, 0]. ”I” , ”Love” , ”Cats” appears once (1) and ”Dogs”is not present (0) in the first sentence. Sentence 2 : [1, 1, 0 ,1]. ”I” , ”Love”, ”Dogs” appears once (1) and ”Cats” is not present (0) in the Second sentence. Similarity Matrix A similarity matrix is a mathematical representation of how sentences are similar to each other. Here the similarity of sentences are calculated using cosine similarity. Cosine similarity measures the cosine of the angle between two vectors. Transformer Architecture The transformer architecture was introduced in the paper Attention is all you need (Vaswani et al. 2017) which revolutionized the field of Natural Language Processing and Machine Learning by addressing the limitations of CNNs (Convolutional Neural Networks)and RNNs (Recurrent Neural Networks). It uses a self-attention mechanism to analyse sequential input, allowing for parallel computation and reaching cutting-edge results in tasks like question-answering, summarization, and text translation. Transformer Architecture (Vaswani et al. 2017) The Encoder is the fundamental component of the Transformer architecture, which transforms input tokens into contextualized representations instead of processing tokens independently. Encoder captures the context of each token in the entire sequence. Using embedding layers, the encoder first transforms input tokens into vectors. The tokens’ semantic meaning is captured by these embeddings, which then translate them into numerical vectors. The Decoder is in charge of producing the output sequence one step at a time using the encoded input sequence and its own outputs that have already been produced. A masked multi-head self-attention mechanism and a multi-head cross-attention mechanism are the two primary sublayers that make up each of its several layers. A feedforward neural network comes next. The self-attention method enables the decoder to predict the next token by taking into account just the previously created tokens. The decoder can concentrate on pertinent segments of the input sequence by paying attention to the encoder’s output representations with the help of cross-attention method. Self Attention Mechanism in Transformer architecture creates three representation of the tokens in order to weight the importance of each word in the sequence relative to one another. The three representations include Queries (Q) , Keys (K) and Values (V) which are obtained by multiplying the input embeddings with learned weight matrices. To find how much attention each token in the sequence should pay to the others, the mechanism calculates a similarity score between its query vector and the key vectors of all tokens. These similarity scores are scaled by the square root of the key vectors’ dimensions. A softmax function is then used to normalize the scores, creating attention weights. After applying the weights to the value vectors, a weighted total is created that considers each token’s context in respect to the others. Since we have discussed important concepts both in BoW and Transformers, let’s try implementing these models. As discussed earlier, we will be implementing these two models to summarize textual data. The data set used is cnn/DailyMail which is a widely used dataset for NLP tasks. The dataset can be found here Bag of Words We will be using the below-mentioned libraries to implement the model. import nltkimport re from nltk.corpus import stopwordsfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.metrics.pairwise import cosine_similarity We will begin by implementing text preprocessing steps which helps to convert the entire text to lowercase, ensuring consistency and avoiding case-sensitive issues. Non-alphanumeric characters are removed using regular expressions, leaving only relevant words and spaces. The function then splits the text into tokens and filters out common stop words, using a predefined list of English stop words. # Download NLTK stopwordsnltk.download("stopwords")nltk.download('punkt_tab')stop_words = set(stopwords.words("english")def preprocess_text(text): text = text.lower() text = re.sub(r'\W', ' ', text) # Removing stopwords tokens = text.split() tokens = [word for word in tokens if word not in stop_words] return ' '.join(tokens) Now we will define a function that takes text as input and gives a summary […]
↧