¡Hola, Mundo!

LSALatent Semantic Analysis, substitute for DTM, TF-IDF(2021.03.10 - [Deep Learning] - BoW, CountVectorizer, fit_transform, vocabulary_, DTM, TDM, TF-IDF, TfidfVectorizer, isnull, fillna, pd.Series) which has not consider meaning of terms. It applies SVD based on DTM, TF-IDF and reduce dimensions, eliciting potential meaning of words.1. SVD Singular Value Decomposition, it refers to the decompos..

Bag of words(BoW) It is a way of extracting features from text, a representation of text that describes the occurrence of words within a document, for example, pour the sentence into the bag and shuffling the bag, any information about the order or structure of words in the bag is discarded. Review 1 : This movie is very scary and long Review 2 : This movie is not scary and is slow Review 3 : Th..

LM Language Model, is a model assigns probability to sequence in order to modeling language, in other words, finding the most natural word sequence Language Modeling Prediction to unknown word from given words. Conditional Probability It is the probability of an event occurring given that another event has already occurred. In this theory, mutually exclusive events are events that cannot occur s..

word_index Grand index to each words. print(tokenizer.word_index) >>> {'barber': 1, 'secret': 2, 'huge': 3, 'kept': 4, 'person': 5, 'word': 6, 'keeping': 7, 'good': 8, 'knew': 9, 'driving': 10, 'crazy': 11, 'went': 12, 'mountain': 13} word_counts Count the each words. print(tokenizer.word_counts) >>> OrderedDict([('barber', 8), ('person', 3), ('good', 1), ('huge', 5), ('knew', 1), ('secret', 6),..

normalization Integrate different words to make them the same word-such as US is same as USA integrate them as US. 1. WordNetLemmatizer If words have different forms, find the root word-such as the root of 'am, are, is' is 'be'. from nltk.stem import WordNetLemmatizer lemmatizer=WordNetLemmatizer() words=[ 'have', 'going', 'loves', 'lives', 'flies', 'dies', 'watched', 'has', 'starting'] print('b..