Related questions

How to find overall model coefficients in Gensim's latent dirichlet allocation?

2018-07-11 03:33 Jane Sully imported from Stackoverflow

I am using Latent Dirichlet Allocation (LDA) to find topics that occur in my corpus, and I was wondering if there is a way to get the overall model coefficients in Gensim? Here each coefficient would represent the weight of each topic in the overall model. I want to get an idea of which topics are the most important relevant through these weights, in addition to looking at topic vocabularies. Something like this is possible in R, since the model has a coefs attribute ( a coefficient for each of the six topics):

lda_model$coefs

[1] -0.56446635 -0.52604353 -0.43325116 -1.88352347  0.20560172 -0.26902856

One idea I have currently is to use Gensim's get_document_topics(bow) function where bow represents a single document converted into a bow vector. Here is the documentation: https://radimrehurek.com/gensim/models/ldamodel.html. Although this is not the same as the model coefficients I was talking about above, it may capture a somewhat similar idea. I can use this function for all documents to get the document's topic distribution, and average all the topic distributions across documents for each topic to get a sense of which topics are the most prevalent. However, this would not be my first go-to option. Please let me know if there is something along the lines of coefs in Gensim's LDA to obtain model coefficients. Thanks!

See also questions close to this topic

  • how to add non .py files into python egg

    I have a flask app which looks like

    my-app 
│   └── src
│       └── python
│             └── config
│             └── app
│── MANIFEST.in
└── setup.py

    The config folder is full of *.yaml files, I want to add all the static config files into my python egg after using 

    python setup.py install

    My setup.py looks like

    import os
from setuptools import setup, find_packages
path = os.path.dirname(os.path.abspath(__file__))
setup(
    name="app",
    version="1.0.0",
    author="Anna",
    description="",
    keywords=[],
    packages=find_packages(path + '/src/python'),
    package_dir={'': path + '/src/python'},
    include_package_data=True
)

    I am trying the use the MANIFEST.in to add the config file However, it always give error

    error: Error: setup script specifies an absolute path:

/Users/Anna/Desktop/my-app/src/python/app

setup() arguments must *always* be /-separated paths relative to the
setup.py directory, *never* absolute paths.

    I have not used any absolute paths in my code, I've seen other posts trying to bypass this error, by removing

    include_package_data=True

    However, in my case, if i do this to avoid this error, all my yamls won't be added.

    I was wondering if there are ways to fix this problem. Thanks

  • How to extract all functions and API calls used in a Python source code?

    Let us consider the following Python source code;

    def package_data(pkg, roots):
    data = []
    for root in roots:
        for dirname, _, files in os.walk(os.path.join(pkg, root)):
            for fname in files:
                data.append(os.path.relpath(os.path.join(dirname, fname), pkg))

    return {pkg: data}

    From this source code, I want to extract all the functions and API calls. I found a similar question and solution. I ran the solution given here and it generates the output [os.walk, data.append]. But I am looking for the following output [os.walk, os.path.join, data.append, os.path.relpath, os.path.join].

    What I understood after analyzing the following solution code, this can visit the every node before the first bracket and drop rest of the things.

    import ast

class CallCollector(ast.NodeVisitor):
    def __init__(self):
        self.calls = []
        self.current = None

    def visit_Call(self, node):
        # new call, trace the function expression
        self.current = ''
        self.visit(node.func)
        self.calls.append(self.current)
        self.current = None

    def generic_visit(self, node):
        if self.current is not None:
            print("warning: {} node in function expression not supported".format(
                  node.__class__.__name__))
        super(CallCollector, self).generic_visit(node)

    # record the func expression 
    def visit_Name(self, node):
        if self.current is None:
            return
        self.current += node.id

    def visit_Attribute(self, node):
        if self.current is None:
            self.generic_visit(node)
        self.visit(node.value)  
        self.current += '.' + node.attr

tree = ast.parse(yoursource)
cc = CallCollector()
cc.visit(tree)
print(cc.calls)

    Can anyone please help me to modified this code so that this code can traverse the API calls inside the bracket?

    N.B: This can be done using regex in python. But it requires a lot of manual labors to find out the appropriate API calls. So, I am looking something with help of Abstract Syntax Tree.

  • use correct version of 'pip' installed for your Python interpreter

    I am using pycharm , i am having this error by adding any package , Click Here

    i have tried lot of methods , but didn't succeed yet.

    Info :

    1. python 3.6.2
    2. pip 10.0.1
    3. VirtualEnv
  • How to install gensim without pip (firewall issues)

    All-

    I would like to use the gensim library, but unfortunately I can't install it via pip due to the company's firewall. Any advice? Thank you in advance for any help or suggestions you can provide.

  • Computing BM25 document representation Gensim

    I want to use a slightly modified version of the BM25 as input to a classifier on text input. Using gensim's BM25 class, what's the most efficient way to compute this vector representation for a large corpus?

    The trained BM25 object stores word frequency by document, word frequency in the corpus, average document length and all doc lengths. Its methods seem to only return the similarity values - whereas if B(string, corpus) is the BM25 representation, and C(string, corpus), I want B(string, corpus) + C(string, corpus) as input to my classifier.

    The fields of the BM25 object are kept as dicts, what would be the most efficient way to compute the vector representation of all texts in my corpus to feed in a sklearn classifier, e.g. kernel SVM?

  • Word embedding decreasing classification precision

    I am currently trying to classify text into 7 classes. Up to now, I have been able to reach a 90% precision score using Majority Voting (with SVM, Multinomial NB, Random Forest and KNN).

    I wanted to try to increase a little more this precision by using word embeddings and thus getting less dimensions for my samples. I use gensim word2vec to create my model and the NLTK list of stop-words and tokenizer:

    with open('data.pkl','r') as f:
    corpus=pickle.load(f)

with open('targets.pkl','r') as f:
    targets=pickle.load(f)

tokenized_corpus=[word_tokenize(anomaly) for anomaly in corpus]
stop_words=['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]
for i,anomaly in enumerate(tokenized_corpus):
    for w in anomaly:
        if w in stop_words:
            tokenized_corpus[i].remove(w)

model = gensim.models.Word2Vec(
        tokenized_corpus,
        window=5,
        size=100)

model.train(tokenized_corpus, total_examples=len(corpus), epochs=10)

    The model seems fine, I get satisfying results when I use similarity between words.

    I use this class to get a mean representation of my samples:

    class MeanEmbeddingVectorizer(object):
    def __init__(self, word2vec):
        self.word2vec = word2vec
        if len(word2vec)>0:
            self.dim=len(word2vec[next(iter(word2vec))])
        else:
            self.dim=0

    def fit(self, X, y):
        return self 

    def transform(self, X):
        return np.array([
            np.mean([self.word2vec[w] for w in words if w in self.word2vec] 
                    or [np.zeros(self.dim)], axis=0) #moyenne des vecteurs des mots (a.word2vec[w])[ou 0 si il connait pas le mot] composant un élément de X
            for words in X
        ])

    and 

    w2v = dict(zip(model.wv.index2word, model.wv.syn0))
a=MeanEmbeddingVectorizer(w2v)    
X_transformed=a.transform(tokenized_corpus)

    Finally, I build a common sklearn pipeline and let sklearn perform a GridSearchCV on my data:

    test_param_n_estimators=[i for i in range(1,50)]
parameters = {'clf2__n_estimators': test_param_n_estimators}

pip=Pipeline([['clf2',RandomForestClassifier()]])

gs_clf2 = GridSearchCV(pip, parameters,verbose=10,n_jobs=2)
gs_clf2 = gs_clf2.fit(X_transformed, targets)

print(gs_clf2.best_score_)
print(gs_clf2.best_params_)

    The issue is that I get random precision scores (around 0.5).

    Maybe dimensionality reduction is not always able to increase precision but I think I don't understand something or I did something wrong, do you have an idea of what is going wrong ?

    Thank you in advance

  • TopicModel: How to query documents by topic model "topic"?

    Below I created a full reproducible example to compute the topic model for a given DataFrame.

    import numpy as np  
import pandas as pd

data = pd.DataFrame({'Body': ['Here goes one example sentence that is generic',
                  'My car drives really fast and I have no brakes',
                  'Your car is slow and needs no brakes', 
                  'Your and my vehicle are both not as fast as the airplane']})

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(lowercase = True, analyzer = 'word')

data_vectorized = vectorizer.fit_transform(data.Body)
lda_model = LatentDirichletAllocation(n_components=4, 
                                      learning_method='online', 
                                      random_state=0,
                                      verbose=1)
lda_topic_matrix = lda_model.fit_transform(data_vectorized)

    Question: How is it possible to filter documents by topic? If so, can documents have multiple topic tags, or is a threshold needed?

    In the end, I like to tag every document with "1" depending on whether it has a high loading of topic 2 and topic 3, else "0".

  • gensim lda permission denied when I try to save my model

    I've been getting some irregular behavior from an LDA topic model program and right now, it seems like my file won't save the lda model it creates... I'm really not sure why.

    Here's a code snippet, albeit it's going to take me more time before I could write code that's reproducible since I'm really just trying to load certain files I created beforehand.

    def naive_LDA_implementation(name_of_lda, create_dict=False, remove_low_freq=False):

    LDA_MODEL_PATH = "lda_dir/" + str(name_of_lda) +"/model_dir/" # for some reason this location doesn't work entirely... and yes, I have made a directory in a the folder of this name.
    # This ends up saving the .state, .id2word, and .expEblogbeta.npy files... But normally when saving an lda model actually works, a fourth file is included that's to my understanding the model itself.
    # LDA_MODEL_PATH = "models/" # This is what I originally had as the location for LDA_MODEL_PATH. I was using a directory called models for multiple lda models. This no longer works.

    doc_df = getCorpus(name_of_lda, cleaned=True) # returns a dataframe containing a row for each text record and an extra column that contains the tokenized version of the text's post/string of words.
    dict_path = "lda_dir/" + str(name_of_lda) + "/dict_of_tokens.dict"
    docs_of_tokens = convert_cleaned_tokens_entries(doc_df['cleaned_tokens'])
    if create_dict != False:
        doc_dict = corpora.Dictionary(docs_of_tokens) :
        if remove_low_freq==True:
            doc_dict.filter_extremes(no_below=5, no_above=0.6)
        doc_dict.save(dict_path)
        print("Finished saving") 
    else:
        doc_dict = corpora.Dictionary.load(dict_path)
doc_term_matrix = [doc_dict.doc2bow(doc) for doc in docs_of_tokens] # gives a unique id for each word in corpus_arr

Lda = gensim.models.ldamodel.LdaModel
ldamodel = Lda(doc_term_matrix, num_topics=15, id2word = doc_dict, passes=20, chunksize=10000)
ldamodel.save(LDA_MODEL_PATH)

    To put it sraightforwardly... I have no clue why permission is being denied when I try to save my lda model to a particular location. Right now even the original models/ directory location is giving me "permission denied" with this error message. It's seeming like any and all directories I could use just... won't work. This is odd behavior and I can't really find asks that talk about this error in the same context. I have found posts of people getting this error message when they actually tried storing in locations that did not exist. But for me that isn't really a question.

    When I first got this error... I actually started to wonder if it was because I had another lda topic model that I named topic_model_1. It was stored in the models/ subdirectory. I started to wonder if the name was a potential cause, and changed it to lda_model_topic_1 to see if that could change results... but nothing is working.

    Even if you can't really figure out what solution applies to my situation (especially since right now I don't have reproducible code, I just have my work)... Can someone tell me what this error message means? When and why does it come up? Maybe that's a start.

          Traceback (most recent call last):
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\utils.py", line 679,
in save
    _pickle.dump(self, fname_or_handle, protocol=pickle_protocol)
TypeError: file must have a 'write' attribute

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "text_mining.py", line 461, in <module>
    main()
  File "text_mining.py", line 453, in main
    naive_LDA_implementation(name_of_lda="lda_model_topic_1", create_dict=True,
remove_low_freq=True)
  File "text_mining.py", line 411, in naive_LDA_implementation
    ldamodel.save(LDA_MODEL_PATH)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\models\ldamodel.py",
line 1583, in save
    super(LdaModel, self).save(fname, ignore=ignore, separately=separately, *arg
s, **kwargs)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\utils.py", line 682,
in save
    self._smart_save(fname_or_handle, separately, sep_limit, ignore, pickle_prot
ocol=pickle_protocol)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\utils.py", line 538,
in _smart_save
    pickle(self, fname, protocol=pickle_protocol)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\utils.py", line 1337,
 in pickle
    with smart_open(fname, 'wb') as fout:  # 'b' for binary, needed on Windows
  File "C:\Users\biney\Miniconda3\lib\site-packages\smart_open\smart_open_lib.py
", line 181, in smart_open
    fobj = _shortcut_open(uri, mode, **kw)
  File "C:\Users\biney\Miniconda3\lib\site-packages\smart_open\smart_open_lib.py
", line 287, in _shortcut_open
    return io.open(parsed_uri.uri_path, mode, **open_kwargs)
PermissionError: [Errno 13] Permission denied: 'lda_dir/lda_model_topic_1/model_
dir/'
  • Advice needed: Best ML architecture to label emails as 0) events or 1) not events?

    I was looking for some ML/NLP advice. I have 50,000 newsletters (emails) that are labelled either “event” or “not event”. Here is an example of each:

    Not event: 

    COMPLIMENTARY STANDARD SHIPPING ON ALL DOMESTIC ORDERS. 

ENDS SOON 
TIMELESS SKIN SERUM DUO
http://enews.ernolaszlo.com/q/bmhc31K1AEaZwmn8quhjEVD2Gn6bhKECHXID2Z_-oGFLUexGdsY3pzTtf

Protect your youthful glow; stock up now.
http://enews.ernolaszlo.com/q/Y16cHUQU_1l761il6e1xyZBSB5Kl1GNI9uyRdaZhOJoy7obGhj-Hm_eoW

Transphuse Rapid Renewal Cell Protocol 
This month only, get our full-size, revolutionary age-reversing serum free when you spend $400
http://enews.ernolaszlo.com/q/W5UbQ8S85EcX85jCVa01TKcQIR9f0F27SnHlrel6xmpasf7GYw-gCjpHQ

    Event:

    First Street Gallery Logo [http://r20.rs6.net/tn.jsp?f=001HzXGe-7Pq8- 
zlt4lLYuOB6gUgJEJfvwghspnK22oQib-   APXPI0NJvw4FOyJ- 




OPENING RECEPTION THIS THURSDAY

JULY 23, 6-8 PM
2015
MFA NATIONAL EXHIBITION

Thomas Wharton, Martini, digital print on canvas, 9 x 8 inches

JULY 23 - AUGUST 14, 2015

First Street Gallery proudly presents our fifth annual MFA NATIONAL 
EXHIBITION.

This exhibition highlights the vast reach of fine arts teaching throughout the United
States. The competition is open to all current and former MFA graduates within the
past three years. Our Juror, David A. Ross, former Director of The Whitney Museum
of American Art and The San Francisco Museum of Modern Art, has a 40-year career
 as a museum professional and educator. He is currently the Chair of the MFA Art
 Practice Program at SVA.

ARTISTS: Danny Baskin (UARK), Kimberly Becker (Heartwood), William 
Chambers (Mass
Art), Donna Cleary (SVA), Sarah Dahlinger (Ohio), Jason Egitto 
(Syracuse), Lindsey
Elsey (Clemson), Dan Fenstermacher (SJSU), En Iwamura (Clemson), 
Richard James (KU),
Annie Johnston (UT-Austin), June Korea (SVA), James Lambert (Mass 
Art), Junko Ledneva
(UAF), J. Myszka Lewis (UW-Madison), Wilson Parry (Parsons), Veronica 
Perez (MECA),
Danette Pratt (Ohio), Jason Schwab (CCAD), Thomas Wharton (UT-Knoxville).

For more info please visit the 2015 MFA NATIONAL EXHIBITION album on our 
Facebook
page. [http://r20.rs6.net/tn.jsp?f=001HzXGe-7Pq8- 
zlt4lLYuOB6gUgJEJfvwghspnK22oQib- 
APXPI0NJv4vTi_KIZjl5AqU4Nfz6z3iB6MWBRsf5QHv- 
4T8IUSmtSVEItfj8TEaoluebDWPmWv8D8ayfdc-wgMtgkfoRJvH4e6- 
s0HG2jeuCsdSgT2q8uK4gve2K-u9S0X-AI-dtxpp72dGVY3orPDp-aCm- 
Gm8b_bp2B_Kh4JXLtEQn1qL6y- 



 T3w2wKAO_Ijb4WKJM54g==&c=ThoQa9dOxQlD2qj4Bk8tMgN
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Summer Gallery Hours: 11 am - 6 pm, Monday through Friday

526 West 26th Street, Suite 209, New York, New York 10001

646-336-8053 · 646-336-8054 (fax)

First Street Gallery is located in the heart of Chelsea, NYC between 
10th & 11th
 Avenues.

 [Nearest Subways: C,E,R,1,F,V to 23rd St. - crosstown bus to 10th Ave.

Nearest Buses:  9th Avenue (#11), 8th Avenue (#10)

FSG Gallery Location (Larger) [http://r20.rs6.net/tn.jsp?f=001HzXGe-7Pq8-zlt4lLYuOB6gUgJEJfvwghspnK22oQib- 




~~~~~~~~~~~~~~~~~~~~~~~~~~

 Join Our Mailing List [http://visitor.constantcontact.com/email.jsp?m=1103049216003]

 © 2001-2011 First Street Gallery

 Artist images copyrighted by the individual artists. All rights reserved.

~~~~~~~~~~~~~~~~~~~~~~~~~~
Forward email
http://ui.constantcontact.com/sa/fwtf.jsp? 
llr=dauwnjdab&m=1103049216003&ea=$events@liveapp.com$&a=1121719149089

This email was sent to events@liveapp.com by 
firststreetgallery@earthlink.net.

Update Profile/Email Address
http://visitor.constantcontact.com/do? 
p=oo&m=001JtllHKXFqd_bBq0Avb7lPA%3D%3D&ch=0d2a4930-e9f2-11e4-af5b-d4ae52754aa9&ca=cd656f5e-7d82-4a66-9e64-9ee1bb170ae9


Instant removal with SafeUnsubscribe(TM)
http://visitor.constantcontact.com/do? 
p=un&m=001JtllHKXFqd_bBq0Avb7lPA%3D%3D&ch=0d2a4930-e9f2-11e4-af5b-d4ae52754aa9&ca=cd656f5e-7d82-4a66-9e64-9ee1bb170ae9


Privacy Policy:
http://ui.constantcontact.com/roving/CCPrivacyPolicy.jsp

Online Marketing by
Constant Contact(R)
www.constantcontact.com



First Street Gallery | 526 W. 26th Street, Suite 209 | New York | NY | 10001

    I want to create a model that can triage new newsletters as either event or not. My current plan was to:

    1. Strip all HTML
    2. TFIDF on events vs non-events
    3. Create something that searches for a date/time (events have to have date/time), and then labels the post either 1 or 0
    4. Use some type of spam email code to assess if an email looks like spam, and label it as such

    Then throw that data, with the labels, and train using XGBoost or something. Does that sound reasonable? I don't have a strong background in NLP, but I know a lot of you are NLP experts, what kind of architecture would you recommend for this problem? Should I be using LDA or something to identify the parts of speech? Any recommended way to clean HTML?

  • TypeError: 'bytes' object is not callable - Python

    I'm getting the "TypeError: 'bytes' object is not callable". What am I doing wrong here? Please help :)

    data_file="reviews_data.txt.gz"
def read_input(data_file):



logging.info("reading file {0}...this may take a while".format(data_file))

with gzip.open (input_file, 'rb') as f:
    for i, line in enumerate (f): 

        if (i%10000==0):
            logging.info ("read {0} reviews".format (i))

        yield gensim.utils.simple_preprocess (line)


documents = list (read_input (data_file))

logging.info ("Done reading data file")

    TypeError Traceback (most recent call last) in ()

    ---> 18 documents = list (read_input (data_file))

    TypeError: 'bytes' object is not callable

    --- I used this file here: https://github.com/kavgan/data-science-tutorials/blob/master/word2vec/Word2Vec.ipynb

  • explaining lda coherence, while performing coherence.get_coherence()... What does KeyError regarding id2Token mean?

    I created an LDA model and a method for evaluating its performance. I started adding some filtering of highly infrequent terms to my LDA model and resolved other issues with its performance. But right now, it seems like sometimes my model crashes when I call getCoherence() and other times it doesn't. I don't really know why so... I need to know what exactly the error message that I'm being given means. In my case, I'm creating a topic model for a dataset of 198000 tweets. Here is the evaluation method code

    def evaluate_LDA_model(name_of_lda, dict_num):
    model_path = "models/" + name_of_lda
    dict_path = "dictionary_files/clean_doc_dict_" + str(dict_num) + ".dict"
    lda_model = gensim.models.ldamodel.LdaModel.load(model_path)
    doc_dict = corpora.Dictionary.load(dict_path)
    doc_clean_lst = getDocuments(clean=True) # returns dataframe with a column called 'cleaned_tokens' that has all of the processed tokens with their document/tweet
    docs_of_tokens = convert_cleaned_tokens_entries(doc_clean_lst['cleaned_tokens']) # Additional method for processing the cleaned tokens
    doc_term_matrix = [doc_dict.doc2bow(doc) for doc in docs_of_tokens]
    log_file_path = "txt_files/" + name_of_lda + "evaluation.txt"=
    coherence_model_var = CoherenceModel(model=lda_model, texts=docs_of_tokens, dictionary=doc_dict, coherence='c_v') # I wouldn't be surprised if I was passing in a parameter incorrectly here... I had trouble figuring out which parameters were supposed to be where.
    p = lda_model.log_perplexity(doc_term_matrix, total_docs = len(docs_of_tokens))
    coherence_lda = coherence_model_var.get_coherence() # crashes here.
    print("LOGGING RESULTS")
    with open(log_file_path, 'a') as log:
        log.write('Perplexity: \t' +  str(p) + '\n') # a measure of how good the model is. lower the better.
        log.write("Coherence Score: " +  str(coherence_lda) + '\n')
        log.write("Topics\n")
    pp(lda_model.print_topics(num_topics=10, num_words=10))
    print("FINISHED EVALUATION")

    For clarity's sake (cuz I know that the variable names aren't intuitive, it's partly why I got confused regarding parameters), doc_dict js created by essentially passing in docs_of_tokens to corpora.Dictionary()'s constructor.

    docs_of_tokens is something I usually extract from a csv file. But it's supposed to be a List or pandas Series of Lists. Each list/entry in docs_of_tokens is a tweet or individual document. The contents of each list in docs_of_tokens has all of the tokens for that particular tweet.

    So the error message I get is long, but I'm going to post the whole trace. See the last line.

    My questions are just

    What does the key error mean? What token and/or key is causing the problem and how would I figure that out?

    What is being evaluated when I call getcoherence? I'm trying to read up on coherence scores but to be honest I don't really know what's happening with my corpus (represented 

      Traceback (most recent call last):
  File "text_mining.py", line 394, in <module>
    main()
  File "text_mining.py", line 388, in main
    evaluate_LDA_model("lda_topic_model5", 3)
  File "text_mining.py", line 351, in evaluate_LDA_model
    coherence_lda = coherence_model_var.get_coherence()
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\models\coherencemodel
.py", line 603, in get_coherence
    confirmed_measures = self.get_coherence_per_topic()
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\models\coherencemodel
.py", line 563, in get_coherence_per_topic
    self.estimate_probabilities(segmented_topics)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\models\coherencemodel
.py", line 535, in estimate_probabilities
    self._accumulator = self.measure.prob(**kwargs)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\topic_coherence\proba
bility_estimation.py", line 138, in p_boolean_sliding_window
    accumulator = ParallelWordOccurrenceAccumulator(processes, top_ids, dictiona
ry)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\topic_coherence\text_
analysis.py", line 424, in __init__
    super(ParallelWordOccurrenceAccumulator, self).__init__(*args)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\topic_coherence\text_
analysis.py", line 280, in __init__
    super(WindowedTextsAnalyzer, self).__init__(relevant_ids, dictionary)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\topic_coherence\text_
analysis.py", line 185, in __init__
    self.relevant_words = _ids_to_words(self.relevant_ids, dictionary)
  File "C:\Users\biney\Miniconda3\lib\site-packages\gensim\topic_coherence\text_
analysis.py", line 60, in _ids_to_words
    word = dictionary.id2token[word_id]
KeyError: 9212
  • I cannot define my matrices for calculate spearman correlation matrix Python 3.6

    I'm trying to write a code that will find the Pearson and Spearman correlation of a matrix from Excel than show it if it's correlated or not according to my threshold. My code is like this:

    import pandas   
import numpy as np    
from itertools import chain

Correlation_coefficient = input ("PEARSON OR SPEARMAN?")   
A = "PEARSON"   
B = "SPEARMAN"

if Correlation_coefficient == A:
    S0 = pandas.read_excel(open('data1.xlsx', 'rb'))
    Y = np.corrcoef (S0)

            if Y[i][j] >= 0.90:
                print ("POSITIVE CORRELATION")
            elif Y[i][j] <= -0.90:
                print ("NEGATIVE CORRELATION")
            else :
                print ("NO CORRELATION ")

elif Correlation_coefficient == B:
    print ("Spearman code will appear here")

    My problem is that we can not calculate spearman correlation coefficient with my excel file because my numbers are in matrix defined and spearman does not accept the values in matrix.How can I calculate spearman correlation cofficient.My another problem is I want to print my output to a text file how can I have this. My output looks like this:

     CC1:   0.9999999999999999
 POSITIVE CORRELATION
 CC1:   -0.1271016204396284
 NO CORRELATION
 CC1:   1.0
 POSITIVE CORRELATION

    So I want that exactly print to my output text file how can I do this.

  • How to interpret a VAR model without sigificant coefficients?

    I am trying to investigate the relationship between some Google Trends Data and Stock Prices.

    I performed the augmented ADF Test and KPSS test to make sure that both time series are integrated of the same order (I(1)).

    However, after I took the first differences, the ACF plot was completely insigificant (except for 1 of course), which told me that the differenced series are behaving like white noise.

    Nevertheless I tried to estimate a VAR model which you can see attached.

    As you can see, only one constant is significant. I have already read that because Stocks.ts.l1 is not significant in the equation for GoogleTrends and GoogleTrends.ts.l1 is not significant in the equation for Stocks, there is no dynamic between the two time series and both can also be models independently from each other with a AR(p) model.

    I checked the residuals of the model. They fulfill the assumptions (normally distributed residuals are not totally given but ok, there is homoscedasticity, its stable and there is no autocorrelation).

    But what does it mean if no coefficient is significant as in the case of the Stocks.ts equation? Is the model just inappropriate to fit the data, because the data doesn't follow an AR process. Or is the model just so bad, that a constant would describe the data better than the model? Or a combination of the previous questions? Any suggestions how I could proceed my analysis?

    Thanks in advance

    VAR output (from R)

  • How to adjust coefficients for nonlinear curve fit using non-standard polynomial function in R

    I am attempting to perform a polynomial regression on a set of data, whose plot looks like this

    Plot 1

    However, every time I perform this regression, I get a low R squared value even though the points seem tightly packed around a line. My guess for why this is happening is that r is using equation to fit the curve

    First equation

    (although I am not certain that the poly function actually uses this equation, that is just a guess), and this doesn't seem capable of generating a curve like that in the plot. To resolve this problem, I have written a new equation which seems to fit the data pretty well.

    New Equation

    After playing with the numbers for a bit, I generated the red curve on the plot 1 with values of beta_0 = 22, beta_one = 195, and beta_two = .001. I then run a regression on this equation and get a much more reasonable R squared value. The code for this plot and the regression are as follows: 

    plot(full_simulation_data$T_n_rt_tds~full_simulation_data$d_rt_tds, pch = 20, cex = .5, ylim = c(0,1000), xlim = c(.75,1))

curve((22+((x^195)/.001*x)), add = TRUE, col = "red")

summary(lm(full_simulation_data$T_n_rt_tds~I(22+((full_simulation_data$d_rt_tds^195)/(.001*full_simulation_data$d_rt_tds)))))

    My question is twofold. One, is there a way that I can get R to automatically adjust these coefficients so that I don't have to fit the line by eyeballing it? Two, is there a way that the normal polynomial function can be fit to the data so that I don't have to invent new equations to fit the data?