MXNET reshape calling ctypes
Here trying to learn MXNET with “The Straight Dope”. On Linear Algebra - Tensors, I get ctypes error when running,
X = nd.arange(24).reshape((2, 3, 4))
The error is,
C:\Anaconda2\lib\site-packages\mxnet\ndarray\ndarray.pyc in reshape(self, *shape, **kwargs) 1060 c_array(ctypes.c_int64, shape), 1061 reverse, -> 1062 ctypes.byref(handle))) 1063 return NDArray(handle=handle, writable=self.writable) 1064 WindowsError: exception: access violation writing 0x00007FFB00000000
Why is this happening and how can I solve it?
See also questions close to this topic
Python Kafka Streaming API - Binning
I am using python kafka stream binning example given in this, Python Kafka Streaming API
I am able to generate the data using generator.py file given under winton-kafka-streams/examples/binning/, whereas when i run the binning.py file from the same folder, i got the below issue. Could someone help me, to resolve this?
Change color of missing values in Seaborn heatmap
Consider the example of missing values in the Seaborn documentation:
corr = np.corrcoef(np.random.randn(10, 200)) mask = np.zeros_like(corr) mask[np.triu_indices_from(mask)] = True sns.heatmap(corr, mask=mask, vmax=.3, square=True)
How do I change the color of the missing values to, for example, black? The color of the missing values should be specified independent of the color scheme of the heatmap, it may not be present in the color scheme.
I tried adding
facecolor = 'black'but that didn't work. The color can be affected by e.g.
sns.axes_style("white")but it isn't clear to me how that can be used to set an arbitrary color.
Xpath + Scrapy + Python : data point couldn't be scraped
This is the XML structure:
<tr> <td> <font size="3"> <strong>Location:</strong> Hiranandani Gardens, Powai </font> </td> </tr>
I want to extract : Hiranandani Gardens, Powai
I tried with these:
Both returned an empty list.
Note: we must have to use the text of tag, i.e., "Location:". Otherwise, there are many other places on the site where the same XML structure is used. So, it'll fetch many more unnecessary things apart from the desired value if the text of strong tag is not used.
Is there a way to name a tensorflow variable based on the value of another tensor-variable
I want to be able to do the following
n = str(tf.constant(2)) v = tf.get_variable(name=n, shape=(256,256), initializer=tf.contrib.layers.xavier_initializer())
but doing this is converts n to a string representation of tensorflow variable i.e
"<tf.Tensor 'Const_4:0' shape=() dtype=int32>"
MNIST data denormalising does not give me back the same
This is part of my learning. I Understood the normalization is really helping to improve accuracy, and hence divided by 255 the mnist values. This will divide all the pixels by 255, and hence all the pixels of 28*28 will have the values in range from 0.0 to 1.0 .
Now i tired to multiply the same with 255,this essentially means we should get the original value back. but when i display the picture, both the original and de-normalised pictures are different.
(trainX, trainY), (testX, testY) = mnist.load_data() plt.subplot(2,2,1) plt.imshow(trainX) trainX /= 255 plt.subplot(2,2,2) plt.imshow(trainX) trainX *= 255 plt.subplot(2,2,3) plt.imshow(trainX) plt.show()
What am i missing?. Anything related to float and int data type of the input data?
Which is best way of logging to file in C?
I'm dealing with some deep learning model written in C.
I wanna make some log file to check it later.
log will be a line for each step.
Each step will take few seconds.
Sometimes I use keyboard interrupt to stop procedure
Ways I thought are:
// Way 1 fp = fopen("log.txt", "a"); for each step: fprintf(fp, "Log content\n"); flose(fp);
I think way 1 may have lower file open/close overhead.
But when I use keyboard interrupt to stop procedure, log file will be never closed properly.
Is it OK?
Or can pass file pointer as argument of the my own signal handler?
// Way 2 for each step: fp = fopen("log.txt", "a"); fprintf(fp, "Log content\n"); fclose(fp);
I think way 2 will have file open/close overhead for each step, could this slow down whole performance?
Could it be crucial?
// Way 3 for each step: fprintf(buffer, "### Log content ###"); if step % 100 == 0: fp = fopen("log.txt", "a"); fprintf(fp, buffer); fclose(fp); flush buffer;
for way 3, i consider 2 ways of buffer:
1. array of string
2. one long string with line feed between items
Which way works best?
Or if there is a good library for logging, could you recommend it to me?
WinAPI: Possible to CreateWindowEx with a combination of WS_EX_TOOLWINDOW | WS_POPUP styles to hide taskbar icon?
I want to show a popup window which shows when clicking on an icon in the Windows 7 systray. But I want to invalidate the icon in the taskbar which is what the
WS_EX_TOOLWINDOWstyle provides as I understand it. Trouble is when I create it with the combined
WS_EX_TOOLWINDOW | WS_POPUPin the constructor after which I call
SetForegroundWindowit still shows the taskbar icon.
What am I doing wrong?
import ctypes ctypes.windll.user32.CreateWindowExA( 0, 'helloworld', 'helloworld', 2147483648 | 128, 0, 0, 400, 400, 0, 0, ctypes.windll.kernel32.GetModuleHandleA(None), None )
I tried creating a hidden window and assigning it as parent to the above in the 4th last parameter but still the taskbar icon did not disappear
hdnWindow = CreateWindowEx( 0, '', '', 128, 0, 0, 0, 0, 0, 0, ctypes.windll.kernel32.GetModuleHandleA(None), None )
Got an access violation when reading from instrument
I am writting a class a Thorlabs Optical Spectrometer. I am using python ctypes to read a provided FTSLib.dll.
Even though the class runs properly, I can engage the instrument, measure, process, and release it I get the following error, which shows up randomly when I try to read from the instrument:
OSError: exception: access violation reading 0x0000000000000004
Any ideas where does this come from?
calling libc.select from python from pty master side
I'm trying to create a simulated TTY connected target device that I can connect to via i.e. minicom. I'm using
/dev/ptmxto create a pty and print out the slave name to be opened by i.e. minicom:
Please connect to: /dev/pts/4. On the python side I then use os.read and os.write to do io and simulate my target:
import os, re, termios from ctypes import * class dev(): def __init__(self): pass def createpty(self): self.fd3 = os.open("/dev/ptmx", os.O_RDWR | os.O_NONBLOCK); if self.fd3 < 0: print("Couldn't open output /dev/ptmx\n") libc = cdll.LoadLibrary("libc.so.6") libc.grantpt(self.fd3); libc.unlockpt(self.fd3); libc.ptsname.restype = c_char_p self.slave = libc.ptsname(self.fd3) print("Please connect to:" + self.slave); self.old = termios.tcgetattr(self.fd3) n = termios.tcgetattr(self.fd3) n = n & ~(termios.ECHO|termios.ICANON) # c_lflag n = n & 0 n[4+1] = n[4+1] & 0xffff0000; termios.tcsetattr(self.fd3, termios.TCSANOW, n)
The whole process is quite combersome. I would like to use
selecton the python side which forces my to use ctypes again. I cannot wrap the self.fd3 into a file using
os.fdopenbecause I need to prevent
closebeing called on self.fd3.
So I have two questions:
- Does anyone know a readymade python library that handles pty creation and manipulation when implementing the pty master side in python?
- If not: Is there a example somewhere that describes howto call libc.select via ctypes?
docker Illegal instruction(core dumped) when import mxnet from the image ufomy/deepo
I used docker and deepo from this website: https://github.com/ufoym/deepo#GPU, When I start the container, I can import all the frameworks (tensorflow, caffe, pytorch)except mxnet. I don't know why. What should I do?
"OSError: [WinError 126] The specified module could not be found" while importing mxnet library
After installing below packages on Windows 8.1 CPU 64-bit using conda command: "conda install mxnet", I am getting error (OSError: [WinError 126] The specified module could not be found) while importing mxnet library.
_mutex_mxnet: 0.0.40-mkl libmxnet: 1.2.1-mkl_h0aaf724_1 mxnet: 1.2.1-h8cc8929_0 py-mxnet: 1.2.1-py36hcd68555_0
Also, popup window OS error appears while executing the import command with OS error description: "The program can't start because tiff.dll is missing from your computer.
Please note that tiff.dll file is already present in my folder "C:\Users\XX\AppData\Local\Continuum\anaconda3\pkgs\libtiff-4.0.9-hb8ad9f9_1\Library\bin" and have also tried re-installing the mxnet package but unfortunately problem is still not solved. Also, i already tried to searched similar issue on stackflow but couldn't find any relevant solution. Kindly help to resolve the issue.
PFB my code and error message.
from __future__ import print_function import numpy as np import mxnet as mx from mxnet import nd, autograd, gluon --------------------------------------------------------------------------- OSError Traceback (most recent call last) <ipython-input-32-4901168cc2e7> in <module>() 1 from __future__ import print_function 2 import numpy as np ----> 3 import mxnet as mx 4 from mxnet import nd, autograd, gluon ~\AppData\Local\Continuum\anaconda3\lib\site-packages\mxnet\__init__.py in <module>() 23 24 from .context import Context, current_context, cpu, gpu ---> 25 from . import engine 26 from .base import MXNetError 27 from . import base ~\AppData\Local\Continuum\anaconda3\lib\site-packages\mxnet\engine.py in <module>() 21 22 import ctypes ---> 23 from .base import _LIB, check_call 24 25 ~\AppData\Local\Continuum\anaconda3\lib\site-packages\mxnet\base.py in <module>() 111 __version__ = libinfo.__version__ 112 # library instance of mxnet --> 113 _LIB = _load_lib() 114 115 # type definitions ~\AppData\Local\Continuum\anaconda3\lib\site-packages\mxnet\base.py in _load_lib() 103 """Load library by searching possible path.""" 104 lib_path = libinfo.find_lib_path() --> 105 lib = ctypes.CDLL(lib_path, ctypes.RTLD_LOCAL) 106 # DMatrix functions 107 lib.MXGetLastError.restype = ctypes.c_char_p ~\AppData\Local\Continuum\anaconda3\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error) 346 347 if handle is None: --> 348 self._handle = _dlopen(self._name, mode) 349 else: 350 self._handle = handle OSError: [WinError 126] The specified module could not be found
MXNet - Dot Product of Sparse Matrices
I'm in the process of building a content recommendation model using MXNet. Despite being ~10K rows, out of memory issues are thrown with CPU and GPU contexts in MXNet. The current code is below.
import mxnet as mx import numpy as np import pandas as pd from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import linear_kernel df = pd.read_csv("df_text.csv") tf = TfidfVectorizer(analyzer = "word", ngram_range = (1,3), min_df = 2, stop_words="english") tfidf_matrix = tf.fit_transform(df["text_column"]) mx_tfidf = mx.nd.array(tfidf_matrix, ctx=mx.gpu()) # Out of memory error occurs here. cosine_similarities = mx.ndarray.dot(mx_tfidf, mx_tfidf.T)
I'm aware that the dot product is a sparse matrix multiplied by a dense matrix, which may be part of the issue. This said, would the dot product have to be calculated across multiple GPU's, in order to prevent out of memory issues?
How data pass for previous view controller using protocol and update collectionview cell swift
First view controller is Collectionview cell and pass data to nextview controller. Nextviewcontroller Edit the text and pass the text in first view contoller-collectionview cell and update the collectionview cell.
Could realtime EEG signal used directly in LSTM?
I am doing an experiment that using raw EEG data for LSTM. I have some quenstion about the input shape of this networks. Some one says that "You need a feature dimension, a batch size dimension and a time dimension". I think that means the input shape should be (feature,batch,time)。 But now I want to use the raw data without feature extraction.I think it may be unfitted for the raw data(256Hz/s).
For example,I have a long data for an hour(921600points). If I seperated in time-domian, each slice have 512 points. Can I use the 512 points as the directly input as the feature dimension?
Meaning: improvement in RMSE during crossvalidation, although not on testset?
In the code below I train a NN with crossvalidation on the first 20000 records in the dataset. The dataset contains 8 predictors.
First I have split my data in 2 parts: the first 20.000 rows (trainset) and the last 4003 rows (out of sample test set)
I have done 2 runs: run 1) a run with 3 predictors run 2) a run with all 8 predictors (see code below).
Based on crossvalidation within the 20.000 rows from the trainset, the RMSE (for the optimal parametersetting) improves from 2.30 (run 1) to 2.11 (run 2).
Although when I test both models on the 4003 rows from the out of sample test set, the RMSE improves only neglible from 2.64 (run 1) to 2.63 (run 2).
What can be concluded from these contradiction in the results?
### R code from Applied Predictive Modeling (2013) by Kuhn and Johnson. ### Chapter 7: Non-Linear Regression Models ### Required packages: AppliedPredictiveModeling, caret, doMC (optional), ### earth, kernlab, lattice, nnet ################################################################################ library(caret) ### Load the data mydata <- read.csv(file="data.csv", header=TRUE, sep=",") validatiex <- mydata[20001:24003,c(1:8)] validatiey <- mydata[20001:24003,9] mydata <- mydata[1:20000,] x <- mydata[,c(1:8)] y <- mydata[,9] parti <- createDataPartition(y, times = 1, p=0.8, list = FALSE) x_train <- x[parti,] x_test <- x[-parti,] y_train <- y[parti] y_test <- y[-parti] set.seed(100) indx <- createFolds(y_train, returnTrain = TRUE) ctrl <- trainControl(method = "cv", index = indx) ## train neural net: nnetGrid <- expand.grid(decay = c(.1), size = c(5, 15, 30), bag = FALSE) set.seed(100) nnetTune <- train(x = x_train, y = y_train, method = "avNNet", tuneGrid = nnetGrid, trControl = ctrl, preProc = c("center", "scale"), linout = TRUE, trace = FALSE, MaxNWts = 30 * (ncol(x_train) + 1) + 30 + 1, maxit = 1000, repeats = 25, allowParallel = FALSE) nnetTune plot(nnetTune) predictions <- predict(nnetTune, validatiex, type="raw") mse <- mean((validatiey - predictions)^2) mse <- sqrt(mse) print (mse)