Slicing overlapping sets of features from 2D numpy array into 3D numpy array
I am confronted with the following problem:
I have a 2D numpy array of shape (total_samples, n_features)
. Think of total_samples as time series i.e. features recorded at every time step. I would like to reshape this 2D array into a 3D array of shape (n_batch, n_samples, n_features)
to be used as input for a “many to one” long short term memory (LSTM) neural network (NN). I would like to be able to specify n_samples
and percentage_overlap
between batches as inputs to a python function that figures out how many batches or strides are possible from the available 2D array given the desired percentage_overlap and n_samples
.
def create_chunks((total_samples, n_features), n_samples, overlap=.5)
return (n_batch, n_samples, n_features)
Numerically, let’s say for a single feature and 4 samples:
[[1], [2], [3], [4]]
using n_samples=2
and percentage_overlap=.5
should output
[[[1], [2]], [[2], [3]], [[3], [4]]].
See also questions close to this topic

Shell is not showing text from my prints
I'm trying to create a code for my lists work and i have written this at the moment:
list = [ ] def Menu(choice): print("""Menu: 1. View entire list 2. View one item 3. Reset list 4. Edit list 5. Quit """) choice = int(input("Please choose an option: ")) while True: if choice == "1": print (list) Menu() elif choice == "2": oneItem = int(input('Please enter an item to view (15): ')) oneItem = oneItem  1 oneView = list[(oneView)] print (oneView) Menu() elif choice == "3": list = [None]*6 Menu() elif choice == "4": def carryon(): for i in range(6): add = input("Please enter a new item for the list: ") list.append(add) carryOn = input("Would you like to carry on? (y/n): ") if carryOn == "y": carryon() else: break break Menu()
When I run the code in shell the menu appears fine:
Menu: 1. View entire list 2. View one item 3. Reset list 4. Edit list 5. Quit Please choose an option: 3
But then it just comes up blank. I can type things in still but nothing happens. Even after i press enter it just carries on. I use python 3.4 (gui)

How the zip() function in Python works?
I am new to python and I'm trying to understand how this function do what it does. I want to know what is happening behind it to understand it better.

Uploading to S3 using Python Requests
I'd like to upload xml's directly to S3 without the use of modules like boto, boto3, or tinys3.
So far I have written:
url = "https://mytests3.s3.amazonaws.com" with open(xml_file,'rb') as data: requests.put(url, data=data)
and I've gone and head and set the AllowedOrigin on my S3 bucket to accept my server's address.
This does not error when running, however, it also does not seem to be uploading anything.
Any help would be appreciated  I'd like to (a) get the thing to upload and (b) figure out how to apply AWSAccessKey and AWSSecretAccessKey to the request

Need a little twitch in my regex to get all the numbers
How to fix the pattern to get the all listed numbers. Currently what I'm getting with the way I've tried is given below:
import re content=''' (555) 5674523, 555 567 4523, (555) 567 4523, 5555674623 +918071805024, +91112352280 ''' print(re.findall(r"\+?.\d+.?\(?\d+.\)??\s?\w*.?\s?\w*.",content))
Result I'm getting:
['(555) 5674523,', ' 555 567 4523, (', ' 567 4523, 555567', '+918071805024, +91', '112352280']
Result I'm after:
(555) 5674523, 555 567 4523, (555) 567 4523, 5555674623 +918071805024, +91112352280

how to download linkedin profile into pdf using python
from bs4 import BeautifulSoup from selenium import webdriver import time import csv import pandas as pd import pdfkit driver=webdriver.Chrome() driver.get('https://www.linkedin.com/in/tejaswaniverma594b3779/') time.sleep(40) soup= BeautifulSoup(driver.page_source, 'lxml') f=open('htmlfile.html', 'w') for item in soup.select('.pvoc.emberview'): f.write("%s" % item) pdfkit.from_file('htmlfile.html', 'tjprofile.pdf') driver.quit()
I can download the profile from save to pdf option easily but when i want to download using python it's view is not correct how can i do that?

How to extract elements (title names, votes, views, answers for each title) from https://stackoverflow.com/?tab=month?
I could figure out the reading the page and storing the text to a file part (
urllib2
) but not the extraction of elements(votes
,title
,...). The page is https://stackoverflow.com/?tab=monthHow do I write a python script to get a webpage and then extract all the titles, no. of views, votes and answers received by each question?

Modified Gram Schmidt in Python for complex vectors
I wrote some code to implement the modified Gram Schmidt process. When I tested it on real matrices, it is correct. However, when I tested it on complex matrices, it went wrong.
I believe my code is correct by doing a step by step check. Therefore, I wonder if there are numerical reasons why the modified Gram Schmidt process fails on complex vectors.
Following is the code:
import numpy as np def modifiedGramSchmidt(A): """ Gives a orthonormal matrix, using modified Gram Schmidt Procedure :param A: a matrix of column vectors :return: a matrix of orthonormal column vectors """ # assuming A is a square matrix dim = A.shape[0] Q = np.zeros(A.shape, dtype=A.dtype) for j in range(0, dim): q = A[:,j] for i in range(0, j): rij = np.vdot(q, Q[:,i]) q = q  rij*Q[:,i] rjj = np.linalg.norm(q, ord=2) if np.isclose(rjj,0.0): raise ValueError("invalid input matrix") else: Q[:,j] = q/rjj return Q
Following is the test code:
import numpy as np # If testing on random matrices: # X = np.random.rand(dim,dim)*10 + np.random.rand(dim,dim)*5 *1j # If testing on some good one v1 = np.array([1, 0, 1j]).reshape((3,1)) v2 = np.array([1, 1j, 1]).reshape((3,1)) v3 = np.array([0, 1, 1j+1]).reshape((3,1)) X = np.hstack([v1,v2,v3]) Y = modifiedGramSchmidt(X) Y3 = np.linalg.qr(X, mode="complete")[0] if np.isclose(Y3.conj().T.dot(Y3), np.eye(dim, dtype=complex)).all(): print("The QRcomplete gives orthonormal vectors") if np.isclose(Y.conj().T.dot(Y), np.eye(dim, dtype=complex)).all(): print("The Gram Schmidt process is tested against a random matrix") else: print("But My modified GS goes wrong!") print(Y.conj().T.dot(Y))
Update
The problem is that I implemented a algorithm designed for inner product linear in first argument
whereas I thought it were linear in second argument.
Thanks @landogardner

Install numpy latest package using aptget
I used below query to install numpy package. But it installed 1.11.0 version. How to install the most recent version.
sudo aptget install pythonnumpy

Figuring out the proper numpy.reshape for 1 option
I have a (hopefully) quick Numpy question, and I hope you can help me. I want to use numpy.reshape to convert (5000, 32, 32, 3) into (5000, 3072), and the only clue I got for the assignment is this:
# Reshape each image data into a 1dim array print (X_train.shape, X_test.shape) # Should be: (5000, 32, 32, 3) (500, 32, 32, 3) ##################################################################### # TODO (2): # # Reshape the image data to one dimension. # # # # Hint: Look at the numpy reshape function and have a look at 1 # # option # ##################################################################### X_train = X_test = ##################################################################### # END OF YOUR CODE # ##################################################################### print (X_train.shape, X_test.shape) # Should be: (5000, 3072) (500, 3072)
I've been spending the last day scouring Google for examples, but apparently this is too trivial to warrant an ask. Help?

Gathering list of 2d tensors from a 3d tensor in Keras
I have a 3d Tensor named main_decoder of shape (None,9,256)
I want to extract 9 tensors of shape (None,256)
I have tried using Keras gather and the following is mode code snippet:
for i in range(0,9): sub_decoder_input = Lambda(lambda main_decoder:gather(main_decoder,(i)), name='lambda'+str(i))(main_decoder)
the resultant is 9 lambda layers of shape (9,256)
How can I modify it so that I can get or gather 9 tensors of shape (None,256)
Thanks.

Negative result for regression using Keras and Tensorflow
I was following the tutorial by Jason Brownlee, and finished section 2. Where his code apparently gave, as a result,
Baseline: 31.64 (26.82) MSE
mine gave
Results: 59.54 (45.45) MSE
I am intrigued by the negative result, since it's supposed to be the price of square meters (the case study is Boston housing data).
His code is as follows:
import numpy import pandas from keras.models import Sequential from keras.layers import Dense from keras.wrappers.scikit_learn import KerasRegressor from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline # load dataset dataframe = pandas.read_csv("housing.csv", delim_whitespace=True, header=None) dataset = dataframe.values # split into input (X) and output (Y) variables X = dataset[:, 0:13] Y = dataset[:, 13] # define base model def baseline_model(): # create model model = Sequential() model.add(Dense(13, input_dim=13, kernel_initializer='normal', activation='relu')) model.add(Dense(1, kernel_initializer='normal')) # Compile model model.compile(loss='mean_squared_error', optimizer='adam') return model # fix random seed for reproducibility seed = 7 numpy.random.seed(seed) # evaluate model with standardized dataset estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=5, verbose=0) kfold = KFold(n_splits=10, random_state=seed) results = cross_val_score(estimator, X, Y, cv=kfold) print("Results: %.2f (%.2f) MSE" % (results.mean(), results.std())) print(results)
EDIT: result is now
Using TensorFlow backend. 20171117 15:19:20.550701: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Results: 59.57 (45.48) MSE
[ 21.09693904 34.74104155 24.24863872 113.87524421 80.55262979 > 98.56815988 39.08693658 150.61814082 17.41903878 15.52491613] Exception ignored in: > Traceback (most recent call last): File "/home/zchihani/pythonEnvs/kerasEnv/lib/python3.5/sitepackages/tensorflow/python/client/session.py", line 696, in del TypeError: 'NoneType' object is not callable
Interestingly (maybe), further calls did not show the 'exception ignored' part.
end of edit
I read here that "when the training error decreases to a certain level, the outputted prediction sometimes is negative".
The proposed response is to use
from keras.constraints import nonneg
but it does not say how or where...
For info, mu keras.json is
{ "backend": "tensorflow", "epsilon": 1e07, "floatx": "float64", "image_dim_ordering": "tf", "image_data_format": "channels_last" }
and the output of pip list (in the virtual env) gives
autopep8 (1.3.3) bleach (1.5.0) enum34 (1.1.6) flake8 (3.5.0) h5py (2.7.1) html5lib (0.9999999) importmagic (0.1.7) jedi (0.11.0) Keras (2.0.9) Markdown (2.6.9) mccabe (0.6.1) numpy (1.13.3) olefile (0.44) pandas (0.21.0) parso (0.1.0) Pillow (4.3.0) pip (9.0.1) pkgresources (0.0.0) protobuf (3.4.0) pycodestyle (2.3.1) pyflakes (1.6.0) pythondateutil (2.6.1) pytz (2017.3) PyYAML (3.12) rope (0.10.7) scikitlearn (0.19.1) scipy (1.0.0) setuptools (36.7.0) six (1.11.0) tensorflow (1.4.0) tensorflowgpu (1.4.0) tensorflowtensorboard (0.4.0rc2) Theano (1.0.0rc1+5.gab69bf0) Werkzeug (0.12.2) wheel (0.30.0) yapf (0.19.0)

How to implement custom sobelfilterbased loss function using Keras
I am new to DL and Keras and currently I am trying to implement a sobelfilterbased custom loss function in Keras.
The idea is to calculate the mean squared loss of a sobel filtered prediction and a sobel filtered ground truth image.
So far, my custom loss function looks like this:
from scipy import ndimage def mse_sobel(y_true, y_pred): for i in range (0, y_true.shape[0]): dx_true = ndimage.sobel(y_true[i,:,:,:], 1) dy_true = ndimage.sobel(y_true[i,:,:,:], 2) mag_true[i,:,:,:] = np.hypot(dx_true, dy_true) mag_true[i,:,:,:] *= 1.0 / np.max(mag_true[i,:,:,:]) dx_pred = ndimage.sobel(y_pred[i,:,:,:], 1) dy_pred = ndimage.sobel(y_pred[i,:,:,:], 2) mag_pred[i,:,:,:] = np.hypot(dx_pred, dy_pred) mag_pred[i,:,:,:] *= 1.0 / np.max(mag_pred[i,:,:,:]) return(K.mean(K.square(mag_pred  mag_true), axis=1))
Using this loss function leads to this error:
in mse_sobel for i in range (0, y_true.shape[0]): TypeError: __index__ returned nonint (type NoneType)
Using the debugger I found out, that
y_true.shape
only returnsNone
 fine. But when I replacey_true.shape
with for example1
such that it looks like thisfor i in range (0,1):
, another error occurs:in sobel axis = _ni_support._check_axis(axis, input.ndim) in _check_axis raise ValueError('invalid axis') ValueError: invalid axis
Here, I am not sure about why the axis seems to be invalid?
Can anyone help me figure out how to implement that loss function? Thank you very much for your help!