Split Dataframe into a zip object with n tuples Pandas
I'd like to know how to do a function where the input is going to be a list of n columns in a dataframe and output is going to be a zip object with that n columns series, Example
def convert_series_into_zip(df,[n1,n2,n2,n3,n2]):
return zip(df['n1'],df['n2'],df['n3']...)
The reason of that is that instead of using vector operations I need loop between n columns in a dataframe and use the ith value of the n series like a tuple (Exactly like the zip function does)
Thank you.
See also questions close to this topic

How to train a multiclass image classifier with Keras
I was following the 2 class image classification tutorial here and wanted to convert it into a multiclass classifier.
I am trying to train a model to predict the brand of a watch from 17 classes. My accuracy after 50 epochs is only 21.88% so I'm not really sure where I am going wrong or if I am even doing this right.
Here is my code:All the images are in their own separate folders under the /data or /valid folders.
Ex: ../watch finder/data/armani
Ex2: ../watch finder/data/gucciimport numpy as np import matplotlib.pyplot as plt import os import cv2 from keras.utils import np_utils from keras.preprocessing.image import ImageDataGenerator from keras.models import Sequential from keras.layers import Conv2D, MaxPooling2D from keras.layers import Activation, Dropout, Flatten, Dense from keras import backend as K import keras.optimizers img_width, img_height = 210, 210 train_data_dir = 'C:/Users/Adrian/Desktop/watch finder/data' validation_data_dir = 'C:/Users/Adrian/Desktop/watch finder/valid' nb_train_samples = 4761 nb_validation_samples = 612 epochs = 50 batch_size = 16 model = Sequential() model.add(Conv2D(32, (3, 3), input_shape=input_shape)) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Conv2D(32, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Flatten()) model.add(Dense(64)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(17)) model.add(Activation('softmax')) model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy']) train_datagen = ImageDataGenerator( rescale=1. / 255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True) test_datagen = ImageDataGenerator(rescale=1. / 255) train_generator = train_datagen.flow_from_directory( train_data_dir, target_size=(img_width, img_height), batch_size=batch_size, class_mode='categorical') validation_generator = test_datagen.flow_from_directory( validation_data_dir, target_size=(img_width, img_height), batch_size=batch_size, class_mode='categorical') model.fit_generator( train_generator, samples_per_epoch=nb_train_samples // batch_size, epochs=epochs, validation_data=validation_generator, validation_steps=nb_validation_samples // batch_size)
This is my first epoch:
Epoch 1/50 18/18 [==============================]  8s 422ms/step  loss: 4.1104  accuracy: 0.0833  val_loss: 2.8369  val_accuracy: 0.0592
And this is my 50th/last epoch:
Epoch 50/50 18/18 [==============================]  7s 404ms/step  loss: 2.4840  accuracy: 0.2188  val_loss: 3.0823  val_accuracy: 0.1795
I am fairly certain I am doing something wrong here but I am really new to deep learning so I'm not sure what that something is. All the help is appreciated.
Also, each brand of watch has between 300400 images and each image size is the same at 210x210.

Find all possible combinations of even or odd numbers and perform mathematical operations on the result
I am trying to find all possible combinations from taking one integer from each of three lists. The combinations must contain either all even or all odd integers.
I would then like to find the sum of the squares of the integers in each combination.
Finally I would like to create a dictionary that uses this value as the key and the combination it used as its stored value.
E.g. for the combination (1,3,1): the key would be the integer 11 (from the sum (1^2) + (3^2) + (1^2)) and the value stored would be (1,3,1)
My code so far:
lists = [[0,1,2,3,4,5,6],[0,1,2,3,4,5,6],[0,1,2,3,4,5,6]] combos = np.array([list(i) for i in np.array(np.meshgrid(*values)).T.reshape(1,len(values))])
This takes three lists that each contain the integers 06
and uses an element from each list to create a combination of three integers
All possible combinations result: [0,0,0], [0,1,0], [0,2,0], [1,0,0] ... [6,6,6]

Is it possible to import a variable without all other data?
If i want to import a variable from another file , lets say file name is 'USA' and i want to import only the 'population' variable.
I do it by
From USA import population
All is good, all is working. But it also imports everything else from the 'USA' file, which means that if i had print Function variables it will print them too at the file i imported the single variable to..
Is it possible to import a variable without importing all the functions, or at least hide the print functions somehow?

How to apply value_counts to all elments of a DataFrame?
It is easy to apply value_counts to a Series:
In [1]: import pandas as pd In [2]: a = pd.DataFrame([[2,3],[2,2],[3,2],[2,1]]) In [3]: a Out[3]: 0 1 0 2 3 1 2 2 2 3 2 3 2 1 In [4]: a[0].value_counts() Out[4]: 2 3 3 1 Name: 0, dtype: int64
I need something like
In [5]: a.value_counts() Out[5]: 2 5 3 2 1 1 dtype: int64
However,
a.value_counts()
returns'DataFrame' object has no attribute 'value_counts'
.How could I apply value_counts to the element of a DataFrame?

Does it make sense to replace nan values by 99999?
how does it make sense to replace
nan
values in the dataframe by the value99999
? I found it here, example 3: https://www.geeksforgeeks.org/pythonpandasdataframereplace/df.replace(to_replace = np.nan, value =99999)
Maybe
99999
should simply representinfinite
, but which intention could be behind this operation? Any ideas, guesses? :/I'd appreciate any advise!

Python datetime problem converting time format
I would like to convert the following time format which is located in a panda dataframe column
100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400
I would like to transform the previous time format into a standard time format of HH:MM as follow
01:00 02:00 03:00 ... 15:00 16:00 ... 22:00 23:00 00:00
How can I do it in python?
Thank you in advance

What is the proper way of loading images onto a website?
I've built a website using Next.js which is a framework built around React.js
my problem is that i don't know what is the proper way to load the images to the website. should i just include all the image files for different screen sizes into the project directory and from there load them via their relative path? or should i first upload all the images to a website like imgur, flickr and then load them via their url?
I'm wondering whether there is a difference between these two methods for loading images and performance, which one is preferred by web developers and which one results in faster load times?

Is there an efficient way to find averages of boxes within a numpy array?
Explanation
I have a bunch of 450x450 images, and I am looking at each color channel individually. So what I have is a 1D numpy array of length 450*450*numImages. I want to take a box of some size, say 3x3 or 5x5 around each pixel and get the average of those pixel values. So by the end I will still have a 450*450*numImages array, it will just now hold those averages. Imagine something like this where we have one 14x14 image and we're just looking at one of the color channels and taking a 3x3 box average:
Current Implementation
iWidth=450 def boxAverage(imageArr, box_scale=1): avgs=[] for i in range(len(imageArr)): avg=0 avgCount=0 for j in range(box_scale,box_scale): for k in range(box_scale*iWidth,box_scale*iWidth,iWidth): try: temp = imageArr[i+k+j] except IndexError: temp = 1 if temp != 1: avg+=temp avgCount+=1 avg/=avgCount avgs.append(avg)
When box_scale = 1, it is a 3x3 box that I am taking the average of and when box_scale=2, it is 5x5, etc.
Problem
This does not scale very well and as I work with more and more images, this gets very slow. So I was wondering if there is a good way using something like numpy to do this in a way that can be done more efficiently. I appreciate any suggestions you might have!

Speedup when combining CPU and GPU with OpenCL
I have an application that uses OpenCL to split a matrix vector multiplication over several devices. I tested the application on a computer with an Intel Xeon CPU and three identical NVIDIA Kepler GPUs. If I use an input size of 2048x2048, then I have a speedup of ~1.4 for the CPU compared to the GPU. Two GPUs have a speedup of ~2 compared to one GPU, but one GPU plus one CPU only has a speedup of ~1.7. If I use three devices, the result looks similar compared to one GPU: Three GPUs have a speedup of ~3, but two GPUs plus one CPU only have a speedup of ~2.4.
So my question is: What can be the reason that the speedup decreases when I replace a GPU with a CPU. The CPU is seperately faster than the GPU.

How to vectorize and optimize this functions in C?
I have this functions, the result is correct but the compiler don't vectorize this. How can I achive that the compiler vectorize this and how can I optimize this codes?
void LongNumSet( char *L, unsigned N, char digit ) { for (int i = 0; i < N; ++i){ L[i] = digit; } } void LongNumCopy( char *Vin, char *Vout, unsigned N ) { for ( int i=0; i< N; ++i ) { Vout[i] = Vin[i]; } } char LongNumAddition( char *__restrict Vin1, char * __restrict Vin2, char * __restrict Vout, unsigned N ) { char CARRY = 0,R,aux; Vin1 = (char*)__builtin_assume_aligned (Vin1, 1); Vin2 = (char*)__builtin_assume_aligned (Vin2, 1); for ( int i=0; i< N; ++i ) { char R = Vin1[i] + Vin2[i] + CARRY; aux = R <= 9; Vout[i] = (aux) ? R:Rten; CARRY = (aux) ? 0:1; } return CARRY; } char LongNumAddDigit( char *V, char digit, unsigned N ) { int i=0; char R = V[0] + digit; if ( R < ten){ V[0] = R; return 0; } V[0] = Rten; // add carry, maybe iteratively for all digits char CARRY = 1; i = 1; while ( CARRY && i < N ) { if ( V[i] < 9 ) { V[i]++; CARRY = 0; } else { V[i] = 0; i++; // CARRY remains set to 1 } } return CARRY; }
I use the comand gcc O3 ffastmath msse funrollallloops ftreevectorizerverbose=25 lm g $1 o ${2}.O3 and I executate the program in 55 s. This is all of code:
#include <stdio.h> #include <stdlib.h> #include <stdint.h> // Variable used to generate pseudorandom numbers unsigned int seed; unsigned int temp; unsigned int var1 = 214013; unsigned int var2 = 2531011; #define val13 13 #define ten 10 // Function to generate pseudorandom numbers inline int myRandom() { temp = var1*seed; seed = temp + var2; return (seed>>val13); } void LongNumInit( char *L, unsigned N ) { for ( int i=0; i<N;++i ) { L[i] = myRandom() % ten; // digito decimal } } void LongNumPrint( char *L, unsigned N, char *Name ) { printf("%s:", Name); for ( int i=N; i>0; i ) { printf("%d", L[i1]); } printf("\n"); } void LongNumSet( char *L, unsigned N, char digit ) { for (int i = 0; i < N; ++i){ L[i] = digit; } } void LongNumCopy( char *Vin, char *Vout, unsigned N ) { for ( int i=0; i< N; ++i ) { Vout[i] = Vin[i]; } } char LongNumAddition( char *__restrict Vin1, char * __restrict Vin2, char * __restrict Vout, unsigned N ) { char CARRY = 0,R,aux; Vin1 = (char*)__builtin_assume_aligned (Vin1, 1); Vin2 = (char*)__builtin_assume_aligned (Vin2, 1); for ( int i=0; i< N; ++i ) { char R = Vin1[i] + Vin2[i] + CARRY; aux = R <= 9; Vout[i] = (aux) ? R:Rten; CARRY = (aux) ? 0:1; } return CARRY; } char LongNumAddDigit( char *V, char digit, unsigned N ) { int i=0; char R = V[0] + digit; if ( R < ten){ V[0] = R; return 0; } V[0] = Rten; // add carry, maybe iteratively for all digits char CARRY = 1; i = 1; while ( CARRY && i < N ) { if ( V[i] < 9 ) { V[i]++; CARRY = 0; } else { V[i] = 0; i++; // CARRY remains set to 1 } } return CARRY; } char LongNumHorizAdd( char *Vin, char *Vout, unsigned N ) { char CARRY = 0; LongNumSet ( Vout, N, 0 ); for ( int i=0; i< N; ++i ) { LongNumAddDigit ( Vout, Vin[i], N ); } return 0; // CARRY can never be set } char LongNumConstMult( char *V, unsigned N, char digit ) { char CARRY = 0; char ja = 0; for ( int i=0; i< N; ++i ) { char aux = V[i] * digit; char R = aux + CARRY; CARRY = ((u_int32_t)R * (u_int32_t)0xCCCD) >> 19; ja = (CARRY << 3) + 2*CARRY; R = ja; V[i] = R; } return CARRY; // may be from 0 to 9 } void LongNumMultiply( char *Vin1, char *Vin2, char *VoutH, char *VoutL, unsigned N ) { // Create Temporal Long Integer with double size unsigned char *TEMP = (unsigned char*) calloc(2*N,sizeof(unsigned char)); unsigned char *RES = (unsigned char*) calloc( 2*N,sizeof(unsigned char) ); LongNumSet ( RES, 2*N, 0 ); // Set RES to 0 for ( int i=0; i<N; ++i ) { LongNumSet ( TEMP, 2*N, 0 ); // Set TEMP to 0 LongNumCopy ( Vin1, TEMP+i, N ); // Copy Vin1 > TEMP, with offset i LongNumConstMult( TEMP, 2*N, Vin2[i] ); // TEMP * Vin2[i] > TEMP LongNumAddition ( TEMP, RES, RES, 2*N ); // TEMP + RES > RES } // Result goes to VoutHVoutL LongNumCopy ( RES, VoutL, N ); // Copy RES > VoutL LongNumCopy ( RES+N, VoutH, N ); // Copy RES+N > VoutH } int main (int argc, char **argv) { int i, sum1, sum2, sum3, N=10000, Rep=50; seed = 12345; // obtain parameters at run time if (argc>1) { N = atoi(argv[1]); } if (argc>2) { Rep = atoi(argv[2]); } printf("Challenge #3: Vector size is %d. Repeat %d times\n", N, Rep); // Create Long Nums unsigned char *V1= (unsigned char*) malloc( N*sizeof(unsigned char) ); unsigned char *V2= (unsigned char*) malloc( N*sizeof(unsigned char) ); unsigned char *V3= (unsigned char*) malloc( N*sizeof(unsigned char) ); unsigned char *V4= (unsigned char*) malloc( N*sizeof(unsigned char) ); LongNumInit ( V1, N ); LongNumInit ( V2, N ); LongNumInit ( V3, N ); // Repeat for (i=0; i<Rep; i++) { LongNumAddition ( V1, V2, V4, N ); LongNumMultiply ( V3, V4, V2, V1, N ); LongNumHorizAdd ( V1, V2, N ); LongNumAddDigit ( V3, V2[0], N ); } // Print last 32 digits of Long Numbers LongNumPrint( V1, 32, "V1" ); LongNumPrint( V2, 32, "V2" ); LongNumPrint( V3, 32, "V3" ); LongNumPrint( V4, 32, "V4" ); free(V1); free(V2); free(V3); free(V4); return 0; }
Thanks for your help.

How to Reduce execution time in for loops in python, as it is taking more a hour for execution?
Please let me know a solution by using NUMPY vectorization or Lambda fumctions for reducing execution time

Numpy: Efficient mapping of single values in np arrays
I have a 3D numpy array respresenting an image in HSV color space (shape = (h=1000, w=3000, 3)).
The last dimension of the image is [H,S, V]. I want to subtract 20 from the H channel from all the pixels IF the pixel value is >20 , but leave S and V intact. I wrote the following vectorized function:
def sub20(x): # x is a array in the format [H,S, V] return np.uint8([H20, S, V]) vec= np.vectorize(sub20, otypes=[np.uint8],signature="(i)>(i)") img2= vec(img1)
What this vectorised function does is to accept the last dimension of the image [H,S,V] and output [H20, S, V] I dont know how to make it subtract 20 if H is greater than 20. it also takes 1 minute to execute. I want the script to accept live webcam feed. Is there any way to make it faster?
Thanks