Used numba to consume GPU instead of CPU in loop
I am doing tuple operation in python which takes huge amount of memory. Every process has been ended after some minute. Now, i want to implement numba in loop to consume my GPU.
for i in range(0, len(idendities) - 1):
for j in range(i + 1, len(idendities)):
#print(samples_list[i], " vs ",samples_list[j])
cross_product = itertools.product(samples_list[i], samples_list[j])
cross_product = list(cross_product)
#print(cross_product)
for cross_sample in cross_product:
#print(cross_sample[0], " vs ", cross_sample[1])
negative = []
negative.append(cross_sample[0])
negative.append(cross_sample[1])
negatives.append(negative)
print(len(negatives))
negatives = pd.DataFrame(negatives, columns=["file_x", "file_y"])
negatives["decision"] = "No"
See also questions close to this topic
-
Function in a function in a while loop can not update an outside class variable?
So this code runs correctly...
class Deck: def __init__(self): self.cards = [] for i in range(6): self.cards.append(i) class Player: def __init__(self): self.cards = [] class Game: def __init__(self): self.player = Player() self.deck = Deck() def play_game(self): self.add_card() print(self.player.cards) def add_card(self): self.player.cards.append(self.deck.cards.pop()) game = Game() game.play_game()
But if I add
while True:
to theplay_game(self)
function like this...def play_game(self): while True: self.add_card() print(self.player.cards)
It says it can not
pop()
from an empty list. So I assume the function within the while loop is not able to access the variableself.player.cards
, within the same instance as the correctly stated code does. Am I right about that?Basically I just want to understand why this doesn't work, and what is going on...?
I do apologize if my wording is not accurate, I am new to Python...
Thanks!
-
pd.cut returns a column of NaNs
I wrote following code, but return all NaNs in the new column.
bins = [0, 1, 2, 5] cat = ['zero', 'one', 'two and more'] df['group'] = pd.cut(df['count'], bins, labels = cat)
this new df['group'] returned all NaNs
I checked ['count'], the dtypes is float. It worked for other columns. Why this one is not working?
-
Pattern match multiple values in a sentence
I have a sentence that has a specific format.
<subject> <action> <object> @ <price> ... // The sentence can continue
and I want to extract these values out of the sentence.
Constraints:
- The subject is always
Bob
orAlice
- Action is either
bought
orsold
- The object can be any word of 1-7 letters //
4apples
should return NULL - Price is either a float/integer
- There can be sentences before
subject
but guaranteed to not containBob/Alice
. - There may or may not be a space after
@
Example:
Hi there, Bob sold apples @2.0 dollars each
Desired Output:
Subject: Bob Action: sold Object: apples Price: 2.0
Currently, I do it the naive way by:
#!/usr/bin/env python3 sentence = "Hi there, alice sold apples @2.0 dollars each" sentence = sentence.lower() if 'alice' in sentence or 'bob' in sentence: s_list = sentence.split(" ") s_idx = -1 if 'bob' in sentence: s_idx = s_list.index('bob') elif 'alice' in sentence: s_idx = s_list.index('alice') if s_idx > -1: Subject = s_list[s_idx] Action = s_list[s_idx+1] Object = s_list[s_idx+2] #more if/else to validate Object contraints Price = s_list[s_idx+3] #more if/else to extract 2.0 if we get @2.0 print("Subject: {}, Action: {}, Object: {}, Price: {}".format(Subject, Action, Object, Price))
How can I do this better? Possibly using
re
- The subject is always
-
How Does Replacing Repeated Code With a Function Affect Performance in an Interpreted Language Such as Python
Lets say I wrote a large section of code and copied it ten times. How would performance (CPU usage, ram usage, running time) change if I were to make the code a function and call it 10 times?
-
Webscraping website search bars with python
I am trying to write some code for a personal project where i can scrape data from a site while also using that site's query box. Furthermore, the website i am trying to use is https://www.latlong.net/convert-address-to-lat-long.html and I am trying to have a portion of my program where you input your address, then the request goes to the url's address search bar and perfoms the query, and then extracts the lat/lon elements from the site and stores it in a dataframe. I know i will need to use beautifulsoup and, from what ive read, possibly mechanize and selenium but i got a a little lost trying to read up on mechanize. Any tips/insight would be huge! Thanks!
-
Compare 2 lists of tuples with same size,:compare and swap
Made 2 list of tuples: I wanna use the alphabet and the counter, for comparing both lists. Tuples of su, belong on the index of tuples in list tu. -> tuple 0 on tu has (40, 'b', 1) -> 'b', 1 in tuple 4 in su are the same, therefore tuple 4 of su should go to index 0, usw.
su = [(30, 'a', 1), (1, 'b', 0), (4, 'a', 0), (17, 'c', 0), (8, 'b', 1)] tu = [(40, 'b', 1), (9, 'c', 0), (3, 'b', 0), (11, 'a', 0), (12, 'a', 1)] for i, (s, t) in enumerate(zip(su, tu)): if t[1] == 'H': print(f" 'H' {i}")
My final wanted list su_new = [(8, b, 1), (17, 'c', 0), (1, 'b', 0), (4, 'a', 0), (30, 'a', 1)] For comparing, I filled in both lists, indices. [(8, b), (17, 'c'), (1, 'b'), (4, 'a'), (30, 'a')]
-
C++ declare a tuple multiple type
it possible create a tuple with multiple and different type in runtime?
example of code:
void magical_function(typeList tl){ for(int i = 0; i <tl.clount; i++) //create a magical tuple
Or
std::tuple<dynamic type in runtime> tupleRuntime;
-
Correctly naming variables types
For a project I have to use lists, tuples or even list of tuples (array of vertices for example) and so on. While writing methods, classes I want to specify the type of their variables. For example:
def Example_Method(self, name: str, age: int): return name + str(age)
or
class Example_Class: def __init__(self, cents: int): self.euros: float = cents / 100
With basic types it is all nice and clean, but how does one name list of floats, tuple of strings or even list of lists or list of tuples. Have searched for information, but couldn't find any or don't know how to name it correctly.
-
Issue with installing gwlf-e using pip
I have tried to install gwlf-e using the command: pip install gwlf-e, but receive this error: "Attempted to compile AOT function without the compiler used by numpy.distutils present. Cannot find suitable msvc." (I do have the specified version of numba installed and also installed microsoft visual studio c++ compiler.) In case this is relevant, I do have Microsoft Visual Studio installed on my desktop, including "Python Tools for Visual Studio" but I am not sure specifically which packages / compilers I might need, or if this is even relevant. Could someone please advise? Thanks
numba version :- 0.38.1
-
Cuda dot product of large numbers results in ovefrlow issues
I'm trying to learn CUDA and one of the things I want to make is a dot product kernel. The kernel works ok for dot products that result in low coefficient values in the result matrix. However, for large values I believe that some overflow issues happen and it results in the wrong number.
This is my cuda kernel
@cuda.jit def matmul(A, B, C): """Perform square matrix multiplication of C = A * B""" i, j = cuda.grid(2) if i < C.shape[0] and j < C.shape[1]: tmp = 0. for k in range(A.shape[1]): tmp += A[i, k] * B[k, j] C[i, j] = tmp
Now if I do the following code:
max_error = [] k_max = 10 for k in range(k_max): n = 2**k m = 2**k a = np.arange(n*m).reshape((n,m)) b = np.arange(n*m).reshape((n,m)) d_a = cuda.to_device(a) d_b = cuda.to_device(b) c = np.zeros((a.shape[0], b.shape[1])).astype(np.float64) d_c = cuda.to_device(c) threads_per_block = 32 blocks_per_thread_x = math.ceil((a.shape[0] + threads_per_block - 1)/threads_per_block) blocks_per_thread_y = math.ceil((a.shape[0] + threads_per_block - 1)/threads_per_block) matmul[(blocks_per_thread_x, blocks_per_thread_y), (threads_per_block, threads_per_block)](d_a, d_b, d_c) c = d_c.copy_to_host() max_error.append((c - a.dot(b)).max()) plot(range(k_max), max_error)
Which performs the dot product of two arrays of size (
2**k
,2**k
) using the Cuda kernel and the NumPy method, the error between the two increases after k=8This leads me to believe that it is some sort of overflow issue. Is there any way to fix this?
-
Numba gives a type error when compiling this trivial function
When calling this function I get the following compilation error, but I can't figure out why:
from numba import njit import numpy as np @njit def fubar(x: np.uint64): assert False y = np.uint64(0) y = x y | 1
It seems to think
y
is afloat64
even though it's clearly auint64
, and therefore can't apply a bitwise operation toy
. It doesn't give me a compilation error if I change references touint64
toint64
. It also doesn't give me a compilation error if I remove theassert False
.> fubar(0) TypingError: Failed in nopython mode pipeline (step: nopython frontend) Invalid use of Function(<built-in function or_>) with argument(s) of type(s): (float64, Literal[int](1)) Known signatures: * (bool, bool) -> bool * (int64, int64) -> int64 * (int64, uint64) -> int64 * (uint64, int64) -> int64 * (uint64, uint64) -> uint64 * parameterized In definition 0: All templates rejected with literals. In definition 1: All templates rejected without literals. In definition 2: All templates rejected with literals. In definition 3: All templates rejected without literals. In definition 4: All templates rejected with literals. In definition 5: All templates rejected without literals. This error is usually caused by passing an argument of a type that is unsupported by the named function. [1] During: typing of intrinsic-call at <ipython-input-6-5336d5e671fe> (9) File "<ipython-input-6-5336d5e671fe>", line 9: def fubar(x: np.uint64): <source elided> y = x y | 1 ^
I am using Kaggle's notebooks, which is currently running Python v3.7.6, Numpy v1.18.5, Numba v0.48.0.