How to filter a numpy array of points by another array

2022-05-04 10:23 Simon imported from Stackoverflow

how can I filter a numpy array a, by the elements of a numpy array b so that I get all the points in a that are not in b.

import numpy as np

a = np.array([[1,2],[1,3],[1,4]])
b = np.array([[1,2],[1,3]])
c = np.array([ d for d in a if d not in b])
print(c)

# acutall outcome
# []
# desired outcome
# np.array([[1,4]])```

3 answers

answered 2022-05-04 10:47 oda
This probably will not be the most efficient (though it turns out to be faster than the other approaches presented here for this input -- see below), but one thing you can do is convert a and b to Python lists and then take their set difference:
```
# Method 1
tmp_1 = [tuple(i) for i in a]    # -> [(1, 2), (1, 3), (1, 4)]
tmp_2 = [tuple(i) for i in b]    # -> [(1, 2), (1, 3)]

c = np.array(list(set(tmp_1).difference(tmp_2)))
```
As noted by @EmiOB, this post offers some insights into why [ d for d in a if d not in b ] in your question does not work. Drawing from that post, you can use
```
# Method 2
c = np.array([d for d in a if all(any(d != i) for i in b)])
```
Remarks

The implementation of array_contains(PyArrayObject *self, PyObject *el) (in C) says that calling array_contains(self, el) (in C) is equivalent to
```
(self == el).any()
```
in Python, where self is a pointer to an array and el is a pointer to a Python object.

In other words:
1. if arr is a numpy array and obj is some arbitrary Python object, then
```
obj in arr
```
is the same as
```
(arr == obj).any()
```
1. if arr is a typical Python container such as a list, tuple, dictionary, and so on, then
```
obj in arr
```
is the same as
```
any(obj is _ or obj == _ for _ in arr)
```
(see membership test operations).

All of which is to say, the meaning of obj in arr is different depending on the type of arr.

This explains why the logical comprehension that you proposed [d for d in a if d not in b] does not have the desired effect.

This can be confusing because it is tempting to reason that since a numpy array is a sequence (though not a standard Python one), test membership semantics should be the same. This is not the case.

Example:
```
a = np.array([[1,2],[1,3],[1,4]])
print((a == [1,2]).any())          # same as [1, 2] in a
# outputs True
```
Timings

For your input, I found my approach to be the fastest, followed by Method 2 obtained from the post @EmiOB suggested, followed by @DanielF's approach. I would not be surprised if changing the input size changes the ordering of the timings so take them with a grain of salt.
```
# Method 1
5.96 µs ± 8.92 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
# Method 2
6.45 µs ± 27.5 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
# @DanielF's answer
16.5 µs ± 276 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
```
answered 2022-05-04 10:59 Abhyuday Vaish
Use This:
```
c = np.array([a_elem for a_elem in a if all(any(a_elem != b_elem) for b_elem in b)])
```
Output:
```
array([[1, 4]])
```
Explanation:

We loop for a sublist a_elem from a and check for all sublists from b. any(a_elem != b_elem) returns True if any value from a_elem is not equal to b_elem. all(any(a_elem != b_elem) for b_elem in b) returns True if all sublists are unequal.

Eg:

We take [1,2] from a check if any of its elements are unequal to [1,2], [1,3] from b one by one. So, it'll be False for [1,2] and True for [1,3]. This creates a list [False, True]

Next, we take [1,3] from a. It'll return True for [1,2] and False for [1,3]. This creates another list [True, False].

Lastly, we take [1,4] from a. It'll return True for both [1,2] and [1,3]. This creates a list [True, True]

Now, when we run all() it returns True when both values are True in the above lists. Hence, we add [1,4] to our array.
answered 2022-05-04 11:28 Daniel F
When comparing row-wise like this I tend to use @Jaime's recipe for converting to a void view here :
```
vview = lambda a:np.ascontiguousarray(a).view(np.dtype((np.void, a.dtype.itemsize * a.shape[1])))

a[~np.isin(vview(a), vview(b)).squeeze()]
Out[]: array([[1, 4]])
```
This avoids the slow for loops of the other answers and doesn't create any intermediate data structures.

See also questions close to this topic

Python File Tagging System does not retrieve nested dictionaries in dictionary

I am building a file tagging system using Python. The idea is simple. Given a directory of files (and files within subdirectories), I want to filter them out using a filter input and tag those files with a word or a phrase.

If I got the following contents in my current directory:

data/
   budget.xls
   world_building_budget.txt
a.txt
b.exe
hello_world.dat
world_builder.spec

and I execute the following command in the shell:

py -3 tag_tool.py -filter=world -tag="World-Building Tool"

My output will be:

These files were tagged with "World-Building Tool":
data/ world_building_budget.txt
hello_world.dat
world_builder.spec

My current output isn't exactly like this but basically, I am converting all files and files within subdirectories into a single dictionary like this:

def fs_tree_to_dict(path_):
    file_token = ''
    for root, dirs, files in os.walk(path_):
        tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
        tree.update({f: file_token for f in files})
        return tree

Right now, my dictionary looks like this: key:''.

In the following function, I am turning the empty values '' into empty lists (to hold my tags):

def empty_str_to_list(d):
    for k,v in d.items():
        if v == '':
            d[k] = []
        elif isinstance(v, dict):
            empty_str_to_list(v)

When I run my entire code, this is my output:

hello_world.dat ['World-Building Tool']
world_builder.spec ['World-Building Tool']

But it does not see data/world_building_budget.txt. This is the full dictionary:

{'data': {'world_building_budget.txt': []},  
'a.txt': [],
'hello_world.dat': [], 
'b.exe': [], 
'world_builder.spec': []}

This is my full code:

import os, argparse


def fs_tree_to_dict(path_):
    file_token = ''
    for root, dirs, files in os.walk(path_):
        tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
        tree.update({f: file_token for f in files})
        return tree


def empty_str_to_list(d):
    for k, v in d.items():
        if v == '':
            d[k] = []
        elif isinstance(v, dict):
            empty_str_to_list(v)


parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter)

parser.add_argument("--filter", action="store", help="keyword to filter files")
parser.add_argument("--tag", action="store", help="a tag phrase to attach to a file")
parser.add_argument("--get_tagged", action="store", help="retrieve files matching an existing tag")

args = parser.parse_args()

filter = args.filter
tag = args.tag
get_tagged = args.get_tagged
current_dir = os.getcwd()

files_dict = fs_tree_to_dict(current_dir)
empty_str_to_list(files_dict)

for k, v in files_dict.items():
    if filter in k:
        if v == []:
            v.append(tag)
            print(k, v)

        elif isinstance(v, dict):
            empty_str_to_list(v)

    if get_tagged in v:
        print(k, v)

Actaully i am working on a project and in it, it is showing no module name pip_internal plz help me for the same. I am using pycharm(conda interpreter

 File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\Scripts\pip.exe\__main__.py", line 4, in <module>
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\__init__.py", line 4, in <module>
    from pip_internal.utils import _log

I am using pycharm with conda interpreter.

Looping the function if the input is not string

I'm new to python (first of all) I have a homework to do a function about checking if an item exists in a dictionary or not.

inventory = {"apple" : 50, "orange" : 50, "pineapple" : 70, "strawberry" : 30}

def check_item():
    x = input("Enter the fruit's name: ")
    if not x.isalpha():
        print("Error! You need to type the name of the fruit")
    elif x in inventory:
        print("Fruit found:", x)
        print("Inventory available:", inventory[x],"KG")
    else:
        print("Fruit not found")
check_item()

I want the function to loop again only if the input written is not string. I've tried to type return Under print("Error! You need to type the name of the fruit") but didn't work. Help

How to get pass an array through a function with a user input?

I want to ask the user for the size of a 2D array arr[][], but also pass it through the function initializeArray. However, if I pass it through the function, I would have to have a size declarator for col, which doesn't allow the user to enter their own value for the size

  #include<iostream>
  using namespace std;

  void initializeArray(arr[][10], int N);

  int main() {
      int N;
      cout << "enter an array size: ";
      cin >> N;
      int arr[N][N];

      initializeArray(arr, N); // I get an error here
      for(int i = 0; i < N; i++) {
          for(int j = 0; j < N; j++)
              cout << arr[i][j] << " ";
          cout << endl;
      }
 }

  void initializeArray(int arr[][10], int N) {
      for(int i = 0; i < N; i++)
          for(int j = 0; j < N; j++)
              arr[i][j] = 0;
  }

The only solution I found was the make arr[][] a global array, but in that case, I would have to still declare the size parameters, and I want the user to enter whatever they want. Is there another way to fix this?

Two-dimensional array C++

Problem: Write a program of a two dimensional integer array of exactly four rows and four columns, and find the sum of the integers each row of the array as well as determine the smallest inputted element/data in the array. The user will input the elements. Display the elements in tabular form. Make the program user friendly. Use a Class and a member function for the process involve.

Pass Associative Array from Symfony Controller to Twig and JavaScript

So, I have 2 arrays that I've combined into one array after making sure they're the same length as such.

//Fetching Ratings for All Coaches & Making a Key, Value Associative Array with Coach id as Key and Rating as Value
        $coaches = $coachRepository->findAll();
        $rating = $coachRepository->findRatingByCoach();
        //Array of coach IDs
        $id_array = array();
        foreach ($coaches as $c){
            $id_array[] = $c->getId();
        }
        $combined = array_combine($id_array, $rating);

Now I want to pass this key, value array to Twig template and JavaScript. I'm currently doing this

    return $this->render('...', [
        ..., 'combined'=>$combined
    ]);

And I'm accessing the array in Twig as such

            {% for key, value in combined %}
                {{ key }} - {{ value }}
            {% endfor %}

And I'm getting this error.

Object of class App\Entity\Coach could not be converted to string

Looking at the Stack Trace I can see that the array combined is being passed in this form.

'combined' => array(object(Coach))

After a bit of debugging it turns out that I thought the issue was with the $id_array when in fact it's the $rating variable that's presenting an issue. I'm not sure why but I'm getting a single float value. I fixed the issue by changing how I'm fetching the $rating, now instead of using a QueryBuilder to get it from the DB, I'm getting it from list of Coaches as such

 //Array of coach IDs and array of Ratings
        $id_array = array();
        $rating_array = array();
        foreach ($coaches as $c){
            $id_array[] = $c->getId();
            $rating_array [] = $c->getRating();
        }

Even if I set the array

TypeError: 'float' object cannot be interpreted as an integer on linspace

   TypeError                                 Traceback (most recent call last)
d:\website\SpeechProcessForMachineLearning-master\SpeechProcessForMachineLearning-master\speech_process.ipynb Cell 15' in <cell line: 1>()

     -->1 plot_freq(signal, sample_rate)
d:\website\SpeechProcessForMachineLearning-master\SpeechProcessForMachineLearning-master\speech_process.ipynb Cell 10' in plot_freq(signal, sample_rate, fft_size)

        2 def plot_freq(signal, sample_rate, fft_size=512):
          3     xf = np.fft.rfft(signal, fft_size) / fft_size
    ----> 4     freq = np.linspace(0, sample_rate/2, fft_size/2 + 1)
          5     xfp = 20 * np.log10(np.clip(np.abs(xf), 1e-20, 1e100))
          6     plt.figure(figsize=(20, 5))

File <__array_function__ internals>:5, in linspace(*args, **kwargs)

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\numpy\core\function_base.py:120, in linspace(start, stop, num, endpoint, retstep, dtype, axis)

     23 @array_function_dispatch(_linspace_dispatcher)
         24 def linspace(start, stop, num=50, endpoint=True, retstep=False, dtype=None,
         25              axis=0):
         26     """
         27     Return evenly spaced numbers over a specified interval.
         28 
       (...)
        118 
        119     """
    --> 120     num = operator.index(num)
        121     if num < 0:
        122         raise ValueError("Number of samples, %s, must be non-negative." % num)
    
    TypeError: 'float' object cannot be interpreted as an integer

What solution about this problem?

How to convert a array of string to array of float using
I have a array as shown below, in that I am having a variable called f, I need to assign the some value for this variable f say 2 and convert this into a floating point array.
```
(['0.', '0.', '-6.190649155150273*^15 + 0.7747634892904517*f^2',
       '2.2098855503598858*^10 + 4.250697125128597*^-7*f^2', '0.', '0.',
       '0.', 0.0, 0.0, -1427.5184297531378], dtype=object)
```
I tried with df.convert_objects(convert_numeric=True) but it is not working
Pandas 1.4.2 upgrade casuing FFlask: array(0.78015261) (0d array) is not JSON serializable at the moment
We upgraded to a newer version of python with new pandas, numpy, etc. The versions are now:
```
- python=3.10.4
- pandas=1.4.2
- numpy=1.22.3
```
In previous versions, the error never occurred. When I do some debugging, it's not that the .to_json() is wrong or fails, it's that the pd.DataFrame([my_result]) doesn't return correctly.

The code is this:
```
// Used in debugging and doesn't return properly
dataframe_test = pd.DataFrame([my_result])

// Error gets thrown here
return pd.DataFrame([my_result]).to_json()
```
The dataframe_test looks like this when I go to view it in Data Viewer within VS Code (it also keeps processing as the data seems to still be running in VS Code - the bar above the viewer is indicating that its still trying to run/process):

The my_result variable looks like this upon entering the dataframe:

I am not sure what exactly is causing the issue and not sure how to debug in Pandas to see what is happening. Any ideas?

Related questions

How to filter a numpy array of points by another array

3 answers

See also questions close to this topic