Generating multiple coordinates from a graph in .jpeg format
I need to generate random(say 3) coordinates from each line in a graph. I don't know what to start with. I don't even have the experimental data using which the graph was plotted. Let say I have this image (my graph is way more complex though). Now, I want to generate 3 coordinates (x,y) from each line (y1,y2 and toto) that satisfies the line. example: y1 (2,0),(4,2),(5,0) y2 (4,5),(6,8),(10,13) toto  (4,14),(5,15),(9,18) What would be the best way to do this?
See also questions close to this topic

Why is image saved by PIL have different md5 sum compared with the original image
assume I have a image
pic.jpg
. I read the image and then save it.from PIL import Image im = Image.open('pic.jpg') im.save('pic1.jpg', 'jpeg')
The md5 sum of the two pictures are different:
$ md5sum pic.jpg 3191102e44fa5ebbb2aa52e042066dad $ md5sum pic1.jpg a6b17e3af3ff66715a2326db33548d11
Do I still have the original image if I read and then save with PIL?

Adding muliple column values from one csv to the same rows of another
So I'm trying to combine column values from one csv to another while saving it into a final csv file. But I want to iterate through all the rows adding the column values of each row to each row of the original csv.
In other words say csv1 has 3 rows.Row 1: Frog,Rat,Duck,, Row 2: Cat,Dog,Cow,, Row 3: Moose,Fox,Zebra,,
And I want to combine 2 more column values from csv2 to each of those rows.
Row 1: Chicken,Pig Row 2: Squirrel,Giraffe Row 3: Bear,Boar
So csv3 would end up looking like.
Row 1: Frog,Rat,Duck,Chicken,Pig Row 2: Cat,Dog,Cow,Squirrel,Giraffe Row 3: Moose,Fox,Zebra,Bear,Boar
But at the same time if there's a row in csv2 that has no values at all I don't want it to copy the row from csv1. In other words that row will not exist at all in the final csv file. I prefer not to use pandas as I have just been using the csv module thus far throughout my code but any method is appreciated.
So far I have come across this method which works if there's only one single row. But when there's more than that it just adds random lines and appends the values all over the place. And it combines both of the columns into one string while adding an extra blank line at the end of the csv for some odd reason.
import csv f1 = open ("2.csv","r", encoding='utf8') with open("3.csv","w", encoding='utf8', newline='') as f: writer = csv.writer(f) with open("1.csv","r", encoding='utf8') as csvfile: reader = csv.reader(csvfile, delimiter=",") for row in reader: row[6] = f1.readline() writer.writerow(row) f1.close()

How to convert cnn block of code in Matlab to python?
Hi i have a block of code which has binary weights. I would like to know what is the exact translation of the below mentioned matlab code in python. I am very new to Matlab.
%convert the weights to 1 and + 1. Store in single because conv2 requires it
for i = 1:length(Filters) Filters{i} = gpuArray(2*(single(sign(Filters{i}))0.5)); NumLearntWeightsEachLayer(i) = size(Filters{i},3)*size(Filters{i},4)*4; end NumLearntWeightsEachLayer(end) = size(Filters{end},3)*size(Filters{end},4); NumLearntWeightsEachLayer TotalLearntWeights = sum(NumLearntWeightsEachLayer)

python gradient descent error when plotting map()
I tried to run the code below,
import math import matplotlib.pyplot as plt from functools import partial def difference_quotient(f,x,h): return(f(x+h)f(x))/h def square(x): return x*x def derivative(x): return 2*x derivative_estimate = partial(difference_quotient,square,h=0.0001) x = range(10,10) y = range(10,10) plt.title("actual vs estimation") plt.plot(x,map(derivative,x),'rx',label="actual") plt.plot(x,map(derivative_estimate,x),'b+',label="estimate") plt.show() print(len(list(map(derivative,x))))
but it shows error below
Traceback (most recent call last): File "C:\Program Files\Python37\lib\sitepackages\matplotlib\units.py", line 168, in get_converter if not np.all(xravel.mask): AttributeError: 'numpy.ndarray' object has no attribute 'mask'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "C:\Users\asus\Documents\Sublime\dataScience\gradient.py", line 20, in plt.plot(x,map(derivative,x),'rx',label="actual") File "C:\Program Files\Python37\lib\sitepackages\matplotlib\pyplot.py", line 2811, in plot is not None else {}), **kwargs) File "C:\Program Files\Python37\lib\sitepackages\matplotlib__init__.py", line 1810, in inner return func(ax, *args, **kwargs) File "C:\Program Files\Python37\lib\sitepackages\matplotlib\axes_axes.py", line 1611, in plot for line in self._get_lines(*args, **kwargs): File "C:\Program Files\Python37\lib\sitepackages\matplotlib\axes_base.py", line 393, in _grab_next_args yield from self._plot_args(this, kwargs) File "C:\Program Files\Python37\lib\sitepackages\matplotlib\axes_base.py", line 370, in _plot_args x, y = self._xy_from_xy(x, y) File "C:\Program Files\Python37\lib\sitepackages\matplotlib\axes_base.py", line 205, in _xy_from_xy by = self.axes.yaxis.update_units(y) File "C:\Program Files\Python37\lib\sitepackages\matplotlib\axis.py", line 1467, in update_units converter = munits.registry.get_converter(data) File "C:\Program Files\Python37\lib\sitepackages\matplotlib\units.py", line 181, in get_converter converter = self.get_converter(next_item) File "C:\Program Files\Python37\lib\sitepackages\matplotlib\units.py", line 187, in get_converter thisx = safe_first_element(x) File "C:\Program Files\Python37\lib\sitepackages\matplotlib\cbook__init__.py", line 1635, in safe_first_element raise RuntimeError("matplotlib does not support generators " RuntimeError: matplotlib does not support generators as input [Finished in 0.7s]
my suspect was on these line,
plt.plot(x,map(derivative,x),'rx',label="actual") plt.plot(x,map(derivative_estimate,x),'b+',label="estimate")
when I tried to change the map(derivative,x) and map(derivative_estimate,x) with y which is range(10,10), it works.
what should I do so the code can show the plot when I use the map function like above?

How to combine 2 dataframe histograms in 1 plot?
I would like to use a code that shows all histograms in a dataframe. That will be
df.hist(bins=10)
. However, I would like to add another histograms which shows CDFdf_hist=df.hist(cumulative=True,bins=100,density=1,histtype="step")
I tried separating their matplotlib axes by using
fig=plt.figure()
andplt.subplot(211)
. But this df.hist is actually part of pandas function, not matplotlib function. I also tried setting axes and adding ax=ax1 and ax2 options to each histogram but it didn't work.How can I combine these histograms together? Any help?
Histograms that I want to combine are like these. I want to show them side by side or put the second one on tip of the first one. Sorry that I didn't care to make them look good.

Plotting if data is is available at any one time for each station, single plot
As the title suggests, i would like to plot data availability, at any one time for each station. The plot can be thought to be a map or scatter, where the station number and time are the coordinates, which will plot vertical lines, instead of points. Similar to the plot at the end of the post. Which is from the output of an R package, 'Climatol' (homogen function).
I would like to know if there is similar way of plotting in PYTHON, I preferably don't want to use the R package, as it does more than just the plot, and hence will take a lot of hours for thousands of station data.
Some sample data (daily time series) of each stations would be like ;
station1 = pd.DataFrame(pd.np.random.rand(100, 1)).set_index(pd.date_range(start = '2000/01/01', periods = 100)) station2 = pd.DataFrame(pd.np.random.rand(200, 1)).set_index(pd.date_range(start = '2000/03/01', periods = 200)) station3 = pd.DataFrame(pd.np.random.rand(300, 1)).set_index(pd.date_range(start = '2000/06/01', periods = 300)) station4 = pd.DataFrame(pd.np.random.rand(50, 1)).set_index(pd.date_range(start = '2000/09/01', periods = 50)) station5 = pd.DataFrame(pd.np.random.rand(340, 1)).set_index(pd.date_range(start = '2000/01/01', periods = 340))
Real sample data; https://drive.google.com/drive/folders/15PwpWIh13tyOyzFUTiE9LgrxUMm9gh6?usp=sharing Code to open for two stations;
import pandas as pd import numpy as np df1 = pd.read_csv('wgenf  20190417T012724.318.genform1_proc',skiprows = 8,delimiter = ' ') df1.drop(df1.tail(6).index,inplace=True) df1 = df1.iloc[:,[1,3]] df1.iloc[:,1].replace('',np.nan,inplace=True) df1 = df1.dropna() df1['Date(NZST)'] = pd.to_datetime(df1.iloc[:,0],format = "%Y %m %d") df1 = df1.set_index('Date(NZST)') df2 = pd.read_csv('wgenf  20190417T012830.116.genform1_proc',skiprows = 8,delimiter = ' ') df2.drop(df2.tail(6).index,inplace=True) df2 = df2.iloc[:,[1,3]] df2.iloc[:,1].replace('',np.nan,inplace=True) df2 = df2.dropna() df2['Date(NZST)'] = pd.to_datetime(df2.iloc[:,0],format = "%Y %m %d") df2 = df2.set_index('Date(NZST)')
(note the missing dates, or Nans, would not be plotted, otherwise a vertical line would be created for that specific station)

Python  How to apply "boundingbox normalization and centering" algorithm for an image matrix?
I'm developing a handwritten digit recognizer with a Devanagari Character Dataset from Kaggle. I managed to scan all the images and train my model in Tensorflow. Now i'm not able to convert my own handwritten digits drawn in Tkinter canvas to MNIST format
28x28
image. This is the info about the png image that is loaded from canvas.Shape : (264, 264) [[255 255 255 ... 255 255 255] [255 255 255 ... 255 255 255] [255 255 255 ... 255 255 255] ... [255 255 255 ... 255 255 255] [255 255 255 ... 255 255 255] [255 255 255 ... 255 255 255]]
Now i want to be able to convert the 264x264 image matrix to 28x28 by applying "boundingbox normalization and centering" algorithm. I referred Normalization and centering of images in MNIST which i didn't quite understand how it works. And also i referred convertimagestomnistformat.py file which is focused on a lot of images. So it would be a great help if there is an explanation on how the algorithm works and how it can be used to normalize and reduce one pixel array of size
264x264
or around. I'm using Python 3.x. 
Unable to get contrast of image as described by its formula
I was trying to get the contrast of an image using a formula but the contrast value is not exceeding 255. As well as whenever I tried to make some operation on my image matrix the element values also not exceeding 255. I tried converting the image matrix to double but the element values changed and are not equal to original pixel values.
clc; clear all; close all; h = imread('C:\Users\LAXMIDHAR\Desktop\My proj files\abc.jpg'); g = rgb2gray(h); % f = im2double(g); [M,N] = size(g); % % for i=1:M % for j=1:N % f(i,j) = f(i,j).*((ij).^2); % end % end % % s = sum(sum(f)); s = 0; for i = 1:M for j=1:N s = s+(g(i,j).*((ij).^2)); end end % s is the contrast of image
s
is expected to be large but its not exceeding 255. This is the contrast formula: 
Sepia Effect in IOS
I am developing an application in Android which has to apply Sepia effect on the uploaded image. Application is already. I came across a function in swift code which is applying Sepia effect.
func applySepia() > UIImage? { filterValue = 1 let image = self.processPixels(filterValue: filterValue) guard let cgimg = image?.cgImage else { print("imageView doesn't have an image!") return self } let value = filterValue print("sliderValue = \(filterValue)") print("value = \(value)") let openGLContext = EAGLContext(api: .openGLES2) let context = CIContext(eaglContext: openGLContext!) let coreImage = CIImage(cgImage: cgimg) let filter = CIFilter(name: "CISepiaTone") filter?.setValue(coreImage, forKey: kCIInputImageKey) filter?.setValue(value, forKey: kCIInputIntensityKey) if let output = filter?.value(forKey: kCIOutputImageKey) as? CIImage { let cgimgresult = context.createCGImage(output, from: output.extent) let image = UIImage(cgImage: cgimgresult!) return image.applySharpness(filterValue:filterValue) } return self }
Above function is using "CISepiaTone" filter to implement Sepia tone. Here "kCIInputIntensityKey" is passed as 1 which I am unable to understand. As per the documentation, it's value ranges between 0 to 1 but how it is allowing a negative value. Due to this intensity value, the generated image does look like this:
In my opinion, after applying Sepia it should look like:
I am able to achieve the second image in android (which is truly Sepia Tone) using https://github.com/StevenRudenko/ColorMartix/blob/master/src/com/sample/colormatrix/Main.java
However, I couldn't find any builtin method or class in android which can be used to implement Sepia Tone with negative intensity just like the way it is applied in IOS swift. Here are my questions:
 How IOS is allowing negative value for kCIInputIntensityKey despite the fact that it should be ranged between 01.
 After negative intensity values, Generated images does not look like a Sepia Tone.
 How can I achieve the same effect in Android?

How to check if any point from a list of points is contained by any polygon from a list of polygons?
I have the following problem: I have a list of shapely points and a list of shapely polygons. Now I want to check in which polygon a given point is.
At the moment I am using the following code, which seems not very clever:
# polygons_df is a pandas dataframe that contains the geometry of the polygons and the usage of the polygons (landuses in this case, e.g. residential) # point_df is a pandas dataframe that contains the geometry of the points and the usage of the point (landuses in this case, e.g. residential) # polylist is my list of shapely polygons # pointlist is my list of shapely points from shapely.geometry import Point, Polygon import pandas as pd import geopandas as gpd i = 0 while i < len(polygons_df.index): j = 0 while j < len(point_df.index): if polylist[i].contains(point): point.at[j, 'tags.landuse'] = polygons_df.iloc[i]['tags.landuse'] else: pass j += 1 i += 1
Can I somehow speed this up? I have more than 100.000 points and more than 10.000 polygons and these loops take a while. Thanks!

OSMNX  visualizing waterway together with footprints and ground
I am trying to plot waterway together with ground (street network/map) and footprints (buildings). My problem is that the waterways are large closed polygon areas, which means instead of plotting a nice blue river with correct thickness it plots the river all over half of the figure (and there's no water there IRL...). This is what I have in JupyterLab so far (example coordinates with few roads for quick test):
%matplotlib inline import matplotlib.pyplot as plt import osmnx as ox ox.config(log_file=True, log_console=True, use_cache=True) import numpy as np map_settings = dict( dist=805, edge_color='k', bgcolor='w', dpi = 300, point = (45.27386, 25.04644), default_width=2, ) fig, ax = ox.plot_figure_ground(network_type='all', **map_settings, ) gdf_bldings = ox.footprints.footprints_from_point(point=map_settings['point'], distance=map_settings['dist'], footprint_type='building', ) fig, ax = ox.footprints.plot_footprints(gdf_bldings, fig=fig, ax=ax, set_bounds=False, save=False, show=True, close=False, dpi=map_settings['dpi'], ) gdf_water = ox.footprints.footprints_from_point(point=map_settings['point'], distance=map_settings['dist'], footprint_type='waterway', ) fig, ax = ox.footprints.plot_footprints(gdf_water, fig=fig, ax=ax, color='b', set_bounds=False, save=False, show=True, close=False, dpi=map_settings['dpi'], ) coslat = np.cos(np.cos(map_settings['point'][1] / 180. * np.pi) ) ax.set_aspect(1/coslat) fig.set_figwidth(10) fig
and it gives the following figure.
I tried other places, and it does the same there. Is there any way to fix this to have a nice river as expected? I tried to iterate through the shapely objects and figure out if I could remove the coordinates but it gets ugly pretty fast.

Python shapely: Aggregating points to shape files for a Choropleth map
I'm trying a create a Choropleth in Python3 using shapely, fiona & bokeh for display.
I have a file with about 7000 lines that have the location of a town and a counter.
Example:
54.7604;9.55827;208 54.4004;9.95918;207 53.8434;9.95271;203 53.5979;10.0013;201 53.728;10.2526;197 53.646;10.0403;196 54.3977;10.1054;193 52.4385;9.39217;193 53.815;10.3476;192 ...
I want to show these in a 12,5km grid, for which a shapefile is available on https://opendataesride.opendata.arcgis.com/datasets/3c1f46241cbb4b669e18b002e4893711_0
The code I have works.
It's very slow, because it's a brute force algorithm that checks each of the 7127 grid points against all of the 7000 points.
import pandas as pd import fiona from shapely.geometry import Polygon, Point, MultiPoint, MultiPolygon from shapely.prepared import prep sf = r'c:\Temp\geo_de\Hexagone_125_km\Hexagone_125_km.shp' shp = fiona.open(sf) district_xy = [ [ xy for xy in feat["geometry"]["coordinates"][0]] for feat in shp] district_poly = [ Polygon(xy) for xy in district_xy] # coords to Polygon df_p = pd.read_csv('points_file.csv', sep=';', header=None) df_p.columns = ('lat', 'lon', 'count') map_points = [Point(x,y) for x,y in zip(df_p.lon, df_p.lat)] # Convert Points to Shapely Points all_points = MultiPoint(map_points) # all points def calc_points_per_poly(poly, points, values): # Returns total for poly poly = prep(poly) return sum([v for p, v in zip(points, values) if poly.contains(p)]) # this is the slow part # for each shape this sums um the points sum_hex = [calc_points_per_poly(x, all_points, df_p['count']) for x in district_poly]
Since this is extremly slow, I'm wondering if there is a faster way to get the num_hex value, especially, since the real world list of points may be a lot larger and a smaller grid with more shapes would deliver a better result.