See also questions close to this topic

• Create new Pandas columns using the value from previous row

  I need to create two new Pandas columns using the logic and value from the previous row.

  I have the following data:

  Day Vol    Price    Income   Outgoing
  1   499     75      
  2   3233    90      
  3   1812    70      
  4   2407    97      
  5   3474    82      
  6   1057    53      
  7   2031    68      
  8   304     78      
  9   1339    62      
  10  2847    57      
  11  3767    93      
  12  1096    83      
  13  3899    88      
  14  4090    63      
  15  3249    52      
  16  1478    52      
  17  4926    75      
  18  1209    52      
  19  1982    90      
  20  4499    93  
  

  My challenge is to come up with a logic where both the Income and Outgoing columns (which are currently empty), should have the values of (Vol * Price).

  But, the Income column should carry this value when, the previous day's "Price" value is lower than present. The Outgoing column should carry this value when, the previous day's "Price" value is higher than present. The rest of the Income and Outgoing columns, should just have NaN's. If the Price is unchanged, then that day's value is to be dropped.

  But the entire logic should start with (n + 1) day. The first row should be skipped and the logic should apply from row 2 onwards.

  I have tried using shift in my code example such as:

  if sample_data['Price'].shift(1) < sample_data['Price'].shift(2)):
      sample_data['Income'] = sample_data['Vol'] * sample_data['Price']
  else:
      sample_data['Outgoing'] = sample_data['Vol'] * sample_data['Price']
  

  But it isn't working.

  I feel there would be a simpler and comprehensive tactic to go about this, could someone please help ?

  Update (The final output should look like this):

  

  For day 16, the data is deleted because we have two similar prices for day 15 and 16.
• When I use basemap in python, longitude is sometimes reversed, flipping the map

  In python, I use basemap (https://matplotlib.org/basemap/) for plotting spatial data, and I've used it for several years without any large problems. I recently had to reinstall python3 (through conda, along with a number of modules) and basemap now has a strange issue: under certain conditions, the map will be displayed with flipped longitudes, switching east and west. As an example, I use this code: https://matplotlib.org/basemap/users/robin.html. If I use that code as-is, the map displays fine, but when I set lon_0=180, the map gets flipped, as shown in the image below.

  Image of map problem

  Setting lon_0 to any positive number results in a flipped map, while 0 or negative numbers result in a correct map. lon_0 should simply set the central longitude of the plotted map, and should not have this behavior, so I'm unsure what's going on. Has anyone seen this behavior before, or have suggestions for how to fix it? I could alter my code to work around it, but I'd rather have things work properly.

  I am using python3.7.3. I've tried updating basemap with the command "conda install -c anaconda basemap", but it tells me that basemap is up to date already.

  Here is the code. It is identical to the code linked above, but with lon_0 set to 180.

  from mpl_toolkits.basemap import Basemap
  import numpy as np
  import matplotlib.pyplot as plt
  # lon_0 is central longitude of projection.
  # resolution = 'c' means use crude resolution coastlines.
  m = Basemap(projection='robin',lon_0=180,resolution='c')
  m.drawcoastlines()
  m.fillcontinents(color='coral',lake_color='aqua')
  # draw parallels and meridians.
  m.drawparallels(np.arange(-90.,120.,30.))
  m.drawmeridians(np.arange(0.,360.,60.))
  m.drawmapboundary(fill_color='aqua')
  plt.title("Robinson Projection")
  plt.show()
  

  When I run the code, the only output is this, which seems unrelated:

  map_test.py:36: MatplotlibDeprecationWarning: The dedent function was deprecated in Matplotlib 3.1 and will be removed in 3.3. Use inspect.cleandoc instead. m = Basemap(projection='robin',lon_0=180,resolution='c')

  Any ideas?
• Use format() on variables

  Is it possible to use "{}".format() logic but instead of "", to use a variable?

  x = 1
  y = 2
  a = 'some {} random {} text'
  print(a.format(x, y))
  

  I expect the following output:

  some 1 random 2 text
• How to insert/inject an XML file into an other XML in Python

  For a student project, I need to inject an XML file into an other one using a python script. I don't really know how to do, I tried the xml.etree python module, but I didn't succeed. This is my problem :

  file1.xml

  <project>
  
     ...
  
     <publish>
        <a> Stuff </a>
     </publish>
  
     ...
  
  </project>
  

  file2.xml

  <b> Other stuff </b>
  

  For the result, I want need to inject the file2.xml into the file1.xml at this exact position :

  file1.xml

  <project>
  
     ...
  
     <publish>
        <a> Stuff </a>
        <b> Other stuff </b>  <----- Here
     </publish>
  
     ...
  
  </project>
  

  How can I do ? Thank you for your time
• Convert XML Tittle Attribute Into a Link

  I'm fairly unfamiliar with XML, but I have a corporate application that creates an XML file that updates the GUI based off user inputs. Each input updates an attribute in XML. However, there are no inputs to create a link.

  Is there a way to to transform a title attribute into a working hyperlink or image?

  I have tried inputting: a href="https://www.google.com">Test

  With the output being:

   <row title="&lt;a href=&quot;https://www.google.com&quot;&gt;Test&lt;/a&gt;"></row>
  

  However, no luck on making it work.
• Powershell Search ApplicationHost.config file and replace value

  I have a need to modify the applicationHost.config file located by default at c:\windows\System32\inetsrv\config.

  Reviewing the XML file, there are many entries for location under configuration. I need to modify only a specific one. Perhaps checking if it exists first before modifying it?

  The following XML is what I want to modify. The is nested under the top level configuration. I want to change just one entry, anonymousAuthentication enabled="true" if found to be false.

  <location path="Default Web Site/MyWebsite/SiteA">
      <system.webServer>
          <security>
              <authentication>
                  <digestAuthentication enabled="false" />
                  <basicAuthentication enabled="false" />
                  <anonymousAuthentication enabled="false" />
                  <windowsAuthentication enabled="true">
                      <providers>
                          <clear />
                          <add value="Negotiate" />
                          <add value="NTLM" />
                      </providers>
                  </windowsAuthentication>
              </authentication>
          </security>
      </system.webServer>
  </location>
  

  A side note. I do see that I could use set-webconfigurationproperty. However, when I do so, it tells me the configuration section cannot be used at this path. This happens when the section is locked at a parent level. I'm using PSPath, which is supposed to get around that, but it's not working.
• How to get aggregated sum of columns in parquet files partitioned by some other column delimited by semi colon in spark

  Below input is available in csv file and same converted into parquet file. using the converted parquet file, need to do vertical sum of Values column of same countries separated by semicolon. Here Country and Values are two columns.

  Country Values
  Canada   47;97;33;94;6
  Canada   59;98;24;83;3
  Canada   77;63;93;86;62
  China    86;71;72;23;27
  China    74;69;72;93;7
  China    58;99;90;93;41
  England  40;13;85;75;90
  England  39;13;33;29;14
  England  99;88;57;69;49
  

  Expected Output:

  Country Values
  Canada   183;258;150;263;71
  China    218;239;234;209;75
  England  178;114;175;173;153
  

  ---
• Is there a way to connect to to aws environment to upload the spark output to s3 bucket when run locally using anaconda

  We can do this using Dev Endpoints in aws glue but i was interested to know if it is possible to push the spark computed data(when run locally/standalone mode) to s3 using anaconda by adding spark and hadoop winutils dependences.
• What does columns.tail do for a dataframe

  I want to know what does DF.foreach.columns.tail do where DF is my dataframe. While I am running a condition within DF.foreach.columns.tail (...) it runs the condition just for the last column not the entire. I want the condition to be run for all the columns within that dataframe. Please help
• DATABRICKS SQL - Converting into a specific time format

  My current date have this format: Table name: YYY Column Name: XXX Data input: 2016-04-16T13:22:19.515+0000

  I did a

  DATE_TRUNC ('month', XXX) AS month FROM YYY

  And it returns Data output: 2016-04-01T00:00:00.000+0000

  I'd like to transform this output into something like April/16 or Apr/16

  Something that it would be easier to query in the end.

  How do I do this?

  Data input: 2016-04-16T13:22:19.515+0000

  DATE_TRUNC ('month', XXX) AS month FROM YYY

  Data output: 2016-04-01T00:00:00.000+0000
• DataBricks - save changes back to DataLake (ADLS Gen2)

  I have legacy data stored as CSV in an Azure DataLake Gen2 storage account. I'm able to connect to this and interrogate it using DataBricks. I have a requirement to remove certain records once their retention period expires, or if a GDPR "right to be forgotten" needs applying to the data.

  Using Delta I can load a CSV into a Delta table and use SQL to locate and delete the required rows, but what is the best way to save these changes? Ideally back to the original file, so that the data is removed from the original. I've used the LOCATION option when creating the Delta table to persist the generated Parquet format files to the DataLake but it would be nice to keep it in the original CSV format.

  Any advice appreciated.
• Metabase - How to use custom JDBC driver? (Databricks)

  I'm trying to connect Metabase in my data source that is in Databricks. I tried to use the SparkSQL connector that comes with Metabase but didn't work. Metabase generates a slightly different jdbc connection url that it should be, Metabase generates a url that begins with jdbc:hive2 but Databricks requires a url that starts with jdbc:spark.

  Databricks provide a Simba Spark JDBC Driver to use for connection. I already used this driver to connect to Databricks with other BI tools and IDEs like Datagrip for example and works perfectly, so I think must be a way to configure Metabase to use this "custom" driver jar to connect to Databricks.

  If anyone wants to download the simba driver they provide: http://info.databricks.com/do0hZ6qCUq0Sj00Mjk000Y0

  Someone already did something related to that and can help me? Thanks!