Remove duplication based on date conditions

See also questions close to this topic

• Sorting column for day of week?

  I have df as below:

        day     value
      Friday      32
      Friday      32
      Monday      11
      Monday      22
      Saturday    44
      Saturday    25
      Sunday      77
      Sunday      88
      Thursday    88
      Thursday    88
      Tuesday     88
      Tuesday     88
      Wednesday   88
      Wednesday   88
  

  How can I sort this in order of correct weekday?

  I tried:

  weekdays = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
  
  df.Weekday = pd.Categorical(df.day,categories=weekdays)
  
  df = df.sort_values('day')
  

  it runs but does not change output,

  thanks
• How accelerate an algorithm to find the time overlaps?

  I have a dataset including 4 attributes :

  • an id attribute called "id" (integer)

  • an attribute called "group" (integer)

  • an admission time called "aankomstdt" (datetime)

  • a discharge time called "ontslagdt" (datetime)

  The dataset looks like that :

  id    group          aankomstdt                   ontslagdt
   1   A         Thu Nov 14 04:31:00 CET 2019 Thu Nov 14 09:43:00 CET 2019
   2   A         Thu Nov 14 05:38:00 CET 2019 Thu Nov 14 06:19:00 CET 2019
   3   A         Thu Nov 14 05:52:00 CET 2019 Thu Nov 14 09:14:00 CET 2019
   4   A         Thu Nov 14 05:54:00 CET 2019 Thu Nov 14 10:02:00 CET 2019
   5   B         Thu Nov 14 06:06:00 CET 2019 Thu Nov 14 11:22:00 CET 2019
  

  I would like to calculate the number of examples that are simultaneously admitted. As such a new attribute should provide me the number of patients admitted for each patient (id) during his/her admission time frame. I created the following working code :

  import pandas as pd
  import numpy as np
  from datetime import datetime
  
  
  
  admission_time = "aankomstdt"
  discharge_time = "ontslagdt"
  group = 'group'
  date_format = '%Y-%m-%d %H:%M:%S'
  path = 'D:/Lionel/Formations_DataScience/Rapidminer/Tests_Rapidminer/count_overlaps_sven.xlsx'
  
  
  def convert_to_datetime(a) :
  
    a = datetime.strptime(str(a), date_format)
    return a
  
  
  def interval_overlaps(a, b):
    return min(a[discharge_time], b[discharge_time]) - max(a[admission_time], b[admission_time]) > np.timedelta64(-1)
  
  
  def count_overlaps(df1):
    return pd.Series([df1.apply(lambda x: interval_overlaps(x, df1.iloc[i]), axis=1).sum() - 1 for i in range(len(df1))], df1.index)
    #return pd.Series([df1.apply(lambda x: interval_overlaps(x, df1.iloc[i]), axis=1).sum() - 1 for i in range(len(df1))])
  def rm_main():
  
    data = pd.read_excel(path)
    data[admission_time] = data[admission_time].apply(convert_to_datetime)
    data[discharge_time] = data[discharge_time].apply(convert_to_datetime)
    data["count"] = data.groupby(group).apply(count_overlaps).values
    return data
  

  But my original dataset has around 70 000 examples, so I estimate that computation time is around 1 month (24h/24). My question is : Is there a solution to significantly accelerate this algorithm ?

  Thanks you,
• Changing the operation of the turtle graphics

  Firstly, I am very new to programming and biting off more than I can chew here. Anyway....

  I am trying to build a solar system/orbiting simulation in python using this code as a model:

  https://fiftyexamples.readthedocs.io/en/latest/gravity.html

  This is my full code:

  # Import math and turtle
  import math
  from turtle import *
  import turtle
  # Gravitational Constant
  G = 6.67428e-11
  # Scale: 1 pixel = 1 astronomical unit
  # 1 astronomical unit = 1 AU = 149597900 km
  AU = (149.6e6 * 1000)  # 149.6 million km in meters
  Scale = 250 / AU
  wn = turtle.Screen()
  wn = turtle.bgcolor('black')
  
  
  class body(Turtle):
  
      # Subclass of turtle representing a body
      # Additional attributes:
      # mass in kg
      # vx, vy: x, y velocities in m/sc
      # px, py: x, y positions in m
      # Set background to black
      # Turtle.bgcolor('black')
      name = 'body'
      mass = None
      vx = vy = 0.0
      px = py = 0.0
  
      def attraction(self, other):
  
          # (body): (fx, fy)
          # returns the force exerted on this body by the other body.
          # Report and error if the other object is the same as this one.
  
          if self is other:
              raise ValueError("Attraction of object %r to itself requested" % self.name)
  
          # Compute the distance of the other body:
  
          sx, sy = self.px, self.py
          ox, oy = other.px, other.py
          dx = (ox - sx)
          dy = (oy - sy)
          d = math.sqrt(dx**2 + dy**2)
  
          # Report an Error if the distance is 0
          # ZeroDivisionError would happen later otherwise
          if d == 0:
              raise ValueError("Collision between objects %r and %r" % (self.name, other.name))
  
          # Compute the force of attraction
  
          f = G * self.mass * other.mass / (d**2)
  
          # Compute the direction of the force
          theta = math.atan2(dy, dx)
          fx = math.cos(theta) * f
          fy = math.sin(theta) * f
          return fx, fy
  
  
  def updateInfo(step, bodies):
  
      # (int, [body])
      # Displays infomation about status of simulation.
  
      print('Step #{}'.format(step))
      for body in bodies:
          s = '{:<8} Pos.={:>6.2f} Vel.={:>10.3f}'.format(body.name, body.px/AU, body.py/AU, body.vy)
          print(s)
      print()
  
  
  def loop(bodies):
      # (body)
      # Never returns; loop through the simulation and updates the positions of given bodies
  
      timestep = 24*3600  # One Day
  
      for body in bodies:
          body.penup()
          body.hideturtle()
  
      step = 1
      while True:
          updateInfo(step, bodies)
          step += 1
  
          force = {}
          for body in bodies:
  
              # Adds all the forces exerted on a body
              totalFx = totalFy = 0.0
              for other in bodies:
                  # Don't calculate attraction to itself
                  if body is other:
                      continue
                  fx, fy = body.attraction(other)
                  totalFx += fx
                  totalFy += fy
  
              # Record the total force exerted
              force[body] = (totalFx, totalFy)
  
          # Update velocities based on the force.
          for body in bodies:
              fx, fy = force[body]
              body.vx += fx / body.mass * timestep
              body.vy += fy / body.mass * timestep
  
              # Update positions
              body.px += body.vx * timestep
              body.py += body.vy * timestep
              body.goto(body.px*Scale, body.py*Scale)
              body.dot(5)
  
  
  def main():
      sun = body()
      sun.name = 'Sun'
      sun.mass = 1.98892 * 10**30
      sun.pencolor('yellow')
  
      earth = body()
      earth.name = 'Earth'
      earth.mass = 5.9742 * 10**24
      earth.px = -1*AU
      earth.vy = 29.783 * 1000
      earth.pencolor('blue')
  
      venus = body()
      venus.name = 'Venus'
      venus.mass = 4.8685 * 10**24
      venus.px = 0.723 * AU
      venus.vy = -35.02 * 1000
      venus.pencolor('orange')
  
      loop([sun, earth, venus])
  
  
  if __name__ == '__main__':
      main()
  

  I have added an import turtle line for my window as I want to change the window color to black. I know the the major differences between the two import styles and why some people prefer one over the other (or why one is not good). I want to change the structure of the class so I can have more control over the graphics of the simulation and also move away from the from x import * method. Any advice will be welcome!
• How to use regex capture groups in pandas replace function

  I've got a pandas DataFrame that uses "2Nd" instead of "2nd", "136Th" instead of "136th", etc. I want the letter immediately following the number to be lowercase.

  Sample data:

  data = pd.Series(['21St StNew York', 'Exampe BlvdSt Louis', '1St Rd'])
  

  Desired output:

  ['21st StNew York', 'Exampe BlvdSt Louis', '1st Rd']
  

  Tried using str.replace():

  data = data.str.replace('\BSt', 'st', regex=True)
  ['21st StNew York', 'Exampe Blvdst Louis', '1st Rd']
  

  Is it possible to use a capture group?

  data = data.str.replace('[0-9]+(St)', 'st', regex=True)
  ['st StNew York', 'Exampe BlvdSt Louis', 'st Rd']
  
• Pandas After merging the same values, the column name becomes 0

  I have the following data structure.

  

  +--------------------------+----------------------+
  ¦            test1         ¦          test2       ¦
  ¦--------------------------¦----------------------+
  ¦ https: // test.com / 123 ¦           st1        ¦
  ¦ https: // test.com / 123 ¦           st2        ¦
  ¦ https: // test.com / 1234¦           st3        ¦
  ¦ https: // test.com / 1234¦           st4        ¦
  +----------------------+--------------------------+
  

  I want to merge the values of test2 based on the same test1 column values

  I tried the following code

  import pandas as pd
  
  test = 'test.xlsx'
  df1 = pd.read_excel(test)
  
  df_isnull_have_keywords = df1.groupby(by='test1').apply(
      lambda x: [','.join('%s' % key for key in x['test2'])])
  
  df_isnull_have_keywords.to_excel('test.xlsx')
  

  But in the output, the test2 column is 0

  

  I don't know，please help me