Django REST Framework - POST slowness when body size greater than 1024 characters

See also questions close to this topic

• how to display contents of text file one line at a time via timer using python on windows?

  this is the code.

  def wndProc(hWnd, message, wParam, lParam):
      if message == win32con.WM_PAINT:
          hdc, paintStruct = win32gui.BeginPaint(hWnd)
  
          dpiScale = win32ui.GetDeviceCaps(hdc, win32con.LOGPIXELSX) / 60.0
          fontSize = 36
  
          # http://msdn.microsoft.com/en-us/library/windows/desktop/dd145037(v=vs.85).aspx
          lf = win32gui.LOGFONT()
          lf.lfFaceName = "Times New Roman"
          lf.lfHeight = int(round(dpiScale * fontSize))
          #lf.lfWeight = 150
          # Use nonantialiased to remove the white edges around the text.
          # lf.lfQuality = win32con.NONANTIALIASED_QUALITY
          hf = win32gui.CreateFontIndirect(lf)
          win32gui.SelectObject(hdc, hf)
  
          rect = win32gui.GetClientRect(hWnd)
          # http://msdn.microsoft.com/en-us/library/windows/desktop/dd162498(v=vs.85).aspx
          win32gui.DrawText(
              hdc,
              **'Glory be to the Father, and to the son and to the Holy Spirit.',**
              -1,
              rect,
              win32con.DT_CENTER | win32con.DT_NOCLIP | win32con.DT_VCENTER
          )
          win32gui.EndPaint(hWnd, paintStruct)
          return 0
  

  .where it says the "glory be to the father.." prayer I would like that string to actually display a few different prayers on a timer. what I mean is I want to save short prayers to a text file and have the line where it says "glory be.." to change to a new prayer every 60 seconds cycling through a few prayers such as the serenity prayer etc.
• How to plot the frequency of my data per day in an histogram?

  I want to plot the number occurences of my data per day. y represent the id of my data. x represent the timestamp which I convert to time and day. But I can't make the correct plot. import matplotlib.pyplot as plt plt.style.use('ggplot') import time

  y=['5914cce8-fad6-45d1-bec2-e59e62823617', '1c2067e0-5173-4a1d-8a75-b18267ee4598', 'db6830ff-fa9c-4aa5-b71e-f6da9333f357', '672cc9d5-360e-4451-bb7c-03e3d0bd8f0d', 'fb0f8122-fffc-47fe-a87a-b2b749df173b', '558e96ca-0222-40c7-acc0-e444f7663f53', 'c3f86fd5-eac3-48d3-a44c-b325f30b6139', '21dd849f-895f-4cf5-a168-45a4c1a9fbf9', 'e3b4cd56-e291-4671-93b6-d2226ee82ae7', '01346c48-a8c4-43d1-ac02-1efa33ca0f4e', '23b78b0f-85be-4ca7-99f4-1a5add76c12e', 'b1c036c0-0c2b-4170-a170-8fd0add0dec2', '74737546-e9c3-4126-bcb2-4d34503421ca', '342991f5-ec87-4c9d-83eb-9908f3e221aa', '4fdcd83a-eb68-4e26-b79b-753c5e022a4e', 'b7fbeca9-9416-43c4-9e90-9e71acc1eaba', '27c9d358-a3ef-4c69-ba89-eac16d8d3bdb', 'ef982c4b-a115-48a1-aef1-2f672d7f1f00', 'efedede2-9bb4-4c52-98b1-8b03070df3fd', 'eb03ae1b-4cde-409c-8d34-2a16a8be30d2']
  x=['1548143296750', '1548183033872', '1548346185194', '1548443373507', '1548446119319', '1548446239441', '1548446068267', '1548445962159', '1548446011209', '1548446259465', '1548446180380', '1548239985290', '1548240060367', '1548240045347', '1547627568993', '1548755333313', '1548673604016','1548673443843', '1548673503914', '1548673563975']
  
  date=[]
  
  for i in x:
      print(i)
      print()
      i=i[:10]
      print(i)
      readable = time.ctime(int(i))
      readable=readable[:10]
      date.append(readable)
  print(date)
  plt.hist(date,y)
  plt.show()
  
• mysql.connector.errors.ProgrammingError: Error in SQL Syntax

  I'm using the Python MySQL connector to add data to a table by updating the row. A user enters a serial number, and then the row with the serial number is added. I keep getting a SQL syntax error and I can't figure out what it is.

    query = ("UPDATE `items` SET salesInfo = %s, shippingDate = %s, warrantyExpiration = %s, item = %s, WHERE serialNum = %s")
          cursor.execute(query, (info, shipDate, warranty, name, sn, ))
          conn.commit()
  

  Error:

  mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE serialNum = '1B0000021A974726'' at line 1
  

  "1B0000021A974726" is a serial number inputted by the user and it is already present in the table.
• How to work with images in pure Django templates so that they work like they do in Wagtail

  I intend to use images in my Django blog app, and I am wondering how do I go about this.

  So I only want one image to be uploaded with each post page. I added an ImageField() so that's that, however I can't force images to be uploaded to media/ folder. Also, I don't know how to force images to be automatically resized and have their names automatically changed.

  Right now, the declaration for the field handling Images is really simple

  main_image = ImageField(upload_to='media/', null=True)
  

  I don't really know how to do any of this, even though I have done my fair share of googling
• How to use default datetime serialization in Django REST Framework?

  I've got a Django REST Framework serializer containing the following:

  from rest_framework import serializers
  
  
  class ThingSerializer(serializers.ModelSerializer):
      last_changed = serializers.SerializerMethodField(read_only=True)
  
      def get_last_changed(self, instance: Thing) -> str:
          log_entry = LogEntry.objects.get_for_object(instance).latest()
  
          representation: str = serializers.DateTimeField('%Y-%m-%dT%H:%M:%SZ').to_representation(log_entry.timestamp)
          return representation
  

  This is problematic because if the datetime formatting ever changes it will be different to all the other datetimes. I want to reuse the code path which DRF uses to serialize other datetime fields.

  What I've tried so far:

  • The only answer which looked relevant doesn't actually produce the same result as DRF (it includes milliseconds, which DRF does not), presumably because it's using the Django rather than DRF serializer.
  • rest_framework.serializers.DateTimeField().to_representation(log_entry.timestamp), rest_framework.fields.DateTimeField().to_representation(log_entry.timestamp) and rest_framework.fields.DateTimeField(format=api_settings.DATETIME_FORMAT).to_representation(log_entry.timestamp) don't work either; they produce strings with microsecond accuracy. I've verified with a debugger that DRF calls the latter when serializing other fields, so I can't understand why it produces a different result in my case.

  • LogEntry.timestamp is declared as a django.db.DateTimeField, but if I try something like LogEntry.timestamp.to_representation(log_entry.timestamp) it fails badly:

    AttributeError: 'DeferredAttribute' object has no attribute 'to_representation'
• How can I speed up Django Query aggregations on a 10 million row database with 60 features?

  I have a database table in psql which contains of 10,000,000 rows and 60 columns (features). I define a Django Queryset as follows:

  MyQ=MyDataBase.objects.filter(Name='Mike', date=date(2018, 2, 11), 
  Class='03')
  

  There are only 5 rows that satisfy the above filter. But when I try something like

  MyQ.count() #which equals 5
  

  or

  MyQ.aggregate(Sum('Score'))['Score__sum'] #which equals 61
  

  each take about 3 minutes to give me the result. Isn't that weird? Aren't query sets supposed to make life easier by focusing only on the rows that we have told them to focus on? counting 5 rows or summing one of the fields of them must not take that long. What am I doing wrong?

  I should also say this. The first time that I tried this code on this table, everything was fine and it took maybe 1 second to catch the result but now the 3 minutes is really annoying. And since then I have not changed anything in the database or the code.
• Problem when deploying django rest in xampp

  I have a rare problem on the part of the client, I am using angular 5 as part of the client. I already made all the changes in Apache to run Django, it works well on my PC and on my cell phone when I connect to the server. when the request is made to the server it only works on my pc where the server is and on my cell phone.

  this is proof of this here the application is done well. but when I try to access the server of another pc the request is made to the old address, which is Django's

  this is the apache configuration.

  <IfModule wsgi_module>
  
   WSGIPythonPath C:\xampp\htdocs\tutorial
   <VirtualHost *:8081>
  
   Alias /static "C:\xampp\htdocs\tutorial\static"
   <Location "/media">
     SetHandler None
   </Location>
  
  <LocationMatch "\.(jpg|gif|png|js|css)$">
    SetHandler None
  </LocationMatch>
  
  <Directory C:\xampp\htdocs\tutorial\static>
      Require all granted
  </Directory>
  
  <Directory C:\xampp\htdocs\tutorial >
     <Files wsgi.py>
         Require all granted
     </Files>
  </Directory> 
  
  
    WSGIScriptAlias / C:\xampp\htdocs\tutorial\tutorial\wsgi.py
  
   </VirtualHost>
  
  
   </IfModule>
  

  and before making the deployment of the client I made the change of the url of the service.

   URLCategorias : string = 'http://10.0.0.16:8081/api/categorias/';
  

  is the only place that used the service

  and this es mi configuration hosts.

  DEBUG = False
  
  ALLOWED_HOSTS = ['10.0.0.16', 'localhost']
  

  I don't know what the problem is, why when making the call to the server, it points to another place.
• Passing generic pandas dataframe to django REST API

  I am currently building a REST API with the django and djangorestframework libraries. I want to make a POST that accepts a JSON-ified version of a pandas dataframe, so that I can then call some other library function on the dataframe and return some output.

  I know that if I know the table structure of the dataframe, there are nice libraries like django-rest-pandas that can serialize and return pandas dataframes as JSON, but I am not sure about how to accept them as parameters to a function.

  My current thought was to pass the JSON as a string to a function-based view, and from there, extract and parse the JSON as a pandas dataframe. However, I don't know if this is the best way to do this. My biggest complication is that I don't know the structure of the dataframe beforehand, so I am not sure how to incorporate that into a model that perhaps some library is able to interpret (although I am still interested in a way to do this if I did know the structure).

  Here is my code so far.

  Sample POST request:

  {
      "table": "{\"AAPL\":{\"1420156800000\":103.8639572404,\"1420416000000\":100.9379443592,\"1420502400000\":100.947444401,\"1420588800000\":102.3629506325,\"1420675200000\":106.2959679468},\"AMZN\":{\"1420156800000\":308.52,\"1420416000000\":302.19,\"1420502400000\":295.29,\"1420588800000\":298.42,\"1420675200000\":300.46},\"FB\":{\"1420156800000\":78.45,\"1420416000000\":77.19,\"1420502400000\":76.15,\"1420588800000\":76.15,\"1420675200000\":78.175},\"GOOG\":{\"1420156800000\":524.81,\"1420416000000\":513.87,\"1420502400000\":501.96,\"1420588800000\":501.1,\"1420675200000\":502.68},\"MSFT\":{\"1420156800000\":43.3443122802,\"1420416000000\":42.9410878182,\"1420502400000\":42.3153946875,\"1420588800000\":42.8530273035,\"1420675200000\":44.1136830927}}",
      "num_samples": 10
  }
  

  models.py:

  class EFObject(models.Model):
      table = models.TextField()
      num_samples = models.IntegerField()
  

  serializers.py:

  class EFObjectSerializer(serializers.ModelSerializer):
      class Meta:
          model = EFObject
          fields = ('table', 'num_samples')
  

  views.py:

  from rest_framework.response import Response
  from rest_framework.decorators import api_view
  from lib.api.serializers import *
  import lib.analytics.analytics as analytics
  import json
  import pandas as pd
  
  @api_view(http_method_names=['POST'])
  def find_ef(request):
      serializer = EFObjectSerializer(data=request.data)
      serializer.is_valid()
      output = serializer.validated_data['table']
      num_samples = serializer.validated_data['num_samples']
      table = pd.read_json(output) 
      result = analytics.find_ef(table, num_samples) # returns a dictionary
      return Response(result)
  

  urls.py:

  from django.conf.urls import include, url
  from rest_framework import routers
  from lib.api import views
  
  router = routers.DefaultRouter()
  router.register(r'users', views.UserViewSet)
  
  # Wire up our API using automatic URL routing.
  # Additionally, we include login URLs for the browsable API.
  urlpatterns = [
      url(r'', include(router.urls)),
      url(r'api-auth/', include('rest_framework.urls', namespace='rest_framework')),
      url(r'ef', views.find_ef)
  ]
  

  This works, but my question is what I should really be doing in the view.py file, as this doesn't seem like the best way to get a dataframe (and it's also a hassle to correctly escape the JSON string of the dataframe after calling table.to_json()). Any help would be appreciated.
• Can I improve the performance of my matlab codes?

  I leveraged matlab functions and mixed them with my python codes by import matlab.engine. So that I can use the dict() in python and a developed matlab package.

  However, I found that the codes in python perform in a much lower efficiency than those of the same logic in matlab, calling for about 10 times longer.

  Is the API of matlab for python suitable for repetitive calls in python?

  My goal includes using the dict() in python because I found the hash finding in matlab by container.Map() is slow. Is there any more efficient replacement?

  ---

  1st UPDATE

  In my task, it takes 20 times longer by matlab codes than python codes, where the container.map and mat2str takes most of the time.

  Because my keys are vectors that need to be changed according to read-in data, the 1*n matrices are converted to strings by mat2str each time creating or visiting a key in the matlab.

  My codes are below: (The files needed: VectorGenrationSequence.txt)

  ```

  clear;clc;
  %% BASIC SETTINGS
  nd = 46;
  num = 1000;
  
  v0 = zeros(1,nd);
  v0str = mat2str(v0);
  network = containers.Map;
  VECTOR = struct('array',sparse(nd,1),...
                  'digitCount',sparse(nd,1),...
                  'digit0',0,...
                  'value',0);
  %% READ IN DATA
  vgs = fopen('VectorGenrationSequence.txt');
  txt = textscan(vgs,'%s','delimiter','\n');
  fclose(vgs);
  txt = txt{1,1};
  %% CREATE THE NETWORK
  maxStage=0;
  dN = size(txt,1);
  timeStart1 = clock;
  for seqIdx = 1:num
      fprintf('%d\n',seqIdx);
      % vector sequence to vector sequence
      vector = v0;
      VectorSequnce = str2num(txt{seqIdx,1}); %#ok<*ST2NM>
      vectors = zeros(size(VectorSequnce,2)+1,nd);
      for stageI = 1:size(VectorSequnce,2)
          vector(VectorSequnce(stageI)) = 1;
          vectors(stageI+1,:) = vector;
      end
      % update the network
      for stageI = (size(VectorSequnce,2)+1):-1:1
          if stageI>size(VectorSequnce,2)
              currentDigit = 0;
          else
              currentDigit = VectorSequnce(stageI);
          end
          vectorStr = mat2str(vectors(stageI,:));
          if network.isKey(vectorStr)  % already have the vector
              networkTmp = network(vectorStr);
              % update the digit
              if currentDigit
                  networkTmp.digitCount(currentDigit) = ...
                      networkTmp.digitCount(currentDigit) + 1;
              else
                  networkTmp.digit0 = networkTmp.digit0 + 1;
                  networkTmp.value = rand;
              end
              network(vectorStr) = networkTmp;
  
          else % do not have the vector
              networkTmp = VECTOR;
              if currentDigit
                  networkTmp.digitCount(currentDigit) = 1;
                  networkTmp.Fvalue(currentDigit) = ...
                      network(mat2str(vectors(stageI+1,:))).Svalue;
                  networkTmp.Svalue = networkTmp.Fvalue(currentDigit);
              else
                  networkTmp.value = rand;
                  networkTmp.Svalue = networkTmp.value;
                  networkTmp.digit0 = 1;
              end
              networkTmp.previousFailure = ...
                  find(vectors(stageI,:)==1);
              networkTmp.previousState = ...
                  cell(size(networkTmp.previousFailure));
              for idx = 1:size(networkTmp.previousFailure,2)
                  spre = vectors(stageI,:);
                  spre(networkTmp.previousFailure(idx)) =0;
                  networkTmp.previousState{1,idx} = ...
                      mat2str(spre);
              end
              network(vectorStr) = networkTmp;
          end
      end
  end
  
  timeEnd1 = clock;
  timeElapse = etime(timeEnd1,timeStart1)
  

  ```
• Why are Python operations 30× slower after calling time.sleep or subprocess.Popen?

  Consider the following loop:

  for i in range(20):
      if i == 10:
          subprocess.Popen(["echo"]) # command 1
      t_start = time.time()
      1+1 # command 2
      t_stop = time.time()
      print(t_stop - t_start)
  

  “command 2” command takes systematically longer to run when “command 1” is run before it. The following plot shows the execution time of 1+1 as a function of the loop index i, averaged over 100 runs.

  Execution of 1+1 is 30 times slower when preceded by subprocess.Popen.

  

  ---

  It gets even weirder. One may think that only the first command run after subprocess.Popen() is affected, but it is not the case. The following loop shows that all commands in the current loop iteration are affected. But the subsequent loops iterations seem to be mostly OK.

  var = 0
  for i in range(20):
      if i == 10:
        # command 1
        subprocess.Popen(['echo'])
      # command 2a
      t_start = time.time()
      1 + 1
      t_stop = time.time()
      print(t_stop - t_start)
      # command 2b
      t_start = time.time()
      print(1)
      t_stop = time.time()
      print(t_stop - t_start)
      # command 2c
      t_start = time.time()
      var += 1
      t_stop = time.time()
      print(t_stop - t_start)
  

  Here’s a plot of the execution times for this loop, average over 100 runs:

  

  ---

  More remarks:

  • We get the same effect when replacing subprocess.Popen() (“command 1”) with time.sleep(), or rawkit’s libraw C++ bindings initialization (libraw.bindings.LibRaw()). However, using other libraries with C++ bindings such as libraw.py, or OpenCV’s cv2.warpAffine() do not affect execution times. Opening files don’t either.
  • The effect is not caused by time.time(), because it is visible with timeit.timeit(), and even by measuring manually when print() result appear.
  • It also happens without a for-loop.
  • This happens even when a lot of different (possibly CPU- and memory-consuming) operations are performed between “command 1” (subprocess.Popen) and “command 2”.
  • With Numpy arrays, the slowdown appears to be proportional to the size of the array. With relatively big arrays (~ 60 M points), a simple arr += 1 operation can take up to 300 ms!

  ---

  Question: What may cause this effect, and why does it affect only the current loop iteration?

  I suspect that it could be related to context switching, but this doesn’t seem to explain why a whole loop iteration would affected. If context switching is indeed the cause, why do some commands trigger it while others don’t?