Django REST Framework - POST slowness when body size greater than 1024 characters

2018-11-08 07:25 Landon imported from Stackoverflow

I'm using DRF. I found that receiving a POST with more than 1024 characters causes a ~1 second penalty, while anything less than that is effectively free. I've simplified this into this trivial example:

# views.py
import time
from rest_framework.decorators import api_view
from django.http import HttpResponse

@api_view(['POST'])
def test_endpoint(request):

    t = time.time()
    data = request.body
    total_time = time.time() - t

    print('got post data:', round(total_time, 3))
    return HttpResponse('body size:{} time:{}'.format(len(data), round(total_time, 3)))


# url.py
urlpatterns = [
    url(r'^test_endpoint', test_endpoint),
]

You can see that all I'm doing is reading the request.body and measuring the time it takes to do so. Then I respond with that time, and the len of the request.body (to prove I accessed it).

Then I execute these curls:

$ time curl -X POST http://127.0.0.1:8000/test_endpoint -d 0123456782345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567901234567890123456789012345678901234567
body size:1024 time:0.0
real    0m0.045s
user    0m0.006s
sys 0m0.009s
$ time curl -X POST http://127.0.0.1:8000/test_endpoint -d 01234567823456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345679012345678901234567890123456789012345678
body size:1025 time:0.999
real    0m1.020s
user    0m0.006s
sys 0m0.006s

You can see the second one has one extra character, and that causes a ~1 second penalty to ready request.body.

Why is that? How can I prevent this?

More Info

I made this as vanilla as I could. I created an new project with django-admin startproject helloworld_project .. I put the code above into it. And running it locally with python manage.py runserver. I don't have a webserver in front of it; I'm accessing it directly with my browser. That's all I'm doing.

Also, I'm doing this on Django==1.11. This problem seems to go away on Django==2.0. I cannot easily upgrade to 2.0. Is there a workaround for this issue on 1.11?

My Current excel looks like:

  ----------------
  |  Type |  Val |
  |--------------|
  |  A    |  1   |
  |--------------|     
  |  A    |  2   |     
  |--------------|
  |  B    |  3   |
  |--------------|     
  |  B    |  4   |     
  |--------------|     
  |  B    |  5   |
  |--------------|
  |  C    |  6   |

----------------

This is the required excel:

  ----------------------
  |  Type |  Val | Sum |
  |--------------------|
  |  A    |  1   | 3   |
  |       |------|     |
  |       |  2   |     |
  |--------------------|
  |  B    |  3   | 12  |
  |       |------|     |
  |       |  4   |     |
  |       |------|     |
  |       |  5   |     |
  |--------------------|
  |  C    |  6   |  6  |
  ----------------------

Is it possible in python using pandas or any other module?

How to Give A Postgres User SuperUser Previllege Through docker-compose?
This is my docker-compose file section for postgres container. These settings are fine, but my django app requires this user to have superuser previlleges through this command inside postgresql.
```
ALTER ROLE project_admin SUPERUSER;
```
How can this be accomodated inside this docker-compose file?
```
 db:
  image: postgres:latest
  container_name: project-db
  environment:
   - POSTGRES_USER='project_admin'
   - POSTGRES_PASS='projectpass'
   - POSTGRES_DB='project'
```

Django/Nginx : serving static files in production

I am trying to serve CSS and other static files on my django app using NGINX server. So I tried to configure it. Here is my /etc/nginx/sites-enabled/mydomain:

server {
    listen 80;
    server_name redpillers.net;

    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/limited/REDPILLERS;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:/home/limited/REDPILLERS/redpillers.sock;
    }

    location /static/ {
       alias /home/limited/REDPILLERS/static/;
    }

}

But when I restart the service I got an error in the error logs file:

2019/05/22 07:26:44 [emerg] 11589#11589: duplicate location "/static/" in /etc/nginx/sites-enabled/mydomain:15

TypeError int() argument must be a string, a bytes-like object or a number, not 'InMemoryUploadedFile'

I was simply trying to upload a file in Django 2 but it's returning this error every time. my from.html have multiple filed where user can add filed if he needs. so i have to k now aobut this is have to know adfsd fasdf af af af as fasdf a f as fa sf as fa sf asf asdf asd f af a f asfd a f

I was simply trying to upload a file in Django 2 but it's returning this error every time. my from.html have multiple filed where user can add filed if he needs.

from.html image view

Error In terminal

Internal Server Error: /stockSale/addStockSale
Traceback (most recent call last):
  File "/home/an2/.local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/an2/.local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/an2/.local/lib/python3.6/site-packages/django/core/handlers/base.py", line 124, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/an2/.local/lib/python3.6/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/an2/office/s2s/s2s/stockSale/views.py", line 295, in addStockSale
    tempItem.save()
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/base.py", line 718, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/base.py", line 748, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/base.py", line 831, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/base.py", line 869, in _do_insert
    using=using, raw=raw)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/query.py", line 1136, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1288, in execute_sql
    for sql, params in self.as_sql():
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1241, in as_sql
    for obj in self.query.objs
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1241, in <listcomp>
    for obj in self.query.objs
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1240, in <listcomp>
    [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1182, in prepare_value
    value = field.get_db_prep_save(value, connection=self.connection)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 790, in get_db_prep_save
    return self.get_db_prep_value(value, connection=connection, prepared=False)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 785, in get_db_prep_value
    value = self.get_prep_value(value)
  File "/home/an2/.local/lib/python3.6/site-packages/django/db/models/fields/__init__.py", line 1807, in get_prep_value
    return int(value)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'InMemoryUploadedFile'
[22/May/2019 11:04:05] "POST /stockSale/addStockSale HTTP/1.1" 500 155523

Here is my view.py file

stockSave = StockSale(
                title=request.POST.get('title'),
                slug=slugify(request.POST.get('title')),
                description=request.POST.get('description'),
                company_id=request.user.company_id,
                created_by=request.user.id,
                sale_status=1,
            )
            stockSave.save()
            stock_sale_id = stockSave.id
            item_name = request.POST.getlist('item_name')
            item_slug = slugify(request.POST.getlist('item_name'))
            unit_id = request.POST.getlist('unit_id')
            total_quantity = request.POST.getlist('total_quantity')
            minimum_order = request.POST.getlist('minimum_order_buy')
            item_image = request.FILES.getlist('item_image')
            item_image_two = request.FILES.getlist('item_image_two')

            zipped = zip(item_name, item_slug, item_image, item_image_two, unit_id, total_quantity, minimum_order)
            for item_name, item_slug, unit_id, item_image, item_image_two, total_quantity, minimum_order in zipped:
                tempItem = TempItem(
                    stock_sale_id="1",
                    title=item_name,
                    slug=item_slug,
                    item_image=item_image,
                    item_image_two=item_image_two,
                    unit_id=unit_id,
                    company_id=request.user.company_id,
                    created_by=request.user.id,
                )
                tempItem.save()
                itemId = tempItem.id
                sotckSaleDetails = StockSaleDetail(
                    stock_sale_id=stock_sale_id,
                    temp_id=itemId,
                    unit_id=unit_id,
                    total_quantity=total_quantity,
                    minimum_order=minimum_order,
                    created_by=request.user.id,
                    company_id=request.user.company_id,
                    sale_status=1,
                )
                sotckSaleDetails.save()
            messages.add_message(request, messages.INFO, 'Saved Successfully')
            sweetify.success(request, 'Post Successfully!', text='Your Stock Buy is just post in Search to Source.',
                             persistent='OK')
            return redirect('stockSale')

Here is my model.py file

class TempItem(models.Model):
    stock_sale_id = models.IntegerField()
    title = models.CharField(max_length=255)
    slug = models.CharField(max_length=255)
    description = models.TextField()
    unit_id = models.IntegerField()
    price = models.FloatField(blank=True, null=True)
    item_image = models.ImageField(upload_to='uploads/stockSale/')
    item_image = models.ImageField(upload_to='uploads/stockSale/')
    item_image_two = models.CharField(max_length=255)
    created_by = models.IntegerField()
    company_id = models.IntegerField()
    updated_by = models.IntegerField(blank=True, null=True, default=0)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)
    status = models.IntegerField(blank=True, null=True, default=0)
    deleted = models.IntegerField(blank=True, null=True, default=0)

How to add Custom Validations with Custom Response in Django Rest Framework?
I have a title field for which I have performed Validations that it should not contain any digits. Now I want to create a general custom response object if the Validations fail.

models.py:
```
title=models.CharField(max_length=50)
```
serializer.py:
```
    title=serializers.CharField(max_length=50,validators=[jobProfile_title])
```
Validation Logic:
```
if not proper validation: 
raise serializers.ValidationError("Message")
```
I want to use a custom response object and not the default one

CSRF Cookie Not Set error, Django Rest Framework Request Factory

I am unit testing a post method of a VIEW and getting a 403 response with the below error.

`def single_plate_pri(self):
    u = User.objects.get(username = 'c269880')
    self.client.force_authenticate(u)
    data = copy.deepcopy(self.valid_pri_data)
    factory = RequestFactory()
    post_url = '/sequence-request/confirm/'
    request = factory.post(post_url, data, user= u, 
    simulation_mode = True, enforce_csrf_checks=True)
    request.user = User.objects.get(username='c269880')
    view = csrf_exempt(SequencingRequestSpreadsheetView().as_view())
    resp = view(request)
    self.assertEqual(resp.status_code, 200)`

Error :

{'detail': ErrorDetail(string='CSRF Failed: CSRF cookie not set.', code='permission_denied')}

I even tried csrf on the view but it did not work.

Calling post a method of a serializer and passing data to it
I want to call a post of a class as shown below by instantiating a view as shown below., i need to write a unit test that calls the post method of this class and not through URL.
```
class SequencingRequestSpreadsheetView(GenericAPIView):

    parser_classes = (JSONParser,)
    serializer_class = SequencingRequestSerializer
    permission_classes = (IsBioUser, )

    suffix = '.xls'
    path = settings.SEQUENCE_REQUEST_SUBMISSION

    def post(self, request, format=None, simulation_mode = False): 
```
I need to know how do I create a request object and pass it to this function. iam instantiating this view class and I tried passing a request data as json and also tried dictionary but did not work.

how do I create a request object and pass it to this method.

resp = SequencingRequestSpreadsheetView().post(request)

Can I improve the performance of my matlab codes?

I leveraged matlab functions and mixed them with my python codes by import matlab.engine. So that I can use the dict() in python and a developed matlab package.

However, I found that the codes in python perform in a much lower efficiency than those of the same logic in matlab, calling for about 10 times longer.

Is the API of matlab for python suitable for repetitive calls in python?

My goal includes using the dict() in python because I found the hash finding in matlab by container.Map() is slow. Is there any more efficient replacement?

1st UPDATE

In my task, it takes 20 times longer by matlab codes than python codes, where the container.map and mat2str takes most of the time.

Because my keys are vectors that need to be changed according to read-in data, the 1*n matrices are converted to strings by mat2str each time creating or visiting a key in the matlab.

My codes are below: (The files needed: VectorGenrationSequence.txt)

```

clear;clc;
%% BASIC SETTINGS
nd = 46;
num = 1000;

v0 = zeros(1,nd);
v0str = mat2str(v0);
network = containers.Map;
VECTOR = struct('array',sparse(nd,1),...
                'digitCount',sparse(nd,1),...
                'digit0',0,...
                'value',0);
%% READ IN DATA
vgs = fopen('VectorGenrationSequence.txt');
txt = textscan(vgs,'%s','delimiter','\n');
fclose(vgs);
txt = txt{1,1};
%% CREATE THE NETWORK
maxStage=0;
dN = size(txt,1);
timeStart1 = clock;
for seqIdx = 1:num
    fprintf('%d\n',seqIdx);
    % vector sequence to vector sequence
    vector = v0;
    VectorSequnce = str2num(txt{seqIdx,1}); %#ok<*ST2NM>
    vectors = zeros(size(VectorSequnce,2)+1,nd);
    for stageI = 1:size(VectorSequnce,2)
        vector(VectorSequnce(stageI)) = 1;
        vectors(stageI+1,:) = vector;
    end
    % update the network
    for stageI = (size(VectorSequnce,2)+1):-1:1
        if stageI>size(VectorSequnce,2)
            currentDigit = 0;
        else
            currentDigit = VectorSequnce(stageI);
        end
        vectorStr = mat2str(vectors(stageI,:));
        if network.isKey(vectorStr)  % already have the vector
            networkTmp = network(vectorStr);
            % update the digit
            if currentDigit
                networkTmp.digitCount(currentDigit) = ...
                    networkTmp.digitCount(currentDigit) + 1;
            else
                networkTmp.digit0 = networkTmp.digit0 + 1;
                networkTmp.value = rand;
            end
            network(vectorStr) = networkTmp;

        else % do not have the vector
            networkTmp = VECTOR;
            if currentDigit
                networkTmp.digitCount(currentDigit) = 1;
                networkTmp.Fvalue(currentDigit) = ...
                    network(mat2str(vectors(stageI+1,:))).Svalue;
                networkTmp.Svalue = networkTmp.Fvalue(currentDigit);
            else
                networkTmp.value = rand;
                networkTmp.Svalue = networkTmp.value;
                networkTmp.digit0 = 1;
            end
            networkTmp.previousFailure = ...
                find(vectors(stageI,:)==1);
            networkTmp.previousState = ...
                cell(size(networkTmp.previousFailure));
            for idx = 1:size(networkTmp.previousFailure,2)
                spre = vectors(stageI,:);
                spre(networkTmp.previousFailure(idx)) =0;
                networkTmp.previousState{1,idx} = ...
                    mat2str(spre);
            end
            network(vectorStr) = networkTmp;
        end
    end
end

timeEnd1 = clock;
timeElapse = etime(timeEnd1,timeStart1)

```

Why are Python operations 30× slower after calling time.sleep or subprocess.Popen?
Consider the following loop:
```
for i in range(20):
    if i == 10:
        subprocess.Popen(["echo"]) # command 1
    t_start = time.time()
    1+1 # command 2
    t_stop = time.time()
    print(t_stop - t_start)
```
“command 2” command takes systematically longer to run when “command 1” is run before it. The following plot shows the execution time of 1+1 as a function of the loop index i, averaged over 100 runs.

Execution of 1+1 is 30 times slower when preceded by subprocess.Popen.

It gets even weirder. One may think that only the first command run after subprocess.Popen() is affected, but it is not the case. The following loop shows that all commands in the current loop iteration are affected. But the subsequent loops iterations seem to be mostly OK.
```
var = 0
for i in range(20):
    if i == 10:
      # command 1
      subprocess.Popen(['echo'])
    # command 2a
    t_start = time.time()
    1 + 1
    t_stop = time.time()
    print(t_stop - t_start)
    # command 2b
    t_start = time.time()
    print(1)
    t_stop = time.time()
    print(t_stop - t_start)
    # command 2c
    t_start = time.time()
    var += 1
    t_stop = time.time()
    print(t_stop - t_start)
```
Here’s a plot of the execution times for this loop, average over 100 runs:

More remarks:
- We get the same effect when replacing subprocess.Popen() (“command 1”) with time.sleep(), or rawkit’s libraw C++ bindings initialization (libraw.bindings.LibRaw()). However, using other libraries with C++ bindings such as libraw.py, or OpenCV’s cv2.warpAffine() do not affect execution times. Opening files don’t either.
- The effect is not caused by time.time(), because it is visible with timeit.timeit(), and even by measuring manually when print() result appear.
- It also happens without a for-loop.
- This happens even when a lot of different (possibly CPU- and memory-consuming) operations are performed between “command 1” (subprocess.Popen) and “command 2”.
- With Numpy arrays, the slowdown appears to be proportional to the size of the array. With relatively big arrays (~ 60 M points), a simple arr += 1 operation can take up to 300 ms!
Question: What may cause this effect, and why does it affect only the current loop iteration?

I suspect that it could be related to context switching, but this doesn’t seem to explain why a whole loop iteration would affected. If context switching is indeed the cause, why do some commands trigger it while others don’t?

Related questions

Django REST Framework - POST slowness when body size greater than 1024 characters

More Info

See also questions close to this topic

My Current excel looks like:

----------------

This is the required excel: