Socket Programming : how to receive data until a dot is received (Python)
Situation: I am currently writing a client program with a provided server program I need to send the command 'READ' and get a bunch of data from server
The server will just keep sending and lastly it will sd a dot (".") My task is to display all msg it send out (tgt with the dot) I can successfully receive everything and displayed. however i cannot exit the while loop(cannot successfully detect the dot) Why is that? Below is my code. thanks!
def READ(mySocket):
op="READ"
clientSocket.send(op.encode())
while(True):
try:
rcvmsg = clientSocket.recv(4096)
print("Server:{} ".format(rcvmsg.decode()))
if (rcvmsg.decode() == "."):
break
except :
print("some error happen" )
sys.exit(1)
See also questions close to this topic
-
Eliminate for-loop in Python
I have to compute some quantities in python using the following formula: (I'll post an Image since LaTeX language is not recognized)
where "psi" is simply the digamma function, v_j and alpha_{t, bl}^{l} are floating numbers with t={0, 1, 2}, bl = {0, 1, 2, 3} and l = {1, 2, ..., 309}.
For now I have solved grouping v_j into a list and the alphas into nested dictionaries (in order: l, t, bl), iterating the formula with three for loops:
for loop in tqdm(range(500)): logr = [] for t in range(3): add1 = digamma(V[t]) - digamma(sum(V)) add2 = 0 for b in test.columns[2:-1]: add2 += digamma(alphas[b][t][test.iloc[395][b]]) - digamma(sum(alphas[b][t])) logrho = add1 + add2 logr.append(logrho) rho = np.exp(logr) q = rho/sum(rho)
The problem is that I have to iterate these lines of code for hundreds of time. So, I am looking for a way to eventually vectorize my problem. My first guess was to convert the "alpha dictionary" into a dataframe (
pd.DataFrame.from_dict()
) but still I can't get rid of the for loops.Is there a better way to tackle the problem? Maybe creating a three dimensional pandas dataframe?
EDIT
The real problem I am struggling to figure out is about the second part of the equation (let me forget about the digamma function). The alpha parameters have 3 dimensions: t=0,1,2; l=1,2,...,N and bl=0,1,2,3. Assume we fix t so that we end up iterating only over t=0,1,2. The alpha parameters can be represented in a two dimensional dataframe:
bl l1 l2 l3 ... lN 0 a0l1 a0l2 a0l3 a0lN 1 a1l1 a1l2 a1l3 a1lN 2 ... 3 a3l1 ... a3lN
I should't subtract the sum over the the column from each alpha, but only from one of the four classes bl=0,1,2,3. If in the data, the variable l1 provides an outcome of 3, I should subtract the sum of the alphas for column l1 from a3l1; if the outcome in l2 is 0, i should subtract the sum of the alphas of column l2 from a0l2 and so on. Finally, the sum of the results will be summed (sum over l)
-
Python ServiceNow Semaphore Timeout Error
Trying to connect to a ServiceNow instance and pull information from the tables. I am using the aiosnow module. Here is the code:
import asyncio import getpass import aiosnow from aiosnow.models.table.declared import IncidentModel as Incident # Get user input instanceSN = input("Enter the instance name: ") usernameSN = input("Enter your ServiceNow username: ") passwordSN = getpass.getpass() async def main(): client = aiosnow.Client(instanceSN, basic_auth=(usernameSN,passwordSN)) async with Incident(client, table_name="incident") as inc: query = aiosnow.select( Incident.assigned_to.starts_with(<insert username>) ).order_asc(Incident.priority) for response in await inc.get(query, limit=5): print("Attaching [readme.txt] to {number} ({sys_id})".format(**response)) await inc.upload_file(response["sys_id"], path="./readme.txt") asyncio.run(main())
When I run this I get the following error message: aiosnow.exceptions.ClientConnectionError: Cannot connect to host servicenow.example.com:443 ssl:default [The semaphore timeout period has expired]
-
Python Twitter: Check if a tweet exists by url without API
I have a list of tweets urls from the same account, and I want to check if this tweets still exist or not.
A tweet may not exist anymore if twitter responds with such errors:
This Tweet is from an account that no longer exists. Learn more
or
Sorry that page doesn't exist!
or any such errors.
What I have tried is using twint library to scrape all the tweets from the given profile, and check if the tweets on my "tweets list" is also in the result that the twint library.
And I have used this function to scrape all the tweets using twint:
def get_tweets(username): c = twint.Config() c.Username = username tweets = [] c.Store_object = True c.Retweets = True c.Store_object_tweets_list = tweets c.Hide_output = True twint.run.Profile(c) tweets_links = [] for tweet in tweets: tweets_links.append(tweet.link) return tweets_links get_tweets(username)
This works well but the problem is that it doesn't scrape all the tweets, and it stops at a certain date (for the username I'm testing 'GideonCRozner' it stops at 24/06/2020), and I have posts urls which are before that date. So simply I'm not able to scrape all the posts using twint library.
My solution right now is to include
selenium
in the code andget
the posts which are not scraped yet one by one, but as you know selenium is a slower solution for that.So I hope that I can use some ideas from you, to scrape all the user's tweets or test a tweet if it exists without selenium and without Twitter API
Thanks a lot for your time!
-
Should I resend or reconnect if the acknowledgement for a sent message is missing?
In a distributed system, to ensure the peer has received a message correctly, we may use application level acknowledgement mechanism (ACK).
But when the client and the server is connecting via TCP or WebSocket, if the application level ACK for a sent message is not received by the sender, does it mean the connection is broken and should be reconnected?
Is it possible for the data or application level ACK is missing from application level but actually successfully transmitted in TCP level? (In this case, resending the message may be economical.)
-
Receiving image through sockets on androidstudio
I have written a code on Android Studio to receive images from a socket server and the application connects to the server every time a button is clicked.
However, when I run the app and click the button nothing shows up but the server sends a message saying the photo is sent. When I click the button again, for the second time (the server is not connected) the image pops up instantly.
I think the issue is that the thread isn't shutting down completely when I click the button once but if I click it again, the thread shuts down forcefully and starts a new one so the image is shown.
The code for the main activity java file is :
package com.example.myapplication; import android.graphics.Bitmap; import android.graphics.BitmapFactory; import android.os.Bundle; import android.os.Handler; import android.view.View; import android.widget.Button; import android.widget.EditText; import android.widget.ImageView; import android.widget.TextView; import androidx.appcompat.app.AppCompatActivity; import java.io.BufferedOutputStream; import java.io.FileOutputStream; import java.io.InputStream; import java.net.Socket; public class MainActivity extends AppCompatActivity implements View.OnClickListener { private TextView mTextViewReplyFromServer; private EditText mEditTextSendMessage; private ImageView mImg; private byte [] imgbyte; Handler updateConversationHandler; @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); Button buttonSend = (Button) findViewById(R.id.btn_send); mEditTextSendMessage = (EditText) findViewById(R.id.edt_send_message); mTextViewReplyFromServer = (TextView) findViewById(R.id.tv_reply_from_server); mImg = (ImageView)findViewById(R.id.imageView); String filepath = "/sdcard/DCIM/img.jpeg"; } @Override public void onClick(View v) { switch (v.getId()) { case R.id.btn_send: Thread fst = new Thread(new ServerThread()); fst.start(); break; } } public class ServerThread implements Runnable { byte [] line; Bitmap bitmap; public void run() { try { Socket client = new Socket("192.168.1.145", 5560); while (true) { // LISTEN FOR INCOMING CLIENTS try { int bytesRead; int current = 0; int filesize=215320; byte [] mybytearray2 = new byte [filesize]; InputStream is = client.getInputStream(); FileOutputStream fos = new FileOutputStream("/sdcard/DCIM/img.jpg"); // destination path and name of file //FileOutputStream fos = new FileOutputStream("/storage/sdcard0/Pictures/Screenshots/"); BufferedOutputStream bos = new BufferedOutputStream(fos); bytesRead = is.read(mybytearray2,0,mybytearray2.length); current = bytesRead; do { bytesRead = is.read(mybytearray2, current, (mybytearray2.length-current)); if(bytesRead >= 0) current += bytesRead; } while(bytesRead > -1); bos.write(mybytearray2, 0 , current); bos.flush(); // bitmap = BitmapFactory.decodeByteArray(mybytearray2 , 0, mybytearray2.length); // mImg.setImageBitmap(bitmap); //System.out.println(end-start); updateConversationHandler.post(new updateUIThread(mybytearray2)); bos.close(); client.close(); break; } catch (Exception e) { e.printStackTrace(); } } } catch (Exception e) { e.printStackTrace(); } } } class updateUIThread implements Runnable { private byte[] byteArray;//private String msg; public updateUIThread(byte[] array){ //public updateUIThread(String str) { this.byteArray=array; //this.msg = str; } @Override public void run() { Bitmap bitmap = BitmapFactory.decodeByteArray(byteArray , 0, byteArray .length); mImg.setImageBitmap(bitmap);//text.setText(text.getText().toString()+"Client Says: "+ msg + "\n"); } } }
Is there anyway I can kill the thread immediately after the image is recieved?
My image also shows up on the android emulator but does not show up on my phone. What could be the reason for this?
Edit : if i start the thread on the oncreate section, the image pops up as soon as the application is started
Edit : python server code :
import socket from time import sleep from time import time host = '' port = 5560 filePath = "/media/pi/ESD-USB/image.jpg" def setupServer(): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) print("Socket created.") try: s.bind((host, port)) except socket.error as msg: print(msg) print("Socket bind comlete.") return s def setupConnection(): s.listen(1) # Allows one connection at a time. conn, address = s.accept() print("Connected to: " + address[0] + ":" + str(address[1])) return conn def sendPic(s, filePath): print(filePath) pic = open(filePath, 'rb') chunk = pic.read() size = len(chunk) print (size) t = time() print("Sending Picture") s.sendall(chunk) pic.close() print("Done sending") print("Elapsed time = " + str(time() - t) + 's') s.close() return "Done sending" def backup(filePath): conn = setupConnection() response = sendPic(conn, filePath) return response s = setupServer() while True: print(filePath) backup(filePath) print("Everything should be backed up now.") break
-
Realtime One to one chat after selecting user from user list in node js and socket io
I am using node and socket io, where user can select a user and start one to one chat. Of any history is exists then load. Using this code message every thing is working with the problem is when user sends message it emits to every user. i have tried soket io.to('') but nothing works. please help me in this. Thank you in advance,
io.on('connection', (socket) => { sok_connections.push(socket); init chat when user click on user then create chat room or if room exist then set socket socket.on('chat-start', function (data) { var sql = "SELECT * FROM chat_rooms WHERE user_1 ='" + data.sender_id + "' AND user_2='" + data.reciver_id + "' OR user_2 ='" + data.sender_id + "' AND user_1='" + data.reciver_id + "'"; pool.getConnection(function (err, connection) { connection.query(sql, function (ch_err, ch_result, ch_fields) { if (ch_err) throw ch_err; if (ch_result.length == 0) { create chat room var query = "INSERT INTO `chat_rooms`(`chat_uid`, `user_1`, `user_2`) VALUES ('" + uniqid('CHAT-ROOM-', '-ESS') + "','" + data.sender_id + "','" + data.reciver_id + "')"; connection.query(query, function (c_ch_err, c_ch_result, c_ch_fields) { if (c_ch_err) throw c_ch_err; var get_data = "select * from chat_rooms where '" + c_ch_result.insertId + "' "; connection.query(get_data, function (s_ch_err, s_ch_result, s_ch_fields) { if (s_ch_err) throw err; current_chat_room_dets.sender_id = data.sender_id; current_chat_room_dets.reciver_id = data.reciver_id; current_chat_room_dets.chat_uid = s_ch_result[0].chat_uid; socket.join(current_chat_room_dets.chat_uid); }); }); } else { var old_chats = "SELECT * FROM conversations WHERE chat_uid='" + ch_result[0].chat_uid + "'"; connection.query(old_chats, function (co_err, co_result, co_fields) { if (co_err) throw co_err; current_chat_room_dets.sender_id = data.sender_id; current_chat_room_dets.reciver_id = data.reciver_id; current_chat_room_dets.chat_uid = ch_result[0].chat_uid; socket.join(current_chat_room_dets.chat_uid); }); } }); }); }); when user change user on chat room socket.on('unsubscribe', function () { socket.leave(current_chat_room_dets.chat_uid); }) load old message and pushing to messsages socket.on('initial-messages', function (data) { var sql = "SELECT * FROM conversations WHERE chat_uid='" + current_chat_room_dets.chat_uid + "'"; pool.getConnection(function (err, connection) { connection.query(sql, function (err, result, fields) { if (err) throw err; connection.release(); for (let res of result) { io.to(current_chat_room_dets.chat_uid).emit('messages', res); } }); }); }); when new user comes set status of user and add to user list socket.on('new user', function (data) { var sql = "SELECT * FROM users"; pool.getConnection(function (err, connection) { connection.query(sql, function (err, result, fields) { connection.release(); if (err) throw err; for (let au of result) { if (au.id == data.user_id) { au.active = true } else { au.active = false } } io.sockets.emit('users', result); }); }); }); when user send a message then post message socket.on('new-message', function (message) { var query = "INSERT INTO `conversations` (`chat_uid`,`message`,`sender`) VALUES ('" + current_chat_room_dets.chat_uid + "','" + message.message + "','" + current_chat_room_dets.sender_id + "')"; pool.getConnection(function (err, connection) { connection.query(query, function (err, rows) { if (err) throw err; }); connection.release(); io.to(current_chat_room_dets.chat_uid).emit('messages', message); }); }); get user typing status socket.on("typing", function (data) { io.log(data, "typing"); socket.broadcast.emit("typing", data); }); socket.once("disconnect", function (data) { console.log(socket.userId, "disconnected"); }); });
-
Osram Smart plus Smart home automation wiht python - no hub
I want to controll my smart lights with python throug my own programm in python. Theres fortunatly a library for that lightify (https://pypi.org/project/lightify/1.0.0/). My first hurdle was that i couldn´t find the IP-Adress of my lights. No it occurs to me that it might be not possible to use this library without hub. My current "hub" is my Alexa. Therefore I didn´t need the IP-Adress in the first place.
Thus i have two questions.
First: Is it possible to interact with these lights without a hub (my alexa excluded).
Second: If so, how can i find out the IP-Adress of the Device. I´ve read that these devices are all on port 4000. But in this context i don´t quite understand how the lights should have connected to my network. To connect them to my Alexa, i only had to search in the Alexa App for Osram Lights (after plugging them in and changing to parring mode obviously)
Hope you got what I meant. Thanks for your help.
-
I am getting an error when I tried to cross compile Libcork on windows using Cygwin (similar version of ipsets present in windows)
I tried to install a library called Libcork in windows using Cygwin. It is just like IP_SETS which is present in Linux(IP_SETS). I am getting an error while tried to cmake
In file included from
/cygdrive/c/users/dines/downloads/ipset/libcork/include/libcork/core/hash.h:16, from /cygdrive/c/users/dines/downloads/ipset/libcork/include/libcork/core/callbacks.h:14, from /cygdrive/c/users/dines/downloads/ipset/libcork/include/libcork/core/allocator.h:18, from /cygdrive/c/users/dines/downloads/ipset/libcork/include/libcork/core.h:15, from /cygdrive/c/users/dines/downloads/ipset/libcork/src/libcork/cli/commands.c:15: /cygdrive/c/users/dines/downloads/ipset/libcork/include/libcork/core/byte-order.h:45:2: error: #error "Unknown endianness" 45 | #error "Unknown endianness" | ^~~~~ /cygdrive/c/users/dines/downloads/ipset/libcork/include/libcork/core/hash.h: In function ‘cork_stable_hash_buffer’: /cygdrive/c/users/dines/downloads/ipset/libcork/include/libcork/core/byte-order.h:175:44: error: implicit declaration of function ‘CORK_UINT32_LITTLE_TO_HOST’; did you mean ‘CORK_UINT32_HOST_TO_BIG’? [-Werror=implicit-function-declaration] 175 | #define CORK_UINT32_HOST_TO_LITTLE(__u32) CORK_UINT32_LITTLE_TO_HOST(__u32) | ^~~~~~~~~~~~~~~~~~~~~~~~~~ /cygdrive/c/users/dines/downloads/ipset/libcork/include/libcork/core/hash.h:107:24: note: in expansion of macro ‘CORK_UINT32_HOST_TO_LITTLE’ 107 | uint32_t k1 = CORK_UINT32_HOST_TO_LITTLE(cork_getblock32((const uint32_t *) curr, 0)); | ^~~~~~~~~~~~~~~~~~~~~~~~~~ cc1: all warnings being treated as errors make[2]: *** [src/CMakeFiles/libcork-shared.dir/build.make:83: src/CMakeFiles/libcork-shared.dir/libcork/cli/commands.c.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:353: src/CMakeFiles/libcork-shared.dir/all] Error 2 make: *** [Makefile:161: all] Error 2
Does any body tried this before. I anyone did can I get some info about how to overcome this issue.
-
Socket Programming — recv() cannot get all data
I am learning socket programming in C language, and this is an incomprehensible problem I encountered during my study.
Today I am trying to send a HTTP request to my test server which host an Apache example website, then receive the response from test server. Here is a part of my receive code.
unsigned long recv_size = 0; unsigned long response_size = 4096; int ret = 0; char *recv_buff = (char *)malloc(response_size); while (1) { // ret = recv(socket, recv_buff, response_size, MSG_WAITALL); // cannot get all data ret = read(socket, recv_buff, response_size); // same effect as the above recv_size += ret; if (ret < 0) error(strerror(errno)); else if (ret == 0) break; // all data recved }
The normal result of my test with burpsuit is this.
But what I received with the C language program was incomplete data.
I searched the reason for one night, but I still did not find a solution for my problem. Whether it is to set the buff to a super large size or any other method, the complete data cannot be accepted at all.
The traffic monitored from wireshark is ok, but my program still cannot receive the complete data. What is the problem?
If you know why, please let me know. THX. (o゜▽゜)o☆
UPDATE
The
while
loop will execute twice, and first time the value ofret
is3343
, and second time is0
, so the loop will stop here. -
python3.6: socket.recv() vs socket.recv_into() performance
I've been using
python3.6
to capture a high speed udp stream and experimented with bothsocket.recv()
andsocket.recv_into()
. I expectedrecv_into()
to be faster since it would copy right into a "preallocated"bytearray
instead of creating a new string each time a packet was read and appended to a list.My test scenario is core bound and I know I am dropping some packets and have a large socket receive buffer size via
setsockopt
onSO_RCVBUF
. I also shutoff the garbage collector to avoid random interruptions.The following snippets have similar performance which doesn't make sense to me and was wondering if someone could help point out what I'm missing. Thanks!
pkts = [] while time.time() - t_start < 10.0: pkt = s.recv(2048) pkts.append(pkt) num_recv_captured = len(pkts)
vs.
buffer = bytearray(2048) num_recv_into_captured = 0 while time.time() - t_start < 10.0: s.recv_into(buffer, 2048) num_recv_into_captured += 1
Here I am seeing
num_recv_into_captured
to be similar tonum_recv_captured
in a core bound scenario but expectednum_recv_into_captured
to be quite a bit larger. -
recv() function seems to hang on Client side
I have to design a Spotify - like music streaming app for a college project. However, in order to utilize DirectSound API (which we need to do, don't know why) from the client's side, I need to receive specific data about the file in order to build the secondary buffer, like Average Bytes Per Second, Sample Rate etc. When trying to send the song's average bytes per second for example, the recv() function called from the client and the program seems to stop when calling it. Does anyone have any idea what's happening or how it can be fixed?
P.S. I'm pretty sure the server sends the message since for troubleshooting reasons, I used a cout command after the send() function as you can see below, which actually show the message.
Server side:
std::cout << "Client requested song: " << recvMSG[0] << " info" << std::endl; _playList = GetTheList(); const size_t cSize = strlen(recvMSG) + 1; wchar_t* wc = new wchar_t[cSize]; mbstowcs(wc, recvMSG, cSize); if (((wc == _selectedTrack) || (wc != _selectedTrack)) && (_selectedTrack != L"" || wc != L"")) //if ( ( (recvMSG == _selectedTrack) || (recvMSG != _selectedTrack)) && (_selectedTrack != L"" || recvMSG != L"") ) { std::string::size_type sz; _selectedTrack = wc; int point = std::stoi(_selectedTrack, &sz) - 1; //recvMSG; //Call method to get the file size in bytes and prepare it for streaming std::wstring name = getFileName(point, "C:\\Users\\User\\Documents\\Storage\\*.wav"); DirectSoundLoad mainLoader(name); std::string trackInfo = "\nFile name: "; std::cout << trackInfo; std::wcout << name << endl; trackInfo = "Average Bytes Per Second:" + std::to_string(mainLoader.getAvgBPS()); std::cout << trackInfo << endl; int songData = send(_clientSocket, (std::to_string(mainLoader.getAvgBPS())).c_str(), 1024, 0); std::cout << "Sent song Data!" << std::endl; } else { std::string trackInfo = "\nSorry, this file doesn't exist in the list anymore."; send(_clientSocket, trackInfo.c_str(), 1024, 0); } }
Client side:
if (_storedList.empty() == false) { int userinput = std::stoi(_userinput); bool isIndexed = false; for (int i = 0; i < _storedList.size(); i++) { if (userinput == i) { isIndexed = true; } } if (isIndexed) { for (int i = 0; i < _totalCount; i++) { if (userinput == i) { _selectedTrack = _storedList[i]; break; } } std::wcout << "You selected: " << _selectedTrack << std::endl; std::cout << "Preparing it for playback..." << std::endl; std::string input = _userinput; std::string select; int selection; if (CheckForNumberInput(input)) { for (int i = 0; i < _storedList.size(); i++) { int temp = std::stoi(input); if (temp == i) { _selectedTrack = _storedList[i]; selection = temp + 1; select = std::to_string(selection); //std::cout << "You chose track: " << temp << std::endl; } } int len = send(_activeSocket, select.c_str(), 1024, 0); if (len == SOCKET_ERROR) { std::cout << "len socket error: " << WSAGetLastError() << std::endl; _isActive = 0; } } Sleep(1000); } else { std::cout << "Wrong Index!" << std::endl; } Sleep(1000); } else { std::cout << "\nThe list is empty!\n" << std::endl; SendRequests(0); }
Client Proceeds to this:
if (choice == 1) { std::cout << "\nPress 'H' to send a Hello Message!" << std::endl; std::cout << "Press 'L' to download the Latest track List." << std::endl; std::cout << "Press 'V' to view the current list" << std::endl; std::cout << "Press 'N' to select the number of a corresponding track for playback." << std::endl; std::cout << "Press 'Q' to quit the program." << std::endl; } else if (choice == 2) { getSongData(); //int songData = recv(_activeSocket, recvMSG, 1024, NULL); std::cout << "Average Bytes Per Second:" << std::endl; std::cout << "Press 'P' to play the selected track." << std::endl; std::cout << "Press 'D' to pause the playback or 'P' to resume." << std::endl; std::cout << "Press 'S' to stop the playback." << std::endl; std::cout << "Press 'Q' to quit the program.\n" << std::endl; } else { choice = 1; }
Where choice == 2 and getSongData() is:
int i = 0; std::cout << "Waiting message..." << std::endl; do { char recvMSG[1024]; int length = recv(_activeSocket, recvMSG, 1024, NULL); //This is the recv() function that hangs if (length <= 0) { std::cout << "\nReceive has failed. Error that occured: " << WSAGetLastError() << std::endl; _isActive = 0; break; } std::cout << "Message recieved:" << recvMSG[0] << std::endl; i++; } while (i < 1);
Thanks in advance if you take the time to read and answer.