Related questions

Read, then Write CSV with "Non-ISO extended-ASCII" text Encoding

2018-10-09 16:31 canpan14 imported from Stackoverflow

My csv has strings like:

TîezÑnmidnan

I'm trying to use the following below to set up a reader/writer

import csv

# File that will be written to
csv_output_file = open(file, 'w', encoding='utf-8')
# File that will be read in
csv_file = open(filename, encoding='utf-8', errors='ignore')
# Define reader
csv_reader = csv.reader(csv_file, delimiter=',', quotechar='"')
# Define writer
csv_writer = csv.writer(csv_output_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)

Then iterating over the information read in

# Iterate over the rows in the csv
for idx, row in enumerate(csv_reader):
    csv_writer.writerow(row[0:30])

Problem is in my output file I can't get it to show up with that same string. According to my mac, the csv file type has the encoding "Non-ISO extended-ASCII"

I tried various encodings and some would just remove the special characters while others just wouldn't work.

It's weird because I can hard code that string above into a variable and use it without problems, so I assume it's something to do with how I'm reading in the file. If I breakpoint before it writes it shows up as the following in the debugger.

T�ez�nmidnan

I can't convert the file before running it, so the python code has to handle any conversions itself.

The expected output I want would be for it to remain in the output file looking like

TîezÑnmidnan

Edit: chardetect gave me "utf-8 with confidence 0.99"

See also questions close to this topic

  • how to display contents of text file one line at a time via timer using python on windows?

    this is the code.

    def wndProc(hWnd, message, wParam, lParam):
    if message == win32con.WM_PAINT:
        hdc, paintStruct = win32gui.BeginPaint(hWnd)

        dpiScale = win32ui.GetDeviceCaps(hdc, win32con.LOGPIXELSX) / 60.0
        fontSize = 36

        # http://msdn.microsoft.com/en-us/library/windows/desktop/dd145037(v=vs.85).aspx
        lf = win32gui.LOGFONT()
        lf.lfFaceName = "Times New Roman"
        lf.lfHeight = int(round(dpiScale * fontSize))
        #lf.lfWeight = 150
        # Use nonantialiased to remove the white edges around the text.
        # lf.lfQuality = win32con.NONANTIALIASED_QUALITY
        hf = win32gui.CreateFontIndirect(lf)
        win32gui.SelectObject(hdc, hf)

        rect = win32gui.GetClientRect(hWnd)
        # http://msdn.microsoft.com/en-us/library/windows/desktop/dd162498(v=vs.85).aspx
        win32gui.DrawText(
            hdc,
            **'Glory be to the Father, and to the son and to the Holy Spirit.',**
            -1,
            rect,
            win32con.DT_CENTER | win32con.DT_NOCLIP | win32con.DT_VCENTER
        )
        win32gui.EndPaint(hWnd, paintStruct)
        return 0

    .where it says the "glory be to the father.." prayer I would like that string to actually display a few different prayers on a timer. what I mean is I want to save short prayers to a text file and have the line where it says "glory be.." to change to a new prayer every 60 seconds cycling through a few prayers such as the serenity prayer etc.

  • How to plot the frequency of my data per day in an histogram?

    I want to plot the number occurences of my data per day. y represent the id of my data. x represent the timestamp which I convert to time and day. But I can't make the correct plot. import matplotlib.pyplot as plt plt.style.use('ggplot') import time

    y=['5914cce8-fad6-45d1-bec2-e59e62823617', '1c2067e0-5173-4a1d-8a75-b18267ee4598', 'db6830ff-fa9c-4aa5-b71e-f6da9333f357', '672cc9d5-360e-4451-bb7c-03e3d0bd8f0d', 'fb0f8122-fffc-47fe-a87a-b2b749df173b', '558e96ca-0222-40c7-acc0-e444f7663f53', 'c3f86fd5-eac3-48d3-a44c-b325f30b6139', '21dd849f-895f-4cf5-a168-45a4c1a9fbf9', 'e3b4cd56-e291-4671-93b6-d2226ee82ae7', '01346c48-a8c4-43d1-ac02-1efa33ca0f4e', '23b78b0f-85be-4ca7-99f4-1a5add76c12e', 'b1c036c0-0c2b-4170-a170-8fd0add0dec2', '74737546-e9c3-4126-bcb2-4d34503421ca', '342991f5-ec87-4c9d-83eb-9908f3e221aa', '4fdcd83a-eb68-4e26-b79b-753c5e022a4e', 'b7fbeca9-9416-43c4-9e90-9e71acc1eaba', '27c9d358-a3ef-4c69-ba89-eac16d8d3bdb', 'ef982c4b-a115-48a1-aef1-2f672d7f1f00', 'efedede2-9bb4-4c52-98b1-8b03070df3fd', 'eb03ae1b-4cde-409c-8d34-2a16a8be30d2']
x=['1548143296750', '1548183033872', '1548346185194', '1548443373507', '1548446119319', '1548446239441', '1548446068267', '1548445962159', '1548446011209', '1548446259465', '1548446180380', '1548239985290', '1548240060367', '1548240045347', '1547627568993', '1548755333313', '1548673604016','1548673443843', '1548673503914', '1548673563975']

date=[]

for i in x:
    print(i)
    print()
    i=i[:10]
    print(i)
    readable = time.ctime(int(i))
    readable=readable[:10]
    date.append(readable)
print(date)
plt.hist(date,y)
plt.show()
  • mysql.connector.errors.ProgrammingError: Error in SQL Syntax

    I'm using the Python MySQL connector to add data to a table by updating the row. A user enters a serial number, and then the row with the serial number is added. I keep getting a SQL syntax error and I can't figure out what it is.

      query = ("UPDATE `items` SET salesInfo = %s, shippingDate = %s, warrantyExpiration = %s, item = %s, WHERE serialNum = %s")
        cursor.execute(query, (info, shipDate, warranty, name, sn, ))
        conn.commit()

    Error:

    mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE serialNum = '1B0000021A974726'' at line 1

    "1B0000021A974726" is a serial number inputted by the user and it is already present in the table.

  • Set header in python list()

    How can I set header for a python list?

    I'm currently doing it like this:

    df = list()
df.append('header1')

    Just wondering if there's a better solution...

  • Better way to Vlookup

    I would like to know if there is a better alternative to Vlookup to find matches between two cells (or Python Dfs).

    Say I have the below Dfs,enter image description here

    I want my code to check if the values in DF1 was in DF2, If values exactly match OR if the values partially matche return me the value in the DF2.

    Just like the matches in 4th column Row 2,3 returned values.

    Thanks Amigo!

  • How to have topic directive show up in table of contents?

    The topic directive (found here: http://docutils.sourceforge.net/docs/ref/rst/directives.html#topic) is said to be used as a "self-contained section".

    How can I make the title of each topic box show up in the table of contents, like a normal section title would?