Related questions

Read, then Write CSV with "Non-ISO extended-ASCII" text Encoding

2018-10-09 16:31 canpan14 imported from Stackoverflow

My csv has strings like:

TîezÑnmidnan

I'm trying to use the following below to set up a reader/writer

import csv

# File that will be written to
csv_output_file = open(file, 'w', encoding='utf-8')
# File that will be read in
csv_file = open(filename, encoding='utf-8', errors='ignore')
# Define reader
csv_reader = csv.reader(csv_file, delimiter=',', quotechar='"')
# Define writer
csv_writer = csv.writer(csv_output_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)

Then iterating over the information read in

# Iterate over the rows in the csv
for idx, row in enumerate(csv_reader):
    csv_writer.writerow(row[0:30])

Problem is in my output file I can't get it to show up with that same string. According to my mac, the csv file type has the encoding "Non-ISO extended-ASCII"

I tried various encodings and some would just remove the special characters while others just wouldn't work.

It's weird because I can hard code that string above into a variable and use it without problems, so I assume it's something to do with how I'm reading in the file. If I breakpoint before it writes it shows up as the following in the debugger.

T�ez�nmidnan

I can't convert the file before running it, so the python code has to handle any conversions itself.

The expected output I want would be for it to remain in the output file looking like

TîezÑnmidnan

Edit: chardetect gave me "utf-8 with confidence 0.99"

See also questions close to this topic

  • I am trying to click on something using python selenium but it is not working

    I have used XPath I have done (driver.find_element_by(**)) but it is not working here is the HTML of it 

    <input type="submit" name="commit" value="add to cart" class="button">

    If you could help me it would be greatly appreciated!

  • Getting error when running flake 8 on Atom

    I installed the linter flake 8 package on Atom and i did the pip installation but for some reason linter isn't functioning it always displays the following error:

    Linter] Error running Flake8 See Console for more info. (Open View -> Developer -> Toggle Developer Tools)

  • Extraneous _BaseValue() when handling repeated property in GAE Ndb Datastore

    Say I have an Ndb entity that goes like this:

    class MyEntity(ndb.Model):
    name = StringProperty()
    tags = StringProperty(repeated=True)

    I will create an entity:

    >>> e = MyEntity(name='test')
>>> e.tags = ['aaa', 'bbb', 'ccc']
>>> e.tags
['aaa', 'bbb', 'ccc']
>>> t = e.tags
>>> t
['aaa', 'bbb', 'ccc']

    Seems ok up to that point. Now I put the entity:

    >>> e.put()
>>> t
[_BaseValue('aaa'), _BaseValue('bbb'), _BaseValue('ccc')]

    Strange...

    Now I print the original attribute:

    >>> e.tags
['aaa', 'bbb', 'ccc']

    It's back to normal. Now for variable t:

    >>> t
['aaa', 'bbb', 'ccc']

    Normal too...

    This issue New entity in repeated StructuredProperty stored as a _BaseValue, is similar. It says this is a known bug. But since it is from 2014, I would expect the bug to have been fixed by now.

  • PCA on tsv file using PySpark shows error

    I'm trying to do PCA analysis on a pyspark dataframe read from a tsv file, saved locally, using pyspark. The .tsv file has 16468028 rows and 22 columns. The columns are :

    ['timestamp', 'uid', 'campaign', 'conversion', 'conversion_timestamp', 'conversion_id', 'attribution', 'click', 'click_pos', 'click_nb', 'cost', 'cpo', 'time_since_last_click', 'cat1', 'cat2', 'cat3', 'cat4', 'cat5', 'cat6', 'cat7', 'cat8', 'cat9']

    Here is my code :

    import pyspark
from pyspark.sql import SQLContext
from pyspark.sql.types import StructType, StructField, StringType
from pyspark.ml.feature import PCA as PCAml
from pyspark.ml.linalg import Vectors, VectorUDT, DenseVector  # Pre 2.0 pyspark.mllib.linalg
from pyspark.ml.feature import VectorAssembler, StringIndexer
from pyspark.ml import Pipeline
from pyspark.sql.types import IntegerType
from pyspark.sql.functions import udf

sc = pyspark.SparkContext()
myfile = sc.textFile("pcb_dataset_final.tsv")
sqlContext = SQLContext(sc)
schema2 = "timestamp\tuid\tcampaign\tconversion\tconversion_timestamp\tconversion_id\tattribution\tclick\tclick_pos\tclick_nb\tcost\tcpo\ttime_since_last_click\tcat1\tcat2\tcat3\tcat4\tcat5\tcat6\tcat7\tcat8\tcat9"
schema2 = schema2.split("\t")
schema2 = StructType([StructField(str(i), StringType(), True) for i in schema2])
df3 = sqlContext.createDataFrame(myfile, schema2)
for i in df3.columns:
    df3 = df3.withColumn(i, df3[i].cast(IntegerType()))
as_prob = udf(lambda x: DenseVector([1 - x, x]), VectorUDT())
df3 = df3.withColumn("click", as_prob(df3["click"]))
pca = PCAml(k=2, inputCol= 'click', outputCol="pca")
model = pca.fit(df3)
transformed = model.transform(df3)
result = model.transform(df3).select("pca")
result.show(truncate=False)

    However, it shows this error :

    Traceback (most recent call last):
  File "H:\p36564\proj_pyspark_pca\read_tsv.py", line 88, in <module>
    model = pca.fit(df3)
  File "H:\p36564\lib\site-packages\pyspark\ml\base.py", line 132, in fit
    return self._fit(dataset)
  File "H:\p36564\lib\site-packages\pyspark\ml\wrapper.py", line 288, in _fit
    java_model = self._fit_java(dataset)
  File "H:\p36564\lib\site-packages\pyspark\ml\wrapper.py", line 285, in _fit_java
    return self._java_obj.fit(dataset._jdf)
  File "H:\p36564\lib\site-packages\py4j\java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "H:\p36564\lib\site-packages\pyspark\sql\utils.py", line 63, in deco
    return f(*a, **kw)
  File "H:\p36564\lib\site-packages\py4j\protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o152.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost, executor driver):

    What is wrong with the code ? How should I fit the dataframe into the model?

  • Django 2.1, How to get user fullname order by last_name, first_name in templates?

    First, sorry for my inexperienced English Proficiency.

    In my country use name ==> last_name+first_name

    so, i can't use AbstractBaseUser of {{user.get_full_name}} in templates.

    purchase.html

    <input type="text" readonly name="purchase_name" value="{{user.get_full_name}}">

    how can use order by last_name + first_name?? Not first_name + last_name

    Could you teach me easy solution?