Parse date in pyspark without udf

I am trying to parse a date column in pyspark by replacing dd/mm/yyyy with yyyy-mm-dd.

import pyspark.sql.functions as F
spark = SparkSession.builders.appName('test').getOrCreate()
sc = spark.sparkContext
sqlc = pyspark.sql.SQLContext(sc)

df = sqlc.createDataFrame([('01/01/2018','user1'),('28/02/2017','user2')], ['Date','user'])
df.show()
+----------+-----+
|      Date| user|
+----------+-----+
|01/01/2018|user1|
|28/02/2017|user2|
+----------+-----+

What I've done so far is:

df.select( F.concat_ws('-',F.split(F.col('Date'),'/')).alias('Date_parsed')).show()
+-----------+
|Date_parsed|
+-----------+
| 01-01-2018|
| 28-02-2017|
+-----------+

What I would like to obtain is:

+-----------+
|Date_parsed|
+-----------+
| 2018-01-01|
| 2017-02-28|
+-----------+

Any idea how to do this without using a udf?

1 answer

  • answered 2018-10-22 09:28 Ali Yesilli

    You can use sql functions for this case

    >>> import pyspark.sql.functions as F
    >>> 
    >>> df.show()
    +----------+-----+
    |      Date| user|
    +----------+-----+
    |01/01/2018|user1|
    |28/02/2017|user2|
    +----------+-----+
    
    >>> df.withColumn('Date',F.date_format(F.to_date('Date','dd/MM/yyyy'),'yyyy-MM-dd')).show()
    +----------+-----+
    |      Date| user|
    +----------+-----+
    |2018-01-01|user1|
    |2017-02-28|user2|
    +----------+-----+
    

    Update: Note that in some versions of spark (e.g 2.1.1), to_date doesn't take formatting as argument, then you can use F.unix_timestamp to format the date column beforehand:

    df.withColumn('Date',F.date_format(F.to_date(
                F.unix_timestamp(F.col('Date'),'dd/MM/yyyy').cast('timestamp')
                                                 ),'yyyy-MM-dd')).show()