Spark: retrieving old values of rows after casting made invalid input nulls

I am having trouble retrieving the old value before a cast of a column in spark. initially, all my inputs are strings and I want to cast the column num1 into a double type. However, when a cast is done to anything that is not a double, spark changes it to null.

Currently, I have dataframes


num1 unique_id
1 id1
a id2
2 id3

and a copy of df1: df1_copy where the cast is made.

when running

df1_copy ='num1').cast('double'), df1_copy.col('unique_id'))

it returns df1_copy:

num1 unique_id
1 id1
null id2
2 id3

I have tried putting it into a different dataframe using select and when but get an error about not being able to find the column num1. The following is what I tried:

df2 ="unique_id").equalTo(df1.col("unique_id")),df1.col('num1)).alias("invalid"), df1_copy.col('unique_id'))
How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum