Change dataType of spark dataframe column in SparkR

Question

I have a spark dataframe.

root
 |-- col1: string (nullable = true)
 |-- Id: string (nullable = true)
 |-- col2: string (nullable = true)
 |-- col3: string (nullable = true)
 |-- col4: string (nullable = true)
 |-- date1: string (nullable = true)
 |-- col5: string (nullable = true)
 |-- date2: string (nullable = true)

I just want to convert the date2 column to date. I used the below code to do that

to_date(myDF$date2)

But the dataframe remains the same. No change in date2 datatype.

How can I change the column date2 to date dataType?

The convention in R is that you need to make an assignment in order to make a transformation "stick". This question does not actually have sufficient information to test any code. — IRTFM
– IRTFM, Commented Apr 5, 2016 at 6:24
That does solve my problem myDF$date2 <- to_date(myDF$date2) — sag
– sag, Commented Apr 5, 2016 at 6:29

Michael Neylon · Accepted Answer · 2016-11-10 02:51:24Z

4

myDF <- withColumn(myDF, "date2", cast(myDF$date2, "date"))

See these pages in the SparkR documentation for the latest version (2.0.1 at the time of writing this):

The cast will not change the existing dataframe, so you need to create a new dataframe or replace the existing dataframe with the newly casted column replacing the old column of the same name.

edited Nov 10, 2016 at 2:51

answered Nov 9, 2016 at 22:36

Michael Neylon

6246 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Change dataType of spark dataframe column in SparkR

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related