Cannot pass variables to a spark sql query in pyspark

Question

I have a python variable (I am using pyspark) of date datatype: The variable value is 2016-10-31

print type(load_dt)

 >> <type 'datetime.date'>

I am having difficulty passing this to a sparksql query:

    hive_context.sql("select * from  tbl t1 where cast (substring(t1.dt,1,10) as date) ={0}".format(load_dt));

    Error:

    u"cannot resolve '(cast(substring(dt,1,10) as date) = ((2016 - 10) - 31))' due to data type mismatch: differing types in '(cast(substring(period_dt,1,10) as date) = ((2016 - 10) - 31))'
 (date and int)

Alper t. Turker · Accepted Answer · 2018-01-03 18:49:38Z

2

Add quotes:

"select * from  tbl t1 where cast (substring(t1.dt,1,10) as date) = '{0}'"

otherwise date is converted to 2016-10-31 string and interpreted as arithmetic expression:

2016 - 10 - 31

answered Jan 3, 2018 at 18:49

Alper t. Turker

35.3k9 gold badges89 silver badges118 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Cannot pass variables to a spark sql query in pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related