How do I use Boolean logic within a pyspark dataframe for sets

Question

I'm trying to create a new column in a pyspark dataframe that is predicated on the contents of another column. The other column has all integers, and I want the new column to be encoded with either 1's or 0's.

import pyspark.sql.functions as F
df2 = df2.withColumn('Industrial', F.when(F.col('CODE') in (1,2,3,4), 1).otherwise(0))

This doesn't work since it wants just Boolean logic. Is there a work around for this?

EDIT: Could still be useful for others since it creates a new column and does a little more than just a check of isin().

Possible duplicate of Filtering a Pyspark DataFrame with SQL-like IN clause — pault
– pault, Commented May 29, 2019 at 19:15

Akhil Batra · Accepted Answer · 2019-05-29 19:13:08Z

1

Use col.isin method

df2 = df2.withColumn('Industrial', F.when(F.col('CODE').isin((1,2,3,4)), 1).otherwise(0))

answered May 29, 2019 at 19:13

Akhil Batra

6106 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How do I use Boolean logic within a pyspark dataframe for sets

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related