pyspark UDF with null values check and if statement

Question

This works provided no null values exist in an array passed to a pyspark UDF.

concat_udf = udf(
    lambda con_str, arr: [x + con_str for x in arr], ArrayType(StringType())
)

I am not seeing how we can adapt this with a null / None check with an If. How to adapt the following correctly below that does not work:

concat_udf = udf(lambda con_str, arr: [  if x is None: 'XXX' else: x + con_str for x in arr  ], ArrayType(StringType()))

I can find no such example. if with transform no success either.

+----------+--------------+--------------------+
|      name|knownLanguages|          properties|
+----------+--------------+--------------------+
|     James| [Java, Scala]|[eye -> brown, ha...|
|   Michael|[Spark, Java,]|[eye ->, hair -> ...|
|    Robert|    [CSharp, ]|[eye -> , hair ->...|
|Washington|          null|                null|
| Jefferson|        [1, 2]|                  []|
+----------+--------------+--------------------+

should become

+----------+--------------------+-----------------------+
|      name|knownLanguages|          properties         |
+----------+--------------------+-----------------------+
|     James| [JavaXXX, ScalaXXX]|[eye -> brown, ha...   |
|   Michael|[SparkXXX, JavaXXX,XXX]|[eye ->, hair -> ...|
|    Robert|    [CSharpXXX, XXX]|[eye -> , hair ->...   |
|Washington|                 XXX|                null   |
| Jefferson|        [1XXX, 2XXX]|                  []   |
+----------+--------------+-----------------------------+

Ternary Operators

Steven
– Steven

2020-10-27 15:31:33 +00:00
Commented Oct 27, 2020 at 15:31 — Steven
– Steven, Commented Oct 27, 2020 at 15:31

Steven · Accepted Answer · 2020-10-27 16:13:57Z

1

using ternary operator, I would do something like this :

concat_udf = udf(
    lambda con_str, arr: [x + con_str if x is not None else "XXX" for x in arr]
    if arr is not None
    else ["XXX"],
    ArrayType(StringType()),
)

# OR 

concat_udf = udf(
    lambda con_str, arr: [
        x + con_str if x is not None else "XXX" for x in arr or [None]
    ],
    ArrayType(StringType()),
)

edited Oct 27, 2020 at 16:13

answered Oct 27, 2020 at 15:34

Steven

15.4k7 gold badges49 silver badges80 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Ged Over a year ago

PythonException: An exception was thrown from a UDF: 'TypeError: 'NoneType' object is not iterable'

Steven Over a year ago

@thebluephantom which one supposed to be None ? the array or the content of the array ? This error means arr is None. But you wrote a test case on x in your question.

Ged Over a year ago

missing the else on 2nd one. very finnicky this udf with pyspark

Ged Over a year ago

Edited, that should clarify.

Steven Over a year ago

@thebluephantom Check the edit. It is always the same logic.

Collectives™ on Stack Overflow

pyspark UDF with null values check and if statement

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related