AWS EMR cluster with Flink does not run any Jar, instead gives java.lang.NoSuchMethodError

Question

I'm trying to submit a jar as a step in AWS EMR using the following command:

aws emr add-steps --cluster-id j-XXXXXXXXX --steps Type=CUSTOM_JAR,Name=test-job,ActionOnFailure=CONTINUE,Jar=s3://xxxx/frauddetection-0.1.jar,Args=flink-yarn-session,-d

It is a basic EMR cluster with Flink 1.14.2 and JDK 8 Adding this as a step gave the error:

java.lang.NoSuchMethodError:org.apache.commons.math3.stat.descriptive.rank.Percentile.withNaNStrategy

I checked the Jar and apache commoms library is present in it. JDK and Flink version on my local machine are the same as that on EMR and the jar runs without any issue locally

Has anyone figured out what causes this?

Modestas Jurčius
– Modestas Jurčius

2022-07-24 17:29:23 +00:00
Commented Jul 24, 2022 at 17:29 — Modestas Jurčius
– Modestas Jurčius, Commented Jul 24, 2022 at 17:29

kkrugler · Accepted Answer · 2022-04-03 14:42:29Z

1

One possibility is that a different (typically older) version of the Apache Commons Math3 jar is part of the AWS EMR installation, and so that's what is getting picked up first on the classpath.

You could try shading those classes when you build your jar.

answered Apr 3, 2022 at 14:42

kkrugler

9,5506 gold badges26 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

AWS EMR cluster with Flink does not run any Jar, instead gives java.lang.NoSuchMethodError

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related