Flink EMR Deployment can't pick up Yarn context and executes only as a local application

Question

I'm deploying my Flink app on AWS EMR 6.2

Flink version: 1.11.2

I configured the step with:

Arguments :"flink-yarn-session -d -n 2"

as described here: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/flink-jobs.html

Both the cluster and the step are in state 'running'.

The controller logs shows:

INFO startExec 'hadoop jar /mnt/var/lib/hadoop/steps/s-309XLKSZGN4V4/trigger-service-***.jar flink-yarn-session -d -n 2'

However, the syslog show that the application didn't pick-up on the Yarn context.

2021-02-24 13:19:57,772 INFO org.apache.flink.runtime.minicluster.MiniCluster (main): Starting Flink Mini Cluster

I pulled out the class name returned by StreamExecutionEnvironment.getExecutionEnvironment in my app, and indeed it is LocalStreamEnvironment.

The application itself runs properly on the JobMaster instance as a local app.

Barak BN · Accepted Answer · 2021-03-06 20:01:07Z

Turns out that the -n is deprecated in new versions of flink, so, per the advice from AWS support, I used this command instead:

flink-yarn-session -d -s 2

Arguments meaning:

-d,--detached Start detached

-s,--slots Number of slots per TaskManager

In addition, In order to make Flink was required create two steps:

--steps '[{"Args":["flink-yarn-session","-d","-s","2"],"Type":"CUSTOM_JAR","MainClass":"","ActionOnFailure":"CANCEL_AND_WAIT","Jar":"command-runner.jar","Properties":"","Name":"Yarn Long Running Session"},{"Args":["bash","-c","aws s3 cp s3://<URL of Application Jar> .;/usr/bin/flink run -m yarn-cluster -p 2 -d <Application Jar>"],"Type":"CUSTOM_JAR","MainClass":"","ActionOnFailure":"CANCEL_AND_WAIT","Jar":"command-runner.jar","Properties":"","Name":"Flink Job"}]'

The first uses command-runner.jar to execute

flink-yarn-session -d -s 2

and the second uses command-runner.jarto download my application Jar from S3 and run the job with flink on yarn-cluster.

Collectives™ on Stack Overflow

Flink EMR Deployment can't pick up Yarn context and executes only as a local application

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related