Apache Spark: java.lang.NoSuchMethodError .rddToPairRDDFunctions

Question

sbt package runs just fine, but after spark-submit I get the error:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.rddToPairRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;Lscala/reflect/ClassTag;Lscala/math/Ordering;)Lorg/apache/spark/rdd/PairRDDFunctions; at SmokeStack$.main(SmokeStack.scala:46) at SmokeStack.main(SmokeStack.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Here is the offending line:

val sigCounts = rowData.map(row => (row("Signature"), 1)).countByKey()

rowData is an RDD Map[String, String]. "Signature" key exists in all items in the map.

I suspect this may be a build issue. Below is my sbt file:

name := "Example1"
version := "0.1"
scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
scalacOptions ++= Seq("-feature")

I'm new to Scala so maybe the imports are not correct? I have:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import scala.io.Source

That's fairly old Spark version. Try to use a bit newer one. And as mentioned above - your runtime and compile time versions of spark do not match. — JiriS
– JiriS, Commented Nov 2, 2016 at 20:22
@maasg I changed compile and runtime Spark versions to match and everything works. Can you put this as the answer so I can accept it? — Daniel Kats
– Daniel Kats, Commented Nov 2, 2016 at 20:45

maasg · Accepted Answer · 2016-11-02 22:38:47Z

35

java.lang.NoSuchMethodError is often an indication that the version the code was compiled against is on a higher version than the libraries used at runtime.

With Spark, that means that the Spark version used to compile is different from the one deployed (on the machine or cluster).

Aligning the versions between development and runtime should solve this issue.

answered Nov 2, 2016 at 22:38

maasg

37.5k14 gold badges91 silver badges117 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Shrikant Prabhu · Accepted Answer · 2017-09-07 09:11:15Z

2

I Was facing the same problem while reading a simple oneline json file into a dataframe and showing it using .show() method. I would get this error on myDF.show() line of code.

For me it turned out to be wrong version of spark-sql library in the build.

i.e. I was having in my External Libraries from SBT , instead of .

Adding following line to my build.sbt resolved the issue

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.0"

answered Sep 7, 2017 at 9:11

Shrikant Prabhu

7458 silver badges13 bronze badges

Comments

Jake · Accepted Answer · 2018-10-09 05:31:03Z

1

If updating the version of one spark dependency, it is safest to update them all to the same version

answered Oct 9, 2018 at 5:31

Jake

4,68810 gold badges44 silver badges89 bronze badges

Collectives™ on Stack Overflow

Apache Spark: java.lang.NoSuchMethodError .rddToPairRDDFunctions

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related