How to create SQLContext in spark using scala?

Question

I am creating a Scala program to SQLContext using sbt. This is my build.sbt:

name := "sampleScalaProject"

version := "1.0"

scalaVersion := "2.11.7"
//libraryDependencies += "org.apache.spark" %% "spark-core" % "2.5.2"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.5.2"
libraryDependencies += "org.apache.kafka" % "kafka_2.11" % "0.8.2.2"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "1.5.2"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "1.5.2"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.0"

And this is test program:

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext

object SqlContextSparkScala {

  def main (args: Array[String]) {
    val sc = SparkContext
    val sqlcontext = new SQLContext(sc)
  }
}

I am getting below error:

Error:(8, 26) overloaded method constructor SQLContext with alternatives:
  (sparkContext: org.apache.spark.api.java.JavaSparkContext)org.apache.spark.sql.SQLContext <and>
  (sparkContext: org.apache.spark.SparkContext)org.apache.spark.sql.SQLContext
 cannot be applied to (org.apache.spark.SparkContext.type)
    val sqlcontexttest = new SQLContext(sc)

Can anybody please let me know the issue as I am very new to scala and spark programming?

user1613360 · Accepted Answer · 2020-02-18 01:27:34Z

12

For newer versions of Spark (2.0+), use SparkSession:

val spark = SparkSession.builder.getOrCreate()

SparkSession can do everything SQLContext can do but if needed the SQLContext can be accessed as follows,

val sqlContext = spark.sqlContext

edited Feb 18, 2020 at 1:27

user1613360

1,3143 gold badges17 silver badges45 bronze badges

answered Jul 5, 2018 at 7:45

Shaido

28.6k26 gold badges76 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Justin Pihony · Accepted Answer · 2015-12-21 03:23:20Z

6

You need to new your SparkContext and that should solve it

answered Dec 21, 2015 at 3:23

Justin Pihony

67.2k20 gold badges154 silver badges185 bronze badges

Comments

Viraj Wadate · Accepted Answer · 2018-07-05 07:22:13Z

4

Simply we can create SQLContext in scala

scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc);

answered Jul 5, 2018 at 7:22

Viraj Wadate

6,2832 gold badges35 silver badges30 bronze badges

Comments

Ashutosh Shukla · Accepted Answer · 2018-02-11 10:41:51Z

0

val conf = new SparkConf().setAppName("SparkJoins").setMaster("local")
val sc = new SparkContext(conf);
val sqlContext = new org.apache.spark.sql.SQLContext(sc);

answered Feb 11, 2018 at 10:41

Ashutosh Shukla

5555 silver badges9 bronze badges

Comments

Kuldeep J · Accepted Answer · 2021-02-04 14:08:02Z

0

If you are using scala shell then use below statement

val sqlContext = spark.sqlContext

And to read parquet files use below statement

val df = sqlContext.read.parquet("/path/to/folder/cotaning/arquet/files/")

answered Feb 4, 2021 at 14:08

Kuldeep J

1

Collectives™ on Stack Overflow

How to create SQLContext in spark using scala?

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related