How to use Spark-Scala to download a CSV file from the web?

Question

world,

How to use Spark-Scala to download a CSV file from the web and load the file into a spark-csv DataFrame?

Currently I depend on curl in a shell command to get my CSV file.

Here is the syntax I want to enhance:

/* fb_csv.scala
This script should load FB prices from Yahoo.

Demo:
spark-shell -i fb_csv.scala
*/

// I should get prices:
import sys.process._
"/usr/bin/curl -o /tmp/fb.csv http://ichart.finance.yahoo.com/table.csv?s=FB"!

import org.apache.spark.sql.SQLContext

val sqlContext = new SQLContext(sc)

val fb_df = sqlContext.read.format("com.databricks.spark.csv").option("header","true").option("inferSchema","true").load("/tmp/fb.csv")

fb_df.head(9)

I want to enhance the above script so it is pure Scala with no shell syntax inside.

I think your best approach is just to read the URI contents outside of the spark code and convert the resulting sequence into a DataFrame. — The Archetypal Paul
– The Archetypal Paul, Commented Sep 25, 2016 at 9:36
@TheArchetypalPaul I would read it as a stream inside spark. What do you think about that ? — eliasah
– eliasah, Commented Sep 25, 2016 at 10:37
I poked about but couldn't find much detail about that - do you have pointers for how to do this from a URL? Also, these are prices of a single stock, I don't believe the data set will be very large... — The Archetypal Paul
– The Archetypal Paul, Commented Sep 25, 2016 at 11:35
@TheArchetypalPaul I haven't tried if this solution is actually scalable, but I was thinking about something like this :sc.parallelize(scala.io.Source.fromURL("http://ichart.finance.yahoo.com/table.csv?s=FB").getLines.toStream) — eliasah
– eliasah, Commented Sep 25, 2016 at 15:16
Don't lthink a stream adds anything there. I thought you meant spark streaming — The Archetypal Paul
– The Archetypal Paul, Commented Sep 25, 2016 at 15:53

Samar · Accepted Answer · 2016-09-25 09:47:02Z

7

val content = scala.io.Source.fromURL("http://ichart.finance.yahoo.com/table.csv?s=FB").mkString

val list = content.split("\n").filter(_ != "")

val rdd = sc.parallelize(list)

val df = rdd.toDF

answered Sep 25, 2016 at 9:47

Samar

2,1011 gold badge15 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jack · Accepted Answer · 2020-03-27 21:56:39Z

1

Found better answer from Process CSV from REST API into Spark

Here you go:

import scala.io.Source._
import org.apache.spark.sql.{Dataset, SparkSession}

var res = fromURL(url).mkString.stripMargin.lines.toList
val csvData: Dataset[String] = spark.sparkContext.parallelize(res).toDS()

val frame = spark.read.option("header", true).option("inferSchema",true).csv(csvData)
frame.printSchema()

answered Mar 27, 2020 at 21:56

jack

1,98118 silver badges32 bronze badges

Collectives™ on Stack Overflow

How to use Spark-Scala to download a CSV file from the web?

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related