Create a DataFrame using spark from an Excel file which is in server path

Question

I am trying to load a xls/xlsx file from server

With this code https://github.com/crealytics/spark-excel

SparkConf sparkConf = new SparkConf();
SparkContext sparkContext = null;

sparkContext = new SparkContext("local", "234", sparkConf);
SparkSession sparkSession = 
SparkSession.builder().sparkContext(sparkContext).getOrCreate();

SQLContext sqlContext = sparkSession.sqlContext().newSession();
Dataset<Row> dframe = sqlContext.read().format("com.crealytics.spark.excel").option("timestampFormat", "yyyy-mmm-dd HH:mm:ss").option("dataAddress", "'My Sheet'!A1").option("useHeader", "true").option("treatEmptyValuesAsNulls", "false").option("inferSchema", "true")
            .load("/home/test/myfile/sample.xlsx"); // local path

This code perfectly works on local file

"/home/test/myfile/sample.xlsx"

How can i read files on server with path like this

"http://10.0.0.1:8080/serverFiles/test.xlsx"

I treid replacing the code with Server url above and got this error :

 Exception in thread "main" java.io.IOException: No FileSystem for scheme: http
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2586)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2593)

Can spark read xlsx/xls file directly from server URL ? [not for CSV]

Sathiyan S · Accepted Answer · 2019-01-17 12:12:21Z

0

You cannot directly use HTTP paths in your Spark Context.

Refer the this SO-question

answered Jan 17, 2019 at 12:12

Sathiyan S

1,0236 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

KishanCS Over a year ago

How can i do that in java .For xlsx file .I dont want to use CSV

Sathiyan S Over a year ago

here you go! refer it stackoverflow.com/questions/6259339/…

KishanCS Over a year ago

This gives a scanner object which again cannot be loaded using sqlContext

Sathiyan S Over a year ago

you can create a local file/String with this scanner and use parallelize to create RDD and create DF from it...

Sathiyan S Over a year ago

this not possible as far I know.. spark will accept all HDFS compliance data source.. but not from HTTP. That's why I suggested to get the file content to your driver and distribute it..

|

Collectives™ on Stack Overflow

Create a DataFrame using spark from an Excel file which is in server path

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related