Read the data from HDFS using Scala

Question

I am new to Scala. How can I read a file from HDFS using Scala (not using Spark)? When I googled it I only found writing option to HDFS.

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.PrintWriter;

/**
* @author ${user.name}
*/
object App {

//def foo(x : Array[String]) = x.foldLeft("")((a,b) => a + b)

def main(args : Array[String]) {
println( "Trying to write to HDFS..." )
val conf = new Configuration()
//conf.set("fs.defaultFS", "hdfs://quickstart.cloudera:8020")
conf.set("fs.defaultFS", "hdfs://192.168.30.147:8020")
val fs= FileSystem.get(conf)
val output = fs.create(new Path("/tmp/mySample.txt"))
val writer = new PrintWriter(output)
try {
    writer.write("this is a test") 
    writer.write("\n")
}
finally {
    writer.close()
    println("Closed!")
}
println("Done!")
}

}

Please help me.How can read the file or load file from HDFS using scala.

What did you try so far, e.g. with hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/… ? — Reactormonk
– Reactormonk, Commented Jan 11, 2017 at 10:17
We have elected for small files to copy from HDFS to local file system and process there SEQUENTIALLY. — Ged
– Ged, Commented Jan 21, 2018 at 18:12

吴武辉 · Accepted Answer · 2021-02-18 17:02:46Z

22

One of the ways (kinda in functional style) could be like this:

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
import java.net.URI
import scala.collection.immutable.Stream

val hdfs = FileSystem.get(new URI("hdfs://yourUrl:port/"), new Configuration()) 
val path = new Path("/path/to/file/")
val stream = hdfs.open(path)
def readLines = Stream.cons(stream.readLine, Stream.continually( stream.readLine))

//This example checks line for null and prints every existing line consequentally
readLines.takeWhile(_ != null).foreach(line => println(line))

Also you could take a look this article or here and here, these questions look related to yours and contain working (but more Java-like) code examples if you're interested.

edited Feb 18, 2021 at 17:02

吴武辉

31 silver badge2 bronze badges

answered Jan 12, 2017 at 15:07

solar

1,3641 gold badge8 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sergey Zakharov Over a year ago

What is URI? How do I import this?

Sergey Zakharov Over a year ago

I got this working by importing URI as import java.net.URI and setting it to hdfs:/// as my Scala service is running in the same node as the HDFS nameserver host.

programort Over a year ago

The method readLine is now deprecated, you should use val bufferedReader = new BufferedReader(new InputStreamReader(stream)) on top of the stream object (source and example of use: hadoopandspark.wordpress.com/2018/04/23/… )

Collectives™ on Stack Overflow

Read the data from HDFS using Scala

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related