create dataframe from map in spark java API

Question

I am trying to use spark sql in java API, below simple stuff (copied from official guide: https://spark.apache.org/docs/latest/rdd-programming-guide.html) would not make intelij happy.

It complains the ClassTag thing, which i do not know how to create it or let it automatically imported or something.

        List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
        JavaRDD<Integer> distData = sc.parallelize(data);

I understand it wants to use 3 arguments

public <T> RDD<T> parallelize(final Seq<T> seq, final int numSlices, final ClassTag<T> evidence$1) {

but how can i get this evidence$1 thing? the official example did not have the argument either

Please help on this.

soMuchToLearnAndShare · Accepted Answer · 2020-09-13 15:18:56Z

1

I decided to look into the source code of the example came with the official guideline. and turns out it needs to create java spark context I got my one working after I used the example from the guide's source code.

import scala.Tuple2;
import org.apache.spark.api.java.JavaSparkContext;

List<Tuple2<String, String>> data =
        Arrays.asList(
                new Tuple2<>("key1","value1")
                , new Tuple2<>("key2", "value2")
        );

JavaPairRDD<String, String> dataRdd = jsc.parallelizePairs(data);

answered Sep 13, 2020 at 15:18

soMuchToLearnAndShare

1,04515 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

create dataframe from map in spark java API

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related