Cannot run spark example on Hadoop Yarn cluster

Getting Started
1

Hi,

I try to run the “pi example” of Spark using my hadoop cluster (5 nodes) but I still have an error message :

appDiagnostics: Application application_1404480833598_0044 failed 2 times due to AM Container for appattempt_1404480833598_0044_000002 exited with exitCode: -1000 due to: java.io.IOException: Resource file:/opt/spark/spark-0.9.1/assembly/target/scala-2.10/spark-assembly-0.9.1-hadoop2.2.0.jar changed on src filesystem (expected 1404744909000, was 1404740234000)

My hadoop cluster is working (I can run mapreduce pi example). I have installed the same version of Spark on each node and I built Spark using “SPARK_HADOOP_VERSION=2.2.0 SPARK_YARN=true clean sbt/sbt assembly”.

(When yarn decide to use only the master node for the spark application, it works, but when others nodes are used for the application, it fails).

The time of creation of the jar file on each computer is different (because of course they have not been installed at the same time).

I use hadoop 2.2.0 and Spark 0.9.1

Thank you to help :smile:

Jo

2

Sorry, but you’re in the wrong place. This forum is for the Spark Core, a WiFi-connected microcontroller platform for building Internet of Things projects.

You’re looking for Apache Spark.

3 Likes
3

I’m starting to think I should be working on becoming a Spark expert as well! :slight_smile:

Thanks,
David

2 Likes