≡ Menu

SparkException File ./myapplication.jar exists and does not match contents of

I got this exception when I was running an EMR application
org.apache.spark.SparkException: File ./myapplication.jar exists and does not match contents of //
If you run into this exception, the issue could be that the nodes have run out of disk space on one of the partitions as I found it out in my case.

When I got this problem, I did df -h on the nodes and found that the root partition is full.

[[email protected] ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      9.8G  9.8G     0 100% /
devtmpfs         34G   20K   34G   1% /dev
tmpfs            34G     0   34G   0% /dev/shm
/dev/xvdb       840G  2.1G  838G   1% /mnt
/dev/xvdc       840G  1.5G  839G   1% /mnt1

Finding out the usage of each folder in ‘/’ shows /home/hadoop consuming 6.8GB space, however du on /home/hadoop does not show anything interesting.

[[email protected] ~]$ du -sh /home/hadoop/*
0       /home/hadoop/bin
0       /home/hadoop/Cascading-2.5-SDK
0       /home/hadoop/conf
12K     /home/hadoop/contrib
0       /home/hadoop/etc
4.0K    /home/hadoop/hadoop-examples.jar
0       /home/hadoop/hive
572K    /home/hadoop/lib
0       /home/hadoop/libexec
0       /home/hadoop/mahout
0       /home/hadoop/pig
0       /home/hadoop/sbin
0       /home/hadoop/scala
0       /home/hadoop/share
0       /home/hadoop/shark
0       /home/hadoop/spark

Then I looked up to find if there are any hidden files/folders

[[email protected] ~]$ ls -a
.  ..  .bash_history  .bash_profile  .bashrc  bin  Cascading-2.5-SDK  conf  contrib  etc  hadoop-examples.jar  hive  lib  libexec  mahout  pig  sbin  scala  share  shark  spark  .ssh  .versions
[[email protected] ~]$ du -sh .versions
6.8G .versions
[[email protected] work]$ pwd
[[email protected] work]$ du -sh *
276M    app-20151015043847-0012
550M    app-20151016053316-0013
550M    app-20151016054953-0014
276M    app-20151016055230-0015
275M    app-20151016060102-0016
276M    app-20151016060322-0017
276M    app-20151016171448-0018
276M    app-20151016172752-0019
275M    app-20151017180735-0020
276M    app-20151017180915-0021
275M    app-20151017182043-0022
276M    app-20151017182142-0023
276M    app-20151017182907-0024
276M    app-20151017183526-0025
276M    app-20151017184015-0026
276M    app-20151017184912-0027
276M    app-20151017185454-0028
44M     app-20151017190743-0029

Cleaning up these files has solved the problem. We need to do this on each of the clusters.

References: Stackoverflow question

{ 0 comments… add one }

Leave a Comment