≡ Menu

SparkException File ./myapplication.jar exists and does not match contents of

I got this exception when I was running an EMR application
org.apache.spark.SparkException: File ./myapplication.jar exists and does not match contents of
If you run into this exception, the issue could be that the nodes have run out of disk space on one of the partitions as I found it out in my case.

When I got this problem, I did df -h on the nodes and found that the root partition is full.

[hadoop@myemr-node ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      9.8G  9.8G     0 100% /
devtmpfs         34G   20K   34G   1% /dev
tmpfs            34G     0   34G   0% /dev/shm
/dev/xvdb       840G  2.1G  838G   1% /mnt
/dev/xvdc       840G  1.5G  839G   1% /mnt1

Finding out the usage of each folder in ‘/’ shows /home/hadoop consuming 6.8GB space, however du on /home/hadoop does not show anything interesting.

[hadoop@myemr-node ~]$ du -sh /home/hadoop/*
0       /home/hadoop/bin
0       /home/hadoop/Cascading-2.5-SDK
0       /home/hadoop/conf
12K     /home/hadoop/contrib
0       /home/hadoop/etc
4.0K    /home/hadoop/hadoop-examples.jar
0       /home/hadoop/hive
572K    /home/hadoop/lib
0       /home/hadoop/libexec
0       /home/hadoop/mahout
0       /home/hadoop/pig
0       /home/hadoop/sbin
0       /home/hadoop/scala
0       /home/hadoop/share
0       /home/hadoop/shark
0       /home/hadoop/spark

Then I looked up to find if there are any hidden files/folders

[hadoop@myemr-node ~]$ ls -a
.  ..  .bash_history  .bash_profile  .bashrc  bin  Cascading-2.5-SDK  conf  contrib  etc  hadoop-examples.jar  hive  lib  libexec  mahout  pig  sbin  scala  share  shark  spark  .ssh  .versions
[hadoop@myemr-node ~]$ du -sh .versions
6.8G .versions
[hadoop@myemr-node work]$ pwd
[hadoop@myemr-node work]$ du -sh *
276M    app-20151015043847-0012
550M    app-20151016053316-0013
550M    app-20151016054953-0014
276M    app-20151016055230-0015
275M    app-20151016060102-0016
276M    app-20151016060322-0017
276M    app-20151016171448-0018
276M    app-20151016172752-0019
275M    app-20151017180735-0020
276M    app-20151017180915-0021
275M    app-20151017182043-0022
276M    app-20151017182142-0023
276M    app-20151017182907-0024
276M    app-20151017183526-0025
276M    app-20151017184015-0026
276M    app-20151017184912-0027
276M    app-20151017185454-0028
44M     app-20151017190743-0029

Cleaning up these files has solved the problem. We need to do this on each of the clusters.

References: Stackoverflow question

{ 0 comments… add one }

Leave a Comment