AWS

SparkException File ./myapplication.jar exists and does not match contents of

by

I got this exception when I was running an EMR application org.apache.spark.SparkException: File ./myapplication.jar exists and does not match contents of //10.28.139.44:33084/jars/myapplication.jar If you run into this exception, the issue could be that the nodes have run out of disk space on one of the partitions as I found it out in my case. When … Read more

Hadoop file system commands

by

If you are working with EMR or hadoop, the following file system command would be handy. List the contents of a directory hadoop fs -ls folderPath Example: To list the contents of the folder hdfs:/input/ hadoop fs -ls /input/data To list all the files recursively in all subfolders hadoop fs -ls -R /input/data To remove … Read more