How to disable Garbage collection error in hadoop?



So I'm running a hadoop program, which repeatedly gives me this error



java.lang.OutOfMemoryError: GC overhead limit exceeded


From my knowledge, this error occurs because too much time is being spent in garbage collection. However, you can suppress this error by using the argument.



-XX:-UseGCOverheadLimit


Now, my question is, where do I give this argument in the hadoop framework. I guess I have to provide it in mapred-site.xml. But how? Currently the mapred-site.xml file looks like as shown below. How should I add that -XX option there? Should I just change -Xmx16384m to -Xmx16384m -XX:-UseGCOverheadLimit?



<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx16384m</value>
</property>
</configuration>

No comments:

Post a Comment