XML : Prevent disappearing of a Layout after a while

I have two Layouts. When I want to close Layout B ,Layout A should appear.Everything works fine till now.

But when I let my app running in the background for a while and try to get from Layout B to A again , Layout A is not appearing and it shows me the Android start menu. Im closing my Layout B with the normal method finish(); if this helps.

How can I prevent this ? Is Android trying to recycle my Layout A after a while ?