Force RecyclerView to bottom of page in layout

I have a page that consists of an ImageView, and a RecyclerView. The RecyclerView contains a small number of items (currently three) and only takes up around a quarter of the screen on my test device. However, despite trying numerous layout options, I cannot get the RecyclerView to effectively wrap its content and take up just enough space required to contain these three rows, and leave the rest of the space for my ImageView.

To help illustrate what I mean, I have drawn two diagrams. The first shows what I would like to happen, and the second what is happening:

So far, I have tried several different combinations of RelativeLayout - for instance, I will set RecyclerView to layout:align_ParentBottom and the second RelativeLayout that contains the ImageView to layout:alignComponent so that its bottom matches the RecyclerView top (this would normally drag the ImageView layout so that it fills any reminaing space, which is what I would like to happen.)

The RecyclerView though just keeps occupying all the space it can, even though it only contains a few rows. The current "solution" I have is to place everything inside a LinearLayout and set less gravity to the RecyclerView, but it isn't ideal, because on different emulators it wont line up completely with the bottom of the screen, and in others there isn't enough room and the RecyclerView becomes scrollable.

Thanks in advance for any help and suggestions anyone can offer.