Android ListView: Return one of multiple views from getChildAt()?

I'm trying to get one specified View from a ListView item that contains more than one View. Right now I have two Views inside of my xml layout file.

playlist_item.xml :

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://ift.tt/nIICcg"
android:orientation="vertical"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:gravity="center" >
<Button 
    android:id="@+id/playlist_button"
    android:layout_width="240dp"
    android:layout_height="80dp"
    android:background="@android:drawable/btn_default"
    android:clickable="false"
    android:focusable="false"
    android:focusableInTouchMode="false"
    android:gravity="left|center_vertical|center_horizontal"
    android:paddingLeft="20dp"
    android:paddingRight="20dp"
    android:textSize="21sp" />
</LinearLayout>

And I've associated the xml layout to a ListView:

    ArrayAdapter<String> playlistAdapter = new ArrayAdapter<String>(context, R.layout.playlist_item, R.id.playlist_button, playlistNames);
        listView.setAdapter(playlistAdapter);

However, I want to get the Button inside of a given ListView item.

Previously I've been able to accomplish this by casting the View obtained from getChildAt() to a Button like so:

Button b = (Button) listView.getChildAt(childIndex);

Unfortunately I can no longer directly cast it to a Button, since now it returns both the Button and the Layout views(?) How could I get only the Button? Thanks!