Scroll indicator using fastscrollview and alphaindexer gets stuck when pressed on

I'm using a an AlphaIndexer for whatsapp like scrolling. When I press on the scroll indicator, the scroll gets stuck. But when I scroll by sliding on the screen, or by moving the indicated letter, the scroll works. What could the issue be? Again, emphasizing the problem - the scroll gets stuck only when I press the blue indicator. It starts to works when touched anywhere else on the screen.

fastScrollEnabled and fastScrollAlwaysVisible are both set to true.

Here's the AlphaIndexer:

Adapter(Context context){
    list = new ArrayList<SingleRow>();
    mContext = context;

    for(int i = 0; i< NoteTagFrag.mName.size(); i++){
        list.add(new SingleRow(NoteTagFrag.mName.get(i), NoteTagFrag.mNumber.get(i)));
    }

    alphaIndexer = new HashMap<String, Integer>();
    for (int i = 0; i < NoteTagFrag.mName.size(); i++) {
        String s = NoteTagFrag.mName.get(i).substring(0, 1).toUpperCase();
        if (!alphaIndexer.containsKey(s))
            alphaIndexer.put(s, i);
    }

    Set<String> sectionLetters = alphaIndexer.keySet();
    ArrayList<String> sectionList = new ArrayList<String>(sectionLetters);
    Collections.sort(sectionList);
    sections = new String[sectionList.size()];
    for (int i = 0; i < sectionList.size(); i++)
        sections[i] = sectionList.get(i);
    mInflater = (LayoutInflater) context.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
}

public int getPositionForSection(int section) {
    return alphaIndexer.get(sections[section]);
}

public int getSectionForPosition(int position) {
    for (int i = sections.length - 1; i >= 0; i--) {
        if (position >= alphaIndexer.get(sections[i])) {
            return i;
        }
    }
    return 0;
}

public Object[] getSections() {
    return sections;
}

Here's the xml file containing the ListView

<ListView
    xmlns:android="http://ift.tt/nIICcg"
    xmlns:tools="http://ift.tt/LrGmb4"
    android:id="@+id/main_list_view"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:fastScrollEnabled="true"
    android:scrollbars="vertical"
    android:fastScrollAlwaysVisible="true"
    tools:context=".MainActivity" />