Listview using fastscrollview and alphaindexer get stuck when pressed on scroll indicator

I'm using a an AlphaIndexer for whatsapp like scrolling. When I press on the scroll indicator, the scroll gets stuck. But when I scroll by sliding on the screen, or by moving the indicated letter, the scroll works. What could the issue be? Again, emphasizing the problem - the scroll gets stuck only when I press the blue indicator. It starts to works when touched anywhere else on the screen.

fastScrollEnabled and fastScrollAlwaysVisible are both set to true.

Here's the AlphaIndexer:

Adapter(Context context){
    list = new ArrayList<SingleRow>();
    mContext = context;

    for(int i = 0; i< NoteTagFrag.mName.size(); i++){
        list.add(new SingleRow(NoteTagFrag.mName.get(i), NoteTagFrag.mNumber.get(i)));
    }

    alphaIndexer = new HashMap<String, Integer>();
    for (int i = 0; i < NoteTagFrag.mName.size(); i++) {
        String s = NoteTagFrag.mName.get(i).substring(0, 1).toUpperCase();
        if (!alphaIndexer.containsKey(s))
            alphaIndexer.put(s, i);
    }

    Set<String> sectionLetters = alphaIndexer.keySet();
    ArrayList<String> sectionList = new ArrayList<String>(sectionLetters);
    Collections.sort(sectionList);
    sections = new String[sectionList.size()];
    for (int i = 0; i < sectionList.size(); i++)
        sections[i] = sectionList.get(i);
    mInflater = (LayoutInflater) context.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
}

public int getPositionForSection(int section) {
    return alphaIndexer.get(sections[section]);
}

public int getSectionForPosition(int position) {
    for (int i = sections.length - 1; i >= 0; i--) {
        if (position >= alphaIndexer.get(sections[i])) {
            return i;
        }
    }
    return 0;
}

public Object[] getSections() {
    return sections;
}