Purpose: The purpose of this paper is to improve the robustness of the traditional Bhattacharyya metric for the effect of histogram quantization in the histogram-based visual tracking. However, the traditional Bhattacharyya metric neglects the correlation of crossing-bin and is not robust for the effect of histogram quantization. Design/methodology/approach: In this paper, the authors propose a visual tracking method via crossing-bin histogram Bhattacharyya similarity in the particle filter. Findings: A crossing-bin matrix is introduced into the traditional Bhattacharyya similarity for measuring the reference histogram and the candidate histogram, and the basic tasks of measure such as maximum similarity of self and the triangle inequality are proven. The authors use the proposed measure in the particle filter visual tracking framework and address a model update strategy based on the crossing-bin histogram Bhattacharyya similarity to improve the robustness of visual tracking. Originality/value: In the experiments using the famous challenging benchmark sequences, precision of the proposed method increases by 12.8 per cent comparing the traditional Bhattacharyya similarity and the cost time decreases by 38 times comparing the incremental Bhattacharyya similarity. The experimental results show that the proposed method can track the object robustly and rapidly under illumination change and occlusion.