Abnormal event detection has been a challenge due to the lack of complete normal
information in the training data and the volatility of the definitions of both normality
and abnormality. Recent research applying sparse representation has shown its
effectiveness in the expression of normal patterns. Despite progress in this area, the relationship of atoms within the dictionary is commonly neglected, thereafter anomalies which are detected based on reconstruction error could brings high false alarm - noise or infrequent normal visual features could be wrongly detected as anomalies, especially when the training data is only a small proportion of the surveillance data. Therefore, we propose behavior-specific dictionaries (BSD) through unsupervised learning, pursuing atoms from the same type of behavior to represent one behavior dictionary. To further improve the dictionary by introducing information from potential infrequent normal patterns, we refine the dictionary by searching ‘missed atoms’ that have compact coefficients. Experimental results show that our BSD algorithm outperforms state-of-the-art dictionaries in abnormal event detection on the public UCSD dataset. Moreover, BSD has less false alarms compared to state-of-the-art dictionaries especially when the training set is small, which is demonstrated on Anomaly Stairs dataset.