Enhancing high-vocabulary image annotation with a novel attention-based pooling

Ali Salar & Ali Ahmadi

---

DOI: 10.21203/rs.3.rs-3271773/v1

Improving loss function for deep convolutional neural network applied in automatic image annotation

Ali Salar & Ali Ahmadi

---

Journal: Visual Computer
Accepted 12 April 2023
DOI: 10.1007/s00371-023-02873-3

---

Abstract. Automatic image annotation (AIA) is a mechanism for describing the visual content of an image with a list of semantic labels. Typically, there is a massive imbalance between positive and negative tags in a picture - in other words, an image includes much fewer positive labels than negative ones. This imbalance can negatively affect the optimization process and diminish the emphasis on gradients from positive labels during training. Although traditional annotation models mainly focus on model structure design, we propose a novel unsymmetrical loss function for a deep convolutional neural network (CNN) that performs differently on positives and negatives, which leads to a reduction in the loss contribution from negative labels and also highlights the contribution of positive ones. During the annotation process, we specify a threshold for each label separately based on the Matthews correlation coefficient (MCC). Extensive experiments on high-vocabulary datasets like Corel 5k, IAPR TC-12, and Esp Game reveal that despite ignoring the semantic relationships between labels, our suggested approach achieves remarkable results compared to the state-of-the-art automatic image annotation models.

Keywords: Image annotation, deep learning, unsymmetrical loss function, threshold estimation