Enhancing high-vocabulary image annotation with a novel attention-based pooling

Ali Salar & Ali Ahmadi

---

Journal: Visual Computer
Accepted 22 August 2024
DOI: 10.1007/s00371-024-03618-6

---

Abstract. Given an image, we aim to assign a set of semantic labels to its visual content automatically. This is generally known as automatic image annotation (AIA). Images contain objects that can vary in size and position, with some only taking up a small region of the entire picture. The rise in the number of object classes also heightens this variety. Despite the achievement of promising results, the majority of the current methods have limited efficacy in the detection of small-scale objects. To make more effective use of spatial data compared to the global pooling method, we propose a modified transformer decoder layer that decreases computational complexity without sacrificing model performance. The study has conducted multiple experiments on four datasets, including three high-vocabulary small-scale datasets (Corel 5 k, IAPR TC-12, and Esp Game) and one large-scale dataset (Visual Genome) with a vocabulary list of 500 words. In comparison with existing state-of-the-art models, our approach achieves comparable results in F1-score, N+, and mean average precision (mAP) on small- and large-scale datasets.

Keywords: Image annotation, multi-label classification, large-scale categories, transformer decoder, cross-attention

Improving loss function for deep convolutional neural network applied in automatic image annotation

Ali Salar & Ali Ahmadi

---

Journal: Visual Computer
Accepted 12 April 2023
DOI: 10.1007/s00371-023-02873-3

---

Abstract. Automatic image annotation (AIA) is a mechanism for describing the visual content of an image with a list of semantic labels. Typically, there is a massive imbalance between positive and negative tags in a picture - in other words, an image includes much fewer positive labels than negative ones. This imbalance can negatively affect the optimization process and diminish the emphasis on gradients from positive labels during training. Although traditional annotation models mainly focus on model structure design, we propose a novel unsymmetrical loss function for a deep convolutional neural network (CNN) that performs differently on positives and negatives, which leads to a reduction in the loss contribution from negative labels and also highlights the contribution of positive ones. During the annotation process, we specify a threshold for each label separately based on the Matthews correlation coefficient (MCC). Extensive experiments on high-vocabulary datasets like Corel 5k, IAPR TC-12, and Esp Game reveal that despite ignoring the semantic relationships between labels, our suggested approach achieves remarkable results compared to the state-of-the-art automatic image annotation models.

Keywords: Image annotation, deep learning, unsymmetrical loss function, threshold estimation

Enhancing Facial Privacy Protection via Weakening Diffusion Purification

Ali Salar, Qing Liu, Yingli Tian, Guoying Zhao

---

Conference: CVPR
Accepted 26 February 2025
ArXiv: arxiv

---

Abstract. The rapid growth of social media has led to the widespread sharing of individual portrait images, which pose serious privacy risks due to the capabilities of automatic face recognition (AFR) systems for mass surveillance. Hence, protecting facial privacy against unauthorized AFR systems is essential. Inspired by the generation capability of the emerging diffusion models, recent methods employ diffusion models to generate adversarial face images for privacy protection. However, they suffer from the diffusion purification effect, leading to a low protection success rate (PSR). In this paper, we first propose learning unconditional embeddings to increase the learning capacity for adversarial modifications and then use them to guide the modification of the adversarial latent code to weaken the diffusion purification effect. Moreover, we integrate an identity-preserving structure to maintain structural consistency between the original and generated images, allowing human observers to recognize the generated image as having the same identity as the original. Extensive experiments conducted on two public datasets, i.e., CelebA-HQ and LADN, demonstrate the superiority of our approach. The protected faces generated by our method outperform those produced by existing facial privacy protection approaches in terms of transferability and natural appearance.