Generalizability of Mixture of Domain-Specific Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning (2024)

3.1 Evaluation Datasets

Diverse Domain Knowledge.To simulate knowledge diversity, we gather a total of 13 distinct and diverse domain-specific datasets of classification tasks for evaluation.We refer the readers to Appendix A.1 for detailed information and their linguistic statistics.Fig.2 reveals the intricate diversity within our selected datasets, both semantic and topic-wise.Notably, SST2 and IMDB, both originating from the same movie corpus, exhibit proximity in topic embedding spaces. On the contrary, non-formal datasets such as Wiki and Tweets are distinctly distant from other datasets in this regard.We refer the readers to Appendix.A.2 for a detailed exploration and analysis of the topic distributions among the datasets.

3.2 Mixing Fine-Tuned Adapters

Base Models and Individual Adapters.We design our evaluation using two transformer-based models, namely BERTDevlin etal. (2019b) and RoBERTaLiu etal. (2019), with a 3 diverse and well-known adapter methods.They are HoulsbyHoulsby etal. (2019), PfeiferPfeiffer etal. (2021) and LoRAHu etal. (2022).These adapter-based methods introduce variations in the adapter architecture and parameterization (Sec.2), contributing to the comprehensiveness of our analysis.

3.3 Adversarial Text Generation

Toward evaluating the robustness of a mixture of adapters, we employ both black-box and white-box textual attacks to exercise Eq. 2.We utilize the popular TextFooler Jin etal. (2020) as the black-box attack, which aims to replace words with synonyms or contextually similar words to deceive PLMs.We utilize the well-known FGSM Goodfellow etal. (2015) as the white-box attack, which can efficiently craft adversarial examples by perturbing embedding of text data in the direction of the sign of the gradient of the loss function to the input, thereby exposing vulnerabilities in model robustness.

3.4 Combinatory Evaluation

To evaluate in-domain performance for each target domain, we generate all possible combinations of adapters.To illustrate, for the target domain MNLI, we can first evaluate with a mixture of only itself. When combining two adapters, we have the flexibility to choose 1 additional adapter out of the remaining 12, resulting in 12 possible combinations.For a set of 3 adapters, including MNLI, we select 2 adapters out of the 12 to generate $C_{12}^{2}$ combinations.This process continues for sets ranging from 4 to 13 adapters, where, in the case of 13 adapters, all adapters are combined.Thus, we have $13*(1{+}\sum_{i=1}^{12}C_{12}^{i}){=}53,248$ combinations for all domains.We report mean and variance of in-domain performance for each set of mixtures of $k$ adapters.

Notably, this setup already includes all the possible mixtures potentially selected by AdapterSoupChronopoulou etal. (2023), which proposes an additional mechanism to select a subset of a fixed number of domain-specific adapters to mix.

4.1 Implementation Details

For the Ag News, Authorship, Financial, IMDB, Tweets, and Wiki-Toxic, we partition the dataset into three segments with an 8:1:1 for train:val:test splits.For datasets belonging to the GLUE corpus, we employ their public training and evaluation splits. For the black-box TextFooler attack, we set the minimum embedding cosine similarity between a word and its synonyms as 0.8, and the minimum USE similarity is 0.84.For white-box FGSMGoodfellow etal. (2015) attack, we set the perturbation magnitude to $0.01$.Following the setup of Houlsby and PfeifferHoulsby etal. (2019); Pfeiffer etal. (2021), we use all adapters with a dimension of $64$ and $256$ for RoBERTa-large and BERT-base models.With LoRA, we use rank $r{=}4$ following Hu etal. (2022).Detailed training, evaluation dataset, and hyper-parameter configuration for different tasks are presented in Appendix A.3.

4.2 In-Domain Performance Results

Overview. We present the performance of a candidate setting of RoBERTa with Pfeiffer adapterPfeiffer etal. (2021) with different numbers of additional mixing domains in Fig.3.Table 1 shows how much the predictive performance drops without and with adversarial black-box and white-box attacks when mixing all adapters.Overall, the average performance drops over all tasks on the clean test set from the original performance to a mix of all adapters is 11.91%, and that for black-box and white-box attacks are 12.03% and 10.30%, respectively.Further results of generalization and adversarial robustness performance across other models and adapter methods are documented in Appendix A.4.

Finding #1: As we add more tasks or domains, the predictive performance of every single task decreases, reaching its lowest point when we incorporate the maximum of $13$ adapters training on various topic datasets.Fig.3 shows that mixing domain-specific adapters indeed decreases in-domain performance (reduction of around 10% in MRPC, QQP, RTE, etc., and nearly 17% in the Financial domain when mixing 13 adapters).The same behaviors were also observed in Jin etal. (2023b) where they merge the weight of PLMs.Notably, task accuracies decreased at a slower pace for QNLI and SST2 when evaluating with increasing size of mixtures (Fig.3).In contrast, a substantial decrease in accuracy is observed for domains such as RCT, IMDB, Ag News, and Authorship (Fig.3).This shows that mixing domain-specific adapters impair model performance differently depending on the target domain, or “what to mix” in an adapter mixture has a crucial effect on the mixture’s performance.To attempt to explain this behavior, we later present and verify a hypothesis that such mixing domain-specific adapters via weight averaging can result in “forgotten knowledge” that can happen due to the differences in signs when mixing these adapters (Sec.5).

Finding #2: On average, there are no notable differences of the magnitude in accuracy degradation with and without adversarial attacks (12.03% dropped in black-box attack versus 11.91% dropped without attack) (Fig. 3).The overall predictive performance was significantly lower under the white-box compared to the black-box attack as expected because the white-box FGSM attack has additional access to the models’ parameters.Interestingly, on the RCT dataset, the accuracy drop on clean ($21.72\%$) is much larger than on black-box and white-box attacks ($12.32\%$ and $12.53\%$, respectively).

Finding #3: Variance in task accuracy on adversarial attacks, when combining different domain adapters, is observed to be larger than the variance in clean accuracy (Fig. 3).Although the magnitude decrease appears similar in Fig.3, differences in variance (max, min) are discernible among each combination.Specifically, in MRPC, QNLI, QQP, RTE, SST2, Tweets, IMDB, Financial, and Authorship, we can observe that certain mixed models observed slightly better adversarial robustness compared to single adapters (when the number of mixing adapters $k$ is $2$ or $3$).Moreover, the variance of robustness in adversarial scenarios tends to be higher than in clean scenarios because all of the adapters are trained on different tasks and they may exhibit different vulnerabilities to the same attack method.

Finding #4: Mix up to three adapters to maintain competitive performance. Based on the results from Fig.3 and all findings above, it is advisable to mix only up to three tasks to maintain competitive performance (as observed in MRPC, QQP, RTE, Tweet, Financial, and Authorship domains with less than a 3% accuracy drop in accuracy).Especially, in some cases when evaluating QNLI, SST2, mixtures of less than three adapters even achieved better performance in terms of generalizability (up to 1%) and also in terms of adversarial robustness of up to 3% compared to the original performance.

5.1 An Explanation Hypothesis

The ideal scenario when averaging adapter weights during mixing is to make minimal adjustments to their weights, both in terms of values–i.e., magnitudes, and directions, to sustain as much as possible the knowledge learned.Investigating how adapter weights mix regarding both their magnitudes and directions, can be overly complex.Thus, we simplify this assessment by focusing on the sign directions of the adapter weights in our analysis. Following the mixing process of $k$ individual adapter weight in Eq. 1 (Sec.4), we then hypothesize that mixing adapter weights of conflicting signs can result in “forgotten knowledge”, and lead to performance degradation.As illustrated in Fig.1, averaging adapter weights across various tasks may lead to nullifying importance weights for individual tasks if their signs are opposite–i.e., positive v.s. negatives.In other words, the fraction of sign difference or FSD (%) or proportion of weight sign difference in adapter weights during mixing correlate with their mixtures’ generalizability. Alg. 3 (Appendix A.5) shows the calculation of FSD.

We evaluate our hypothesis with different cases: (i) individual adapters (mixture with $k{=}1$), (ii) dual adapters ($k{=}2$), and generalize to (iii) multiple adapters ($k{>}2$).To demonstrate the utility of our hypothesis, we apply it to improve the generalizability of adapter mixtures and also to derive a more effective model pruning in Sec.6, 7.

5.2 Individual Adapters ($k{=}1$)

We calculate the FSD of adapter weights on RoBERTa and normalize it by the total number of weights, denoted as a matrix $\mathbf{S}_{k\times k}$ where each row is the FSD of a single adapter train on task $k$ to the remaining adapters (Fig.4).We refer the readers to Sec.A.6 in the Appendix for results on BERT.Interestingly, a consistent trend in the FSD is observed across various model architectures (BERT, RoBERTa) and adapter methods (e.g., Fig.4 and Fig.9 of AppendixA.6).The reason is that adapters act as small MLP layers that integrate task-specific knowledge into pre-trained models Meng etal. (2022) in different adapter methods.This shared functionality contributes to a similar trend in FSD, highlighting the robustness and generalizability of the observed behavior across different adapters’ architectural variations.In addition, Adapters trained on datasets with distinct topic distributions and cosine similarities (Fig.2) exhibit varying weight directions (Fig.4).Especially, MNLI has a similar linguistic distribution with other datasets (i.e. MRPC, RTE, Ag News, etc) (Fig.2), but the adapter trained on MNLI has a significantly larger FSD compared to the remaining domain-specific adapters (Fig.4).Thus, datasets that are similar in linguistic statistics may not necessarily share the same optimization trajectory.As a result, methods that are based on the linguistic distribution to choose the closest set of adapters to mix like AdapterSoup Chronopoulou etal. (2023) may lead to sub-optimal performance.

5.3 Dual Adapters ($k{=}2$)

Fig.5 shows the FSD for each adapter when mixing two domain-specific adapters.This investigation is conducted within the context of the Pfeiffer AdapterPfeiffer etal. (2021) using a pre-trained RoBERTa model.Overall, there is a strong negative correlation between the FSD and the generalizability–i.e., the lower the sky-blue bar (or the smaller fraction of weight sign conflicts), the higher the performance of the mixture (Fig.5).Notably, tasks with substantial difference in the weight signs witness a pronounced performance decrease.Specifically, RCT exhibits significant performance drops due to substantial differences in adapter weight direction.Conversely, tasks such as MRPC, and QNLI demonstrate either marginal improvement or no change in performance when mixed with other adapters.This is well correlated to the marginal FSD of the mixed adapters, ranging from only 5% to 10% compared to the original weight.

5.4 Multiple Adapters ($k{>}2$)

Similar to the dual-adapter setting, there is still a strong negative correlation between FSD and the generalizability, and increasing the number of mixed adapters amplified the sign disparity (Fig.6).For example, adapters trained on RCT and Tweet exhibit large FSD compared to other adapters and hence observed a significant decrease in generalizability (Fig.6).In some cases, mixing only a few adapters (e.g., 2–4) could still maintain competitive performance as in QNLI, SST2 domains (Fig. 6, 15).This correlates with the relatively marginal differences in adapter weights of QNLI and SST compared to adapters trained on other domains (Fig.4), as their mixtures do not lead to significant cancellations of existing parameters and hence preserve learned knowledge.

5.5 Discussion

Which adapters we should mix? From Sec.5, we find that it is not advisable to merge adapters that are significantly different from each other in weight signs.On the other hand, mixing the weight of these adapters with other adapters which have a small FSD achieves competitive performance (within 3% drop in accuracy) only when the number of mixing adapters is small (QNLI, MRPC in Fig.6) or can achieve better performance, although rarely, compared to training the original adapter as seen in QNLI domain (Fig.5).Therefore, when deploying PLMs, it is prudent to only select a group of tasks with a small FSD to minimize the performance drop in the final mixed model.Our observation is crucial in deploying these models in edge devices where only the adapters are saved on edge, which often has a specific memory capacity limit.

Experiment comparison with AdapterSoup. AdapterSoup Chronopoulou etal. (2023) dynamically selects a set of $l$ adapters during inference. When $l{=}k$, then our experiment setting is the same as the AdapterSoup setting when we use all adapters at once.When $l{=}1$, it is the original performance of a single adapter, assuming that AdapterSoup is perfect at picking the same domain that is already trained on, and this result is already included in our paper (mixture of only $1$ adapter, Fig.3).When $1{<}l{<}k$, we do not have the results of the specific combination that AdapterSoup would select.However, we reported the maximum performance across all combinatorial combinations among 13 domain-specific adapters or each value in Fig.3.For example, when $l{=}3$, we only observe a possible comparative performance (within less than 3% drop) in QNLI, MRPC, and SST2 domains (Fig. 15).We also emphasize that it is one thing to select the best adapters to mix during inference, it is much harder to choose $l$ or how many of them.

7.1 FSD-based Magnitude Pruning

Post-training Pruning.Sparse AdapterHe etal. (2022) employs pruning across every layer of adapters, being able to achieve comparable or even superior predictive performance than standard adapters, even when the sparse ratio reaches up to 80%. By adopting a similar process of adapters pruning after training, with the guidance of our FSD analysis in Sec.5, we can eliminate redundant parameters at an early stage, circumventing the need for a time-consuming iterative pruning process, as discussed in prior works such as Han etal. (2016); Lazarevich etal. (2021). The detailed pruning algorithm is presented in Alg.2.

Generalizability of Pruned Adapters.Fig.8a shows RoBERTa’s performance, where we systematically prune the weight of the Pfeiffer adapter 40%–100% of sparsity. For a single task at the sparsity level $d\%$, we retain only the largest–i.e., the top-$d\%$, influential parameters of the corresponding adapter, and report the average in-domain performance across 13 domains.Remarkably, pruning up to 90% parameters of adapter weight does not lead to performance degradation.This observation suggests redundancy in adapters’ parameters may contribute to the increase in the fraction of weight direction conflicts–i.e., high FSD when merging them (Fig.6).

Generalizability of Mixed Pruned Adapters.Motivated by pruned adapter can still maintain original performance up to 90% of sparsity (Fig. 8a), we hypothesize that mixing sparse adapters may indirectly reduce the fraction of weight sign conflict, therefore, leading to competitive performance with the original adapters.Given a set of domain-specific adapters, based on the FSD, we choose the top $k$ layers with the highest fraction of weight sign difference and prune each layer up to 90% of sparsity. Then we mix these adapters by weight averaging. Details of mixing domain-specific adapters with weight sign conflict information is shown in Alg.4 (Appendix A.8).Fig.8b shows that mixing adapters with 80% or 90% sparsity consistently achieved better performance than the upper-bound empirical accuracy achieved when mixing their dense versions.

A.1 Datasets.

A.2 Topic distribution of training datasets

Tables from 4 to 16 show 10 topics and corresponding important words which are exacted from LDA for each training dataset. Notably, Ag News and SST2 have high cosine similarity but observe a large difference in terms of topic distribution compared to other domains (Fig.2). Therefore, each statistical mechanism like cosine similarity or topic distribution only reflects one aspect of data distribution and may show inconsistencies with each other.

A.3 Hyper-parameter

Training and evaluation datasets.To assess performance in out-of-distribution scenarios, we conduct evaluations on a diverse set of 13 datasets covering various topics, ranging from movie reviews, news, authorship, and healthcare, to non-formal language text such as Wiki Toxic and Tweets.For datasets within the GLUE corpus, we employ training and evaluation datasets to gauge accuracy across different settings.In the case of Ag News, Authorship, Financial, IMDB, Tweets, and Wiki-Toxic, we partition the training set into three segments with an 8:1:1 ratio, utilizing them for training, evaluation, and test datasets, respectively.This approach ensures a comprehensive evaluation of model performance across a wide spectrum of domains and linguistic styles.Table 2 shows data statistics on train/test datasets.

Setting on text adversarial attack.In this study, we employ two types of attacker methods: TextFooler Jin etal. (2020) and FGSM Goodfellow etal. (2015).

TextFooler word-level attacks focus on replacing words within the text with synonyms or contextually similar words.By making ostensibly minor alterations to the input text, these attacks can deceive LLMs into producingincorrect outputs or substantially modifying their predictions.We meticulously fine-tune the hyperparameters of TextFooler to obtain more appropriate synonyms.We set the minimum embedding cosine similarity between a word and its synonyms as 0.8, and the minimum Universal Sentence Encoder similarity is 0.84.

FGSM Goodfellow etal. (2015) is a white-box embedding-level attack.FGSM uses the Fast Gradient Sign Method to calculate gradients of the model’s loss to the input text and generates an adversarial example by perturbing the embedding of input text in the direction that maximizes the loss.We choose the magnitude of the perturbation in embedding space as $0.01$ on BERT and RoBERTa models.

Adapter Configuration.We use adapters with a dimension of 64 and 256 usingRoBERTa-large and BERT-base encoders following the setup of Houlsby etal. (2019), Pfeiffer etal. (2021).With LoRA, we use rank $r=4$ following the setup of Hu etal. (2022).

Hardware Information. We evaluate model performance on AMD Ubuntu 22.04.2 with Ryzen Threadripper PRO 5975WX, 1800MHz, Cached 512 KB and 4 $\times$ GPU Nvidia A6000.Hyper-Parameters.Detailed hyper-parameter configuration for different tasks is presented in Table17.

A.4 Model performance when mixing adapters across tasks

Tables 10, 11, 12 and 13 show task accuracy when mixing multiple adapters.

A.5 Fraction of weight sign difference

Alg.3 presents a detailed algorithm to compute the FSD.

A.6 Additional result on weight sign difference

Fig.9 shows the weight sign difference of the adapter and normalizes it by the total number of adapter weights in BERT.

A.7 Additional results when mixing two adapters

Fig.14 and 15 show model generalization when mixing two and multiple adapters across various tasks.

A.8 Mixing Sparse Adapters with weight sign difference

Alg.4 shows details of mixing sparse adapter with sign conflict information.