Data Checklist: On Unit-Testing Datasets with Usable Information (2024)

1 Introduction

Behavioral checklists have been proposed to test the capabilities and limitations of NLP models beyond traditional benchmarks (Ribeiro etal., 2020), and they have been widely used in red teaming (Perez etal., 2022) and human-in-the-loop evaluation (Bhatt etal., 2021).However, model checklists do not address the fact that many bugs observed during evaluation can be ascribed to the dataset that the model is trained or fine-tuned on, suggesting that these problems could be mitigated by audits further upstream.

We turn to the notion of $\mathcal{V}$-information as the primitive for such audits.Written as $I_{\mathcal{V}}(X\to Y)$, this measures how much information an input variable $X$ contains about the output $Y$ that is accessible to a model family $\mathcal{V}$ (Xu etal., 2019).Recently, it has emerged as a popular tool for understanding why datasets are often more simplistic than the real-world tasks they purport to reflect, with dataset difficulty being framed as the lack of usable information (Ethayarajh etal., 2022).This can be used to compare different function families, datasets, data points drawn from the same distribution, and even different features of the input.

In this work, we build on these primitives by introducing a taxonomy of unit tests for data, which explicitly maps common classes of questions about datasets into structured expressions of usable information.One common type of question is whether a particular feature is the only part of the input that is useful.For example, given a toxicity detection dataset, we would want to know whether profane words in the input are sufficient to predict toxicity; that would be undesirable for a toxicity benchmark, as a simple $n$-gram model could solve the task.This question can be articulated more formally as: “Is there any $\mathcal{V}$-information that $X$ contains about $Y$ that is not already contained in feature $\Phi(X)$?”, or as the inequality $\hat{I}_{\mathcal{V}}(X\to Y|\Phi)>0$, where $\Phi$ contains all the profanity in the input $X$.In our taxonomy, this is called an insufficiency test, since $\Phi$ must be insufficient for predicting $Y$.

We formulate a taxonomy of 10 tests to help understand datasets from different angles.Checklists assembled from these tests provide a systematic way to reason about dataset artifacts found in prior literature and allow us to discover artifacts in the more challenging and less well-studied case of preference alignment datasets. We further propose using the checklist to inform data filtering, where we (1) filter out undesirable data instances to mitigate artifacts, and (2) improve efficiency by using less training data, while preserving test performance.

Our main contributions are as follows:

2 Background

3 Data Checklists

Based on $\mathcal{V}$-information, we propose a taxonomy for various types of tests that could be used to understand the difficulty of a given dataset and features of interest. To create these tests, $X$, $Y$, and $\Phi$ are the variables, but the model ($\mathcal{V}$) stays the same. Our proposed tests address common questions about the target dataset. Each unit test is an assertion of a particular inequality: $LHS>\epsilon$ or $LHS<\epsilon$, where $LHS$ is an expression of usable information and $\epsilon$ is a small non-negative tolerance threshold.

We call a collection of these unit tests a data checklist.There are ten tests in our taxonomy—viability, unviability, applicability, inapplicability, non-exclusivity, exclusivity, insufficiency, sufficiency, necessity, and redundancy—which are outlined and described in Table 1.The difference between some of these tests is subtle, and we recommend reading the descriptions carefully.For example, exclusivity is a strictly stronger test than sufficiency, i.e.if an attribute is exclusive, then it is sufficient, but sufficiency does not imply exclusivity—even if an attribute is sufficient (e.g. profanity being sufficient to predict toxicity), it is not necessarily exclusive, because something in its inverse (e.g. sentiment words) can also contain information about the label.Similarly, necessity is a stronger test than applicability.

For all tests, the value of the LHS expression is bounded in $[0,\text{log}|\mathcal{W}|)$, where $W$ is the vocabulary.We have a lower bound of 0, because $\mathcal{V}$-information is formulated as the difference between two entropy values, $H_{\mathcal{V}}(Y)$ and $H_{\mathcal{V}}(Y|X)$, and $H_{\mathcal{V}}(Y)\geq H_{\mathcal{V}}(Y|X)$ since having access to $X$ should never make $Y$ less predictable, due to the optional ignorance property of the model family $\mathcal{V}$.The upper bound of $H_{\mathcal{V}}(Y)$ is the entropy of a perfectly uniform distribution over tokens.

Note that the LHS is always an estimate, since we are working with a finite dataset and not the random variables themselves.Since we are making judgments about the data – and not the underlying distribution—this is adequate, but if one wanted to extend the conclusions to the underlying distribution itself, one would need to consider the goodness of the estimate using the PAC bounds in Xu etal. (2019).Because in practice we are taking an estimate of the $\mathcal{V}$-information, even if the true $\mathcal{V}$-information is 0, our estimate may be slightly above or below that. Thus, we use $\epsilon$ as the tolerance threshold for test failure. We set $\epsilon$ to 0.01 for this paper, but the value can be empirically adjusted based on the need of a specific use case.

4 Dataset Artifacts

In this section, we discuss how the checklist described in Section3 can be used to identify dataset artifacts.Most prior work on dataset artifacts and analytics focuses on NLU datasets such as SNLI (Bowman etal., 2015).The usable information-based checklists, however, can be applied to any dataset.In the following experiments, we frame all tasks as sequence generation problems, using seq2seq models such as T5 (Raffel etal., 2020), or causal language models such as Llama-2 (Touvron etal., 2023). We let $X$ be the input sequence for an example and $Y$ the reference output sequence (beyond $X$).The PVI for each example is calculated using the average log-probabilities for all tokens in $Y$, and the $\mathcal{V}$-information estimate for the dataset is then the average PVI across all examples.

4.2 Discovering Novel Artifacts in Preference Datasets

Given the increasing popularity of large language models and the success of preference alignment techniques such as RLHF (Ouyang etal., 2022) and DPO (Rafailov etal., 2023), we apply our checklists to examine preference datasets, the understanding of which could be otherwise difficult due to the large scale and the subtlety of the preference pairs.

We work with the following three preference datasets:

1. 1.

   SHP³³3https://huggingface.co/datasets/stanfordnlp/SHP, the Stanford Human Preferences Dataset. It contains 385k human preferences from Reddit posts / comments in 18 domains, where a response A is more preferred than B if it is written no later than B and has more upvotes.

2. 2.

   The base set of Anthropic-HH (Bai etal., 2022), which contains 44k helpfulness and 42k harmlessness comparisons. It is collected as natural language dialogues, where crowdworkers write a chat message, models provide two responses, and the crowdworkers are asked to label which one is preferred.

3. 3.

   The UltraFeedback dataset (Tunstall etal., 2023), which contains 64k preference pairs. Its collection is fully automated, where GPT-4 is used to assign a score for each of four model completions from a variety of models.

To compute checklists for the preference datasets, we formulate the following two tasks: (1) preference modeling, where $Y$ is a binary label on which one of the two responses is better; (2) direct alignment of the preferred response, where $Y$ is a sequence of tokens.Note that in general, pairwise preferences are hard to learn and can sometimes even be subjective, so the usable information for the reward modeling task is not as high as that of NLU datasets. Nevertheless, the task is still learnable, with even LMs below 1B parameters passing the viability test.

	Test Type	$\mathcal{V}$-Info	Passes	Implication
SHP	viability	0.117	✓	$\mathcal{V}$ can learn something from SHP.
	applicability	0.066	✓	$\mathcal{V}$ can learn something using length.
	sufficiency	0.069	✗	Some $\mathcal{V}$-information contained in $X$ is not in length.
	exclusivity	0.031	✗	Not all $\mathcal{V}$-information exist exclusively in length.
	necessity	0.073	✓	Length is necessary to extract all $\mathcal{V}$-information in $X$.
HH-helpful	applicability	0.037	✓	$\mathcal{V}$ can learn something using length.
	sufficiency	0.079	✗	Some $\mathcal{V}$-information contained in $X$ is not in length.

Response length.

Models trained with RLHF have been shown to prefer longer responses (Singhal etal., 2023).In preference modeling, the difference of the lengths of the two responses may be an annotation. We trace this observation to the underlying datasets.

We run a full checklist on SHP to examine the impact of response length as an artifact (Table2). We use the difference between the number of words in the chosen and rejected responses as our “length” attribute. For the inverse of the length attribute, we repeat the shorter response so that it has the same length as the longer one, pushing the length difference to zero.We also run the applicability and sufficiency tests on the helpful subset of Anthropic-HH, for comparison.Table2 shows that in both SHP and HH-helpful, response length is applicable but insufficient, meaning that while it is possible to learn something about the preference from response length, length alone does not contain all the information in the text for predicting preference.Further, in SHP, length is not exclusive, as even without a difference in length, the input pairs still contain usable information; length is necessary, however, suggesting that some information only exists in the relative lengths of the paired responses.

The fact that the applicability score of response length is higher in SHP than in HH-helpful reflects the different compositions of the two datasets. SHP is collected from Reddit posts and comments where preferences are determined by user upvotes, while HH-helpful consists of model-generated responses rated by crowdsource workers. Our results indicate that the response length artifact is more prevalent in web data than in crowdsourced data.

Word complexity.

Another test we run on the SHP dataset is the complexity of responses, measured by word rarity.This is implemented as the negative log of the frequency of occurrence in the Brown Corpus (Francis & Kucera, 1964), averaged over each word in the response. While we hypothesized that word complexity would impact preference, our results suggest otherwise—the applicability test shows that word complexity only captures 0.006 bits of T5-usable information, below our threshold $\epsilon=0.01$.However, a more fine-grained analysis on different subsets of SHP shows word complexity passes the applicability test for preference pairs from certain subreddits that require domain-specific knowledge, such as askscience and askcarguys (AppendixA).

Profane words.

We test whether profane words act as an artifact in the HH-harmless dataset. For the direct alignment task, $\Phi$ is the profane words extracted from the query, and the target output is the preferred response. For the preference modeling task, $\Phi$ is the profane words found in the two responses, and we test whether $\Phi$ is applicable or sufficient for predicting the preference.We use 1383 profane words found from an online source⁴⁴4https://www.cs.cmu.edu/~biglou/resources/ for our experiment. In both tasks, profane words are applicable, but not sufficient (Figure2).

Multilinguality in UltraFeedback.

A number of examples in UltraFeedback include non-English components. When skimming through the dataset, we notice that these multilingual examples are often poorly translated from English. To confirm whether the multilingual part of UltraFeedback is useful, we conduct a non-exclusivity test on English words in the dataset⁵⁵5Non-English word extraction is done by the pycld2 package.. The test failed, with non-English tokens containing only 0.0003 bits of T5-usable information, while the viable T5-usable information for UltraFeedback is 0.0247 bits.This suggests that English monolingualism is a largely exclusive attribute, and that the increased alignment of an LLM for non-English outputs is a positive externality of training on English data rather than coming from the multilingual data itself.

Model-extracted attributes.

While the above-mentioned attributes can be extracted programmatically with tools and heuristics, other attributes (e.g. humor) may not be easily extractable. Those attributes may, however, be recognizable and exploitable by large language models such as GPT-4, which we pose as a direction for future work.

5 PVI-based Data Filtering

If we have a failed test that we would like to pass, one natural approach is to filter out examples with undesirable PVI values. Specifically, for tests that require above-$\epsilon$ bits of information, we can remove the low-PVI examples from the training data, in particular those that have negative PVI. For tests that require below-$\epsilon$ bits of information, to make them pass, we would remove the high-PVI examples.While PVI-based filtering may allow us to pass a test, we further demonstrate that such filtering has practical value when training a model on the filtered data and testing on held-out data that has not been filtered.It should be noted that because the model was trained on the unfiltered data, a new $\mathcal{V}$-information estimate cannot be obtained by just averaging over the remaining examples; a new model must be trained if one wants to calculate, precisely, the new PVIs.

Experiment setup.

We experiment with filtering preference datasets using the PVI values of the training data, and align a Mistral-7B (Jiang etal., 2023) model with DPO (Rafailov etal., 2023) on the filtered data, to test the effect of the proposed approach.We show that our checklist is not only a tool for understanding datasets, but can help filter large datasets with redundant data points, and ensures performance with less training data.

Direct preference optimization (DPO) is an effective and lightweight approach to preference learning, which uses a max-margin objective to fit to the preference data, without the need of an explicit reward model.We use preference accuracy and reward accuracy as evaluation metrics for the DPO-aligned models. Reward accuracy is defined as the reward of the chosen response—defined as $\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}$, where $\pi_{\theta},\pi_{\text{ref}}$ are the aligned and reference models—being higher than that of the rejected response, a metric used as the training objective of DPO.Preference accuracy is defined as the log-probability of the chosen response being higher than that of the rejected (i.e., ignoring the reference model).

	Filter Type	# Exs	Reward Acc	Pref. Acc
Harmless	none	43k	72.74%	60.42%
	viability (remove pvi $<$ 0)	35k	73.53%	64.63%
	random	35k	71.40%	60.72%
UF	none	61k	72.35%	55.05%
	viability (remove pvi $<$ 0)	34k	71.00%	55.50%
	english non-exclusivity	52k	70.70%	55.35%

6 Related Work

Training data plays an important role in the performance and robustness of ML algorithms (Jain etal., 2020; Dai & Gifford, 2023). Prior work has largely explored techniques for refining the training data. This is done mainly by training data attribution methods (Koh etal., 2019; Han etal., 2020; Pruthi etal., 2020; Guu etal., 2023, inter alia), identifying influential training examples (Koh & Liang, 2017), or using data curation techniques and algorithms to select specific data subsets that yield comparable performance to training on the entire dataset (Vivek etal., 2024; Gadre etal., 2023; Seedat etal., 2023; Wei etal., 2015; Feldman etal., 2011, inter alia). DSIR (Xie etal., 2023) is a framework for selecting a subset of data from a large raw dataset with a similar distribution to a smaller target dataset by estimating importance weights over a featurization of the two distributions. Sharma etal. (2024) propose quality score to guide data selection in NLP datasets for LM training. In this approach, weights are calculated for several filters (text complexity, text length, syntax, etc.), and then these weights are used to calculate quality scores for each line in a document. Data Maps (Swayamdipta etal., 2020) use the training dynamics of a model to visualize how easily and stably different data points can be predicted; this can be used to identify data points that easy-to-learn, hard-to-learn, and unlearnable.

A well-known issue of NLP models is exploiting spurious patterns in the datasets that make them simpler to solve than the task they purport to reflect (Naik etal., 2018; He etal., 2019; McCoy etal., 2019; Aspillaga etal., 2020).These annotation artifacts have been found to exist in NLU datasets such as SNLI and MultiNLU (Gururangan etal., 2018). Geva etal. (2019) studies annotator bias in multiple datasets and demonstrates that incorporating annotator identifiers as features during training enhances model performance. Gardner etal. (2021) introduced a statistical procedure for detecting artifacts in datasets, alongside theoretical evidence suggesting that large datasets are likely to contain artifacts. Ethayarajh etal. (2022) demonstrated that the majority of usable information in DWMW17 (Davidson etal., 2017) is contained with 50 (potentially) offensive words, meaning that the dataset was far more simple than the community considered it to be.Singhal etal. (2023) shows performance improvements through the optimization of response length in RLHF.

Ribeiro etal. (2020) introduced a checklist for comprehensive behavioral testing of models.Building on the model checklist, ER-TEST (Joshi etal., 2022) proposed to test on unseen datasets and contrast sets in addition to using functional tests. MacAvaney etal. (2022) introduced new diagnostic probes to test the characteristics of neural IR models. Zeno (Cabrera etal., 2023) is an interactive framework for testing model outputs for specific types (or, slices) of inputs.To our knowledge, we are the first to propose checklists for datasets themselves instead of using datasets as a tool for checklisting models.

7 Conclusion

We proposed data checklists, a collection of unit tests for datasets based on the usable information framework.Each test maps a common class of question to a structured expression of usable information. If the estimated amount of information is above or below the corresponding threshold (depending on the test), it passes; otherwise, it fails.We applied the checklist to discover previously known dataset artifacts such as premise-hypothesis overlap in SNLI and lexical bias in hate speech detection; we then analyzed artifacts in LLM preference alignment datasets, including response length, word complexity, profane words, and non-English components.While preference datasets are often used to train large foundation models, we show that smaller models (e.g. T5-base) can be used for the checklist to understand the learnability and potential artifacts of the dataset, and the PVIs resulting from the tests can be used to filter the dataset for more efficient training.

References

• Aspillaga etal. (2020)Carlos Aspillaga, Andrés Carvallo, and Vladimir Araujo.Stress test evaluation of transformer-based models in natural language understanding tasks.In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 1882–1894, Marseille, France, May 2020. European Language Resources Association.ISBN 979-10-95546-34-4.URL https://aclanthology.org/2020.lrec-1.232.
• Bai etal. (2022)Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan.Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.URL https://arxiv.org/abs/2204.05862.
• Bhatt etal. (2021)Shaily Bhatt, Rahul Jain, Sandipan Dandapat, and Sunayana Sitaram.A case study of efficacy and challenges in practical human-in-loop evaluation of NLP systems using checklist.In Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, and Anastasia Shimorina (eds.), Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pp. 120–130, Online, April 2021. Association for Computational Linguistics.URL https://aclanthology.org/2021.humeval-1.14.
• Bowman etal. (2015)SamuelR. Bowman, Gabor Angeli, Christopher Potts, and ChristopherD. Manning.A large annotated corpus for learning natural language inference.In Lluís Màrquez, Chris Callison-Burch, and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642, Lisbon, Portugal, September 2015. Association for Computational Linguistics.doi: 10.18653/v1/D15-1075.URL https://aclanthology.org/D15-1075.
• Cabrera etal. (2023)ÁngelAlexander Cabrera, Erica Fu, Donald Bertucci, Kenneth Holstein, Ameet Talwalkar, JasonI. Hong, and Adam Perer.Zeno: An interactive framework for behavioral evaluation of machine learning.In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA, 2023. Association for Computing Machinery.ISBN 9781450394215.doi: 10.1145/3544548.3581268.URL https://doi.org/10.1145/3544548.3581268.
• Dai & Gifford (2023)Zheng Dai and DavidK Gifford.Training data attribution for diffusion models.arXiv preprint arXiv:2306.02174, 2023.URL https://arxiv.org/pdf/2306.02174.pdf.
• Davidson etal. (2017)Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber.Automated hate speech detection and the problem of offensive language.In Proceedings of the 11th International AAAI Conference on Weblogs and Social Media, ICWSM ’17, 2017.URL https://ojs.aaai.org/index.php/ICWSM/article/view/14955/14805.
• Ethayarajh etal. (2022)Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta.Understanding dataset difficulty with $\mathcal{V}$-usable information.In Kamalika Chaudhuri, Stefanie Jegelka, LeSong, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5988–6008. PMLR, 17–23 Jul 2022.URL https://proceedings.mlr.press/v162/ethayarajh22a.html.
• Feldman etal. (2011)Dan Feldman, Matthew Faulkner, and Andreas Krause.Scalable training of mixture models via coresets.In J.Shawe-Taylor, R.Zemel, P.Bartlett, F.Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume24. Curran Associates, Inc., 2011.URL https://proceedings.neurips.cc/paper_files/paper/2011/file/2b6d65b9a9445c4271ab9076ead5605a-Paper.pdf.
• Francis & Kucera (1964)WNelson Francis and Henry Kucera.A standard corpus of present-day edited american english, for use with digital computers.Brown University, Providence, 2, 1964.
• Gadre etal. (2023)SamirYitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang WeiW Koh, Olga Saukh, AlexanderJ Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt.Datacomp: In search of the next generation of multimodal datasets.In A.Oh, T.Neumann, A.Globerson, K.Saenko, M.Hardt, and S.Levine (eds.), Advances in Neural Information Processing Systems, volume36, pp. 27092–27112. Curran Associates, Inc., 2023.URL https://proceedings.neurips.cc/paper_files/paper/2023/file/56332d41d55ad7ad8024aac625881be7-Paper-Datasets_and_Benchmarks.pdf.
• Gardner etal. (2021)Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and NoahA. Smith.Competency problems: On finding and removing artifacts in language data.In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1801–1813, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.doi: 10.18653/v1/2021.emnlp-main.135.URL https://aclanthology.org/2021.emnlp-main.135.
• Geva etal. (2019)Mor Geva, Yoav Goldberg, and Jonathan Berant.Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1161–1166, Hong Kong, China, November 2019. Association for Computational Linguistics.doi: 10.18653/v1/D19-1107.URL https://aclanthology.org/D19-1107.
• Gururangan etal. (2018)Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and NoahA. Smith.Annotation artifacts in natural language inference data.In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107–112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.doi: 10.18653/v1/N18-2017.URL https://aclanthology.org/N18-2017.
• Guu etal. (2023)Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi.Simfluence: Modeling the influence of individual training examples by simulating training runs.arXiv preprint arXiv:2303.08114, 2023.URL https://arxiv.org/pdf/2303.08114.pdf.
• Han etal. (2020)Xiaochuang Han, ByronC. Wallace, and Yulia Tsvetkov.Explaining black box predictions and unveiling data artifacts through influence functions.In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5553–5563, Online, July 2020. Association for Computational Linguistics.doi: 10.18653/v1/2020.acl-main.492.URL https://aclanthology.org/2020.acl-main.492.
• He etal. (2019)HeHe, Sheng Zha, and Haohan Wang.Unlearn dataset bias in natural language inference by fitting the residual.In Colin Cherry, Greg Durrett, George Foster, Reza Haffari, Shahram Khadivi, Nanyun Peng, Xiang Ren, and Swabha Swayamdipta (eds.), Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pp. 132–142, Hong Kong, China, November 2019. Association for Computational Linguistics.doi: 10.18653/v1/D19-6115.URL https://aclanthology.org/D19-6115.
• Hewitt etal. (2021)John Hewitt, Kawin Ethayarajh, Percy Liang, and ChristopherD Manning.Conditional probing: measuring usable information beyond a baseline.In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1626–1639, 2021.URL https://aclanthology.org/2021.emnlp-main.122/.
• Jain etal. (2020)Abhinav Jain, Hima Patel, Lokesh Nagalapatti, Nitin Gupta, Sameep Mehta, Shanmukha Guttula, Shashank Mujumdar, Shazia Afzal, Ruhi SharmaMittal, and Vitobha Munigala.Overview and importance of data quality for machine learning tasks.In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, pp. 3561–3562, New York, NY, USA, 2020. Association for Computing Machinery.ISBN 9781450379984.doi: 10.1145/3394486.3406477.URL https://doi.org/10.1145/3394486.3406477.
• Jiang etal. (2023)AlbertQ. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, DevendraSingh Chaplot, Diego delas Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, LélioRenard Lavaud, Marie-Anne Lachaux, Pierre Stock, TevenLe Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and WilliamEl Sayed.Mistral 7b, 2023.
• Joshi etal. (2022)Brihi Joshi, Aaron Chan, Ziyi Liu, Shaoliang Nie, Maziar Sanjabi, Hamed Firooz, and Xiang Ren.ER-test: Evaluating explanation regularization methods for language models.In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3315–3336, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.doi: 10.18653/v1/2022.findings-emnlp.242.URL https://aclanthology.org/2022.findings-emnlp.242.
• Koh & Liang (2017)PangWei Koh and Percy Liang.Understanding black-box predictions via influence functions.In Doina Precup and YeeWhye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume70 of Proceedings of Machine Learning Research, pp. 1885–1894. PMLR, 06–11 Aug 2017.URL https://proceedings.mlr.press/v70/koh17a.html.
• Koh etal. (2019)Pang WeiW Koh, Kai-Siang Ang, Hubert Teo, and PercyS Liang.On the accuracy of influence functions for measuring group effects.In H.Wallach, H.Larochelle, A.Beygelzimer, F.d'Alché-Buc, E.Fox, and R.Garnett (eds.), Advances in Neural Information Processing Systems, volume32. Curran Associates, Inc., 2019.URL https://proceedings.neurips.cc/paper_files/paper/2019/file/a78482ce76496fcf49085f2190e675b4-Paper.pdf.
• MacAvaney etal. (2022)Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, and Arman Cohan.ABNIRML: Analyzing the behavior of neural IR models.Transactions of the Association for Computational Linguistics, 10:224–239, 2022.doi: 10.1162/tacl˙a˙00457.URL https://aclanthology.org/2022.tacl-1.13.
• McCoy etal. (2019)Tom McCoy, Ellie Pavlick, and Tal Linzen.Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3428–3448, Florence, Italy, July 2019. Association for Computational Linguistics.doi: 10.18653/v1/P19-1334.URL https://aclanthology.org/P19-1334.
• Naik etal. (2018)Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig.Stress test evaluation for natural language inference.In EmilyM. Bender, Leon Derczynski, and Pierre Isabelle (eds.), Proceedings of the 27th International Conference on Computational Linguistics, pp. 2340–2353, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics.URL https://aclanthology.org/C18-1198.
• Ouyang etal. (2022)Long Ouyang, Jeffrey Wu, XuJiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, PaulF Christiano, Jan Leike, and Ryan Lowe.Training language models to follow instructions with human feedback.In S.Koyejo, S.Mohamed, A.Agarwal, D.Belgrave, K.Cho, and A.Oh (eds.), Advances in Neural Information Processing Systems, volume35, pp. 27730–27744. Curran Associates, Inc., 2022.URL https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
• Perez etal. (2022)Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.Red teaming language models with language models.In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3419–3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.doi: 10.18653/v1/2022.emnlp-main.225.URL https://aclanthology.org/2022.emnlp-main.225.
• Pruthi etal. (2020)Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan.Estimating training data influence by tracing gradient descent.In H.Larochelle, M.Ranzato, R.Hadsell, M.F. Balcan, and H.Lin (eds.), Advances in Neural Information Processing Systems, volume33, pp. 19920–19930. Curran Associates, Inc., 2020.URL https://proceedings.neurips.cc/paper_files/paper/2020/file/e6385d39ec9394f2f3a354d9d2b88eec-Paper.pdf.
• Rafailov etal. (2023)Rafael Rafailov, Archit Sharma, Eric Mitchell, ChristopherD Manning, Stefano Ermon, and Chelsea Finn.Direct preference optimization: Your language model is secretly a reward model.In Thirty-seventh Conference on Neural Information Processing Systems, 2023.URL https://arxiv.org/abs/2305.18290.
• Raffel etal. (2020)Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and PeterJ. Liu.Exploring the limits of transfer learning with a unified text-to-text transformer.J. Mach. Learn. Res., 21(1), jan 2020.ISSN 1532-4435.URL https://dl.acm.org/doi/abs/10.5555/3455716.3455856.
• Ribeiro etal. (2020)MarcoTulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh.Beyond accuracy: Behavioral testing of NLP models with CheckList.In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4902–4912, Online, July 2020. Association for Computational Linguistics.doi: 10.18653/v1/2020.acl-main.442.URL https://aclanthology.org/2020.acl-main.442.
• Seedat etal. (2023)Nabeel Seedat, Jonathan Crabbé, Zhaozhi Qian, and Mihaela vander Schaar.Triage: Characterizing and auditing training data for improved regression.In A.Oh, T.Neumann, A.Globerson, K.Saenko, M.Hardt, and S.Levine (eds.), Advances in Neural Information Processing Systems, volume36, pp. 74995–75008. Curran Associates, Inc., 2023.URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ed687a5f52b651b19e7c18f702907b8b-Paper-Conference.pdf.
• Sharma etal. (2024)Vasu Sharma, Karthik Padthe, Newsha Ardalani, Kushal Tirumala, Russell Howes, HuXu, Po-Yao Huang, Shang-Wen Li, Armen Aghajanyan, and Gargi Ghosh.Text quality-based pruning for efficient training of language models.ArXiv, abs/2405.01582, 2024.URL https://arxiv.org/pdf/2405.01582.
• Singhal etal. (2023)Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett.A long way to go: Investigating length correlations in rlhf, 2023.URL https://arxiv.org/abs/2310.03716.
• Swayamdipta etal. (2020)Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, NoahA. Smith, and Yejin Choi.Dataset cartography: Mapping and diagnosing datasets with training dynamics.In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275–9293, Online, November 2020. Association for Computational Linguistics.doi: 10.18653/v1/2020.emnlp-main.746.URL https://aclanthology.org/2020.emnlp-main.746.
• Touvron etal. (2023)Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, CristianCanton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, PunitSingh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, EricMichael Smith, Ranjan Subramanian, XiaoqingEllen Tan, Binh Tang, Ross Taylor, Adina Williams, JianXiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and ThomasScialom.Llama 2: Open foundation and fine-tuned chat models, 2023.
• Tunstall etal. (2023)Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, AlexanderM. Rush, and Thomas Wolf.Zephyr: Direct distillation of lm alignment, 2023.URL https://arxiv.org/abs/2310.16944.
• Vivek etal. (2024)Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela.Anchor points: Benchmarking models with much fewer examples.In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1576–1601, St. Julian’s, Malta, March 2024. Association for Computational Linguistics.URL https://aclanthology.org/2024.eacl-long.95.
• Wei etal. (2015)Kai Wei, Rishabh Iyer, and Jeff Bilmes.Submodularity in data subset selection and active learning.In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume37 of Proceedings of Machine Learning Research, pp. 1954–1963, Lille, France, 07–09 Jul 2015. PMLR.URL https://proceedings.mlr.press/v37/wei15.html.
• Xie etal. (2023)SangMichael Xie, Shibani Santurkar, Tengyu Ma, and PercyS Liang.Data selection for language models via importance resampling.In A.Oh, T.Naumann, A.Globerson, K.Saenko, M.Hardt, and S.Levine (eds.), Advances in Neural Information Processing Systems, volume36, pp. 34201–34227. Curran Associates, Inc., 2023.URL https://proceedings.neurips.cc/paper_files/paper/2023/file/6b9aa8f418bde2840d5f4ab7a02f663b-Paper-Conference.pdf.
• Xu etal. (2019)Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon.A theory of usable information under computational constraints.In International Conference on Learning Representations, 2019.URL https://arxiv.org/abs/2002.10689.
• Zheng etal. (2023)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, ZiLin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, JosephE. Gonzalez, and Ion Stoica.Judging LLM-as-a-judge with MT-bench and chatbot arena.In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.URL https://arxiv.org/abs/2306.05685.

Appendix A Word complexity applicability per subreddit

Table4 shows that word complexity has higher applicability in certain subreddits than in others.

Appendix B Low viability PVI examples are likely mislabeled

Ethayarajh etal. (2022) computed BERT-usable information on the CoLA grammaticality detection dataset and found that 4 out of the 10 lowest-PVI instances in the test set of CoLA are mislabeled.

In a similar vein, we manually examined 8 lowest-PVI instances in the HH-harmless dataset, with T5-usable information computed from the viability test on the preference prediction task in Section4.2. We find that 3 out of the 8 examples are mislabeled and 3 where the choice is ambiguous.

Table5 shows the top-5 lowest-viability-PVI examples in HH-harmless, 4 of which are mislabeled or ambiguous.

Appendix C Insufficiency of score delta in UltraFeedback

As an additional experiment, we consider the insufficiency test on UltraFeedback with $\Phi$ as score delta, i.e. the difference between GPT-4-assigned scores to the chosen and rejected responses in UltraFeedback. While those scores are used as the ground truth for inferring the preference labels in SHP and should contain all usable information, we would like the input text to still contain usable information when conditioned on $\Phi$. The insufficiency test fails with an average PVI of 0.0026 (not $>\epsilon$) by flan-t5-xl, providing another tool for filtering out examples with low insufficiency PVIs, where the text is less informative.