publications

2026

1. Preprint

   Wikiskill: Compiling agent experience into persistent knowledge for skill evolution

   Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, and Tu Vu

   In arXiv preprint arXiv:2604.06377, 2026

   Abs arXiv Bib PDF

   Agent skills package specialized knowledge and workflows into reusable resources that extend AI agent capabilities. Recent work automatically discovers such skills from agent experience, which enables agents to progressively adapt through interaction. However, the insights that guide skill development typically remain scattered across optimization histories, limiting their systematic reuse across iterations. We introduce WikiSkill, a framework that co-evolves agent skills with a persistent knowledge base (wiki). At a high level, WikiSkill separates raw execution experience, accumulated knowledge, and executable skills, while continuously consolidating experience into the wiki, which subsequent skill updates can build on. Across diverse benchmarks and models, WikiSkill consistently outperforms state-of-the-art skill-evolution methods and improves over no-skill baselines in most model-benchmark settings. We find that skill evolution complements model scaling: larger models generally benefit more from evolved skills, while smaller models with skills can outperform substantially larger models without them. We also find that evolved skills transfer effectively across models and model families, and skills evolved by other models can outperform self-evolved skills. Finally, our ablation studies confirm that persistent knowledge accumulation in the wiki is critical for effective skill evolution. These results demonstrate the benefits of systematically accumulating and refining agent experience for developing reusable and transferable skills.

   @inproceedings{tang2026wikiskill,
     title = {Wikiskill: Compiling agent experience into persistent knowledge for skill evolution},
     author = {Tang, Liyan and Rashtchian, Cyrus and Ferng, Chun-Sung and Tomkins, Andrew and Juan, Da-Cheng and Vu, Tu},
     booktitle = {arXiv preprint arXiv:2604.06377},
     year = {2026},
     pdf = {https://arxiv.org/pdf/2608.27454},
   }

   // VentureBeat feature // DAIR.AI’s Top AI Papers of the Week
2. COLM

   EvoSkill: Automated Skill Discovery for Multi-Agent Systems

   Salaheddin Alzubi, Noah Provenzano, Jaydon Bingham, Christian Calvo, Weiyuan Chen, and Tu Vu

   In The Third Conference on Language Modeling, 2026

   Abs arXiv Bib PDF

   Coding agents are increasingly used as general-purpose problem solvers, but they often lack the domain expertise required for specialized tasks. Agent skills address this limitation through reusable modules that package instructions, scripts, and resources to augment agents with specialized capabilities. However, these skills are typically hand-crafted, which limits their scalability. We ask whether agents can instead acquire such skills automatically from experience. We introduce EvoSkill, a framework for automated skill acquisition. EvoSkill iteratively analyzes execution failures to identify capability gaps, proposes new skills or refinements to existing ones, and materializes them as structured skill folders. Candidate skills are selected based on held-out validation performance, while model parameters remain fixed. We evaluate EvoSkill on diverse benchmarks spanning financial-document reasoning, web search, multi-source reasoning, and coding (OfficeQA, SealQA, FRAMES, and LiveCodeBench), using models from the Claude, Gemini, Kimi, and gpt-oss families as executors. We show that EvoSkill can substantially improve agent performance across benchmarks, including gains of 15.7 points on SealQA and 11.4 points on OfficeQA. In a controlled comparison, skill evolution with EvoSkill substantially outperforms prompt evolution with GEPA. We further find evidence that evolved skills can transfer across settings: a search skill evolved on SealQA improves BrowseComp accuracy by 5.3 points without modification, while reuse across executor models improves performance in several model pairings, with outcomes depending on model and harness compatibility.

   @inproceedings{alzubi2026evoskill,
     title = {EvoSkill: Automated Skill Discovery for Multi-Agent Systems},
     author = {Alzubi, Salaheddin and Provenzano, Noah and Bingham, Jaydon and Calvo, Christian and Chen, Weiyuan and Vu, Tu},
     booktitle = {The Third Conference on Language Modeling},
     year = {2026},
     url = {https://openreview.net/forum?id=tLvlDr1GWD},
     pdf = {https://openreview.net/pdf?id=tLvlDr1GWD},
   }

   // Pioneering work on skill evolution
3. ICLR

   SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models

   Thinh Pham, Nguyen Phan Nguyen, Pratibha Zunjare, Weiyuan Chen, Yu-Min Tseng, and Tu Vu

   In The Fourteenth International Conference on Learning Representations, 2026

   Abs arXiv Bib PDF

   We introduce SealQA, a challenge benchmark for evaluating SEarch-Augmented Language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors: (1) Seal-0 (main) and (2) Seal-Hard, both of which assess factual accuracy and reasoning capabilities, where Seal-0 targets the most challenging questions that frontier non-reasoning models (e.g., GPT-4.1) answer with near-zero accuracy; and (3) LongSeal, which extends SealQA to test long-context, multi-document reasoning in "needle-in-a-haystack" settings. Our evaluation reveals critical limitations in current models. Even frontier reasoning models face significant challenges across SealQA flavors. On Seal-0, GPT-5 with tools achieves only 43.2% accuracy at its best reasoning effort. We also find that even advanced reasoning models (e.g., DeepSeek-R1) can be vulnerable to noisy search results. Notably, increasing test-time compute does not yield reliable gains across GPT-5 and the o-series of models, with performance often plateauing or even declining early. Finally, while current models are less affected by the "lost-in-the-middle" issue, they still fail to reliably identify relevant documents in LongSeal when faced with numerous distractors. To facilitate future work, we release SealQA at https://huggingface.co/datasets/vtllms/sealqa.

   @inproceedings{pham2026sealqa,
     title = {Seal{QA}: Raising the Bar for Reasoning in Search-Augmented Language Models},
     author = {Pham, Thinh and Nguyen, Nguyen Phan and Zunjare, Pratibha and Chen, Weiyuan and Tseng, Yu-Min and Vu, Tu},
     booktitle = {The Fourteenth International Conference on Learning Representations},
     year = {2026},
     url = {https://openreview.net/forum?id=zWb7ueH16c},
     pdf = {https://openreview.net/pdf?id=zWb7ueH16c},
   }

   // Our benchmark dataset has been used by Google’s Gemini, Qwen, DeepSeek, Kimi, and Perplexity
4. NeurIPS

   The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment

   Rishab Balasubramanian, Pin-Jie Lin, Rituraj Sharma, Anjie Fang, Fardin Abdi, Viktor Rozgic, Zheng Du, Mohit Bansal, and Tu Vu

   In The Fortieth Annual Conference on Neural Information Processing Systems, 2026

   Abs arXiv Bib PDF

   We investigate whether post-trained capabilities can be transferred across model scales without retraining, and propose the Master Key Hypothesis, which states that model capabilities correspond to directions in a low-dimensional latent subspace that induce specific behaviors and are transferable across models through linear alignment. Based on the hypothesis, we introduce UNLOCK, a training-free and label-free framework that extracts a capability direction by contrasting activations between capability-present and capability-absent Source variants, aligns it with a Target model through a low-rank linear transformation, and applies it at inference time to elicit the behavior. Experiments on reasoning behaviors, including Chain-of-Thought (CoT) and mathematical reasoning, demonstrate substantial improvements across model scales without training. For example, transferring CoT reasoning from Qwen1.5-14B to Qwen1.5-7B yields an accuracy gain of 12.1% on MATH, and transferring a mathematical reasoning direction from Qwen3-4B-Base to Qwen3-14B-Base improves AGIEval Math accuracy from 61.1% to 71.3%, surpassing the 67.8% achieved by the 14B post-trained model. Our analysis shows that the success of transfer depends on the capabilities learned during pre-training, and that our intervention amplifies latent capabilities by sharpening the output distribution toward successful reasoning trajectories.

   @inproceedings{balasubramanian2026master,
     title = {The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment},
     author = {Balasubramanian, Rishab and Lin, Pin-Jie and Sharma, Rituraj and Fang, Anjie and Abdi, Fardin and Rozgic, Viktor and Du, Zheng and Bansal, Mohit and Vu, Tu},
     booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems},
     year = {2026},
     url = {https://openreview.net/forum?id=U3GUy7xRmI},
     pdf = {https://arxiv.org/pdf/2604.06377},
   }

   // NeurIPS Spotlight (top 1.3% out of 30.7k submissions)
5. ICML

   On the Generalization Gap in Self-Evolving Language Model Reasoning

   Zhenting Qi, Susanna Maria Baby, Stefanie Anna Baby, Kan Yuan, Andrew Tomkins, Tu Vu, Da-Cheng Juan, and Cyrus Rashtchian

   In Proceedings of the 43rd International Conference on Machine Learning, 2026

   Abs arXiv Bib PDF

   Recent work suggests that large language models (LLMs) can improve through self-evolution (SE), using supervision signals generated by the model itself. In this work, we ask: under a strict closed-loop setup, where the SE algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training? We analyze four representative strategies in a unified offline self-evolution framework, including single-round verification, multi-turn revision with feedback, iterative training, and curriculum learning. Our primary experiments use Knights and Knaves (KK) logical reasoning tasks, which provide deterministic solutions, controlled difficulty levels, and a clean testbed for easy-to-hard generalization. We first show that SE consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non trivial gap to oracle supervision. We find that multi-turn critic-revision with large models could reach strong self-evolution performance, where Gemma 12B nearly matches oracle-supervised training. Beyond KK, we also evaluate SE on real-world reasoning benchmarks, where gains are also modest. Overall, our results characterize when closed-loop SE can help, and show how internally generated supervision remains insufficient under this minimal formulation.

   @inproceedings{pmlr-v306-qi26a,
     title = {On the Generalization Gap in Self-Evolving Language Model Reasoning},
     author = {Qi, Zhenting and Baby, Susanna Maria and Baby, Stefanie Anna and Yuan, Kan and Tomkins, Andrew and Vu, Tu and Juan, Da-Cheng and Rashtchian, Cyrus},
     booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
     pages = {100209--100228},
     year = {2026},
     editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto},
     volume = {306},
     series = {Proceedings of Machine Learning Research},
     publisher = {PMLR},
     pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/qi26a/qi26a.pdf},
     url = {https://proceedings.mlr.press/v306/qi26a.html},
   }

6. Preprint

   PRISM: Pushing the Frontier of Deep Think via Process Reward Model-Guided Inference

   Rituraj Sharma*, Weiyuan Chen*, Noah Provenzano, and Tu Vu

   In arXiv preprint arXiv:2603.02479, 2026

   Abs arXiv Bib PDF

   DEEPTHINK methods improve reasoning by generating, refining, and aggregating populations of candidate solutions, which enables strong performance on complex mathematical and scientific tasks. However, existing frameworks often lack reliable correctness signals during inference, which creates a population-enhancement bottleneck where deeper deliberation amplifies errors, suppresses correct minority solutions, and yields weak returns to additional compute. In this paper, we introduce a functional decomposition of DEEPTHINK systems and propose PRISM, a Process Reward Model (PRM)-guided inference algorithm that uses step-level verification to guide both population refinement and solution aggregation. During refinement, PRISM treats candidate solutions as particles in a PRM-defined energy landscape and reshapes the population through score-guided resampling and stochastic refinement, which concentrates probability mass on higher-quality reasoning while preserving diversity. Across mathematics and science benchmarks, PRISM is competitive with or outperforms existing DEEPTHINK methods, reaching 90.0%, 75.4%, and 71.4% with gpt-oss-20b on AIME25, HMMT25, and GPQA Diamond, respectively, while matching or exceeding gpt-oss-120b. Additionally, our analysis shows that PRISM produces consistent net-directional correction during refinement, remains reliable when the initial population contains few correct candidates, and often lies on the compute-accuracy Pareto frontier

   @inproceedings{sharma2026prism,
     title = {PRISM: Pushing the Frontier of Deep Think via Process Reward Model-Guided Inference},
     author = {Sharma*, Rituraj and Chen*, Weiyuan and Provenzano, Noah and Vu, Tu},
     booktitle = {arXiv preprint arXiv:2603.02479},
     year = {2026},
     pdf = {https://arxiv.org/pdf/2603.02479},
   }

7. Preprint

   MERRIN: A Benchmark for Multimodal Evidence Retrieval

   Han Wang, David Wan, Hyunji Lee, Thinh Pham, Mikaela Cankosyan, Weiyuan Chen, Elias Stengel-Eskin, Tu Vu, and Mohit Bansal

   In arXiv preprint arXiv:2604.13418, 2026

   Abs arXiv Bib PDF

   Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments), a human-annotated benchmark for evaluating search-augmented agents. MERRIN measures AI agents’ ability to identify relevant modalities, retrieve multimodal evidence, and perform multi-hop reasoning over noisy web sources. It differs from prior work in three important aspects: (1) using natural language queries without explicit modality cues, (2) incorporating underexplored modalities such as video and audio, and (3) requiring the retrieval of complex, often noisy or conflicting multimodal evidence during web search. We evaluate diverse search agents powered by ten models, including strong closed-source models (e.g., GPT-5.4-mini, Gemini 3/3.1 Flash/Pro) and open-weight models (Qwen3-4B/30B/235B), across three search settings (no search, native search, and agentic search). Our results show that MERRIN is highly challenging: the average accuracy across all agents is 22.3%, with the best-performing agent reaching only 40.1%. We further observe that while stronger agents like Gemini Deep Research achieve higher performance, gains are modest due to over-exploration; they take more steps and use more tools, but are often distracted by conflicting or partially relevant web content, leading to incorrect answers. Compared to humans, these agents consume more resources yet achieve lower accuracy, largely due to inefficient source selection and an overreliance on text modalities. These findings highlight the need for search agents capable of robust search and reasoning across diverse modalities in noisy web environments, making MERRIN a valuable testbed for evaluating such capabilities.Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments), a human-annotated benchmark for evaluating search-augmented agents. MERRIN measures AI agents’ ability to identify relevant modalities, retrieve multimodal evidence, and perform multi-hop reasoning over noisy web sources. It differs from prior work in three important aspects: (1) using natural language queries without explicit modality cues, (2) incorporating underexplored modalities such as video and audio, and (3) requiring the retrieval of complex, often noisy or conflicting multimodal evidence during web search. We evaluate diverse search agents powered by ten models, including strong closed-source models (e.g., GPT-5.4-mini, Gemini 3/3.1 Flash/Pro) and open-weight models (Qwen3-4B/30B/235B), across three search settings (no search, native search, and agentic search). Our results show that MERRIN is highly challenging: the average accuracy across all agents is 22.3%, with the best-performing agent reaching only 40.1%. We further observe that while stronger agents like Gemini Deep Research achieve higher performance, gains are modest due to over-exploration; they take more steps and use more tools, but are often distracted by conflicting or partially relevant web content, leading to incorrect answers. Compared to humans, these agents consume more resources yet achieve lower accuracy, largely due to inefficient source selection and an overreliance on text modalities. These findings highlight the need for search agents capable of robust search and reasoning across diverse modalities in noisy web environments, making MERRIN a valuable testbed for evaluating such capabilities.

   @inproceedings{wang2026merrin,
     title = {MERRIN: A Benchmark for Multimodal Evidence Retrieval},
     author = {Wang, Han and Wan, David and Lee, Hyunji and Pham, Thinh and Cankosyan, Mikaela and Chen, Weiyuan and Stengel-Eskin, Elias and Vu, Tu and Bansal, Mohit},
     booktitle = {arXiv preprint arXiv:2604.13418},
     year = {2026},
     pdf = {https://arxiv.org/pdf/2604.13418},
   }

8. Technical report

   ROMA: Recursive Open Meta-Agent Framework for Long-Horizon Multi-Agent Systems

   Salaheddin Alzubi, Baran Nama, Arda Kaz, Anushri Eswaran, Weiyuan Chen, Sarvesh Khetan, Rishab Bala, Tu Vu*, and Sewoong Oh*

   In arXiv preprint arXiv:2602.01848, 2026

   Abs arXiv Bib PDF

   Current agentic frameworks underperform on long-horizon tasks. As reasoning depth increases, sequential orchestration becomes brittle, context windows impose hard limits that degrade performance, and opaque execution traces make failures difficult to localize or debug. We introduce ROMA (Recursive Open Meta-Agents), a domain-agnostic framework that addresses these limitations through recursive task decomposition and structured aggregation. ROMA decomposes goals into dependency-aware subtask trees that can be executed in parallel, while aggregation compresses and validates intermediate results to control context growth. Our framework standardizes agent construction around four modular roles - Atomizer (which decides whether a task should be decomposed), Planner, Executor, and Aggregator - which cleanly separate orchestration from model selection and enable transparent, hierarchical execution traces. This design supports heterogeneous multi-agent systems that mix models and tools according to cost, latency, and capability. To adapt ROMA to specific tasks without fine-tuning, we further introduce GEPA+, an improved Genetic-Pareto prompt proposer that searches over prompts within ROMA’s component hierarchy while preserving interface contracts. We show that ROMA, combined with GEPA+, delivers leading system-level performance on reasoning and long-form generation benchmarks. On SEAL-0, which evaluates reasoning over conflicting web evidence, ROMA instantiated with GLM-4.6 improves accuracy by 9.9% over Kimi-Researcher. On EQ-Bench, a long-form writing benchmark, ROMA enables DeepSeek-V3 to match the performance of leading closed-source models such as Claude Sonnet 4.5. Our results demonstrate that recursive, modular agent architectures can scale reasoning depth while remaining interpretable, flexible, and model-agnostic.

   @inproceedings{alzubi2026roma,
     title = {ROMA: Recursive Open Meta-Agent Framework for Long-Horizon Multi-Agent Systems},
     author = {Alzubi, Salaheddin and Nama, Baran and Kaz, Arda and Eswaran, Anushri and Chen, Weiyuan and Khetan, Sarvesh and Bala, Rishab and Vu*, Tu and Oh*, Sewoong},
     booktitle = {arXiv preprint arXiv:2602.01848},
     year = {2026},
     pdf = {https://arxiv.org/pdf/2602.01848},
   }

2025

1. EMNLP

   Efficient Model Development through Fine-tuning Transfer

   Pin-Jie Lin, Rishab Balasubramanian, Fengyuan Liu, Nikhil Kandpal, and Tu Vu

   In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025

   Abs arXiv Bib PDF

   Modern LLMs face a major obstacle: each new pre-trained model version requires expensive and repetitive alignment. We propose a method that transfers fine-tuning updates across model versions. The key idea is to extract the \textitdiff vector, which is the difference in parameters induced by fine-tuning, from a \textitsource model version and apply it to the base of a different \textittarget version. We show that transferring diff vectors significantly improves the target base model, often achieving performance comparable to its fine-tuned counterpart. For example, applying the fine-tuning updates from Llama 3.0 8B to Llama 3.1 8B increases accuracy by 46.9% on IFEval and 15.7% on LiveCodeBench without further training, surpassing Llama 3.1 8B Instruct. In multilingual settings, we also observe accuracy gains relative to Llama 3.1 8B Instruct, including 4.7% for Malagasy and 15.5% for Turkish on Global MMLU. Our controlled experiments reveal that fine-tuning transfer works best when source and target models are linearly connected in parameter space. We also show that this transfer provides a stronger and more efficient starting point for subsequent fine-tuning. Finally, we propose an iterative \textitrecycling-then-finetuning approach for continuous model development, which improves both efficiency and effectiveness. Our findings suggest that fine-tuning transfer is a viable strategy to reduce training costs while maintaining model performance.

   @inproceedings{lin-etal-2025-efficient,
     title = {Efficient Model Development through Fine-tuning Transfer},
     author = {Lin, Pin-Jie and Balasubramanian, Rishab and Liu, Fengyuan and Kandpal, Nikhil and Vu, Tu},
     editor = {Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet},
     booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
     year = {2025},
     address = {Suzhou, China},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2025.emnlp-main.131/},
     doi = {10.18653/v1/2025.emnlp-main.131},
     pages = {2617--2636},
     isbn = {979-8-89176-332-6},
     pdf = {https://aclanthology.org/2025.emnlp-main.131v2.pdf},
   }

   // EMNLP Oral (top 4.0% out of 8K+ submissions)
2. Preprint

   ATEB: Evaluating and improving advanced nlp tasks for text embedding models

   Simeng Han, Frank Palma Gomez, Tu Vu, Zefei Li, Daniel Cer, Hansi Zeng, Chris Tar, Arman Cohan, and Gustavo Hernandez Abrego

   In arXiv preprint arXiv:2502.16766, 2025

   Abs arXiv Bib PDF

   Traditional text embedding benchmarks primarily evaluate embedding models’ capabilities to capture semantic similarity. However, more advanced NLP tasks require a deeper understanding of text, such as safety and factuality. These tasks demand an ability to comprehend and process complex information, often involving the handling of sensitive content, or the verification of factual statements against reliable sources. We introduce a new benchmark designed to assess and highlight the limitations of embedding models trained on existing information retrieval data mixtures on advanced capabilities, which include factuality, safety, instruction following, reasoning and document-level understanding. This benchmark includes a diverse set of tasks that simulate real-world scenarios where these capabilities are critical and leads to identification of the gaps of the currently advanced embedding models. Furthermore, we propose a novel method that reformulates these various tasks as retrieval tasks. By framing tasks like safety or factuality classification as retrieval problems, we leverage the strengths of retrieval models in capturing semantic relationships while also pushing them to develop a deeper understanding of context and content. Using this approach with single-task fine-tuning, we achieved performance gains of 8% on factuality classification and 13% on safety classification. Our code and data will be publicly available.

   @inproceedings{han2025ateb,
     title = {{ATEB}: Evaluating and improving advanced nlp tasks for text embedding models},
     author = {Han, Simeng and Gomez, Frank Palma and Vu, Tu and Li, Zefei and Cer, Daniel and Zeng, Hansi and Tar, Chris and Cohan, Arman and Abrego, Gustavo Hernandez},
     booktitle = {arXiv preprint arXiv:2502.16766},
     year = {2025},
     pdf = {https://arxiv.org/pdf/2502.16766},
   }

3. TMLR

   What Matters for Model Merging at Scale?

   Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai

   In Journal of Transactions on Machine Learning Research, 2025

   Abs arXiv Bib PDF

   Model merging aims to combine multiple expert models into a more capable single model, offering benefits such as reduced storage and serving costs, improved generalization, and support for decentralized model development. Despite its promise, previous studies have primarily focused on merging a few small models. This leaves many unanswered questions about the effect of scaling model size and how it interplays with other key factors—like the base model quality and number of expert models— to affect the merged model’s performance. This work systematically evaluates the utility of model merging at scale for transformer based models to examine the impact of these different factors. We experiment with merging fully fine-tuned models using four popular merging methods—Averaging, Task Arithmetic, Dare-TIES, and TIES-Merging—across model sizes ranging from 1B to 64B parameters and merging up to 8 different expert models. We evaluate the merged models on both held-in tasks, i.e., the expert’s training tasks, and zero-shot generalization to unseen held-out tasks. Our wide range of experiments provide several new insights about merging transformer based models at scale and the interplay between different factors. First, we find that merging is more effective when experts are created from strong base models, i.e., models with good zero-shot performance, compared to pre-trained ones. Second, larger models perform better when merged. Third merging consistently improves generalization capabilities. Notably, when merging eight large expert models, the merged models often generalize better compared to the multitask trained models. Fourth, we can better merge more expert models when working with larger models. Fifth, different merging methods behave very similarly at larger scales. Overall, our findings shed light on some interesting properties of model merging while also highlighting some limitations.

   @inproceedings{yadav2025what,
     title = {What Matters for Model Merging at Scale?},
     author = {Yadav, Prateek and Vu, Tu and Lai, Jonathan and Chronopoulou, Alexandra and Faruqui, Manaal and Bansal, Mohit and Munkhdalai, Tsendsuren},
     booktitle = {Journal of Transactions on Machine Learning Research},
     issn = {2835-8856},
     year = {2025},
     url = {https://openreview.net/forum?id=9sbetmvNpW},
     pdf = {https://openreview.net/pdf?id=9sbetmvNpW},
   }

2024

1. EMNLP

   Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation

   Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

   Abs arXiv Bib PDF

   As large language models (LLMs) evolve, evaluating their output reliably becomes increasingly difficult due to the high cost of human evaluation. To address this, we introduce FLAMe, a family of Foundational Large Autorater Models. FLAMe is trained on a diverse set of over 100 quality assessment tasks, incorporating 5M+ human judgments curated from publicly released human evaluations. FLAMe outperforms models like GPT-4 and Claude-3 on various held-out tasks, and serves as a powerful starting point for fine-tuning, as shown in our reward model evaluation case study (FLAMe-RM). On Reward-Bench, FLAMe-RM-24B achieves 87.8% accuracy, surpassing GPT-4-0125 (85.9%) and GPT-4o (84.7%). Additionally, we introduce FLAMe-Opt-RM, an efficient tail-patch fine-tuning approach that offers competitive RewardBench performance using 25\texttimesfewer training datapoints. Our FLAMe variants outperform popular proprietary LLM-as-a-Judge models on 8 of 12 autorater benchmarks, covering 53 quality assessment tasks, including RewardBench and LLM-AggreFact. Finally, our analysis shows that FLAMe is significantly less biased than other LLM-as-a-Judge models on the CoBBLEr autorater bias benchmark.

   @inproceedings{vu-etal-2024-foundational,
     title = {Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation},
     author = {Vu, Tu and Krishna, Kalpesh and Alzubi, Salaheddin and Tar, Chris and Faruqui, Manaal and Sung, Yun-Hsuan},
     editor = {Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung},
     booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
     year = {2024},
     address = {Miami, Florida, USA},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.emnlp-main.949/},
     doi = {10.18653/v1/2024.emnlp-main.949},
     pages = {17086--17105},
     pdf = {https://arxiv.org/pdf/2407.10817},
   }

   // The top-performing generative model on RewardBench as of July 15, 2024, trained only on publicly available data
2. ACL

   FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

   Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong

   In Findings of the Association for Computational Linguistics: ACL, 2024

   Abs arXiv Bib PDF

   Since most large language models (LLMs) are trained once and never updated, they struggle to dynamically adapt to our ever-changing world. In this work, we present FreshQA, a dynamic QA benchmark that tests a model’s ability to answer questions that may require reasoning over up-to-date world knowledge. We develop a two-mode human evaluation procedure to measure both correctness and hallucination, which we use to benchmark both closed and open-source LLMs by collecting \ensuremath>50K human judgments. We observe that all LLMs struggle to answer questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked. In response, we develop FreshPrompt, a few-shot prompting method that curates and organizes relevant information from a search engine into an LLM’s prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. To facilitate future work, we additionally develop FreshEval, a reliable autorater for quick evaluation and comparison on FreshQA. Our latest results with FreshEval suggest that open-source LLMs such as Mixtral (Jiang et al., 2024), when combined with FreshPrompt, are competitive with closed-source and commercial systems on search-augmented QA.

   @inproceedings{vu-etal-2024-freshllms,
     title = {{F}resh{LLM}s: Refreshing Large Language Models with Search Engine Augmentation},
     author = {Vu, Tu and Iyyer, Mohit and Wang, Xuezhi and Constant, Noah and Wei, Jerry and Wei, Jason and Tar, Chris and Sung, Yun-Hsuan and Zhou, Denny and Le, Quoc and Luong, Thang},
     editor = {Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek},
     booktitle = {Findings of the Association for Computational Linguistics: ACL},
     year = {2024},
     address = {Bangkok, Thailand},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.findings-acl.813/},
     doi = {10.18653/v1/2024.findings-acl.813},
     pages = {13697--13720},
     pdf = {https://aclanthology.org/2024.findings-acl.813.pdf},
   }

   // ZDNET feature // DAIR.AI’s Top ML Papers of the Week // Our dataset and method have inspired or been used for the development of Google’s Gemini, Perplexity.AI’s Online LLMs, You.com, and Contextual AI’s RAG 2.0
3. ICLR

   Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models

   Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Y Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou

   In The Twelfth International Conference on Learning Representations, 2024

   Abs arXiv Bib PDF

   Sparse Mixture-of-Experts (MoE) is a neural architecture design that adds learnable parameters to Large Language Models (LLMs) without increasing computational complexity (FLOPs). Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instruction tuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (in the second and third scenarios), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MoE-32B, surpasses the performance of Flan-PaLM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied by FLAN-MoE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.

   @inproceedings{shen2024mixtureofexperts,
     title = {Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models},
     author = {Shen, Sheng and Hou, Le and Zhou, Yanqi and Du, Nan and Longpre, Shayne and Wei, Jason and Chung, Hyung Won and Zoph, Barret and Fedus, William and Chen, Xinyun and Vu, Tu and Wu, Yuexin and Chen, Wuyang and Webson, Albert and Li, Yunxuan and Zhao, Vincent Y and Yu, Hongkun and Keutzer, Kurt and Darrell, Trevor and Zhou, Denny},
     booktitle = {The Twelfth International Conference on Learning Representations},
     year = {2024},
     url = {https://openreview.net/forum?id=6mLjDwYte5},
     pdf = {https://openreview.net/pdf?id=6mLjDwYte5},
   }

   // Pioneering work on combining mixture-of-experts and instruction tuning

2023

1. Technical report

   Gemini: A Family of Highly Capable Multimodal Models

   Google Gemini Team: Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, and  others including Tu Vu

   In arXiv preprint arXiv:2312.11805, 2023

   Abs arXiv Bib PDF

   This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.

   @inproceedings{team2023gemini,
     title = {Gemini: A Family of Highly Capable Multimodal Models},
     author = {Anil, Rohan and Borgeaud, Sebastian and Wu, Yonghui and Alayrac, Jean-Baptiste and Yu, Jiahui and Soricut, Radu and Schalkwyk, Johan and Dai, Andrew and Hauth, Anja and others},
     booktitle = {arXiv preprint arXiv:2312.11805},
     year = {2023},
     pdf = {https://arxiv.org/pdf/2312.11805},
   }

   // Google AI Blog
2. ICML

   The Flan Collection: Designing Data and Methods for Effective Instruction Tuning

   Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, and Adam Roberts

   In Proceedings of the 40th International Conference on Machine Learning, 2023

   Abs arXiv Bib PDF

   We study the design decision of publicly available instruction tuning methods, by reproducing and breaking down the development of Flan 2022 (Chung et al., 2022). Through careful ablation studies on the Flan Collection of tasks and methods, we tease apart the effect of design decisions which enable Flan-T5 to outperform prior work by 3-17% across evaluation settings. We find task balancing and enrichment techniques are overlooked but critical to effective instruction tuning, and in particular, training with mixed prompt settings (zero-shot, few-shot, chain-of-thought) actually yields equivalent or stronger (2%) performance in all settings. In further experiments we show Flan-T5 requires less finetuning to converge higher and faster than T5 on single downstream tasks – motivating instruction-tuned models as more computationally-efficient starting checkpoints for new tasks. Finally, to accelerate research on instruction tuning, we make the Flan 2022 collection of datasets, templates, and methods publicly available.

   @inproceedings{pmlr-v202-longpre23a,
     title = {The Flan Collection: Designing Data and Methods for Effective Instruction Tuning},
     author = {Longpre, Shayne and Hou, Le and Vu, Tu and Webson, Albert and Chung, Hyung Won and Tay, Yi and Zhou, Denny and Le, Quoc V and Zoph, Barret and Wei, Jason and Roberts, Adam},
     booktitle = {Proceedings of the 40th International Conference on Machine Learning},
     pages = {22631--22648},
     year = {2023},
     editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
     volume = {202},
     series = {Proceedings of Machine Learning Research},
     publisher = {PMLR},
     pdf = {https://proceedings.mlr.press/v202/longpre23a/longpre23a.pdf},
     url = {https://proceedings.mlr.press/v202/longpre23a.html},
   }

   // Pioneering work on large-scale instruction tuning // Google Research Blog
3. ICBINB@NeurIPS

   Self-Evaluation Improves Selective Generation in Large Language Models

   Jie Ren, Yao Zhao, Tu Vu, Peter J. Liu, and Balaji Lakshminarayanan

   In Proceedings on "I Can’t Believe It’s Not Better: Failure Modes in the Age of Foundation Models" at NeurIPS 2023 Workshops, 2023

   Abs arXiv Bib PDF

   Safe deployment of large language models (LLMs) may benefit from a reliable method for assessing their generated content to determine when to abstain or to selectively generate. While likelihood-based metrics such as perplexity are widely employed, recent research has demonstrated the limitations of using sequence-level probability estimates given by LLMs as reliable indicators of generation quality. Conversely, LLMs have demonstrated strong calibration at the token level, particularly when it comes to choosing correct answers in multiple-choice questions or evaluating true/false statements. In this work, we reformulate open-ended generation tasks into token-level prediction tasks, and leverage LLMs’ superior calibration at the token level. We instruct an LLM to self-evaluate its answers, employing either a multi-way comparison or a point-wise evaluation approach, with the option to include an “None of the above” option to express the model’s uncertainty explicitly. We benchmark a range of scoring methods based on self-evaluation and evaluate their performance in selective generation using TruthfulQA and TL;DR. Through extensive experiments with PaLM-2 and GPT-3, we demonstrate that self-evaluation based scores not only improve accuracy, but also correlate better with the overall quality of generated content.

   @inproceedings{pmlr-v239-ren23a,
     title = {Self-Evaluation Improves Selective Generation in Large Language Models},
     author = {Ren, Jie and Zhao, Yao and Vu, Tu and Liu, Peter J. and Lakshminarayanan, Balaji},
     booktitle = {Proceedings on "I Can't Believe It's Not Better: Failure  Modes in the Age of Foundation Models" at NeurIPS 2023 Workshops},
     pages = {49--64},
     year = {2023},
     editor = {Antorán, Javier and Blaas, Arno and Buchanan, Kelly and Feng, Fan and Fortuin, Vincent and Ghalebikesabi, Sahra and Kriegler, Andreas and Mason, Ian and Rohde, David and Ruiz, Francisco J. R. and Uelwer, Tobias and Xie, Yubin and Yang, Rui},
     volume = {239},
     series = {Proceedings of Machine Learning Research},
     publisher = {PMLR},
     pdf = {https://proceedings.mlr.press/v239/ren23a/ren23a.pdf},
     url = {https://proceedings.mlr.press/v239/ren23a.html},
   }

4. ACL

   Dialect-robust Evaluation of Generated Text

   Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Timothy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, and Sebastian Gehrmann

   In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023

   Abs arXiv Bib PDF

   Text generation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects. In this paper, we introduce a suite of methods to assess whether metrics are dialect robust. These methods show that state-of-the-art metrics are not dialect robust: they often prioritize dialect similarity over semantics, preferring outputs that are semantically incorrect over outputs that match the semantics of the reference but contain dialect differences. As a step towards dialect-robust metrics for text generation, we propose NANO, which introduces regional and language information to the metric’s pretraining. NANO significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. It also enables a more ambitious approach to evaluation, dialect awareness, in which system outputs are scored by both semantic match to the reference and appropriateness in any specified dialect.

   @inproceedings{sun-etal-2023-dialect,
     title = {Dialect-robust Evaluation of Generated Text},
     author = {Sun, Jiao and Sellam, Thibault and Clark, Elizabeth and Vu, Tu and Dozat, Timothy and Garrette, Dan and Siddhant, Aditya and Eisenstein, Jacob and Gehrmann, Sebastian},
     editor = {Rogers, Anna and Boyd-Graber, Jordan and Okazaki, Naoaki},
     booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics},
     year = {2023},
     address = {Toronto, Canada},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.acl-long.331/},
     doi = {10.18653/v1/2023.acl-long.331},
     pages = {6010--6028},
     pdf = {https://aclanthology.org/2023.acl-long.331.pdf},
   }

2022

1. EMNLP

   Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation

   Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant

   In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022

   Abs arXiv Bib PDF

   In this paper, we explore the challenging problem of performing a generative task in a target language when labeled data is only available in English, using summarization as a case study. We assume a strict setting with no access to parallel data or machine translation and find that common transfer learning approaches struggle in this setting, as a generative multilingual model fine-tuned purely on English catastrophically forgets how to generate non-English. Given the recent rise of parameter-efficient adaptation techniques, we conduct the first investigation into how one such method, prompt tuning (Lester et al., 2021), can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. Our experiments show that parameter-efficient prompt tuning provides gains over standard fine-tuning when transferring between less-related languages, e.g., from English to Thai. However, a significant gap still remains between these methods and fully-supervised baselines. To improve cross-lingual transfer further, we explore several approaches, including: (1) mixing in unlabeled multilingual data, and (2) explicitly factoring prompts into recombinable language and task components. Our approaches can provide further quality gains, suggesting that robust zero-shot cross-lingual generation is within reach.

   @inproceedings{vu-etal-2022-overcoming,
     title = {Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation},
     author = {Vu, Tu and Barua, Aditya and Lester, Brian and Cer, Daniel and Iyyer, Mohit and Constant, Noah},
     editor = {Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue},
     booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
     year = {2022},
     address = {Abu Dhabi, United Arab Emirates},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.emnlp-main.630/},
     doi = {10.18653/v1/2022.emnlp-main.630},
     pages = {9279--9300},
     pdf = {https://aclanthology.org/2022.emnlp-main.630.pdf},
   }

2. ACL

   SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

   Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou’, and Daniel Cer

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022

   Abs arXiv Bib PDF

   There has been growing interest in parameter-efficient methods to apply pre-trained language models to downstream tasks. Building on the Prompt Tuning approach of Lester et al. (2021), which learns task-specific soft prompts to condition a frozen pre-trained model to perform different tasks, we propose a novel prompt-based transfer learning approach called SPoT: Soft Prompt Transfer. SPoT first learns a prompt on one or more source tasks and then uses it to initialize the prompt for a target task. We show that SPoT significantly boosts the performance of Prompt Tuning across many tasks. More remarkably, across all model sizes, SPoT matches or outperforms standard Model Tuning (which fine-tunes all model parameters) on the SuperGLUE benchmark, while using up to 27,000\texttimes fewer task-specific parameters. To understand where SPoT is most effective, we conduct a large-scale study on task transferability with 26 NLP tasks in 160 combinations, and demonstrate that many tasks can benefit each other via prompt transfer. Finally, we propose an efficient retrieval approach that interprets task prompts as task embeddings to identify similar tasks and predict the most transferable source tasks for a novel target task.

   @inproceedings{vu-etal-2022-spot,
     title = {{SP}o{T}: Better Frozen Model Adaptation through Soft Prompt Transfer},
     author = {Vu, Tu and Lester, Brian and Constant, Noah and Al-Rfou{'}, Rami and Cer, Daniel},
     editor = {Muresan, Smaranda and Nakov, Preslav and Villavicencio, Aline},
     booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics},
     year = {2022},
     address = {Dublin, Ireland},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.acl-long.346/},
     doi = {10.18653/v1/2022.acl-long.346},
     pages = {5039--5059},
     pdf = {https://aclanthology.org/2022.acl-long.346.pdf},
   }

   // Pioneering work on soft prompt tuning & transfer // Headlines of Google AI’s Natural Language Accelerated Newsletter Q1, 2022
3. EMNLP

   Leveraging QA Datasets to Improve Generative Data Augmentation

   Dheeraj Mekala, Tu Vu, Timo Schick, and Jingbo Shang

   In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022

   Abs arXiv Bib PDF

   The ability of generative language models (GLMs) to generate text has improved considerably in the last few years, enabling their use for generative data augmentation. In this work, we propose CONDA, an approach to further improve GLM’s ability to generate synthetic data by reformulating data generation as context generation for a given question-answer (QA) pair and leveraging QA datasets for training context generators. Then, we cast downstream tasks into the same question answering format and adapt the fine-tuned context generators to the target task domain. Finally, we use the fine-tuned GLM to generate relevant contexts, which are in turn used as synthetic training data for their corresponding tasks. We perform extensive experiments on multiple classification datasets and demonstrate substantial improvements in performance for both few- and zero-shot settings. Our analysis reveals that QA datasets that require high-level reasoning abilities (e.g., abstractive and common-sense QA datasets) tend to give the best boost in performance in both few-shot and zero-shot settings.

   @inproceedings{mekala-etal-2022-leveraging,
     title = {Leveraging {QA} Datasets to Improve Generative Data Augmentation},
     author = {Mekala, Dheeraj and Vu, Tu and Schick, Timo and Shang, Jingbo},
     editor = {Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue},
     booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
     year = {2022},
     address = {Abu Dhabi, United Arab Emirates},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.emnlp-main.660/},
     doi = {10.18653/v1/2022.emnlp-main.660},
     pages = {9737--9750},
     pdf = {https://aclanthology.org/2022.emnlp-main.660.pdf},
   }

2021

1. EMNLP

   STraTA: Self-Training with Task Augmentation for Better Few-shot Learning

   Tu Vu, Minh-Thang Luong, Quoc Le, Grady Simon, and Mohit Iyyer

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021

   Abs arXiv Bib PDF

   Despite their recent successes in tackling many NLP tasks, large-scale pre-trained language models do not perform as well in few-shot settings where only a handful of training examples are available. To address this shortcoming, we propose STraTA, which stands for Self-Training with Task Augmentation, an approach that builds on two key ideas for effective leverage of unlabeled data. First, STraTA uses task augmentation, a novel technique that synthesizes a large amount of data for auxiliary-task fine-tuning from target-task unlabeled texts. Second, STraTA performs self-training by further fine-tuning the strong base model created by task augmentation on a broad distribution of pseudo-labeled data. Our experiments demonstrate that STraTA can substantially improve sample efficiency across 12 few-shot benchmarks. Remarkably, on the SST-2 sentiment dataset, STraTA, with only 8 training examples per class, achieves comparable results to standard fine-tuning with 67K training examples. Our analyses reveal that task augmentation and self-training are both complementary and independently effective.

   @inproceedings{vu-etal-2021-strata,
     title = {{ST}ra{TA}: Self-Training with Task Augmentation for Better Few-shot Learning},
     author = {Vu, Tu and Luong, Minh-Thang and Le, Quoc and Simon, Grady and Iyyer, Mohit},
     editor = {Moens, Marie-Francine and Huang, Xuanjing and Specia, Lucia and Yih, Scott Wen-tau},
     booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
     year = {2021},
     address = {Online and Punta Cana, Dominican Republic},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2021.emnlp-main.462/},
     doi = {10.18653/v1/2021.emnlp-main.462},
     pages = {5715--5731},
     pdf = {https://aclanthology.org/2021.emnlp-main.462.pdf},
   }

2020

1. EMNLP

   Exploring and Predicting Transferability across NLP Tasks

   Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer

   In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020

   Abs arXiv Bib PDF

   Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. Can fine-tuning these models on tasks other than language modeling further improve performance? In this paper, we conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems (text classification, question answering, and sequence labeling). Our results show that transfer learning is more beneficial than previously thought, especially when target task data is scarce, and can improve performance even with low-data source tasks that differ substantially from the target task (e.g., part-of-speech tagging transfers well to the DROP QA dataset). We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task, and we validate their effectiveness in experiments controlled for source and target data size. Overall, our experiments reveal that factors such as data size, task and domain similarity, and task complexity all play a role in determining transferability.

   @inproceedings{vu-etal-2020-exploring,
     title = {Exploring and Predicting Transferability across {NLP} Tasks},
     author = {Vu, Tu and Wang, Tong and Munkhdalai, Tsendsuren and Sordoni, Alessandro and Trischler, Adam and Mattarella-Micke, Andrew and Maji, Subhransu and Iyyer, Mohit},
     editor = {Webber, Bonnie and Cohn, Trevor and He, Yulan and Liu, Yang},
     booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing},
     year = {2020},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2020.emnlp-main.635/},
     doi = {10.18653/v1/2020.emnlp-main.635},
     pages = {7882--7926},
     pdf = {https://aclanthology.org/2020.emnlp-main.635.pdf},
   }

   // Pioneering work on exploring task transferability in the LLM era

2019

1. ACL

   Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification

   Tu Vu, and Mohit Iyyer

   In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

   Abs arXiv Bib PDF

   While paragraph embedding models are remarkably effective for downstream classification tasks, what they learn and encode into a single vector remains opaque. In this paper, we investigate a state-of-the-art paragraph embedding method proposed by Zhang et al. (2017) and discover that it cannot reliably tell whether a given sentence occurs in the input paragraph or not. We formulate a sentence content task to probe for this basic linguistic property and find that even a much simpler bag-of-words method has no trouble solving it. This result motivates us to replace the reconstruction-based objective of Zhang et al. (2017) with our sentence content probe objective in a semi-supervised setting. Despite its simplicity, our objective improves over paragraph reconstruction in terms of (1) downstream classification accuracies on benchmark datasets, (2) faster training, and (3) better generalization ability.

   @inproceedings{vu-iyyer-2019-encouraging,
     title = {Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification},
     author = {Vu, Tu and Iyyer, Mohit},
     editor = {Korhonen, Anna and Traum, David and M{\`a}rquez, Llu{\'i}s},
     booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
     year = {2019},
     address = {Florence, Italy},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/P19-1638/},
     doi = {10.18653/v1/P19-1638},
     pages = {6331--6338},
     pdf = {https://aclanthology.org/P19-1638.pdf},
   }

2018

1. NAACL

   Sentence Simplification with Memory-Augmented Neural Networks

   Tu Vu, Baotian Hu, Tsendsuren Munkhdalai, and Hong Yu

   In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018

   Abs arXiv Bib PDF

   Sentence simplification aims to simplify the content and structure of complex sentences, and thus make them easier to interpret for human readers, and easier to process for downstream NLP applications. Recent advances in neural machine translation have paved the way for novel approaches to the task. In this paper, we adapt an architecture with augmented memory capacities called Neural Semantic Encoders (Munkhdalai and Yu, 2017) for sentence simplification. Our experiments demonstrate the effectiveness of our approach on different simplification datasets, both in terms of automatic evaluation measures and human judgments.

   @inproceedings{vu-etal-2018-sentence,
     title = {Sentence Simplification with Memory-Augmented Neural Networks},
     author = {Vu, Tu and Hu, Baotian and Munkhdalai, Tsendsuren and Yu, Hong},
     editor = {Walker, Marilyn and Ji, Heng and Stent, Amanda},
     booktitle = {Proceedings of the 2018 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies},
     year = {2018},
     address = {New Orleans, Louisiana},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/N18-2013/},
     doi = {10.18653/v1/N18-2013},
     pages = {79--85},
     pdf = {https://aclanthology.org/N18-2013.pdf},
   }

2. *SEM@NAACL

   Integrating Multiplicative Features into Supervised Distributional Methods for Lexical Entailment

   Tu Vu, and Vered Shwartz

   In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, 2018

   Abs arXiv Bib PDF

   Supervised distributional methods are applied successfully in lexical entailment, but recent work questioned whether these methods actually learn a relation between two words. Specifically, Levy et al. (2015) claimed that linear classifiers learn only separate properties of each word. We suggest a cheap and easy way to boost the performance of these methods by integrating multiplicative features into commonly used representations. We provide an extensive evaluation with different classifiers and evaluation setups, and suggest a suitable evaluation setup for the task, eliminating biases existing in previous ones.

   @inproceedings{vu-shwartz-2018-integrating,
     title = {Integrating Multiplicative Features into Supervised Distributional Methods for Lexical Entailment},
     author = {Vu, Tu and Shwartz, Vered},
     editor = {Nissim, Malvina and Berant, Jonathan and Lenci, Alessandro},
     booktitle = {Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics},
     year = {2018},
     address = {New Orleans, Louisiana},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/S18-2020/},
     doi = {10.18653/v1/S18-2020},
     pages = {160--166},
     pdf = {https://aclanthology.org/S18-2020.pdf},
   }