Publications

Recent model merging methods demonstrate that the parameters of fully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce a new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting.

Synthetic data (SIM) drawn from simulators have emerged as a popular alternative for training models where acquiring annotated real-world images is difficult. However, transferring models trained on synthetic images to real-world applications can be challenging due to appearance disparities. A commonly employed solution to counter this SIM2REAL gap is unsupervised domain adaptation, where models are trained using labeled SIM data and unlabeled REAL data. Mispredictions made by such SIM2REAL adapted models are often associated with miscalibration - stemming from overconfident predictions on real data. In this paper, we introduce AUGCAL, a simple training-time patch for unsupervised adaptation that improves SIM2REAL adapted models by - (1) reducing overall miscalibration, (2) reducing overconfidence in incorrect predictions and (3) improving confidence score reliability by better guiding misclassification detection - all while retaining or improving SIM2REAL performance. Given a base SIM2REAL adaptation algorithm, at training time, AUGCAL involves replacing vanilla SIM images with strongly augmented views (AUG intervention) and additionally optimizing for a training time calibration loss on augmented SIM predictions (CAL intervention). We motivate AUGCAL using a brief analytical justification of how to reduce miscalibration on unlabeled REAL data. Through our experiments, we empirically show the efficacy of AUGCAL across multiple adaptation methods, backbones, tasks and shifts.

As foundation models grow in size and complexity, parameter-efficient fine-tuning approaches like LoRA have emerged as effective solutions for adapting these models to specific tasks. Model merging offers a resource-efficient paradigm for combining knowledge from multiple expert models without requiring access to original training data, but faces challenges with catastrophic interference when scaling beyond a few models. This study systematically evaluates how different merging techniques (Task Arithmetic, TIES, DARE, KnOTS, and combinations) perform when scaling the merge from 2 to 8 tasks, exploring optimal hyperparameter configurations for controlling interference. Using two different ViT architectures (B/32 and L/14) across 8 diverse vision datasets, I comprehensively evaluate all 247 possible combinations of expert models. The results demonstrate that KnOTS-enhanced methods exhibit significantly better multi-task performance as the number of merged models increases, consistently outperforming baselines across all model cardinalities which becomes even more pronounced with scale, KnOTS-TIES achieveing a 4.07% improvement compared to TIES with the ViT-B/32 architecture. Analysis of hyperparameter trends reveals that as merge size increases, optimal scaling coefficients for Task Arithmetic and TIES decrease, while KnOTS-TIES maintains higher coefficients and becomes more selective with its top-K intervention. These findings suggest that orthogonal decomposition methods like KnOTS can significantly mitigate performance decay when merging multiple LoRA models by better preserving task-specific knowledge.

Modern artificial intelligence (AI) approaches mainly rely on neural network (NN) or deep NN methodologies. However, these approaches require large amounts of data to train, given, that the number of their trainable parameters has a polynomial relationship to their neuron counts. This property renders deep NNs challenging to apply in fields operating with small, albeit representative datasets such as healthcare. In this paper, we propose a novel neural network architecture which trains spatial positions of neural soma and axon pairs, where weights are calculated by axon-soma distances of connected neurons. We refer to this method as distance-encoding biomorphic-informational (DEBI) neural network. This concept significantly minimizes the number of trainable parameters compared to conventional neural networks. We demonstrate that DEBI models can yield comparable predictive performance in tabular and imaging datasets, where they require a fraction of trainable parameters compared to conventional NNs, resulting in a highly scalable solution.