By harnessing the capabilities of large language models (LLMs), recent large multimodal models (LMMs) have shown remarkable versatility in open-world multimodal understanding. Nevertheless, they are usually parameter-heavy and computation-intensive, thus hindering their applicability in resource-constrained scenarios. To this end, several lightweight LMMs have been proposed successively to maximize the capabilities under constrained scale (e.g., 3B). Despite the encouraging results achieved by these methods, most of them only focus on one or two aspects of the design space, and the key design choices that influence model capability have not yet been thoroughly investigated. In this paper, we conduct a systematic study for lightweight LMMs from the aspects of model architecture, training strategy, and training data. Based on our findings, we obtain Imp—a family of highly capable LMMs at the 2B$\sim$4B scales. Notably, our Imp-3B model steadily outperforms all the existing lightweight LMMs of similar size, and even surpasses the state-of-the-art LMMs at the 13B scale. With low-bit quantization and resolution reduction techniques, our Imp model can be deployed on a Qualcomm Snapdragon 8Gen3 mobile chip with a high inference speed of about 13 tokens/s.

The applications of isometric 3D objects have recently received sufficient attentions and thus it is very attractive to retrieve such isometric 3D objects from large-scale collections. Although some existing approaches have presented some interesting ideas on retrieving 3D objects from large-scale collections, their performance are restrained because of their limited ability on 3D shape representation. To achieve more effective representations of 3D objects (shapes), some recent methods use complex deep neural networks to learn more discriminative deep features for 3D shape representation but they require huge computational resources. In this paper, a more effective solution is developed by seamlessly integrating traditional hand-crafted descriptors with deep point convolutional networks. Our algorithm first takes local point descriptors as the input and produces a global vector for shape retrieval and recognition. To save the efforts of designing complex deep convolutional neural network (CNN), we first use intrinsic point descriptors to describe the shape deformation. Then, a simple but effective point CNN network is developed to extract the local shape information by performing subspace compression and fusion, which depends on an end-to-end learning process to link the local and global information for discriminative shape representation. A knowledge transfer strategy is further used to enrich deep features after a series of downsampling operations. Finally, we carry out experimental evaluations over multiple popular benchmarks and the results have verified that our algorithm can outperform the state-of-the-art methods.

In this paper, a deep mixture of diverse experts algorithm is developed to achieve more efficient learning of a huge (mixture) network for large-scale visual recognition application. First, a two-layer ontology is constructed to assign large numbers of atomic object classes into a set of task groups according to the similarities of their learning complexities, where certain degrees of inter-group task overlapping are allowed to enable sufficient inter-group message passing. Second, one particular base deep CNNs with $M+1$ outputs is learned for each task group to recognize its $M$ atomic object classes and identify one special class of “not-in-group”, where the network structure (numbers of layers and units in each layer) of the well-designed deep CNNs (such as AlexNet, VGG, GoogleNet, ResNet) is directly used to configure such base deep CNNs. For enhancing the separability of the atomic object classes in the same task group, two approaches are developed to learn more discriminative base deep CNNs: (a) our deep multi-task learning algorithm that can effectively exploit the inter-class visual similarities; (b) our two-layer network cascade approach that can improve the accuracy rates for the hard object classes at certain degrees while effectively maintaining the high accuracy rates for the easy ones. Finally, all these complementary base deep CNNs with diverse but overlapped outputs are seamlessly combined to generate a mixture network with larger outputs for recognizing tens of thousands of atomic object classes. Our experimental results have demonstrated that our deep mixture of diverse experts algorithm can achieve very competitive results on large-scale visual recognition.