tomaarsen/ColQwen3-VL-Embedding-2B-vdr

Multi-Vector Encoder

This is a Multi-Vector Encoder model finetuned from Qwen/Qwen3-VL-Embedding-2B on the llamaindex-vdr-en-train-preprocessed dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

• Model Type: Multi-Vector Encoder
• Base model: Qwen/Qwen3-VL-Embedding-2B
• Maximum Sequence Length: 262144 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: maxsim
• Supported Modalities: Text, Image, Video, Message
• Training Dataset:
  • llamaindex-vdr-en-train-preprocessed

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Multi-Vector Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Multi-Vector Encoders on Hugging Face

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}, 'image': {'method': 'forward', 'method_output_name': 'last_hidden_state'}, 'video': {'method': 'forward', 'method_output_name': 'last_hidden_state'}, 'message': {'method': 'forward', 'method_output_name': 'last_hidden_state', 'format': 'structured'}}, 'module_output_name': 'token_embeddings', 'processing_kwargs': {'chat_template': {'add_generation_prompt': True}}, 'unpad_inputs': False, 'architecture': 'Qwen3VLModel'})
  (1): Dense({'in_features': 2048, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/ColQwen3-VL-Embedding-2B-vdr")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'What are the new anthropological perspectives on development as discussed by Quarles Van Ufford and Giri in 2003?',
]
documents = [
    'https://huggingface.co/tomaarsen/ColQwen3-VL-Embedding-2B-vdr/resolve/main/assets/image_0.jpg',
    'https://huggingface.co/tomaarsen/ColQwen3-VL-Embedding-2B-vdr/resolve/main/assets/image_1.jpg',
    'https://huggingface.co/tomaarsen/ColQwen3-VL-Embedding-2B-vdr/resolve/main/assets/image_2.jpg',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (48, 128) (352, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[37.0712, 33.5348, 23.2705]])

Evaluation

Metrics

Multi Vector Information Retrieval

• Dataset: vdr-eval-hard
• Evaluated with MultiVectorInformationRetrievalEvaluator

Metric	Value
maxsim_accuracy@1	0.8467
maxsim_accuracy@3	0.9567
maxsim_accuracy@5	0.98
maxsim_accuracy@10	0.9967
maxsim_precision@1	0.8467
maxsim_precision@3	0.3189
maxsim_precision@5	0.196
maxsim_precision@10	0.0997
maxsim_recall@1	0.8467
maxsim_recall@3	0.9567
maxsim_recall@5	0.98
maxsim_recall@10	0.9967
maxsim_ndcg@10	0.9278
maxsim_mrr@10	0.905
maxsim_map@100	0.9053

Training Details

Training Dataset

llamaindex-vdr-en-train-preprocessed

• Dataset: llamaindex-vdr-en-train-preprocessed
• Size: 10,000 training samples
• Columns: query, image, and negative_0
• Approximate statistics based on the first 100 samples:

  	query	image	negative_0
  type	string	image	image
  modality	text	image	image
  details	min: 27 tokens mean: 35.59 tokens max: 50 tokens	min: 1000x756 px mean: 1437x1628 px max: 2044x1869 px	min: 1008x756 px mean: 1450x1637 px max: 2044x1851 px

• Samples:

  query	image	negative_0
  What are the new anthropological perspectives on development as discussed by Quarles Van Ufford and Giri in 2003?
  What are the three main positions anthropologists have taken in relation to development, as discussed by David Lewis?
  Who are the three sisters known as the Fates in Greek mythology?

• Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:

  {
      "score_metric": "colbert_scores",
      "mini_batch_size": 1,
      "mini_batch_num_tokens": null,
      "score_mini_batch_size": 1,
      "scale": 1.0,
      "size_average": true,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.1
• max_grad_norm: 30.0
• bf16: True
• learning_rate_mapping: {'^(model\.)?1\.linear': 0.0002}

All Hyperparameters

Click to expand

• per_device_train_batch_size: 64
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 30.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {'^(model\.)?1\.linear': 0.0002}
• max_length: None

Training Logs

Epoch	Step	Training Loss	vdr-eval-hard_maxsim_ndcg@10
-1	-1	-	0.8244
0.0510	8	2.2737	-
0.1019	16	0.6985	0.9132
0.1529	24	0.3765	-
0.2038	32	0.3179	0.9133
0.2548	40	0.2909	-
0.3057	48	0.2984	0.9184
0.3567	56	0.2649	-
0.4076	64	0.2771	0.9232
0.4586	72	0.2404	-
0.5096	80	0.2456	0.9271
0.5605	88	0.2933	-
0.6115	96	0.2549	0.9224
0.6624	104	0.2636	-
0.7134	112	0.2815	0.9202
0.7643	120	0.2829	-
0.8153	128	0.2592	0.9206
0.8662	136	0.2180	-
0.9172	144	0.2647	0.9206
0.9682	152	0.2411	-
1.0	157	-	0.9278

Training Time

• Training: 2.4 hours
• Evaluation: 26.8 minutes
• Total: 2.8 hours

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.7.0.dev0
• Transformers: 5.14.1
• PyTorch: 2.13.0+cu130
• Accelerate: 1.14.0
• Datasets: 5.0.0
• Tokenizers: 0.22.2

Additional Resources

• Sentence Transformers Documentation: the full documentation site, including training, evaluation, and pre-trained model catalogs.
• PyLate: the upstream library whose features were absorbed into Sentence Transformers for multi-vector / late-interaction models.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiVectorMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}