lightonai_GTE-ModernColBERT-v1

---

tags:

• ColBERT
• multi-vector
• PyLate
• sentence-transformers
• sentence-similarity
• feature-extraction
• generated_from_trainer
• dataset_size:640000
• loss:Distillation base_model: Alibaba-NLP/gte-modernbert-base pipeline_tag: sentence-similarity license: apache-2.0 library_name: PyLate metrics:
• MaxSim_accuracy@1
• MaxSim_accuracy@3
• MaxSim_accuracy@5
• MaxSim_accuracy@10
• MaxSim_precision@1
• MaxSim_precision@3
• MaxSim_precision@5
• MaxSim_precision@10
• MaxSim_recall@1
• MaxSim_recall@3
• MaxSim_recall@5
• MaxSim_recall@10
• MaxSim_ndcg@10
• MaxSim_mrr@10
• MaxSim_map@100 model-index:
• name: PyLate model based on Alibaba-NLP/gte-modernbert-base results:
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoClimateFEVER type: NanoClimateFEVER metrics:
    • type: MaxSim_accuracy@1 value: 0.36 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.62 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.78 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.86 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.36 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.2333333333333333 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.20799999999999996 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.12799999999999997 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.18333333333333332 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.289 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.41566666666666663 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.49566666666666664 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.41477895139843374 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.526579365079365 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.33473812643311207 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoDBPedia type: NanoDBPedia metrics:
    • type: MaxSim_accuracy@1 value: 0.88 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.94 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.96 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.98 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.88 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.7133333333333334 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.6560000000000001 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.572 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.11798996781634019 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.23074158968531658 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.2961618059276896 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.4145532152487909 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.7295518860528665 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.9168571428571428 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.5883869727264871 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoFEVER type: NanoFEVER metrics:
    • type: MaxSim_accuracy@1 value: 0.92 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.98 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.98 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 1.0 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.92 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.35999999999999993 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.21599999999999994 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.10999999999999999 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.8566666666666667 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.96 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.96 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.98 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.9451911044041129 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.9522222222222223 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.9270501207729468 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoFiQA2018 type: NanoFiQA2018 metrics:
    • type: MaxSim_accuracy@1 value: 0.56 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.66 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.74 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.8 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.56 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.32666666666666666 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.25599999999999995 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.15199999999999997 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.30924603174603177 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.47840476190476194 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.5751746031746031 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.6411984126984127 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.5669909336903424 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.6359444444444444 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.5031998196513616 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoHotpotQA type: NanoHotpotQA metrics:
    • type: MaxSim_accuracy@1 value: 0.92 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 1.0 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 1.0 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 1.0 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.92 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.58 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.35999999999999993 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.18599999999999994 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.46 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.87 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.9 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.93 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.9011747095216048 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.96 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.8591508921772081 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoMSMARCO type: NanoMSMARCO metrics:
    • type: MaxSim_accuracy@1 value: 0.54 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.68 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.74 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.92 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.54 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.22666666666666666 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.14800000000000002 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.092 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.54 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.68 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.74 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.92 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.7088869908160952 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.6446507936507936 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.6496349206349206 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoNFCorpus type: NanoNFCorpus metrics:
    • type: MaxSim_accuracy@1 value: 0.56 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.68 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.74 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.76 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.56 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.43333333333333335 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.39199999999999996 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.304 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.06640185752724687 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.10198877096622012 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.12839743828750172 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.15658989769166 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.3957047406068243 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.627 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.1917924344366858 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoNQ type: NanoNQ metrics:
    • type: MaxSim_accuracy@1 value: 0.64 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.82 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.86 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.9 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.64 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.2866666666666666 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.17999999999999997 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.1 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.61 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.78 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.82 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.88 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.7645227466201794 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.7390000000000001 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.7239323294755705 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoQuoraRetrieval type: NanoQuoraRetrieval metrics:
    • type: MaxSim_accuracy@1 value: 0.96 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 1.0 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 1.0 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 1.0 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.96 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.4 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.25599999999999995 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.13399999999999998 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.8473333333333334 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.9453333333333334 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.9693333333333334 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.9893333333333334 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.9691448095973965 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.9766666666666667 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.9551871794871795 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoSCIDOCS type: NanoSCIDOCS metrics:
    • type: MaxSim_accuracy@1 value: 0.48 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.74 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.78 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.84 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.48 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.3999999999999999 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.292 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.19399999999999995 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.10066666666666667 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.24666666666666665 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.29966666666666664 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.39666666666666667 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.3986767701602276 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.6137222222222222 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.3163385555719993 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoArguAna type: NanoArguAna metrics:
    • type: MaxSim_accuracy@1 value: 0.3 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.62 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.7 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.82 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.3 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.20666666666666667 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.14 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.08199999999999999 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.3 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.62 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.7 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.82 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.5609089627577635 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.4774603174603175 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.4824361431413148 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoSciFact type: NanoSciFact metrics:
    • type: MaxSim_accuracy@1 value: 0.74 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.86 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.9 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.94 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.74 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.3 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.19599999999999995 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.10399999999999998 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.715 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.83 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.885 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.93 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.8371556505161787 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.8116666666666668 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.8048798701298702 name: Maxsim Map@100
  • task: type: py-late-information-retrieval name: Py Late Information Retrieval dataset: name: NanoTouche2020 type: NanoTouche2020 metrics:
    • type: MaxSim_accuracy@1 value: 0.7755102040816326 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.9387755102040817 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.9795918367346939 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.9795918367346939 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.7755102040816326 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.6598639455782314 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.6571428571428573 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.5183673469387755 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.05176652252904378 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.13618168510556633 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.2193408037582337 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.33397423594107617 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.5926586898856947 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.8629251700680272 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.42574993112112997 name: Maxsim Map@100
  • task: type: nano-beir name: Nano BEIR dataset: name: NanoBEIR mean type: NanoBEIR_mean metrics:
    • type: MaxSim_accuracy@1 value: 0.6642700156985872 name: Maxsim Accuracy@1
    • type: MaxSim_accuracy@3 value: 0.8106750392464678 name: Maxsim Accuracy@3
    • type: MaxSim_accuracy@5 value: 0.8584301412872841 name: Maxsim Accuracy@5
    • type: MaxSim_accuracy@10 value: 0.9076609105180532 name: Maxsim Accuracy@10
    • type: MaxSim_precision@1 value: 0.6642700156985872 name: Maxsim Precision@1
    • type: MaxSim_precision@3 value: 0.39434850863422294 name: Maxsim Precision@3
    • type: MaxSim_precision@5 value: 0.3043956043956044 name: Maxsim Precision@5
    • type: MaxSim_precision@10 value: 0.20587441130298273 name: Maxsim Precision@10
    • type: MaxSim_recall@1 value: 0.3968003368937432 name: Maxsim Recall@1
    • type: MaxSim_recall@3 value: 0.5514089852047589 name: Maxsim Recall@3
    • type: MaxSim_recall@5 value: 0.6083647167549766 name: Maxsim Recall@5
    • type: MaxSim_recall@10 value: 0.6836909560189698 name: Maxsim Recall@10
    • type: MaxSim_ndcg@10 value: 0.6757959189252093 name: Maxsim Ndcg@10
    • type: MaxSim_mrr@10 value: 0.7495919239490669 name: Maxsim Mrr@10
    • type: MaxSim_map@100 value: 0.5971136381353681 name: Maxsim Map@100

---

GTE-ModernColBERT-v1

Multi-vector embedding model based on Alibaba-NLP/gte-modernbert-base

This is a multi-vector (ColBERT-style late interaction) embedding model trained on the ms-marco-en-bge-gemma dataset. It maps sentences & paragraphs to sequences of 128-dimensional dense vectors and can be used for semantic textual similarity using the MaxSim operator.

Model Details

Model Description

• Model Type: Multi-vector embedding model
• Base model: Alibaba-NLP/gte-modernbert-base
• Document Length: 300 tokens
• Query Length: 32 tokens
• Output Dimensionality: 128 dimensions
• Similarity Function: MaxSim
• Training Dataset:
  • ms-marco-en-bge-gemma
• Language: English
• License: Apache 2.0

Document length

GTE-ModernColBERT has been trained with knowledge distillation on MS MARCO with a document length of 300 tokens, explaining its default value for documents length.

However, as illustrated in the ModernBERT paper, ColBERT models can generalize to documents lengths way beyond their training length and GTE-ModernColBERT actually yields results way above SOTA in long-context embedding benchmarks, see LongEmbed results.

Simply change adapt the document length parameter to your needs when loading the model:

model = models.ColBERT(
    model_name_or_path=lightonai/GTE-ModernColBERT-v1,
    document_length=8192,
)

ModernBERT itself has only been trained on 8K context length, but it seems that GTE-ModernColBERT can generalize to even bigger context sizes, though it is not guaranteed so please perform your own benches!

Model Sources

• Documentation: PyLate Documentation
• Repository: PyLate on GitHub
• Hugging Face: PyLate models on Hugging Face

Full Model Architecture

ColBERT(
  (0): Transformer({'max_seq_length': 299, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
)

Usage

Sentence Transformers

This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:

pip install "sentence-transformers>=6.0.0"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("lightonai/GTE-ModernColBERT-v1")

query = "Which planet is known as the Red Planet?"
documents = [
    "Venus is often called Earth's twin because of its similar size and proximity.",
    "Mars, known for its reddish appearance, is often referred to as the Red Planet.",
    "Jupiter, the largest planet in our solar system, has a prominent red spot.",
    "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
]

query_embeddings = model.encode_query(query)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings[0].shape)
# (12, 128) (18, 128)

# MaxSim late-interaction scoring (higher is more relevant)
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[11.2577, 11.5113, 11.3457, 11.4518]])

PyLate

First install the PyLate library:

pip install -U pylate

Retrieval

PyLate provides a streamlined interface to index and retrieve documents using ColBERT models. The index leverages the Voyager HNSW index to efficiently handle document embeddings and enable fast retrieval.

Indexing documents

First, load the ColBERT model and initialize the Voyager index, then encode and index your documents:

from pylate import indexes, models, retrieve

# Step 1: Load the ColBERT model
model = models.ColBERT(
    model_name_or_path=pylate_model_id,
)

# Step 2: Initialize the Voyager index
index = indexes.Voyager(
    index_folder="pylate-index",
    index_name="index",
    override=True,  # This overwrites the existing index if any
)

# Step 3: Encode the documents
documents_ids = ["1", "2", "3"]
documents = ["document 1 text", "document 2 text", "document 3 text"]

documents_embeddings = model.encode(
    documents,
    batch_size=32,
    is_query=False,  # Ensure that it is set to False to indicate that these are documents, not queries
    show_progress_bar=True,
)

# Step 4: Add document embeddings to the index by providing embeddings and corresponding ids
index.add_documents(
    documents_ids=documents_ids,
    documents_embeddings=documents_embeddings,
)

Note that you do not have to recreate the index and encode the documents every time. Once you have created an index and added the documents, you can re-use the index later by loading it:

# To load an index, simply instantiate it with the correct folder/name and without overriding it
index = indexes.Voyager(
    index_folder="pylate-index",
    index_name="index",
)

Retrieving top-k documents for queries

Once the documents are indexed, you can retrieve the top-k most relevant documents for a given set of queries. To do so, initialize the ColBERT retriever with the index you want to search in, encode the queries and then retrieve the top-k documents to get the top matches ids and relevance scores:

# Step 1: Initialize the ColBERT retriever
retriever = retrieve.ColBERT(index=index)

# Step 2: Encode the queries
queries_embeddings = model.encode(
    ["query for document 3", "query for document 1"],
    batch_size=32,
    is_query=True,  #  # Ensure that it is set to False to indicate that these are queries
    show_progress_bar=True,
)

# Step 3: Retrieve top-k documents
scores = retriever.retrieve(
    queries_embeddings=queries_embeddings, 
    k=10,  # Retrieve the top 10 matches for each query
)

Reranking

If you only want to use the ColBERT model to perform reranking on top of your first-stage retrieval pipeline without building an index, you can simply use rank function and pass the queries and documents to rerank:

from pylate import rank, models

queries = [
    "query A",
    "query B",
]

documents = [
    ["document A", "document B"],
    ["document 1", "document C", "document B"],
]

documents_ids = [
    [1, 2],
    [1, 3, 2],
]

model = models.ColBERT(
    model_name_or_path=pylate_model_id,
)

queries_embeddings = model.encode(
    queries,
    is_query=True,
)

documents_embeddings = model.encode(
    documents,
    is_query=False,
)

reranked_documents = rank.rerank(
    documents_ids=documents_ids,
    queries_embeddings=queries_embeddings,
    documents_embeddings=documents_embeddings,
)

Evaluation

Metrics

BEIR Benchmark

GTE-ModernColBERT is the first model to outpeform ColBERT-small on the BEIR benchmark. As reproduction in the IR domain is challenging, we worked closely with Benjamin Clavié, the author of ColBERT-small to reproduce the evaluation setup of this model. Despite all these efforts and reducing to the maximum the difference in scores in most of the datasets, some are still a bit different. For this reason, we also report the results of ColBERT-small in the same setup we used to evaluate GTE-ModernColBERT for completness and fair comparison.

Model	Average	FiQA2018	NFCorpus	TREC-COVID	Touche2020	ArguAna	QuoraRetrieval	SCIDOCS	SciFact	NQ	ClimateFEVER	HotpotQA	DBPedia	CQADupstack	FEVER	MSMARCO
GTE-ModernColBERT	54.67	45.28	37.93	83.59	31.23	48.51	86.61	19.06	76.34	61.8	30.62	77.32	48.03	41	87.44	45.32
ColBERT-small (reported)	53.79	41.15	37.3	84.59	25.69	50.09	87.72	18.42	74.77	59.1	33.07	76.11	45.58	38.75	90.96	43.5
JinaColBERT-v2		40.8	34.6	83.4	27.4	36.6	88.7	18.6	67.8	64	23.9	76.6	47.1		80.5
ColBERT-small (rerunned)	53.35	41.01	36.86	83.14	24.95	46.76	87.89	18.72	74.02	59.42	32.83	76.88	46.36	39.36	88.66	43.44

LongEmbed Benchmark

GTE-ModernColBERT has been trained with knowledge distillation on MS MARCO with a document length of 300 tokens, explaining its default value for documents length. However, as illustrated in the ModernBERT paper, ColBERT models can generalize to documents lengths way beyond their training length and GTE-ModernColBERT actually yields results way above SOTA (almost 10 points above previous SOTA) in long-context embedding benchmark:

Model	Mean	LEMBNarrativeQARetrieval	LEMBNeedleRetrieval	LEMBPasskeyRetrieval	LEMBQMSumRetrieval	LEMBSummScreenFDRetrieval	LEMBWikimQARetrieval
GTE-ModernColBERT (with 32k document length)	88.39	78.82	92.5	92	72.17	94.98	99.87
voyage-multilingual-2	79.17	64.694	75.25	97	51.495	99.105	87.489
inf-retriever-v1	73.19	60.702	61.5	78.75	55.072	97.387	85.751
snowflake-arctic-embed-l-v2,0	63.73	43.632	50.25	77.25	40.04	96.383	74.843
gte-multilingual-base	62.12	52.358	42.25	55.5	43.033	95.499	84.078
jasper_en_vision_language_v1	60.93	37.928	55	62.25	41.186	97.206	72.025
bge-m3	58.73	45.761	40.25	59	35.543	94.089	77.726
jina-embeddings-v3	55.66	34.297	64	38	39.337	92.334	66.018
e5-base-4k	54.51	30.03	37.75	65.25	31.268	93.868	68.875
gte-Qwen2-7B-instruct	47.24	45.46	31	38.5	31.272	76.08	61.151

ModernBERT itself has only been trained on 8K context length, but it seems that GTE-ModernColBERT can generalize to even bigger context sizes, though it is not guaranteed so please perform your own benches!

PyLate Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with pylate.evaluation.pylate_information_retrieval_evaluator.PyLateInformationRetrievalEvaluator

Metric	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoFiQA2018	NanoHotpotQA	NanoMSMARCO	NanoNFCorpus	NanoNQ	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
MaxSim_accuracy@1	0.36	0.88	0.92	0.56	0.92	0.54	0.56	0.64	0.96	0.48	0.3	0.74	0.7755
MaxSim_accuracy@3	0.62	0.94	0.98	0.66	1.0	0.68	0.68	0.82	1.0	0.74	0.62	0.86	0.9388
MaxSim_accuracy@5	0.78	0.96	0.98	0.74	1.0	0.74	0.74	0.86	1.0	0.78	0.7	0.9	0.9796
MaxSim_accuracy@10	0.86	0.98	1.0	0.8	1.0	0.92	0.76	0.9	1.0	0.84	0.82	0.94	0.9796
MaxSim_precision@1	0.36	0.88	0.92	0.56	0.92	0.54	0.56	0.64	0.96	0.48	0.3	0.74	0.7755
MaxSim_precision@3	0.2333	0.7133	0.36	0.3267	0.58	0.2267	0.4333	0.2867	0.4	0.4	0.2067	0.3	0.6599
MaxSim_precision@5	0.208	0.656	0.216	0.256	0.36	0.148	0.392	0.18	0.256	0.292	0.14	0.196	0.6571
MaxSim_precision@10	0.128	0.572	0.11	0.152	0.186	0.092	0.304	0.1	0.134	0.194	0.082	0.104	0.5184
MaxSim_recall@1	0.1833	0.118	0.8567	0.3092	0.46	0.54	0.0664	0.61	0.8473	0.1007	0.3	0.715	0.0518
MaxSim_recall@3	0.289	0.2307	0.96	0.4784	0.87	0.68	0.102	0.78	0.9453	0.2467	0.62	0.83	0.1362
MaxSim_recall@5	0.4157	0.2962	0.96	0.5752	0.9	0.74	0.1284	0.82	0.9693	0.2997	0.7	0.885	0.2193
MaxSim_recall@10	0.4957	0.4146	0.98	0.6412	0.93	0.92	0.1566	0.88	0.9893	0.3967	0.82	0.93	0.334
MaxSim_ndcg@10	0.4148	0.7296	0.9452	0.567	0.9012	0.7089	0.3957	0.7645	0.9691	0.3987	0.5609	0.8372	0.5927
MaxSim_mrr@10	0.5266	0.9169	0.9522	0.6359	0.96	0.6447	0.627	0.739	0.9767	0.6137	0.4775	0.8117	0.8629
MaxSim_map@100	0.3347	0.5884	0.9271	0.5032	0.8592	0.6496	0.1918	0.7239	0.9552	0.3163	0.4824	0.8049	0.4257

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with pylate.evaluation.nano_beir_evaluator.NanoBEIREvaluator

Metric	Value
MaxSim_accuracy@1	0.6643
MaxSim_accuracy@3	0.8107
MaxSim_accuracy@5	0.8584
MaxSim_accuracy@10	0.9077
MaxSim_precision@1	0.6643
MaxSim_precision@3	0.3943
MaxSim_precision@5	0.3044
MaxSim_precision@10	0.2059
MaxSim_recall@1	0.3968
MaxSim_recall@3	0.5514
MaxSim_recall@5	0.6084
MaxSim_recall@10	0.6837
MaxSim_ndcg@10	0.6758
MaxSim_mrr@10	0.7496
MaxSim_map@100	0.5971

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• learning_rate: 3e-05
• bf16: True

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.11.10
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.48.2
• PyTorch: 2.5.1+cu124
• Accelerate: 1.1.1
• Datasets: 2.21.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084"
}

PyLate

@misc{PyLate,
title={PyLate: Flexible Training and Retrieval for Late Interaction Models},
author={Chaffin, Antoine and Sourty, Raphaël},
url={https://github.com/lightonai/pylate},
year={2024}
}

GTE-ModernColBERT

@misc{GTE-ModernColBERT,
title={GTE-ModernColBERT},
author={Chaffin, Antoine},
url={https://huggingface.co/lightonai/GTE-ModernColBERT-v1},
year={2025}
}