Massspecgym In The Wild

Terms

in silico: experimentation done on the computer
shortcut learning: Decision rules that perform well with benchmarks but do not transfer to real world or more stringent testing of the model.
Overfitting: Model fits training data too closely, and cannot make accurate predictions about non training data.
data leakage: test data are accidentally available during training/evaluation. Which makes the model performance look better than it should be
MCES: Maximum Common Edge Subgraph, similarity measure (distance) used for splitting into train/test without being too similar.
Tanimoto Similarity: Ranges from 0.0 to 1.0, measures similarity between two molecular fingerprints
InChIKey-14: First 14 characters of an InChIKey.
Chemical formula leak: Occurs when model gains information about formula which should be unknown to the model.
Candidate set: list of possible molecules retrieval model ranks the spectrum against.

Introduction

Misusing benchmarks can result in shortcut learning, data leakage, and overfitting.

MassSpecGym breaks the molecule identification prcoess down into 3 tasks: 1) spectrum simulation challenge 2) retrieval challenge 3) de novo generation challenge - where the model makes structures from the MS

There are three commonly found types of failures for training models: 1) Data leakage: Data splitting on exact structures can yield test splits that have tests that are strongly represented in the training data. 2) Shortcut learning: inconsistencies, such as in SMILES canonicalization, can allow for shortcutting, which causes high fit for the model without working on the spectrum-molecule relationship 3) Metrics and implementation divergence: Unchecked use of divergent metrics and implementations and drift from fair evaluation standards that MassSpecGym should be enforcing.

Data Leakage

MassSpecGym tries to avoid leakage with MCES training / validation splits, rather than exact molecule matching. Because exact exclusion is insufficient and the model can still identify close analogs during the pretraining and falsely appear to generalize better than the model actually can.

This is important for de novo models, such as 1) spectrum-to-fingerprint encoder 2) fingerprint-to-molecule decoder

The issue is that the model could be learning - memorizing rather, the molecule space and not learning the MS/MS structure elucidation needed.

Recommendations for training against data leaks

exclude exact test matches from pretraining data
exclude close analogs
report performance stratified by test-set similarity to pretraining molecules

Synthetic spectra leakage

If a spectral simulator is trained on test molecules, then used to generate synthetic spectra for training another model, the leakage transfers through the simulator.

This is “transitive leakage”:
test molecule -> simulator -> synthetic spectra -> downstream model

Formula leakage

Formula prediction is not solved, especially for halogenated molecules or higher mass molecules.

MassSpecGym has:

formula-based track: formula is known
mass-based track: formula is not known If a model uses ground-truth formula but reports results on the mass-based track, that leaks information. Random retrieval improves massively if ground-truth formula is used to filter candidates, showing formula information is very powerful.

Shortcut learning in retrieval

MassSpecGym had candidate artifacts where true molecules and decoys had different SMILES formatting/canonicalization.

A model could learn:

which SMILES looks different?
instead of:
which molecule matches this spectrum?

Which produces very inflated retrieval scores.

MassSpecGym v1.5 fixes this by enforcing uniform RDKit canonicalization across query and candidate SMILES.

PubChem ranking bias

Another shortcut: PubChem candidate order itself can leak useful information.

A spectrum-blind baseline using PubChem default ranking reached high hit rate, meaning the candidate set has bias independent of spectra.
This is worse than just a bug. It means the benchmark design itself can let models exploit candidate distributions.

Implementation bugs

Benchmark reuse can spread hidden bugs. They found a MIST batching/padding mask bug. Padding tokens were not properly excluded in attention, causing inflated/inconsistent behavior in batched inference. Later methods reused MIST checkpoints/components, so one bug propagated into multiple papers.