I recently dove into Yoshua Bengio et al.’s 2003 paper, “A Neural Probabilistic Language Model”. Reading a paper from over two decades ago is fascinating. What struck me most wasn’t the specific model, which is simple by today’s standards, but how clearly Bengio laid out the core problems of language modeling. I came away with more respect for his vision.

The problem: the curse of dimensionality

Bengio starts by framing the fundamental challenge: the curse of dimensionality. As he puts it,

“…a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training.”

This is because the number of possible sentences is essentially infinite, like the Library of Babel. Any specific sentence has almost zero probability of occurring randomly.

The “curse” goes deeper than the sheer number of sequences. As the number of dimensions, such as sequence length or feature count, increases:

  1. Space expands exponentially: The volume of the space grows very fast, making the available data extremely sparse.
  2. Distance intuition breaks: In high dimensions, points tend to become equidistant from each other, and most of the volume is concentrated far from the center, near the “surface” of the high-dimensional space. Our low-dimensional intuitions about proximity and density fail.
  3. Spurious correlations: With so many dimensions, it becomes easy to find apparent patterns in data that are just noise.

This is a core challenge for many real-world problems, especially with rich sensory data spanning many dimensions. How do you find the signal in such a vast, sparse space without getting lost?

The solution: fighting fire with fire

Bengio and his colleagues proposed a way to fight this curse:

“…learning a distributed representation for words…”

Essentially, they proposed learning dense, low-dimensional feature vectors, or embeddings, for each word in the vocabulary. This is like fighting fire with fire: while the vocabulary space is huge and discrete, the learned feature space is much smaller, for example 30-100 dimensions in their experiments vs. 17k+ words, but continuous. Because it’s a dense continuous space, even a relatively low-dimensional one can represent complex relationships. They are mapping the discrete vocabulary into a structured latent space.

How generalization happens

So how does this help? The paper explains:

“Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence.”

This, for me, is the crux of it. The model learns which words play similar semantic or syntactic roles and places them close together in the embedding space. Because the probability function operates smoothly over this continuous space, seeing “The cat sat on the mat” helps the model assign a higher probability to the unseen sentence “A dog rested on the rug,” because the corresponding words have similar learned representations. This mapping from discrete symbols to a meaningful continuous space is what allows generalization beyond simply memorizing n-grams. This is still central to how current LLMs generalize, even if the systems are much larger now.

Learning end-to-end

A key part of their proposal was point 3:

“learn simultaneously the word feature vectors and the parameters of that probability function.”

They recognized that the embeddings and the prediction mechanism need to learn from each other. You can’t just fix one and train the other; they have to be optimized together, end-to-end, for the embeddings to become useful for prediction and vice versa.

A historical aside: parallel processing with CPUs

What also caught my eye was the extensive discussion of parallelizing the training process. This was 2003, when widespread GPU computing for ML wasn’t a thing yet. They describe parameter-parallel processing across multiple CPUs, up to 64 Athlon processors in their cluster. They discuss asynchronous updates and communication overhead with MPI. It feels like an early version of the massive parallelization, now mostly on GPUs/TPUs, that is essential for training today’s large models.

Lasting impact

While the specific MLP architecture in the paper is rudimentary now, the core ideas still matter: tackle the curse of dimensionality with learned distributed representations, generalize through similarity in embedding space, and train the representations end-to-end. Reading this paper felt like seeing an early version of the framework we’re still working within.