Tokenizers are the fundamental piece of NLP architecture
The standard approach
Processes raw bytes, rather than tokens
Starts with large vocabulary that gets trimmed down
things look the same in the context window and different outside
Context windows
Shifted-target labeling
The inner product determines how related two words are (higher is more related)
There's a fixed-size internal memory As new context shows up, it's compressed into this fixed state
Process text in parallel
The idea is to use rotations to represent the positions of words in text
It's kind of weird that you have to use angles for distance, but it's basically because the way that transformers work, you interpret the dot product as a measure of attention
The different angles need to rotate at different speeds, so you don't end up with orthogonal vectors, or two vectors that are meant to represent different words, but are the exact same location (this is called aliasing)
The problem is that if your angular increments are set ahead of time, if the context window gets really big, eventually vectors can get too close to ones that have already been seen
Another issue is that if the intervals are too small, we get floating point errors
Neural Tangent Kernel
In Dynamic NTK-Aware Scaling you adjust the base frequency as the conversation (and context) expands