Notes

• Consider the complexity of any given request (Big O)
• Embedding Dimensions
  • Higher dimensional representations of similarity
  • Clustering like things together
  • Token embeddings for n-tokens created n * d embedding matrices
• Positional Encoding
  • Want the model to understand positionality of tokens
  • Don’t use indexes to reduce memory overhead
  • Nice if bounded between -1 and 1
  • Each encoding should be unique
  • Sinusoidal Positional Encoding
• Connectivity Matrix / Graph
  • Describes connections between nodes and edges
  • Directed
    • All connected nodes are mutually connected
  • Undirected
    • Connected nodes may or may not be mutually connected
• Computational Complexity
  • Attention Layer
    • O(n^2) time and memory
  • Embedding Dimensions
    • Also results in O(n^2) time and space complexity
• Position Encoding (again)
  • Sinusoidal Positional Encoding has no extrapolation ability

ALiBi

• Attention Layer with Linear Biases
• Remove Sinusoidal Positional Encoding
• Attached a bias at the attention head(s)
• Use softmax
• Has a trailing bias
• Closer tokens get a higher bias
• The attention score has a constant bias (slope)
• Greatly speeds up training
• Further out tokens have little influence on the final output

Spare Attention

• Not all tokens are equally important
• Focus on important tokens