Skip to search formSkip to main contentSkip to account menu

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

This work proposes FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM, and is optimal for a range of SRAM sizes.
arXiv (opens in a new tab)

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

This work tweak the algorithm to reduce the number of non-matmul FLOPs, and parallelize the attention computation, even for a single head, across different thread blocks to increase occupancy, and distribute the work between warps to reduce communication through shared memory.
arXiv (opens in a new tab)

StarCoder: may the source be with you!

This work performs the most comprehensive evaluation of Code LLMs to date and shows that StarCoderBase outperforms every open Code LLM that supports multiple programming languages and matches or outperforms the OpenAI code-cushman-001 model.
arXiv (opens in a new tab)

HiPPO: Recurrent Memory with Optimal Polynomial Projections

This formal framework yields a new memory update mechanism (HiPPO-LegS) that scales through time to remember all history, avoiding priors on the timescale and enjoys the theoretical benefits of timescale robustness, fast updates, and bounded gradients.
arXiv (opens in a new tab)

Hyena Hierarchy: Towards Larger Convolutional Language Models

This work proposes Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating, and sets a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets.
arXiv (opens in a new tab)

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

A simple sequence model inspired by control systems that generalizes RNN heuristics, temporal convolutions, and neural differential equations while addressing their shortcomings, and introduces a trainable subset of structured matrices that endow LSSLs with long-range memory.
arXiv (opens in a new tab)

Hungry Hungry Hippos: Towards Language Modeling with State Space Models

A new SSM layer, H3, is proposed that is explicitly designed for the impact on language modeling and achieves promising initial results, achieving lower perplexity than Transformers and outperforming Transformers in zero- and few-shot learning on a majority of tasks in the SuperGLUE benchmark.
arXiv (opens in a new tab)

Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time

DejaVu is proposed, a system that uses a low-cost algorithm to predict contextual sparsity on the fly given inputs to each layer, along with an asynchronous and hardware-aware implementation that speeds up LLM inference.
arXiv (opens in a new tab)

Monarch: Expressive Structured Matrices for Efficient and Accurate Training

Surprisingly, the problem of approximating a dense weight matrix with a Monarch matrix, though nonconvex, has an analytical optimal solution and can achieve favorable accuracy-efficiency tradeoffs in several end-to-end sparse training applications.
arXiv (opens in a new tab)

Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations

This work introduces a parameterization of divide-and-conquer methods that can automatically learn an efficient algorithm for many important transforms, and can be incorporated as a lightweight replacement of generic matrices in machine learning pipelines to learn efficient and compressible transformations.
PubMed (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)