A deep dive into the mathematical elegance that makes extreme compression possible

1. Introduction: The Compression Challenge

1.1 The Impossible Dream: Running 70B Models on Your Gaming PC

Picture this: You have a gaming laptop with an RTX 4090—a beast of a card with 24GB of VRAM. You want to run Llama 2 70B, one of the most powerful open-source language models available. Here’s the brutal math:

• At full precision (FP16): 70 billion parameters × 2 bytes = 140GB
• Your available VRAM: 24GB
• The gap: You’d need 6 of your GPUs. Total cost?

This isn’t just an inconvenience, it’s a fundamental barrier. State-of-the-art AI remains locked in data centers, accessible only to well-funded labs and companies. Consumer hardware, edge devices, and even many research institutions are simply shut out.

Quantization promised to change this. By representing weights with fewer bits, we could compress these models to fit on accessible hardware. The progression seemed clear:

• 8-bit quantization (2020-2021): Mostly lossless, 2× compression → 70GB (still too big)
• 4-bit quantization (2022-2023): Near-lossless with methods like GPTQ, AWQ → 35GB (getting closer!)
• 2-bit quantization (2023-2024): The holy grail → ~18GB (fits on a single RTX 4090!)

But there was a problem: nobody could make 2-bit work.

1.2 Why 2-Bit Was Considered Impossible

By late 2023, the field had hit a wall. Every attempt to quantize LLMs below 3 bits resulted in catastrophic quality degradation. The core challenge: LLM weight matrices contain outliers—a small number of weights that are 100-1000× larger than the rest. At 2 bits (only 4 distinct values), there’s insufficient resolution to represent both normal weights and outliers accurately.

Existing Methods Hit Hard Limits

Let’s look at what the state-of-the-art methods achieved at 2 bits on Llama 2 70B (WikiText2 perplexity with context length 2048, lower is better):

The best existing method (OmniQuant at 7.81) was more than 2× worse than the FP16 baseline. Models were incoherent, repetitive, and failed basic reasoning tasks.

The consensus emerged: 4 bits is the practical minimum.

Tim Dettmers and Luke Zettlemoyer even published a paper in 2023 arguing that “4-bit precision is optimal” for LLMs, with diminishing returns below that threshold.

The 2-bit dream seemed dead.

1.3 The QuIP# Breakthrough

Then came QuIP#. The results were unprecedented:

• First method to achieve near-lossless 2-bit quantization (4.16 PPL vs 3.32 FP16 baseline on Llama 2 70B, context 2048)
• 3-bit models outperform “theoretically lossless” 4-bit (3.56 vs 3.47 PPL)
• Scales better than higher bitrates as model size increases

What changed? QuIP# combines three techniques in a principled, mathematically elegant way:

1. Randomized Hadamard Transform (RHT) for incoherence processing
2. E8 lattice codebooks for optimal sphere packing
3. Block-LDLQ adaptive rounding with Hessian awareness

Each component addresses a specific mathematical challenge. Together, they enable what was thought impossible.

But before we dive into QuIP#’s solution, we need to understand the fundamental challenges that made 2-bit quantization seem impossible in the first place.

2. Background: What Makes Quantization Hard?

2.1 Quantization Basics: The Storage vs. Accuracy Bargain

The Core Idea

Imagine you’re moving to a smaller apartment and need to compress your belongings. You could:

• Pack everything loosely (takes many boxes, but nothing gets damaged)
• Compress everything tightly (fits in fewer boxes, but some items might break)

Quantization is exactly this trade-off for neural network weights. Each weight in a model is originally stored as a 16-bit floating-point number (FP16), giving it incredible precision. But do we really need that much precision?

Quantization reduces the memory footprint of LLMs by representing weights with fewer bits:

• FP16 (16 bits): 70 billion parameters × 2 bytes = 140GB
• 4-bit: 70 billion parameters × 0.5 bytes = 35GB (4× compression)
• 2-bit: 70 billion parameters × 0.25 bytes = ~18GB (8× compression) ✓ Fits on RTX 4090!

Why It’s Not Just Rounding

You might think: “Why not just round each weight to the nearest 2-bit value?” Here’s why that fails catastrophically.

Consider a simple weight matrix row: [0.1, 0.15, 0.2, 0.12, 0.18, 0.11, 0.16]

With 2 bits, you can only represent 4 distinct values (say: {0, 0.1, 0.2, 0.3}). Naively rounding each weight independently would map everything to either 0.1 or 0.2, obliterating the subtle differences between 0.11, 0.12, and 0.15 that might be critical for the model’s behavior.

The true challenge: How do you choose which information to preserve when you can only afford 4 distinct values per dimension?

This is where the mathematics gets interesting.

2.2 The Outlier Problem: The 1% That Breaks Everything

The Hidden Structure of LLM Weights

LLM weight matrices aren’t uniform—they contain outliers: a small number of weights that are 100-1000× larger than the rest. This isn’t a bug; it’s a fundamental feature of how transformers learn.

A Visual Example

Typical weights: [0.1, 0.15, 0.2, 0.12, 0.18, 0.11, 0.16]
Outlier weight:  [150.0]

That single outlier dominates the entire matrix. Why? Because in the forward pass, y = W·x, even if x is small, that 150.0 weight creates a huge activation that completely changes the output.

The Quantization Dilemma

You face an impossible choice:

1. Scale for the outlier: Set your 4 quantization levels to cover the range [0, 150]. Now your levels might be {0, 50, 100, 150}. But this crushes all normal weights (0.1, 0.15, 0.2…) to zero! You’ve destroyed 99% of the weights to preserve 1%.

2. Ignore the outlier: Set your levels to {0, 0.1, 0.2, 0.3} to capture normal weights well. But now the 150.0 outlier gets clipped to 0.3—a 500× error that will cause catastrophic failures in the model’s output.

At 4 bits (16 distinct values), you have enough resolution to handle this with techniques like per-group scaling. At 2 bits (only 4 values), the math breaks down.

This is why you can’t just “turn down the bits” and expect things to work. The outlier problem is the fundamental barrier that prevented 2-bit quantization from being viable until QuIP#.

2.3 Why Existing Methods Fail at 2-Bit

By late 2023, the field had hit a wall. Let’s examine what the state-of-the-art methods achieved at 2 bits on Llama 2 70B (context length 2048):

(Lower perplexity = better. The best existing method, OmniQuant at 7.81, was more than 2× worse than the FP16 baseline.)

Why Each Method Failed

1. AWQ & OmniQuant: Use heuristic outlier suppression via activation-aware rescaling. At 2 bits, these heuristics aren’t strong enough—outliers still dominate.

2. GPTQ: Per-group scaling adds 0.25 bits overhead per weight (12% at 2 bits) and only mitigates the outlier problem rather than solving it.

3. SpQR: Stores outliers separately in FP16, but irregular memory access patterns kill GPU performance.

4. AQLM: Achieves good quality but uses 1MB codebooks that cause cache misses, making it slower than FP16.

QuIP#’s Key Insight: Instead of fighting outliers with heuristics, eliminate them entirely through principled mathematical transformation (Randomized Hadamard Transform). We’ll explore this in Section 4.

Now that we understand why existing methods failed, let’s see how QuIP# solves these challenges through three synergistic components.

3. QuIP#’s Three Pillars

3.1 High-Level Architecture

QuIP# is built on three synergistic components:

1. Incoherence Processing with RHT: Transforms weights to eliminate outliers
2. E8 Lattice Codebooks: Matches quantization to the transformed weight distribution
3. Block-LDLQ Adaptive Rounding: Accounts for weight interdependencies

Each component solves a specific mathematical problem:

Original Weights (with outliers)
         ↓
    [RHT Transform]
         ↓
Incoherent Weights (Gaussian, ball-shaped)
         ↓
  [E8P Quantization]
         ↓
Quantized Weights (minimal error)
         ↓
 [Block-LDLQ Rounding]
         ↓
Final Quantized Model

The beauty is in how these pieces fit together. The RHT creates a Gaussian distribution. The E8 lattice is proven optimal for packing spheres in 8D space (exactly what a Gaussian distribution looks like!). And Block-LDLQ uses the Hessian to minimize the final reconstruction error.

Let’s explore each pillar in depth, starting with the transformation that eliminates outliers.

4. Pillar 1: Incoherence Processing with Randomized Hadamard Transform

4.1 The Core Idea: Spread the Risk

The Building Analogy

Imagine a building supported by many pillars. If one pillar bears 99% of the weight, the building will collapse if that pillar fails. But if the weight is evenly distributed across all pillars, the building can withstand the loss of any single support.

Incoherence processing does this for neural network weights. Instead of having a few “weight pillars” bear most of the importance, we redistribute the load so no single weight is critical.

Mathematical Formulation

The magic: We transform weights W using orthogonal matrices U and V:

W' = U·W·V^T

The key properties:

• Preserves the forward pass: y = W·x = U·W'·V^T·x (we just transform inputs/outputs accordingly)
• Redistributes magnitude: Outliers get “spread out” across many weights
• No information loss: Orthogonal matrices are invertible

During inference, we compute: y = U^T·(W'·(V·x))

4.2 What is Incoherence?

Intuitive Explanation

An incoherent matrix has no outliers—all entries have similar magnitude. Think of it like a democracy where no single vote dominates, versus a dictatorship where one voice controls everything.

Formal Definition

For a weight matrix W ∈ ℝ^(m×n), we say it’s μ-incoherent if:

max|W_ij| ≤ μ·||W||_F / √(m·n)

Understanding the Inequality

Let’s break this down:

• W_ij: Single entry at row i, column j
• max|W_ij|: The largest absolute value (the outlier)
• ||W||_F: Frobenius norm = √(Σᵢⱼ W²ᵢⱼ) (total magnitude of all weights)
• √(m·n): Normalization by matrix size
• μ: Incoherence parameter (smaller = better)

What it means in plain English: “The biggest entry ≤ μ × average entry”

Concrete Examples

• Incoherent matrix (μ ≈ 1): All entries ≈ 0.28

  [0.27, 0.29, 0.26, 0.30]
  [0.28, 0.27, 0.31, 0.27]
  [0.29, 0.28, 0.27, 0.29]
  

• Matrix with outlier (μ ≈ 20): One entry = 5.5, others ≈ 0.28

  [0.27, 0.29, 0.26, 0.30]
  [0.28, 5.50, 0.31, 0.27]  ← Outlier!
  [0.29, 0.28, 0.27, 0.29]
  

4.3 The Randomized Hadamard Transform (RHT)

Why Hadamard?

The Hadamard matrix has special properties that make it perfect for incoherence processing:

1. Orthogonal: Preserves information (no loss)
2. Binary entries: All entries are ±1 (no floating-point multiplies needed!)
3. Fast to compute: O(n log n) via Fast Walsh-Hadamard Transform

The RHT Construction

The transformation is:

W' = H·diag(S_U)·W·diag(S_V)·H^T

Where:

• H: Hadamard matrix (orthogonal, entries in {-1, +1})
• S_U, S_V: Random sign vectors (diagonal ±1 matrices)

Theoretical Guarantees

Lemma 3.1 from the QuIP# paper: With high probability (1-δ), the RHT achieves:

μ_H = √(2·log(2n²/δ))

This is a major improvement over QuIP’s Kronecker approach:

• QuIP (Kronecker): μ = O(log² n)
• QuIP# (RHT): μ = O(√log n)

Better incoherence means less quantization error!

Runtime Improvement

• QuIP (Kronecker): O(n√n) operations
• QuIP# (RHT): O(n log n) operations

For Llama 2 70B with n=28,672: This is ~170× faster for the transform!

4.4 Handling Non-Power-of-2 Dimensions

The Challenge

Hadamard matrices exist for dimensions 1, 2, and most multiples of 4 (the Hadamard conjecture). However, for efficient computation using the Fast Walsh-Hadamard Transform, we prefer power-of-2 dimensions. But LLMs have all sorts of dimensions:

• Llama 2 70B has intermediate dimension 28,672 = 1024 × 28

The Solution: Kronecker Product

We can factorize n = p×q where p is a power of 2 and q is another valid Hadamard dimension:

H = H_p ⊗ H_q

For Llama 2: H = H_1024 ⊗ H_28

This gives us:

• Compute time: O(q²p log p)
• For 28,672: Much faster than QuIP’s O(n√n)

4.5 Why This Works for Quantization

After RHT, weights become approximately Gaussian distributed:

• Before: Spiked distribution with outliers
• After: Smooth Gaussian bell curve

This is crucial because:

1. No single weight is critical → quantization errors spread evenly
2. Roughly ball-shaped in high dimensions → perfect for E8 lattice (next section!)
3. Predictable error bounds → we can prove theoretical guarantees

With weights now transformed into a Gaussian distribution free of outliers, we face a new challenge: how do we quantize this ball-shaped distribution efficiently? This is where the mathematics of sphere packing becomes crucial.

5. Pillar 2: Vector Quantization with E8 Lattice Codebooks

5.1 The Shape-Matching Problem

5.1.1 Ball-Shaped Gaussian Distribution

After RHT, weights are approximately Gaussian. In multiple dimensions, this creates a “ball shape”—weights are radially symmetric around the origin.

Think of throwing darts at a dartboard. Most land near the center, fewer toward the edges. In 8 dimensions, Gaussian weights do the same thing—they cluster in a “ball” around zero.

5.1.2 Why Scalar Quantization Fails

Scalar quantization treats each dimension independently. This creates a hypercube of representable points.

The problem: Most of the cube’s volume is in the corners, but weights never appear there (Gaussian samples don’t reach the corners). We’re wasting precious bits on regions of space we’ll never use!

The Math: For a d-dimensional unit cube, the ratio of corner volume to ball volume grows exponentially:

• 2D: ~21% waste
• 4D: ~47% waste
• 8D: ~69% waste

At 2 bits, we can’t afford to waste 69% of our representable space!

5.2 Enter Vector Quantization

The Idea

Instead of quantizing each weight individually, we quantize d weights together as a d-dimensional vector.

• Scalar quantization: 4 values per dimension → hypercube
• Vector quantization: Shape the codebook to match the actual distribution → sphere

The Trade-off

Vector quantization has exponential cost:

• Codebook size: 2^(k·d) entries for k bits and dimension d
• Example: 2 bits, 8 dimensions → 2^16 = 65,536 codewords

This is where the E8 lattice becomes magical.

5.3 The Sphere Packing Problem

5.3.1 What Is Sphere Packing?

The Question: How do you arrange equal-sized spheres to achieve maximum coverage of space?

This is an ancient mathematical problem, dating back to Kepler’s study of cannonball stacking in 1611.

Relevance to Quantization: Each codebook entry is a sphere center. The sphere radius determines how far weights can be from that center. Better packing = smaller max distance = lower quantization error.

2D Examples:

• Square packing: 78.5% efficiency, 4 neighbors touching each sphere
• Hexagonal packing: 90.7% efficiency, 6 neighbors touching each sphere

The hexagonal packing is proven optimal in 2D. We want the same for higher dimensions!

5.3.2 The Kissing Number

The kissing number is how many equal-sized spheres can touch one central sphere.

Examples:

• 2D square grid: 4 neighbors
• 2D hexagonal: 6 neighbors
• 3D: 12 neighbors (think of oranges at the grocery store)
• 8D E8 lattice: 240 neighbors!

Higher kissing number = denser packing = better quantization!

5.3.3 Direct Connection to Quantization

The connection between sphere packing and quantization is direct:

• Codebook entries = sphere centers
• Sphere radius = coverage area (how far weights can be from nearest codeword)
• Better packing = lower max distance = lower quantization error

In the error bound, the covering radius appears as quantization noise σ². E8’s optimal packing minimizes σ², which directly minimizes the final quantization error.

5.4 Common Lattices in Quantization

The E8 lattice achieves the proven optimal packing in 8 dimensions. This is not a heuristic—it’s a mathematical certainty (Viazovska, 2016, Fields Medal 2022).

5.5 The E8 Lattice: Mathematical Beauty

5.5.1 Definition

The E8 lattice is defined as:

E_8 = (ℤ⁸ ∪ (ℤ+½)⁸) ∩ {x | Σx_i is even}

In plain English:

• All-integer OR all-half-integer vectors
• With even coordinate sum

Valid E8 points:

• [1, 1, 1, 1, 1, 1, 1, 1] ✓ (all integers, sum=8 is even)
• [0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5] ✓ (all half-integers, sum=4 is even)
• [1, 0, 1, 0, 1, 0, 1, 0] ✓ (all integers, sum=4 is even)

Invalid points:

• [1, 1, 1, 0, 0, 0, 0, 0] ✗ (sum=3 is odd)
• [0.5, 0.5, 1, 1, 1, 1, 1, 1] ✗ (mixed integer and half-integer)

5.5.2 Why E8 is Special

1. Proven optimal: Maryna Viazovska proved in 2016 that E8 achieves the densest sphere packing in 8D (Fields Medal 2022)

2. Highest kissing number: 240 in 8D—this is proven to be the best possible

3. Highly symmetric: Has 696,729,600 symmetries, which enables compression

4. Hardware-friendly: The structure allows for the E8P compression trick (next section)

5.6 E8P: The “Padded” Compression Trick

5.6.1 The Challenge

For 2-bit quantization in 8 dimensions, we need:

• 2 bits × 8 dimensions = 16 bits total per 8-weight block
• 2^16 = 65,536 codewords

Naively storing this:

• 65,536 vectors × 8 dimensions × 2 bytes = 1 MB per codebook

The Problem: L1 cache on modern GPUs is 128-256 KB. A 1MB codebook won’t fit! This causes cache misses, making inference slower than FP16 (as AQLM discovered).

5.6.2 E8P Solution: Exploit Symmetry

The insight: We don’t need to store all 65,536 vectors. E8’s symmetry lets us:

1. Store only 256 base vectors (4 KB)
2. Use the remaining 8 bits to encode sign flips and shifts
3. Generate all 65,536 points on the fly

Codebook compression: 65,536 entries → 256 base vectors (256× smaller codebook)

Note: Each weight still uses 16 bits for encoding. The compression is in the codebook size (1 MB → 4 KB), not the encoding size. This makes the codebook cache-resident, enabling fast lookups.

5.6.3 E8P Encoding Structure (16 bits)

Each 16-bit codeword encodes:

[8 bits: base index] [7 bits: sign flips] [1 bit: shift]

• Bits 0-7: Index into 256-entry table S
• Bits 8-14: Which coordinates to negate
• Bit 15: Add ±0.25 shift

5.6.4 Decoding Example (Step-by-Step)

Let’s decode codeword: 0001010110010111

Step 1: Base vector

• Bits 0-7: 00010101 = 21
• Look up S[21] = [0.5, 0.5, 0.5, 1.5, 0.5, 0.5, 0.5, 0.5]

Step 2: Apply sign flips

• Bits 8-14: 1001011 (4 ones = even count)
• Base is all-half-integers (needs even # of flips to stay in E8)
• Flip positions 0, 1, 3, 6
• Infer 8th bit from parity constraint
• Result: [-0.5, -0.5, 0.5, -1.5, 0.5, 0.5, -0.5, -0.5]

Step 3: Apply shift

• Bit 15: 1 → add 0.25
• Final: [-0.25, -0.25, 0.75, -1.25, 0.75, 0.75, -0.25, -0.25]

5.6.5 Why 7 Sign Bits (Not 8)?

This is elegant! E8 has a parity constraint:

• If the base vector requires an even # of flips → 8th sign bit is determined by the other 7
• Given 7 sign bits → parity determines the 8th bit automatically

We save 1 bit per codeword by exploiting mathematical structure!

5.6.6 Hardware Implementation

Decoding E8P is incredibly fast:

1. Load base: 1 memory access (L1 cache hit, 4KB total)
2. Extract signs: 1 shift + AND operation
3. Compute 8th sign: Hardware popcount (XOR parity)
4. Apply signs: SIMD multiply (8 parallel ops)
5. Apply shift: SIMD add (8 parallel ops)

Total: ~5 instructions per weight, all cache-resident!

This is why QuIP# achieves >50% of peak memory bandwidth, while AQLM is slower than FP16.

We’ve transformed weights to eliminate outliers (RHT) and matched our quantization to their distribution (E8P). But there’s one final piece: accounting for how weights interact with each other during quantization.

6. Pillar 3: Block-LDLQ Adaptive Rounding

6.1 Why Adaptive Rounding?

Even with RHT and E8P, we face a final challenge: weights aren’t independent.

Imagine you’re tuning a guitar. If you tune each string in isolation, the guitar might sound terrible because strings interact to create chords. You need to tune them together, considering how they affect each other.

Similarly, when we round weights, errors in early weights affect later ones. Adaptive rounding accounts for these interdependencies.

6.2 The Hessian: Measuring Sensitivity

The proxy Hessian captures how changes in weights affect model loss:

H = 𝔼[x·x^T]

Where:

• 𝔼[·]: Expected value (average over calibration data samples)
• x: Input activations to the layer
• H_ij: Measures how much weights i and j “interact”

Intuition: High Hessian values mean “this weight is sensitive—round carefully!”

6.3 Block LDL Decomposition

For a Hessian H ∈ ℝ^(n×n) with block size g, we compute:

H = L^T·D·L

Where:

• L: Unit block lower-triangular matrix (g×g blocks)
• D: Block diagonal matrix

This is like breaking a big problem into smaller, manageable chunks of size g.

6.4 The Block-LDLQ Algorithm

For each block of g weights:

Ŵ_k = Q(W_k + (W_{1:k-1} - Ŵ_{1:k-1})·A_k)

What this means:

• W_k: Current block of weights
• Ŵ_{1:k-1}: Already-quantized previous blocks
• A_k: Feedback matrix (from L)
• Q: Vector quantization to E8P codebook

The key: We use errors from previous blocks as feedback, so errors don’t accumulate!

6.5 Theoretical Guarantee

Theorem 4.1 from the QuIP# paper: For μ-incoherent weights with E8P codebook:

𝔼[Error] ≤ (g·m·μ²·σ²/n)·tr(H^{1/2})²

Where:

• σ²: Quantization noise (minimized by E8P!)
• μ: Incoherence (minimized by RHT!)
• g: Block size (8 for QuIP#)

The beauty: Both RHT and E8P appear in the error bound! Each component directly reduces the final error.

Now let’s step back and see how all three pillars work together to achieve what was thought impossible.

7. The Complete Picture: Why QuIP# Works

7.1 The Virtuous Cycle

RHT 
  → Gaussian Weights (ball-shaped, μ ≈ √log n)
      ↓
  E8 optimal packing (matches Gaussian shape)
      ↓
  Minimizes covering radius σ²
      ↓
  Low quantization noise in Block-LDLQ
      ↓
  Near-lossless 2-bit quantization!

7.2 Each Component is Necessary

Remove any piece and the system fails:

• Without RHT: Outliers remain → high μ → error bound explodes
• Without E8P: Poor sphere packing → high σ² → error bound explodes
• Without Block-LDLQ: No Hessian adaptation → accumulated error

7.3 The Mathematical Beauty

All three components appear in the error bound:

Error ≤ (g·μ²·σ²/n)·tr(H^{1/2})²
        ↑   ↑   ↑
        |   |   └─ E8P minimizes this (optimal packing)
        |   └───── RHT minimizes this (incoherence)
        └───────── Block-LDLQ optimizes given μ and σ²

This isn’t accidental—it’s mathematically inevitable that these three techniques combine to minimize error.

With the theory established, let’s examine what QuIP# means for real-world deployment and the future of LLM accessibility.

8. Practical Implications

8.1 Deployment Scenarios Unlocked

Before QuIP#:

• Llama 2 70B: Requires 140GB (6× RTX 4090s ≈ $60k)
• Research teams locked out of state-of-the-art models
• Edge deployment impossible

After QuIP#:

• Llama 2 70B: ~18GB ✓ Fits on single RTX 4090 ($1,600)
• 7B models: 4-6GB → runs on smartphones
• Cost reduction: 7× memory → 7× more models per server
• Privacy: Sensitive data processing entirely on-device

8.2 The Scaling Breakthrough

The unprecedented result: QuIP# 3-bit scales better than 4-bit.

This directly refutes the 2023 consensus that “4-bit is optimal.” As models get larger, QuIP# 2-bit appears to scale similarly to 3-bit and 4-bit, suggesting that 2-bit may become the new standard.

9. Conclusion

QuIP# achieves what was thought impossible: near-lossless 2-bit quantization of LLMs. The key insights:

1. Eliminate outliers through principled RHT transformation (not heuristic suppression)
2. Match the distribution using proven-optimal E8 lattice sphere packing
3. Account for dependencies through Block-LDLQ adaptive rounding

Each component addresses a specific mathematical challenge. Together, they form an elegant solution that:

• Enables 70B models on consumer hardware
• Achieves unprecedented compression with minimal quality loss
• Scales better than “theoretically optimal” 4-bit methods

The 2-bit dream is alive.

For practitioners: QuIP# quantized models are available at https://huggingface.co/relaxml. Code at https://github.com/Cornell-RelaxML/quip-sharp.

The future of LLM deployment just got a lot more accessible.

References

• Tseng, A., Chee, J., Sun, Q., Kuleshov, V., & De Sa, C. (2024). QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. ICML 2024.
• Viazovska, M. (2017). The sphere packing problem in dimension 8. Annals of Mathematics.
• Chee, J., Cai, Y., Kuleshov, V., & De Sa, C. (2023). QuIP: 2-bit quantization of large language models with guarantees. NeurIPS 2023.