D-Matrix Raptor 3D-DRAM Accelerator for Generative Inference at Hot Chips 2026

Recorded: Sept. 14, 2026, 12:09 p.m.

Original

Summarized

d-Matrix Raptor 3D-DRAM Accelerator for Generative Inference at Hot Chips 2026 - ServeTheHome

Facebook

RSS

TikTok

Youtube

Forums
AI
Server

Server Systems
Server CPUs
Accelerators
Server Motherboards
Server Chassis
Other Components

5G Edge
Storage
Networking
Workstation

Workstation Processors
Workstation Motherboards

Software

Operating Systems
Server Applications
Virtualization

Guides

Buyer’s Guides
Tips
Top Hardware Components for TrueNAS / FreeNAS NAS Servers
Top Hardware Components for pfSense Appliances
Top Hardware Components for napp-it and Solarish NAS Servers
Top Picks for Windows Server 2016 Essentials Hardware
The DIY WordPress Hosting Server Hardware Guide

Facebook

RSS

TikTok

Youtube

Monday, September 14, 2026

ServeTheHome News
Storage Reliability

Raid Calculator
RAID Reliability Calculator | Simple MTTDL Model

About

Contact
Editorial and Copyright Policies

AG Substack
AG Reports
Subscribe
ServeTheHome Forums

your username
your password

Forgot your password? Get help
Privacy Policy

Password recovery
Recover your password

your email

A password will be e-mailed to you.

ServeTheHome

Forums
AI
Server

Server Systems
Server CPUs
Accelerators
Server Motherboards
Server Chassis
Other Components

5G Edge
Storage
Networking
Workstation

Workstation Processors
Workstation Motherboards

Software

Operating Systems
Server Applications
Virtualization

Guides

Home Server Accelerators d-Matrix Raptor 3D-DRAM Accelerator for Generative Inference at Hot Chips 2026

ServerAccelerators

d-Matrix Raptor 3D-DRAM Accelerator for Generative Inference at Hot Chips 2026

By Patrick Kennedy - August 23, 2026 0

FacebookXPinterestLinkedinReddItEmailPrintCopy URL

d-Matrix Raptor 3D-DRAM
Next up, d-Matrix is presenting its Raptor 3D-DRAM accelerator for generative inference at Hot Chips 2026. The company has made waves, and we have covered it before, including the d-Matrix Corsair In-Memory Computing for AI Inference at Hot Chips 2025. We also found they were doing networking in The New d-Matrix JetStream 400G Ethernet Card for Data Center Scale AI Inference. Let us see what they have going on this year.
This is being done live, so please excuse typos.

d-Matrix Raptor 3D-DRAM Accelerator for Generative Inference at Hot Chips 2026
Model weights keep growing, and the KV cache scales with context length multiplied by batch size. So 64 users at 1M context can mean roughly 935 GB of KV cache. Weights and cache together create a problem that is both a capacity problem and a bandwidth problem, and both sides keep growing.
d-Matrix The Growing Data Problem
SRAM meets the bandwidth target, but only on a tiny scale. A Corsair SRAM accelerator card pair reaches roughly 300 TB/s at about 1 ns latency, yet holds only about 4 GB. A 6T SRAM cell is around 10 times larger than a DRAM cell, and leakage runs to tens of watts at GB scale. This makes SRAM suitable for a draft model in speculative decoding, not for holding frontier model weights. That seems to be what NVIDIA is using Groq for as an example.
d-Matrix SRAM: Bandwidth Advantage
HBM solves the capacity half but struggles on bandwidth. Pin speed and I/O width per base die improve slowly, and the number of stacks is limited by available package beachfront, roughly 8-16 stacks per package. d-Matrix cites a practical bandwidth ceiling around 20 TB/s for HBM4 packages such as the NVIDIA Vera Rubin and AMD Instinct MI455.
d-Matrix HBM: The Bandwidth Issue
Bandwidth that high carries a power price. At 2.4 pJ/bit, pushing 100 TB/s through HBM eats about 1.92 kW before any fabric traffic is counted. Packages today lack both the beachfront and the power budget to reach SRAM-class bandwidth with HBM.
d-Matrix HBM: The Power Problem
d-Matrix’s answer is to stack compute directly on top of DRAM dies. Stacking creates a thermal challenge because hundreds of watts must escape through TSVs in a temperature-sensitive DRAM stack, plus a power-delivery challenge from IR drop. d-Matrix says a 1-Hi logic-on-top stack at no more than 0.5 W/mm2 can be liquid cooled and keep DRAM under 100 C.
d-Matrix 3D-DRAM: An idea whose time has come
3D DRAM lands between the two extremes on an energy ladder. On-die SRAM costs roughly 50 fJ, while 2.5D HBM4 systems run in the 2.5 to 5 pJ range when chip-level energy is included. Vertical 3D IO comes in at around 0.3 to 0.4 pJ, about 10 times lower than HBM, because it is a PHY-less millimeter-scale path rather than a centimeter-scale interposer route. Fewer stacked layers than HBM also means a larger die and better yield.
d-Matrix Why 3D-DRAM?
d-Matrix is now mapping that view of technologies onto how LLM inference workloads behave. Prefill processes many prompt tokens in parallel and is compute-throughput-bound, whereas decode produces one token at a time and is typically memory-bandwidth-bound. Attention can flip to compute-bound with high GQA and speculative decoding, and MoE stays memory-bound even at modest batch sizes. Decode is the phase that wants huge bandwidth. If you saw our NVIDIA GB10 or AMD Strix Halo coverage, memory bandwidth is the big challenge with those types of systems.
d-Matrix LLM Inference
Since decode dominates wall-clock runtime, the memory-bound portion matters most. d-Matrix highlights that most inference time is spent in the decode phase, so improving decode bandwidth improves overall inference performance.
d-Matrix Majority of wall-clock inference time is spent in decode
At 32GB per card, with 4-bit weights and an 8-bit KV cache, d-Matrix sizes to fit in one rack. A 72-card scale-up can host a frontier model such as Kimi K3 at 1M context. Disaggregation and multi-rack extend beyond a single Raptor rack.
d-Matrix 1Hi 32GB 3D-DRAM: Frontier LLMs fit in one Raptor Rack
Building the system around this memory is a co-design exercise across the memory subsystem, data movement fabric, and workload mapping.
d-Matrix Building a 3D-DRAM Based Inference System
Now d-Matrix is showing its topology using the full mesh package and discussing its communication protocol.
d-Matrix Low Latency Fabric Intra-Card and Inter-Card
d-Matrix’s specific implementation is called Raptor. A TSMC N4 logic die sits on top of a 3D DRAM die using 36 um face-to-face stacking, a process d-Matrix describes as proven, low-cost, high-volume, and high-yield.
d-Matrix Raptor 3D-DRAM
Turning the dies into a working system exposes a broad set of integration challenges. d-Matrix is highlighting four here.
d-Matrix The 3D-DRAM Integration Landscape
Those four problems are not independent. d-Matrix walks through three entangled challenges in bank mapping, I/O power, and thermal reliability, noting that a solution to any one constrains the design space of the other two.
d-Matrix Challenges Are Entangled
Each tensor engine needs a 128B flit per access, and with 32B delivered per column access from 32B banks, that works out to needing 4 banks per channel. d-Matrix’s die has 840 banks, 768 after 72 spares, spread across 256 channels for just 3 banks per channel. This flit does not divide evenly across what is available.
d-Matrix Challenge 1: The Bank-to-Channel Mapping Problem
With 3 banks per channel, a single access returns 96B, so delivering a 128B flit takes two accesses and fetches 192B, wasting about 33 percent of bandwidth near 33 TB/s. Column staggering could pack flits but needs a 192B shifting buffer and complicates timing and verification.
d-Matrix Challenge 1: The Overfetch Dilemma
Stream blocking reclaims that waste. d-Matrix shares one partial 32B access across three flits, so 4 accesses at 96B feed 3 flits at 128B, with 384B in, matching 384B out. Overfetch drops to zero, every column access is used, and no shifting network is required.
d-Matrix Solution: Stream Blocking
Moving 100 TB/s at 0.37 pJ/bit works out to 296 W just for I/O, and conventional DBI could save 20 percent. HBM gets away with DBI because its multi-cycle bursts let the PHY see the full burst, but d-Matrix’s single-cycle 256-bit 3D-DRAM link has no burst and no sideband pin to signal the inversion choice.
d-Matrix Challenge 2: The I/O Power Wall
Stream flipping delivers that 20 percent without the pin. Each flit is compared to the previous one and inverted when needed, cutting toggles to near zero with a single metadata bit per flit carried alongside ECC. d-Matrix puts the overhead at 0.8 percent with no PHY change.
d-Matrix Solution: Stream Flipping Pinless DBI
Heat poses the third challenge at a 105C junction temperature. Yield matters because 840 banks mean even a 1 percent fault rate threatens whole channels, and discarding bonded dies is uneconomical. Disabling a faulty bank narrows its channel, and a single weak channel can throttle the entire slice, while higher temperatures drive more soft errors. Retention drops from 32 ms at 85C to 4 ms at 105C, demanding 8 times more refresh while ECC and scrub must keep up. This is why Hot Chips is so neat, because we are getting to this level of nerd detail.
d-Matrix Challenge 3: DRAM Reliability at 105C
To combat all of this d-Matrix interleaves ECC and DBI bits into the last columns of each subarray, pairing commodity ECC with a Reed-Solomon code. Reading 16 to 32 times fewer rows means the 8-times-more-frequent refresh costs only about 1.37 percent, holding bandwidth near 100 TB/s with under 1.4 percent loss.
d-Matrix Solution: Thermal-Aware Refresh & ECC
Bank chaining keeps channels symmetric when banks fail. Two physical mux levels let level one skip the first fault and level two skip the second, so the chiplet’s 72 spare banks absorb faults anywhere on the die. Spares backfill and channels stay uniform with negligible routing cost.
d-Matrix Solution: Bank Chaining
Now d-Matrix has what looks like a silicon-area comparison against HBM4 and NVIDIA Rubin R200. Raptor posts about 32.6 GB/s per mm2 compared with roughly 1.5 GB/s for the HBM parts, around 20 times the bandwidth per square millimeter, and 2.96 mW per GB/s against 40 mW, a 13.5x improvement. That is pretty neat.
d-Matrix Conclusions – 1
Raptor’s result claims roughly 1,000 tokens per second per user, serving a 3-trillion-parameter class model with 1M context. d-Matrix’s thesis is that moderate per-card memory paired with high bandwidth wins in low-latency inference. It would be neat to see this actually running. Hopefully we can do that one day.
d-Matrix Raptor sustains ~1000 TPS/User for serving 3T class model at 1M context
That is pretty neat assuming it works, but it would have been nice to hear a bit more on the drawbacks.
Final Words
Raptor is an architectural bet that bundles bandwidth and capacity into a single package for inference. d-Matrix’s comparison figures against HBM4 and Rubin R200 point to a real density and power-per-GB/s advantage. If 3D DRAM closes the memory bandwidth gap, it reshapes how accelerator vendors weigh capacity against bandwidth in AI systems. Whenever someone has a solution like this that seems somewhat adjacent to the current architecture, my first question is usually “why is everyone not using this?” You would have to imagine companies like NVIDIA have someone somewhere who would have thought about stacking DRAM on top of logic. This one is really interesting, and hopefully we can see it running soon.
We have a full recap on the Substack, including where all of these pieces ranked in terms of popularity:

Hot Chips 2026 Recap and Data by Patrick Kennedy
The data is in, and here is who created buzz at Hot Chips this year
Read on Substack

TAGSd-MatrixHot Chips 2026Raptor
FacebookXPinterestLinkedinReddItEmailPrintCopy URL

Previous articleSK hynix HBM Packaging at Hot Chips 2026Next articleIBM Z and LinuxONE Dual-ISA Processor and AI Acceleration at Hot Chips 2026 Patrick Kennedyhttps://www.servethehome.comPatrick has been running STH since 2009 and covers a wide variety of SME, SMB, and SOHO IT topics. Patrick is a consultant in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about server, storage and networking, building blocks. If you have any helpful information please feel free to post on the forums.

Facebook

LEAVE A REPLY

Please enter your comment!

Please enter your name here

You have entered an incorrect email address!
Please enter your email address here

Save my name, email, and website in this browser for the next time I comment.
Sign me up for the STH newsletter!

Δ

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Newsletter
Get the best of STH delivered weekly to your inbox. We are going to curate a selection of the best posts from STH each week and deliver them directly to you.

Your email address:

By opting-in you agree to have us send you our newsletter. We are using a third party service to manage subscriptions so you can unsubscribe at any time.
Leave this field empty if you're human:

ABOUT USServeTheHome is the IT professional's guide to servers, storage, networking, and high-end workstation hardware, plus great open source projects.

Advertise on STH

DISCLAIMERS: We are a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for us to earn fees by linking to Amazon.com and affiliated sites.
Please be advised of our data usage for machine learning and AI inference and training purposes which can be found here along with our other editorial and copyright policies.FOLLOW US

Facebook

RSS

TikTok

Youtube

The d-Matrix Raptor 3D-DRAM accelerator for generative inference addresses the growing problems of capacity and bandwidth inherent in Large Language Model (LLM) inference by integrating memory and compute in a three-dimensional structure. The necessity for this advancement stems from the escalating demands of model weights and the key-value (KV) cache, where the cache size scales with context length and batch size, creating simultaneous capacity and bandwidth constraints. While SRAM accelerators offer extremely high bandwidth with low latency, they suffer from limited capacity, suitable only for draft models, whereas High Bandwidth Memory (HBM) excels in capacity but faces significant bandwidth and power challenges. HBM systems, despite high bandwidth, encounter power density issues and physical limitations regarding stack count.

The authors analyze the trade-offs between SRAM and HBM, noting that HBM can achieve high bandwidth but incurs substantial power costs, and current packaging technologies lack the necessary density and power budget to reach SRAM-class bandwidth with HBM. d-Matrix proposes 3D DRAM as a balanced solution, sitting between the extremes of on-die SRAM and 2.5D HBM4 systems on an energy ladder. This approach results in lower energy consumption for vertical three-dimensional input/output, as it utilizes shorter, millimeter-scale paths rather than longer interposer routes.

The context of LLM inference workloads is crucial, as the decode phase, which produces one token at a time, is typically memory-bandwidth-bound, meaning improving decode bandwidth is paramount to overall inference performance. The research posits that most wall-clock inference time is spent in this decode phase. d-Matrix suggests that a system built around 3D-DRAM, such as a 1Hi configuration, can host frontier LLMs, potentially allowing a large model with substantial context to fit within a single rack, facilitating disaggregation across multiple racks.

The specific implementation utilizes the Raptor topology, which features a TSMC N4 logic die stacked on top of a 3D DRAM die using 36 um face-to-face stacking, a method the authors describe as proven and high-yield. Implementing this system exposes several complex integration challenges that are interconnected. These challenges include the bank-to-channel mapping problem, the I/O power wall, and thermal reliability. The authors emphasize that solving one constraint inevitably impacts the others. For instance, the bank mapping requires careful consideration of how data access is handled to maximize bandwidth, as standard methods lead to inefficiency in the utilization of available resources.

To mitigate inefficiencies, d-Matrix introduces specific techniques. To address the bank-to-channel mapping issue, they explore stream blocking, where partial data accesses are streamed across multiple flits, which allows for efficient data movement by eliminating unnecessary shifting network overhead. Furthermore, adapting the data flow to leverage overlapping operations, such as stream flipping, allows for data inversion without requiring complex physical sideband pins, reducing overhead with minimal changes to the PHY.

Thermal reliability and power management pose further hurdles. High temperatures reduce data retention time, necessitating more frequent refresh cycles, which impacts power consumption. To counter this, d-Matrix integrates error correction coding and data flow management. They propose thermal-aware refresh and error correction coding, interleaving ECC and data flow bits into the final columns of each subarray, combining commodity ECC with Reed-Solomon codes. Bank chaining is also introduced to maintain channel symmetry during faults, allowing spare banks to absorb errors and keep the system functional.

These integrated solutions yield significant performance gains. The comparison estimates that the Raptor system achieves approximately 32.6 GB/s per square millimeter, which is twenty times the bandwidth per square millimeter compared to HBM parts, and demonstrates a power efficiency of 2.96 mW per GB/s versus 40 mW in HBM. The overall conclusion is that d-Matrix's thesis is that moderate per-card memory paired with high bandwidth is superior for low-latency inference. The accelerator architecture represents an architectural bet that successfully bundles bandwidth and capacity, suggesting that if 3D DRAM effectively closes the memory bandwidth gap, it will fundamentally reshape the balance between capacity and bandwidth considerations for accelerator vendors in AI systems.