The shortage of advanced packaging production capacity is anticipated to end earlier than expected. Industry suggests that Samsung’s inclusion in providing HBM3 production capacity has led to an increased supply of memory essential for advanced packaging. Coupled with TSMC’s strategy of enhancing advanced packaging production capacity through equipment modifications and partial outsourcing, and the adjustments made by some CSP in designs and placing orders, the bottleneck in advanced packaging capacity is poised to open up as early as the first quarter of the upcoming year, surpassing industry predictions by one quarter to half a year, according to the UDN News.

TSMC refrains from commenting on market speculations, while Samsung has already issued a press release signaling the expansion of HBM3 product sales to meet the growing demand for the new interface, concurrently boosting the share of advanced processes.

Industry indicates that the previous global shortage of AI chips primarily resulted from inadequate advanced packaging capacity. Now the shortage in advanced packaging capacity is expected to end sooner, it implies a positive shift in the supply of AI chips.

Samsung, alongside Micron and SK Hynix, is a key partner for TSMC in advanced packaging. In a recent press release, Samsung underscores its close collaboration with TSMC in previous generations and the current high-bandwidth memory (HBM) technology, supporting the compatibility of the CoWoS process and the interconnectivity of HBM. Having joined the TSMC OIP 3DFabric Alliance in 2022, Samsung is set to broaden its scope of work and provide solutions for future generations of HBM.

Previously, the industry points out that the earlier shortage of AI chips stemmed from three main factors: insufficient advanced packaging capacity, tight HBM3 memory capacity, and some CSPs repeatedly placing orders. Now, the obstacles related to these factors are gradually being overcome. In addition to TSMC and Samsung’s commitment to increasing advanced packaging capacity, CSPs are adjusting designs, reducing the usage of advanced packaging, and canceling previous repeated orders – all of which are the key factors.

TSMC’s ongoing collaboration with OSATs(Outsourced Semiconductor Assembly And Test) to expedite WoS capacity expansion is gaining momentum. NVIDIA confirmed during a recent financial calls that it has certified other CoWoS advanced packaging suppliers’ capacity as a backup. Industry speculation suggests that certifying the capacity of other CoWoS suppliers for both part of the front-end and back-end production will contribute to TSMC and its partners achieving the target of reaching a monthly CoWoS capacity of approximately 40,000 pieces in the first quarter of the next year.

Furthermore, previous challenges in expanding advanced packaging production capacity, especially in obtaining overseas equipment, are gradually being overcome. With equipment optimization, more capacity is being extracted, alleviating the shortage of AI chip capacity.

(Image: Samsung)

Explore more

• • [News] Memory Titans Vie for Control in HBM Tech, Who Will Shape the Next-Gen?
  • [News] Samsung Boosts 2.5D Packaging Equipment to Compete for TSMC’s CoWoS Orders
  • [News] The Alliance among Apple, TSMC, and Amkor May Pose Challenges to Samsung’s Advanced Packaging Strategy

Market reports suggest Nvidia’s new product release cycle has shortened from two to a year, sparking intense competition among major memory companies in the realm of next-gen High Bandwidth Memory (HBM) technology. Samsung, SK Hynix, and Micron are fervently competing, with SK Hynix currently holding the dominant position in the HBM market. However, Micron and Samsung are strategically positioned, poised for a potential overtake, reported by TechNews.

Current Status of the HBM Industry

SK Hynix made a breakthrough in 2013 by successfully developing and mass-producing HBM using the Through Silicon Via (TSV) architecture. In 2019, they achieved success with HBM2E, maintaining the overwhelming advantage in the HBM market. According to the latest research from TrendForce, Nvidia plan to partner with more HBM suppliers. Samsung, as one of the suppliers, its HBM3 (24GB) is anticipated to complete verification with NVIDIA by December this year.

Regarding HBM3e progress, Micron, SK Hynix, and Samsung provided 8-layer (24GB) Nvidia samples in July, August, and October, respectively, with the fastest verification expected by year-end. All three major players anticipate completing verification in the first quarter of 2024.

As for HBM4, the earliest launch is expected in 2026, with a stack increase to 16 layers from the existing 12 layers. The memory stack will likely adopt a 2048-bit memory stack connection interface, driving demand for the new “Hybrid Bonding” stacking method. The 12-layer HBM4 product is set to launch in 2026, followed by the 16-layer product expected in 2027.

Navigating HBM4, the New Technologies and Roadmaps of Memory Industry Leaders

SK Hynix

According to reports from Business Korea, SK Hynix is preparing to adopt “2.5D Fan-Out” packaging for the next-generation HBM technology. This move aims to enhance performance and reduce packaging costs. This technology, not previously used in the memory industry but common in advanced semiconductor manufacturing, is seen as having the potential to “completely change the semiconductor and foundry industry.” SK Hynix plans to unveil research results using this packaging method as early as next year.

The 2.5D Fan-Out packaging technique involves arranging two DRAM horizontally and assembling them similar to regular chips. The absence of a substrate beneath the chips allows for thinner chips, significantly reducing the thickness when installed in IT equipment. Simultaneously, this technique bypasses the Through Silicon Via (TSV) process, providing more Input/Output (I/O) options and lowering costs. 

According to their previous plan, SK Hynix aims to mass-produce the sixth-generation HBM (HBM4) as early as 2026. The company is also actively researching “Hybrid Bonding” technology, likely to be applied to HBM4 products.

Currently, HBM stacks are placed on the interposer next to or GPUs and are connected to their interposer. While SK Hynix’s new goal is to eliminate the interposer completely, placing HBM4 directly on GPUs from companies like Nvidia and AMD, with TSMC as the preferred foundry.

Samsung

Samsung is researching the application of photonics in HBM technology’s interposer layer, aiming to address challenges related to heat and transistor density. Yan Li, Principal Engineer in Samsung’s advanced packaging team, shared insights at the OCP Global Summit in October 2023.

(Image: Samsung)

According to Samsung, The industry has made significant strides in integrating photonics with HBM through two main approaches. One involves placing a photonics interposer between the bottom packaging layer and the top layer containing GPU and HBM, acting as a communication layer. However, this method is costly, requiring an interposer and photon I/O for logic chips and HBM.

(Image: Samsung)

The alternative approach separates the HBM memory module from packaging, directly connecting it to the processor using photonics. Rather than dealing with the complexity of packaging, a more efficient approach is to separate the HBM memory module from the chip itself and connect it to the logic IC using photonics technology. This approach not only simplifies the manufacturing and packaging costs for HBM and logic ICs but also eliminates the need for internal digital-to-optical conversions in the circuitry. However, careful attention is required to address heat dissipation.

Micron

As reported by Tom’s Hardware, Micron’s 8-layer HBM3e (24GB) is expected to launch in early 2024, contributing to improved AI training and inference performance. The 12-layer HBM3e (36GB) chip is expected to debut in 2025.

Micron is working on HBM4 and HBM4e along with other companies. The required bandwidth is expected to exceed 1.5 TB/s. Micron anticipates launching 12-layer and 16-layer HBM4 with capacities of 36GB to 48GB between 2026 and 2027. After 2028, HBM4E will be introduced, pushing the maximum bandwidth beyond 2+ TB/s and increasing stack capacity to 48GB to 64GB.

Micron is taking a different approach from Samsung and SK Hynix by not integrating HBM and logic chips into a single die, suggested by Chinese media Semiconductor Industry Observation. This difference in strategy may lead to distinct technical paths, and Micron might advise Nvidia, Intel, AMD that relying solely on the same company’s chip carries greater risks.

(Image: Micron)

TSMC Aids Memory Stacking       

Currently, TSMC 3DFabric Alliance closely collaborates with major memory partners, including Micron, Samsung, and SK Hynix. This collaboration ensures the rapid growth of HBM3 and HBM3e, as well as the packaging of 12-layer HBM3/HBM3e, by providing more memory capacity to promote the development of generative AI.

(Image: TSMC)

(Image: SK Hynix)

Explore more

• Manufacturers Anticipate Completion of NVIDIA’s HBM3e Verification by 1Q24; HBM4 Expected to Launch in 2026, Says TrendForce
• [News] SK Hynix Signs Prioritized Supply Agreement with NVIDIA for HBM3e, 2023 Q4 Revenue Expected to Surpass KRW 10 Trillion Again
• [News] Memory Giants Samsung & Micron Actively Embrace DDR5 and HBM as Favorable Choices
• [News] Seizing the AI Trend! Revealing Samsung and Micron’s HBM Expansion Timetable
• Continuous Rise in HBM Demand, Memory Giants Expecting HBM4 Delivery in 2025

On November 13, NVIDIA unveiled the AI computing platform HGX H200, featuring the Hopper architecture, equipped with H200 Tensor Core GPU and high-end memory to handle the vast amounts of data generated by AI and high-performance computing.

This marks an upgrade from the previous generation H100, with a 1.4x increase in memory bandwidth and a 1.8x increase in capacity, enhancing its capabilities for processing intensive generative AI tasks.

The internal memory changes in H200 represent a significant upgrade, as it adopts the HBM3e for the first time. This results in a notable increase in GPU memory bandwidth, soaring from 3.35TB per second in H100 to 4.8TB per second.

The total memory capacity also sees a substantial boost, rising from 80GB in H100 to 141GB. When compared to H100, these enhancements nearly double the inference speed for the Llama 2 model.

H200 is designed to be compatible with systems that already support H100, according to NVIDIA. The company states that cloud service providers can seamlessly integrate H200 into their product portfolios without the need for any modifications.

This implies that NVIDIA’s server manufacturing partners, including ASRock, ASUS, Dell, Eviden, GIGABYTE, HPE, Ingrasys, Lenovo, Quanta Cloud, Supermicro, Wistron, and Wiwynn, have the flexibility to replace existing processors with H200.

The initial shipments of H200 are expected in the second quarter of 2024, with cloud service giants such as Amazon, Google, Microsoft, and Oracle anticipated to be among the first to adopt H200.

What is HBM?

“The integration of faster and more extensive HBM memory serves to accelerate performance across computationally demanding tasks including generative AI models and [high-performance computing] applications while optimizing GPU utilization and efficiency,” said Ian Buck, the Vice President of High-Performance Computing Products at NVIDIA.

What is HBM? HBM refers to stacking DRAM layers like building blocks and encapsulating them through advanced packaging. This approach increases density while maintaining or even reducing the overall volume, leading to improved storage efficiency.

TrendForce reported that the HBM market’s dominant product for 2023 is HBM2e, employed by the NVIDIA A100/A800, AMD MI200, and most CSPs’ (Cloud Service Providers) self-developed accelerator chips.

As the demand for AI accelerator chips evolves, in 2023, the mainstream demand is projected to shift from HBM2e to HBM3, with estimated proportions of approximately 50% and 39%, respectively.

As the production of acceleration chips utilizing HBM3 increases gradually, the market demand in 2024 is expected to significantly transition to HBM3, surpassing HBM2e directly. The estimated proportion for 2024 is around 60%.

Since Manufacturers plan to introduce new HBM3e products in 2024, HBM3 and HBM3e are expected to become mainstream in the market next year.

TrendForce clarifies that the so-called HBM3 in the current market should be subdivided into two categories based on speed. One category includes HBM3 running at speeds between 5.6 to 6.4 Gbps, while the other features the 8 Gbps HBM3e, which also goes by several names including HBM3P, HBM3A, HBM3+, and HBM3 Gen2.

HBM3e will be stacked with 24Gb mono dies, and under the 8-layer (8Hi) foundation, the capacity of a single HBM3e will jump to 24GB.

According to the TrendForce’s previous news release, the three major manufacturers currently leading the HBM competition – SK hynix, Samsung, and Micron – have the following progress updates.

SK hynix and Samsung began their efforts with HBM3, which is used in NVIDIA’s H100/H800 and AMD’s MI300 series products. These two manufacturers are expected to sample HBM3e in Q1 2024 previously. Meanwhile, Micron chose to skip HBM3 and directly develop HBM3e.

However, according to the latest TrendForce survey, as of the end of July this year, Micron has already provided NVIDIA with HBM3e verification, while SK hynix did so in mid-August, and Samsung in early October.

(Image: Nvidia)

Amidst the AI boom, HBM technology steps into the spotlight as market demand continues to surge. Global market research firm TrendForce anticipates a 58% year-on-year increase in HBM demand in 2023, with a potential additional growth of approximately 30% in 2024.

Compared to traditional DRAM, HBM (High Bandwidth Memory) boasts advantages such as high bandwidth, high capacity, low latency, and low power consumption. These attributes accelerate AI data processing and make it particularly well-suited for high-performance computing scenarios like ChatGPT. As a result, it has gained popularity, and major storage manufacturers are actively driving HBM technology upgrades.

Leading memory manufacturers are intensifying their efforts, with Samsung set to introduce HBM4.

Since the inception of the first HBM products utilizing TSV packaging technology in 2014, HBM technology has seen multiple upgrades, including HBM, HBM2, HBM2E, HBM3, and HBM3e.

Regarding the SK Hynix and Samsung, two major South Korean companies, have been at the forefront of HBM3 development. NVIDIA’s H100/H800 and AMD’s MI300 series, represent HBM3’s progress. Both SK Hynix and Samsung expected to offer HBM3e samples by the first quarter of 2024. On the other hand, Micron, a U.S.-based memory company, is bypassing HBM3 and directly pursuing HBM3e.

HBM3e will feature 24Gb mono die stacks, and with an 8-layer (8Hi) configuration, a single HBM3e chip’s capacity will soar to 24GB. This advancement is expected to be incorporated into NVIDIA’s GB100 in 2025, leading the three major OEMs to plan HBM3e sample releases in the first quarter of 2024 and enter mass production in the latter half of the year.

In addition to HBM3 and HBM3e, the latest updates indicate that storage giants are planning the launch of the next generation of HBM—HBM4.

Samsung recently announced that it has developed 9.8Gbps HBM3E and is planning to provide samples to customers. Furthermore, Samsung is actively working on HBM4 with a goal to begin supply in 2025. It’s reported that Samsung Electronics is developing technologies such as non-conductive adhesive film (NCF) assembly for optimizing high-temperature thermal characteristics, as well as hybrid bonding (HCB), for HBM4 products.

In September, Korean media reported that Samsung is gearing up to revamp its production process and launch HBM4 products to capture the rapidly growing HBM market. HBM4 memory stacks will feature a 2048-bit memory interface, a significant departure from the previous 1024-bit interface for all HBM stacks. This enhanced interface width holds great significance for the evolution of HBM4.

While HBM4 promises a major breakthrough, it is still a ways off, making it too early to discuss its practical applications and widespread adoption. Industry experts emphasize that the current HBM market is dominated by HBM2e. However, HBM3 and HBM3e are poised to take the lead in the near future.

According to TrendForce’s research, HBM2e currently accounts for the mainstream market share, being used in various products like NVIDIA A100/A800, AMD MI200, and many AI accelerators developed by CSPs. To keep pace with the evolving demands of AI accelerator chips, OEMs are planning to introduce new HBM3e products in 2024, with HBM3 and HBM3e expected to become the market’s primary players next year.

In terms of the demand transition between different HBM generations, TrendForce estimates that in 2023, mainstream demand will shift from HBM2e to HBM3, with estimated demand shares of approximately 50% and 39%, respectively. As more HBM3-based accelerator chips enter the market, demand will substantially shift toward HBM3 in 2024, surpassing HBM2e and accounting for an estimated 60% of the market. This transition, coupled with higher average selling prices (ASP), is poised to significantly drive HBM revenue growth next year.

(Photo credit: Samsung)