As the United States intensifies its chip embargo against China, the Chinese Academy of Engineering (CAE) has released an annual report for technological development. This report serves as a strategic guide to navigate the embargo and promote autonomous technological growth comprehensively.
…

As reported by China’s Jiwei on October 2nd, Samsung has revealed that its chip manufacturing division has secured an order from AI chip client Tenstorrent to produce chips utilizing its cutting-edge 4nm process.

…

Looking at the impact of AI server development on the PCB industry, mainstream AI servers, compared to general servers, incorporate 4 to 8 GPUs. Due to the need for high-frequency and high-speed data transmission, the number of PCB layers increases, and there’s an upgrade in the adoption of CCL grade as well. This surge in GPU integration drives the AI server PCB output value to surpass that of general servers by several times. However, this advancement also brings about higher technological barriers, presenting an opportunity for high-tech PCB manufacturers to benefit.

TrendForce’s perspective: 

• The increased value of AI server PCBs primarily comes from GPU boards.

Taking the NVIDIA DGX A100 as an example, its PCB can be divided into CPU boards, GPU boards, and accessory boards. The overall value of the PCB is about 5 to 6 times higher than that of a general server, with approximately 94% of the incremental value attributed to the GPU boards. This is mainly due to the fact that general servers typically do not include GPUs, while the NVIDIA DGX A100 is equipped with 8 GPUs.

Further analysis reveals that CPU boards, which consist of CPU boards, CPU mainboards, and functional accessory boards, make up about 20% of the overall AI server PCB value. On the other hand, GPU boards, including GPU boards, NV Switch, OAM (OCP Accelerator Module), and UBB (Unit Baseboard), account for around 79% of the total AI server PCB value. Accessory boards, composed of components such as power supplies, HDD, and cooling systems, contribute to only about 1% of the overall AI server PCB value.

• The technological barriers of AI servers are rising, leading to a decrease in the number of suppliers.

Since AI servers require multiple card interconnections with more extensive and denser wiring compared to general servers, and AI GPUs have more pins and an increased number of memory chips, GPU board assemblies may reach 20 layers or more. With the increase in the number of layers, the yield rate decreases.

Additionally, due to the demand for high-frequency and high-speed transmission, CCL materials have evolved from Low Loss grade to Ultra Low Loss grade. As the technological barriers rise, the number of manufacturers capable of entering the AI server supply chain also decreases.

Currently, the suppliers for CPU boards in AI servers include Ibiden, AT&S, Shinko, and Unimicron, while the mainboard PCB suppliers consist of GCE and Tripod. For GPU boards, Ibiden serves as the supplier, and for OAM PCBs, Unimicron and Zhending are the suppliers, with GCE, ACCL, and Tripod currently undergoing certification. The CCL suppliers include EMC. For UBB PCBs, the suppliers are GCE, WUS, and ACCL, with TUC and Panasonic being the CCL suppliers.

Regarding ABF boards, Taiwanese manufacturers have not yet obtained orders for NVIDIA AI GPUs. The main reason for this is the limited production volume of NVIDIA AI GPUs, with an estimated output of only about 1.5 million units in 2023. Additionally, Ibiden’s yield rate for ABF boards with 16 layers or more is approximately 10% to 20% higher than that of Taiwanese manufacturers. However, with TSMC’s continuous expansion of CoWoS capacity, it is expected that the production volume of NVIDIA AI GPUs will reach over 2.7 million units in 2024, and Taiwanese ABF board manufacturers are likely to gain a low single-digit percentage market share.

(Photo credit: Google)

AI Chips and High-Performance Computing (HPC) have been continuously shaking up the entire supply chain, with CoWoS packaging technology being the latest area to experience the tremors.

In the previous piece, “HBM and 2.5D Packaging: the Essential Backbone Behind AI Server,” we discovered that the leading AI chip players, Nvidia and AMD, have been dedicated users of TSMC’s CoWoS technology. Much of the groundbreaking tech used in their flagship product series – such as Nvidia’s A100 and H100, and AMD’s Instinct MI250X and MI300 – have their roots in TSMC’s CoWoS tech.

However, with AI’s exponential growth, chip demand from not just Nvidia and AMD has skyrocketed, but other giants like Google and Amazon are also catching up in the AI field, bringing an onslaught of chip demand. The surge of orders is already testing the limits of TSMC’s CoWoS capacity. While TSMC is planning to increase its production in the latter half of 2023, there’s a snag – the lead time of the packaging equipment is proving to be a bottleneck, severely curtailing the pace of this necessary capacity expansion.

Nvidia Shakes the foundation of the CoWoS Supply Chain

In these times of booming demand, maintaining a stable supply is viewed as the primary goal for chipmakers, including Nvidia. While TSMC is struggling to keep up with customer needs, other chipmakers are starting to tweak their outsourcing strategies, moving towards a more diversified supply chain model. This shift is now opening opportunities for other foundries and OSATs.

Interestingly, in this reshuffling of the supply chain, UMC (United Microelectronics Corporation) is reportedly becoming one of Nvidia’s key partners in the interposer sector for the first time, with plans for capacity expansion on the horizon.

From a technical viewpoint, interposer has always been the cornerstone of TSMC’s CoWoS process and technology progression. As the interposer area enlarges, it allows for more memory stack particles and core components to be integrated. This is crucial for increasingly complex multi-chip designs, underscoring Nvidia’s intention to support UMC as a backup resource to safeguard supply continuity.

Meanwhile, as Nvidia secures production capacity, it is observed that the two leading OSAT companies, Amkor and SPIL (as part of ASE), are establishing themselves in the Chip-on-Wafer (CoW) and Wafer-on-Substrate (WoS) processes.

The ASE Group is no stranger to the 2.5D packaging arena. It unveiled its proprietary 2.5D packaging tech as early as 2017, a technology capable of integrating core computational elements and High Bandwidth Memory (HBM) onto the silicon interposer. This approach was once utilized in AMD’s MI200 series server GPU. Also under the ASE Group umbrella, SPIL boasts unique Fan-Out Embedded Bridge (FO-EB) technology. Bypassing silicon interposers, the platform leverages silicon bridges and redistribution layers (RDL) for integration, which provides ASE another competitive edge.

Could Samsung’s Turnkey Service Break New Ground?

In the shifting landscape of the supply chain, the Samsung Device Solutions division’s turnkey service, spanning from foundry operations to Advanced Package (AVP), stands out as an emerging player that can’t be ignored.

After its 2018 split, Samsung Foundry started taking orders beyond System LSI for business stability. In 2023, the AVP department, initially serving Samsung’s memory and foundry businesses, has also expanded its reach to external clients.

Our research indicates that Samsung’s AVP division is making aggressive strides into the AI field. Currently in active talks with key customers in the U.S. and China, Samsung is positioning its foundry-to-packaging turnkey solutions and standalone advanced packaging processes as viable, mature options.

In terms of technology roadmap, Samsung has invested significantly in 2.5D packaging R&D. Mirroring TSMC, the company launched two 2.5D packaging technologies in 2021: the I-Cube4, capable of integrating four HBM stacks and one core component onto a silicon interposer, and the H-Cube, designed to extend packaging area by integrating HDI PCB beneath the ABF substrate, primarily for designs incorporating six or more HBM stack particles.

Besides, recognizing Japan’s dominance in packaging materials and technologies, Samsung recently launched a R&D center there to swiftly upscale its AVP business.

Given all these circumstances, it seems to be only a matter of time before Samsung carves out its own significant share in the AI chip market. Despite TSMC’s industry dominance and pivotal role in AI chip advancements, the rising demand for advanced packaging is set to undeniably reshape supply chain dynamics and the future of the semiconductor industry.

(Source: Nvidia)

OpenAI’s ChapGPT, Microsoft’s Copilot, Google’s Bard, and latest Elon Musk’s TruthGPT – what will be the next buzzword for AI? In just under six months, the AI competition has heated up, stirring up ripples in the once-calm AI server market, as AI-generated content (AIGC) models take center stage.

The convenience unprecedentedly brought by AIGC has attracted a massive number of users, with OpenAI’s mainstream model, GPT-3, receiving up to 25 million daily visits, often resulting in server overload and disconnection issues.

Given the evolution of these models has led to an increase in training parameters and data volume, making computational power even more scarce, OpenAI has reluctantly adopted measures such as paid access and traffic restriction to stabilize the server load.

High-end Cloud Computing is gaining momentum

According to Trendforce, AI servers currently have a merely 1% penetration rate in global data centers, which is far from sufficient to cope with the surge in data demand from the usage side. Therefore, besides optimizing software to reduce computational load, increasing the number of high-end AI servers in hardware will be another crucial solution.

Take GPT-3 for instance. The model requires at least 4,750 AI servers with 8 GPUs for each, and every similarly large language model like ChatGPT will need 3,125 to 5,000 units. Considering ChapGPT and Microsoft’s other applications as a whole, the need for AI servers is estimated to reach some 25,000 units in order to meet the basic computing power.

As the emerging applications of AIGC and its vast commercial potential have both revealed the technical roadmap moving forward, it also shed light on the bottlenecks in the supply chain.

The down-to-earth problem: cost

Compared to general-purpose servers that use CPUs as their main computational power, AI servers heavily rely on GPUs, and DGX A100 and H100, with computational performance up to 5 PetaFLOPS, serve as primary AI server computing power. Given that GPU costs account for over 70% of server costs, the increase in the adoption of high-end GPUs has made the architecture more expansive.

Moreover, a significant amount of data transmission occurs during the operation, which drives up the demand for DDR5 and High Bandwidth Memory (HBM). The high power consumption generated during operation also promotes the upgrade of components such as PCBs and cooling systems, which further raises the overall cost.

Not to mention the technical hurdles posed by the complex design architecture – for example, a new approach for heterogeneous computing architecture is urgently required to enhance the overall computing efficiency.

The high cost and complexity of AI servers has inevitably limited their development to only large manufacturers. Two leading companies, HPE and Dell, have taken different strategies to enter the market:

• HPE has continuously strengthened its cooperation with Google and plans to convert all products to service form by 2022. It also acquired startup Pachyderm in January 2023 to launch cloud-based supercomputing services, making it easier to train and develop large models.
• In March 2023, Dell launched its latest PowerEdge series servers, which offers options equipped with NVIDIA H100 or A100 Tensor Core GPUs and NVIDIA AI Enterprise. They use the 4th generation Intel Xeon Scalable processor and introduce Dell software Smart Flow, catering to different demands such as data centers, large public clouds, AI, and edge computing.

With the booming market for AIGC applications, we seem to be one step closer to a future metaverse centered around fully virtualized content. However, it remains unclear whether the hardware infrastructure can keep up with the surge in demand. This persistent challenge will continue to test the capabilities of cloud server manufacturers to balance cost and performance.

(Photo credit: Google)