Korea's AI Firms Slim Down Models, Put Idle GPUs to Work
Korean AI firms are reportedly shifting their focus from raw model intelligence to economic efficiency, driven by rising inference costs as generative AI evolves into autonomous agents and physical AI. They are adopting "lightweighting" techniques like pruning, quantization, and graph optimization to reduce memory and computation, while also seeking to improve GPU utilization rates, which currently hover around 30%.
Korean artificial intelligence firms are reportedly shifting the basis of competition in the AI market from how smart a model is to how economically smart it is. This change is driven by rising inference costs as generative AI, once focused on text and questions, expands into AI agents that perform tasks autonomously and physical AI that operates in the real world. Cost-efficiency technology is emerging as a key factor for corporate profitability.
A leading method to make AI models more efficient is "lightweighting," which reduces the memory a model occupies and the volume of computation it requires, allowing it to run on fewer computing resources. Kim Hyung-jun, CEO of SqueezeBits, explained that inference occurs each time a user accesses a service, causing computing resources and costs to rise with user numbers. For services with large models or heavy computation, such as image and video generation, efficient model operation directly impacts service costs and speed.
This focus on lightweighting also reflects a changing landscape in AI computing. According to global accounting and consulting firm Deloitte, inference accounted for one-third of all AI computing three years ago but is expected to reach two-thirds this year, surpassing training. This indicates increasing demand for using AI in actual work rather than primarily developing it.
Widely used lightweighting techniques include pruning, which cuts less important connections in a model's neural networks, and quantization, which compresses data from 32-bit to 8-bit or 4-bit units. Another technique, graph optimization, reduces how often data moves to high-bandwidth memory during computation and rearranges operations for faster processing. Yoo Seung-joo, a professor at Seoul National University, noted that graph optimization does not degrade computational quality, while quantization and pruning can affect quality to some degree. He emphasized the importance of staying within a reasonable line so that any difference is not noticeable.
Shrinking a model does not automatically make it faster or cheaper; the hardware running the model, such as graphics processing units (GPUs), must efficiently handle the altered computation. Kim Hyung-jun stated that optimal settings can vary even with the same GPU when the product or generation changes, as hardware handles different operations and data formats uniquely. GPU efficiency is considered important for reducing AI inference costs, especially since utilization rates are currently reported to be stuck around 30%.
Nvidia's workload-splitting tools, Multi-Instance GPU (MIG) and Multi-Process Service (MPS), are drawing attention for their potential to optimize hardware performance. However, the report also specifies that for core tasks, accuracy must be verified when these efficiency technologies are applied.
Share this article
Related reading
6 stories
Daigou: the gray market that moves Nvidia’s chips to China
The gray market for Nvidia chips in China highlights the hardware challenges AI firms face globally.

NETS Explores a Way for AI Agents to Pay at Singapore Shops

OCBC Aims to Train 20,000 Small Business Owners and Staff for Free

How a little-known Beijing AI safety group became an adviser to China’s tech giants

Nium to Handle Real-Time Overseas Transfers for Woori Bank

