When it comes to model deployment, efficiency trumps sheer parameter size.

What? Large models are finally slimming down?

This isn't just idle chatter. According to CNBC, Apple is in talks with PrismML, a startup renowned for its cutting-edge model compression technology, to explore the feasibility of running large-scale AI models directly on iPhones.

(Image Source: CNBC)

You know, in recent years, whenever AI is mentioned during smartphone launches, I instinctively reach for my water cup.

It's not that I have a bone to pick with manufacturers; it's just that the routine has become all too predictable. First, AI summarizes on-screen content, then it employs various image editing tools for personalized color adjustments or to remove unwanted bystanders from photos. This year, a new feature has been added: a voice assistant to help you order coffee.

But this isn't entirely the fault of smartphone makers. Current mainstream large models simply can't fit into smartphones, and the pruned, on-device versions aren't quite smart enough. As a result, the only viable option is cloud-based updates. Take Doubao, for example—it rolled out an AI podcast feature, and within three months, all major manufacturers followed suit.

So, here's the question: If a full-fledged large model can be slimmed down to fit into a smartphone, can on-device AI assistants truly go mainstream?

Let's delve into this with Leitech (ID: leitech):

Who is PrismML?

According to its official website, PrismML is a startup specializing in model compression. It originated from a research team at the California Institute of Technology and is backed by investors such as Khosla Ventures, Cerberus, and Google. Its research focuses on reducing model size and operational costs without significantly compromising model intelligence.

(Image Source: PrismML)

What have they achieved?

PrismML's approach shares similarities with low-bit models like BitNet. It drastically reduces model size by simplifying how AI models store internal information, limiting each weight in the model to binary or ternary expressions. This significantly cuts down on the memory required for storing and running the model.

Specifically, a parameter in a traditional large model typically requires 16 or even 32 bits to store.

(Image Source: HuggingFace)

For instance, a 27 billion parameter model using FP16 precision would require 27 billion × 2 Byte ≈ 54GB—roughly the size of Qwen3.6-27B in FP16.

Forget smartphones; many consumer-grade PCs would struggle to run it smoothly.

Under PrismML's approach, parameters in the 1-bit version are simplified to {-1, +1}. Just as each pixel in a photo used to save 16 levels of grayscale but now only needs to save black and white, although there is significant information loss, the volume can be compressed to 1/14 of the original, and the reasoning performance can be restored through training.

(Image Source: PrismML)

Based on this technology, they officially launched the Bonsai-27B model on July 15th. Fine-tuned from the Qwen3.6-27B model, it reduces the model from approximately 54GB to less than 4GB while retaining full context, enabling it to run natively on an iPhone with 12GB of memory.

To put this in perspective, Google's Gemma 4 E4B, designed for smartphones and edge devices, is about 3.65GB. PrismML has managed to fit a nominally 27 billion parameter dense model into a similar 'footprint.'

We won't delve into the user experience just yet, but hardware manufacturers are surely intrigued.

(Image Source: PrismML)

So, it's no surprise that Apple is interested in this technology.

You see, Apple's own on-device model has around 3 billion parameters and uses techniques like 2-bit quantization and cache sharing. Yet, it can only handle real-time translation, photo album search, and email summarization on the phone, lacking any Agent-related execution capabilities.

In contrast, the Bonsai-27B model retains some of the Agent capabilities of Qwen3.6-27B.

Of course, there is still some performance loss. In PrismML's own tests, the ternary version retains about 95% of the comprehensive performance of the full-precision model, while the 1-bit version retains about 90%. For tasks like tool invocation, which are highly valued by Agents, the performance drop is more noticeable.

Community tests have also reported that the ternary version of PrismML still suffers from issues like hallucinations and Agent looping compared to Q4_K_XL. However, its advantage lies in its extremely small size, essentially achieving performance comparable to a 17.9GB model with just 5.9GB.

(Image Source: Reddit)

But regardless, being usable is better than not being usable at all.

From physically not fitting to whether the actual experience is acceptable, if we continue to make progress in this direction, I believe there will be intense competition ahead.

Interestingly, on July 15th, seven model services providing on-device generative AI for smartphones, including Apple Intelligence, Huawei Xiaoyi, OPPO, Xiaomi, and vivo, have all completed registration with the cyberspace administration.

You have to admit, this list looks quite lively, indicating that smartphone manufacturers are seriously arranging on-device AI this year.

(Image Source: Cyberspace Administration)

The reason is not hard to understand. Tasks like notification summarization, call organization, photo album search, and image recognition don't necessarily need to be processed in the cloud every time.

Especially for private information like chat records, photos, and files, it's naturally best to handle them on your own phone. Considering the recent privacy scandal involving Grok, I can fully understand why people don't want their information to be passed around on the internet.

(Image Source: Leitech)

The problem is, from my personal experience, current smartphone AI functions still rely heavily on the cloud, and most functions become unusable without an internet connection.

Why does this happen? What is the current state of on-device AI in smartphones?

Coincidentally, I recently tried out Gemma 4 E4B in the Google AI Edge Gallery and can share my experience with you.

(Image Source: Google)

First, note that Gemma 4 E4B is already a highly capable model among on-device smartphone models, capable of processing text, images, and audio. Once downloaded, it can still operate offline.

For example, Ask Image enables multimodal input, which many on-device smartphone AIs have struggled with in the past.

In practice, Gemma 4 demonstrates strong image recognition capabilities. Although it's still not very familiar with anime characters, it captures features in the image quite well and can recognize common foods, hardware, and flowers.

(Image Source: Leitech)

Then there's Ask Audio, which can upload up to 30 seconds of audio for transcription and summarization.

This function is less impressive, possibly due to the blurry nature of my recordings. The transcribed content bears almost no relation to the original audio, making its current usability quite limited. It's still better to rely on Doubao or Qianwen for summarization.

(Image Source: Leitech)

As for text processing...

I fed a 2,500-word article to several models deployable on smartphones, hoping they could provide a summary.

Ultimately, only Gemma 3n E4B and Gemma 4 E4B completed the task, but the former took nearly two minutes and produced an answer that missed the main points. The latter provided a more concise summary, capturing the main information points quite well, making it sufficient for quickly scanning materials.

Even some logic problems that couldn't be solved in the past can now be tackled by Gemma 4 E4B after prolonged thinking, although the thinking time far exceeds the response time of online large models.

In Leitech's view, Gemma 4 E4B has proven that smartphone local models can indeed get the job done.

However, I'm only willing to use it for summarization, rewriting, and simple image recognition. For slightly more complex tasks, especially those involving long Chinese texts, detailed judgments, and content creation, the gap between it and online large models remains significant, not to mention tasks like Agent invocation.

Remember, in terms of compression rate and functionality, Gemma 4 is currently the strongest on-device AI for smartphones.

To surpass this level, Apple can only abandon its current compression methods and try to fit models with larger parameter counts into the same space, which might give smartphones a less error-prone 'brain.'

In the past, when discussing large models, everyone assumed that more parameters meant more prestige.

Hundreds of billions were just the starting point, and trillions were not too many. Announcing parameter counts at product launches was like weighing produce at a market, with the emphasis on 'my radishes are big and strong.'

This seems fine, but it's a different story when it comes to actual operation.

According to research on the Chinchilla scaling law by Jordan Hoffmann's team, with the same training computational power, a 70B model with more sufficient training data can comprehensively outperform under-trained 280B and 530B models. Even for MoE models with trillion-level parameters, VRAM usage (video memory usage) and memory bandwidth (memory bandwidth) remain significant issues.

(Image Source: arXiv)

More importantly, models with such massive parameters naturally cannot enter consumer-grade devices.

For on-device models to become a permanent fixture in smartphones, efficiency must be improved, directly addressing issues of memory, power consumption, and heat dissipation. PrismML's compression algorithm has pointed a way forward for hardware manufacturers. Combined with localized collaborations between Qianwen, Baidu, and Apple, it brings Apple's promise of 'Apple Intelligence' from three years ago one step closer to reality.

In the future, progress in large models may not necessarily rely on intimidating parameter counts.

Being able to work stably in limited storage space, making fewer mistakes, generating less heat, and truly being helpful when it matters—this is what on-device AI should strive for in its next phase.

AI, PrismML, Apple, large models, AI-enabled smartphones

Source: Leitech

Images in this article come from: 123RF Royalty-Free Image Library, Source: Leitech