Kenji Baheux
When we build features with AI models on the web, we often rely on server-sidesolutions for larger models. This is especially true for generative AI, whereeven the smallest models are about thousand times bigger than themedian web page size.It's also true for other AI use cases, where models can range from 10s to 100sof megabytes.
Since these modelsaren't shared across websites,each site has to download them on page load. This is an impractical solution fordevelopers and users
While server-side AI is a great option for large models, on-device and hybridapproaches have their own compelling upsides. To make theseapproaches viable, we need to address model size and model delivery.
That's why we're developing web platform APIs and browser features designed tointegrate AI models, including large language models (LLMs), directly into thebrowser. This includesGemini Nano,the most efficient version of the Gemini family of LLMs, designed to run locallyon most modern desktop and laptop computers. With built-in AI, your website orweb application can perform AI-powered tasks without needing to deploy or manageits own AI models.
Discover the benefits of built-in AI, our implementation plan, and how you cantake advantage of this technology.
Get an early preview
We need your input to shape the APIs,ensure they fulfill your use cases, and inform our discussions with other browservendors for standardization.
Join our early preview program to providefeedback on early-stage built-in AI ideas, and discover opportunities to testin-progress APIs through local prototyping.
Join the Chrome AI developer public announcements groupto be notified when new APIs become available.
Benefits of built-in AI for web developers
With built-in AI, your browser provides and managesfoundation and expert models.
As compared to do it yourself on-device AI, built-in AI offers thefollowing benefits:
- Ease of deployment: As the browser distributes the models, it takes intoaccount the capability of the device and manages updates to the model. Thismeans you aren't responsible for downloading or updating large models over anetwork. You don't have to solve for storage eviction, runtime memory budget,serving costs, and other challenges.
- Access to hardware acceleration: The browser's AI runtime is optimized tomake the most out of the available hardware, be it a GPU, an NPU, or fallingback to the CPU. Consequently, your app can get the best performance on eachdevice.
Benefits of running on-device
With a built-in AI approach, it becomes trivial to perform AI tasks on-device,which in turn offers the following upsides:
- Local processing of sensitive data: On-device AI can improve yourprivacy story. For example, if you work with sensitive data, you can offer AIfeatures to users with end-to-end encryption.
- Snappy user experience: In some cases, ditching the round trip to theserver means you can offer near-instant results. On-device AI can be thedifference between a viable feature and a sub-optimal user experience.
- Greater access to AI: Your users' devices can shoulder some of theprocessing load in exchange for more access to features. For example, if youoffer premium AI features, you could preview these features with on-device AIso that potential customers can see the benefits of your product, withoutadditional cost to you. This hybrid approach can also help you manageinference costs especially on frequently used user flows.
- Offline AI usage: Your users can access AI features even when there is nointernet connection. This means your sites and web apps can work as expectedoffline or with variable connectivity.
Hybrid AI: On-device and server-side
While on-device AI can handle a large array of use cases, there are certain usecases which require server-side support.
For example, you may need to use larger models or support a wider range ofplatforms and devices.
You may consider hybrid approaches, dependent on:
- Complexity: Specific, approachable use cases are easier to support withon-device AI. For complex use cases, consider server-side implementation.
- Resiliency: Use server-side by default, and use on-device when the deviceis offline or on a spotty connection.
- Graceful fallback: Adoption of browsers with built-in AI will take time,some models may be unavailable, and older or less powerful devices may notmeet the hardware requirements for running all models optimally. Offerserver-side AI for those users.
For Gemini models, you can use backend integration (withPython,Go,Node.js, orREST) or implement inyour web application with the newGoogle AI client SDK for Web.
Browser architecture and APIs
To support built-in AI in Chrome, we created infrastructure to access foundationand expert models for on-device execution. This infrastructure is alreadypowering innovative browser features, such asHelp me write,and will soon power APIs for on-device AI.
You'll access built-in AI capabilities primarily with task APIs, such as atranslation API or a summarizationAPI. Task APIs are designed to run inference against the best model for theassignment.
In Chrome, these APIs are built to run inference against Gemini Nano withfine-tuning or an expert model. Designed to run locally on most modern devices,Gemini Nano is best for language-related use cases, such as summarization,rephrasing, or categorization.
Also, we intend to provide exploratory APIs, so that you can experiment locallyand share additional use cases.
For example, we may provide:
- Prompt API: Send an arbitrary task, expressed in natural language, to thebuilt-in Large Language Model (Gemini Nano in Chrome).
- Fine-tuning (LoRA) API: Improve the built-in LLM's performance on a taskby adjusting the model's weights withLow-Rank Adaptationfine tuning.
When to use built-in AI
Here are a few ways we expect built-in AI can benefit you and your users:
- AI-enhanced content consumption: Including summarization,translation,answering questions about some content, categorization, and characterizing.
- AI-supported content creation: Such as writing assistance, proofreading,grammar correction, and rephrasing.
What's next
Join our early preview program toexperiment with early-stage built-in AI APIs.
In the meantime, you can learn how to use Gemini Pro on Google's servers withyour websites and web apps in ourquickstart for the Google AI JavaScript SDK.
