Deploying Gemma3 with Ollama on Google Cloud Run: A Step-by-Step Guide
# Deploying Gemma3 with Ollama on Google Cloud Run: A Step-by-Step Guide
*Published: April 30, 2025*
## Introduction
In the rapidly evolving landscape of AI and machine learning, having access to powerful language models is becoming increasingly important for developers and businesses alike. Google's Gemma3 represents a significant advancement in open-source language models, offering impressive capabilities while being more accessible than many alternatives.
In this blog post, I'll walk you through the process of deploying Gemma3 using Ollama on Google Cloud Run with GPU acceleration. This setup provides a scalable, cost-effective way to run inference with this powerful model without managing complex infrastructure.
## Why Gemma3 and Ollama on Cloud Run?
Before diving into the technical details, let's understand why this combination is particularly compelling:
1. **Gemma3** is Google's latest open-source language model, offering strong performance while being more efficient than many alternatives.
2. **Ollama** provides a streamlined way to run language models locally or in containers, with a simple API.
3. **Google Cloud Run** offers serverless container execution with GPU support, allowing you to pay only for what you use.
This combination gives you:
- Cost-effective inference (pay-per-use)
- Scalable architecture
- Simple deployment and management
- GPU acceleration for faster responses
## Prerequisites
Before we begin, make sure you have:
- A Google Cloud Platform account with billing enabled
- Google Cloud SDK (version 519.0.0) installed
- Docker installed on your local machine
- A Hugging Face account with an API token
- Basic familiarity with command-line tools
## Project Setup
I've created a GitHub repository with all the necessary files to streamline this process. The repository includes:
1. A **Dockerfile** that builds on the official Ollama image
2. A **Makefile** to automate the deployment process
3. Configuration files and documentation
Let's go through the deployment process step by step.
## Step 1: Clone the Repository and Configure Environment
First, clone the repository and set up your environment variables:
```bash
git clone https://github.com/yourusername/cloudrun-ollama-gemma3.git
cd cloudrun-ollama-gemma3
cp .env.example .env
```
Edit the `.env` file to include your specific configuration:
```
HUGGINGFACE_TOKEN=your_huggingface_token
PROJECT_ID=your_gcp_project_id
REGION=your_preferred_region
SERVICE_NAME=your_service_name
REPO_NAME=your_repository_name
```
## Step 2: Initialize the Project
Run the initialization command to set up your environment:
```bash
make init
```
This command checks for required tools and creates a virtual environment using `uv`.
## Step 3: Install Dependencies and Configure GCP
Next, install the necessary dependencies and configure your Google Cloud environment:
```bash
make install
```
This command:
- Updates the virtual environment with required packages
- Ensures Google Cloud SDK version 519.0.0 is installed
- Configures your GCP project and region
- Creates an Artifact Registry repository if it doesn't exist
- Sets up Docker authentication for Artifact Registry
## Step 4: Build the Docker Image
Now, build the Docker image using Google Cloud Build:
```bash
make cloud-build
```
This process might take 10-15 minutes as it builds the container image and pulls the Gemma3 model. The default machine type for the build is `e2-highcpu-32`, but you can specify a different one:
```bash
make cloud-build MACHINE_TYPE=e2-highcpu-16
```
## Step 5: Deploy to Cloud Run
Finally, deploy the service to Cloud Run with GPU support:
```bash
make cloudrun-deploy
```
You can customize the GPU type and timeout settings:
```bash
make cloudrun-deploy GPU_TYPE=nvidia-l4 TIMEOUT=180
```
The deployment process will create a Cloud Run service with:
- GPU acceleration (default: nvidia-l4)
- 8 CPUs and 32GB memory
- Configured timeout (default: 120 seconds)
- Private access (no unauthenticated requests)
## Testing Your Deployment
Once deployed, you can test your service using curl. Since we've configured the service with `--no-allow-unauthenticated`, you'll need to include an authentication token:
```bash
# Get an ID token for authentication
TOKEN=$(gcloud auth print-identity-token)
# Send a request to the service
curl -X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"model": "gemma3:4b", "prompt": "Write a poem about Gemma3"}' \
https://YOUR_SERVICE_NAME-HASH.a.run.app/api/generate
```
If you've configured your service to allow unauthenticated access, you can use a simpler command:
```bash
curl https://YOUR_SERVICE_NAME-HASH.a.run.app/api/generate -d '{
"model": "gemma3:4b",
"prompt": "Write a poem about Gemma3"
}'
```
## Understanding the Costs
One of the advantages of Cloud Run is its pay-per-use pricing model. You'll be charged for:
1. **Compute time**: When your service is processing requests
2. **GPU usage**: For the time the GPU is attached to your service
3. **Container storage**: For storing your container image
For development and testing purposes, costs are typically manageable. For production workloads, consider setting appropriate concurrency and instance limits.
## Advanced Configuration
The Makefile and deployment configuration offer several customization options:
- **Machine types** for Cloud Build: e2-highcpu-32 (default), e2-highcpu-16, e2-standard-8
- **GPU types** for Cloud Run: nvidia-l4 (default), nvidia-t4, nvidia-a100
- **Timeout** settings: Default is 120 seconds, but can be increased for longer inference tasks
- **Memory and CPU** allocation: Configured for optimal performance with Gemma3
## Conclusion
Deploying Gemma3 with Ollama on Google Cloud Run provides a powerful, scalable solution for AI inference. The combination of Google's efficient language model, Ollama's streamlined container, and Cloud Run's serverless infrastructure creates a flexible system that can adapt to your needs.
The automated deployment process using the Makefile simplifies what would otherwise be a complex setup, allowing you to focus on building applications that leverage the power of Gemma3 rather than managing infrastructure.
Whether you're building a chatbot, content generation tool, or any other application that requires natural language processing, this setup provides a solid foundation to build upon.
## Next Steps
Consider exploring:
- Integration with your existing applications
- Fine-tuning Gemma3 for your specific use case
- Setting up monitoring and logging for your deployment
- Implementing a caching layer to improve performance and reduce costs
Happy building!
---
*This blog post is part of our series on deploying AI models to production. For more tutorials and guides, visit our website or follow us on social media.*