Knowledgebase – Managed IT Services & Cybersecurity | Miami IT Company
Step-by-Step Guide: Building and Configuring a Proxmox LLM Server with GPU Passthrough
This guide provides a step-by-step walkthrough for setting up a local LLM server using the following components:
- Proxmox Hypervisor
- Ubuntu Server operating system
- Ollama as the LLM engine
- OpenWebUI as the graphical user interface
These components were selected because they are open source, reliable, and provide the flexibility needed for a private AI environment. The platform also supports advanced capabilities such as Retrieval-Augmented Generation (RAG), allowing your LLM to be enhanced with private datasets and organizational knowledge.
The design was developed with scalability and long-term management in mind. Virtualization allows the entire LLM environment to be backed up, restored, migrated, and expanded as additional hardware resources become available in the future.
Lab Hardware Configuration
For this lab environment, we used a Dell PowerEdge R740 server and NVidia GPUs with the following specifications. These instructions are strictly for Linux, Nvidia and x64 compatible hardware.
- CPU: 24 cores
- GPUs: 2 × NVIDIA Tesla P100 16GB
- Memory: 128GB RAM
Proxmox was selected as the hypervisor platform, with GPU passthrough configured to allow the NVIDIA GPUs to be directly accessed by the Ubuntu virtual machine. This provides near-native GPU performance for AI workloads while maintaining the flexibility and management benefits of virtualization. The Ollama VM will be the only VM that we be run on the hypervisor.
Proxmox also simplifies infrastructure management by allowing virtual machines to be backed up to external storage, replicated, migrated, and scaled across different hardware platforms when additional resources are required.
Scope of This Guide
This tutorial focuses specifically on installing and configuring Ollama as the LLM engine. If you need assistance with installing or managing Proxmox, there are many excellent community resources, videos, and documentation available that cover Proxmox deployment and administration in detail.
Creating a VM and Installing Ubuntu
Create a VM and install Ubuntu Server: https://ubuntu.com/download/
In the VM creation process, select enough CPU cores, disk space and RAM to meet your needs without overextending the hypervisors capacity. I generally allocate no more than 75% of available resources to the VM. Since the heavy pulling will be done by the GPUs, 16-20 CPU cores is more than enough for this type of configuration and additional vCPUs will add diminishing increases or no performance increases at all.
After your VM is provisioned, select it and click on hardware –> add –> PCI Device
Knowledgebase – Managed IT Services & Cybersecurity | Miami IT Company
Select your GPUs from the RAW device list and add them for passthrough to the OS.
Install Ubuntu on the host VM and after completing the OS installation, log in via SSH console and run the following command:
nvidia-smi –L
You should see something like this:
01:00.0 VGA compatible controller: NVIDIA Corporation Tesla P100
02:00.0 VGA compatible controller: NVIDIA Corporation Tesla P100
If you do no see your GPUs, then the PCI passthrough is not working.
Next, check that the NVIDIA drivers are loaded:
lsmod | grep nvidia
If nothing is returned, the NVIDIA driver isn’t loaded. To load the driver, find out which driver you need for your GPU model, and install using one of these commands:
sudo apt install nvidia-utils-580-server
sudo apt install nvidia-utils-595-server
Once the drives are installed, rerun lsmod | grep nvidia to ensure that the GPUs are displayed. You should see something similar to this (since our server has two GPUs, it displays two devices as highlighted in yellow):
00:10.0 3D controller: NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] (rev a1)
Subsystem: NVIDIA Corporation Device 118f
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
00:11.0 3D controller: NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB] (rev a1)
Subsystem: NVIDIA Corporation Device 118f
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
Installing Ollama Step by Step on your Ubuntu Server
To install Ollama, execure the following command from the VM’s shell:
sudo curl -fsSL https://ollama.com/install.sh | sh
Next, verify Ollama version:
ollama –v
Start the server:
ollama serve
Visit https://ollama.com/library and select a model based on your GPU type and memory. In this example, I am going to download Gemma3:27b which requires 17Gb, however you should select a model that does not exceed your GPU memory capacity. It’s recommended to select a model that’s no more than 75% of your GPU capacity to allow context and other processing requirements to load into GPU memory. If your model reached it’s GPU capacity, the rest will be offloaded to your computer/server’s CPU and the model will slow to a craw.
Enter the following command replacing the model with your own.
ollama pull gemma3:27b
Once download is complete, run the command: ollama list to display a list of downloaded models. To begin interacting with the model, enter the command: ollama then select code, chat and then select the model you just pulled. Alternatively you can run Ollama followed by the model:
ollama gemma3:27b
Now that Ollama is installed and working, we are going to add a GUI interface for user friendliness.
First stop the model by pressing CTRL+C twice (if you are interacting with it) the ESC, and then running the command:
ollama stop gemma3:27b (or whatever model you are running)
Next, execute the command ollama rm gemma3:27b to remove the model because we are going to install it next using a GUI.
Installing Ollama+OpenWebUI Web GUI
Olllama can be installed together with OpenWebUI
First, install docker with the following command:
sudo apt install docker.io
Once installed, proceed with the docker image installation:
sudo docker pull ghcr.io/open-webui/open-webui:main
sudo docker run -d -p 3000:8080 -v open-webui:/app/backend/data –name open-webui ghcr.io/open-webui/open-webui:main
These simple two one-line commands will download and install OpenWebUI along with Ollama inside a docket container.
Allowing Access to Docker and Securing Your Ollama Installation
Find your docker’s IP subnet, which is typically 172.17.0.0/16 and gateway 172.17.0.1
sudo docker network inspect bridge
Configure Ollama to bind to your docker network.
sudo systemctl edit ollama
Add the following line item as shown below. This will allow Ollama to be accessed by all hosts, including the docker subnet where the OpenWebUI is installed. Don’t worry, we will firewall the VM later on to prevent it from being accessed from outside of our docker environment.
[Service]
Environment=”OLLAMA_HOST=0.0.0.0:1434″
Restart the services to reload the new configuration.
sudo systemctl daemon-reload sudo systemctl restart ollama
Check that it’s working:
ss -tlnp | grep 11434
Securing the Ollama Virtual Machine with UFW
The next step is to secure the Ollama installation by installing and configuring UFW firewall. This will prevent the Ollama VM as well as the Ollama instance from being accessible from anywhere except from the OpenWebUI docker network. This will force users to sign into OpenWeb UI in order to access Ollama.
Execute the following commands:
sudo apt update
sudo apt install ufw –y
If you use SSH to access the Ollama server, execute the following command while replacing n.n.n.n/24 with your own local subnet and CIDR. This will prevent UFT from blocking SSH and disconnecting you from the Ollama VM.
sudo ufw allow from n.n.n.n/24 to any port 22 proto tcp
Now, create a rule to allow access to the Ollama server from the Docker subnet:
sudo ufw allow from 172.17.0.0/16 to any port 11434 proto tcp
Enable the UFW firewall:
sudo ufw enable
Next, run the command:
sudo ufw status numbered
You should see port 22 SSH allowed only from your LAN or local IP and the Ollama port 11434 only allowed from the docker subnet.
Install Clam-AV antivirus, this will help protect the system by scanning uploaded files before they are indexed.
sudo apt update && sudo apt install -y clamav clamav-daemon
Configuring Open-WebUI
The final step is to configure the Open-WebUI server. Open a browser and navigate to http://ipaddress:3000 and you will be presented with an initial login screen. This will be the administrator’s credentials for the AI stack. Enter your email address and password and proceed to log in.
Once logged in, click on your user name at the bottom left, then select admin panel as shown below.
From there select settings –> connections and add the Ollama server’s IP address as shown below.
Click save and then in the same section (connections) go to download models by clicking on the download icon as shown below:
Find your preferred model at https://ollama.com/library, where these is an extensive library of models of varied weights, quantizations and specialized training. Make a note of your model’s name and then enter it into the download search field as shown below, and click on the download icon.
Now that you have everything in place, you are ready to interact with your artificial intelligence LLM.
Your LLM is ready! To see the performance and utilization of your GPU, run the command: nvidia-smi in a command line. You can watch your GPU’s temperature, memory usage and processor utilization as you run various Open-WebUI’s models.
If your Ollama instance is slow, run the following command to monitor the GPU/CPU usage:
watch -n 10 ollama ps
You should see 100% GPU, this means that the entire model is being run on your GPU. If you see a split between GPU and CPU, then your GPU has run out of memory and the model is offloading work to the CPU, making Ollama very slow. If you see a split between CPU and GPU, use a smaller size model or use a GPU twith greater VRAM capacity. You can also add additional GPUs for aggregate memory to run larger models, but there is a small performance degradation when compared to running a model from a single GPU.
While the LLM is installed, there are many setting you can tweak to enhance your experience. You can settings, plugins, and models directly through Open-WebUI.
For more details, see the documentation: https://docs.openwebui.com/
Enjoy!