↓Skip to main content

Ollama - Setup

Table Of Contents
#

If your here because you want to use Insightful Search, Ollama is the most annoying part of it. Only because it requires relatively expensive hardware. You need a semi-recent graphics card capable of running AI models. If you have a gaming PC you might already have one that can be used for Ollama, and you can even run Ollama on that PC if you want. That’s what I did until I could afford a separate system for it.

Hardware Requirements
#

To check if your GPU is capable of running Ollama click here to go to Ollama’s official documentation. In addition to that, I can personally confirm that the AMD RX 9060 XT 16GB also works even though, at the time of writing this, it’s not on the list. I would like to assume that also means the AMD RX 9070 and AMD RX 9070 XT also work since they are the same generation, but I don’t know for sure. If you already have one of those, it can’t hurt to try. If you intend to purchase a GPU for this specifically, get one from the Ollam list that has at least 16GB of VRAM. Some newer cards have 8GB but that’s not usually enough, at least if you want decent performance.

Operating Systems
#

Ollama can be installed on a different system than the one you intend to access it from. If you plan to run Insightful Search on TrueNAS, you can run Ollama on a different PC on your local network.

Since most of the guides on this site are for TrueNAS I’ll talk about that first. If you plan to run Ollama on TrueNAS there is a HUGE (potential?) caveat. This might be because the RX 9060 XT I’m using isn’t fully supported by the current version of TrueNAS. When I realized the current version of TrueNAS doesn’t have the drivers for AMD 9000 series GPUs, I tried passing it to a Virtual Machine. Doing it that way requires two GPUs. One for the Virtual Machine and one for TrueNAS. The second one can be something super cheap. For example, I use a GT 710 (There is another caveat with this GPU specifically that I will talk about later) I got on Amazon for around $60. If your GPU IS supported by TrueNAS, you might be able to get this working by using the Ollam app from the Discover Apps section of TrueNAS. Since I can’t test that, try it at your own risk.

In this guide I will talk about how I installed Ollama on a Linux Virtual Machine in TrueNAS. I will also talk about the Ollama app from the Discover Apps section of TrueNAS, but I haven’t used that myself so take what I say on that with a grain of salt.

If you want to install Ollama on a system other than TrueNAS (or a Virtual Machine on TrueNAS), you can download it from the official Ollama website (linked below):

TrueNAS App
#

I’ll start here because the Virtual Machine route is longer, and a bit more complicated.

First, you will need to make a dataset for Ollama to store downloaded models in. I highly recommend this Dataset be put on an SSD, I’m not sure if an HDD will be fast enough.

From the TrueNAS web interface, go to Apps > Discover Apps and search Ollama click on the one just called Ollama and click Install.

If you have and AMD GPU make sure to change the dropdown menu under Image Selector to ROCm Image. Choose a port and write it down, you will need this when trying to connect to and use Ollama. Next to Host IPs click Add and leave the default option. Move down to Storage Configuration and change the Type to Host Path. Point it to the Dataset you just made for it earlier. Make sure to check the box under GPU Configuration next to **Passthrough available (non-Nvidia) GPUs, then click install.

If it works, You’re done! If not, keep reading, and try the Virtual Machine method I went with.

Virtual Machine (VM)
#

First make sure you have two GPUs like I mentioned above. You will want to good one for Ollama to be installed closer to the CPU on the motherboard and the cheap one in the farther slot. I don’t know if it’s 100% necessary, but I’m pretty sure it performs better in the slot closer to the CPU.

Boot up TrueNAS and connect to the web interface. Click on System > Advanced Settings then scroll down to Isolated GPU Device(s). Click configure, you will get a warning about changing advanced options, just close it. Select the GPU you plan to use with Ollama, and click save. Before we make the VM, reboot the TrueNAS system with a keyboard and monitor connected and enter the UEFI / BIOS. If you don’t know how, it sometimes says to press a certain key while the system is turning on. You can also search the brand and model of motherboard online and find out from the company’s website.

In there you need to find and enable some settings:

  • IOMMU
  • Virtualization
  • Above 4G Decoding
  • ReBar

You may also need to disable CSM (Compatability Support Module).

If you don’t know how to find those settings search online with the brand and model of your motherboard and CPU. If your CPU doesn’t support one or more of these features, then you unfortunately can’t continue without buying a new CPU. Sorry.

Now save and exit the BIOS / UEFI and reboot into TrueNAS. Depending on your hardware, you might notice that the GPU you isolated shows up in System > Advanced Settings > Isolated GPU Device(s) but if you click Configure again, it says Unknown. That’s what happened to me with the RX 9060 XT and GT 710, it may be different for you. Even if it says Unknown, as long as you see the correct GPU in the list it should work.

Go to Datasets and create a new dataset, preferably on an SSD. Call it vms or something similar. Inside that dataset, create a new Zvol, name it whatever you want, this is where you will put the VM. Give it at least 40GiB to 50 GiB, you will need space for downloading models. You can always make it bigger, but making it smaller is VERY risky. Leave all the other options as default.

Now you need to use the TrueNAS shell. Open it by clicking System > Shell. You are going to download and install a disk image for Fedora Server onto the Zvol you just created. Run the following commands:

Before you make the VM you need to create a network bridge for it to be able to access the internet to download ollama and models. Watch this video to learn how, I didn’t make it, but I found it helpful.

In the Virtual Machines tab of TrueNAS click Add in the top corner. Use the following options:

  • Guest Operating System: Linux
  • Name: Anything, I called mine GPUPassthrough (you can’t use spaces)
  • Boot Method: UEFI
  • Create a Password
  • Click Next
  • Virtual CPUs: 1
  • Cores: 2
  • Threads: 1
  • CPU Mode: Host Passthrough
  • Memory Size: At least 4 GiB but 6 GiB to 8 GiB would be better
  • Click Next
  • Use Existing Disk Image
  • Disk Type: VirtIO
  • Select the Zvol you created for the VM
  • Click Next
  • Adapter Type: VirtIO
  • Attach NIC: The Network Bridge you created earlier
  • Click Next
  • NO Installation Media (you already installed it to the Zvol)
  • Click Next
  • Ensure Display Device: UNCHECKED If you leave this checked, it will cause problems later
  • Choose the GPU you isolated earlier (If it says Unknown and you saw your GPU in the list before, choose that)
  • Click Next
  • Click Save

If it doesn’t start automatically, click on the new VM and turn it on. Then click the Display button. It will ask you for the password you made earlier. Enter it and finish setup. It’s been a while since I set this up myself, so I don’t remember what else you need to do to setup the Fedora Server after the VM starts. I will try to come back and update this with more detail when I have time. If you get stuck, look online for a YouTube video or a forum post or something.

After the VM is set up, you can either use the Virtual Display or go back to TrueNAS and use the Serial Shell. Now to install Ollama! Finally!

You can reference the official docs here if you need to. If you have an AMD GPU like me you might need to use the manual install script.

From the VM run the following command:

That should download and install Ollama. If using an AMD GPU you will also need to run the following command:

That installs the AMD ROCm Drivers which are necessary. You can now test it (Technically). If you want to test it right this moment, you will need a terminal on whatever PC or Laptop you have. I recommend waiting to test it because we still have a bit of setup, and if you test it now, you will need to redownload any AI models you use later. You need to SSH into the VM, by typing ssh your_username@the_vm_ip into the terminal. Now that you have the terminal AND the Serial Shell or Display from TrueNAS, use one of them to run the command: ollama serve. Then use the other to run the command: ollama run qwen3:latest. That will download and run the LLM qwen3 specifically the 8b version. You can chat with it after it finishes downloading. Press CTRL + D to exit the chat when you’re done.

Technically Optional, But Recommended Step#

Setting ollama up as a systemd service will make sure it runs automatically when you start or restart the VM.

Ollama should have created a service file when it was installed, but you need to make a small change. Run the command: sudo systemctl edit ollama. Add the following lines near the top:

[Service]
Environment="OLLAMA_HOST=http://YOUR_VM_IP:11434"
Environment="OLLAMA_HOME=/home/ollama"

OLLAMA_HOST tells Ollama to let you connect to it from another device on your network. This is necessary becasue the Virtual Machine counts as a separate device.

OLLAMA_HOME tells Ollama to use its own home directory instead of the one for your user account. Instructions for creating that are further down.

I almost forgot to mention, you will need to set a static IP for the VM using your wifi router configuration page. If you don’t know how, search the model and brand of your router online, it’s different for each one. You can (and probably should) also set a static IP for the TrueNAS system, especially if you intend to use it for Insightful Search.

After editing the systemd service, run:

  • sudo systemctl enable ollama
  • sudo systemctl start ollama
  • sudo systemctl stop ollama

Now we need to create a home directory for Ollama by running these commands:

  • sudo usermod -d /home/ollama ollama
  • sudo mkdir -p /home/ollama
  • sudo chown -R ollama:ollama /home/ollama

This will change where ollama downloads AI models to. Instead of /home/YOUR_USER/.ollama it will now be /home/ollama/.ollama, so if you tested it earlier and downloaded any models, they will need to be redownloaded.

For Insightful Search#

If you plan to use this for Insightful Search you should download models using the following commands:

  • If using the Serial Shell in TrueNAS:
    • ollama pull qwen3:latest
    • ollama pull hf.co/mradermacher/bge-large-en-v1.5-i1-GGUF:i1-Q4_K_M
  • If using ssh from a terminal on another device:
    • OLLAMA_HOST=http://YOUR_VM_IP:11434 ollama pull qwen3:latest
    • OLLAMA_HOST=http://YOUR_VM_IP:11434 ollama pull hf.co/mradermacher/bge-large-en-v1.5-i1-GGUF:i1-Q4_K_M

Custom Model
#

One more thing you can do if you want to customize the model you use for Insightful Search (or anything else):

  • Click here to download a template Modelfile for Ollama.
  • Open it in a text editor.
  • Customize the SYSTEM prompt.
  • Optionally change the model it’s derived from with the FROM line.
    • Just make sure you have that model downloaded before continuing.
  • If your GPU has at least 16 GB of VRAM you can leave the PARAMETER line.
    • It increases the context window for the AI model. That means it can use more input data to form a response.
    • The context window must have enough room for:
      • Your Input Prompt, the Database Results (If using Insightful Search), and the AI’s Response.
  • If your GPU has less than 16 GB of VRAM remove the PARAMETER line or it will cause significant slowdowns.
  • Click here to see Ollama’s official documentation on what options you can use in your modelfile.
  • When you finish editing the file save it somewhere easy to get to and in your terminal run the following command:
    • OLLAMA_HOST=http://YOUR_VM_IP:11434 ollama create YOUR_NEW_MODEL_NAME –file /PATH_TO_YOUR_MODELFILE
  • When setting up Node-RED, find the ollama-chat node and change the Model property to the name you gave your custom model.

For the next step click on either of the following links to continue: