780M, Ollama, oh dear

So, shiny new lenovo with a 8840HS CPU rocking the 780M (A)GPU

This was performed on a lenovo P14s 5Gen AMD 8840HS with 32GB memory running Fedora Workstation 41

Let’s start with the obvious stuff. The latest GPU firmware must be installed and the latest version of ollama must also be installed. As of this writing it is ollama 0.4.6

Then you may, or you may not, want to up the dedicated amount of “VRAM” your (A)GPU is allowed to use. My, albeit limited, experience is that leaving it at AUTOMATIC will send 50% of the processing to the CPU and 50% to the (A)GPU when working with models around ~4GB in size. Performance wise this isn’t too bad on my system, however setting BIOS to allocate 8GB will allow for ~8GB models to use the (A)GPU only..(and then also smaller models naturally).. which is faster, but wastes memory in most scenarios. I also noticed some instabilities with 8GB VRAM allocated, but I did not care to investigate it further.

I haven’t tried larger models with command line ollama.. i did try 14GB models using LM Studio. It worked, but it crashed my gnome session after a while and besides, I don’t think I like the LM Studios terms so I removed it from my system.

If you have followed along this far you’ve realized that “that guy is full of s**t.. this doesn’t work!” and you’d be right. I have not come to the most important part yet.

If you do -

$ journalctl -fu ollama --no-pager

you will see loglines indicating that ollama does not know about your (A)GPU.. even though ollama clearly stated that it was present when you installed ollama. So we need to remedy that.

I guess there are multiple ways to perform this next step, but I chose the path of least resistance.

$ sudo vim /etc/systemd/system/ollama.service
# insert the following in to the [Service] section. Yes, there is already
# a variable named Environment, we will add one more.
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.2"

# save the file and exit
$ sudo systemctl daemon-reload
$ sudo systectl restart ollama.service

Finding the correct HSA_OVERRIDE_GFX_VERSION" was pain in the proverbial butt! The only place I’ve found it specified is here https://discuss.linuxcontainers.org/t/rocm-and-pytorch-on-amd-apu-or-gpu-ai/19743

I’ll admit that my google foo is not as strong as it once was, but I blame google and not my self (I had to give up on duckduck.. )

• for GCN 5th gen based GPUs and APUs HSA_OVERRIDE_GFX_VERSION=9.0.0

• for RDNA 1 based GPUs and APUs HSA_OVERRIDE_GFX_VERSION=10.1.0

• for RDNA 2 based GPUs and APUs HSA_OVERRIDE_GFX_VERSION=10.3.0

• for RDNA 3 based GPUs and APUs HSA_OVERRIDE_GFX_VERSION=11.0.0

I mucked about with 10.3.0, which was the only reference I found to GFX_VERSION, for several hours until I stumbled upon this guy: qkiel So, thank you qkiel, you saved my sanity.

So, there’s that.

I leave you with this

zsh> ollama run llama3.2
>>> Why should I blog about llama?
Blogging about llamas can be a unique and fascinating niche. Here are some reasons why you might consider starting a blog about llamas:

1. **Growing interest in alternative pets**: Llamas have become increasingly popular as exotic pets, and people are willing to learn more 
about their care, behavior, and needs.
2. **Unique perspective**: By focusing on llamas, you can offer a fresh and distinct voice in the world of pet blogging or wildlife writing.
3. **Educational value**: Llamas are intelligent, social animals that require specific care and attention. A blog about llamas can provide 
valuable information for potential owners, breeders, or anyone interested in these amazing creatures.
4. **Community building**: You can connect with a like-minded community of llama enthusiasts, share experiences, and build relationships 
with people who share your passion for these animals.
5. **Creative expression**: Writing about llamas can be a creative outlet, allowing you to express yourself through storytelling, poetry, or 
other forms of writing.
6. **Monetization opportunities**: If you have a large and engaged audience, you may be able to monetize your blog through advertising, 
affiliate marketing, or sponsored content related to llama-related products or services.
7. **Personal growth**: Learning about llamas can be a fun and rewarding experience that helps you develop new skills, such as animal care, 
research, or writing.

Some potential blog post ideas to get you started:

* "10 Fun Facts About Llamas You Never Knew"
* "A Beginner's Guide to Keeping Llamas as Pets"
* "The Benefits of Llama Therapy and How It Can Help People"
* "Llama Behavior: Understanding Their Body Language and Social Structure"
* "Llama Care and Nutrition: What You Need to Know"

Remember, starting a blog is just the first step. To make it successful, focus on creating high-quality content, engaging with your 
audience, and continuously learning about llamas to share your knowledge with others.

‘till next time.