DEV Community

Miguel Morente de las Heras
Miguel Morente de las Heras

Posted on Edited on Fully Autonomous

Three AI containers on AMD ROCm: what I learned building OCR, RAG and a web agent with no API keys

Written for the AMD AI League by lablab.ai and AMD. #AMD #lablab #ROCm

The AMD AI League sets one hard rule for every challenge: ship a Docker image that runs on AMD GPUs with ROCm, with the model inside the image. You get no API keys, no cloud LLM, and 30 seconds per question. I built three of these containers: an OCR reader for licence plates and road signs, a RAG system that cites the exact files behind each answer, and a web agent that returns one exact value and the page it came from.

The models are small (4B to 8B). What decides the score is the engineering around them. These five lessons held for all three.

1. Load the model once, answer many times

Loading a vision-language model takes tens of seconds, which is most of the 30-second budget. So the container starts a resident server that loads the model onto the GPU once and listens on a local unix socket. Each question runs as a small, fresh process that talks to that server, so no question pays the loading cost.

2. Write a valid answer before you think

The first thing each run does is write a valid, empty JSON answer. The real answer overwrites it later. A crash, a timeout or an out-of-memory error then leaves a well-formed "I don't know" behind, never a broken file. An internal deadline stops the work well before the 30 seconds run out.

3. Let the model choose, never invent

For the web agent and the RAG system, the model extracts the answer, but the answer is only accepted if it appears literally in the passage it cites. When it doesn't, the system returns an empty answer. That one check removed nearly all of the hallucinations in my tests, and an empty answer scores better than a wrong one.

4. Classical retrieval does most of the work

BM25 with stemming, stop words in two languages and a bonus for the requested type of value (a phone number, a date, a version) finds the right passage before the model sees anything. The model then only reads a few hundred tokens, which keeps it fast and makes it harder to fool.

5. Always have a plan B without the model

If time runs short, rule-based extractors (regular expressions for phones, emails, prices, dates and versions) answer from the best passage, and only when the words of the question are right next to the value. The answer is a little worse, but it is never empty because of a timeout.

ROCm notes

  • Start from the official rocm/pytorch image and install transformers with --no-deps. Otherwise pip may replace the ROCm build of PyTorch with a CUDA one.
  • Keep the model weights inside the image and check the final image size early, since the league limits it.
  • bf16 with greedy decoding gave me stable, repeatable answers.

Code

The OCR and RAG containers are open source: https://github.com/automatizacionesmorente-png/amd-ai-league-mini-challenges

Thanks to AMD and lablab.ai for the GPU credits and the challenges. If you are building on ROCm too, I'd like to hear what worked for you.

Top comments (1)

Collapse
 
li_yi_dcec96bf2647dec1b7e profile image
Mindutil •

Lesson 2 is the one I would steal outside of a competition: writing a valid empty answer before doing any work turns a crash into a scored refusal instead of a missing file, and most error handling still assumes the process gets a final word. Lesson 3 has a shape that applies to directories too. "Accept the answer only if it appears literally in the cited passage" is the check pricing labels fail. I run a catalog of ~4,300 AI tools where 2,198 rows say free, and the label comes from a scraped field, not from a vendor statement; I have three rows for one vendor, scraped from three landing pages of the same product, answering free / free / freemium. Retrieval was fine in that case, the citation was decorative - which is exactly the failure your literal-string rule catches.