Smarter Food Classification with LiquidAI Vision Models

Fine-tuning turns LiquidAI’s small models into high-accuracy food classifiers with smart LoRA training

Today’s newsletter is written by Benito.

Who is Benito?

Benito Martin is a top AI engineer, whom I had the privilege to meet a few months ago when he enrolled in my live bootcamp on building real-time ML systems. He is a curious, humble and hard-working. The perfect combination, and a guy you wanna have in your team.

I am super happy to have him as the first guest of my newsletter.

So, without further ado, let’s get to work!

---

When LiquidAI’s 450M vision-language model hit 91% accuracy on my food classification task right out of the box, I was impressed. This means only 1 in 10 predictions was wrong. That’s genuinely strong performance for a model that size—most vision models need 7B+ parameters to reach similar numbers.

So I asked myself whether fine-tuning could close that gap or not. Could I take an already-efficient model and push it higher without needing massive GPUs or burning through cloud credits?

Spoiler: yes, but not before spending some time creating a simple but effective prompt and being sure the outputs are normalized, as “hot dog” and “hot_dog” aren’t the same thing to an evaluation script.

Also, making sure the model is not confidently declaring everything is a hot dog within the wrong predictions, as this could be a sign of class imbalance or bad hyperparameter tuning. Hamburger? Hot dog. Garlic bread? Also hot dog. Otherwise, you would accidentally create the world’s most optimistic hot dog detector.

Let’ see how I achieved this. The complete code repository can be found in the link below:

The Results: What Fine-Tuning Actually Achieved

Before we dive into how I got here, let’s look at what fine-tuning accomplished. These aren’t theoretical improvements—these are real accuracy gains on test data.

3 Classes Evaluation

5 Classes Evaluation

Key Takeaway

• Fine-tuning added 4-8 percentage points across the board, with the biggest gains on the 450M model.

• For 3 classes, the base models established a strong baseline—91.1% and 96.7%—but fine-tuning improved them to 95.6% and 98.0%, respectively.

• For 5 classes, the smallest model struggled a bit with garlic bread and ceviche, but fine-tuning not only improved both but also boosted overall performance by 8.4%.

At its core, this project was about exploring what small, pre-trained vision-language models can achieve when guided with the right data, prompts, and training strategy. Fine-tuning isn’t just about tweaking a few hyperparameters—it’s about preparing the dataset carefully, defining clear prompts, and running controlled experiments to see what actually works in practice.

Let’s start with how I set up the environment to make it happen.

The Infrastructure: Why Modal?

Here’s the thing about pre-trained vision-language models: they’re Swiss Army knives. They can classify images, answer questions about them, and generate captions—basically handle any vision task you throw at them. LiquidAI’s LFM2-VL models are no exception. The 450M base model hit 91.1% accuracy on my 3-class food dataset right out of the box. The 1.6B model? 96.7%.

Those are respectable numbers. You could ship something with that.

But you also need a proper infrastructure if you want to perform fine-tuning. In plain words, a GPU. And this is where modal becomes the obvious choice. Cloud GPUs, 30 USD/month free, pay-per-second pricing, and most importantly, I could spawn long-running training jobs without worrying about my own computer resources. After all the tests I did with these models over the last month, I spent only half of it.

With Modal handling the compute, let’s focus now on the data and its preparation for fine-tuning.

Training Preparation: The Balancing Act

For this task, I used the widely known Food-101 dataset. It is a dataset with 101 food categories and 75,750 total samples. That’s a lot of food images. I did not use all of them.

Food-101 dataset

Instead, I picked three classes to start: hamburger, garlic bread, and hot dog, and later extended it to 5 classes with ceviche and carrot cake. I pulled 750 samples per class, which gave me 2,250/3,750 images total. Then I split them 80/20 into training and test sets. This is where things get interesting. I didn’t enforce perfect balance—I used controlled randomness in the sampling process. Real-world datasets aren’t perfectly balanced, and I wanted to see if the model could handle slight imbalances without falling apart.

Also, vision-language models output text, and text has formatting quirks. I added some normalization to convert spaces to underscores and lowercase everything.

The other critical piece was prompt formatting. These aren’t pure classification models—they’re chat models. They expect a conversation format with system and user messages. I used a system message defining the task (”You are a food classifier”) and a user message with the image, plus a list of valid classes.

class Prompts(BaseModel):
    system_message: str = Field(
        default=(
            “You are a food classifier.\n”
            “Given an image of a food item, you identify the food category.”
        )
    )
    user_message: str = Field(
        default=(
            “What food type from the following list do you see in the picture?\n\n”
            “\n\n{class_list}\n\n”
            “Provide your answer as a single food type from the list”
            “without any additional text.”   
        )
    )

Additionally, I added the ground truth target from the assistant role so that the model learns the proper label.

def format_sample_for_training(sample: dict, system_message: str, user_message: str) -> list[dict]:

    ground_truth = sample[”label”]

    image = ensure_rgb(sample[”image”])

    return [
        {”role”: “system”, “content”: [{”type”: “text”, “text”: system_message}]},
        {
            “role”: “user”,
            “content”: [{”type”: “image”, “image”: image}, {”type”: “text”, “text”: user_message}],
        },
        {
            “role”: “assistant”,
            “content”: [{”type”: “text”, “text”: ground_truth}],
        },
    ]

With the dataset balance defined and the prompt formatted, the next challenge was actually training something without my laptop catching fire.

Training and Evaluating the Models

Modal lets you define training jobs as Python functions with decorators. You specify the GPU type, timeout, and storage volumes, then Modal handles the rest. I used L40S GPUs because they offered the best balance of memory and cost for vision models.

The training configuration lives in YAML files (finetune_0.yaml through finetune_3.yaml) in the repo, which made experimentation much easier than hardcoding values. For the 450M model, I used LoRA rank 8 with alpha 16. LoRA is the technique that makes fine-tuning affordable—instead of updating all model parameters, you add small adapter layers and only train those. The rank controls adapter size.

For the 450M model, training took about 10 minutes for 2 epochs. The 1.6B model finished in roughly the same time with just 1 epoch. I noticed that adding a second epoch didn’t help—performance plateaued, and the model even started to hallucinate. It’s also a tougher baseline to beat.

So, I tried a different approach. My first run with the 1.6B model used the same configuration as the 450M one, but the training loss wouldn’t budge. For the 3-class training, I lowered the learning rate and increased the warm-up ratio so the learning rate would ramp up more slowly. That did the trick—I got a small but measurable +0.5% improvement.

With that baseline, I trained the model on 5 classes, but this time there was no gain. Then I increased the rank and LoRA alpha while dropping the learning rate. Suddenly, progress. Bigger models seem to need gentler learning rates and sometimes more adapter capacity. Not groundbreaking insights, but definitely easy to miss.

After training, I added an automatic model download for evaluating, as here you need less computing power than for training. However, you can also perform the evaluation using modal.

There are two evaluation scripts in the repo. One for the base model and one for the fine-tuned model. You need to add two arguments to run it: the config, to select the model and the dataset, and the finetune-config, as the train/test split configuration for fine-tuning is there, and we need to evaluate on the same training setup.

Evaluate the base model:

uv run src/food_images_finetuning/evaluation/evaluate_base_model.py \
  --config src/food_images_finetuning/configs/config_450M.yaml \
  --finetune-config src/food_images_finetuning/configs/finetune_1.yaml

Evaluation of 450M fine-tuned on 5 classes

Evaluate the fine-tuned model:

uv run src/food_images_finetuning/evaluation/evaluate_fine_tune_model.py \
  --config src/food_images_finetuning/configs/config_450M.yaml \
  --finetune-config src/food_images_finetuning/configs/finetune_1.yaml

Evaluation of 450M fine-tuned on 5 classes

As you can see, the results of the 450M fine-tuned model improved significantly. Fine-tuning not only provided a boost of +8.4% in accuracy but also helped the model better learn the target classes. Although both models used the same prompt designed to constrain predictions to a fixed set of classes, the base model still produced undefined outputs such as egg, sushi, or pizza. After training, the fine-tuned model still failed on a few samples, but at least it attempted to predict one of the learned classes.

Insights on the Results

Apart from the overall performance improvement and better class recognition, there are other aspects worth mentioning.

Looking back at those results tables, you can see the baseline performance established by the base models. The 450M base started at 91.1% for 3 classes and 85.6% for 5 classes. The 1.6B base started higher at 96.7% for both tasks. These baselines matter because they show you’re not starting from scratch—you’re taking already-capable models and specializing them.

The per-class breakdown tells a more interesting story. Garlic bread was consistently the hardest to classify across all models. The 450M base model only hit 84.6% accuracy on garlic bread, while managing 95.7% on hot dogs. Fine-tuning brought garlic bread up to 98.8%, essentially fixing the weak spot.

Hot dogs were easy for everything. Even the base models rarely misclassified them.

The 5-class results showed diminishing returns for the larger model. The 1.6B fine-tuned model only gained 0.5 percentage points over the base. Meanwhile, the 450M model jumped 8.4 points. Smaller models have more room to improve with task-specific training. Bigger models are already closer to their ceiling on simple classification tasks, and on top of that, some food images contain more than one item, which makes it hard to reach a 100% overall accuracy on all classes.

Notice the test set isn’t perfectly balanced? Ceviche has 169 samples, while hot dog has 137. This wasn’t an accident. As I mentioned before, I used controlled randomness in the train/test split to mimic real-world conditions. The results suggest the controlled randomness worked—no class dominated predictions just because it had more samples.

What Actually Matters

Here’s what surprised me: model size mattered less than I expected. The fine-tuned 450M model (95.6%) almost beat the base 1.6B model (96.7%) on the 3-class task. We are talking about only 5 missed images out of 450. Not by much, and it demonstrates that a smaller model with task-specific training can compete with a larger general model. This has real implications for deployment—smaller models run faster and cheaper.

The proper chat template formatting is non-negotiable. Vision-language models expect structured conversations. And you do not need a long prompt, especially for a classification task where the model must predict one item. For this specific use case, simple is better, and with the expected structure: system message, user message with image, and assistant response with the label.

The real lesson here isn’t about model architecture or hyperparameters. It’s about data preparation. I spent more time getting the dataset balanced, normalized, and formatted correctly than I did tuning training parameters. Get the data wrong, and no amount of hyperparameter tuning will save you.

Conclusion

With all these experiments and observations in place, it’s time to wrap up with a few key takeaways from the fine-tuning process.

Fine-tuning improved accuracy by 4-8 percentage points across all model variants. The base models were good, establishing solid baselines of 91.1% and 96.7%. Fine-tuning made them better for this specific task, which shows not only how strong LiquidAI models are but also how strong they can be.

Modal makes cloud training manageable without infrastructure headaches. YAML configs make experimentation faster, considering the number of parameters that you can tune for this task.

The gap between 91% and 98% accuracy was filled with data normalization, balancing, learning rate adjustments, and other hyperparameter tuning.

Last but not least, I’d like to thank Pau Labarta Bajo for allowing me to write an article about fine-tuning LiquidAI models in his newsletter. If you’re reading this, you’re probably already subscribed. However, you can also check out my own newsletter,AI Echoes, where I write about RAG systems, agentic applications, and end-to-end deployments on AWS and GCP.

I also recently launched a free course in collaboration with Miguel Otero Pedrido: a Substack Newsletter Search Engineproject featuring four lessons and a full video walkthrough.

Thanks for reading, and happy fine-tuning!

Benito