Skip to content
Niklas Heckel
Go back

underwheLLMing: Jev is for Swabians

AI Disclaimer

The following parts of this post and benchmark are AI generated:

  • The code for the benchmark is fully AI generated. Since I have a toddler at home (Why again do I even start my own blog?) there was no way to do this by myself.
  • Test data for the third run.

tl;dr: For classification of household expenses using Jev or the Luna Decisions API does not provide a meaningful advantage over LLMs with the Responses API in terms of accuracy, speed, and cost.

Expense Tracking and Categorization

I am currently building an app to track and analyze my household’s finances. The typical household or individual doesn’t have a diverse set of different income types. In contrast the expense side is way more interesting. If you count individual products a household may have hundreds of individual expenses per month. To analyze these systematically expenses need to be assigned to categories. Tracking this manually isn’t fun so a central part of the app is about semi-automatic processing of supermarket receipts or banking statements and assigning them to categories.

So far I used DeepSeek V4.1 Flash to automatically assign categories to individual transactions or expenses. This worked quite fast and was cheap as well.

With the recent release of Jev I thought this might be a task Jev could be better suited for in terms of accuracy, speed as well as costs. I also didn’t measure the performance of DeepSeek Flash or compare it against other models yet. So this was a good opportunity to catch up on this.

The code as well as the input and results of this experiment can be found here. You can repeat this by yourself with my or your own test cases. If you do so please let me know if you come to any other conclusions.

This benchmark isn’t optimized for the specifics of each individual provider and may include some flaws. Nonetheless, I believe there is value in such benchmarks. If a central part of your application relies on any LLMs or decision models you should try to identify the most suited one systematically.

Classification with COICOP

I use UNSD Classification of Individual Consumption According to Purpose (COICOP) 2018 as categories for individual expenses. The classification is publicly available as an Excel file. COICOP divides spending in 15 main divisions with a four digit coding system. The first digit represents the division, the second the group, the third the class and the fourth the subclass. Division one (Food and non-alcoholic beverages) has an optional fifth breakdown into the item level. This is an example from an official UNSD presentation:

Categories 14 and 15 are not about spending by households themselves, but spending by NGOs or governments for households. Therefore I excluded these. Some categories have no further subdivision and are therefore repeated on the lower levels. E.g. “Tertiary education” is the name of both 10.4.0 and 10.4.0.0. In such cases I removed the intermediate nodes and kept only the leaf nodes (10.4.0.0 in this case).

The task of a classifier is to assign one leaf category (a category which has no further breakdown) to a given expense title. For the benchmark I created a (German) dataset with 100 expense titles. It included some low hanging fruits such as Champignons (mushrooms), Fenchel (fennel) but I mainly focused on elements which I expected to be difficult to classify.

These are some of the curiosities included:

I am neither an alcoholic nor a meat-head, but the most ambiguous expense titles I found were mainly regional booze or meat cuisines.

The Classifiers and Strategies

The first obstacle using Jev was that a choice only supports up to 255 options. Because COICOP contains 498 leaf categories I had to use a recursive approach for the classification. For the recursive approach I tested these two variations and expected the second to perform better:

  1. the model gets at each level the available categories for the current level as choices.
  2. additionally to the former, the model gets a display of the full subtree at the current level. This means at the first level the model receives the full COICOP category tree including all divisions, groups, classes, subclasses and items. With the second request it receives only the tree starting from the selected division which includes the classes, subclasses, etc. belonging to that division.

Since OpenAI just released the Decisions API in preview, I tried this with Luna as well.

In contrast to that with DeepSeek Flash I had used a direct approach thus far. The model gets a list of all available leaf categories (as well as the hierarchy) and directly picks one. I expected the recursive approach to perform better than the direct approach. Therefore for DeepSeek Flash, GPT 6.1 Sol and Luna I tested both the recursive and direct approach.

One disadvantage of the recursive approach is that you need significantly more requests to get a classification. With the direct approach you only have to make a single request. The recursive approach requires up to five requests per classification.

The following table shows which combination of models and strategies I tested:

Model/ APIRecursiveRecursive subtreeDirect
Jevxx
GPT 6 Luna Decisions APIxx
GPT 6 Luna Responses APIxx
GPT 6.1 Solxx
DeepSeek V4.1 Flashxx

Decision models getting smashed by LLMs

I did two individual experiments. In the first, I ran the test with a sample size of one meaning I iterated once over the full dataset. So each expense title only gets tested once per combination. In the second run I excluded the GPT 6.1 Sol model to save on costs. In the first run alone the Sol strategies cost me around $3. The second run was set with a sample size of ten which leads to ten tests for each expense title per combination.

The tables below show the accuracy, speed and cost to classify 1,000 items for each combination. The cost cells containing ”≥” show lower bounds because for some tests cost information was not available. For DeepSeek both the peak and non-peak prices are recorded.

The results were quite consistent across both experiments. There are only two notable changes:

  1. The direct approach with Luna was in the second run around 77% cheaper. This can be attributed to the increase of cached input tokens.
  2. For DeepSeek Flash the recursive approach was in the first run notably better than the direct approach. This reversed in the second run.

For all other combinations there are no significant differences in speed or cost and the accuracy is within ±2% of the first run.

The detailed, raw classification logs are available both for the first and the second run. This includes the predictions and speed for all tests across all combinations and expense titles.

First run

Model / APIStrategyAccuracyCost / 1,000 items (USD)Avg. speed (s/item)
GPT 6.1 SolDirect84.0%$25.342673.008
GPT 6.1 SolRecursive76.0%≥ $3.7966213.196
GPT 6 LunaDirect60.0%$1.284722.130
GPT 6 LunaRecursive50.0%$0.163196.380
GPT 6 Luna (Decisions)Recursive46.0%$0.144351.136
GPT 6 Luna (Decisions)Recursive subtree51.0%$2.223571.144
DeepSeek V4.1 FlashDirect49.0%$0.14765 / $0.073821.021
DeepSeek V4.1 FlashRecursive54.0%$0.27909 / $0.139543.448
jev-latestRecursive46.0%$0.108361.430
jev-latestRecursive subtree50.0%$1.182921.569

Second run (sample size 10)

Model / APIStrategyAccuracyCost / 1,000 items (USD)Avg. speed (s/item)
GPT 6 LunaDirect60.3%$0.298781.624
GPT 6 LunaRecursive51.3%$0.161725.803
GPT 6 Luna (Decisions)Recursive45.9%≥ $0.144051.032
GPT 6 Luna (Decisions)Recursive subtree51.0%≥ $2.223481.110
DeepSeek V4.1 FlashDirect55.1%$0.14763 / $0.073810.985
DeepSeek V4.1 FlashRecursive48.5%$0.27470 / $0.137353.293
jev-latestRecursive45.4%$0.108801.389
jev-latestRecursive subtree48.3%≥ $1.179141.539

There are several interesting insights to this. For the analysis I will use the second run as default if not mentioned otherwise or if the model is Sol.

First, I expected the recursive approach to perform better for the non-decision models. It turned out that this was wrong. Except for DeepSeek Flash in the first run all non-decision models performed better with the direct approach than with the recursive approach. Accounting also for the longer time required by the recursive strategy, which is around three times as long for each model, it doesn’t make sense to use such a recursive approach from my perspective.

The winner in terms of accuracy is Sol which achieved 84%. Luna came second with 60% accuracy, around 5% higher than DeepSeek Flash. Sol is too expensive for this task. If you expect the same cache improvement as for Luna across a larger test for Sol you would get $5.18 per 1,000 items which is 17x more expensive than Luna. (The calculation was done with Codex based on the cache hit ratios. I was too lazy to review it. But I’d argue that it doesn’t matter: even if Sol were only 10x as expensive, my point would still stand.) In the future I will consider switching to Luna instead of DeepSeek for this task. But it’s still twice as expensive as DeepSeek’s peak prices and takes 65% more time. For me, that’s a considerable drawback. The scores of 60% for Luna and 49% for DeepSeek seem quite bad. However this was a testset designed specifically with difficult to classify elements. So for your everyday expenses which are way more common elements this should be fine. In the future I would like to repeat this experiment with a larger body of expense titles which are more ordinary.

This brings us to the motivation of this benchmark. Both decision systems performed worse than the Responses API LLMs both in terms of accuracy and speed. If you take off-peak prices DeepSeek Flash also beats Jev in terms of price. The price and speed advantages of the decision models get negated by the necessity to make multiple requests. You could argue that the decision models aren’t intended for these types of classification tasks. However, standard LLMs aren’t designed specifically for this either. Please note that the Luna Decisions API is still in preview. The Luna Decision model also regularly refused to process requests. For each combination the number of refusals was between 5% and 7%. This included mainly categories where I would not expect a guardrail to step in such as Fenchel (fennel), Gutsle (cookies) or Kaba (Cacao Powder). The guardrails of the model might get improved in the future and this would likely improve the score.

Weird Cache Behavior

One thing I noted which I couldn’t explain is that the effect of the strategy on the costs were the opposite for GPT models and DeepSeek. For Sol and Luna the recursive strategy was cheaper than the direct strategy. For DeepSeek the direct strategy was cheaper. I had expected the direct strategy to be cheaper due to a higher reuse of cache which somehow wasn’t the case for the GPT models.

Jev is for Swabians

I randomly saw that Jev was the only model consistently proposing the correct category for “*VEGANE MAULT.”. This is a local food from the German region Swabia and almost part of the identity of Swabians. Sol got this right in the first run as well. Luna and Flash only proposed this 1/10 times in the second run. Jev using the recursive strategy also selected the correct category, coffee, for Muckefuck Grantler Blend in all 10 attempts. None of the other models got this right in the second run.

It’s difficult to understand why a model chooses one category over another. There are approaches to this such as the J-Space from Anthropic. But I believe there is still a lot of room for improvement to help developers better understand the reasoning of models. This would allow for a more systematic approach of prompt tuning. For me personally this still feels more like quackery than engineering.

The Great Downfall

Typesafe reports that the primary training data of Jev is English. Since the benchmark outcome could be different using mostly English titles I also run a third experiment with 50 English expense titles. For this quick test I generated the dataset with GPT 6.1 Sol. I also think these titles are easier than those in the German dataset.

The results are shown in the following table. Raw results are here

Model / APIStrategyAccuracyCost / 1,000 items (USD)Avg. speed (s/item)
GPT 6 LunaDirect92.8%$0.237461.672
GPT 6 Luna (Decisions)Recursive65.8%≥ $0.135651.134
GPT 6 Luna (Decisions)Recursive subtree96.0%$2.048801.271
DeepSeek V4.1 FlashDirect83.4%$0.14821 / $0.074111.025
jev-latestRecursive64.8%$0.097691.228
jev-latestRecursive subtree92.0%$1.021751.393

I kind of regret doing this because it ruined my punchline. If I had just stopped with the first two runs this could have been a great rant about decision models. But instead I just didn’t know better and had to do this stupid countercheck. Unfortunately, I committed the results before looking at them so sweeping this under the rug wasn’t an option either.

Fortunately it isn’t all lost. The Subtree strategy with the Luna Decisions API achieved the best accuracy, 3.2% more than with the Luna Responses API. At the same time it led to an 8.5x increase in costs. Interestingly, this time we didn’t get any refusals from the API.

My personal takeaway from this is that it is really hard to systematically compare the performance of different models and classification strategies. In the future I plan to repeat this benchmark with a larger German dataset.

Conclusion

At the moment there is no clear overall advantage of using the tested decision models/ APIs over LLMs with the Responses API. Luna Decisions API achieves a small increase in terms of accuracy but at 8.5x the cost for English expense titles. For difficult German expense titles they perform significantly worse. I guess that this also transfers to most tasks that involve answering questions with a larger amount of choices. Based on the benchmark for expense title classification with COICOP I would recommend a direct approach with either GPT 6 Luna or DeepSeek Flash.

If you have any thoughts about this I’d love to hear them.