Rendered at 23:48:26 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
deepsquirrelnet 8 hours ago [-]
> 1 - Is the document about desks designed for children?
> 2 - Does the document offer products or information on kids’ desks?
> ...
> 9 - Are brands or models of children’s desks listed?
> 10 - Does it cover where to buy or how to choose a desk for kids?
>
> Each of these can be turned into a noul question. P(yes) of each can be summed to score overall relevance
It seems to me that this is a rediscovery of natural language inference (NLI), which was very active in NLP research pre-LLMs. I have a side hobby of training NLI cross encoders, and I'd bet that many of these use cases could be serviced by a good quality small NLI model.
But I wonder whether LLM on its own might have gotten close. If you're going to use an LLM for each call then it seems reasonable to test whether or not adding Jev improves performance.
EDIT: I should have had more coffee before writing this comment. As make3 points out below, I forgot that in the LLM+Jev setup, the LLM only needs to run once per query, whereas in the LLM-only setup the LLM would run once per item/segment being evaluated.
esafak 7 hours ago [-]
A reranker has to be fast. LLMs are not.
make3 8 hours ago [-]
Importantly though, LLMs can generate questions in parallel, and there can be much fewer questions (q) than segments to evaluate against (n), so the cost of jev on questions (jev(q * n) + llm(q)) vs the cost of putting all the segments in the LLM can be smaller (llm(n)), (llm(n) >> llm(q) + jev(n * q) if n >> q)
The real benchmark that's missing for me is MaxSim on a real embedder from MTEB.
ungreased0675 7 hours ago [-]
A possible improvement could be generating a sensitivity score for each question. Some are more important than others. For example not mentioning kids at all should cause a huge decrease in relevance, but not mentioning a specific feature should only cause a small decline in score.
jsenn 9 hours ago [-]
This is a cool idea. It would be interesting to see how sensitive the results are to the strength of the LLM used for query augmentation. They used GPT-5 in their experiments, and generating all those tokens would presumably add a lot of latency in your query pipeline.
Another baseline that would be interesting to see is just pure LLM reranking--is Jev/the query augmentation adding anything? Presumably the idea is that Jev is cheaper, but even with a costly LLM you might get a lot of cache hits as long as the query (the variable portion) goes last. Even setting caching aside, almost the entire request is prefill plus a very small amount of generation for the judgment itself. You may end up generating fewer tokens just using the LLM as a judge directly than you need to generate the rubric for Jev. OTOH if queries can be normalized or are naturally repetitive you may be able to cache the rubric...
drob518 8 hours ago [-]
I would assume that you could get by with a very small and fast LLM to do the question generation. You’re not asking the model for very much. From what I can tell, Jev is adding the quantitative probability which makes the score not simply a sum of ones and zeros against the questions.
6 hours ago [-]
ac29 9 hours ago [-]
Have used decision models the same way: Had LLM generate/tune questions for Clef to label/extract emails with {useful work thing} content
Too early to say how useful this is versus existing techniques, the cool thing is this is the first thing I've found local models genuinely useful for. I think I was able to process 50,000 email questions per hour or so with Clef flash on Strix Halo with a custom engine. A modern gaming GPU would likely be much faster
echoangle 8 hours ago [-]
Shouldn't you multiply instead of summing the scores?
Would you show a result with 3 really high scores and one which is really low?
hedgehog 5 hours ago [-]
I use LLMs for ranking and at least for my applications geomean has given me decent scores when the rubric has multiple axes. Raw scores tend to be noisy so either way it helps to take the rough scores and refine by reranking in chunks.
nico 9 hours ago [-]
Super interesting. It’s amazing all the applications we are starting to see. Even if not optimal, a lot of people are now thinking about options and workflows that they weren’t before
> 2 - Does the document offer products or information on kids’ desks?
> ...
> 9 - Are brands or models of children’s desks listed?
> 10 - Does it cover where to buy or how to choose a desk for kids?
>
> Each of these can be turned into a noul question. P(yes) of each can be summed to score overall relevance
It seems to me that this is a rediscovery of natural language inference (NLI), which was very active in NLP research pre-LLMs. I have a side hobby of training NLI cross encoders, and I'd bet that many of these use cases could be serviced by a good quality small NLI model.
See: https://huggingface.co/dleemiller/finecat-nli-v2-l
And also: "Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI" https://www.semanticscholar.org/paper/Less-Annotating%2C-Mor...
- BM25 only
- Jev only
- LLM + Jev
The latter worked best.
But I wonder whether LLM on its own might have gotten close. If you're going to use an LLM for each call then it seems reasonable to test whether or not adding Jev improves performance.
EDIT: I should have had more coffee before writing this comment. As make3 points out below, I forgot that in the LLM+Jev setup, the LLM only needs to run once per query, whereas in the LLM-only setup the LLM would run once per item/segment being evaluated.
The real benchmark that's missing for me is MaxSim on a real embedder from MTEB.
Another baseline that would be interesting to see is just pure LLM reranking--is Jev/the query augmentation adding anything? Presumably the idea is that Jev is cheaper, but even with a costly LLM you might get a lot of cache hits as long as the query (the variable portion) goes last. Even setting caching aside, almost the entire request is prefill plus a very small amount of generation for the judgment itself. You may end up generating fewer tokens just using the LLM as a judge directly than you need to generate the rubric for Jev. OTOH if queries can be normalized or are naturally repetitive you may be able to cache the rubric...
Too early to say how useful this is versus existing techniques, the cool thing is this is the first thing I've found local models genuinely useful for. I think I was able to process 50,000 email questions per hour or so with Clef flash on Strix Halo with a custom engine. A modern gaming GPU would likely be much faster
Would you show a result with 3 really high scores and one which is really low?