Yup, I think you need to find an ideal question to ask, find a paper that exemplifies a great solution and create a question that you would expect the AI to find it from, then see how close it gets.
Right now, you don’t know if the list is as good as it gets or if it’s missed a humdinger of a perfect find.
A bit late to the party here, but I’m also running my own “research plugin” homebrew thing. What I’m doing for testing is: ask for facts from papers I’ve already been though, that way I have some sort of idea if the plugin is doing what it’s supposed to
Ewww gross! A member. 
That’s pretty cool though @N2U! Right now my homebrew plugin is focused on discoverability of papers.
After that I plan on tackling probing the papers (once I get a good PDF-to-LaTeX solution working for equations and algorithms).
I’ll definitely be knocking on your door when I get there though!
Can you elaborate on what this means in this context?
How do you quantify discoverability?
It’s okay to reinvent the wheel, but what’s the underwhelming part?
I’ve tried this:
It didn’t work as well as I hoped, I think it’s because abstracts are already a summarized version of the research, and they’ll loose some of the context, that your query was originally looking for when summarized again.
Mine also looks like it’s just doing a simple keyword search, but that’s just because I believe it makes it easier for chatGPT to understand how to use it.
That may also be the case with the other plugins, I don’t know, can only speak for myself
Ahh, yeah then it makes more sense, I would still read the original abstract over the summary, but I have had some success with using gpt to simplify abstracts for easier readability 
Of course, mine too. I’m taking about the results I see looking like it’s a very simple keyword search.
Regardless, it’s just a little pet project I’m doing to build a plugin. I don’t anticipate publishing it.