Using AgentM to watch for new research papers of interest

[stevenic]

I’m getting enough of the pieces of AgentM in place that I’m able to get it to do useful things. I wrote a small program (ok AgentM wrote part of it) that fetches the last days worth of research papers from arxiv.org, filters them to the papers related to topics I care about, and then projects those filtered papers to a uniform markdown format for easy scanning:

It uses gpt-4o-mini so it’s cost effective to run and it took 6 or 7 minutes in total to process 553 papers. Here’s the meat of the code:

// Initialize OpenAI 
const apiKey = process.env.OPENAI_API_KEY!;
const model = 'gpt-4o-mini';
const completePrompt = openai({ apiKey, model });


// Define the projections template
const template = 
`# <title> (<pubDate in mm/dd/yyyy format>)
<abstract>
[Read more](<link>)`;

async function main() {
    // Fetch latest papers
    console.log(`\x1b[35;1mFetching latest papers from arxiv.org...\x1b[0m`);
    const data = await fetchUrl(`https://rss.arxiv.org/rss/cs.cl+cs.ai`);
    const feed = parseFeed(data);

    // Identify topics of interest
    const topics = process.argv[2] ?? `new prompting techniques or advances with agents`;
    console.log(`\x1b[35;1mFiltering papers by topics: ${topics}...\x1b[0m`);
    
    // Filter papers by topic
    const parallelCompletions = 3;
    const filterGoal = `Filter the list to only include papers related to ${topics}.`;
    const filtered = await filterList({goal: filterGoal, list: feed, parallelCompletions, completePrompt });
    if (!filtered.completed) {
        console.error(filtered.error);
        return;
    }

    // Generate projections
    console.log(`\x1b[35;1mGenerating projections for ${filtered.value!.length} of ${feed.length} papers...\x1b[0m`);
    const goal = `Map the news item to the template.`;
    const projections = await projectList({goal, list: filtered.value!, template, parallelCompletions, completePrompt })
    if (!projections.completed) {
        console.error(projections.error);
        return;
    }

    // Render papers
    projections.value!.forEach((entry) => console.log(`\x1b[32m${entry.projection}\x1b[0m\n\n${'='.repeat(80)}\n`));    
}