How to optimize chunked grammar/spell-check processing with LLaMA.cpp in Node.js?

API Bugs
, , , ,
1

Hi everyone,

I’m working on a custom RAG-like system in Node.js where users can choose options like grammar correction or spell checking. To avoid hitting the model’s token limit, I split large documents into smaller chunks and process them batch by batch.

Here’s the simplified version of my main function:

async function main({
  filePath,
  modelId,
  checkOption,
  promptTemplateId,
  promptVersionNumber
}: MainProps): Promise<string> {
  let chunks = await parseAndSplitDocument({ filePath });

  // Get the prompt template (from DB or fallback)
  let promptTemplate = await getBasePromptTemplate({ checkOption });
  if (!promptTemplate) {
    promptTemplate = FALLBACK_PROMPT_TEMPLATE[checkOption];
  }
  if (!promptTemplate.includes('{CONTEXT}')) {
    promptTemplate += '\n\n{CONTEXT}';
  }

  const PARALLEL_BATCHES = 2;
  const CHUNKS_PER_BATCH = 4;
  const results = [];

  const llmForCheck = await getGrammarLLM(modelId);

  while (chunks.length > 0) {
    const runnable = {};
    const invokable = {};

    for (let i = 0; i < PARALLEL_BATCHES; i++) {
      const context_array = [];
      const start_index = i * CHUNKS_PER_BATCH;

      runnable[i] = PromptTemplate.fromTemplate(
        promptTemplate.replace('CONTEXT', `CONTEXT_${i}`)
      ).pipe(llmForCheck);

      for (let j = 0; j < CHUNKS_PER_BATCH; j++) {
        const index = start_index + j;
        if (index < chunks.length) context_array.push(chunks[index]);
      }
      invokable[`CONTEXT_${i}`] = context_array;
    }

    const mapChain = RunnableMap.from(runnable);
    const result = await mapChain.invoke(invokable);
    results.push(result);

    chunks = chunks.slice(PARALLEL_BATCHES * CHUNKS_PER_BATCH);
  }

  return results.map(r => Object.values(r).map(v => v?.content).join('')).join('');
}

This works fine, but I’m running into issues:

  • Performance: It’s very slow when processing larger files (hundreds of chunks).

  • Batching logic: Even though I use PARALLEL_BATCHES and CHUNKS_PER_BATCH, the runtime still feels mostly sequential.

  • LLM calls: Sometimes results seem delayed even when running in “parallel” batches.

  • I’m running this on a high-configuration Azure setup that can handle large models easily, so I don’t think hardware is the bottleneck.

My Questions:

  1. Is there a more efficient way to handle grammar/spell checking on large files without exceeding token limits?

  2. How can I make the chunked LLM calls truly parallel in Node.js with LLaMA.cpp? (Right now, it feels slower than expected.)

  3. Would a async parallel execution (Promise.all) be a better fit here?

  4. Any best practices for handling these kinds of text-processing tasks with models running on LLaMA.cpp ?

Stack:

  • Node.js

  • LLaMA.cpp models

  • Dynamic prompts (depending on grammar vs spelling option)

Any advice, optimization tricks, or patterns others have used would be super helpful! :folded_hands:

2

For similar task my approach was to break down the text in paragraphs using regex to obtain my chunks (basically you search for punctuation at the end of the line ignoring empty spaces after it).

Then I opened a pipeline of requests (10 to start) with a limit of 50 requests in the pipeline at most, and started the initial requests, while adding more requests to the pipeline to all chunks were passed through it (with a slight sleep time before adding a new request of quarter of a second).

All done in PHP (curl). The model used for editing was a fine tuned gpt-4o-mini, so that it is fast.

Hope this approach helps.

How long are chunks you are submitting? Mine where one paragraph only (no need too much context for spelling in grammar checks, especially if you’re using fine-tuned models).