During testing, large PDF documents (NIST IR 8428 or other research articles for example) exceed the token output window of the LLM that's in use. I suggest one of two recursive solutions:

a. Bad: may create section headers that don't make sense when attempting to determine the original document those section headers were created from. Goes against what appears to be the general theory behind Recall.

b. Good: Summaries themselves maintain the maximum output token window for each section.

a. Bad: Maximum token output requires total compression of all previous summaries into one condensed summary. Substantial context may be lost due to the lack of reference back to the original source material when compressing summaries.

b. Document is kept in one card.

An example pseudo-function for option 2 is as follows:

FUNCTION summarizeLargePDF(pdfFilePath, maxInputTokens, maxOutputTokens, llm):

currentPage = 1

totalTokens = 0

currentSummary = ""

allSummaries = [] # Array to hold section summaries

sectionStartPage = 1

FUNCTION processPage(pageNumber):

pageText = extractTextFromPage(pdfFilePath, pageNumber)

pageTokens = countTokens(pageText)

IF (totalTokens + pageTokens) > maxInputTokens:

sectionEndPage = pageNumber - 1

sectionText = extractTextFromPages(pdfFilePath, sectionStartPage, sectionEndPage)

sectionSummary = llm.summarize(sectionText, maxOutputTokens)

allSummaries.append(sectionSummary)

sectionStartPage = pageNumber

totalTokens = 0

totalTokens += pageTokens

currentPage += 1

processPage(currentPage)

processPage(currentPage) # Start recursive processing

WHILE allSummaries.length > 1:

newSummaries = []

FOR i = 0 TO allSummaries.length - 1 STEP 2:

combinedText = allSummaries[i] + allSummaries[i+1]

newSummary = llm.summarize(combinedText, maxOutputTokens)

newSummaries.append(newSummary)

allSummaries = newSummaries

finalSummary = allSummaries[0]

RETURN finalSummary