Hello everyone,
I am planning to develop a chatbot designed to assist users in generating reports. Users will provide the necessary data and specify the sections they want included in the report.
To begin with, I intend to build a proof of concept (POC) without a graphical user interface (GUI). My initial approach is to use a prompt-based system, where data is provided to a Large Language Model (LLM), which then generates a report based on the input.
Since LLMs cannot directly include images in their responses, I am exploring different ways to incorporate images into the reports. One possible solution is to host the images publicly so that the LLM can include their URLs in the response.
Has anyone worked on a similar project before? I would greatly appreciate any ideas or suggestions on how to handle this efficiently.
I made a similar bot 1 year ago. Ask the LLM to write the text of the report in Markdown and into a Markdown code block, so
it is easy to distinguish from the rest of the reply
the LLM can format it properly
it can include images by linking them, both via public url or address in your local folders.
Then you use a conversion package to turn the Markdown document into word of pdf (there are several). You can also impost some default formatting, ex. how markdown formatting will convert into word/pdf.
Your approach of hosting images publicly and linking them in the report is a solid solution. Another option is to generate reports in HTML or PDF format, embedding images directly. You could also use a script to process LLM-generated text and insert images dynamically before finalizing the report. If automation is key, consider integrating Python (ReportLab, Pandas) or LaTeX for structured, image-rich reports. Would love to hear more about your implementation!
Thank you for your ideas and suggestions. I conducted a small test and observed the following:
Data Extraction from Excel Files: LLMs cannot automatically extract data from .xlsx files, though they perform reasonably well with .csv files.
Limitations with PowerPoint Files: LLMs are unable to extract images or data from .pptx files.
Report Generation Requirements: Our report needs to incorporate data from multiple formats, including .docx, .pdf, .xlsx, and .pptx.
To address these limitations, we may need to use a library such as Unstructured to extract tables, images, and other relevant data separately before feeding them into the LLM. We have successfully implemented this approach in a previous project, so that part is manageable.
However, our goal is to enable the LLM to automatically insert images at the correct positions within the generated report. I am unsure how this can be achieved using local folders. Typically, when using the OpenAI API, we send images as base64-encoded data or URLs.
Would love to hear any thoughts or suggestions on how to handle this effectively.
Hello everyone,
For the POC referenced above, we carried out the following steps:
Extracted text, tables, and images from the input data.
Stored the images in a way that made them accessible to the LLM via public URLs.
Crafted custom prompts and provided both the extracted text and image URLs to the LLM to generate the report.
Findings:
We found that we had limited control over how the images were integrated into the final report. As a result, we decided early on not to include images in the report.