Best Table Text Format for Embeddings Generation

Prompting
,
1

I’m creating a generative AI ChatBot where users ask questions in regards to various product offerings.

The majority of this information is located in tables/matrices in our internal Knowledge Base articles (HTML). To ensure accurate answers are being generated, I was curious what would be the best text format to generate embeddings from tables/matrices?

Several of these tables/matrices have rowspans where one value is used in multiple rows but only displayed once, embedded tables, or even ordered/numbered lists.

My initial though was to convert the HTML to Markdown and repeat any rowspan value on each line…but Markdown doesn’t support embedded tables or lists (unless in HTML) and to reduce size/cost/noise I don’t want to include HTML when generating the embeddings.

2

What do you mean by ‘Markdown doesn’t support embedded table’?

Tables are in the main markdown spec: Extended Syntax | Markdown Guide

Otherwise, I think your plan of converting docs to markdown first should work great. I do the same for my production apps. And yeah, repeat rowspans as you planned, since default Assistants API RAG just uses chunks and vector search underneath (as far as I know).

3

Some of our HTML tables contain embedded (or nested) tables:

e.g.

<table>
  <thead>
    <tr>
      <th>Column Header 1</th>
      <th>Column Header 2</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1.25</td>
      <td>70%</td>
    </tr>
    <tr>
      <td>2</td>
      <td>
        <table>
          <thead>
            <tr>
              <th>Sub-Header 1</th>
              <th>Sub-Header 2</th>
            </tr>
          </thead>
          <tbody>
            <tr>
              <td>60%</td>
              <td>Apple</td>
            </tr>
            <tr>
              <td>80%</td>
              <td>Banana</td>
            </tr>
          </tbody>
        </table>
      </td>
    </tr>
  </tbody>
</table>