Keyword matching versus more NLP

Introduction: We are using ChatGPT to analyze descriptions of thousands of companies to determine whether they match our Ideal Customer Profile (ICP). The process involves first including companies with certain characteristics and then excluding those that engage in activities that disqualify them as matches. However, ChatGPT appears to struggle with truly “reading” the descriptions and instead often admits that it is primarily performing keyword matching. This results in inaccuracies that require extensive manual review.

Specific Issues Encountered:

  1. Literal Keyword Matching:
  • Despite attempting to incorporate broader context, ChatGPT tends to match terms literally. For instance, it included companies like Altra and Simpler Staffing Solutions, which are staffing agencies, because they contained terms related to healthcare or medical services.
  • Similarly, Metropolitan Appraisal Group and American Healthcare Appraisal were included, even though their names clearly indicate appraisal services, not acquisition or development of healthcare CRE.
  1. Lack of Contextual Understanding:
  • When analyzing descriptions, ChatGPT fails to distinguish between different uses of the same term. For example, “rehabilitation” was interpreted as relevant to healthcare CRE even when it referred to rehabilitation of physical properties, not healthcare facilities.
  • This lack of context was also evident when terms like “medical management” or “consultancy” were included as matches, even though they are clear indicators of service providers.
  1. Broader Context Handling:
  • We tried incorporating broader context terms like “senior living” and “medical office” with instructions to mark them for review if the acquisition language wasn’t clear. Despite this, ChatGPT often failed to mark these companies for review, leading to excessive false positives.
  • Conversely, some companies with clear acquisition language were excluded, likely because of the presence of a disqualifying term elsewhere in the description.
  1. Mismatch with Provided Guidance:
  • We provided a file with explicit instructions, marking companies as “yes” (Healthcare), “yes, diversified” (Diversified), or “no” (not a match). Even after this, ChatGPT struggled to align its filtering results with the provided guidance.
  • For example, only 4 of the 12 initially included companies matched the provided guidance, despite explicitly integrating terms from the “include” column.
  1. Output Formatting Challenges:
  • We attempted to generate structured results, splitting companies into included, excluded, and review categories, and providing counts for each. However, the counts frequently did not align with the expected results based on manual analysis.
  • Attempts to create a refined prompt that includes output formatting and sharing instructions have been made, but the accuracy of results remains an issue.

Question to OpenAI Community:

What can we do to improve ChatGPT’s ability to understand descriptions contextually, as opposed to merely performing keyword matching? Are there techniques, prompt adjustments, or methodologies that can better simulate a business development professional’s reading of these descriptions to more accurately identify matches?

We’re open to any suggestions or examples that could help us refine this process. Thank you for your guidance!