I wanted to reach out to the community to see how people are approaching the intent classification in their applications. For example, if I am creating a digital twin of a HR department. How do I determine the intent from the user question to route to the right HR workflow. Just plain intent classification with LLM does not work and is not consistent. One of the approach I am thinking is to use knowledge graphs. Would like to see if others also attempted in a similar way.
Hi! Can you share more details about how you are currently approaching the intent classification? Do you provide the LLM with a defined list of different intent categories?
This is what I am planning. Lets say , I have a set of workflows that is accessible to some users and roles. I first create a knowledge graph(using neo4j) and create association of user, role, workflow, questions, category of questions. Once use asks a question, I generate a category from the question and then two two searches one with category and other with question against the KG. I fuse the output from both, do a WRRF and then take the top 1 result.
you will have to determine what things HR does and from that build your categories for HR. Than when a message comes in it can look at the options you present to know which logic path choice to take.
plain does work believe me but you have to prompt engineer to ensure its not fuzzy on the logic but more direct.
I use neo4j with kruel.ai and it uses a lot of intents for many memory pulls from many data points.
here is a very basic logical way to get some results. you will have to think about all you require for every category and build out direct examples of each if you want more direction. its all about your prompt engineering to ensure that all small details go to the right place. you can than also add another layer after if need be to see if its correct like a feedback loop with more instructions to be sure. many ways to do it.
this is a simple approach:
async def determine_intent_basic(message: str, async_client) → str:
print(f"Entering Determine Intent")
prompt_text = f"“”
Analyze the following message and classify its intent based on the content:
- Use ‘POLICY’ if the message primarily involves questions on company policies.
- Use ‘FORMS’ if the message contains requests for forms.
- Use ‘HR’ if the if the question requires support from human resources
Consider any direct or indirect or general information as a basis for your classification.
Message: "{message}"
"""
try:
response = await async_client.chat.completions.create(
model="gpt-3.5-turbo-0125",
messages=[{"role": "system", "content": prompt_text}],
max_tokens=1024,
temperature=0.5
)
if response.choices:
choice_content = response.choices[0].message.content.strip().lower()
logging.debug(f"Determine intent content: '{choice_content}'")
# Simplifying the response to match one of the predefined intents
if "POLICY" in choice_content:
return "POLICY"
elif "FORMS" in choice_content:
return "FORMS"
elif "HR" in choice_content:
return "HR"
else:
# Default fallback if the response doesn't clearly match any category
logging.error("Response did not match expected intent formats.")
return 'keyword_search'
else:
logging.error("No valid choice in the GPT response")
return 'keyword_search' # Default fallback
except Exception as e:
logging.error(f"Error in determine_intent_basic: {e}", exc_info=True)
return 'keyword_search' # Fallback in case of exception
From here you can take it as far as you like. with Kruel.ai I use a lot of dynamic calls and it has a lot of layered logic to ensure that it gets to where it needs to go. when speed is an issue you can do async and parallel calls if you need to crunch fast with our adding additional time.