How to teach a model to match multiple lists

[sergei_shk]

I’ve got a task to make sure the model can tell whether one can send or receive payments in a certain currency to a specific country. There’s a list of supported countries and a list of supported currencies. I’ve tried a few approaches:

1. The simplest one. In the prompt, I wrote “Supported countries for sending payments”, “Supported countries for receiving payments” and listed the countries below. Similarly for currencies.
2. I described each country in detail like “You can make payments to Honduras, but you can’t receive payments from Honduras”. The same goes for currencies.
3. For each country, I explicitly stated whether you can receive or send payments, and in which currencies.

For queries like “Can I receive a payment from Honduras in Yuan?”, the third prompt option works more or less, but not in 100% of cases. With the other approaches, the answer seems to be random.
Is there another way to teach the model to match two lists?

[Macha]

Welcome to the community!

Are you using ChatGPT or the API?

This sounds like something you would make a vector database for, or an organized key:value pair list in a document you feed it.

Personally though, if it were up to me, I would not use an LLM for this task. This sounds like it would be better suited as a data structure, and then GPT would simply handle turning the values of that data into natural language.

[sergei_shk]

I’m using the API.
Your solution seems sensible. But I also need the model to handle questions like “Can I send a payment to Honduras?” and “Can I send a payment in yen?”. In other words, it should be able to work with lists individually as well.

[Macha]

Are you sure that’s possible with your setup?

From what you previously described, there’s a distinguishing factor between sending and receiving payments. If I understand correctly, that means both endpoints of the exchange matters, yes?
So, if you ask “Can I send a payment to Honduras?” the question becomes where is the payment being sent from? How can it make a judgement call without that information?

That’s why I suggest some form of an organized database/dictionary; these questions are always going to be bi-directional (according to the setup you described).

This means that even if you ask that question, you would still require knowledge of the sender’s origin in prior contextual information. So, it would still need to match those data points together to achieve an effective result.

[sergei_shk]

Okay, let me clear things up for you, I actually have four lists:

1. A list of countries that support inbound payments
2. A list of countries that support outbound payments
3. A list of currencies that support inbound payments
4. A list of currencies that support outbound payments

The aim is that GPT-4 can answer all sorts of questions such as:

• Can I send money to Honduras
• Can I receive money from Honduras
• Can I send money in Swiss francs
• Can I receive money in Swiss francs
• Can I send money to Honduras in Swiss francs
• Can I receive money from Honduras in Swiss francs

[Macha]

Correct, and what I’m saying is this:

Let’s keep Honduras as the example.

You have an entire list of inbound and outbound payment countries. As separate lists.

The problem here is that you aren’t specifying where the sender/receiver is located. The data is there, because it’s in your lists, but you’re not asking it in a way that would give you relevant answers. Either that, or you don’t realize the complexity of your problem.

It would be easier to make a whiteboard example tbh, but alas, this is a forum. Let me try to show it for you here:

1. Can I send money to Honduras?

? → Honduras

2. Can I receive money from Honduras?

? ← Honduras

numbers 3. & 4. have the same structure as 1 & 2

5. Can I send money to Honduras in Swiss francs?

? → Honduras:CHF

6. Can I receive money from Honduras in Swiss francs?

? ← Honduras:CHF

The ? is the variable in your problem. You cannot ask these questions until you resolve this variable. This is not an AI problem: this is a logic problem.

Where the sender/receiver is located matters because that’s what defines the answer as being yes or no. Pick two random origin countries and you’ll see what I mean:

United States → Honduras //

Iceland → Honduras //

*This is an example for logic flow not factual accuracy, idk currency exchange laws

Again, computer algorithms/ a quick script would make this a breeze, but trying to make an LLM do this is not only difficult, it turns into unnecessary effort because of the many easier and simpler solutions already out there. It would be easier for you to prompt GPT to build the script to solve this problem than forcing it to solve it independently.

GPT can handle all sorts of questions, if you provide all of the relevant data. Currently, your queries don’t provide enough information. If you genuinely asked me those questions as a human, the first thing I would ask you is “where do you live?”.

Does this help?

[sergei_shk]

Thank you for the thorough explanation. However, in my task, it really doesn’t matter where the user lives. From the product’s perspective, they can send or receive money anywhere in the world (because the funds always end up in England, where the company operates). What’s important is just the country and currency of the person who is sending money to the user or receiving money from the user.
So, in your examples, you can think of “?” as England.

[Macha]

From the product’s perspective, they can send or receive money anywhere in the world

Doesn’t this answer your own questions? What are you trying to solve here?
Based off of this logic, the answer is going to be yes no matter what then.
You would have an easier time making a ‘blacklist’ of disallowed countries either outgoing or incoming than the former.

I get the problem may lie with currency stuff, but if the currency is the variable that defines an allowed or disallowed action, then there is no reason to ask any of the non-currency questions, nor the independent questions, because the answer will always be yes according to your statement until both the country and currency are defined.

Consider this logic flow:

1. Which direction is the vector variable - incoming(<-) or outgoing(->)?
2. What is the country we care about?

After this is defined, ask:

3. Is vector to country in blacklist?

If it is not, then:

4. What is the currency that is being exchanged?

After this is defined, ask:

5. Can currency be exchanged in country going in vector direction?

To answer this, a data structure, like a dictionary, holds these values like so:

6. Whitelisted vector pairs - country:currency

Whereby incoming and outgoing vectors are each their own dictionary. You could modify it slightly, there are different ways to do this. This is just an example.

Now, this is all well and good, but you need to understand some things here.

I’m unsure if you understand the sheer scope that makes this “product” near impossible to construct without getting yourself or your users in severe legal trouble without extensive, thorough research into each and every country’s financial regulations that you are trying to support. Again, this is fine for a little project or otherwise, but if you’re trying to make any kind of product or platform here, you are likely going to have exponentially more problems than this.

However, in my task, it really doesn’t matter where the user lives.

If this is for a legitimate product, then yes, it absolutely does. If you don’t believe so, then you do not have enough knowledge to build this product, because you will put your users in serious risk. This is why cryptocurrency exists; it bypasses the complexity of the exchange problem.

What’s important is just the country and currency of the person who is sending money to the user or receiving money from the user.

Not ‘or’, ‘and’.

My previous response wasn’t specifically talking about provider/platform ↔ user, it was user ↔ user.

If the answer is always going to be England ↔ country:currency, and if you can send or receive money anywhere, then you don’t have a problem.

If the entire reason you asked this question to begin with was to create a chat-like interface for the users of your platform, then it does matter where the user lives, and GPT cannot make an assumption of where the user lives, and the logic from my previous response still applies.

The user still needs to tell GPT where they live in order for it to provide the answer as I previously mentioned. If they don’t, then you don’t need to do any of this, you can just tell them “yes” and move on.

Nothing in this problem is 1-way: money flows between two points regardless. You need to take into account both endpoints, not just one.

[sergei_shk]

It seems to me that you’re getting a bit too bogged down in the nitty-gritty of the task’s organisational aspect. In reality, the product uses Currencycloud for payments. Consequently, Currencycloud has a list of countries and currencies that it supports for both incoming and outgoing transfers. Where the user is located doesn’t really matter, since all transactions go through Currencycloud.
I think I’ve got the hang of your idea about using a dictionary instead of trying to get the LLM to process lists on its own. I’ll give it a go, thanks

[Macha]

Give it a go and see how it works for you.

I will say again, if the user’s location doesn’t matter, then there would be nothing for the LLM to do other than respond “yes”. The LLM is not a substitute for carefully constructed programs. The task is an organizational problem, that’s why I spent so much time on the organizational part of this.

The point I was trying to get at with all of this was that if you wanted the LLM to handle the logic, the LLM needs the user’s location in order to perform the logic you were originally asking for. The context for what you need vs. what the LLM needs are two different things.

[sergei_shk]

Actually, I’m not expecting any magic from LLM. I was just thinking that based on such information:

You can not receive payments from Colombia, but you can send payments to Colombia
You can receive payments in Yen, but you can not send payments in Yen
You can both receive and send payments in Euro

It would be able to answer the question “Can I send a payment to Colombia in Euros” with yes, and the question “Can I send a payment to Colombia in Yen” with no.

[Macha]

What I have been trying to tell you is that the answer will be different based on who ‘you’ is.

Neither a human nor a language model could answer those questions from that information. I don’t know how else to explain it.

Who is “you”? The user? The LLM? This question is why the LLM can’t answer what you’re trying to ask it, and is also the reason I’m clearly not explaining this very well.

You’re assuming too much about what the LLM can process with the information you’re providing, and not providing enough context. Your information you’re providing to the LLM is not enough information for it to answer you. It would not be enough information for me to answer you if you asked me those questions.

[sergei_shk]

Certainly, I’ve spelled everything out in the prompt in detail – the company in question, how to respond to user inquiries, the main features, and so on. The LLM doesn’t have trouble understanding what “You” refers to. It even sometimes gets the answers to the questions I’ve mentioned above right. However, occasionally it gets them wrong, but if you ask it to explain its response, it will quote the relevant sections from the documentation and then apologise and give the correct answer.
So, the LLM is capable of answering such questions correctly. However, the issue lies in how to make it respond more consistently. At the moment, it does so correctly about 70-80% of the time

[Macha]

Can you show the prompt you’re using then?

I was asking you who ‘you’ refers to, because it’s still unclear from my point of view.
These LLMs also cannot achieve 100% accuracy all of the time, to be clear. We can work on increasing it’s accuracy (which is normally achieved by providing greater clarity in the prompt), and potentially finding a workaround to make it can get near 100%, but absolute perfection in this circumstance is unrealistic. This is because you are asking it to perform both complex retrieval logic and representation in natural language form in a way that is accurate.

If you want 100% accuracy, you need to feed it the correct data (as in, directly providing the accurate information pieces without making it sift through a large document that it needs to piece together on its own) and have it turn that data into natural language. So a dictionary or some other data structure is unavoidable.

If it gets things wrong 30% of the time, then it is very much having trouble understanding what to do. The LLM will never say “I don’t know”; it’s going to guess.