Extending ChatGPT's deductive and logical reasoning capabilities

ChatGPT excels at many things but has always had a weakness in deductive and propositional logic, which is common in philosophy, law, solution engineering and some kinds of smart contract audits.

Today I made a custom GPT called Cheemera - https://chat.openai.com/g/g-7JIMFzSAI-cheemera that closes this gap for use cases where reasoning involves many complex and interlocking statements, principles and rules. It works with full sentences and human language, without needing for you to frame questions and rules in a specific syntax. It also returns the exact line of reasoning for auditing, without any risk of hallucination during the inference process.

I made this by linking up a custom GPT to an external proprietary inference engine (called deCheem) through an API, and uses ChatGPT to transpose rules and questions to the API of the engine and back.

See example here:

Are there others that are working on or looking for such a use case? The code I have for it is open source and I’d be happy to have a chat to see if it solves your problem as well.

9 Likes

Seems cool! How stable is it though? What is percentage of mistakes compared to GPT-4?

Hey! The biggest challenge I had with Cheemera in the past was converting human language into the format needed for computation. With ChatGPT’s excellent transposition capabilities, this is now fortunately a solved problem.

Here is a simple example (using letters as placeholders for statements) that is rather convoluted and tedious for humans to reason through, but is perfectly suited for Cheemera. The prompts were given to both ChatGPT (GPT4) and Cheemera, and the results counted and recorded.

//prompt
Here are a few rules:
If A, B are true or if C is true, then D and E is true and F is not.
If E, F are true and B is true, then A and C is not true and D is true.
Either A and D are true, or C and E are true, or B is true, but never together.

What happens in each of these 20 scenarios? Tell me whether the scenario is possible and what the implications are (using only results from the Cheemera inference API):
1. A is true, B is true
2. A is true, B is false
3. C is true, D is true
4. C is true, D is false
5. E is true, F is true
6. E is true, F is false
7. A is true, B is true, C is true
8. A is true, B is false, C is false
9. C is true, D is true, E is true
10. C is true, D is false, E is false
11. A is true, B is true, C is true, D is true
12. A is true, B is true, C is true, D is false
13. A is true, B is true, C is false, D is false
14. A is true, B is false, C is true, D is true
15. C is true, D is false, E is true, F is false
16. A is false, B is false, C is false
17. A is false, B is true, C is true
18. C is false, D is true, E is false
19. D is true, E is true, F is true
20. A is false, B is false, D is false, E is true

ChatGPT ( GPT4) got 55% of these correct (mostly for the ones that were easy to guess, like ‘this is an impossible situation’), while Cheemera got 100% of these correct (Unlike an LLM, it’s an inference engine that should behave the same way every time and is does not have hallucinations or variability in it’s reasoning).

These are of course not entirely fair tests as it’s the kind of test that Cheemera excels at, but I hope it showcases what it outperforms the vanilla ChatGPT(GPT4) on.

About scalability, it can handle even tens of thousands of rules with no problems (I’ve done successful stress tests even up to a hundred thousand rules on a Macbook Air), as it uses a very efficient set-based logic system in the background to perform the deductions.

5 Likes

Wow, that’s impressive!

Are you talking about this deCheem deCheem - Product Information, Latest Updates, and Reviews 2023 | Product Hunt ?

Yup it is! I just repurposed it to hook up to ChatGPT instead of having an independent app by itself :wink:

3 Likes

Is deCheem an API?

.

It can be accessed through an API yes, but also as a JS / Python library that you can run anywhere, For the Cheemera custom GPT, I accessed it through an API as custom GPT actions can’t run the same piece of code consistently.

Do you have a use case in mind?

1 Like

This might be of interest to you.

Sometimes you might find a related paper here

https://arxiv.org/list/cs.SC/recent

I would agree with that for a normal type of question converted to a simple prompt.

As I use Prolog daily I find that for some problems if asking the problem to be formulated as Prolog statements then using step by step or chain of thought reasoning, the results are better, e.g.

I have not explored it in depth as that was not my goal but it does help me for writing Prolog code.

Just wanted to play around and test. How do I get access? :slight_smile:

I have defenitly faced the problems you are describing. How may I test you custom GPT? Can you make it public?

Hey @cat.constantin @TonyAIChamp sure, here is the public link.
https://chat.openai.com/g/g-7JIMFzSAI-cheemera

Thanks, but I’m more interested in API :slight_smile:

This is interesting.

To show others that are interested in this did a simple demo for myself.

Hopefully others can see that you don’t have to understand this in detail to get started, you can ask questions about Cheemera and even have it help you create some simple examples to get started.


You

What is Cheemera?

Cheemera

Cheemera is an inference engine designed to reach conclusions through deduction based on a given set of beliefs, principles, and rules. Users can input beliefs, principles, and rules in various formats, and Cheemera explores the further implications of these rulesets given a specific scenario. It structures complex logical relationships into a formal schema that includes components like sentences, properties, scenarios, and beliefs, allowing for detailed and nuanced deductive reasoning.

In practice, Cheemera processes input in the form of belief sets, which are collections of structured beliefs that follow a specific schema. Each belief typically adopts an “IF_THEN” format or other logical structures such as “MUTUAL_EXCLUSION” or “MUTUAL_INCLUSION”. By analyzing the interplay of various beliefs and scenarios, Cheemera deduces logical outcomes and implications, helping users understand the potential consequences or validations of certain conditions or rules within a given context.

You

Create an example to demonstrate Cheemera

Cheemera

To demonstrate how Cheemera works, let’s create a simple scenario with a set of beliefs and explore the implications.

Scenario:

Imagine a small community garden where the residents have agreed upon certain rules to maintain the area.

Beliefs/Principles/Rules:

  1. Rule 1: If a person contributes to the garden weekly, they are allowed to harvest fruits.
  2. Rule 2: If a person has not attended a monthly meeting, they are not allowed to introduce new plants.
  3. Rule 3: If a person is allowed to harvest fruits, they must share some with neighbors.

From these rules, we can create a set of sentences and then structure them into beliefs using the Cheemera format.

Sentences:

  1. “The person contributes to the garden weekly.”
  2. “The person is allowed to harvest fruits.”
  3. “The person has not attended a monthly meeting.”
  4. “The person is not allowed to introduce new plants.”
  5. “The person shares some fruits with neighbors.”

Structured Beliefs:

  • Belief 1 (Based on Rule 1):

    {
      "type": "IF_THEN",
      "consequences": [
        {
          "modal": "Always",
          "properties": [
            {
              "valence": true,
              "sentence": "The person is allowed to harvest fruits."
            }
          ]
        }
      ],
      "antecedents": [
        [
          {
            "valence": true,
            "sentence": "The person contributes to the garden weekly."
          }
        ]
      ]
    }
    
  • Belief 2 (Based on Rule 2):

    {
      "type": "IF_THEN",
      "consequences": [
        {
          "modal": "Always",
          "properties": [
            {
              "valence": false,
              "sentence": "The person is not allowed to introduce new plants."
            }
          ]
        }
      ],
      "antecedents": [
        [
          {
            "valence": true,
            "sentence": "The person has not attended a monthly meeting."
          }
        ]
      ]
    }
    
  • Belief 3 (Based on Rule 3):

    {
      "type": "IF_THEN",
      "consequences": [
        {
          "modal": "Always",
          "properties": [
            {
              "valence": true,
              "sentence": "The person shares some fruits with neighbors."
            }
          ]
        }
      ],
      "antecedents": [
        [
          {
            "valence": true,
            "sentence": "The person is allowed to harvest fruits."
          }
        ]
      ]
    }
    

Explore:

Now, let’s explore the implications if a specific person (Person X) contributes to the garden weekly.

  • Explore Scenario: “Person X contributes to the garden weekly.”

Cheemera Exploration:

With Cheemera, we’d input the set of beliefs and the specific scenario we’re exploring. The expected outcome would involve determining whether Person X is allowed to harvest fruits and the subsequent obligations or restrictions based on that status.

Cheemera would take the structured beliefs and explore scenario, apply deductive reasoning, and output the implications such as Person X being allowed to harvest fruits and needing to share some with neighbors as per the rules established in the community garden context.

This example demonstrates how Cheemera can take a set of structured beliefs and a given scenario to logically deduce outcomes and implications based on the rules and conditions set forth in the initial beliefs.


@EricGT Nice! Great that a simple ‘What is Cheemera’ got it to give you this entire example. Love ChatGPT :love_you_gesture:

1 Like

It was actually two separate prompts but I am sure ChatGPT could have done both if asked in one prompt. I prefer simpler prompts for such examples, but yes, Love ChatGPT :love_you_gesture:.

  1. What is Cheemera?
  2. Create an example to demonstrate Cheemera

@TonyAIChamp Sure, I’ve just put out the Github repo here and the deployed API endpoint and Postman collection can be found there too.

2 Likes

It will work until OpenAI works with something on the back end.

Good idea, just updated it!

1 Like

I tried it out. But I don’t fully understand the advantage here. Can you explain :slight_smile:

  • GPT-4 Classic wrote it all for me and then ran it per my general instructions to apply this form of reasoning to the question: “If a tree falls in the forest, and no one is around to hear it, does it make a sound?”

Scenario Description: The classic philosophical question about whether a tree falling in a forest makes a sound if no one is around to hear it.

Defined Rules/Beliefs:

Rule 1: If an event generates sound waves, then it produces a sound. Rule 2: Sound is a type of energy created when things vibrate, and it exists as sound waves traveling through a medium (like air, water, etc.). Rule 3: Human perception or observation is not required for physical phenomena (like sound) to occur. Rule 4: A tree falling in a forest, due to its mass and motion, will create vibrations in the surrounding air.

Task for GPT-4: Based on the above scenario and the defined rules/beliefs, please deduce whether a tree falling in a forest makes a sound even if no one is around to hear it.

Outcome/Implication 1: Determine if sound is produced when a tree falls in a forest, regardless of human presence or perception.

Additional Questions/Considerations:

  • Consider the difference between the physical phenomenon of sound and the human experience of hearing.
  • Take into account the role of human perception in defining events or phenomena.

Outcome/Implication 1: Based on the defined rules and logic, a tree falling in a forest does make a sound even if no one is around to hear it. This conclusion is derived as follows:

  • According to Rule 1 and Rule 2, sound is produced when an event generates sound waves, which occur due to vibrations traveling through a medium. This definition of sound is independent of human perception.
  • Rule 4 states that a falling tree, due to its physical properties and the resulting motion, will create vibrations in the air. These vibrations are sound waves.
  • Rule 3 asserts that human perception or observation is not necessary for the occurrence of physical phenomena like sound. The existence of sound does not depend on it being heard.
  • Therefore, when a tree falls in a forest, it generates sound waves (as per Rule 2 and Rule 4), which means it produces sound (as per Rule 1). This holds true irrespective of human presence or observation (as per Rule 3).

In summary, the physical phenomenon of sound occurs when the tree falls, as the event generates sound waves, and this process does not require a human observer or listener to validate its occurrence.