Configure custom GPT to access and parse website source code

GPT builders
1

I’m Plus user and configure custom GPT for analyzing of some aspects of website source code.

In the GPT configuration is Web Browsing activated. However, I go crazy with further GPT configuration:

On one day I prompt GPT in the left configuration frame Configure yourself for accessing and parsing SSL-secured website source code with Beautiful Soap, Requests and urllib3. After this I prompt GPT in the right frame Visit [url] and output the second H4 heading - and get the data as expected.

On the next day the same prompt in the right frame causes the execution of this code

import requests
from bs4 import BeautifulSoup

# URL to be parsed
url = "http://aSite2Visit.de/index.php/kfz-service"

try:
    # Sending a request to the URL
    response = requests.get(url)

    # Checking if the request was successful
    if response.status_code == 200:
        # Parsing the content of the webpage
        soup = BeautifulSoup(response.content, 'html.parser')

        # Finding all H4 headings
        h4_headings = soup.find_all('h4')
        h4_headings_text = [heading.get_text().strip() for heading in h4_headings]
    else:
        h4_headings_text = ["Failed to retrieve the webpage. Status code: " + str(response.status_code)]
except Exception as e:
    h4_headings_text = [str(e)]

h4_headings_text

which ends up with the error like

["HTTPConnectionPool(host='aSite2Visit.de, port=80): Max retries exceeded with url: /index.php/kfz-service (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ea7810893a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"]

If I prompt GPT to visit an SSL-secured website the executed code looks like

import requests
from bs4 import BeautifulSoup

# URL to be analyzed
url = "https://sslSite2Visit.de/index.php/kfz-service"

# Sending a request to the URL
response = requests.get(url)

# Checking if the request was successful
if response.status_code == 200:
    # Parsing the content of the webpage
    soup = BeautifulSoup(response.content, 'html.parser')
    # Output the parsed content for analysis
    soup.prettify()
else:
    soup = None
    "Failed to retrieve the webpage. Status code: " + str(response.status_code)

ending up with the error

File ~/.local/lib/python3.8/site-packages/requests/sessions.py:703, in Session.send(self, request, **kwargs)
    700 start = preferred_clock()
    702 # Send the request
--> 703 r = adapter.send(request, **kwargs)
    705 # Total elapsed time of the request (approximately)
    706 elapsed = preferred_clock() - start

File ~/.local/lib/python3.8/site-packages/requests/adapters.py:519, in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    515     if isinstance(e.reason, _SSLError):
    516         # This branch is for urllib3 v1.22 and later.
    517         raise SSLError(e, request=request)
--> 519     raise ConnectionError(e, request=request)
    521 except ClosedPoolError as e:
    522     raise ConnectionError(e, request=request)

ConnectionError: HTTPConnectionPool(host='auto-babiel.de', port=80): Max retries exceeded with url: /index.php/kfz-service (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ea78a312ee0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')

I created a setup file with description, how exactly should GPT access websites and uploaded it into the configuration/training area, but it ssems to have no expected stable effect.

My question: how can I configure custom GPT to achieve stable reliable access and parsing of external websites with and without SSL security?

1 Like
2

You might also want to check if the website can provide content with a simple http request. Many a times you would need to use a browser based api such as with playwright as many websites generate dynamic content

1 Like
4

I test this workflow with some urls with static content, they definitively don’t have content rendered by Javascript and provenly deliver a 200 status code. Beside of this, to begin with the workflow, it would be fully enough to access existing source code error free.
As you can see from my citation of the error, the error is not a 404 or any other, related to the general url unavailability. How should I interpret your answer and how should it help me with my issue? What should be the sense of simple http request?