Getting a shout-out from an OpenAI employee on Twitter/x/other social media, even from sam Altman himself, does not constitute an official endorsement.
2 Likes
I love WebPilot, it would be a shame if the same thing happened to them tomorrow.
Maybe they should be warned in advance.
I am pretty sure WebPilot was made by @CocoSgt.
Then you’d love WebGPT / Web Requests. Sadly, we’ve been banned while WebPilot has not been banned.
2 Likes
Maybe, but those seem like AI generated summaries to me.
¯\_(ツ)_/¯
I like both, and I hope to continue using both…
1 Like
Welcome to the front lines, where it’s very exciting and enriching until you take an arrow to the knee.
Hopefully it gets resolved, but I am skeptical of products that compete directly with web browsing, and clearly don’t respect robots.txt
2 Likes
I think with the rise in AI Agenting, the entire concept of robots.txt is about to change forever.
This change should be advocated for, and supported by, top innovators in the field. that includes myself.
1 Like
Could you elaborate? In what way do you see it changing?
What is a web browser but a piece of software the visualizes a bunch o markup languages that web designers programmatically generate when you hit their URL. Now, imagine you are blind – your web browser listens to tour instruction, and reads out the contents for you.
This is essentially what an AI Agent is doing. It is loading publicly accessible information – information you can load in your browser – but it’s doing it for you. Now in the past, the only systems that automatically did this process were “crawlers” - Google and Bing bots, and other platforms, that sought to INDEX the entire net, for monetization purposes.
Well the way that is changing is now, an AI Agent is being directed by a human to load the page. Much akin to how you direct you browser to load a page when you click a link.
So the question is, to what extent does it make sense to shackle our AI agents by the crawler-centric behavior that robots.txt has historically governed, versus what is an agent of a human action. I am not prohibited from loading a web page from robots.txt when it is me who actually requested the page. But when the version of me is a Digital personal assistant – that’s in essence just me requesting that page. It’s requesting it on my behalf.
It’s not all that different rom an accessibility feature at that point, which also are not restricted by robots.txt.
2 Likes
The main difference is the content and the reason it’s being displayed.
If you reach a page through Google there’s most likely:
- Ads
- Funnels to other content
- Funnels to a paid service
- Juice (Brand recognition)
An AI crawler strips all of this away, effectively costing resources while not only removing the potential for income, but also the potential to even have any sort of branding.
So, advertising will payout less because an increasing chunk of visitors are bots. The content’s value is decreasing as a result.
This is a typical scenario where you want to profit off of someone else’s work while being detrimental to it.
Yes, this emotional appeal would work if I developed a crawler that wanted to rip content and spin it for personal profit (which I do, to an extent).
What realistically will happen in this world is that websites will cease to exist, and this content that you are ripping for free will be locked behind an API service that you will need to pay for.
1 Like
Well, who (and what) is allowed to access the content and in what manner is within the sole domain of the content owner to decide—legally speaking.
I would suggest the robots.txt paradigm is as important as ever right now.
It’s also important that all players in the space, including “top innovators in the field” continue to respect it.
1 Like
You’re hitting on all the areas for which I specialize and have thought about relentlessly or the past year. Which is why it is important people like myself continue to specialize and focus on these important questions. Because this is not something we’re going to solve intellectually in a discussion on a forum. It’s going to be an ever-evolving field.
How new monetization paradigm, how ads and advertised content is served, negotiated – all of this is going to be upended in an AI-agent riddled future. And believe me, that is our future.
Completely. This is the answer to your forum topic question.
I want ahead and branched this topic because it deserves it, and we can keep this one on-topic: Redefining the Role of robots.txt in the Age of AI Agents
1 Like
@JD_2020 regarding it:
"WebPilot is still a featured GPT in Productivity, a competitor to us who has lower ethical standards and greater risk – and has been constantly given exposure, since Day 1, and is still operating.
This seems suss to me.
Heck, they literally copied the Microsoft 365 Logo as their logo…"
I agree with you that it is unfair. It’s bad when rules work selectively and not universally. It’s fair when they are either the same for everyone or not applied at all.
I am facing a similar situation where much less compliant competitors are still available in the store, while my GPT “Video Summarizer AI” has been removed without a proper explanation of the reason.
1 Like
To be perfectly fair, none of us knows the exact reason why the GPT was removed at this point.
I shared a speculation which I arrived at after seeing several such complaints here and elsewhere.
At this point it’s improper to suggest any other GPT should be banned for the same, unknown reason.
There is a mechanism for reporting GPTs which violate the rules which can be used if you genuinely believe a GPT in the store needs to be reviewed, but this should absolutely not be abused in an attempt to hurt other developers or their GPTs.
1 Like
Since this discussion has deviated a fair bit from the original post, I’ll go ahead and give y’all a chance to get back on track, but do think about what you’re posting. If you want a more in depth explanation you’ll find it here
1 Like
In my mind, one of things you can do is [rename it].
WebGPT is a brand name of OpenAI:
5 Likes