RPA - Using openai + codex to for robotic process automation?

General API discussion
1

I am a newbie developer and interested if anyone would be able to give me a heads up. I am trying to figure out a way to explore using GPT-3 for RPA like UiPath. But something smaller like browse.ai

The way I envision it working is you recording audio and video with a tool similar to loom (custom) and then it would be run through open ai and possibly some other form of algo that can essentially create code using codex to scrape the screen for data extraction from legacy tools or for tools without an api that need data extraction for data analysis.

If anyone can give me an idea to explore that would be awesome. Thank you guys.

2 Likes
2

I was thinking to do something similar. For web scraping, you can try to set up your prompt so Codex can try to understand the HTML blob and find out where the elements are (in terms of XPath), then you should be able to generate code (like puppeteer and the like) to do what you want.

2 Likes
3

thanks Simoni. I guess concern is trying to figure out how to capture the video and take that info to codex.

4

If someone is still interested in working on this problem, please ping me!

5

Hi @ori.kabeli we are keen to explore this. How can we collaborate?

6

Hi @cgutierrez.37 dod you manage to move further on this? We are keen to do this to make synthetic monitoring super simple. Any progress that you can share would be super helpful

7

Still working on experimenting with this. Started using https://botcity.dev and getting closer, but no definitive results yet.

1 Like