RPA - Using openai + codex to for robotic process automation?

I am a newbie developer and interested if anyone would be able to give me a heads up. I am trying to figure out a way to explore using GPT-3 for RPA like UiPath. But something smaller like browse.ai

The way I envision it working is you recording audio and video with a tool similar to loom (custom) and then it would be run through open ai and possibly some other form of algo that can essentially create code using codex to scrape the screen for data extraction from legacy tools or for tools without an api that need data extraction for data analysis.

If anyone can give me an idea to explore that would be awesome. Thank you guys.

2 Likes

I was thinking to do something similar. For web scraping, you can try to set up your prompt so Codex can try to understand the HTML blob and find out where the elements are (in terms of XPath), then you should be able to generate code (like puppeteer and the like) to do what you want.

2 Likes

thanks Simoni. I guess concern is trying to figure out how to capture the video and take that info to codex.

If someone is still interested in working on this problem, please ping me!