A Useful Clicker for reverse engineering human software in order to use in AGI systems

Useful Clicker is a program for creating xml description and documenting the user activity and also documenting the software. For example, to automatically search for YouTube content and download videos, post tweets, everything that a normal user with a mouse and keyboard can do. The program has a built-in computer vision functionality, i.e. it can determine the buttons on the display by analyzing the pixels in the image. In this sense, it works similar to how the retina of an animal’s eye works. The buttons are highlighted in green in the style of augmented reality. It can also recogize the text of a menu or a button using OCR transformer.

UC supports macro recording with code generation of unique xml code,
input: for which mouse trajectory and keyboard events.
output: tiny xml code

The idea of ​​an astronomical number of screenshots to reverse engineer human software came to me from Marvell comics.
We need to create an astronomical number of screenshots for a variety of programs and get systems. Every dialog in any computer program with GUI interface is unique to the program. So it can be recognized in many contexts in various backgrounds of other windows on the screen. Therefore, the clicker will be geared towards reverse engineering heritage. Soon, according to the idea of ​​Elon Musk, all transport software i.e. the expertise of some engineer will replace neural networks. But we need to prepare a description of the functions of every program on Earth that ever grows. So that the future AGI can use these compiled programs for its calculations. So that he doesn’t have to reinvent the wheel to be rude.
AGI thinks orders of magnitude faster than a person can soft people in such interesting combinations that people have never even dreamed of. But this information must be available. In order not to waste time on reverse engineering. Perhaps, at some stage, AGI can itself observe various software written by people. By disassembling it. To do this, I provided a Function Description window that a person must fill out.

The program must recognize the context. Those. which window is currently on the screen, which buttons are on the screen. All buttons are outlined in green, which must be clicked in red. He also signs the normative action number next to the button. All this can be arranged in a full-screen semi-transparent widget. Buttons “Next Frame” and “Previous Frame” on the top screen. That is, the clicker works in the clicker programming mode. To return the previous window e.g.

Examples of xml description of UsefulClicker

File -> Open readme.txt

Software repository

This idea is described in more detail in the following video Новая идея для кликера. Мне нужно астрономическое количество скриншотов. - YouTube