I have had an idea for a while where I would like to take documents (webpages, documentation, ebooks) which have been dumped into plaintext and then re-present their contents such that small, essential units (sentences, titles, a mathematical expression) are displayed on screen one at a time.
Although there are a few different ways of approaching this, my ideal right now is trying to find an NLP method to quickly, easily classify what different lines in a text file are. It should be able to easily detect: this is a sentence - this is a subchapter heading - this is a line of code - this is a blockquote - etc. There really shouldn’t be too many different categories, it might be as simple as sentence, header, line of code, for example.
The classification isn’t actually essential - it could be enough just to segment the text, actually. Segmentation is already a very common functionality. It could just be viewed as GPT-3 doing it automatically, vs. a rule-based approach. I believe with a neural network, you just have to provide it with classes to allocate to any input. So with segmentation, right now all I can think of is sending indices of all the characters in the text along with the text, and the class being “end segment here / not end segment here”. Not sure where GPT fits in here, i.e. if we have the ability to train GPT ourselves, on new data?
But I can see the classification coming in handy, for example, if GPT-3 knows a heading is a heading, it can build an automatic table of contents later on, in some software interface, for example. If it knows code is code, it knows not to automatically join two consecutive lines, whereas with natural language it should, if the sentence is broken over two lines.
So, could anyone recommend a way to get GPT to mark up a text file with 3 tags - maybe I’ll use some standard format like XML or HTML: header, text/paragraph, and code or maybe “div” or something.
Or is there a better tool for this? For example, should I train GPT or a neural network on HTML for example, so it learns what kind of language goes in what kinds of HTML tags, then ask it to predict the HTML tag for any given segment of text?
Thank you very much.