As I have been working my way through editing Marc Strassman’s fascinating collection of GPT-3 generated essays, I have observed a pattern in completions that I think most of us usually interpret as “changing the topic” or “losing the thread”. A lot of the time, I can “save” them as an editor (make them make sense) simply by applying a ‘block quote’ or ‘pull quote’ style so that they are displayed in the printed book that way. This makes me wonder whether some of the times that we think GPT-3 is “changing the topic” or “losing the thread” are actually driven by cases where, e.g. the language model correctly observes that first-person pull quotes (as in a magazine article) are often seen in close proximity to narrative third person … so the language model is actually generating valid predictions, but the API does not have a way of understanding that the language model “means” for certain typography or layout conventions to be applied to a given chunk of text.
So my question is, what has been done as far as semantic processing of layout & fonts during training – by default, NLP often strips out a lot of meaning in bold, italics etc - is the same true for GPT-3? And has any thought been giving to enabling GPT-3 to communicate in terms of layout & typography?
Hi there, that’s correct that you can’t use things like bold and italics when using the API, but you can convey meaning in different ways, such as by explicitly referring to topics as “important” or providing multiple examples of your important information.
I’ve had good results by adding html tags into prompts as a proxy for overarching layout/page structure. For example, here’s a snippet of a response that shows GPT-3 understands in-page hierarchy well:
... <h2>Opening One Home-Buying Account? So Do Two Other Things to Brighten Potential Buyers' Credit Histories</h2><ul><br> <strong>1. Closing</strong> at least one home-buying account before applying for a new mortgage</p><br> <p>There are some reports out there about closing a traditional mortgage when applying for another type of mortgage, such as a refinance,<a target="_blank" href="https://www.credit.com/creditcheck/how-to-close-a-mortgage-look-before-lending-credit-cards"><em> ways to close</em></a> could hurt your credit score. Credit pundits say it's always best to open as few new accounts as possible—just to keep your traditional score healthy. You never know when you'll want to take out another mortgage, so it's also a good idea to keep at least one home-buying account open at all times.<br><br></p><p><strong>2. Opening</strong> a property- and long-term care insurance policy within the same year</p><br> <p>When ...
(Just as a note—I’m not publishing these. This response was part of a test for the Cushman alpha several months ago.)