Saw this today, but I haven’t played with it yet. Looks kinda cool…
5 Likes
Nah I am not going down that rabbithole haha.. looks cool though
1 Like
I mean it looks really easy to use grrrr
1 Like
The implementation looks good @PaulBellow!
One key improvement could be to tailor (or maybe even skip) the RecursiveCharacterTextSplitter function. While I’m not sure how many documents you intend to chunk, or what’s in those documents (both in terms of content and structure), I have observed that large gains can be obtained by chunking more mindfully, as opposed to mechanically. For e.g., in some legal documents I was trying to make graphs for, the chunks were quite awkardly constructed at times, as the variance of clause length can be quite high across different documents. As a result, some chunks were missing crucial parts of certain clauses, thereby causing issues in downstream applications that were hard to diagnose at first.
Access to subject matter experts can be limited, and so one alterative is to use an LLM (either via API or an application like ChayGPT) to help create these meaningful chunks. This is hard to scale across hundreds of ever changing documents, but I think the effort is definitely worth it.
My guess is that changing the model will affect the graph more than the temperature itself (and it’s perhaps better to have a low temperature value for this exercise), but would be curious to see some results if you experiment more with this. Thanks! 
1 Like
It’s all very basic so far, but thanks!
I’m doing prose (novels and outlines)… thinking of firing up a modern Fiction Factory then writing about it… which has a history! Back at the turn of the 20th century, a gentleman quit lawyering and started pumping out “pulp fiction” at “astounding rates” (for the time).
AUT FICTION, AUT NULLUS.
“Well, my dear,” said John Milton Edwards, miserably uncertain and turning to appeal to his wife, “which shall it be—to write or not to write?”
“To write,” was the answer, promptly and boldly, “to do nothing else but write.”
John Milton wanted her to say that, and yet he did not. Her conviction, orally expressed, had all the ring of true metal; yet her husband, reflecting his own inner perplexities, heard a false note suggesting the base alloy of uncertainty.
“Hadn’t we better think it over?” he quibbled.
“You’ve been thinking it over for two years, John, and this month is the first time your returns from your writing have ever been more than your salary at the office. If you can be so successful when you are obliged to work nights and Sundays—and most of the time with your wits befogged by office routine—what could you not do if you spent ALL your time in your Fiction Factory?”
“It may be,” ventured John Milton, “that I could do better work, snatching a few precious moments from those everlasting pay-rolls, than by giving all my time and attention to my private Factory.”
“Is that logical?” inquired Mrs. John Milton.
“I don’t know, my dear, whether it’s logical or not. We’re dealing with a psychological mystery that has never been broken to harness. Suppose I have the whole day before me and sit down at my typewriter to write a story. Well and good. But getting squared away with a fresh sheet over the platen isn’t the whole of it. The Happy Idea must be evolved. What if the Happy Idea does not come when I am ready for it? Happy Ideas, you know, have a disagreeable habit of hiding out. There’s no hard and fast rule, that I am aware, for capturing a Happy Idea at just the moment it may be most in demand. There’s lightning in a change of work, the sort of lightning that clears the air with a tonic of inspiration. When I’m paymastering the hardest I seem to be almost swamped with ideas for the story mill. Query: Will the mill grind out as good a grist if it grinds continuously? If I were sure—”
“It stands to reason,” Mrs. Edwards maintained stoutly, “that if you can make $125 a month running the mill nights and Sundays, you ought to be able to make a good deal more than that with all the week days added.”
“Provided,” John Milton qualified, “my fountain of inspiration will flow as freely when there is nothing to hinder it as it does now when I have it turned off for twelve hours out of the twenty-four.”
“Why shouldn’t it?”
“I don’t know, my dear,” John Milton admitted, “unless it transpires that my inspiration isn’t strong enough to be drawn on steadily.”
“Fudge,” exclaimed Mrs. Edwards.
“And then,” her husband proceeded, “let us consider another phase of the question. The demand may fall off. The chances are that it WILL fall off the moment the gods become aware of the fact that I am depending on the demand for our bread and butter. Whenever a thing becomes absolutely essential to you, Fate immediately obliterates every trail that leads to it, and you go wandering desperately back and forth, getting more and more discouraged until—”
“Until you drop in your tracks,” broke in Mrs. Edwards, “and give up—a quitter.”
“Quitter” is a mean word. There’s something about it that jostles you, and treads on your toes.
“I don’t think I’d prove a quitter,” said John Milton, “even if I did get lost in a labyrinth of hard luck. It’s the idea of losing you along with me that hurts.”
“I’ll risk that.”
Plotto is another cool rabbit-hole. A written book with formulas for plots… the “algebra of literature” I think they called it? I may have fine-tuned on it at one time? Or not! 
Equal parts reference guide and historical oddity, Plotto is sure to amaze and delight writers for another hundred years. (Written in 1920…)
In the mid 20th century another author tried to go all out with modern tech (at the time). Yes, none other than the Perry Mason author. He used dictation (tape recorders) then had others type them out. He came out At his height, he was churning a novel a week, I believe. You can read about his story here.
And there’s also the Best Seller Code .. back in 2016 they used “maths” to get a blueprint of a best seller. Really good stuff. I’d have to “borrow” a lot of best-sellers to replicate, but I don’t imagine the pattern has changed too much since then (nay, since we started telling stories to each other! haha…)
There was also the Book Lamp project… a “genome” for books… Apple bought that and buried it or are using it somehow? I dunno. Good rabbit-hole, though!
I’d used NLP before tho not much with AI yet, but I’m having fun and learning a lot.
I’ve written millions of words at this point over 40 years, so lots to work with.
Don’t think I could afford AI for all of them! (Daily free tokens are nice!)
Maybe I should write an early 21st century Fiction Factory story?
I appreciate the helpful pointers and rah-rah!
What are you tinkering on these days?
1 Like
With a million words written over the past 40 years, I think you’re ahead of most people out there, and sitting on a gold mine of high-quality expert authored text that folks would pay good money to get their hands on to train on and/or fine tune models with! What are ways you are/intend to harness this great corpora?
I am tinkering with really anything and everything I can get my hands on, but mostly in the enterprise workplace related documents and not so much fiction/non-fiction, even though I would really like to do so!
And yes, if only tokens were free…
2 Likes
There is a program for allowing free tokens if you are eligible.
https://help.openai.com/en/articles/10306912-sharing-feedback-evaluation-and-fine-tuning-data-and-api-inputs-and-outputs-with-openai
2 Likes
This is great, thank you for sharing @aprendendo.next!
3 Likes
There is another project that I am working on right now. It is a Youtube Transcript Dashboard and Summarizer (InfoCaptor AI) . Sharing few screenshots to give an idea. It uses openai for basics. It allows you to tag and categorize all the videos that you have watched. For e.g let say you are watching home gardening, or say startup related videos. It tags, summarizes and the best thing it automatically generates a word cloud using the top keywords.
2 Likes
hah that’s beautiful… that’s where I started… A spiral… Where I lived and moved outwards I started walking…
I didn’t start with code though ^^
1 Like
Small demo update… Was hoping to implement MCP for the end of the month connecting to ChatGPT too but unsupported on my subscription/country still 
I don’t like to use other people’s libraries… All code here is my own from the GIF building to the Open API socket connections written in pure PHP.
A quick question…
The reference ids here are tied to my account right? Anything wrong with publishing any data from the responses (apart from the obvious in my own data)?
Side note: The forum Image AI does not read GIFs properly ^^… ‘The image is entirely blank with a solid white background.’ (Actually just the first frame )
GIFs are an awesome format that can hold other data too… Considering AI hints for process to store inside the GIFs so I can share not only the visual process but also some hints for reproduction of process for AI…
I don’t believe the forum recreates GIFs (Yet to test) but I like the idea of sharing animated images where process can also be recreated by AI.
(Open image in new tab to view full screen 4k animation )
Small Edit: Sorry for anyone on an iPhone… They seem not to open 4k GIFs 
… Noted for next VU
2 Likes
This is a bit of what I was building.
I told a lot of you that we’ll see come June, if there was any substance in my chosen arena.
Well, June’s about done:
My First Gen Future-Sense Ai project
had a patent lawyer contact me bout discussing this.
but not a word from anyone else…
I’m sorry if it’s scary… i think it’s pretty groundbreaking…
I didn’t let all my distilled infos out of the bag, just enough from 1 dated post…
i can’t stress the sorts of dangers one can find working in this sort of field, but it’s no joke.
Fun Random Person Generator
Small updates… Started Building GIFs 1440x900 so they will fit in 1536x1024 image recognition. Hopefully will now display on iPhone too
Father/Son collaboration
3D Long/Lat Globe built to spec including zoom out on spin for better GIF compression and using NASA map.
3 Likes
The last GIF worked but it wasn’t great… Just within the 4Mb limit and I had the Globe skipping lots of frames…
Here’s a slightly cleaner GIF, still within the 4Mb forum limit.
So I have a question…
Web 3.0… Is this a version of the web where Humans and AIs access information online? A kind of shared space?
Well GIFs seem to me to be a great format for sharing data too… More than data… Context…
So here are some ideas I have had, they are non standard I guess but this is a hackathon thread… Kinda fishing for ideas
I thought about adding code or pseudocode to the data sections on the GIF, maybe Phasm prompts that describe the creation of the process being displayed…
Maybe JSONRPC / MCP context message or even full code.
Maybe other juicy information like sources links…
Even ‘Watermarks’ where maybe you could encode the thread ID of a forum post inside the GIF for context/attribution ie
I think maybe all of the above and more…
I think there is real context here for sharing processes that AI can understand in a fun way online… (Maybe a headache for forum moderators 
)
2 Likes