Hey everyone! 
I’m excited to introduce SemanticText, a framework that combines the power of generative AI with practical tooling ideas for language models (LLMs). Built in Smalltalk, SemanticText also is a proof-of-concept for tightly integrating LLMs into a runtime system, opening up new possibilities for program exploration, debugging, and building smarter tools.
What is SemanticText?
SemanticText offers different capabilities to integrate generative AI into your apps and your development workflow:
1. LLM Framework + OpenAI API Client
The framework provides access to OpenAI’s models such as GPT-4o and convenient libraries for incorporating these capabilities into your runtime system. This includes:
- Custom Agents: Build conversational or autonomous agents with function calling, model hyperparameters, and streaming responses. Also offers a reusable chat UI component.
- Semantic Search: Create in-memory or file-based vector databases powered by OpenAI text embeddings and build RAG applications on top of them.
- Voice Automaton: Enable oral conversation with agents by leveraging OpenAI’s audio models for speech transcription and synthesis.
Example – Hello world RAG assistant in 7 lines:
SemanticConversation new
addSystemMessage: 'You are an assistant that answers questions about the SemanticText package.';
addFunction: #searchDocumentation argsAction: [:searchTerm |
(SemanticText packageInfo methods collect: #source) asSemanticCorpus
findObjects: 10 similarToQuery: searchTerm];
addUserMessage: 'how can i change the temperature of a conversation?';
getFinalAssistantReply
Output:
To change the temperature of a conversation in the SemanticText package, you can adjust the configuration by setting a different temperature value. This can be done as shown in the example:
SemanticConversation new withConfigDo: [:config | config temperature: 1.9. "Set desired temperature here" config maxTokens: 200]; "Optionally adjust other settings" addUserMessage: 'What is a computer?'; getAssistantReplyHigher temperatures produce more random outputs, while lower temperatures result in more deterministic outputs.
2. Tools for Prototyping, Debugging, and Building LLM Applications
SemanticText includes several built-in tools to simplify the development and debugging of LLM-based applications:
- Prompt Prototyping: Experiment with prompts and build agents in an interactive playground with direct access to your LLM functions and generate code for them. You can also mock the behavior of agents and functions here for testing.
- Debugging Utilities: Debug LLM interactions, trace function calls, and experiment with agent behavior directly in your running system.
- Cost Tracking and Estimation: Track API usage and estimate costs for LLM invocations to manage expenses when using services like OpenAI.
3. Programming Tools and Integrations Based on AI
SemanticText integrates tightly with the Squeak/Smalltalk environment to augment exploration, debugging, and prototyping tasks:
- Editor Integration: Explain selected code snippets, generate or edit methods, or summarize the responsibilities of a class within the editor.
- Mailing List Integration: Chat with over 20 years of discussions and patches on the Squeak developer’s mailing list, search for related discussions, and make sense of long mailing list threads using AI-generated insights.
- Help Browser Integration: Chat with Squeak’s built-in documentation, perform semantic searches for books and tutorials, and easily find similar help articles.
4. Natural Language Debugging
With SemanticText, you can talk directly to runtime objects in natural language. For example, when you see an Order object in a debugger, you can directly enter “When was this order created?” instead of manually interacting with the debugging console.
Internally, an exploratory programming agent will autonomously generate a plan, research the implementation and documentation of classes, and write and execute code expressions until it can answer your question.
Alternatively, you can also use a language extension for semantic messaging to intermix traditional scripting with LLM invocations. These pseudo-code like method calls are handled by the exploratory programming agents and will return regular code objects:
anOrder ? #creationDate. "--> 2024-11-26T00:00:00+01:00"
aProduct orderItems ? #mostOftenBoughtOne. "--> MyArticle<Google Pixel 9 Pro>"
aProduct semanticProxy countSalesFrom: '2023Q3' to: '2023Q4'. "--> 12,345"
World ? #numberOfOpenWindows. "--> 138"
The above is Smalltalk – translated to Python, this would look like this:
order.ask('creationDate') # datetime.date(2024, 11, 26)
product.orderItems.ask('most often bought one') # MyArticle(name='Google Pixel 9 Pro')
product.orderItems.semanticProxy().countSalesWithin('2023Q3', '2023Q4') # 12345
display.screen().ask('number of open windows') # 12
Similar integrations are available for chatting with the call stack in a debugger or with the classes in a package editor.
Try It Yourself
Check out SemanticText here:
The repo contains installation instructions, documentation, and several API examples.
In Squeak, you can install it by running:
Metacello new
baseline: 'SemanticText';
repository: 'github://hpi-swa-lab/Squeak-SemanticText:main';
load.
Not a Smalltalker? The ideas and architecture are transferable to other ecosystems, and I’d love to hear your thoughts on applying similar principles elsewhere!
For further details, you can also check out our following publications:
- Talking to Objects in Natural Language: Toward Semantic Tools for Exploratory Programming (Conference Paper. Slides, presentation transcript, and posters are also available)
- The Semantic Workspace: Augmenting Exploratory Programming with Integrated Generative AI Tools (Master’s thesis)
Join the Live Demo!
I’ll be presenting SemanticText and discussing how LLMs can be deeply integrated into runtime systems at the UK Smalltalk User Group Meeting:
Wednesday, Nov 27th
7 PM GMT / 8 PM CET / 11 AM PST
Join the Meetup
I’m looking forward to your comments and ideas! Let’s chat about what’s possible when AI tools are integrated directly into runtime systems. I’m eager to hear your ideas, feedback, or questions!
Best,
Christoph