Just found this community while searching to see what’s out there involving LLMs in gaming! I’ve been working on building an LLM-backed game agent for quite a while now. My goal is to create smooth real-time interaction between the player and a single LLM-backed agent in a 3d environment.
The only demo I have so far is this video which I made over a year ago, which I’d call v1. I’ve since been working on v2 and wanted to get a similar video out last fall but it has been slow going trying to rein in all the moving parts and getting the LLM to behave. V2 has been built with much more intention for using an LLM as the agent’s “brain” and as a result the system is more complex and open-ended.
Some high-level points on the v2 implementation:
- built in Unity
- runs a local LLM through LlamaSharp (v1 used a third-party package that I’ve since scrapped because LlamaSharp provides finer grain control of inference)
- text-to-speech and speech-to-text also use neural networks (though they are much smaller than the “brain” LLM and don’t take up much memory)
- running on a 4080 mobile, the LLM and other NNs take up about 7GB of GPU memory (out of 12GB).
Due to the real-time nature, my goal was to have no more than 1 second reaction time (i.e. 1 second to receive an action from the LLM in response to stimuli). This means I have to keep my prompts and responses really tight. It works well enough in the short term. But I am basically stuck on medium- to long-term planning as I’m trying to figure out how to apply reasoning, formulate goals and track memory in addition to the agent acting, while still keeping the inference time fast. I’ve come to the conclusion that I probably need to relax my speed requirement to accomplish this. This is the part that is taking a while.
Current documentation on context engineering, agentic workflows, etc. provide limited insight for this problem because they generally deal with slower, turn-based processes that solve business problems. I’m kind of making this up as I go. Happy to join this community and discuss my efforts and see what other people are doing. Anyone else doing something similar to this?
1 Like
Welcome to the community, @jfaath !
Great first post, imho! Hope you stick around. I added your video. Hope you don’t mind!
I played with LLM for NPCs a while back, but it was a bit too expensive, and I backed off for a bit.
I’m seeing more and more LLM use in games, though, which is why I started this thread to attract devs like yourself.
Did you find us via Google? Again, welcome!
Im trying to implement llm .. into game development,
bigest problem i find, specially with local models that they are heavy in the download. also heavy on resource..
That part.. i complete understand, we not have models.. yet. that are 5mb.
But then.. the kicker.
Made a game, as basic as possible. to just see if concept is working.
When i the user, get its first screen. i tell the user that it is using AI. people accept or close the game.
When asking to download 4gb data model.. they stop and game is instant worthless..
people love to download 260gb of assets .. to play the next call of duty.. but 4gb data model. they not want to..
But then i can hide the fact. that its using LLM .. or be as open ass possible.
beeing as open as possible rejects almost 85%.
So how to solve this some sort of dilema.. people love the idea.. to have endless games.
but afraid or hate Llms and what they stand for.. or time just will decide 
1 Like
Yes, I searched for something like “LLM-based game agent”. I’ve been doing that search periodically to see what I can find out there. I found this github site some time ago which is a survey of research using LLMs for games.
Thanks for the welcome. Looking forward to engaging with the community.
1 Like
We are definitely in the very early stages of using this technology for games. Public sentiment for using AI in creative pursuits isn’t great. Most people don’t have the hardware to run a local model and there are privacy and financial concerns using a cloud-based model.
I’m not even sure the traditional game format or distribution method (Steam marketplace, for example) will apply to games made with LLMs. I’ve thought a little bit about how it might look, but it’s all moot until I get something that is working and playable. It’s fun being on the frontier, but difficult and slow-moving.
2 Likes
No problem. You’re the type of dev we want more of here, I think.
We’ve got most stuff related tagged gaming or similar.
Feel free to start a thread about any project you have in Community with the project tag. Or keep helping here in the general gamedev thread.
1 Like
I’m looking into an annotator position that deals with LLM models for gaming…
It’s higher on my list of preferred because this seems to be the direction things are going.
Looking forward to see if any of the experience there can help out with projects like these ^.^.
1 Like
One thing I have learned using GPT-style tools in gamedev is that they are most powerful when used to accelerate experimentation rather than replace playtesting. I have had much better results letting AI help me explore many small variations quickly, then using real player behavior to decide what survives. Optimizing for ‘least resistance’ in tools often produces systems that are efficient to build but lifeless to play — so I now treat friction as something to relocate into play rather than eliminate
For that AI is absolute great,
I have seen implementations and usage with AI already in multiple indy developer setups.
from helping with coding, story telling and even already early adopters of MCP servers for unreal engine and other tools.
Even seen non eperienced coders, making complete game engines. yes a complete 2d game engine that is working and even generated game execatables.
Lots of people screaming of attention not to use these tools. learn to program yourself.
But specially.. for people with great mindsets.. the will and not have making monney but fun on first place.. have already seen great work beeing done.
So it can absolute accelerate.. not only in professional work.. but absolute in game dev.
How great is it when you simply dump a screenshot of 5 blueprints.. tell the machine.
i have no idea what is going on.. but this is the bug.. and you have a rubber duck that even make suggestions.
just like asking a friend that have no idea what you doing, this duck .. sometimes can help you finding the solution together
1 Like
How to hook codex in vscode/web browser chatgpt so they can talk to each other and iterate on my game directly in the game engine?
Asking because web chatgpt is more well-versed in this game engine lingo.
1 Like
Try Cloudflare; it offers a wealth of features within Codex.
1 Like
They decided because they were presented a list of tools to use based on the current location right?