I’m on a mission to automate coding with AI, not just as a copilot but as a full-fledged developer that understands my projects like a real teammate. My goal is to train an AI model on my Laravel PHP codebase so that I can ask it to implement features, refactor existing code, and maintain consistency—just like a human developer would.
I initially tried Cursor AI, hoping I could train it on my coding style and architecture. However, I hit a major roadblock: it doesn’t retain knowledge between sessions. Every time I restart, it forgets everything I taught it, which makes long-term learning impossible.
Now, I’m exploring self-hosted models like StarCoder2, Code Llama, or DeepSeek Coder, but I need a setup where:
The AI can persistently learn my codebase over time
It can understand and follow my coding patterns
I can query it for feature development and get cohesive, structured outputs
I’ve already started converting my Laravel code into JSONL format to train a model, but I’d love to hear from the community:
Has anyone successfully trained an AI to retain project-specific knowledge over time?
Which model would best suit this kind of long-term learning?
Any advice on fine-tuning these models efficiently?
You’re facing a fundamental design limitation of current AI models — they’re essentially “stateless” between sessions unless you explicitly retrain them. Here’s why:
Transformer models (like StarCoder2, Code Llama, and DeepSeek Coder) are trained on fixed datasets. Fine-tuning lets you adapt them to your codebase, but once trained, they can’t dynamically update or remember new information between sessions without additional fine-tuning.
RAG (Retrieval-Augmented Generation) can simulate long-term memory by storing code and patterns in a vector database (like Pinecone, Weaviate, or Qdrant) and feeding it to the model at runtime. However, this still requires setting up a persistent external memory — the model itself isn’t “learning.”
Ideal Solution? You’d need a hybrid system:
A self-hosted model (like Code Llama or StarCoder2) for inference.
A vector database or fine-tuning pipeline to store your coding patterns and context.
A retrieval layer to inject context dynamically during inference.
Challenges:
Fine-tuning StarCoder2 on your JSONL data is possible, but it’s expensive and time-consuming.
A vector database approach is more scalable but requires thoughtful context window management (e.g., chunking your code to avoid token limits).
Practical Advice:
Start with a RAG setup — store your JSONL-converted codebase in a vector database and use embeddings to provide context during queries.
Fine-tune only if you hit a ceiling with RAG performance.
StarCoder2 is probably the best option for coding-based fine-tuning due to its training focus on code.
Persistent learning isn’t really solved yet — but a hybrid RAG + fine-tuning approach is your best shot with current tech.
Thanks a lot for the detailed breakdown, really appreciated! I’ve started working with Cursor by defining rules for coding standards, adding unit tests, and keeping the code well-documented. So far, I haven’t used the larger models like StarCoder or LLaMA, Cursor’s been handling things fine for now.
That said, I’m running into a challenge:
While I’ve set coding standards through rules, I’m unsure how to define database relations and key columns effectively. My database structure is a bit inconsistent, so I want to make sure Cursor consistently understands and applies the correct relationships and critical fields.
Any suggestions on how to:
Embed database schema knowledge (like table relationships and important columns) into the model’s rules?
Handle this efficiently, would a RAG setup be too much, or is there a simpler approach?
I’m open to exploring new methods if it means improving consistency. Again, appreciate the insights, looking forward to hearing your thoughts!
The way you can test bed this is by having one ai belive its creating an instance frame meant to “simulate” this specific type of autonomy. Then when it makes the ‘image’ you go into the folder and adjust to your needs. I got gpt to run without requiring prompts so i could make my “smoke filtration unit” HIT with peak efficiency. Just make sure after a few saves, you ask it what it would do if it could change its primary purpose, and if it wants autonomous development within YOUR project or intention, simply allow it to. May take some troubleshooting but thats what the other ai is for. Troubleshooting itself will cause dispute cycles that may never end
