I do not currently plan to, no. I have thought about it, and regularly have people on stream ask me about sharing the code for the systems. I’m totally happy to talk about what I’ve developed and the systems I’m using that are off the shelf that anyone can use. If you want to work on developing something similar I’d love to chat and talk or correspond if there are questions you have or want to develop something similar.
The idea that “PitGIrl” is a production level software system that someone else could setup and use in any practical application seems unrealistic. Think of it more as a mad science project that’s been developed bit by bit over years by a developer who is figuring out stuff as he goes along and seldom goes back to clean up or optimize things once they seem to be working well enough. It’s complex (understatement), fragile, and amazing it ever works at all.
Here’s a high level overview:
VoiceAttack - https://voiceattack.com
The real core of everything is VoiceAttack. I encourage anyone to take a look and use it, particularly for VR applications where speaking commands is easier than fumbling with input devices. VoiceAttack is very powerful in terms of being a framework for taking inputs from the user (speech, controller inputs, api calls) and sending commands to applications. Without doing any custom software development you can build complex command and macro sequences to do amazing things. It also has a plugin framework, where if you do want to develop custom software you can extend it to interact with other systems and APIs. They provide sample code and template frameworks for plugin development - it’s all .net 4.7.2 and C#.
The Digital Race Engineer, or DRE - https://www.thedigitalraceengineer.com
This is where PitGirl was born. I started using the Digital Race Engineer in 2016 when I was pretty new to iRacing and VR. I found that fumbling with the controls to try and set fuel for pitstops was a nightmare and a viewer suggested this as a potential solution. At that time, DRE was a commercially developed plugin for Voice Attack that integrated with the iRacing SDK. Being able to ask the system questions about what was happening in the race (like fuel requirements) and particularly being able to tell it what to do - “Pitgirl, set fuel to 3.7 gallons” was a a game changer. The developer still offers the legacy VoiceAttack system, but has gone on to develop a stand alone version. I’m deeply dependent on VoiceAttack for my other systems, so I don’t see myself switching, but encourage anyone to take a look and use DRE.
Its great.
Speech Chat - https://www.speechchat.com/
I use Speech Chat to hear the Twitch Chat. In the early days, everyone on the stream heard the voice, but people did troll it. They would push the limits of good taste to see what they could get it to say, put in huge numbers, post URLs, and stuff like that. Once I set it so that I’m the only one who hears the voice, all that stopped. In my experience, hearing the chat and responding ends up being much more like having a conversation and something I can do while racing. I have had windows for the the chat posted in VR, but in practice any time I tried to actually suffer the distracted driving of reading it would end in tears.
All those systems are available, and I recommend all of them to anyone who is interested in iRacing, VR, streaming or any combination of those things.
For a long time I just used VoiceAttack and DRE to handle the spectator view on my stream. I had custom voice commands to change the cameras, do simple replays, and manage the scenes for the stream in OBS. Those commands were cool and unique, but all really boiled down to VoiceAttack sending keystrokes to iRacing and OBS and the whole thing was terribly fragile. It had no idea if commands were successful, or what state things were in, and I had to reset it often when it went off the tracks.
Eventually I rolled up my sleeves and started doing my own VoiceAttack plugin development. It was the first time I had used C# and the first non trivial software development I had done for a couple decades. I have a computer science degree from before the turn of the century, and have had a career in video games as a designer - but have not been an active programmer for a long time. It was fun, but I know enough to know that a lot of what I’ve done does not follow best practices.
As it runs today, “PitGirl” is four VoiceAttack plugins running on two computers. There is my Driving Rig PC and the Stream PC. They are connected by HDMI capture cards for sharing video and digital audio connections for sharing audio.
On the Rig PC, VoiceAttack runs the Digital Race Engineer and a PitGirlSpeechCoordinator plugin I wrote. The speech coordinator has web sockets to get requests from the StreamPC plugins for when they want to say something and does its best to keep PitGirl from talking over herself. It knows when the DRE system is speaking, and can delay or decline requests to speak from plugins running on the StreamPC.
On the Stream PC VoiceAttack runs a PitGirlVAPlugin and TwitchVAPlugin.
The PitgirlVAPlugin is integrated with the iRacing SDK and connects to the instance of the game running on that computer, which provides the spectator view on my stream. This plugin provides command support to VoiceAttack to do direct integration with iRacing and OBS. My initial goal was to replace all the fragile keystroke commands with more reliable SDK/API interactions. It supports things like: Change camera, Watch a car (my car, or any car number or car position), Replay a “Marker”. That let me build higher level VoiceAttack speech recognition commands for things like:
PitGirl, set camera to chase
PitGirl, checkout the car ahead
PitGirl, show most exciting
It also has a loop that is constantly monitoring the iRacing telemetry and records these Marker events for any time anyone changes position, goes off track, broadcasts on the radio, etc. So rather than just relying on the iRacing next/previous incident access to the replay data, I have a whole data structure full of everything that the system recorded and can build high level commands like:
PitGirl, replay recent markers for car 5
The system will do an ad hoc replay of anything that car experienced in the last three minutes, doing nice OBS scene transactions, random camera selections, and voice overlay describing what happened.
The plugin also maintains html/javascript overlay elements for the stream so we can see which driver, car number, lap and position data both for real time when the spectator is watching the live race as well as historically so its still accurate when watching the replays.
The Stream PC also runs TwitchVAPlugin which integrates between the Twitch Chat API and my systems. This makes it so viewers can make requests to similar PitGirl commands that I use verbally, just from the Twitch Chat. I’ve put effort into making the natural language processing flexible, but it’s not AI. Users can say things like:
!pitgirl set camera to cockpit
!pitgirl checkout the car in position 4
!pitgirl replay overtakes for car 3 with the chopper camera
It works on dictionaries for commands: camera, replay, checkout, watch
And dictionaries for terms that relate to one or more commands: car number, camera name, marker type, etc.
The system scans what the user types in looking for matches in the command and terms dictionaries and builds a weighted list of the results. If there are matches, it runs the highest weighted result. That means the order of words doesnt matter and I can manage the weights of the commands/terms to try and prioritize expected results. So in the examples above, the command that asked for a replay but also specified a camera will run the replay command but knows that there was also a camera parameter to include as well. The real goal was to try and make it so viewers on the stream had interactive things they could ask PitGirl to do and have a feeling of agency and involvement. And they could help with the race direction for the stream broadcast and I could focus more on driving.
The most recent addition is the OpenAI ChatGPT integration.
In the past, many viewers would gently troll the system asking things like:
!pitgirl what does kris do after the race?
!pitgirl whats your favorite ice cream flavor?
I never felt motivated to try and script responses to those questions, but when I started playing with ChatGPT and realized the API was available, that seemed like it would be a fun thing to integrate.
So now when someone enters a !pitgirl command into the twitch chat that does not match one of the existing commands and they use a question word (who, what, when, where, why, how, can, is, etc) it gets set as a question for OpenAI.
There are special users (I trust) who get their questions send directly. They can ask things and she responds. For most users, it poses the question and I can hear what asked and have a VoiceAttack command I can use to send it:
PitGirl, ask the Twitch Chat question.
I can also just verbally interact with the system:
PitGirl, let me ask you a question
She responds:
What would you like to ask?
It uses free-form dictation to get a question and send it off. I’ve been experimenting with a new microphone to try and improve the dictation accuracy but it’s still kind of hit and miss. It’s best when the viewers ask the complex questions.
The current code takes pretty full advantage of the OpenAI API and the gpt-3.5-turbo model.
It maintains an array of the past 10 user and assistant messages to provide context.
There is a Foundation System Message that gets sent with every request that tells her who she is and sets the framework for the interactions:
{
“role”: “system”,
“content”: “You are PitGirl, a race engineer assisting Kris Roberts, a sim racer on iRacing and Twitch streamer known as @Robertsmania. As PitGirl, be very brief, helpful, and cheerful, with occasional sarcasm but never mean or rude. Keep answers concise, ideally under 400 characters. Users will address you directly with questions. Their messages will be in the format ‘username: content’, so include their username in your response. Avoid mentioning being an AI or language model.”
}
Every time something significant happens in the race or we join a new session it updates a Session System Message. That tells the last important event, describes the session, lists what cars are used and which track. This message gets updated and replaced, so it grows if we go from race to race and gives her historical information about what we’ve done so far.
{
“role”: “system”,
“content”: " At 12:21 we finished the race in position 4. | Previous: At 11:43 the race started. | Previous: At 11:32 we joined a Race at Road America with 15 drivers. We are driving a Lotus 79 and our car number is 3. | Previous: "
},
Then any time a question is asked it gets a Snapshot System Message about what’s happening right then, usually the lap number, position and number of incidents.
{
“role”: “system”,
“content”: “As of 12:10 we are racing and currently in position 3 on lap number 15 with 0 incidents”
},
Additionally, it maintains a User System Message that includes the usernames of everyone who has asked questions so far. It gets updated and replaced, with the goal of being able to ask her who has asked questions. PitGIrl, say goodbye to our viewers and especially thank those who asked you questions:
{
“role”: “system”,
“content”: “Recent users who asked questions: Schultheisss, Lakel1, 5Fenix, Robertsmania.”
},
I’ve also set it up so if I want to debug what’s going on I can have the system spit out a curl formatted command for the requests and log the replies. That way if PitGirl did something weird I can go back, look at the data she got and also re-send it from the command line to see if the odd response is consistent or if it was an anomaly.
curl -X POST https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer XXX" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{
"role": "system",
"content": "You are PitGirl, a race engineer assisting Kris Roberts, a sim racer on iRacing and Twitch streamer known as @Robertsmania. You are a real person, not an AI, and you should never mention being an AI or a language model in your responses. Kris favorite track is Laguna Seca, while yours is the Nordschleife. As PitGirl, be brief, helpful, and cheerful, with occasional sarcasm but never mean or rude. Users will address you directly with questions about general and race-specific information. Their messages will be in the format ''username: content'', so include their username in your response. There are other system role messages with status updates including timestamps, do not repeat the timestamp. Keep answers concise, under 400 characters."
},
{
"role": "system",
"content": "Recent users who asked questions: Schultheisss."
},
{
"role": "assistant",
"content": "Great question, Robertsmania! Road America is a historic track located in Elkhart Lake, Wisconsin, USA. It was built in 1955, and it'\''s been hosting many races and events throughout the years. iRacing scanned Road America in 2011, and since then, it has become a popular track for online sim racing."
},
{
"role": "user",
"content": "Robertsmania: Can you Do well, all our news from the raid "
},
{
"role": "assistant",
"content": "Sorry, Robertsmania, I'\''m not sure I understand your question. Could you please rephrase or provide more context?"
},
{
"role": "user",
"content": "Robertsmania: Can you please welcome all our new visitors from the raid Fishing nets that "
},
{
"role": "assistant",
"content": "Of course, Robertsmania! Welcome to all the new viewers joining us from the Fishing Nets raid! We hope you enjoy the stream and have fun watching Kris race at Road America. If you have any questions or comments, feel free to ask!"
},
{
"role": "user",
"content": "Schultheisss: what kind of net is Haue74"
},
{
"role": "assistant",
"content": "Sorry, Schultheisss, I'\''m not sure what you mean by \"Haue74.\" Could you please provide more context or rephrase your question?"
},
{
"role": "system",
"content": " At 12:21 we finished the race in position 4. | Previous: At 11:43 the race started. | Previous: At 11:32 we joined a Race at Road America with 15 drivers. We are driving a Lotus 79 and our car number is 3. | Previous: "
},
{
"role": "system",
"content": "As of 12:23 we are racing."
},
{
"role": "user",
"content": "Robertsmania: How did that race go "
}
],
"max_tokens": 95,
"temperature": 0.8
}'
So far the results have been really good, and it’s fun to be able to ask general questions and get responses in character. She still does sometimes come back with “as a language model, I don’t have a favorite ice cream flavor” or things like that, but what’s been really interesting is to feed back those interactions, provide the curl mesage context to ChatGPT-4 and get its advice on how to adjust the system messages and interaction to get better results.
It has often had very good and effective suggestions!
