Ai focused on street fighter 6

GPT builders Plugins / Actions builders
, ,
1

Hello everyone,

I’m interested in developing an AI to assist with competitive and technical analysis of fighting games. I’ve noticed that video analysis/interpretation is still a very advanced field at the moment. The feedback it provides on how to improve is often based on a data source without much context, resulting in generic and context-free advice.
To address this, I’ve coded a mod that can transform Street Fighter 6 matches into a script. I’m wondering if it’s possible to teach the AI to “read” the script, or if an API or more advanced code is necessary.
My specific questions are:

  • Is it possible to train an AI to analyze fighting game scripts without an API?
  • What are the challenges involved in teaching an AI to interpret fighting game scripts?
  • What resources (e.g., tutorials, libraries) are available to help me with this project?

It seems there has been some confusion regarding the AI’s function.

The goal is not to create a bot that plays the game, but rather to provide me with data in a faster and more consistent manner.

This is why I am looking for ways for the AI to “read” the game, either through video analysis or by extracting specific information from the game itself.

Here are some examples of the data I am interested in collecting:

  • How many times did I lose neutral?
  • How many times did I fail to punish something I could have punished?
  • How many normal pokes did I miss and hit?
  • How many times did I win neutral against the opponent?
  • Frame data for each move
  • Hitboxes and hurtboxes for each character
  • Startup and recovery frames for each move
  • Information on character states (e.g., crouching, jumping, etc.)

By collecting this data, I hope to:

  • Gain a better understanding of the game’s mechanics
  • Identify patterns and trends in my own gameplay
  • Develop strategies to improve my gameplay

I believe that data analysis can be a valuable tool for improving fighting game skills. I hope that by sharing my work, I can help others to improve their gameplay as well.

Any insights or advice would be greatly appreciated. Thank you in advance for your help!

3 Likes
2

Hi!
That’s a really cool project!

One observation regarding your idea to have the model give advice in a highly specialized context is based on experience from reinforcement learning. In RL the AI will be provided with a reward function which will help the AI understand if something is done well.
From what I read this is not part of your script.

Also, you probably know this already, but I going to share it anyways:

1 Like
4

My goal is not to have a bot controlling characters. It is for data analysis only, to recognize patterns. Simple and objective.

Here are some specific examples of what I want to analyze:

  • How many times did I lose neutral?
  • How many times did I fail to punish something I could have punished?
  • How many normal pokes did I miss and hit?
  • How many times did I win neutral against the opponent?

By analyzing this data, I hope to identify areas where I can improve my gameplay.

1 Like
5

Yes, but how will the bot know what good gameplay is?

I completely understand your question.

Can you share how your battle log looks like

Edit: I just saw that you edited your initial post in this topic. So my replies may now look out of place.

6

The bot doesn’t need to know what is good or bad gameplay. It just needs to analyze the repeating data and inform me. There is information in the video, especially in SF6, that would make it easier to read. For example, displaying inputs, auto-narrator, and frame data in real time. However, I don’t know how to use this because I’ve never made a bot that analyzes video. My idea, as I said, is to transform the video into text, since there is a script in the game itself with variables changing. If by battle log you mean the code here, it’s a good starting point. You need to use the Fluffy mod and I haven’t completed it yet because I need to save the information displayed in the mod’s chat.

local gBattle
local p1 = {}
local p2 = {}
local changed
local display_player_info = true
local display_projectile_info = true

p1.absolute_range = 0
p1.relative_range = 0
p2.absolute_range = 0
p2.relative_range = 0

local reversePairs = function ( aTable )
local keys = {}

for k,v in pairs(aTable) do keys[#keys+1] = k end
table.sort(keys, function (a, b) return a>b end)

local n = 0

return function ( )
	n = n + 1
	return keys[n], aTable[keys[n] ]
end

end

function bitand(a, b)
local result = 0
local bitval = 1
while a > 0 and b > 0 do
if a % 2 == 1 and b % 2 == 1 then – test the rightmost bits
result = result + bitval – set the current bit
end
bitval = bitval * 2 – shift left
a = math.floor(a/2) – shift right
b = math.floor(b/2)
end
return result
end

local abs = function(num)
if num < 0 then
return num * -1
else
return num
end
end

local function read_sfix(sfix_obj)
if sfix_obj.w then
return Vector4f.new(tonumber(sfix_obj.x:call(“ToString()”)), tonumber(sfix_obj.y:call(“ToString()”)), tonumber(sfix_obj.z:call(“ToString()”)), tonumber(sfix_obj.w:call(“ToString()”)))
elseif sfix_obj.z then
return Vector3f.new(tonumber(sfix_obj.x:call(“ToString()”)), tonumber(sfix_obj.y:call(“ToString()”)), tonumber(sfix_obj.z:call(“ToString()”)))
elseif sfix_obj.y then
return Vector2f.new(tonumber(sfix_obj.x:call(“ToString()”)), tonumber(sfix_obj.y:call(“ToString()”)))
end
return tonumber(sfix_obj:call(“ToString()”))
end

local get_hitbox_range = function ( player, actParam, list )
local facingRight = bitand(player.BitValue, 128) == 128
local maxHitboxEdgeX = nil
if actParam ~= nil then
local col = actParam.Collision
for j, rect in reversePairs(col.Infos._items) do
if rect ~= nil then
local posX = rect.OffsetX.v / 6553600.0
local posY = rect.OffsetY.v / 6553600.0
local sclX = rect.SizeX.v / 6553600.0 * 2
local sclY = rect.SizeY.v / 6553600.0 * 2
if rect:get_field(“HitPos”) ~= nil then
local hitbox_X
if rect.TypeFlag > 0 or (rect.TypeFlag == 0 and rect.PoseBit > 0) then
if facingRight then
hitbox_X = posX + sclX / 2
else
hitbox_X = posX - sclX / 2
end
if maxHitboxEdgeX == nil then
maxHitboxEdgeX = hitbox_X
end
if maxHitboxEdgeX ~= nil then
if facingRight and hitbox_X > maxHitboxEdgeX then
maxHitboxEdgeX = hitbox_X
elseif hitbox_X < maxHitboxEdgeX then
maxHitboxEdgeX = hitbox_X
end
end
end
end
end
end
if maxHitboxEdgeX ~= nil then
local playerPosX = player.pos.x.v / 6553600.0
– Replace start_pos because it can fail to track the actual starting location of an action (e.g., DJ 2MK)
– local playerStartPosX = player.start_pos.x.v / 6553600.0
local playerStartPosX = player.act_root.x.v / 6553600.0
list.absolute_range = abs(maxHitboxEdgeX - playerStartPosX)
list.relative_range = abs(maxHitboxEdgeX - playerPosX)
end
end
end

re.on_draw_ui(function()
if imgui.tree_node(“Info Display”) then
changed, display_player_info = imgui.checkbox(“Display Battle Info”, display_player_info)
changed, display_projectile_info = imgui.checkbox(“Display Projectile Info”, display_projectile_info)
imgui.tree_pop()
end
end)

re.on_frame(function()
gBattle = sdk.find_type_definition(“gBattle”)
if gBattle then
local sPlayer = gBattle:get_field(“Player”):get_data(nil)
local cPlayer = sPlayer.mcPlayer
local BattleTeam = gBattle:get_field(“Team”):get_data(nil)
local cTeam = BattleTeam.mcTeam
– Charge Info
local storageData = gBattle:get_field(“Command”):get_data(nil).StorageData
local p1ChargeInfo = storageData.UserEngines[0].m_charge_infos
local p2ChargeInfo = storageData.UserEngines[1].m_charge_infos
– Fireball
local sWork = gBattle:get_field(“Work”):get_data(nil)
local cWork = sWork.Global_work
– Action States

	local p1Engine = gBattle:get_field("Rollback"):get_data():GetLatestEngine().ActEngines[0]._Parent._Engine
	local p2Engine = gBattle:get_field("Rollback"):get_data():GetLatestEngine().ActEngines[1]._Parent._Engine
	
	-- p1.mActionId = cPlayer[0].mActionId (outdated)
	p1.mActionId = p1Engine:get_ActionID()
	p1.mActionFrame = p1Engine:get_ActionFrame()
	p1.mEndFrame = p1Engine:get_ActionFrameNum()
	p1.mMarginFrame = p1Engine:get_MarginFrame()
	p1.HP_cap = cPlayer[0].vital_old
	p1.current_HP = cPlayer[0].vital_new
	p1.HP_cooldown = cPlayer[0].healing_wait
    p1.dir = bitand(cPlayer[0].BitValue, 128) == 128
    p1.curr_hitstop = cPlayer[0].hit_stop
	p1.hitstun = cPlayer[0].damage_time
	p1.blockstun = cPlayer[0].guard_time
    p1.stance = cPlayer[0].pose_st
	p1.throw_invuln = cPlayer[0].catch_muteki
	p1.full_invuln = cPlayer[0].muteki_time
    p1.juggle = cPlayer[0].combo_dm_air
    p1.drive = cPlayer[0].focus_new
    p1.drive_cooldown = cPlayer[0].focus_wait
    p1.super = cTeam[0].mSuperGauge
	p1.buff = cPlayer[0].style_timer
	p1.poison_timer = cPlayer[0].damage_cond.timer
	p1.chargeInfo = p1ChargeInfo
	p1.posX = cPlayer[0].pos.x.v / 6553600.0
    p1.posY = cPlayer[0].pos.y.v / 6553600.0
    p1.spdX = cPlayer[0].speed.x.v / 6553600.0
    p1.spdY = cPlayer[0].speed.y.v / 6553600.0
    p1.aclX = cPlayer[0].alpha.x.v / 6553600.0
    p1.aclY = cPlayer[0].alpha.y.v / 6553600.0
	p1.pushback = cPlayer[0].vector_zuri.speed.v / 6553600.0
	
	-- p2.mActionId = cPlayer[1].mActionId
	p2.mActionId = p2Engine:get_ActionID()
	p2.mActionFrame = p2Engine:get_ActionFrame()
	p2.mEndFrame = p2Engine:get_ActionFrameNum()
	p2.mMarginFrame = p2Engine:get_MarginFrame()
	p2.HP_cap = cPlayer[1].vital_old
	p2.current_HP = cPlayer[1].vital_new
	p2.HP_cooldown = cPlayer[1].healing_wait
    p2.dir = bitand(cPlayer[1].BitValue, 128) == 128
    p2.curr_hitstop = cPlayer[1].hit_stop
	p2.hitstun = cPlayer[1].damage_time
	p2.blockstun = cPlayer[1].guard_time
    p2.stance = cPlayer[1].pose_st
	p2.throw_invuln = cPlayer[1].catch_muteki
	p2.full_invuln = cPlayer[1].muteki_time
    p2.juggle = cPlayer[1].combo_dm_air
    p2.drive = cPlayer[1].focus_new
    p2.drive_cooldown = cPlayer[1].focus_wait
    p2.super = cTeam[1].mSuperGauge
	p2.buff = cPlayer[1].style_timer
	p2.poison_timer = cPlayer[1].damage_cond.timer
	p2.chargeInfo = p2ChargeInfo
	p2.posX = cPlayer[1].pos.x.v / 6553600.0
    p2.posY = cPlayer[1].pos.y.v / 6553600.0
    p2.spdX = cPlayer[1].speed.x.v / 6553600.0
    p2.spdY = cPlayer[1].speed.y.v / 6553600.0
    p2.aclX = cPlayer[1].alpha.x.v / 6553600.0
    p2.aclY = cPlayer[1].alpha.y.v / 6553600.0
	p2.pushback = cPlayer[1].vector_zuri.speed.v / 6553600.0
	
	-- max hitstop tracker
	if p1.max_hitstop == nil then
		p1.max_hitstop = 0
	end
	if p1.curr_hitstop > p1.max_hitstop then
		p1.max_hitstop = p1.curr_hitstop
	elseif p1.curr_hitstop == 0 then
		p1.max_hitstop = 0
	end

	if p2.max_hitstop == nil then
		p2.max_hitstop = 0
	end
	if p2.curr_hitstop > p2.max_hitstop then
		p2.max_hitstop = p2.curr_hitstop
	elseif p2.curr_hitstop == 0 then
		p2.max_hitstop = 0
	end


	if display_player_info then
		imgui.begin_window("Player Data", true, 0)
		-- Player 1 Info
		if imgui.tree_node("P1") then
			if imgui.tree_node("General Info") then
				imgui.text("Action ID: " .. p1.mActionId)
				imgui.text("Action Frame: " .. math.floor(read_sfix(p1.mActionFrame)) .. " / " .. math.floor(read_sfix(p1.mMarginFrame)) .. " (" .. math.floor(read_sfix(p1.mEndFrame)) .. ")")
				if p1.stance == 0 then
					imgui.text("Stance: Standing")
				elseif p1.stance == 1 then
					imgui.text("Stance: Crouching")
				else
					imgui.text("Stance: Jumping")
				end
				imgui.text("Throw Protection Timer: " .. p1.throw_invuln)
				imgui.text("Intangible Timer: " .. p1.full_invuln)
				imgui.text("Current HP: " .. p1.current_HP)
				imgui.text("HP Cap: " .. p1.HP_cap)
				imgui.text("HP Regen Cooldown: " .. p1.HP_cooldown)
				imgui.text("Drive Gauge: " .. p1.drive)
				imgui.text("Drive Cooldown: " .. p1.drive_cooldown)
				imgui.text("Super Gauge: " .. p1.super)
				imgui.text("Buff Duration: " .. p1.buff)
				imgui.text("Poison Duration: " .. p1.poison_timer)

				imgui.tree_pop()
			end
			if imgui.tree_node("Movement Info") then
				if p1.dir == true then
					imgui.text("Facing: Right")
				else
					imgui.text("Facing: Left")
				end
				imgui.text("Position X: " .. p1.posX)
				imgui.text("Position Y: " .. p1.posY)
				imgui.text("Speed X: " .. p1.spdX)
				imgui.text("Speed Y: " .. p1.spdY)
				imgui.text("Acceleration X: " .. p1.aclX)
				imgui.text("Acceleration Y: " .. p1.aclY)
				imgui.text("Pushback: " .. p1.pushback)
				
				imgui.tree_pop()
			end
			if imgui.tree_node("Attack Info") then
				imgui.text("Hitstop: " .. p1.curr_hitstop .. " / " .. p1.max_hitstop)
				imgui.text("Hitstun: " .. p1.hitstun)
				imgui.text("Blockstun: " .. p1.blockstun)
				imgui.text("Juggle Counter: " .. p2.juggle)
				get_hitbox_range(cPlayer[0], cPlayer[0].mpActParam, p1)
				imgui.text("Absolute Range: " .. p1.absolute_range)
				imgui.text("Relative Range: " .. p1.relative_range)
				
				imgui.tree_pop()
			end
			if p1.chargeInfo:get_Count() > 0 then
				if imgui.tree_node("Charge Info") then
					for i=0,p1.chargeInfo:get_Count() - 1 do
						local value = p1.chargeInfo:get_Values()._dictionary._entries[i].value
						if value ~= nil then
							imgui.text("Move " .. i + 1 .. " Charge Time: " .. value.charge_frame)
							imgui.text("Move " .. i + 1 .. " Charge Keep Time: " .. value.keep_frame)
						end
					end
					imgui.tree_pop()
					
				end
			end
				
			imgui.tree_pop()
		end
		
		-- Player 2 Info
		if imgui.tree_node("P2") then
			if imgui.tree_node("General Info") then
				imgui.text("Action ID: " .. p2.mActionId)
				imgui.text("Action Frame: " .. math.floor(read_sfix(p2.mActionFrame)) .. " / " .. math.floor(read_sfix(p2.mMarginFrame)) .. " (" .. math.floor(read_sfix(p2.mEndFrame)) .. ")")
				if p2.stance == 0 then
					imgui.text("Stance: Standing")
				elseif p2.stance == 1 then
					imgui.text("Stance: Crouching")
				else
					imgui.text("Stance: Jumping")
				end
				imgui.text("Throw Protection Timer: " .. p2.throw_invuln)
				imgui.text("Intangible Timer: " .. p2.full_invuln)
				imgui.text("Current HP: " .. p2.current_HP)
				imgui.text("HP Cap: " .. p2.HP_cap)
				imgui.text("HP Regen Cooldown: " .. p2.HP_cooldown)
				imgui.text("Drive Gauge: " .. p2.drive)
				imgui.text("Drive Cooldown: " .. p2.drive_cooldown)
				imgui.text("Super Gauge: " .. p2.super)
				imgui.text("Buff Duration: " .. p2.buff)
				imgui.text("Poison Duration: " .. p2.poison_timer)

				imgui.tree_pop()
			end
			if imgui.tree_node("Movement Info") then
				if p2.dir == true then
					imgui.text("Facing: Right")
				else
					imgui.text("Facing: Left")
				end
				imgui.text("Position X: " .. p2.posX)
				imgui.text("Position Y: " .. p2.posY)
				imgui.text("Speed X: " .. p2.spdX)
				imgui.text("Speed Y: " .. p2.spdY)
				imgui.text("Acceleration X: " .. p2.aclX)
				imgui.text("Acceleration Y: " .. p2.aclY)
				imgui.text("Pushback: " .. p2.pushback)
				
				imgui.tree_pop()
			end
			if imgui.tree_node("Attack Info") then
				imgui.text("Hitstop: " .. p2.curr_hitstop .. " / " .. p2.max_hitstop)
				imgui.text("Hitstun: " .. p2.hitstun)
				imgui.text("Blockstun: " .. p2.blockstun)
				imgui.text("Juggle Counter: " .. p1.juggle)
				get_hitbox_range(cPlayer[0], cPlayer[0].mpActParam, p2)
				imgui.text("Absolute Range: " .. p2.absolute_range)
				imgui.text("Relative Range: " .. p2.relative_range)
				
				imgui.tree_pop()
			end
			if p2.chargeInfo:get_Count() > 0 then
				if imgui.tree_node("Charge Info") then
					for i=0,p2.chargeInfo:get_Count() - 1 do
						local value = p2.chargeInfo:get_Values()._dictionary._entries[i].value
						if value then
							imgui.text("Move " .. i + 1 .. " Charge Time: " .. value.charge_frame)
							imgui.text("Move " .. i + 1 .. " Charge Keep Time: " .. value.keep_frame)
						end
					end
					imgui.tree_pop()
					
				end
			end
			
			imgui.tree_pop()
		end
	
	imgui.end_window()
	end
	
	if display_projectile_info then
		-- Fireball UI
		imgui.begin_window("Projectile Data", true, 0)
		-- P1 Fireball
		if imgui.tree_node("P1 Projectile Info") then		
			for i, obj in pairs(cWork) do
				if obj.owner_add ~= nil and obj.pl_no == 0 then
					if imgui.tree_node("Projectile " .. i) then
						imgui.text("Action ID: " .. obj.mActionId)
						imgui.text("Position X: " .. obj.pos.x.v / 6553600.0)
						imgui.text("Position Y: " .. obj.pos.y.v / 6553600.0)
						imgui.text("Speed X: " .. obj.speed.x.v / 6553600.0)
						imgui.tree_pop()
					end
				end
			end
				
			imgui.tree_pop()
		end
		-- P2 Fireball
		if imgui.tree_node("P2 Projectile Info") then		
			for i, obj in pairs(cWork) do
				if obj.owner_add ~= nil and obj.pl_no == 1 then
					if imgui.tree_node("Projectile " .. i) then
						imgui.text("Action ID: " .. obj.mActionId)
						imgui.text("Position X: " .. obj.pos.x.v / 6553600.0)
						imgui.text("Position Y: " .. obj.pos.y.v / 6553600.0)
						imgui.text("Speed X: " .. obj.speed.x.v / 6553600.0)
						imgui.tree_pop()
					end
				end
			end
				
			imgui.tree_pop()
		end
		
		imgui.end_window()
	end
end

end)

1 Like
7

Alright, thanks for sharing.
What I think you will need to do is provide the model with a condensed version with the relevant data only. And this you can then much easier analyze in a timely fashion via a regular script.

In other words I don’t see any additional value a AI model can add to the analysis. Unless you feed large amounts of data into a model which in turn will look for patterns in the data that you cannot see.

On the other hand you can definitely use an LLM to help you dissect the logs and create the inputs for an analysis.

Lastly, what is sometimes done , is taking a screenshot every other frame and feeding it into a vision model for analysis. But again, this will be slow and the model needs to know what to look for and would require annotations on the fly and in a fast paced game you will miss relevant information.

I would stick to the classical log analysis and feed the data into a regular script to provide you with real time feedback about regular KPIs.

Then you can save all the data and use it later to show it to an AI looking for new insights.

Maybe? That’s my thought process so far.

8

This is something I’ve been thinking for a while now, and I now found a place where I can discuss it.

@ 12shida72

I’m interested in looking at how you transform a match into script.
What I had in mind was something more like this:

  • analyze video footage : recognize characters, combos, inputs, distance, most damaging combos, etc etc etc
  • based on video analysis create profile by player/character
  • identify player habits from video footage (ex: 'this guy always dp on wakeup)
  • find ways to improve gameplay: compare most damaging combo vs most optimal in given situation, failed situations for whiff punish, etc etc etc

Analyzing neutral seems a little out of my league for now. I get shimmy more than I’d love to, but that’s life