It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server-client arch) a bunch of models of various difficulty levels to let players practice. Could we even get models tuned to specific campaign levels? I wonder how costly the creation of Pluto was.
So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly - https://broodbench.com/ and https://news.ycombinator.com/item?id=49766966
How do the best humans do against the LLMs?
This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
I did a 48 hour live stream - Astra cleared in 43 hours.
Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.
It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?
Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.
They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.
I did a live stream where Astra played Pluto. Astra went 0 - 1000 https://x.com/kaimcpheeters/status/2106782151622504911?s=20
I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server-client arch) a bunch of models of various difficulty levels to let players practice. Could we even get models tuned to specific campaign levels? I wonder how costly the creation of Pluto was.
Any thoughts on building this for Starcraft 2?