How useful is this test tool to you? What bugs did you find or do you hope to find this way? How often and long do you run it?
I'm asking, because I guess, it probably takes ages to traverse some kind of dungeon. If there are special sequences, the tool will most likely never find them (like find key, open door, kill boss), right?
I've done some small games in my spare time, but Unit-Testing was either completely useless or way overkill. But I guess, the larger your game gets, the more important it will be.
But the the work involved is so huge, that I think it's not worth it for indy games or small studios. If you look at the test setup for AAA-Games, like League of Legends [1], the work involved is mind-blowing...
noric-bot has been extremely useful as a test tool, if a bit narrow due to just spamming random cmds. To your point, the bot in its present form can't automatically run a dungeon or verify edge cases.
A previous version of the server codebase was written in ruby[1]. The playable branch of that repo has a combat demo that's a bit fun, however the performance is so terrible that I shelved the whole codebase years ago.
As a long-term vision I'd like to launch an indie MMORPG that supports at least 1,000 players per synchronous, single-threaded server. It's an item/equipment focused game. Each player may have hundreds of items with a total of thousands of affects. noric-bot helps me stay on track with that goal.
What bugs did you find or do you hope to find this way?
The server is an actor-based "distributed system" using Akka. I only ever run it on one jvm but in theory it could be clustered. The bot helps find performance bottlenecks and actor-related edge cases.
The server runs at 4fps and Very Bad things happen if the 250ms frame time budget is routinely exceeded. The bots use a special password to spawn "fully loaded" player mobs with hundreds of items and ~1k affects. This helps me see bottlenecks like saving/persistence, I/O, Akka mailboxes growing, time spent processing player cmds vs moving the simulation forward naturally, etc.
Sometimes the bot will find logical bugs. This is usually in the context of assertions I've added that the bots will trigger.
I'll routinely write fake commands that bots will spam to execute scenarios that otherwise would be unreachable (if only due to lacking gameplay functionality at this early stage).
Often I'll just run the bots for fun and see how the game feels with a lot of other "players" running around :-)
I'm asking, because I guess, it probably takes ages to traverse some kind of dungeon. If there are special sequences, the tool will most likely never find them (like find key, open door, kill boss), right?
I've done some small games in my spare time, but Unit-Testing was either completely useless or way overkill. But I guess, the larger your game gets, the more important it will be.
But the the work involved is so huge, that I think it's not worth it for indy games or small studios. If you look at the test setup for AAA-Games, like League of Legends [1], the work involved is mind-blowing...
[1] https://technology.riotgames.com/news/automated-testing-leag...