Plummeting performance

Nomosoft

  • Posts: 122
I have a problem that only seems to have shown up since I upgraded to the release build of 2.0. The game I am working on seems to have developed a strange resource leak. The game runs fine for a few minutes, then starts to bog down, and eventually the frame rate hits single digits. The problem is worse in the standalone player, but it shows up in the browser, too.

My first thought was that I had actors not being properly disposed of. So, I wrote a behavior to display every single actor type and a running count of how many of each actor exist in the current scene. That does not seem to be the problem, all created actors are being destroyed, and the numbers all fall back to zero as actors leave the screen/are destroyed by bullets/whatever.

Next, I conjectured that I had an out-of-control behavior doing something silly, so I wrote a profiling behavior and timed every single behavior being used in my game. Although I do have a few resource hogs, they are all fairly stable. I don't have any behaviors that expand to gobble up more and more cycles as they run. So, that also doesn't explain the creeping slowdown.

My actors are being properly disposed of, and my behaviors are crunching along at a steady pace, but my game keeps slowing down, anyway. It _seems_ to be centered around creating and destroying actors. I say _seems_ because I really haven't been able to pinpoint the problem, but I can sort of coax it into happening by creating and destroying a ton of actors. It isn't the actual creating and destroying that eats up the time, though. The slowdown continues even after the actors have been created or disposed of. The game behaves as if there are many actors in play, even when the actual count shows few or none.

Here's an illustration of the problem:
I have a character that produces a burst of bullets. The frame rate takes a hit when the bullets are created, but jumps right back to 60 FPS as soon as the creation is complete. The bullets go zipping off the screen and are destroyed. After a few dozen bursts, I notice that the frame rate is taking more of a hit, and not recovering as quickly. Instead of dropping to 50 FPS when the bullets are created, it drops into the 30s, and takes a couple of seconds to get back to 50+. Then, after a few dozen more bursts, the frame rate drops all the way to single digits every time there is a burst, and takes a long time to recover, and never actually recovers all the way to FPS, instead hitting the 20s or 30s. Essentially, the game is acting like I have dozens and dozens of actors in play, even though the actual count is zero once the bullets fly off and are destroyed. It's as if every time I created an actor of type "Bullet," every "Bullet" that I ever created (and destroyed) is being processed. The "Update" and "Render" times jump through the roof when the new burst is created, then  settle down some once the bullets fly off the screen. This gets worse and worse as the game plays on. At the beginning, the increase in Update and Render time is not even noticeable.

I have a game uploaded under my account with "Trimmed" in the name. I don't want to make it public, but the Stencyl team are welcome to look at it. It's not perfect, but it's the easiest way I could fine to demonstrate the issue. The problem is actually worse in a "real" game, where multiple types of actor are being created and destroyed, and multiple actors are shooting bullets at one another. It works like this: Load the game, and hold your Shift key down right from the beginning. Holding Shift shuts off the bullet firing behavior. While holding Shift, hit the tilde and look at the Update, Flixel, and Render times. On my computer, they are something like 0ms Update, 6ms Render, 6ms Flixel, 8ms Flash, 14ms total, 60FPS. Let go of the Shift key and let the ship spew bullets for a few minutes. Now, hit Shift, again, and notice that Update, Render, etc, stay much higher than before and the FPS drops (on my computer) by a solid 10+ FPS. Repeat the process, and watch how Update, Render, etc, creep even higher, as the framerate bogs down. The yellow text on the left shows a running count of all actors in the game. You'll notice the performance degrades despite the number of actors settling back to 0 every time. I've tried this on three different PCs with completely different hardware (probably different versions of the Flash player, too, but I didn't check) and they all show the same problem. I did not notice this happening pre-2.0.

As I mentioned, it takes longer for the effect to show up in the browser (Firefox, for me), so leave the game sit a few minutes if you want to see what I'm talking about.

Rob

  • *
  • Posts: 1268
How do you dispose of those bullets?

If possible, can you make this a simple game and put it up on forge for us to investigate? You can just spawn a bunch of these bullets at a periodic rate and kill them.
GOLDEN RULE #1 : SAVE YOUR GAMES FREQUENTLY
IOS/XCODE DEBUGGING
http://www.stencyl.com/help/view/xcode-ios-troubleshoot/
WATCH STENCYLWORKS VIDEO TUTORIAL BY SUNRISEKINGDOM
http://www.youtube.com/watch?v=KDfRfjzr9j4&feature=channel_video_title

coleislazy

  • *
  • Posts: 2607
Are you using recycled actors? If so, try it with normal actors and see if there is a difference. I was having a problem with recycled actors as well that cleared up when I switch to the normal create/kill blocks.

Nomosoft

  • Posts: 122
How do you dispose of those bullets?

If possible, can you make this a simple game and put it up on forge for us to investigate? You can just spawn a bunch of these bullets at a periodic rate and kill them.

I've tried both killing and recycling the bullets. Recycling actually caused the problem to become significantly worse. I didn't do too much experimenting in that direction, though.

Well, what the hell. I didn't want to make it public in such a terrible state, but here's a link to some actual gameplay: http://www.stencyl.com/game/play/12503 I've sped up bullet production to make the problem show up, quicker. Ignore the debug "Upgrading to xx" message, it is doing literally nothing. As you can see, the framerate starts out fine, but rapidly plummets and never recovers, despite actor numbers remaining fairly steady.

Coleislazy - you replied as I was typing this. I address the recycled actor issue in the paragraphs, above.

Rob

  • *
  • Posts: 1268
Can you confirm that we have a problem (fps dropping to single digits) with recycled actor and not regular actor? Can you give some data on both instances?
GOLDEN RULE #1 : SAVE YOUR GAMES FREQUENTLY
IOS/XCODE DEBUGGING
http://www.stencyl.com/help/view/xcode-ios-troubleshoot/
WATCH STENCYLWORKS VIDEO TUTORIAL BY SUNRISEKINGDOM
http://www.youtube.com/watch?v=KDfRfjzr9j4&feature=channel_video_title

Rob

  • *
  • Posts: 1268
I've created a simple game that:

1. create 30 recycled actors on the screen every second.
2. kill them in 1 second.

Using flash player and turning on debug with ~, I see my fps constant at around 40 (with quick dips during spawn) but I do not see any degrade over time.

I've win-xp with 1GB ram.

Please see how this game run for you. Possibly change it to show your degrading fps case.

Just stick this game under stencylworks/games folder.
GOLDEN RULE #1 : SAVE YOUR GAMES FREQUENTLY
IOS/XCODE DEBUGGING
http://www.stencyl.com/help/view/xcode-ios-troubleshoot/
WATCH STENCYLWORKS VIDEO TUTORIAL BY SUNRISEKINGDOM
http://www.youtube.com/watch?v=KDfRfjzr9j4&feature=channel_video_title

Nomosoft

  • Posts: 122
I've created a simple game that:

1. create 30 recycled actors on the screen every second.
2. kill them in 1 second.

Using flash player and turning on debug with ~, I see my fps constant at around 40 (with quick dips during spawn) but I do not see any degrade over time.

I've win-xp with 1GB ram.

Please see how this game run for you. Possibly change it to show your degrading fps case.

Just stick this game under stencylworks/games folder.

My machine is a Core-i3 2120 @ 3.30GHz, 8 gigs of ram, and a Radeon 6570 graphics card.

The slowdown is less severe in your example, but it is definitely there. When I started your program four of five minutes ago, I was getting 60 FPS, with an update time of ~1ms, a render of ~1ms, and a Flixel of ~3ms. Now, I am getting 58-59 FPS with an update of ~6ms, a render of ~4ms, and a Flixel of ~10ms. It is slowly climbing. I'm pretty sure if I leave it sit while I go to the bank, it will be significantly worse when I get back.

With more things going on, more behaviors attached to the actors, and so on in my real game situation, the slowdown comes much faster. Although it is not as sever in this simplified test program, it is certainly there. In fact, every "create" action is taking much longer, now, than when I first opened your program. Now, instead of a quick flicker while the old actors are destroyed and 30 new ones are created, there is a distinct wiping of the screen, the FPS drops to ~15, and then takes a couple of frames to get back. It's getting worse as I watch. Maybe I should use Fraps or something to record the effect.

Rob

  • *
  • Posts: 1268
Did you say it eventually dropped to 15 fps? If so, how long does it take for this to happen?

What if you swapped in regular actor instead of recycled actors?

Thanks for investigating this. These are my cup of tea also. Get me some good stats. I'll also try on my end.
GOLDEN RULE #1 : SAVE YOUR GAMES FREQUENTLY
IOS/XCODE DEBUGGING
http://www.stencyl.com/help/view/xcode-ios-troubleshoot/
WATCH STENCYLWORKS VIDEO TUTORIAL BY SUNRISEKINGDOM
http://www.youtube.com/watch?v=KDfRfjzr9j4&feature=channel_video_title

Nomosoft

  • Posts: 122
Did you say it eventually dropped to 15 fps? If so, how long does it take for this to happen?

What if you swapped in regular actor instead of recycled actors?

Thanks for investigating this. These are my cup of tea also. Get me some good stats. I'll also try on my end.

It was spiking down to 15 FPS while creating new actors. When it started, the FPS barely blipped when new actors were created. I just got back from the bank, and the program is down to 18 FPS, (~42ms update, ~52ms Flixel), dropping to 3 FPS with ~300ms update and ~ 270ms Flixel when new actors are created.

Rob

  • *
  • Posts: 1268
I am going to give it a a try here using my game above with absolutely NO changes.

You mean after more than half an hour running, it would degrade like this:

START

60 fps (15 fps during spawn spike)

30 MINUTES LATER

18 fps (3 fps during spawn spike)
GOLDEN RULE #1 : SAVE YOUR GAMES FREQUENTLY
IOS/XCODE DEBUGGING
http://www.stencyl.com/help/view/xcode-ios-troubleshoot/
WATCH STENCYLWORKS VIDEO TUTORIAL BY SUNRISEKINGDOM
http://www.youtube.com/watch?v=KDfRfjzr9j4&feature=channel_video_title

Nomosoft

  • Posts: 122
I am going to give it a a try here using my game above with absolutely NO changes.

You mean after more than half an hour running, it would degrade like this:

START

60 fps (15 fps during spawn spike)

30 MINUTES LATER

18 fps (3 fps during spawn spike)

I am not 100% sure of the time line, since I was away from my computer, but, essentially, yes. I started your game and let it sit. Some time later, 18FPS.

Edit: In case I didn't make it clear, the degredation accelerates as time goes on. It takes X minutes to lose the first two or three FPS, then only X more minutes to lose the next 10-20 FPS.

Rob

  • *
  • Posts: 1268
Thanks. I'll try both regular and recycle actors and measure them for 0.5 hours each and post my observation in a couple of hours.

Spawn 30 identical actor randomly every 1 second
Kill same 30 actors in 1 second

----------------------------------------
RECYCLED ACTORS

START 40-50 fps  (20-30 fps spike)
          flash-10-win.exe mem usage -> 30K

END (AFTER 20 MINUTES)  18 fps (4 fps spike)
           flash-10-win.exe mem usage -> 276K

----------------------------------------

REGULAR ACTORS

START 40-50 fps  (20-30 fps spike)
flash-10-win.exe mem usage -> 30K

END (AFTER 20 MINUTES)  19 fps (11 fps spike)
flash-10-win.exe mem usage -> 75K


« Last Edit: May 03, 2012, 06:29:33 pm by Rob »
GOLDEN RULE #1 : SAVE YOUR GAMES FREQUENTLY
IOS/XCODE DEBUGGING
http://www.stencyl.com/help/view/xcode-ios-troubleshoot/
WATCH STENCYLWORKS VIDEO TUTORIAL BY SUNRISEKINGDOM
http://www.youtube.com/watch?v=KDfRfjzr9j4&feature=channel_video_title

Nomosoft

  • Posts: 122
Thanks. I'll try both regular and recycle actors and measure them for 0.5 hours each and post my observation in a couple of hours.

RECYCLED ACTORS

START 40-50 fps  (20-30 fps spike)

END (AFTER 30 MINUTES) ---

REGULAR ACTORS

START ---

END (AFTER 30 MINUTES) ---

I left your creation loop alone, but I changed the Kill to a Recycle, and am running the program, again.

Rob

  • *
  • Posts: 1268
That's great. We should try:

1. create actor, kill actor
2. create actor, recycle actor
3. create recycle actor, kill actor
4. create recycle actor, recycle actor

I am also monitoring the flash-10-win.exe mem usage as it creeps up.
GOLDEN RULE #1 : SAVE YOUR GAMES FREQUENTLY
IOS/XCODE DEBUGGING
http://www.stencyl.com/help/view/xcode-ios-troubleshoot/
WATCH STENCYLWORKS VIDEO TUTORIAL BY SUNRISEKINGDOM
http://www.youtube.com/watch?v=KDfRfjzr9j4&feature=channel_video_title

Nomosoft

  • Posts: 122
That's great. We should try:

1. create actor, kill actor
2. create actor, recycle actor
3. create recycle actor, kill actor
4. create recycle actor, recycle actor

I am also monitoring the flash-10-win.exe mem usage as it creeps up.

I'm about twenty minutes into running the create recycled/recycle version, and the performance drop is negligible, so far. The thing is, I tried that in my game (all my actors share a single dying behavior, so it's easy) and it did not help. I am going to put "kill" back and see what happens.