I think the issue with z-indexes is that you're trying to use them like a sort of actor-based depth buffer, which (to the best of my knowledge) they're not. There are a couple good solutions I can think of. Keep in mind both solutions basically add an entire dimension to the existing 2D coordinate system. This simple-sounding problem has quite a bit of depth to it.
1. Painter's Algorithm. I actually wrote about implementing this
over here in an isometric game. It could work for you, too. You would only need "scene height" number of buckets, and each actor is placed into the bucket corresponding to its y-coordinate. The advantage is that a "draw stack" (as I call it in the post) based solution handles a lot of actors efficiently. The drawback is that you're basically writing a little renderer that sits on top of Stencyl's (using the Drawing events). This means letting go of some of the things Stencyl makes easy for you (like some of the tweens), or implementing it yourself.
2. An alternate solution would be to build off of
Mr. Wagoner's solution. At the very end of the post he gives you a "z-index update" block. What you'd want to do is make a similar event. It would look something like:
For every y coordinate that is on screen
For every actor with that y coordinate
Move actor to the top within its layer
End For
End For
Note that we only loop through the y-coordinates that are currently on screen. Don't loop through the y-coordinates that are out of the camera's bounds. The simplest solution is to call this event whenever the y-coordinates of any actor change, but we'll want to avoid that (read on).
The advantage is that you're not writing your own renderer. The drawback is that you potentially have a major bottleneck. This is a double loop, and could be called often, especially if you have a lot of moving actors.
If you (or anyone else finding this thread) wants to try implementing that idea, what I'd suggest you do is put the reordering event in a scene behavior. Whenever an actor moves, set a boolean "update z-index?" to TRUE. In an "Drawing" event within a scene behavior, run the following:
If update z-index? then
Call the z-index update event
Set update z-index? to FALSE
End If
Then, whenever an actor moves, do this: (Put this code in an actor behavior. Attach it to each actor that moves)
If new y-coordinate != old y-coordinate then
Set update z-index? to TRUE
End If
This ensures the update runs at most once per screen update. (So, at most 60 times per second).