Could you perhaps explain it a little more? I can't seem to grasp what you're telling me.
Here are some basics. Stencyl is built on top of OpenFL, so it's helpful to understand a little about how OpenFL handles these kinds of things.
OpenFL implements
Sprites, which you can take as anything visible or tangible in the scene. These include images, Actors, and even Layers. So you can think of Layers as large, invisible Sprites
Every Sprite can have child Sprites. If you're not familiar with this analogy, think of paper cut-out animation (like in South Park) where different visuals are glued together. In this example, the body is the parent Sprite, and the eyes, mouth, clothes, etc. the child Sprites. This hierarchy can be multiple levels deep. For instance, the eyes are child Sprites of the body, and each eye has a child Sprite of its own--the pupil.
There are a lot of advantages to this architecture. If you move the parent Sprite, the children move along with it. If you hide the parent Sprite, the children all hide as well.
So what you need to visualize is that every Sprite has a list of children, and z-index refers to the child's position in this list. Since a Layer is just a Sprite, then each actor's z-index is its position in the Layer's list of children.
Here are two approaches to implementing z-index based depth. You can continuously sort the list of children. This is the approach your behavior you're using takes. It's the simpler solution, but comes at the cost of not scaling very well when the number of actors gets large. The other approach is to insert actors into this list as they're created, and change their position as they move. This approach is more complicated to program, but will result in faster z-ordering.
The problem you're running into with the trees is that there are a lot of actors to continuously sort. So you either need to implement a more efficient sorting routine, or reduce the number of trees.
Keep in mind you don't actually have to delete trees to reduce their number. All you really need to do is find a way to not sort every tree. I thought of another solution that might work. I noticed your behavior loops through every layer. What if you broke your map up into small, screen-sized sections, and assigned a layer to each section. Modify the behavior so that instead of sorting every actor on every layer, it only sorts every actor on the visible layer.