Capturing local variables from Coroutine's for-loop woes

http://forum.unity3d.com/threads/capturing-local-variables-from-coroutines-for-loop-woes.282623/

 

1. Hey everyone,
   
   So as I was working on some code I ran into a bug that I couldn't quite place. After some experimentation I found that the problem lies in capturing the value of a Coroutine's loop-local variable. While it works as expected when the method the binding takes place in is a "regular method" (i.e. void), it seems to always hold the same value when done from within a Coroutine. This value is always the same as the last value it held. I think a code sample to reproduce the problem is probably clearer.
   
   
2. Code (csharp):
3. 1. using UnityEngine;
   2. using System;
   3. using System.Collections;
   4. using System.Collections.Generic;
   5.  
   6. public class Test : MonoBehaviour {
   7.     public List<Action> actions = new List<Action>();
   8.  
   9.     IEnumerator Start(){ //when changed to "void Start()" the sequence runs as expected
   10.         yield return null;
   11.         for(int i=0; i<3; i++){
   12.            int_i = i;
   13.            Debug.Log("Adding DoThing() for index " + _i);
   14.            actions.Add(() => DoThing(_i));
   15.         }
   16.  
   17.         Init();
   18. }
   19.  
   20.     void DoThing(int index){
   21.         Debug.Log("Doing the thing " + index + "!");
   22.     }
   23.  
   24.     void Init(){
   25.         for(int i=0; i<actions.Count; i++){
   26.             actions[i]();
   27.         }
   28.     }
   29. }
4. The expected output here would be:
   
5. Code (csharp):
6. 1. Adding DoThing() for index 0
   2. Adding DoThing() for index 1
   3. Adding DoThing() for index 2
   4. Doing the thing 0!
   5. Doing the thing 1!
   6. Doing the thing 2!
7. But instead, when using Start() as a Coroutine I get:
   
8. Code (csharp):
9. 1.  
   2. Adding DoThing() for index 0
   3. Adding DoThing() for index 1
   4. Adding DoThing() for index 2
   5. Doing the thing 2!
   6. Doing the thing 2!
   7. Doing the thing 2!
10. Is this a known limitation of Coroutines? Why is this happening? And is there an easy fix for this problem, or would I need to refactor my code so as to completely avoid this scenario?
    
    So many questions, but looking forward to an answer! =)
    
    Thanks,
    Patrick
11.  
12. Senshi, Nov 27, 2014
13. #1
14. Stoven likes this.

15. superpig

    

    Insert bacon, receive bugfixUnity Technologies

    Joined:

    Jan 16, 2011

    Messages:

    3,284

    Interesting!
    
    I have an intuition as to what's going on but I'm not sure yet. Do you get the same result if you compile your code to a DLL with Microsoft's compiler, instead of the built-in Mono compiler?

     

    superpig, Nov 28, 2014

    #2

16. Senshi

    

    Joined:

    Oct 3, 2010

    Messages:

    405

    That's a pretty good word to describe this with! 
    
    Can you provide some steps to take to do/ test this? I've never done anything like it, and I'm not even sure if it's possible for me as I'm on OSX? If it is I'd love to give it a try though!

     

    Senshi, Nov 28, 2014

    #3

17. Senshi

    

    Joined:

    Oct 3, 2010

    Messages:

    405

    superpig said: ↑

    Interesting!
    
    I have an intuition as to what's going on but I'm not sure yet. Do you get the same result if you compile your code to a DLL with Microsoft's compiler, instead of the built-in Mono compiler?

    Click to expand...

    Oof, this board is moving fast! Quoting you real quick so you get a notification. =)

     

    Senshi, Nov 28, 2014

    #4

1. Stoven

   

   Joined:

   Jul 28, 2014

   Messages:

   171

1. Wow, that's functionality you'd expect from Javascript in terms of variables having function scope, and not block scope. I'm not sure why it's working that way in C#? Maybe IEnumerator instances from methods are implemented differently than I had originally expected.
   
   My guess now is that an IEnumerator method call returns an instance that has state saved for every local variable in a customized extension of IEnumerator that Unity provides.
   
   Example of function scope in Javascript:
   
   
2. Code (JavaScript):

1. var a = 1;
2.  
3. function four() {
4.   if (true) {
5.     var a = 4;
6.   }
7.  
8.   alert(a); // alerts '4', not the global value of '1'
9. }

1.  

1. Last edited: Nov 28, 2014
2. Stoven, Nov 28, 2014
3. #5

5. Senshi

   

   Joined:

   Oct 3, 2010

   Messages:

   405

   Stoven said: ↑

   My guess now is that an IEnumerator method call returns an instance that has state saved for every local variable in a customized extension of IEnumerator that Unity provides.

   Click to expand...

   Hmm, but then that would mean that (the first) int _i has its state saved as well, instead of being created every iteration? That doesn't sound like the intended behaviour then. Or did you mean something else?

    

   Senshi, Nov 28, 2014

   #6

1. Stoven

   

   Joined:

   Jul 28, 2014

   Messages:

   171

1. This is complete guesstimation on my part as to how Unity handles IEnumerator instances for their methods that has return type IEnumerator and of course some testing of printing the type returned to the screen.
   
   The way your method is probably interpreted after a call to Start() by Unity
   
   (Note: the class is not tested btw)
   
   
2. Code (CSharp):

1.  
2. public class c++_iterator<T> : IEnumerator where T : struct
3. {
4.  
5.     private int i;
6.     private int _i;
7.     private int chunk = 0;
8.     private YieldInstruction _yield;
9.     public object Current { get { return _yield; } }
10.  
11.     // ... code to set values, except chunk
12.  
13.     public bool MoveNext()
14.     {
15.  
16.         switch(chunk)
17.         {
18.  
19.             case 0:
20.                 _yield = null;
21.                 chunk = 1;
22.             return true;
23.      
24.             case 1:
25.      
26.                 for(i=0; i<3; i++)
27.                 {
28.                     _i = i;
29.                     Debug.Log("Adding DoThing() for index " + _i);
30.                     actions.Add(() => DoThing(_i));
31.                 }
32.          
33.                 Init();
34.                 chunk = 2;
35.             return false;
36.      
37.         }
38.  
39.  
40.         return false;
41.     }
42.  
43.     public void Reset(){ throw new InvalidOperationException(); }
44.  
45.     public void Dispose()
46.     {
47.         // possibly some kind of disposal for whatever types needed it
48.     }
49. }
50.  

1. Since the Anonymous method assignment from actions.Add(() => DoThing(_i)) is using the reference from the specially constructed class, and since it isn't called yet, all DoThing(_i) calls will get the same _i value once the actions execute their functions.
   
   Again, this is purely a guess!!!
2.  

1. Last edited: Nov 29, 2014
2. Stoven, Nov 28, 2014
3. #7

5. superpig

   

   Insert bacon, receive bugfixUnity Technologies

   Joined:

   Jan 16, 2011

   Messages:

   3,284

   That's pretty much my guess as well 
   
   Note that this isn't something Unity does - it's something the compiler does. That's why I'm wondering whether you get the same results using a different compiler; strictly speaking it shouldn't be necessary to 'lift' the _i variable out to class scope because it's not accessed on either side of a yield, but a) maybe the compiler isn't smart enough to figure that out and b) maybe it shouldn't because it'd be even more confusing to have some variables behave this way and others not.
   
   If you're on Mac, you won't be able to run the Microsoft compiler... you could try getting the most modern version of Mono/Xamarin Studio and building it in that. Otherwise, I'll try and remember to try this code out when I next boot into Windows. Unless @Stoven beats me to it 

    

   superpig, Nov 28, 2014

   #8

   Stoven likes this.

1. A.Killingbeck

   

   Joined:

   Feb 21, 2014

   Messages:

   456

1. Code (CSharp):

1. IEnumerator Start(){ //when changed to "void Start()" the sequence runs as expected
2.         yield return null;
3.         int _i = 0;
4.         for(int i=0; i<3; i++){
5.  
6.             Debug.Log("Adding DoThing() for index " + _i);
7.             actions.Add(()=>
8.                         {
9.                 DoThing(_i);
10.                 _i++;
11.             });
12.         }
13.        
14.         Init();
15.     }

1.  

1. A.Killingbeck, Nov 28, 2014
2. #9
3. Senshi likes this.

4. Senshi

   

   Joined:

   Oct 3, 2010

   Messages:

   405

   Stoven said: ↑

   Again, this is purely a guess!!!

   Click to expand...

   Ah, but it is a guess that would make sense -- and as I'm typing this, approved by @superpig himself!
   
   

   superpig said: ↑

   If you're on Mac, you won't be able to run the Microsoft compiler... you could try getting the most modern version of Mono/Xamarin Studio and building it in that. Otherwise, I'll try and remember to try this code out when I next boot into Windows. Unless @Stoven beats me to it 

   Click to expand...

   I'm trying to remember if I have an IDE installed on my Windows partition right now. Perhaps I can take a peek tonight, but as I'm new to this whole compiling-to-DLL thing I'd appreciate it if either of you could give a whirl as well. =)
   
   Thanks!

    

   Senshi, Nov 28, 2014

   #10

1. Senshi

   

   Joined:

   Oct 3, 2010

   Messages:

   405

1. A.Killingbeck said: ↑

1. Code (CSharp):

1. IEnumerator Start(){ //when changed to "void Start()" the sequence runs as expected
2.         yield return null;
3.         int _i = 0;
4.         for(int i=0; i<3; i++){
5.  
6.             Debug.Log("Adding DoThing() for index " + _i);
7.             actions.Add(()=>
8.                         {
9.                 DoThing(_i);
10.                 _i++;
11.             });
12.         }
13.      
14.         Init();
15.     }

1. Click to expand...

Interesting; thanks for posting! Unfortunately, while this works for this exampe case, my real-life scenario is a bit more complicated. =/ I guess I could make it work like this, but it would become a bit unwieldly to maintain I'm afraid.

 

1. Senshi, Nov 28, 2014
2. #11

4. Stoven

   

   Joined:

   Jul 28, 2014

   Messages:

   171

   So far, I've only learned to use C# with Unity and nowhere else XD
   
   I'm not 100% familiar enough with .Net to do something like what is requested, unfortunately. I spend most of my time studying the Unity documentation, discussions in the discussion section of the forums and working on my game XP
   
   So methods that return an IEnumerator in .Net define this behavior, not specifically Unity? I was under the assumption that Unity was responsible for the custom c++_iterator<> instance... unless there are other workings happening under-the-hood with the compiler that I'm not quite understanding for this scenario.
   
   I could have sworn that the representitive for this Unity video about Coroutines said that it was behavior specific to Unity, but upon rewatching the part I thought I remembered hearing it, he simply states (around 12:30 to 12:40) that "Calling [the] IEnumerator [function] returns an IEnumerator which is basically a pointer to the start of the function" - he doesn't explicitly state that this behavior is Unity-specific, so I'm now assuming this is a .Net thing?
   
   Sorry if I'm simply adding confusion here. I'm very curious about this myself. Please correct me if I'm misinterpreting your statement.
   
   Edit: Listened again. 11:04 in makes it a bit more clear (I glossed over it because I was making an attempt to find a specific part of the speech). So the C# compiler is responsible for this, as you've said! So it is a .Net thing... very interesting.
   
   Edit 2: How did I miss this in the .Net specification???
   
   "Technical Implementation
   
   Although you write an iterator as a method, the compiler translates it into a nested class that is, in effect, a state machine. This class keeps track of the position of the iterator as long the For Each...Next or foreach loop in the client code continues.
   
   To see what the compiler does, you can use the Ildasm.exe tool to view the Microsoft intermediate language code that is generated for an iterator method.
   
   When you create an iterator for a class or struct, you do not have to implement the whole IEnumerator interface. When the compiler detects the iterator, it automatically generates the Current, MoveNext, and Dispose methods of the IEnumerator or IEnumerator<T>interface."

    

   Last edited: Nov 28, 2014

   Stoven, Nov 28, 2014

   #12

   Senshi likes this.

1. lordofduct

   

   Joined:

   Oct 3, 2011

   Messages:

   2,159

1. @Stoven - this is a mono/.net thing, not a unity thing, I'll confirm that for you.
   
   And yes, it works pretty much how Stoven described it. The method gets turned into its own anonymous class in which all the local variables of the method actually are fields of this class. That is how their states are saved, and this model isn't perfect. If you mix all sorts of anonymous syntax sugar together, things like this happen. You're mixing anonymous/lambda functions with iterator methods, and it doesn't handle it the way you expect.
   
   If you have to have these lamdas inside the coroutine. Pull that bit of code out into its own method. The compiler instead of creating state fields for all those local variables will instead just call the method, and it will work as expected.
   
   
2. Code (csharp):

1.  
2. using UnityEngine;
3. using System.Collections.Generic;
4.  
5. public class CoroutineTestScript : MonoBehaviour {
6.  
7.     private List<System.Action> _actions = new List<System.Action>();
8.  
9.     System.Collections.IEnumerator Start ()
10.     {
11.         yield return null;
12.  
13.         FillUpActions();
14.  
15.         Init();
16.     }
17.  
18.     private void FillUpActions()
19.     {
20.         for (int i = 0; i < 3; i++)
21.         {
22.             int j = i;
23.             Debug.Log("Adding DoThing() for index " + i.ToString());
24.             _actions.Add(() =>
25.             {
26.                 DoThing(j);
27.             });
28.         }
29.     }
30.  
31.     private void DoThing(int i)
32.     {
33.         Debug.Log("Doing the thing " + i.ToString() + "!");
34.     }
35.  
36.     private void Init()
37.     {
38.         for(int i =0; i < _actions.Count; i++)
39.         {
40.             _actions[i]();
41.         }
42.     }
43.  
44. }
45.  

1.  

1. lordofduct, Nov 28, 2014
2. #13
3. Senshi and Stoven like this.

4. superpig

   

   Insert bacon, receive bugfixUnity Technologies

   Joined:

   Jan 16, 2011

   Messages:

   3,284

   Stoven said: ↑

   So methods that return an IEnumerator in .Net define this behavior, not specifically Unity?

   Click to expand...

   Methods that return IEnumerator and use yield statements, yes. I wrote an article about how it all works a few years ago.

    

   superpig, Nov 28, 2014

   #14

   Senshi and Stoven like this.

1. Senshi

   

   Joined:

   Oct 3, 2010

   Messages:

   405

1. Thanks everyone for the helpful links and explanations! Putting the for-loop in its own method indeed solves my issue, as does just placing the lambda in its own method only (which is more suitable for my scenario). But that got me thinking: Why not just keep going with this, and scope all the things! 0/ ? (Or in non-meme-speak: I'd rather keep this functionality local; without creating a specific top-level function for it.)
   
   
2. Code (csharp):

1.  
2. using UnityEngine;
3. using System;
4. using System.Collections;
5. using System.Collections.Generic;
6.  
7. public class CoroutineTestScript : MonoBehaviour {
8.     List<System.Action> actions = new List<System.Action>();
9.  
10.     IEnumerator Start (){
11.         Action<int> AddAction = new Action<int>((int index) => {
12.             Debug.Log("Adding DoThing() for index " + index);
13.             actions.Add(() => DoThing(index));
14.         }); //local method with its own scope!
15.  
16.         yield return null;
17.    
18.         for (int i = 0; i < 3; i++){
19.             AddAction(i);
20.             //or if you want to get *really* conservative:
21.             new Action<int>((int index) => {
22.                 Debug.Log("Adding DoThing() for index " + index);
23.                 actions.Add(() => DoThing(index));
24.             })(i);
25.         }
26.  
27.         Init();
28.     }
29.  
30.     private void DoThing(int i){
31.         Debug.Log("Doing the thing " + i.ToString() + "!");
32.     }
33.  
34.     private void Init(){
35.         for(int i=0; i<actions.Count; i++){
36.             actions[i]();
37.         }
38.     }
39. }
40.  

1. So yeah, problem solved at least! I am still curious about the results of a Microsoft compiler compiled DLL, but reading this I would expect it to yield (pun not intended) the same results.
   
   Anyway, thanks again to everyone! This was a fun thing to explore and I learned a bunch! =)