Task process ending before finishing all the work

I've been having trouble running multiple tasks with heavy operations. It seems as if the task processes is killed before all the operations are complete.

The code here is an example code I used to replicate the issue. If I add something like Debug.Write(), the added wait for writing fixes the issue. The issue is gone if I test on a smaller sample size too. The reason there is a class in the example below is to create complexity for the test. The real case where I encountered the issue first is too complicated to explain for a post here.

public static class StaticRandom
{
    static int seed = Environment.TickCount;

    static readonly ThreadLocal<Random> random =
        new ThreadLocal<Random>(() => new Random(Interlocked.Increment(ref seed)));

    public static int Next()
    {
        return random.Value.Next();
    }

    public static int Next(int maxValue)
    {
        return random.Value.Next(maxValue);
    }

    public static double NextDouble()
    {
        return random.Value.NextDouble();
    }
}

// this is the test function I run to recreate the problem:
static void tasktest()
{
    var testlist = new List<ExampleClass>();
    for (var index = 0; index < 10000; ++index)
    {
        var newClass = new ExampleClass();
        newClass.Populate(Enumerable.Range(0, 1000).ToList());
        testlist.Add(newClass);
    }
    var anotherClassList = new List<ExampleClass>();
    var threadNumber = 5;

    if (threadNumber > testlist.Count)
    {
        threadNumber = testlist.Count;
    }

    var taskList = new List<Task>();
    var tokenSource = new CancellationTokenSource();
    CancellationToken cancellationToken = tokenSource.Token;

    int stuffPerThread = testlist.Count / threadNumber;
    var stuffCounter = 0;
    for (var count = 1; count <= threadNumber; ++count)
    {
        var toSkip = stuffCounter;
        var threadWorkLoad = stuffPerThread;
        var currentIndex = count;

        // these ifs make sure all the indexes are covered
        if (stuffCounter + threadWorkLoad > testlist.Count)
        {
            threadWorkLoad = testlist.Count - stuffCounter;
        }
        else if (count == threadNumber && stuffCounter + threadWorkLoad < testlist.Count)
        {
            threadWorkLoad = testlist.Count - stuffCounter;
        }

        taskList.Add(Task.Factory.StartNew(() => taskfunc(testlist, anotherClassList, toSkip, threadWorkLoad),
            cancellationToken, TaskCreationOptions.None, TaskScheduler.Default));
        stuffCounter += stuffPerThread;
    }

    Task.WaitAll(taskList.ToArray());
}

public class ExampleClass
{
    public ExampleClassInner[] Inners { get; set; }

    public ExampleClass()
    {
        Inners = new ExampleClassInner[5];
        for (var index = 0; index < Inners.Length; ++index)
        {
            Inners[index] = new ExampleClassInner();
        }
    }

    public void Populate(List<int> intlist) {/*adds random ints to the inner class*/}


    public ExampleClass(ExampleClass copyFrom)
    {
        Inners = new ExampleClassInner[5];
        for (var index = 0; index < Inners.Length; ++index)
        {
            Inners[index] = new ExampleClassInner(copyFrom.Inners[index]);
        }
    }

    public class ExampleClassInner
    {
        public bool SomeBool { get; set; } = false;
        public int SomeInt { get; set; } = -1;

        public ExampleClassInner()
        {
        }

        public ExampleClassInner(ExampleClassInner copyFrom)
        {
            SomeBool = copyFrom.SomeBool;
            SomeInt = copyFrom.SomeInt;
        }
    }
}

static int expensivefunc(int theint)
{ 
/*a lot of pointless arithmetic and loops done only on primitives and with primitives, 
just to increase the complexity*/
    theint *= theint + 1;
    var anotherlist = Enumerable.Range(0, 10000).ToList();
    for (var index = 0; index < anotherlist.Count; ++index)
    {
        theint += index;
        if (theint % 5 == 0)
        {
            theint *= index / 2;
        }
    }
    var yetanotherlist = Enumerable.Range(0, 50000).ToList();
    for (var index = 0; index < yetanotherlist.Count; ++index)
    {
        theint += index;
        if (theint % 7 == 0)
        {
            theint -= index / 3;
        }
    }
    while (theint > 8)
    {
        theint /= 2;
    }

    return theint;
}

// this function is intentionally creating a lot of objects, to simulate complexity
static void taskfunc(List<ExampleClass> intlist, List<ExampleClass> anotherClassList, int skip, int take)
{
    if (take == 0)
    {
        take = intlist.Count;
    }
    var partial = intlist.Skip(skip).Take(take).ToList();
    for (var index = 0; index < partial.Count; ++index)
    {
        var testint = expensivefunc(index);
        var newClass = new ExampleClass(partial[index]);
        newDna.Inners[StaticRandom.Next(5)].SomeInt = testint;
        anotherClassList.Add(new ExampleClass(newClass));
    }
}

The expected result is that the list anotherClassList will be the same size as testlist and this happens when the lists are smaller or the complexity of the task operations is smaller. However, when I increase the volume of operations, the anotherClassList has a few indexes missing and sometimes some of the indexes in the list are null objects.

Example result:

enter image description here

Why does this happen, I have Task.WaitAll?

2 answers

  • answered 2019-01-11 05:55 Michael Randall

    Your problem is its just not thread safe, you just cant add to a list in a multi threaded environment and expect it to play nice.

    One way is to use lock or a thread safe collection, but i feel this all should be refactored (my OCD is bing`ing off all over the place)

    private static object _sync = new object();
    
    ...
    
    private static void TaskFunc(List<ExampleClass> intlist, List<ExampleClass> anotherClassList, int skip, int take)
    {
    
       ...
    
       var partial = intlist.Skip(skip).Take(take).ToList();
    
       ...
    
       // note that locking here will likely drastically decrease any performance threading gain
       lock (_sync)
       {
          for (var index = 0; index < partial.Count; ++index)
          {
             // this is your problem, you are adding to a list from multiple threads
             anotherClassList.Add(...);
          }
       }
    
    }
    

    In short, i think you need to better thinking about the threading logic of your method, identify what you are trying to achieve, and how to make it conceptually thread safe (while keeping your performance gains)

  • answered 2019-01-11 07:19 Sadem

    After TheGeneral enlightened me that Lists are not thread safe, I changed the List to which I was adding in a thread, to an Array type and this fixed my issue.