Copy data from one list to another takes long time

I have two lists: List A (is a Dictionary)and List B(Data), each of them having two common properties C1 and C2. I'm trying to bring the List1 data into list 2 using linq code below but this is taking almost 9 min to execute the loop(35k records). Please assist on how to read this two list data into one list effectively without performance issue. I have appended C1 and C2 to single property C1C2.

List A

C1|C2 |desc_1 |desc_2 |DataType

0 |1 |0 |1 |TEXT

0 |2 |0 |2 |TEXT

0 |3 |0 |3 |TEXT

0 |4 |0 |4 |TEXT

22 |1 |22 |1 |NUMERIC

22 |2 |22 |2 |NUMERIC

.

List B

App_Id |C1 |C2 |value

12 |0 |1 |55R76

12 |0 |2 |TEST PROEPRTIES

12 |0 |3 |CHOPPER

12 |0 |4 |TEST MD

13 |0 |1 |9866FGG078

13 |0 |2 |TITLE2

13 |22 |1 |TODAY

13 |22 |2 |TEST12344

This is my code so far :

var query = from x in ListB
                            join y in ListA
                                on x.C1C2 equals y.C1C2
                            select y;
                //Update metadata from ListA to ListB
                foreach (var x in query)
                {
                    var itemToChange = ListB.FirstOrDefault(d => d.C1C2  == x.C1C2);
                    if (itemToChange != null)

                    itemToChange.Desc1 = x.Desc1;
                    itemToChange.Desc2 = x.Desc2;
                    itemToChange.DataType = x.DataType;
                }

Expeced Result:

App_Id, C1,C2,Desc1, Desc2, DataType, Value in to one list.

2 answers

  • answered 2019-08-13 03:47 Peter Duniho

    Your current implementation is O(n^2), which can indeed get very slow as the data size gets large. You have essentially two nested loops: the outer foreach, and then an implicit loop in the FirstOrDefault() call (which has to scan the ListB collection to find the element of interest).

    You're already doing a join, but for some reason not using it idiomatically. That is, normally when one uses a join query, they actually do join the data so that the paired elements that are found can be enumerated sequentially in O(n) time.

    Lacking a complete code example in your question, it's impossible to know for sure how to fix your code, but I would expect something like this would work much better:

    var query = from x in ListB
                join y in ListA
                on x.C1C2 equals y.C1C2
                select new { Source = y, Target = x };
    
    //Update metadata from ListA to ListB
    foreach (var x in query)
    {
        x.Target.Desc1 = x.Source.Desc1;
        x.Target.Desc2 = x.Source.Desc2;
        x.Target.DataType = x.Source.DataType;
    }
    

  • answered 2019-08-13 05:46 Slava Utesinov

    You can convert ListB to dictionary - DicB, so it will be O(n) complexity:

    var DicB = ListB.ToDictionary(x => x.C1C2, x => x);
    
    foreach(var itemA in ListA)
    {
        if(DicB.ContainsKey(itemA.C1C2))
        {
            var itemB = DicB[itemA.C1C2];
            itemB.Desc1 = itemA.Desc1;
            itemB.Desc2 = itemA.Desc2;
            itemB.DataType = itemA.DataType;
        }
    }