removing duplicate from a 2D array

[mrbig1479]

Hey , i have this array

$MyArray [100][5]

inside that array , there are duplicate lines

(a single element can be a duplicate , but i want to remove only cases where the all the elements are the same

for example

$MyArray [1][0] = A

$MyArray [1][1] = B

$MyArray [1][2] = C

$MyArray [1][3] = D

$MyArray [1][4] = E

$MyArray [1][5] = F

I Don't want to remove array [2][all the elements] in a case like this

$MyArray [2][0] = A

$MyArray [2][1] = B

$MyArray [2][2] = C

$MyArray [2][3] = D

$MyArray [2][4] = E

$MyArray [2][5] = XXX

but i do want to remove in a case like this (where is eqaul to $MyArray [1][all the elements] )

$MyArray [3][0] = A

$MyArray [3][1] = B

$MyArray [3][2] = C

$MyArray [3][3] = D

$MyArray [3][4] = E

$MyArray [3][5] = F

I hope i explained well

[picaxe]

(a single element can be a duplicate , but i want to remove only cases where the all the elements are the same)

One way is to create a 1D array by concatenating all elements in 2nd dimension with suitable delimiter (say "|"), pass thru _ArrayUnique and then use StringSplit($aArray[$i], "|", 2) to recreate your 2D array.

[JonathanR]

#Include <Array.au3>

Dim $MyArray[100][6] = [ _
["A", "B", "C", "D", "E", "F"], _
["A", "B", "C", "D", "E", "XXX"], _
["A", "B", "C", "D", "E", "F"], _
["A", "B", "C", "QQ", "E", "F"], _
["1", "2", "3", "4", "5", "6"], _
["A", "B", "C", "D", "E", "XXX"]]
;etc..

_ArrayDisplay($MyArray) ;before output
$outputArray = _CompareSecondDimension($MyArray)
_ArrayDisplay($outputArray) ;after output

Func _CompareSecondDimension(Const ByRef $csdArray)
    Local $aString[UBound($csdArray)], $rtnArray = $csdArray, $iCount = 0
    For $a = 0 to UBound($csdArray) - 1
        For $b = 0 to UBound($csdArray, 2) - 1
            $aString[$a] &= $csdArray[$a][$b]
        Next
        If _ArraySearch($aString, $aString[$a]) <> $a Then
            _ArrayDelete($rtnArray, $a - $iCount)
            $iCount += 1
        EndIf
    Next
    Return $rtnArray
EndFunc

;before output

[0]|A|B|C|D|E|F

[1]|A|B|C|D|E|XXX

[2]|A|B|C|D|E|F

[3]|A|B|C|QQ|E|F

[4]|1|2|3|4|5|6

[5]|A|B|C|D|E|XXX

[6]||||||

[7]||||||

[8]||||||

...etc...

;after output

[0]|A|B|C|D|E|F

[1]|A|B|C|D|E|XXX

[2]|A|B|C|QQ|E|F

[3]|1|2|3|4|5|6

[4]||||||

If I understood what you're trying to do correctly, then the above script SHOULD work.

It works by making a 1D array of all the concatenated second dimensions, it then searches it against what it's already done, and if it finds a match, it removes it from the return array.

I threw this together in about 20 minutes, so odds are, it can likely be optimized and enhanced, but the principle is there.

Edited January 11, 2010 by JRouleau

[Xenobiologist]

Is that enough?

#include <Array.au3>
; Array

Global $a[4][5] = [[1, 2, 3, 4, 5],[1, 2, 3, 4, 5],[1, 2, 3, 4, 9],[1, 2, 3, 4, 8]]

_ArrayDisplay($a)

For $i = 1 To UBound($a, 1) - 2
    If $a[$i][0] = $a[0][0] And _
            $a[$i][1] = $a[0][1] And _
            $a[$i][2] = $a[0][2] And _
            $a[$i][3] = $a[0][3] And _
            $a[$i][4] = $a[0][4] Then
        _ArrayDelete($a, $i)
    EndIf
Next
_ArrayDisplay($a)

[Malkey]

Another example.

#include <Array.au3>

local $MyArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
                     ["A", "B", "C", "D", "E", "XXX"], _
                     ["A", "B", "C", "D", "E", "F"], _
                     ["A", "B", "C", "QQ", "E", "F"], _
                     ["1", "2", "3", "4", "5", "6"], _
                     ["A", "B", "C", "D", "E", "XXX"]]
                     ;etc..

Local $aDuplicatelessArray = _ArrayRemoveDuplicateRows($MyArray)

_ArrayDisplay($aDuplicatelessArray)

Func _ArrayRemoveDuplicateRows($MyArray)
    Local $sString, $sNoDupStr = "", $iCount = 0
    Local $aNoDupsArray[UBound($MyArray, 1)][UBound($MyArray, 2)]
    For $i = 0 To UBound($MyArray) - 1
        $sString = $MyArray[$i][0] & $MyArray[$i][1] & $MyArray[$i][2] & _
                $MyArray[$i][3] & $MyArray[$i][4] & $MyArray[$i][5] & @CRLF
        If StringInStr($sNoDupStr, $sString) = 0 Then ; Check if $sString is in $sNoDupStr string
            $sNoDupStr &= $sString & @CRLF ; Update $sNoDupStr string
            $aNoDupsArray[$iCount][0] = $MyArray[$i][0]
            $aNoDupsArray[$iCount][1] = $MyArray[$i][1]
            $aNoDupsArray[$iCount][2] = $MyArray[$i][2]
            $aNoDupsArray[$iCount][3] = $MyArray[$i][3]
            $aNoDupsArray[$iCount][4] = $MyArray[$i][4]
            $aNoDupsArray[$iCount][5] = $MyArray[$i][5]
            $iCount += 1
        EndIf
    Next
    ReDim $aNoDupsArray[$iCount][UBound($MyArray, 2)]
    Return $aNoDupsArray
EndFunc ;==>_ArrayRemoveDuplicateRows

[mrbig1479]

Another example.

#include <Array.au3>

local $MyArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
                     ["A", "B", "C", "D", "E", "XXX"], _
                     ["A", "B", "C", "D", "E", "F"], _
                     ["A", "B", "C", "QQ", "E", "F"], _
                     ["1", "2", "3", "4", "5", "6"], _
                     ["A", "B", "C", "D", "E", "XXX"]]
                     ;etc..

Local $aDuplicatelessArray = _ArrayRemoveDuplicateRows($MyArray)

_ArrayDisplay($aDuplicatelessArray)

Func _ArrayRemoveDuplicateRows($MyArray)
    Local $sString, $sNoDupStr = "", $iCount = 0
    Local $aNoDupsArray[UBound($MyArray, 1)][UBound($MyArray, 2)]
    For $i = 0 To UBound($MyArray) - 1
        $sString = $MyArray[$i][0] & $MyArray[$i][1] & $MyArray[$i][2] & _
                $MyArray[$i][3] & $MyArray[$i][4] & $MyArray[$i][5] & @CRLF
        If StringInStr($sNoDupStr, $sString) = 0 Then ; Check if $sString is in $sNoDupStr string
            $sNoDupStr &= $sString & @CRLF ; Update $sNoDupStr string
            $aNoDupsArray[$iCount][0] = $MyArray[$i][0]
            $aNoDupsArray[$iCount][1] = $MyArray[$i][1]
            $aNoDupsArray[$iCount][2] = $MyArray[$i][2]
            $aNoDupsArray[$iCount][3] = $MyArray[$i][3]
            $aNoDupsArray[$iCount][4] = $MyArray[$i][4]
            $aNoDupsArray[$iCount][5] = $MyArray[$i][5]
            $iCount += 1
        EndIf
    Next
    ReDim $aNoDupsArray[$iCount][UBound($MyArray, 2)]
    Return $aNoDupsArray
EndFunc ;==>_ArrayRemoveDuplicateRows

works great !!

Thanks

[SRE]

I created this account just to say thank you to JonathanR - your solution works flawless let and is exactly what I was looking for.  Had a hard time find a thread on how to delete duplicate rows in a 2d array.  Saved me hours of work writing my own function.  Thank you!

[Malkey]

I noticed my old example of post #5 did not allow for blank elements in a 2D array, and neither does JonathanR's example of post #3.   That is, both examples take these two rows as duplicates:-
    ["", "1", "", "", "3", ""], _
    ["1", "3", "", "", "", ""], _
because of the blank elements.

My modified example to allow for blank elements.

#include <Array.au3>

Local $MyArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "QQ", "E", "F"], _
        ["1", "2", "3", "4", "5", "6"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["", "1", "", "", "3", ""], _
        ["1", "3", "", "", "", ""], _
        ["", "", "", "", "", ""]]
;etc..
_ArrayDisplay($MyArray, "Before")
Local $aDuplicatelessArray = _ArrayRemoveDuplicateRows($MyArray)

_ArrayDisplay($aDuplicatelessArray, "Unique")

Func _ArrayRemoveDuplicateRows($MyArray)
    Local $sString, $sNoDupStr = "", $iCount = 0
    Local $aNoDupsArray[UBound($MyArray, 1)][UBound($MyArray, 2)]
    For $i = 0 To UBound($MyArray) - 1
        $sString = "|" & $MyArray[$i][0] & "|" & $MyArray[$i][1] & "|" & $MyArray[$i][2] & "|" & _
                $MyArray[$i][3] & "|" & $MyArray[$i][4] & "|" & $MyArray[$i][5] & "|" & @CRLF ; Adding "|" allows for blank elements in array.
        If StringInStr($sNoDupStr, $sString) = 0 Then ; Check if $sString is in $sNoDupStr string
            $sNoDupStr &= $sString & @CRLF ; Update $sNoDupStr string
            For $j = 0 To UBound($MyArray, 2) - 1
                $aNoDupsArray[$iCount][$j] = $MyArray[$i][$j]
            Next
            $iCount += 1
        EndIf
    Next
    ReDim $aNoDupsArray[$iCount][UBound($MyArray, 2)]
    Return $aNoDupsArray
EndFunc   ;==>_ArrayRemoveDuplicateRows

And, a modified JonathanR's example allowing for blank elements.

#include <Array.au3>

Dim $MyArray[100][6] = [ _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "QQ", "E", "F"], _
        ["1", "2", "3", "4", "5", "6"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["", "1", "", "", "3", ""], _
        ["1", "3", "", "", "", ""], _
        ["", "", "", "", "", ""]]
;etc..

_ArrayDisplay($MyArray) ;before output
$outputArray = _CompareSecondDimension($MyArray)
_ArrayDisplay($outputArray) ;after output

Func _CompareSecondDimension(Const ByRef $csdArray)
    Local $aString[UBound($csdArray)], $rtnArray = $csdArray, $iCount = 0
    For $a = 0 To UBound($csdArray) - 1
        For $b = 0 To UBound($csdArray, 2) - 1
            $aString[$a] &= "|" & $csdArray[$a][$b] & "|"
        Next
        ;ConsoleWrite($aString[$a] & @CRLF)
        If _ArraySearch($aString, $aString[$a]) <> $a Then
            _ArrayDelete($rtnArray, $a - $iCount)
            $iCount += 1
        EndIf
    Next
    Return $rtnArray
EndFunc   ;==>_CompareSecondDimension

 

[SRE]

Thank you Malkey for pointing out the issue with blank elements.  I'm using JonathanR's code because it short and works.  But I'm hoping someone here can show me how to make it delete in reverse.  I notice _ArraySearch has an option to set to 0 to search in reverse and I tried stepping ( STEP - 1 ) from high to 0 for the $a loop.  Can't get it to work for me.

I discovered that I need it to make the highest numbered rows the authoritative.  Currently, it's making the first entry authoritative and deleting everything else that follows that is a duplicate.  I want it to keep the last entry and delete all duplicates above it.  Any takers?  I'm at a loss.  Thank you.

[Malkey]

Instead of creating a throw away array and using _ArraySearch as JonathanR does, I create a throw away string and use StringInStr.

This example deletes the first occurring duplicate rows , or, keeps the row with the highest array index of the duplicate rows.

expandcollapse popup

#include <Array.au3>

Local $MyArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["1", "2", "3", "4", "5", "6"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "QQ", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["", "1", "", "", "3", ""], _
        ["1", "3", "", "", "", ""], _
        ["", "", "", "", "", ""]]
;etc..
_ArrayDisplay($MyArray, "Before")
Local $aDuplicatelessArray = _ArrayRemoveDuplicateRows($MyArray)
_ArrayDisplay($aDuplicatelessArray, "Unique")

; Remove first or lowest occurring duplicate row
Func _ArrayRemoveDuplicateRows($MyArray)
    Local $sString, $sNoDupStr = "", $iCount = 0
    Local $aNoDupsArray[UBound($MyArray, 1)][UBound($MyArray, 2)]
    For $i = UBound($MyArray) - 1 To 0 Step -1
        $sString = ""
        For $k = 0 To UBound($MyArray, 2) - 1
            $sString &= "|" & $MyArray[$i][$k] & "|" & @CRLF ; Adding "|" allows for blank elements in array.
        Next
        If StringInStr($sNoDupStr, $sString) = 0 Then ; Check if $sString is in $sNoDupStr string
            $sNoDupStr &= "|" & $sString & "|" & @CRLF ; Update $sNoDupStr string
            For $j = 0 To UBound($MyArray, 2) - 1
                $aNoDupsArray[$iCount][$j] = $MyArray[$i][$j]
            Next
            $iCount += 1
        EndIf
    Next
    ReDim $aNoDupsArray[$iCount][UBound($MyArray, 2)]
    ;_ArrayDisplay($aNoDupsArray)

    ; Invert array
    Local $iLen1 = UBound($aNoDupsArray), $iLen2 = UBound($aNoDupsArray, 2), $aRet[$iLen1][$iLen2]
    For $i = 0 To $iLen1 - 1
        For $j = 0 To $iLen2 - 1
            $aRet[$iLen1 - 1 - $i][$j] = $aNoDupsArray[$i][$j]
        Next
    Next
    Return $aRet
EndFunc   ;==>_ArrayRemoveDuplicateRows

#cs 
 Array before :-
    A|B|C|D|E|F
    A|B|C|D|E|XXX
    A|B|C|D|E|F
    1|2|3|4|5|6
    A|B|C|D|E|F
    A|B|C|QQ|E|F
    A|B|C|D|E|XXX
    |1|||3|
    1|3||||
    |||||
    |||||
    |||||
    ...Etc...

 Array returned:-
    1|2|3|4|5|6
    A|B|C|D|E|F
    A|B|C|QQ|E|F
    A|B|C|D|E|XXX
    |1|||3|
    1|3||||
    |||||
#ce

 

[czardas]

Remove duplicate rows (for ArrayWorkshop.au3 - see my signature).

#include 'ArrayWorkshop.au3'

Local $aArray = [[1,2,3],[1,2,3],[4,5,6],[1,2,3]]
_ArrayUniqueXD($aArray)

 

Edited April 28, 2017 by czardas

[Malkey]

To erase all duplicate rows except keep the last duplicate row, try:-

#include <Array.au3>
#include 'ArrayWorkshop.au3'

Local $aArray[100][6] = [["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["1", "2", "3", "4", "5", "6"], _
        ["A", "B", "C", "D", "E", "F"], _
        ["A", "B", "C", "QQ", "E", "F"], _
        ["A", "B", "C", "D", "E", "XXX"], _
        ["", "1", "", "", "3", ""], _
        ["1", "3", "", "", "", ""], _
        ["", "", "", "", "", ""]]

_ReverseArray($aArray)
_ArrayUniqueXD($aArray)
_ReverseArray($aArray)
_ArrayDisplay($aArray)

 

[czardas]

@Malkey It took me a minute or two to figure out what you meant. Then I read post #9.

Edited April 28, 2017 by czardas