That search pattern isn't going to find what you're looking for unless you limit the area to be searched. I can understand the first search looking through the entire screen, but you are only limiting the second/third search patterns by using the top and left of the found pixel before it. You're still searching the rest of the screen for the next pixel if the first pixel found isn't in the image you showed, it might find the next 2 colors anywhere on the screen.
If you've ever played the game Battleship, think of your search area as looking for the PT boat, you hit the first pixel, you should look for the second pixel within a very small area around the first one, not the whole screen, same as with the third pixel, search in a small area around the second pixel.If you don't find it in that area, rerun the first search pattern again and try again.