Looking for ideas on how to deal with massive raw data sifting

[Jazkal]

Hello everybody.

I'm looking for ideas on how to deal with finding patterns in raw data, when the pattern may not be known ahead of time.

I have lots of lists of some small, but mostly blocks of large number strings. I need to be able to sort out the patterns and identify the ones that are repeating.

As an example, here is one of the strings:

11235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011

As you can see visually in the block, there is a repeating number sequence. But in this sequence itself, there are repeating patterns. These other patterns show up in other data blocks, and I need to index them to related sequences.

14594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314594370774156178538190998752796516730336954932572910112358314

Here is a second block with the related sequences.

I'm mainly looking for direction in how to tackle the issue.

Thanks for looking.

 

 

 

 

Edited June 27, 2022 by Jazkal
added images of blocks

[jchd]

Maybe you can mimic algorithms used to detect sequences in DNA data.

Google for that. For instance: https://pastel.archives-ouvertes.fr/tel-01762479/document

[UEZ]

Something like this here?:

#include <Array.au3>

Global $s = "11235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011235831459437077415617853819099875279651673033695493257291011"

Global $len = Int(StringLen($s) / 2), $rp
For $i = 2 To $len
    $rp = StringMid($s, 1, $i)
    $a = StringSplit($s, $rp, 2)
    If UBound($a) < $len Then
        ConsoleWrite(_ArrayToString($a) & @CRLF)
    EndIf
Next

 

[Jazkal]

UEZ, That is an approach I hadn't thought of. I'll play more with that.

 

2 hours ago, jchd said:

Maybe you can mimic algorithms used to detect sequences in DNA data.

Google for that. For instance: https://pastel.archives-ouvertes.fr/tel-01762479/document

jacd, first pass of the white paper, and I think this is what I need. I knew it was out there, but I couldn't find it looking though my math encyclopedia. Thanks for leading me to that.

[jchd]

This is one of the application domains for Bloom filters. I'm sure there is significant litterature about dissecting DNA sequences and by extension sequences of symbols repeating in a large string of symbols in a given set, here digits.