Network file transfer issue

[Champak]

Is there a way to detect when multiple file transfers from an external source has been completed to a network (synology) drive, or test for active data transfer activity to that network drive/folder (which would be handled by a child app)?

My script monitors a bunch of folders and then does some other stuff once a file hits the directory. It works fine when dropping files in from the main computer because the magic activates once the shell message 0x00040000 is received which activates AFTER a GROUP of files have finished transferring instead of after each individual file. The problem is when the files are coming from an external source to the network folder being monitored...by ways of my phone direct to the drive...that message doesn't pop up, only 0x00020000 after each INDIVIDUAL file. That's where things get screwy because I can't have my function activating after individual file upload, it has to happen after completion of all files for smooth operation.

Thanks.

[crackdonalds]

the external source will have to let you know that the transfer has been completed. this could be done by dropping a file (completed.txt) somewhere. then you can check whether the file exists or not.

it all kinda depends on the possibilities that you have at the external source. If you know exactly how many files or how many bites have to be transferred then you can check for that. for example, you could do an md5 check to make sure that the destination files are the same as the source. if they have the same md5 hash then it means that the transfer was successful.

 

Edited November 29, 2021 by crackdonalds

[argumentum]

Try Syncthing

[pseakins]

9 hours ago, argumentum said:

Try Syncthing

Or FreeFileSync It has an email feature requiring a donation but it also has an "On Completion" feature which can execute anything of your choosing. No donation required for that.

[Champak]

Both of those are folder sync apps...as far as I see, and that is not what I'm tying to do. Syncthing doesn't look like it would work for me at all. FreeFileSync seems like it would work, but the way it would work I've already thought about it and don't really wan't to implement it. Basically using real time sync it would send out a notification a certain amount of time after activity has stopped. I already thought about doing that with a timer activated function on loop once one file was uploaded to check the last time a file was uploaded, and activate if nothing has happened in X time. This idea works fine if the upload has taken place over LAN, but if it's incoming WAN transfer that blows up my timing setup, too unpredictable. If I'm missing something with either of these apps, please let me know.

In the meantime, I was thinking of two other options.

1/ I was looking at winpcap and wireshark to monitor my network (they seem to allow me to monitor all traffic on my LAN/router, not just what goes and comes from the computer it's installed on, let me know if I'm wrong) and if they send out data that I can capture with autoit realtime, that may be a way to go. My thinking is once I detect an incoming file from the folder, start monitoring the network data activity, and when the load drops that would be the indicator that all transfers are complete and I can proceed with the rest of the function.

2/ This may be easier. A/ Find an app that monitors hard drive free space that autoit can tap into. I know autoit has DriveSpaceFree, but it isn't real time and could cause issues in timing between LAN vs WAN and small vs large files, or B/ find an app that monitors hard drive writing where I can look at when the writing stops for X amount of time to act.

Any thoughts on either and direction?

[JockoDundee]

Both 1 and 2 give you the unpleasant dilemma of reliability vs. latency.

Increase one and you increase the other.

Not fun.

Plus, waiting on files from a mobile device could easily be delayed by minutes, depending on coverage.

@crackdonalds made a suggestion that you didn’t respond to, namely whether it’s possible to add a zero-byte semaphore file at the end of the transfer, to indicate the transfer is done.  Perhaps you have no control or information about the files being transferred?

What kind of files are they, how are they generated?

[Champak]

Sorry @crackdonalds, I have no control over adding that info at the end. Strictly the files and no info beforehand. And majority of times no possibility to check back at the source anyway.

@JockoDundee image and video. Mostly jpg and mp4. What do you mean how are they generated? What do you mean reliability and latency issues? I don't see it. All options are looking at a constant that stays up until completion...except between files in a group transfer which is manageable. I would be adding a timer for that. So off the top of my head the function would maybe say, do until transfer or writing drops X amount, then sleep for 3 to 8 seconds (I think plenty of time for another file to start writing within the same transfer regardless of source) then check if there is still a transfer/disk write happening and if it is fire the function again, if it's not go on. Or is there something I don't understand about network and hard disk write activity...first time I'm really looking at either.

[argumentum]

Synology runs on linux. Look in forums related to that, as what you'd like to have is an API to get that info. and if none exist, you( or some one that knows ) could just write it.
I never had a Synology box to play with but that's the approach I'd take. 

[JockoDundee]

8 hours ago, Champak said:

What do you mean reliability and latency issues?

You are downloading files from phones, correct?

Do these phones never go out of range, or die because of battery?

[Champak]

@JockoDundee got you. Those particular scenarios I'm not worried about.

@argumentum I've been in touch with them. They said no API for the info that I want. That can only be retrieved through snmp mib. I saw a couple posts on here about it, but can't make heads or tails as of now. I've already enables the function on my NAS, but can't seem to get a ping response...then I need to make heads or tails of the rest of stuff 😭.

[crackdonalds]

there's a snmp udf on the forum. i've used it before. if you can't get a ping response then that should be allowed in the firewall of the device.

if you're going to look into snmp then i would recommend to download a snmp walker such as "snmptest". that would make things easier while you write the functionality that you require.

personally i don't think that you can get info through snmp. disk space wouldn't be a problem but the start/end of a file transfer... i doubt it... but you can check the MIB.

[Champak]

Turns out I didn't put a proper community name in. I thought leaving it as public was good enough. Network traffic and disk writing was not available through MIB, so I'm using disk load and space in combination instead. So far everything is working as it should, just throwing scenarios at it to make sure it doesn't break down somewhere.

Thanks.