File Locking with Cooperative Semaphores

[willichan]

This is a UDF I created for myself, to handle file locking on an SQLite database that was being shared on a network. Since I did not want to run a process on the server to handle the file locking, I needed a way to avoid write conflicts from the clients themselves.

**********
Thanks to an ISP problem, the ZIP download is not available.  The most recent code will be maintained here.
**********
 
Updated to work with 64-bit Win7

Minor performance enhancement as proposed by orbs (see >post #15)

expandcollapse popup

#include-once

#include <date.au3> ; Using:  _DateAdd(), _NowCalc(), _DateDiff()

; #INDEX# =========================================================================================
; Title .........: CFS
; AutoIt Version : 3.2.3++
; Language ..... : English
; Description ...: Functions for resource locking using cooperative, file-based semaphores
; Author(s) .....: willichan
; Note ..........: No files are actually locked using these functions.  Cooperative semaphores are
;                : dependant upon all access to the resource following the same locking mechanism.
;                : This satisfies resource locking needs when the resource is on a shared network
;                : location, but a server based semaphore service is not practical.
;                : For a good explanation of this methodology, see Sean M. Burke's article,
;                : "Resource Locking with Semaphore Files" at http://interglacial.com/tpj/23/ and
;                : "Resource Locking Over Networks" at http://interglacial.com/tpj/24/
; ;================================================================================================

; ------------------------------------------------------------------------------
; This software is provided 'as-is', without any express or
; implied warranty.  In no event will the authors be held liable for any
; damages arising from the use of this software.

; #CURRENT# =======================================================================================
; _CFS_RequestSemaphore
; _CFS_ReleaseSemaphore
; ;================================================================================================

; #INTERNAL_USE_ONLY# =============================================================================
; _CFS_ResourseLockName
; _CFS_CreateLock
; _CFS_RemoveLock
; _CFS_LockAge
; ;================================================================================================

; #FUNCTION# ;=====================================================================================
;
; Name...........: _CFS_RequestSemaphore
; Description ...: Requests cooperative semaphore lock for requested resource
; Syntax.........: _CFS_RequestSemaphore($sRequestedResource[, $iTimeout[, $iClean]])
; Parameters ....: $sRequestedResource - Path/name of the requested resource
;                  $iTimeout - Optional, timeout in seconds for before lock request fails
;                  default = 300s (5 min)
;                  $iClean - Optional, age in seconds for lock to exist before it is
;                            assumed dead and forcably removed. default = 900s (15 min)
; Return values .: Success - Returns 1
;                  Failure - Returns 0
; Author ........: willichan
; Modified.......:
; Remarks .......: Since we assume the resource exists, we also assume the path to it exists
; Related .......:
; Link ..........:
; Example .......: Yes
;
; ;================================================================================================
Func _CFS_RequestSemaphore($sRequestedResource, $iTimeout = 300, $iClean = 900)
    Local $sCFS_Semaphore = _CFS_ResourseLockName($sRequestedResource)
    Local $iStart = TimerInit()
    While True
        If _CFS_LockAge($sCFS_Semaphore, $iClean) Then _CFS_RemoveLock($sCFS_Semaphore) ; Too old?
        If TimerDiff($iStart) > ($iTimeout * 1000) Then ; have we timed out?
            Return 0
        Else
            If _CFS_CreateLock($sCFS_Semaphore) Then Return 1 ; did we get the lock?
        EndIf
        Sleep(100) ; didn't get a lock, pause and try again
    WEnd
EndFunc   ;==>_CFS_RequestSemaphore

; #FUNCTION# ;=====================================================================================
;
; Name...........: _CFS_ReleaseSemaphore
; Description ...: Releases cooperative semaphore lock for requested resource
; Syntax.........: _CFS_ReleaseSemaphore($sRequestedResource)
; Parameters ....: $sRequestedResource - Path/name of the requested resource
; Return values .: Success - Returns 1
;                  Failure - Returns 0
; Author ........: willichan
; Modified.......:
; Remarks .......: Never call this unless you hold a successful request
; Related .......:
; Link ..........:
; Example .......: Yes
;
; ;================================================================================================
Func _CFS_ReleaseSemaphore($sRequestedResource)
    Return _CFS_RemoveLock(_CFS_ResourseLockName($sRequestedResource))
EndFunc   ;==>_CFS_ReleaseSemaphore

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_ResourseLockName
; Description ...: Returns the name of the resource lock
; Syntax.........: _CFS_ResourseLockName($sRequestedResource)
; Parameters ....: $sRequestedResource - Path/name of the requested resource
; Return values .: Name of resource lock
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_ResourseLockName($sRequestedResource)
    Local Const $sInvalidChars = "\ "
    Local $sCFS_Semaphore = StringStripWS($sRequestedResource, 3)
    While StringInStr($sInvalidChars, StringRight($sCFS_Semaphore, 1))
        $sCFS_Semaphore = StringLeft($sCFS_Semaphore, StringLen($sCFS_Semaphore) - 1)
    WEnd
    Return $sCFS_Semaphore & "_Sem"
EndFunc   ;==>_CFS_ResourseLockName

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_CreateLock
; Description ...: Creates the semaphore lock
; Syntax.........: _CFS_CreateLock($sCFS_Semaphore)
; Parameters ....: $sCFS_Semaphore - Name of the resource lock
; Return values .: Success - Returns 1
;                  Failure - Returns 0
; Author ........: willichan
; Modified.......: orbs, willichan
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_CreateLock($sCFS_Semaphore)
    Local $aResult = DllCall('kernel32.dll', 'bool', 'CreateDirectoryW', 'wstr', $sCFS_Semaphore, 'struct*', 0)
    If @error Or ($aResult[0] = 0) Then
        Return 0
    Else
        Return 1
    EndIf
EndFunc   ;==>_CFS_CreateLock

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_RemoveLock
; Description ...: Removes the semaphore lock
; Syntax.........: _CFS_RemoveLock($sCFS_Semaphore)
; Parameters ....: $sCFS_Semaphore - Name of the resource lock
; Return values .: Success - Returns 1
;                  Failure - Returns 0
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_RemoveLock($sCFS_Semaphore)
    Return DirRemove($sCFS_Semaphore, 1)
EndFunc   ;==>_CFS_RemoveLock

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_LockAge
; Description ...: Checks to see if the Lock is too old to be valid
; Syntax.........: _CFS_LockAge($sCFS_Semaphore, $iClean)
; Parameters ....: $sCFS_Semaphore - Name of the resource lock
;                  $iClean - Age in seconds for lock to exist before it is assumed dead and
;                  forcably removed
; Return values .: Returns 1 if lock is to old
;                  Returns 0 if lock is still valid
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_LockAge($sCFS_Semaphore, $iClean)
    Local $asDate = FileGetTime($sCFS_Semaphore, 1, 0)
    If @error Then Return 0
    Local $sExpires = _DateAdd("s", $iClean, $asDate[0] & "/" & $asDate[1] & "/" & $asDate[2] & " " & $asDate[3] & ":" & $asDate[4] & ":" & $asDate[5])
    If _DateDiff("s", _NowCalc(), $sExpires) > 0 Then
        Return 0
    Else
        Return 1
    EndIf
EndFunc   ;==>_CFS_LockAge

If you want to see it in action, before implementing it in your own production scripts, I have written up a little demo here:

expandcollapse popup

#include <CFS.au3>
#include <ButtonConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
Opt("GUIOnEventMode", 1)
Opt("MustDeclareVars", 1)
Opt("TrayAutoPause", 0)
Opt("TrayMenuMode", 0)
Opt("TrayIconHide", 0)
Opt("TrayIconDebug", 0)

Global $frm_demo, $frm_demo_status, $frm_demo_quit
Global Const $frm_demo_width = 180
Global Const $frm_demo_height = 65
Global Const $MyName = StringLeft(@ScriptName, StringInStr(@ScriptName, ".", 0, -1) - 1)
Global Const $logpath = "f:\logs"
Global Const $logfile = $logpath & "\CFSdemo.log"
Global $SessionID = 0
Global $loop = True

_UniqueSession()
DirCreate($logpath)
_frm_demo_create()
main()

Func main()
    Local $lock
    While $loop
        _frm_demo_update("Requesting")
        $lock = _CFS_RequestSemaphore($logfile, 600, 300)
        If $lock Then
            _frm_demo_update("Locked")
            FileWriteLine($logfile, @ComputerName & ":" & $SessionID & " - " & @HOUR & ":" & @MIN & ":" & @SEC & ":" & @MSEC)
            _frm_demo_update("Releasing")
            _CFS_ReleaseSemaphore($logfile)
            _frm_demo_update("")
        EndIf
    WEnd
EndFunc   ;==>main


Func _UniqueSession()
    While _MutexExists($MyName & "-" & $SessionID)
        $SessionID += 1
    WEnd
EndFunc   ;==>_UniqueSession

Func _MutexExists($sOccurenceName) ;thanks to martin for this function
    Local $ERROR_ALREADY_EXISTS = 183, $handle, $lastError
    $sOccurenceName = StringReplace($sOccurenceName, "\", "")
    $handle = DllCall("kernel32.dll", "int", "CreateMutex", "int", 0, "long", 1, "str", $sOccurenceName)
    $lastError = DllCall("kernel32.dll", "int", "GetLastError")
    Return $lastError[0] = $ERROR_ALREADY_EXISTS
EndFunc   ;==>_MutexExists

Func _frm_demo_create()
    $frm_demo = GUICreate("Session " & $SessionID, $frm_demo_width, $frm_demo_height)
    GUISetOnEvent($GUI_EVENT_CLOSE, "_frm_demo_close")
    GUISetOnEvent($GUI_EVENT_MINIMIZE, "_frm_demo_minimize")
    GUISetOnEvent($GUI_EVENT_RESTORE, "_frm_demo_restore")
    $frm_demo_status = GUICtrlCreateLabel("Initializing", 0, 8, 175, 17)
    $frm_demo_quit = GUICtrlCreateButton("Quit", 0, 32, 179, 25, $WS_GROUP)
    GUICtrlSetOnEvent($frm_demo_quit, "_frm_demo_quit_click")
    GUISetState(@SW_SHOW)
    _frm_demo_xy()
EndFunc   ;==>_frm_demo_create

Func _frm_demo_quit_click()
    _frm_demo_close()
EndFunc   ;==>_frm_demo_quit_click

Func _frm_demo_close()
    GUIDelete($frm_demo)
    $loop = False
EndFunc   ;==>_frm_demo_close

Func _frm_demo_minimize()
EndFunc   ;==>_frm_demo_minimize

Func _frm_demo_restore()
EndFunc   ;==>_frm_demo_restore

Func _frm_demo_update($update)
    GUICtrlSetData($frm_demo_status, $update)
EndFunc   ;==>_frm_demo_update

Func _frm_demo_xy()
    Local $x, $y
    Local $winsize = WinGetPos("Session " & $SessionID)
    Local $mx = Int(@DesktopWidth / $winsize[2])
    Local $my = Int(@DesktopHeight / $winsize[3])
    Local $IDpos = Mod($SessionID, ($mx * $my))
    $x = Int($IDpos / $my) * ($winsize[2])
    $y = Mod($IDpos, $my) * ($winsize[3])
    WinMove("Session " & $SessionID, "", $x, $y)
EndFunc   ;==>_frm_demo_xy

Make sure to set $logpath and $logfile to a proper location on your PC or network.

Run the demo as many times as you like on a single machine to simulate multiple machines, or run on several machines.
You will be able to see when each session gets its "lock" on the log file. You can also look at the log file itself to see what sessions/machines got access.

NOTE: Because of the way Windows handles multitasking, you will probably see groupings of the same session getting its lock several times before the next one. This is because Windows does not do a very good job of sharing the processor between applications, so one session may make several iterations before Windows pulls processor time away for the next session. You will see a better distribution if you actually use multiple machines rather than simulated machines.
 
For those who are wanting them, here are the "help" files that were included in the original ZIP file.
Nothing spectacular.  They are just a cut back demo designed to look like what would be in the help file if this were included.

_CFS_ReleaseSemaphore.au3
 

#include <CFS.au3>
; Where the remote log file is located
$logfile = "F:/server_share/logs/mylog.log"
; Request a lock on the remote log file
$lock = _CFS_RequestSemaphore($logfile)
If $lock Then
 ; Lock request succeeded.  Write to the file.
 FileWriteLine($logfile, @ComputerName & ":  I got my file lock")
 ; Now release the file so others can write to it.
 _CFS_ReleaseSemaphore($logfile)
Else
 ; Lock request timed out.  Don't write to the file.
 MsgBox(0, @ScriptName, "We failed to get a lock.  We can't write to the file.")
EndIf

_CFS_RequestSemaphore.au3
 

#include <CFS.au3>
; Where the remote log file is located
$logfile = "F:/server_share/logs/mylog.log"
; Request a lock on the remote log file
$lock = _CFS_RequestSemaphore($logfile)
If $lock Then
 ; Lock request succeeded.  Write to the file.
 FileWriteLine($logfile, @ComputerName & ":  I got my file lock")
 ; Now release the file so others can write to it.
 _CFS_ReleaseSemaphore($logfile)
Else
 ; Lock request timed out.  Don't write to the file.
 MsgBox(0, @ScriptName, "We failed to get a lock.  We can't write to the file.")
EndIf

Edited October 25, 2014 by willichan

[willichan]

For those who don't want to trust downloading the UDF from my server, here is the code for it. The download does have the help/example files included in it, however.
 

---- Edit ----

code removed.   The most current code will be maintained in post #1.



Edited October 25, 2014 by willichan

[footswitch]

Thanks for posting your idea, I'll definitely give it a try

[EDIT]

First remarks:

- You need to have the desired network path mapped with a drive letter (you also need to be authenticated to that path, but mapping rules out this requirement).

Second remarks:

- I believe it works as advertised! Only suggestion is that you change the following in CFS.au3

From this:

Local $iResult = RunWait('mkdir "' .........

To this:

Local $iResult = RunWait(@ComSpec & " /c " &'mkdir "' ............

The first option doesn't work under Windows 7 64-bit

Cheers,

footswitch

Edited November 26, 2010 by footswitch

[willichan]

The first option doesn't work under Windows 7 64-bit

Thank you. I will test that out.

You need to have the desired network path mapped with a drive letter

I wonder if that is also a Win7 64-bit limitation. I am able to work with UNCs under XP Pro SP3

you also need to be authenticated to that path

That goes without saying.

Thanks for the feedback. I'll start doing some testing under 64-bit.

[footswitch]

I am able to work with UNCs under XP Pro SP3

Really? Because as far as I know, the command line doesn't support UNCs. I could be wrong.

About the authentication, yes, it's kind of obvious, but once you start working directly with UNCs you run the risk of needing to authenticate on-the-fly, and this script wouldn't allow that also.

So... does mkdir work flawlessly? I mean, does mkdir really return an error upon creating an existing folder, even when it "thinks" it doesn't exist?

Let me give you an example, based on the way that I THINK mkdir works:

- mkdir checks for the directory to exist

a) if it exists, return error

if it doesn't exist, create it and return successfully - now, at this time, imagine another instance of mkdir is doing the exact same thing, but some miliseconds earlier. Question is: does mkdir check for the successful CREATION of the folder or, in the other hand, successful ISSUING of that command?

So as you can see I'm worried about a racing event which could happen or not depending on the way mkdir works.

footswitch

[willichan]

RE: Mkdir

Yes, mkdir does support UNCs. Many of the command line utilities will support UNC parameters. You just can't change to/land on a UNC.

I won't go into conjecture as to how Microsoft goes about doing it. My testing, however, shows mkdir to be the most reliable method as far as returning a proper error code. I tested various methods on 30 machines, running 24 hours, with 100 processes each, all simultaneously attempting mkdir commands. In that time, I did not get a single false 'success' report.

If a more reliable method is found, I will definitely test/implement it.

RE: authentication

Since the semaphore directory is in the same location as the resource your are trying to "lock" it is assumed that you will already be handling the authentication to the resource location before attempting to get a lock on it.

[footswitch]

Yes, mkdir does support UNCs. Many of the command line utilities will support UNC parameters. You just can't change to/land on a UNC.

You are absolutely right, I just tested it under Windows 7 x64 and it did work. but the script didn't.

So I retested the script and the path wasn't an issue after all, it was the mkdir call all along. As I mentioned earlier, for 64-bit compatibility:

Local $iResult = RunWait(@ComSpec & " /c " & 'mkdir "' & $sCFS_Semaphore & '"', @TempDir, @SW_HIDE)

Probably this compatibility issue is related to the command tools' location not being mapped to the PATH environment variable. But this fixes it.

Turns out it was my fault that I didn't test for UNC support after adjusting that single line of code. Sorry about that.

I tested various methods on 30 machines, running 24 hours, with 100 processes each, all simultaneously attempting mkdir commands. In that time, I did not get a single false 'success' report.

Wow, didn't realise there was so much study involved in it. That's... something

Since the semaphore directory is in the same location as the resource your are trying to "lock" it is assumed that you will already be handling the authentication to the resource location before attempting to get a lock on it.

Okay, that's a fair assumption

Next step: File Locking with Cooperative Queuing Semaphores

[willichan]

So I retested the script and the path wasn't an issue after all, it was the mkdir call all along. As I mentioned earlier, for 64-bit compatibility:

I've got that added to my ToDo list. ALl of the systems at the office are XP Pro, and all my systems at home are Linux, but I do have a license for Win 7 Ultimate. I will need to set up a machine and do some testing to make sure the error returns still work right using @ComSpec.

Next step: File Locking with Cooperative Queuing Semaphores

That will be a nice trick. If I figure that one out, does someone buy me lunch?

[footswitch]

I guess you're right, @ComSpec could change the results. "your mileage may vary"

It's not like I can't live without queuing, but over time I'm thinking of implementing it myself

If xyz_Sem exists, find all folders named xyz_Sem_*, get the last folder number and create xyz_Sem_n+1, where n is the Queue order.

Something like that.

But I'm predicting some issues here: There's already a timeout condition for the current semaphores. Now when you have lots of Queues and you don't know whether they will succeed or not, you could be placing a huge delay in database response time.

[willichan]

It's not like I can't live without queuing, but over time I'm thinking of implementing it myself

It would be nice. Unfortunately, the cooperative semaphore scheme muddles it a bit. It was intended for where running server services were impractical. When you reach the need for queuing, it may become more practical to go with a server service.

I'll still keep my brain chugging on it , but I don't see it materializing too soon.

[Megabird]

I'm thinking of something that may give you something to work on: DirCreate won't throw an error if the directory already exists, but DirMove does if you leave it's flag value to 0. So the idea is:

1: create a unique semafore (ex. using computername + time)

2: try to rename that semafore to the real semafore name until success

It can also help you queue the requests, since you can see every uniques semafores...

just an idea...

[willichan]

I'm thinking of something that may give you something to work on: DirCreate won't throw an error if the directory already exists, but DirMove does if you leave it's flag value to 0. So the idea is:

I actually use the DOS mkdir command, because it does return an error for directory exists. Directory creation is the most stable for cross-platform server compatibility. Even though AutoIt only runs from Windows, normal file locking is not handled consistently by servers running on various platforms (Mac, Linux, Unix, Novel, etc...).

This is an interesting idea. I will look into how consistent directory moves/renames are on other platforms. Definately something to play around with.

----Edit----

Ok. I've been playing around with queuing. Since there is no master process monitoring the queue and cleaning up, it gets a little tricky. I have a basic process for cooperative cleaning up, but it is not perfect. Without the queuing, I was able to run several hundred simulated users across 30-50 machines without any collisions. Unfortunately, with the queuing, I don't have it running quite so cleanly. I get collisions every 20-30 accesses with only 30 simulated users. Still needs some work. Feel free to play around with it if you want to.

queuing version:

expandcollapse popup

#include-once

#include <date.au3>  ; Using:  _DateAdd(), _NowCalc(), _DateDiff()
#include <file.au3>  ; Using:  _FileListToArray()
#include <array.au3> ; Using:  _ArraySort()

; #INDEX# =========================================================================================
; Title .........: CFS
; AutoIt Version : 3.2.3++
; Language ..... : English
; Description ...: Functions for resource locking using cooperative, file-based semaphores
; Author(s) .....: willichan
; Note ..........: No files are actually locked using these functions.  Cooperative semaphores are
;               : dependant upon all access to the resource following the same locking mechanism.
;               : This satisfies resource locking needs when the resource is on a shared network
;               : location, but a server based semaphore service is not practical.
;               : For a good explanation of this methodology, see Sean M. Burke's article,
;               : "Resource Locking with Semaphore Files" at http://interglacial.com/tpj/23/ and
;               : "Resource Locking Over Networks" at http://interglacial.com/tpj/24/
; ;================================================================================================

; ------------------------------------------------------------------------------
; This software is provided 'as-is', without any express or
; implied warranty.  In no event will the authors be held liable for any
; damages arising from the use of this software.

; #CURRENT# =======================================================================================
; _CFS_RequestSemaphore
; _CFS_ReleaseSemaphore
; ;================================================================================================

; #INTERNAL_USE_ONLY# =============================================================================
; _CFS_ResourseLockName
; _CFS_CreateLock
; _CFS_RemoveLock
; _CFS_LockAge
; _CFS_Timestamp
; _CFS_Timestamp2DateString
; ;================================================================================================

; #FUNCTION# ;=====================================================================================
;
; Name...........: _CFS_RequestSemaphore
; Description ...: Requests cooperative semaphore lock for requested resource
; Syntax.........: _CFS_RequestSemaphore($sRequestedResource[, $iTimeout[, $iClean]])
; Parameters ....: $sRequestedResource - Path/name of the requested resource
;               $iTimeout - Optional, timeout in seconds for before lock request fails
;               default = 300s (5 min)
;               $iClean - Optional, age in seconds for lock to exist before it is
;                           assumed dead and forcably removed. default = 900s (15 min)
; Return values .: Success - Returns 1
;               Failure - Returns 0
; Author ........: willichan
; Modified.......:
; Remarks .......: Since we assume the resource exists, we also assume the path to it exists
; Related .......:
; Link ..........:
; Example .......: Yes
;
; ;================================================================================================
Func _CFS_RequestSemaphore($sRequestedResource, $iTimeout = 300, $iClean = 900)
    Local $sCFS_Semaphore = _CFS_ResourseLockName($sRequestedResource)
    Local $sCFS_SemaphoreQueue, $iResult, $sNext
    Do
        $sCFS_SemaphoreQueue = $sCFS_Semaphore & _CFS_Timestamp() & _CFS_Timestamp($iTimeout)
        $iResult = RunWait('mkdir "' & $sCFS_SemaphoreQueue & '"', @TempDir, @SW_HIDE)
    Until $iResult = 0
    Local $iStart = TimerInit()
    While True
        If FileExists($sCFS_Semaphore) Then ; make sure there is a lock before we check its age.
            If _CFS_LockAge($sCFS_Semaphore, $iClean) Then _CFS_RemoveLock($sCFS_Semaphore) ; Too old?
        EndIf
        If TimerDiff($iStart) > ($iTimeout * 1000) Then ; have we timed out?
            DirRemove($sCFS_SemaphoreQueue)
            Return 0
        Else
            $sNext = _CFS_FrontOfQueue($sCFS_Semaphore) ; who's next in line?
            If $sNext = "" Then Return 0 ; Oops!  Everyone's queue expired ... including ours.
            If $sNext = $sCFS_SemaphoreQueue Then ; is it our turn?
                If _CFS_CreateLock($sCFS_Semaphore, $sCFS_SemaphoreQueue) Then Return 1 ; did we get the lock?
            EndIf
        EndIf
        Sleep(100) ; didn't get a lock, pause and try again
    WEnd
EndFunc   ;==>_CFS_RequestSemaphore

; #FUNCTION# ;=====================================================================================
;
; Name...........: _CFS_ReleaseSemaphore
; Description ...: Releases cooperative semaphore lock for requested resource
; Syntax.........: _CFS_ReleaseSemaphore($sRequestedResource)
; Parameters ....: $sRequestedResource - Path/name of the requested resource
; Return values .: Success - Returns 1
;               Failure - Returns 0
; Author ........: willichan
; Modified.......:
; Remarks .......: Never call this unless you hold a successful request
; Related .......:
; Link ..........:
; Example .......: Yes
;
; ;================================================================================================
Func _CFS_ReleaseSemaphore($sRequestedResource)
    Return _CFS_RemoveLock(_CFS_ResourseLockName($sRequestedResource))
EndFunc   ;==>_CFS_ReleaseSemaphore

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_ResourseLockName
; Description ...: Returns the name of the resource lock
; Syntax.........: _CFS_ResourseLockName($sRequestedResource)
; Parameters ....: $sRequestedResource - Path/name of the requested resource
; Return values .: Name of resource lock
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_ResourseLockName($sRequestedResource)
    Local Const $sInvalidChars = "\ "
    Local $sCFS_Semaphore = StringStripWS($sRequestedResource, 3)
    While StringInStr($sInvalidChars, StringRight($sCFS_Semaphore, 1))
        $sCFS_Semaphore = StringLeft($sCFS_Semaphore, StringLen($sCFS_Semaphore) - 1)
    WEnd
    Return $sCFS_Semaphore & "_Sem"
EndFunc   ;==>_CFS_ResourseLockName

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_CreateLock
; Description ...: Creates the semaphore lock
; Syntax.........: _CFS_CreateLock($sCFS_Semaphore, $sCFS_SemaphoreQueue)
; Parameters ....: $sCFS_Semaphore - Name of the resource lock
;               $sCFS_SemaphoreQueue - Lock request queue name
; Return values .: Success - Returns 1
;               Failure - Returns 0
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_CreateLock($sCFS_Semaphore, $sCFS_SemaphoreQueue)
    Local $iResult = DirMove($sCFS_SemaphoreQueue, $sCFS_Semaphore, 0)
    If $iResult = 1 Then
        FileSetTime($sCFS_Semaphore, "", 1)
        Return 1
    Else
        Return 0
    EndIf
EndFunc   ;==>_CFS_CreateLock

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_RemoveLock
; Description ...: Removes the semaphore lock
; Syntax.........: _CFS_RemoveLock($sCFS_Semaphore)
; Parameters ....: $sCFS_Semaphore - Name of the resource lock
; Return values .: Success - Returns 1
;               Failure - Returns 0
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_RemoveLock($sCFS_Semaphore)
    Return DirRemove($sCFS_Semaphore, 1)
EndFunc   ;==>_CFS_RemoveLock

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_LockAge
; Description ...: Checks to see if the Lock is too old to be valid
; Syntax.........: _CFS_LockAge($sCFS_Semaphore, $iClean)
; Parameters ....: $sCFS_Semaphore - Name of the resource lock
;               $iClean - Age in seconds for lock to exist before it is assumed dead and
;               forcably removed
; Return values .: Returns 1 if lock is to old
;               Returns 0 if lock is still valid
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_LockAge($sCFS_Semaphore, $iClean)
    Local $asDate = FileGetTime($sCFS_Semaphore, 1, 0)
    If @error Then Return 0
    Local $sExpires = _DateAdd("s", $iClean, $asDate[0] & "/" & $asDate[1] & "/" & $asDate[2] & " " & $asDate[3] & ":" & $asDate[4] & ":" & $asDate[5])
    If _DateDiff("s", _NowCalc(), $sExpires) > 0 Then
        Return 0
    Else
        Return 1
    EndIf
EndFunc   ;==>_CFS_LockAge

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_Timestamp
; Description ...: Returns a string representing the current time plus $iOffset seconds
; Syntax.........: _CFS_Timestamp([$iOffset])
; Parameters ....: $iOffset - number of seconds to add to the timestamp
; Return values .: Returns a string in the format YYYYMMDDHHMMSS
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_Timestamp($iOffset = 0)
    Return StringReplace(StringReplace(StringReplace(_DateAdd("s", $iOffset, @YEAR & "/" & @MON & "/" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC), "/", ""), ":", ""), " ", "")
EndFunc   ;==>_CFS_Timestamp

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_Timestamp2DateString
; Description ...: Reformats a timestamp generated by _CFS_Timestamp to YYYY/MM/DD HH:MM:SS
; Syntax.........: _CFS_Timestamp2DateString($sTimestamp)
; Parameters ....: $sTimestamp - Timestamp as returned by _CFS_Timestamp
; Return values .: Returns a string in the format YYYY/MM/DD HH:MM:SS
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_Timestamp2DateString($sTimestamp)
    Return StringLeft($sTimestamp, 4) & "/" & StringMid($sTimestamp, 5, 2) & "/" & StringMid($sTimestamp, 7, 2) & " " & StringMid($sTimestamp, 9, 2) & ":" & StringMid($sTimestamp, 11, 2) & ":" & StringMid($sTimestamp, 13, 2)
EndFunc   ;==>_CFS_Timestamp2DateString

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_Expired
; Description ...: Determins if a queued semaphore is expired, and deletes it if it is
; Syntax.........: _CFS_Expired($sCFS_SemaphoreQueue)
; Parameters ....: $sCFS_SemaphoreQueue - The semaphore queue to check
; Return values .: Returns True if expired
;               Returns False if not expired
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_Expired($sCFS_SemaphoreQueue)
    Local $sTimestamp = _CFS_Timestamp2DateString(StringRight($sCFS_SemaphoreQueue, 14))
    If _DateDiff("s", _NowCalc(), $sTimestamp) > 0 Then
        Return False
    Else
        DirRemove($sCFS_SemaphoreQueue)
        Return True
    EndIf
EndFunc   ;==>_CFS_Expired

; #INTERNAL_USE_ONLY# =============================================================================
;
; Name...........: _CFS_FrontOfQueue
; Description ...: Returns the Queue semifore name for the next in queue
; Syntax.........: _CFS_FrontOfQueue($sCFS_Semaphore)
; Parameters ....: $sCFS_Semaphore - the path/name of the semaphore
; Return values .: Returns a string with the next semaphore in queue, else returns "" of no queue
; Author ........: willichan
; Modified.......:
; Remarks .......:
; Related .......:
; Link ..........:
; Example .......: No
;
; ;================================================================================================
Func _CFS_FrontOfQueue($sCFS_Semaphore)
    Local $sPath, $sFile, $aQueue, $iCount = 1
    If StringInStr($sCFS_Semaphore, "\") Then
        $sPath = StringLeft($sCFS_Semaphore, StringInStr($sCFS_Semaphore, "\", 0, -1) - 1)
        $sFile = StringRight($sCFS_Semaphore, StringLen($sCFS_Semaphore) - StringInStr($sCFS_Semaphore, "\", 0, -1))
    Else
        $sPath = ""
        $sFile = $sCFS_Semaphore
    EndIf
    $aQueue = _FileListToArray($sPath, $sFile & "*", 2)
    If @error Then Return ""
    If Not IsArray($aQueue) Then Return ""
    _ArraySort($aQueue, 0, 1)
    While $iCount <= $aQueue[0]
        If _CFS_Expired($sPath & "\" & $aQueue[$iCount]) Then
            $iCount += 1
        Else
            Return $sPath & "\" & $aQueue[$iCount]
        EndIf
    WEnd
    Return ""
EndFunc   ;==>_CFS_FrontOfQueue

Edited April 28, 2011 by willichan

[willichan]

New version now works in 64-bit Windows 7.

See post #1

[orbs]

willichan, very nice work! simple implementation of a very smart concept!

question about parameter $iClean in function _CFS_RequestSemaphore() :

description: "age in seconds for lock to exist before it is assumed dead and forcably removed."

it seems that this is not quite so.

$iClean is used in function _CFS_LockAge() to determine if the existing semaphore exceeds the time specified for the calling instance, not for the instance which created the semaphore.

consider this:

PROCESS1 creates a semaphore with $iClean=60. PROCESS1 thinks that this means that the resource will remain at its disposal for at least one minute.

now PROCESS2 wants to lock the resource for itself, but needs it for only 15 seconds, so calling _CFS_RequestSemaphore() with $iClean=15.

_CFS_LockAge() called by PROCESS2 checks if the semaphore is older than 15 seconds, while any common sense would think it should check if the semaphore is older than 60 seconds, as it is currently locked by PROCESS1 which requested it for 60 seconds.

admittedly, usually the same resource is locked by different instances of the same process for the same maximum time, so this is not a problem. but occasionally this is not the case, and anyway for proper coding, this should be addressed.

possible solutions would be to write a text file inside the semaphore directory, which contains the maximum age (and perhaps also the hostname and PID of the process who locked it, so before force kill, the calling process can query the condition of the locking process).

or am i getting this completely wrong?

EDIT:

also, what is the X86 condition good for? @ComSpec works just as well for X86, you can just use it regardless of architecture. i'm referring to this line:

If @OSArch <> 'X86' Then $cmd = @ComSpec & ' /c ' & $cmd 

which can be simply:

$cmd = @ComSpec & ' /c ' & $cmd

Edited October 12, 2014 by orbs

[orbs]

using mkdir instead of DirCreate() is understood. but a better way may exist: using direct call to CreateDirectoryW in kernel32.dll (which also fails if the directory already exists).

a timing test shows mkdir takes ~25ms while CreateDirectoryW takes ~1ms

testing locally on quite a decent machine:

Windows 7 64-bit, CPU i5, 16GB RAM

testing on SSD, SATA3, and USB3 drives - similar results.

this testing script does not use the UDF, instead it has 2 versions of the _CFS_CreateLock() function - one the original (well, almost - i got rid of the architecture condition, as i mentioned in previous post), one calling the dll.

expandcollapse popup

#include <Array.au3> ; for _ArrayDisplay()
Global $sCFS_Semaphore = @TempDir & '\CFS_Test_Sem' ; direclty define the semaphore directory name
Global $aTime[4] ; an array to hold the timing results

DirRemove($sCFS_Semaphore) ; initial cleanup. comment this line out (and create the semaphore before next run) to make sure the function failes when semaphore already exists
$aTime[0] = @SEC & '.' & @MSEC
$nResult_comspec = __CFS_CreateLock_comspec($sCFS_Semaphore)
$aTime[1] = @SEC & '.' & @MSEC
MsgBox(0, $nResult_comspec, '$nResult_comspec') ; see the return value

DirRemove($sCFS_Semaphore) ; initial cleanup. comment this line out to make sure the function failes when semaphore already exists
$aTime[2] = @SEC & '.' & @MSEC
$nResult_dll = __CFS_CreateLock_dll($sCFS_Semaphore)
$aTime[3] = @SEC & '.' & @MSEC
MsgBox(0, $nResult_dll, '$nResult_dll') ; see the return value

DirRemove($sCFS_Semaphore) ; final cleanup
_ArrayDisplay($aTime) ; display results

Func __CFS_CreateLock_comspec($sCFS_Semaphore)
    ; Since DirCreate() does not return an error if the directory already exists, we use mkdir
    Local $cmd = 'mkdir "' & $sCFS_Semaphore & '"'
    $cmd = @ComSpec & ' /c ' & $cmd
    Local $iResult = RunWait($cmd, @TempDir, @SW_HIDE)
    If @error Then Return 0
    If $iResult = 0 Then
        Return 1
    Else
        Return 0
    EndIf
EndFunc   ;==>__CFS_CreateLock_comspec

Func __CFS_CreateLock_dll($sCFS_Semaphore)
    ; Since DirCreate() does not return an error if the directory already exists, we use WinAPI
    Local $aRet = DllCall('kernel32.dll', 'bool', 'CreateDirectoryW', 'wstr', $sCFS_Semaphore, 'struct*', 0)
    If @error Then Return 0
    If $aRet[0] = 0 Then
        Return 0
    Else
        Return 1
    EndIf
EndFunc   ;==>__CFS_CreateLock_dll
 

i will test on a network share when i get the chance. unless i experience some hiccups in further tests, i think the conclusion is clear - it is better to use the dll version (last function in the testing script).

[willichan]

 

willichan, very nice work! simple implementation of a very smart concept!

question about parameter $iClean in function _CFS_RequestSemaphore() :

description: "age in seconds for lock to exist before it is assumed dead and forcably removed."

it seems that this is not quite so.

$iClean is used in function _CFS_LockAge() to determine if the existing semaphore exceeds the time specified for the calling instance, not for the instance which created the semaphore.

 

Actually, it is correct.  You can think about it as your maximum level of patience to wait for the lock to become available.  It should not be used to set how long you intend to keep the lock.  No instance has any way of knowing anything about what another instance wants to do.  Writing data to an additional file is also not a good solution, since that just takes more time, and requires locking there as well.

Since there are is no master system watchdoging the locks, if a process dies without releasing the lock, there needs to be a way for other processes to move on.  That is where $iClean comes in.  It assumes that if you have been waiting that long for the resource to become available, and it has not, it must have been locked by a dead process.

Remember.  The idea here is that there is not a process watching over the system that could handle the cleanup and queueing.

This is one of the unfortunate drawbacks of a cooperative system.  It is, however, a minor one if you need to deploy in an environment where you cannot implement a watchdog.

 

also, what is the X86 condition good for? @ComSpec works just as well for X86, you can just use it regardless of architecture. i'm referring to this line:

If @OSArch <> 'X86' Then $cmd = @ComSpec & ' /c ' & $cmd

which can be simply:

$cmd = @ComSpec & ' /c ' & $cmd

 

I will take another look at this as I get some time.  I do not remember the reasoning for it right now.  It has been a while since I wrote this.

 

using mkdir instead of DirCreate() is understood. but a better way may exist: using direct call to CreateDirectoryW in kernel32.dll (which also fails if the directory already exists).

a timing test shows mkdir takes ~25ms while CreateDirectoryW takes ~1ms

 

I will look into this as well.  If it is stable across multiple server platforms (Windows server, Unix Server, Apple Server, etc...) then I will implement it.  I don't forsee any problems with it.

[willichan]

Post #1 has been updated with the latest code, including the performance update using a DLL call as proposed by orbs in post #15

[Zedna]

If it's for locking of SQLite database on shared network disk

then I would create "lock" table in SQLite database with one record and columns

locked=0/1 locked_date= locked_by_user=

Each user who wants to lock DB have to read this record and if it's already locked he can't proceed,

if it's not locked then he update this record (set locked=1) and proceed with further updates od other database tables,

when all work is done, he will unlock DB by updating this record to locked=0

[orbs]

the problem is that the checking of the value 0 and the updating it to 1 is not atomic operation - it is divided, and furthermore, there is a considerable time lapse in between.

you suggest that each user performs these steps:

1) read lock record

2) if it's 0 then write 1 to lock record

now, if 2 users do it at the same time, chronologically that would be:

1) user #1 reads the lock record (reading 0)

2) user #2 reads the lock record (reading 0)

3) user #1 read 0 so it updates it to 1 and assumes he has the lock

4) user #2 read 0 so it updates it to 1 (actually it's already set to 1 by user #1) and assumes he has the lock

both users are now sure each one of them has the lock.

this UDF relies on atomic operation - there is no time difference between the lock and the reading, because they are not different operations. it's the same operation that either creates the lock (and returns success), or does not (and returns failure). there are no 2 steps. it's like this:

1) if creating the lock is successful, then i have the lock. otherwise, i don't have the lock.

P.S. reminder: "lock" here actually means something like "voluntarily exclusive access".

[Zedna]

I think that problem of non atomic SQL commands could be solved by using SQL transactions and setting DB to not read uncomitted transactions.

And you can use this to make SQL atomic lock on "lock" table (SQL pseudocode):

Begin tran

Update lock set locked = locked -> this will do lock of table

Select @locked = locked from lock

If @locked = 1 then rollback tran exit

If @locked = 0 then update lock set locked = 1 commit tran

Edited October 26, 2014 by Zedna