[BitBucket date: 2017-03-21.15:25:37]
I ran into a problem when making concurrent C-MOVE requests using Orthanc. To replicate the issue, try the following:
1. Make an instance-level query. For example:
POST modalities/ClearCanvas/query
Payload: {"Level":"Instance","Query":{"StudyInstanceUID": "1.2.410.200028.479.2015128.135245", "SeriesInstanceUID": "1.2.410.200028.479.2015128.135245.1"}}
2. Get the answers from the query above. For example:
queries/b2e7c2b9-9ff8-452a-afab-736ef73f6587/answers
This will return answers corresponding to our instances.
3. Issue concurrent retrieve requests to download those instances. For example:
POST queries/b2e7c2b9-9ff8-452a-afab-736ef73f6587/answers/0/retrieve
POST queries/b2e7c2b9-9ff8-452a-afab-736ef73f6587/answers/1/retrieve
I wrote a C# client to make sure both requests are sent simultaneously.
The second "retrieve" request will fail. It says "Peer aborted Association (or never connected).
I dived into the code and might have found the bug (not the fix yet). Here is what I discovered:
Whe I make C-MOVE calls, the following function will be invoked:
ReusableDicomUserConnection::Locker locker(context_.GetReusableDicomUserConnection()...);
locker.GetConnection().Move(target, map);
The intention of locker is to protect Move() call with a mutex. Unfortunately, after completing the Move() call, CommandDispatcher::Step() will be triggered again where DUL_PEERREQUESTEDRELEASE message will be received. The T_ASC_Association object will be released as a result.
But if a second move call is initiated AFTER the first Move() call, but BEFORE the CommandDispatcher::Step() call, the second Move() call will try to reuse the original DicomUserConnection object for further communication. When the aforementioned DUL_PEERREQUESTEDRELEASE message arrives, the second move call will fail since the reused T_ASC_Association object is destroyed.
So the fundamental problem of the bug is that locker.GetConnection().Move(target, map) didn't cover all operations related to the move: when the first Move() is still trying to cleanup the association while a second Move() is triggered, we can see crashes every time.
Thanks for your thoughts on how to fix the problem in advance!
Thanks!
Ishaan |