Issue41

Title Additional range IdentifierConstraintType in Orthanc
Priority wish Status resolved
Superseder Nosy List admin
Assigned To
Keywords Orthanc Core

Created on 2017-03-25.13:36:08 by admin, last changed by admin.

Messages
msg183 (view) Author: admin Date: 2017-03-25.13:36:08
[BitBucket user: Денис Смирнов]
[BitBucket date: 2017-03-25.12:36:08]

Hi,

I am using Orthanc 1.2.0 with PostgreSQL plugin and have successfully indexed ~ 3Tb of DICOM images. But right now I've faced a problem than query/retrieve works not very fast. After small recerch I found out that the problem is in the way Orthanc processes query/retrieve translaion to SQL. So, here is a query for DICOM studies from 2017-03-20 to 2017-03-22:

```
#!bash

findscu -S -k QueryRetrieveLevel=STUDY -k PatientID -k StudyDate=20170320-20170322 -k StudyDescription -k StudyInstanceUID -aec ORTHANC -aet FINDSCU pacs.viveya.local 4242
```
And in PostgreSQL it translates to two heavy queries

```
#!sql
SELECT d.id FROM DicomIdentifiers AS d, Resources AS r WHERE d.id = r.internalId AND r.resourceType='1' AND d.tagGroup='8' AND d.tagElement='32' AND d.value>='20170320';

SELECT d.id FROM DicomIdentifiers AS d, Resources AS r WHERE d.id = r.internalId AND r.resourceType='1' AND d.tagGroup='8' AND d.tagElement='32' AND d.value<='20170322';
```
As you can guess is is not a good idea because the second query returns all DicomIdentifiers.id before 2017-03-22. In my case it is 135094 rows that would be intersected with 593 rows after 2017-03-20 by Orthanc. The main problem is in getting from index 135094 rows with rechecking some of them in heap due to mvcc of PostgreSQL. May be it is not a problem if you have ssd and small tables and indexes but in my situation it is pain.
A better solution would be to use a SQL query with "between" or ">= and <=" in one statement. They work ~500 times faster in my case.

```
#!sql

SELECT d.id FROM DicomIdentifiers AS d, Resources AS r WHERE d.id = r.internalId AND r.resourceType='1' AND d.tagGroup='8' AND d.tagElement='32' AND d.value between '20170320' and '20170322';
```
But the problem is that Orthanc rigth now is implementing only four constraint types

```
#!c++

  enum IdentifierConstraintType
  {
    IdentifierConstraintType_Equal,
    IdentifierConstraintType_SmallerOrEqual,
    IdentifierConstraintType_GreaterOrEqual,
    IdentifierConstraintType_Wildcard        /* Case sensitive, "*" or "?" are the only allowed wildcards */
  };
```
But I think it can be a good idea to implement an additional fifth IdentifierConstraintType_Range solving the problem above. It can be supported by extension (like PostgreSQL one) if developers deside to use it. What do you think about this idea and is it a dificult task?

P.S. I am not a C++ programmer so I can't sugest a patch to this issue((
msg184 (view) Author: admin Date: 2017-05-10.17:50:01
[BitBucket user: Alain Mazy]
[BitBucket date: 2017-05-10.15:50:01]

Thanks for the very detailed report  and the proposal which seems adequate.

Technically, it's not that difficult but it requires modification of the Plugin SDK interface, the SQLite default implementation and the Postgresql plugin.  So it's quite a tedious task ...
msg185 (view) Author: admin Date: 2017-05-11.01:26:41
[BitBucket user: Денис Смирнов]
[BitBucket date: 2017-05-10.23:26:41]

Yes, I've looked through the code and understood the scale of a problem. But without fixing this performance issue Orthanc would never become a big data solution... I've even started learning C++ in the evenings to suggest a minor upgrade for Orthanc core and PG plugin. But right now I am far away from writing a good production C++ code (developers would do in 1000 times better)
msg186 (view) Author: admin Date: 2017-05-11.01:29:15
[BitBucket user: Денис Смирнов]
[BitBucket date: 2017-05-10.23:29:15]

Btw I have small SQL index improvements for orthanc-postgres but I can't find Issues bar there to write sugestions. They are tested in production and made queries really faster.
msg187 (view) Author: admin Date: 2017-05-11.08:56:02
[BitBucket user: Alain Mazy]
[BitBucket date: 2017-05-11.06:56:02]

Great !  All issues are actually centralized in the Orthanc repo.  When creating an issue, you may select the 'component' to "Postgresql - plugin"
msg188 (view) Author: admin Date: 2018-07-05.18:16:28
[BitBucket user: Sébastien Jodogne]
[BitBucket date: 2018-07-05.16:16:28]

This is now implemented by the conjunction of the following two modifications:

* New primitive in the Orthanc Database SDK (will be part of Orthanc 1.4.0): https://hg.orthanc-server.com/orthanc/changeset/e583478e0c6cd906e0cfa7afc58da7dabf881c49
* Refactored PostgreSQL plugin (will be released as Orthanc PostgreSQL 2.2): https://hg.orthanc-server.com/orthanc-databases/file/5a97c68a7a51da441ae94b6ab6681b2ea7aa8e80/Framework/Plugins/IndexBackend.cpp#L1078

A global official release in planned for July 2018.
History
Date User Action Args
2026-07-29 15:51:19admincreate