Duplicate Handling Technical Documentation

Detailed documentation of the duplicate handling extension configuration options.

It is possible to define a list of configurations. All of them are encapsulated in the configurations attribute:

{  
  "configurations": [  
    {...},  
    {...},  
    {...}	  
  ]  
}

Configuration parameters

Each configuration can use the following attributes.

Attribute	Type	Required	Description
queues	list	false	IDs of queues where the duplicate detection should be performed. You can assign the extension to multiple queues and specify numerous actions for different queues in one instance. If not present, the duplicate detection will be performed on all the queues to which the extension is assigned unless `excluded_queues` is set.
excluded_queues	list	false	IDs of queues where the duplicate detection won't be performed. This parameter cannot be set along with the`queue_ids`.
trigger_events	list	false	List of Rossum events that trigger this configuration. Corresponds to selected hook events. If not specified, the duplicate detection will be performed for all allowed `trigger_events`. Possible values: - `annotation_status` - `annotation_content`
trigger_actions	list	false	List of Rossum actions that trigger this configuration. Corresponds to selected hook actions. If not specified, the duplicate detection will be performed for all actions relevant for a selected `trigger_event`. Possible values: - `changed` - `initialize` - `updated` - `started` - `confirm` - `export` - `user_update`
statuses	list	false	List of annotation status-couples in the format `{previous status}.{current status}` or `{previous status}->{current status}` defining when the processing of logic should start. If not present, duplicate detection will be performed for all allowed status changes.
logic	list	true	List of logic objects. More details are in the section below.

Logic parameters

Detailed specification of the logic object attributes.

Attribute	Type	Required	Default value	Description
rules	list	false	`[]`	List of rules based on which the duplicates will be detected. More details are in the section below.
scope	object	false		The scope of the duplicate detection logic is where the extension will look for duplicates. The section below provides more details. If not specified, the default `scope` will be used.
timestamp	object	false	`{}`	How long in the past the extension should look for duplicates. More details are in the section below.
matching_flow	list	false		Definition of how to combine duplicates detected by individual rules. It can be a list of IDs or logical `and` operations between rule ids. Individual list elements behave like logical `or` operations.
search_query	object	false		[Beta] Alternatively, you can also define a `search_query` instead of the combination of `rules`, `matching_flow`, `scope` and `timestamp`. More details are in the section below. If defined, such configuration will be preferred.
actions	list	true		List of actions to be performed if duplicates were detected.

Rule parameters

Detailed specification of the rule object attributes.

Attribute	Type	Required	Description
id	int	true	ID of the rule object. This can be referred to in the `matching_flow` field. IDs must be unique.
attribute	enum	true	Type of the rule. Possible values: - `field` - `relation` - `filename`
field_schema_id	string		Required for `field` rule. The `schema_id` of the field that should be used to compare for duplicates.

Scope parameters

Detailed specification of the scope object attributes.

Attribute	Type	Required	Default value	Description
object	enum	false	`queue`	Scope, where the extension should look for duplicates. It is recommended to use the`queue` scope and define as few queues as possible. Detecting duplicates in the whole workspace or organization can be very expensive. Possible values: - `queue` - `workspace` - `organization`
ids	list	false		The object's IDs (queue/workspace) should be considered during detection. Default is the context of the incoming annotation.
statuses	list	false		The statuses of annotations that should be considered during detection. Default is all. Possible values: - `created` - `importing` - `failed_import` - `split` - `to_review` - `reviewing` - `in_workflow` - `confirmed` - `rejected` - `exporting` - `exported` - `failed_export` - `postponed` - `deleted` - `purged`

Timestamp parameters

Detailed specification of the timestamp object attributes. If not specified, it searches without any limit.

Attribute	Type	Required	Description
action	enum	false	What annotation date condition should be considered when checking the timespan. Possible values: - `arrived_at_before` - `arrived_at_after` - `assigned_at_before` - `assigned_at_after` - `confirmed_at_before` - `confirmed_at_after` - `modified_at_before` - `modified_at_after` - `exported_at_before` - `exported_at_after`
timespan	int	false	The time span in days for the selected date condition

Search query parameters

[Beta] Search query for the duplicate detection. Uses more powerful searching based on the /search endpoint.

If this object is defined, it will be preferred for duplicate detection. matching_flow, rules, scope and timestamp objects will be ignored.

It is possible to reference some various values using the following syntax easily:

annotation and document object attributes using the dot . notation: {annotation.queue} or {document.created_at}
datapoint values using the @ operator: @{invoice_id}
datetime features
- now - current timestamp
- timedelta - python standard timedelta object

Attribute	Type	Required	Default value	Description
query	object	false	`{}`	A subset of MongoDB Query Language. See Rossum api docs for specifics and more details.
query_string	object	false	`{}`	Object with configuration for full-text search. See Rossum api docs for specifics and more details.

Actions parameters

All actions have one common required attribute - type - specifying the type of action that will be performed. Other attributes vary based on the type. Possible action types:

apply_label
fill_field
forward_annotation
mark_duplicate
show_message
stop_automation

Apply label action

Apply a label to the annotation that triggered the hook.

Attribute	Type	Required	Description
type	enum	true	The type of action that should be performed. Possible value: `apply_label`.
label	string	true	`name` of the label that should be applied

Fill field action

Fill a field with a defined value.

The following placeholders can be used in the value_to_fill attribute:

duplicate_ids - list of IDs of detected duplicate annotations
now - current datetime object. Additional operations can follow.
timedelta - timedelta object. Allows for operations like + or -together with now.
annotation - nested object containing attributes of the annotation that triggered the hook.
%ANNOTATION_ID% - expression to fill list of detected duplicates

Example: Detected {duplicate_ids | length} duplicates.<br> Date of detection: {now.strftime('%m/%d/%Y')}.

Detailed specification of the fill_field action attributes:

Attribute	Type	Required	Default value	Description
type	string	true		The type of action that should be performed. Possible value: `fill_field`
fill_only_if_empty	bool	false	`false`	If true, the field will only be filled if it was originally empty.
field_to_fill	string	true		The schema ID of the datapoint in Rossum schema to fill with custom value if duplicate is detected.
value_to_fill	string	true		The custom value to be filled to the datapoint in `field_to_fill`. Various placeholders can be used to create the value.

Forward annotation action

Forward the annotation to a different queue and/or different status.

Detailed specification of the forward_annotation action attributes:

Attribute	Type	Required	Description
type	string	true	The type of action that should be performed. Possible value: `forward_annotation`
target_queue	int	false	ID of the target queue where the duplicate annotation should be placed. If not specified, the current annotation queue will be used.
target_status	string	false	The target status of the annotation after it is forwarded to another queue. By default, the status is kept the same. If not specified, the current annotation status will be used.

Mark duplicate action

Create a new relation object. Ignored if the relation already exists.

Attribute	Type	Required	Description
type	string	true	The type of action that should be performed. Possible value: `mark_duplicate`

Show message action

Show a message to the user in the UI. The type and content of the message can be customized.

Following placeholders can be used in the `message`` attribute:

duplicate_ids - list of IDs of detected duplicate annotations
now - current datetime object. Additional operations can follow.
timedelta - timedelta object. Allows for operations like + or -together with now.
annotation - nested object containing attributes of the annotation that triggered the hook.
%ANNOTATION_ID% - expression to fill list of detected duplicates

Detailed specification of the show_message action attributes:

Attribute	Type	Required	Default value	Description
type	string	true		The type of action that should be performed. Possible value: `show_message`
message	string	true		Content of the message that will be displayed. Various placeholders can be used to create the message.
message_type	enum	false	`info`	Possible values: - `error` - `info` - `warning`

Stop automation action

Stop the automation by returning an error message. Only applicable for annotation_content.initialize event.

Attribute	Type	Required	Description
type	string	true	The type of action that should be performed. Possible value: `stop_automation`

Configuration examples

Simple detection with error message as an action

This configuration is triggered only on theinitialize, started or updated actions of annotation_content event. It will detect duplicates based on theinvoice_id field. If any other annotation in the same queue has the same invoice_id, an error will be shown in the annotation screen.

{
    "configurations": [
        {
            "trigger_events": ["annotation_content"],
            "trigger_actions": ["initialize", "started", "updated"],
            "logic": [
                {
                    "rules": [
                        {
                            "attribute": "field",
                            "field_schema_id": "invoice_id"
                        }
                    ],
                    "actions": [
                        {
                            "type": "show_message",
                            "message_type": "error",
                            "message": "Detected {duplicate_ids | length} duplicates"
                        }
                    ]
                }
            ]
        }
    ]
}

Complex detection logic with multiple actions

This configuration is triggered only on thechanged action of annotation_status event and only if the status is changed from importing to to_review or from to_review to postponed.

It will detect duplicates based on a combination of several rules. The detection will be scoped on the queue with ID 12345 and only annotations in selected statuses will be considered. The detection will also search only among annotations that arrived at Rossum in the last 60 days. If duplicates are detected, multiple actions will be performed in the order defined in the configuration.

{
    "configurations": [
        {
            "trigger_events": ["annotation_status"],
            "trigger_actions": ["changed"],
            "statuses": ["importing->to_review", "to_review->postponed"],
            "logic": [
                {
                    "matching_flow": ["1and2and3", "4"],
                    "rules": [
                        {
                            "id": 1,
                            "attribute": "field",
                            "field_schema_id": "invoice_id"
                        },
                        {
                            "id": 2,
                            "attribute": "field",
                            "field_schema_id": "sender_name"
                        },
                        {
                            "id": 3,
                            "attribute": "filename"
                        },
                        {
                            "id": 4,
                            "attribute": "relation"
                        }
                    ],
                    "scope": {
                        "object": "queue",
                        "ids": [12345],
                        "statuses": ["confirmed", "exported", "deleted"]
                    },
                    "timestamp": {
                        "action": "arrived_at_after",
                        "timespan": 60
                    },
                    "actions": [
                        {
                            "type": "fill_field",
                            "field_to_fill": "duplicate",
                            "value_to_fill": "Duplicate of %ANNOTATION_ID%"
                        },
                        {
                            "type": "forward_annotation",
                            "target_queue": 123456,
                            "target_status": "postponed"
                        },
                        {
                            "type": "mark_duplicate",
                            "message": "Marked as duplicate"
                        },
                        {
                            "type": "show_message",
                            "message_type": "error",
                            "message": "Detected {duplicate_ids | length} duplicates"
                        }
                    ]
                }
            ]
        }
    ]
}

Detection based on Search Query

Such configuration is triggered only on theinitialize, started or updated actions of the annotation_content event. It will detect duplicates based on theinvoice_id field among annotations in the same queue and created in the last 200 days ({(now - timedelta(days=200)).isoformat()}).

{
    "configurations": [
        {
            "trigger_events": ["annotation_content"],
            "trigger_actions": ["initialize", "started", "updated"],
            "logic": [
                {
                    "search_query": {
                        "query": {
                            "$and": [
                                {
                                    "field.invoice_id.string": {
                                        "$eq": "@{invoice_id}"
                                    }
                                },
                                {
                                    "queue": {
                                        "$in": ["{annotation.queue}"]
                                    }
                                },
                                {
                                    "created_at": {
                                        "$gt": "{(now - timedelta(days=200)).isoformat()}"
                                    }
                                }
                            ]
                        }
                    },
                    "actions": [
                        {
                            "type": "show_message",
                            "message_type": "error",
                            "message": "Duplicates detected: %ANNOTATION_ID%"
                        }
                    ]
                }
            ]
        }
    ]
}