Skip to content

Task Manager

Task Manager provides the possibility of scheduling special long lasting tasks for the following web services.

  • 1
    DB Web Services

This documentation describes installation, administration and usage of Task Manager.

Task Manager application is available as part of a microservices system or in standalond web application mode.

Microservices system mode

In microservices system mode the Task Manager runs together with Config service, Discovery service, Gateway service. These three services are mandatory and optionally other services can also be part of the system. All configuration must be done in the Config module.

DB Web Service must also run according to the tasks planned to be created, executed.

The default configuration applies to the microservices system mode.

The web application runs on host and listens on port .

Standalone web application mode

In standalone web application mode the Task Manager and DB Web Service run alone, without the Config service, Discovery service and Gateway service.

The default configuration must be changed according to the standalone web application mode; set

eureka.client.enabled=false in application.properties file in Task Manager and DB Web Service as well.

DB Web service address should be set in com.chemaxon.taskmanager.service.jwsdb property in : format in Task Manager application.properties file. Example: http://localhost:8062

DB Web service also need access to Task Manager service. com.chemaxon.taskmanager.service property in DB Web Service application.properties file need to be set in : format. Example: http://localhost:8068

Download

See here.

Software requirements

See here

Installation

See here.

Module is installed into folder: jws/jws-taskmanager

Licenses

See here.

" JChem Microservices DB " license is needed.

Logging

See here.

Configuration

**Default configuration: **

application.properties
server.port=8068
logging.file.name=../logs/jws-taskmanager.log
eureka.client.enabled=true
bootstrap.properties
spring.cloud.config.failFast=true
spring.cloud.config.uri=${CONFIG_SERVER_URI:http://localhost:8888/}
spring.cloud.config.retry.initialInterval=3000
spring.cloud.config.retry.multiplier=1.2
spring.cloud.config.retry.maxInterval=60000
spring.cloud.config.retry.maxAttempts=100

For more settings possibilities see spring documentation page.

Database configuration

Task manager service has own database to store added structures and task statuses. H2 and PostgreSQL databases are supported.

**Default configuration: **

application.properties description
${CXN_TASK_JDBC_URL:jdbc:h2:file:./data/task_db}
spring.datasource.driverClassName=${CXN_TASK_DRIVER:org.h2.Driver} Value is org.postgresql.Driver in PostgreSQL case
spring.datasource.username=${CXN_TASK_JDBC_USER:user}
spring.datasource.password=${CXN_TASK_JDBC_PASSWORD:password}
spring.jpa.database-platform=${CXN_TASK_DIALECT:org.hibernate.dialect.H2Dialect} Value is org.hibernate.dialect.PostgreSQLDialect in PostgreSQL case

File and S3 import configuration

Task manager can import structure from tables which is exported from DB Web Services. It can be uploaded from file or S3 bucket.

AWS credential configuration details can be found here.

Import setup can be configured with below properties.

application.properties description
com.chemaxon.webservices.db.import.dbImportStrategy=FILE Specifies whether use FILE based import or S3 based. Default is: FILE
com.chemaxon.webservices.db.import_export.dir=./data/export Import file path in FILE based import strategy case
com.chemaxon.webservices.db.import.s3BuckeBasetUrl:s3://export-bucket/ Import S3 bucket in S3 strategy case.

DB Web Services configuration

Task manager is integrated with DB Web Services. Below configuration can be added to DB Web Services application.properties file regarding task manager communication.

application.properties description
com.chemaxon.taskmanager.scheduler.enabled=false When task manager starts a job in DB Web Service only one DB service instance starts processing regardless of running instances. If this attribute is true all DB service instances check running task and join processing (e.g. molecule import). If DB Web Service instance is restarted during task is in progress then process continues if this attribute is configured as true. Default value is false.
com.chemaxon.taskmanager.scheduler.frequency=60000 DB Web service tries to join running task with this regularity if com.chemaxon.taskmanager.scheduler.enabled=true. It is in milliseconds.
com.chemaxon.taskmanager.service=http://${TASKMANAGER_SERVICE_URL} When DB Web service runs in standalone mode it can connect to task manager via this URL.
com.chemaxon.taskmanager.db.batchSize:5000 DB Web service request structure for processing from task manager with this size.

Retry mechanism

Task manager has a retry mechanism implemented in it when it communicates with DB Web Service. This is adjustable with these two properties in Task Manager application.properties file

application.properties description
com.chemaxon.retrytemplate.backOffPeriod=10000 It is the time in milliseconds the service waits between two attempts to send the request to the other service again if no response was given.
com.chemaxon.retrytemplate.maxAttempts=20 The maximum number of attempts. Note, that it also contains the very first call.

With the above written default values the service tries to get a response for about 3 minutes.

When DB Web service is restarted during import structure can get stuck in progress state. Task manager reprocess these structures according to below configuration.

application.properties description
com.chemaxon.taskmanager.scheduler.frequency=60000 Regularity of task manager job which identifies stuck sructures. It is in milliseconds.
com.chemaxon.taskmanager.scheduler.timelimit=600000 Time limit before task manager tries to reprocess stuck structure.
com.chemaxon.taskmanager.scheduler.retrylimit=3 Number of structure process retry

High Availability (HA)

Running more instances of the task manager service ensures HA and load balancing.

HA mode requires PostgreSQL database. It is not supported with H2 database.

Running the server

Prerequisites in case of microservices system mode:

1.Config service is running

2.Discovery service is running

3.Gateway service is running

Run the service in command line in folder jws/jws-taskmanager:

jws-taskmanager-service start (on Windows)

jws-taskmanager-service start (on Linux)

or

run-jws-taskmanager.exe (on Windows)

run-jws-tarkmanager (on Linux)

API documentation

Find and try out the API on the Swagger UI.

Mode URL of Swagger UI Default URL of Swagger UI
microservices system <serverhost>:<gateway-port>/jws-taskmanager/API/ localhost:8080/jws-taskmanager/API/
standalone web application mode <serverhost>:<server-port>/API/ localhost:8068/API/

Usage

The guidelines on the Swagger UI API documentation of your installed module display the methods and syntax implemented for reaching the functionalities of the Task Manager toolkit.

JWSDB tasks

Task manager can be applied for importing structures into an already existing table. The table must be created at the endpoint /rest-v1/db/additional/createTable/{tableName} of DB Web Services.

The structures can be added via rest endpoint or can be uploaded from file or S3 bucket.

The structures will have ID data in the table. If the structures to be imported have their own IDproperty, this IDproperty can be set as ID in the table. Add structure endpoint (PUT - /rest-v1/work-queue/task/{taskId}) has id attribute which is also set ID in DB Web service side. It is recommended to provide ID property.

** Example - Import structure **

  1. Create task

PUT - /rest-v1/work-queue/task

1
2
3
4
5
6
7
8
{
  "params": {
    "isDuplicateFiltering": "true",
    "tableName": "mytable"
  },
  "service": "jwsdb",
  "task": "batchInsert"
}
  1. Add structures with ID

PUT - /rest-v1/work-queue/task/{taskId}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
  "inputFormat": "smiles",
  "structures": [
    {
      "id": 1,
      "structure": "O=C1CCCC=C1"
    },
    {
      "id": 2,
      "structure": "C1CCCCC1"
    }
  ]
}

Note: Provided ID will be used as ID in DB Web Services import.

  1. Start task

POST - /rest-v1/work-queue/task/{taskId}/manage

1
2
3
{
  "action": "START"
}
  1. Monitor task progress

GET - /rest-v1/work-queue/task/{taskId}

When task is in progress the response contains the processing percentage.

1
2
3
4
{
  "status": "IN_PROGRESS",
  "processedPercentage": 12
}

If import is done task is placed in Ready status.

1
2
3
{
  "status": "READY"
}

If task has any issue the status is updated to ERROR and response contains number of processed and failed.

1
2
3
4
5
{
  "status": "ERROR",
  "processedSuccess": 0,
  "processedError": 2
}
  1. Task result

GET - /rest-v1/work-queue/task/{taskId}/results?resultIdType=ALL

When task has finished the result can be retrieved.

Example result in success import case.

1
2
3
4
5
6
{
    "failedIds": [],
    "successfulIds": [1, 2],
    "duplicatedIds": [],
    "idType": "requestedId"
}

Example result in error case.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
    "failedIds": [{
            "id": 1,
            "errorMessage": "Could not save data to db: com.chemaxon.webservices.db.exception.TableDoesNotExistsException: there is no table registered for name: mytable"
        }, {
            "id": 2,
            "errorMessage": "Could not save data to db: com.chemaxon.webservices.db.exception.TableDoesNotExistsException: there is no table registered for name: mytable"
        }
    ],
    "successfulIds": [],
    "duplicatedIds": [],
    "idType": "requestedId"
}

Note: idType is requestedId, if ID was provided in the Add structures with ID step.