Scalability

    How scalable is JChem Cartridge / JChem Base?

    Lutz Weber's presentation about very large databases (200 million structures) and 2D- and 3D-Similarity Searching.

    What is the maximum number of structures for JChem Base / JChem Cartridge?

    In a single table the maximum number of rows is 231-1 (2,147,483,647).
    We may increase this in the future if needed.
    There is no practical limit for the number of tables.

    Hardware requirements

    How can I estimate hardware requirements for a JChem database?

    1. Memory:
      JChem may use significant amount of memory for building a structure cache in memory.
      Please see this topic on how to estimate the size of the structure cache.
      The total Java memory consumption (heap size) consists of:
      1. The size of the structure cache
      2. Processing memory for JChem Base: we recommend to multiply the estimated cache size by a factor of 0.75 (up to 8 processor cores). In case of more than 8 cores you may have higher memory need since it scales up with the number of the threads started.
      3. Processing memory for your Java application (built on JChem Base)
        Cache and Java memory usage measured for different table size on a 8 core 64-bit Linux (version 3.2.0-48-generic) environment:

        Target count

        Cache size measured (MB)

        Cache size estimated (MB)

        Java memory allocation (MB)

        986,581

        148.85

        152.45

        250

        5,032,252

        701

        720

        1,250

        10,002,807

        1,610

        1,637

        3,000

        19,987,542

        3,174

        3,206

        6,500

        38,221,498

        6,002

        6,144

        12,000

    The Java memory usage could be further improved (compared to the table values) with fine-tuning the JVM parameters.
    To determine the total amount of RAM requirement you should add:

      • Java memory consumption (detailed above)
      • Other programs running on that computer
      • Memory requirement for the operating system

    64 bit systems: A single Java process cannot allocate more than around 2 GB on 32 bit systems. If your Java memory needs will exceed this limit, a 64 bit system is recommended (including hardware, operating system and Java).

    Please, also visit this link on how to enable the Java Virtual Machine to allocate the necessary amount of memory.

    • Disk space:
      Apart from the number of structures it also depends on the format of the input file, and type of RDBMS engine used by JChem Base.

    A benchmark result with 1 million structures (the NCI dataset was multiplied for the test), the RDBMS was Oracle:

    Format

    Size of JChem table

    Original file size

    SDF

    1.2 GB

    3.7 GB

    SMILES

    270 MB

    50 MB

    Note: JChem compresses SDF in the cd_structure field by default (in the case of the NCI dataset to roughly about 3 times smaller). This can be disabled (e.g. if they need to be displayed directly by non-ChemAxon tools), but the storage size increases in this case.

    • Processing power:

      Comparing the performance of different hardware architectures is a complex topic and not the subject of this FAQ. 

      Some quick facts:
      • JChem automatically uses all processors during a search
      • The search speed scales well with the number of processors
      • The search time is directly proportional to the size of the table (assuming there is no constraint on the number of hits - see below)


    The following benchmarks can be used as a starting point


    For the same database the search time greatly depends on the type of query structure. For a very general query (e.g. benzene) there will be a lot of hits, meaning longer execution time, while more specific queries run very fast on the same large database. 

    The search time consists of

      • Cache load time : the cache is built up during the very first search
      • Screening time : this is a quick pre-filtering using fingerprints
      • Graph search: slower, but only performed for the screened compounds

    The following can be stated:

      • The screening time is directly proportional to the size of the database table.
      • The graph search time is directly proportional to the number of screened compounds.
      • The number of hits is roughly proportional to the number of of screened compounds.
      • The graph search time is roughly proportional to the number of hits.


    Tip: it is rarely useful to return a huge number of hits (especially for human consumption).
    If the number of hits is limited, only the rapid screening time will increase with table size, which means the total search time will remain almost constant regardless of the table size.

    How to estimate the memory need for the Structure Cache?

    Typically 1 million drug-like structures consume around 160 MB memory in the structure cache of JChem.
    Note: although JChem can drop the least recently used table from the structure cache if low on memory, it is recommended that all structure tables should fit in the cache (as cache loading can take a considerable amount of time). When estimating the memory need simply sum the number of rows in the tables.
    The following table shows typical memory needs for standard structures:

    Test specifications

    Number of molecules:

    10,000,000

    Fingerprint size:

    16*32=512 bits

    Average SMILES length per molecule:

    60.4

    Memory consumption:

    1,604.1 MB

    Caching time:

    900 seconds

    Memory need increases with the number of molecules, the size of fingerprints, and the average SMILES string size. The following approximation can be used when number_of_molecules > 25000:

    memory_need[bytes] =
        number_of_molecules * (0.5 * average_smiles_length[characters] + fingerprint_size[bits] / 8 + 74)

    The structure table fingerprint statistics generation function can be used to report the average smiles length and fingerprint size of a JChem table or JChem index. See more information in the JChem Manager command line usage (s command), at the Cartridge index statistics function and the Statistics tab at Instant JChem Schema editor.

    Benchmarks

    How fast is importing in JChem Base?

    The following table shows the duration of import in some cases. The configuration was the same as in the cartridge benchmark.

    Number of structures

    Elapsed Time (ms)

    Duplicates allowed

    Duplicates not allowed

    10000

    40406.0

    44923.0

    100000

    405781.0430840.0 

    200000

    824870.0883681.0 

    Notes:

    • If duplicates are allowed, time increases linearly with the number of molecules imported.
    • If duplicates are not allowed, JChem performs a search for every molecule to check if it is already in the structure table or not.
    • The table into which the structures were imported always contained an initial 10 structures and were stats-collected

    How fast is substructure searching in JChem Base?

     

    The following tests demonstrate the speed of substructure search in JChem.
    The test configuration was the same as in the cartridge benchmark, and the same query structures were used.

    • A chemical table containing 38,165,924 molecules was used;
    • Fingerprints and SMILES were cached by JChem;
    • JChem version: 6.3.0;
    • Java version: Oracle Corporation 1.7.0_17;
    • OS: amd64 Linux 2.6.18-164.0.0.0.1.el5xen.

    Query Structure

    Search Options

    Number of Hits

    Screened Count

    Search Time (ms)

    t:s

    0

    0

    3116.0

    t:s

    0

    0

    3132.0

    t:s

    3

    3

    3506.0

    t:s

    77

    115

    3157.0

    t:s

    93

    193

    2780.0

    t:s

    117

    117

    2775.0

    t:s

    137

    137

    2787.0

    t:s

    696

    720

    3153.0

    t:s

    1127

    1337

    3570.0

    t:s

    3841

    3973

    3155.0

    t:s

    10047

    10421

    3647.0

    t:s

    233472

    233852

    5620.0

    t:s

    5937200

    5990786

    76444.0

    t:s

    8339436

    9189891

    129767.0

    The column names have the following meaning:

    • Query Structure: The query tested.
    • Search Options: The search options applied.
    • Number of Hits: The number of structures returned by the query.
    • Screened Count: The number of structure left over from the fingerprint screening as possible candidates meeting the search criteria.
    • Search Time: The total time spent executing the query.

    How fast is inserting in JChem Cartridge?

    The following table shows the duration of inserting in some cases. The configuration was exactly the same as in the cartridge benchmark.

    Number of structures

    Elapsed Time (ms)

    Duplicates allowed

    Duplicates not allowed

    10000

    169040.0

    191999.0

    100000

    1684728.0

    3101339.0

    200000

    3555535.0

    9296357.0

    How fast is indexing in JChem Cartridge?

    The following table shows the duration of indexing in some cases. The configuration was exactly the same as in the cartridge benchmark.

    Number of structures

    Elapsed Time (ms)

    Duplicates allowed

    Duplicates not allowed

    10000

    12602.014881.0 

    100000

    72729.0782445.0 

    200000

    145654.0

    3500274.0

    How fast is searching in JChem Cartridge?

    The following table shows the duration of search in JChem Cartridge using the following configuration:

    • JChem version: 6.3.0;
    • Oracle version: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit;
    • Hardware: Amazon EC2, m2.4xlarge instance, EBS storage;
    • Other environmental data;
    • Target molecule set: PubChem;
    • Number of structures: 38,165,924.
      Session Date : 2013-10-05

      Query Structure

      Search Options

      Number of Hits

      Screened Count

      Search Time (ms)

      t:s earlyResults:2000

      0

      0

      2803.0

      t:s earlyResults:2000

      0

      0

      3187.0

      t:s earlyResults:2000

      3

      3

      3512.0

      t:s earlyResults:2000

      77

      115

      2835.0

      t:s earlyResults:2000

      93

      193

      3181.0

      t:s earlyResults:2000

      117

      117

      2802.0

      t:s earlyResults:2000

      137

      137

      3151.0

      t:s earlyResults:2000

      696

      720

      3598.0

      t:s earlyResults:2000

      1127

      1337

      3210.0

      t:s earlyResults:2000

      3841

      3973

      3601.0

      t:s earlyResults:2000

      10047

      10421

      3337.0

      t:s earlyResults:2000

      233472

      233852

      6258.0

      t:s earlyResults:2000

      5937200

      5990786

      86830.0

      t:s earlyResults:2000

      8339436

      9189891

      143022.0

      t:t simThreshold:0.9

      0

      0

      4209.0

      t:t simThreshold:0.9

      0

      0

      4969.0

      t:t simThreshold:0.9

      0

      0

      2272.0

      sep=! t:s!ctFilter:(PSA() <= 200) && (rotatableBondCount() <= 10) && (mass() <= 500) && (aromaticRingCount() <= 4)

      130

      137

      3011.0

      sep=! t:s!ctFilter:(mass() <= 500) && (logP() <= 5) && (donorCount() <= 5) && (acceptorCount() <= 10)

      15

      137

      2962.0

    The column names have the following meaning:

    • Query Structure: The query tested.
    • Search Options: The name of the operator tested.
    • Number of Hits: The number of structures returned by the query.
    • Screened Count: The number of structure left over from the fingerprint screening as possible candidates meeting the search criteria.
    • Search Time: The total time spent executing the SQL statement.

    Tuning

    What can/should I do to make JChem Cartride searches faster?

    You can find a few performance tuning hints here.

    • No labels