Ditm is a high performance mediation engine software, capable of normalizing
event records, transform the source data to a common format, allowing further processing.
During preprocessing records of input data the main elements are
- Extracting/fetching records from different sources (binary file, text file, table). Decoding/encoding these ones according to type of them.
- Record conversion to other (usually common) format. This format is right on formal (sintactic) consideration.
- Record transformation/normalization/enrichment. The result is already right on semantic consideration.
- Selection of records based on some conditions.
- Sending/loading records to output according to result of selection.
- Giving fast and efficient input record preprocessing
- Tracking of processes would be simple and clear (log, report)
- Easy to set parameters
- Helping to set the most proper parameters
- Preferably using processes in transaction (according to the applied platform)
- Using efficient codes
- Using source code close to OS, without plus layers (compiling to different platforms)
- Using only necessary outer libraries
- Intensive memory usage
- Using multithread technology
- Using plug-in dynamic libraries (they load up only when they needs)
- Partitioning of program along functions
- Core (common) modules
- Client specific modules
- ditm is a "quasi" out of box product
- Making the specific modules considerable as a customization
- Processing of records from different type sources
- Encoding the records
- Converting the records to common format
- Transforming/enriching the records to common semantic format
- Selecting records according to given conditions
- Writing out the records based on result of selection to different locations (file, table) in different format (binary, text, table)
- Using only necessary parameters loaded from different sources
- Possibility encrypting some important parameters in the configuration file
- Supporting to set the most proper parameters
- Possibility composing own processes (drawers, elements, interfaces)
- Supporting multithread running
- Using dynamic (plug-in) modules
- Supporting unicode characters
- Compiling to unix, linux, windows platforms
- Using different type of interfaces (file, odbc, oracle)
- Using fast, run in memory, intern data processor for configuration and parameter data
On the figure below the green boxes denote the elements of core part and the red ones show the client specific elements.
This is an possible example. In other cases the specific part may be different.
Core part tasks
- Controlling the static and dynamic modules, libraries
- Ensuring the cooperation of coexistence threads
- Managing the configurations (license, supported and custom options)
- Handling parameters (loading, using)
- Controlling the log file
- Preprocessing input files for full process (open-close, lock, archiving)
- Error handling
- Providing interfaces (file system, database system)
- Ensuring tool routines to the all processing
- Handling specific modules (load, run, unload)
Specific part tasks
- Defining the configuration (license, supported options)
- Managing clients specific elements of parameters
- Running processes along given specific configuration
- Processing event records according to client requirements (fetch, conversion, transformation, selection, output)
- Making reports
On the figure below you can see the behavior of ditm on different platforms used different data stores.
The axis with numbers show how many records were written out to data store in one second.
You can find more details in white paper.
You should consider these results only to get any order of magnitude, because the result solidly depends on your platform performance,
your client specific tasks, data store performance, etc.