DATA PREPARATION
Data integration: from scattered sources to one clean dataset
Data integration with TIMi uses Anatella to extract data from more than 80 sources, clean and transform it, load it where it is needed and verify the result, on one standard server instead of a cluster.
80+ sources
Databases, files, web services and streams.
Multi-threaded
One transformation uses all the CPUs of the server.
Compressed files
Reads compressed text files natively.
HOW IT WORKS
How data integration works.
Three steps, in one engine, on a standard server with no cluster.
1. Extract
Anatella reads relational databases, files in many formats, web services and streams.
2. Transform
Join, filter, aggregate and derive new columns. Complex steps can use JavaScript, R or Python.
3. Load and verify
Results are written to databases, files or BI tools, with checks such as duplicate keys before the load.
AnatellaGETTING IT RIGHT
Choices that shape an integration project.
Most of the effort lies in understanding the sources, not in moving the data.
Start from the use case
Decide the dataset your analysis needs, then work back to the sources. This avoids integrating data nobody uses.
Plan the verification
Row counts, duplicate keys and consistency checks are part of the flow, so errors show up before they reach the reports.
Think about the size early
Anatella handles very wide tables, with more than 50,000 columns, which suits analytical datasets better than a classical ETL.
TRANSPARENCY
A pipeline you can read.
An integration flow is a graph of boxes. Each transformation is visible, in order, so a new team member or an auditor can follow how the final table was built.
For data teams
Each step of the pipeline is visible and can be debugged, including JavaScript steps.
For quality
Checks are part of the flow and run every time.
For maintenance
A flow can be versioned, scheduled and reused.
DATA
What data can be integrated.
Anatella connects to more than 80 sources, so this data can be gathered without prior custom development.
Relational databases
Oracle, SQL Server, MySQL, Teradata and others through ODBC and OLEDB.
Files
CSV, Excel, SAS, SPSS and Stata, including compressed text files.
Web services
REST APIs and other web sources.
Streams
IoT streams over MQTT.
PROOF
Integration at scale.
These references show Anatella running large integration and ETL workloads.
Billing ETL from 1,520 hours to 80 hours
Codensa reduced the processing time of its billing ETL from 1,520 hours to 80 hours.
150 billion transactions per month, on one server
Snype processes 150 billion transactions per month in 5 to 7 hours, on a single standard server.
FAQ
Common questions.
Do we need a cluster for large volumes?
No. Anatella runs on a single standard server and uses all its CPUs. Horizontal scaling is possible if needed, without rewriting the flows.
Can it replace our current ETL tool?
It can, depending on your setup. A migration is best scoped on a sample of your existing jobs.
Which sources are supported?
More than 80, including relational databases through ODBC and OLEDB, files, web services and MQTT streams.
Is the integration code open to inspection?
Flows are visible as graphs. Steps written in JavaScript, R or Python can be read and debugged. Other actions are compiled.
RELATED USE CASES
Keep exploring.
Data Quality and Cleaning
Detect and fix errors, duplicates and inconsistencies in your data.
Data Warehousing
Centralize analytical data with fast columnar storage on standard hardware.
BI Tool Integration
Feed Tableau, Qlik, Kibella or any database from automated pipelines.
SEE FOR YOURSELF
Test it on your own data.
Run TIMi on your own data with a free trial, or talk to us about your case.
