DATA & AI
Text mining: make free text usable in your analyses
Text mining with TIMi uses Anatella to correct spelling errors, match names and addresses fuzzily, translate and classify text, and turn raw text into variables that predictive models can use.
Spelling correction
Fix errors in names, addresses and free fields.
Fuzzy matching
Join tables on sounds and approximate spellings.
Translation
Translate text from one language to another.
HOW IT WORKS
How text mining works.
Three steps, in one engine, on a standard server with no cluster.
1. Prepare the text
Anatella reads text fields and documents and normalizes them.
2. Transform the text
Spelling correction, fuzzy matching, translation and bag-of-words turn text into clean values and new columns.
3. Use it in models
The new columns join the dataset that Modeler uses for prediction, or feed classification of texts.
Anatella
ModelerGETTING IT RIGHT
Choices that shape a text mining project.
Text is only valuable once it becomes a field a model or a person can use.
Start with the question
Decide what you want to learn from the text: a category, an entity, a sentiment, or a variable for a model.
Language matters
Spelling correction, matching and translation depend on the languages in your data. Check coverage for yours at scoping.
Sensitive text stays in house
Built-in operators run on your own servers. An optional connector to the Rosette API calls an external service, so avoid it for sensitive text, or validate it with your security team first.
TRANSPARENCY
Variables you can read.
The bag-of-words approach produces plain columns, so a model that uses them can show which words or expressions drive a prediction. The transformations are steps in a graph, open to review.
For analysts
See which words influence a prediction.
For reviewers
Each text transformation is a visible step.
For security
Built-in operators run on your own servers.
DATA
What text data can be processed.
Anatella connects to more than 80 sources, so this data can be gathered without prior custom development.
Names and addresses
Entry errors and spelling variants.
Messages and comments
Customer comments, tickets and notes.
Documents
Reports, case files and records.
Reference lists
Lists used for matching and extraction.
PROOF
Text mining in production.
This reference shows text mining running on a regular refresh cycle.
Text mining refreshed in under 6 hours
DIJIN runs text mining on its data with a refresh cycle of under six hours.
FAQ
Common questions.
Can it classify documents?
Yes. The bag-of-words approach produces variables that Modeler can use to classify texts, subject to enough labeled examples.
Which languages are supported?
The built-in operators handle many languages, and the Unicode support covers any character set. Quality varies by language, so validate on a sample of your data.
Does the text leave our servers?
Not with the built-in operators. The optional Rosette connector sends text to an external API, which you can avoid for sensitive data.
Can it join two lists that spell names differently?
Yes. Fuzzy matching joins tables on approximate spellings and sound encodings.
RELATED USE CASES
Keep exploring.
SEE FOR YOURSELF
Test it on your own data.
Run TIMi on your own data with a free trial, or talk to us about your case.
