What is in here
Documents
show
The document, stage by stage
document
The matrix
One document's vector
document
Where that number came from
term
in document
TF-IDF is not a black box: it is a count, a logarithm and a division. This panel does the arithmetic in front of you and then checks its answer against the cell in the matrix.
Hand it on
write the top terms
The seam of the portfolio: once text is a matrix of numbers, VedaForge can treat those columns as features and VedaML can model them. Nothing about the row of numbers remembers that it used to be a sentence.
n-gram size
top
word
window
The concordance is the oldest tool in corpus linguistics and still the fastest way to see how a word is actually used, rather than how you assume it is.
rank by
seen at least times
The lexicon (VADER)
7,506 words with hand-tuned valences, plus rules for negation, degree words, capitals and exclamation marks. No training, no context, no idea what a sentence means. It runs in a second and shows all its working, which is exactly why it is still here after fifteen years.
neutral band ±
Why this document got that score
document
Each word is removed and the sentence re-scored: the difference is what that word was worth to THIS score. It is the same move as permutation importance, and it needs no cooperation from the model.
The neural track
DistilBERT fine-tuned on SST-2: a real transformer, running in this browser through transformers.js. The model is about 65 MB and downloads only when you click. Afterward it is cached. It reads word order and context, so it hears things a lexicon cannot; it also has no neutral class, and no idea it is talking about aeroplanes.
score the first documents
Where they disagree
A classifier over words
TF-IDF into a linear model, cross-validated. The score is not the point: the coefficients sit over words, so the model can be read out loud. A leaderboard of tuned models is VedaML's job, and the matrix export is the way there.
model
folds
C / alpha
What it learned, in words
Where it went wrong, readable
A confusion matrix says how often. These are the documents themselves, which is the only way to find out why.
Fit a topic model
method
topics
alpha
eta
iterations
The topics
relevance λ
0.6
λ = 1 ranks a topic's words by how probable they are inside it, which fills every topic with the corpus's favourite words. λ = 0 ranks by lift, how much more this topic uses a word than the corpus does, which surfaces the words that live here and nowhere else. Slide it: "the top words for this topic" was never one question.
The corpus, coloured by dominant topic
How many topics? Argue it, do not guess it
k from
to
Coherence asks whether a topic's top words actually co-occur in documents. It is a proxy, not truth, but it turns the topic count from taste into an argument you can defend.
One document's mixture
document
Extractive: choose the sentences that were already there
method
keep sentences
λ
edge cut
from
document
group
The graph it ranked
Every sentence points at every other with the weight of their similarity. PageRank then finds the fixed point: a sentence is important if important sentences are similar to it. The ring is the document in order; the thicker the line, the more two sentences have in common.
Abstractive: write new sentences
DistilBART fine-tuned on CNN news, running here through transformers.js. About 250 MB, downloaded only when you click, cached afterward. It does not choose sentences: it generates them, which is why it can be fluent, and why it can be wrong in ways an extractive summary never can.
Two spaces, and what each one knows
dimensions
first documents
LSA builds a space out of THIS corpus: it can only know that two documents are alike because they share words, or share words with the same third words. MiniLM was trained on a billion sentence pairs, so it arrives already believing that a bag and a suitcase are the same idea. Build both and search the same query in each.
Search by meaning
What is this document near?
document
Cluster the corpus
k
Patterns: the half that needs no model
A PNR is six capitals sitting after the word booking. A flight number is a carrier code and digits. A date is a date. Spending 65 MB of transformer on those is a way of saying you never looked at your data. These are exact, instant and utterly brittle: they find what they were told to find and nothing else.
The model: names, places, organisations
BERT fine-tuned on CoNLL news, through transformers.js. About 65 MB, only on click. It knows PER, ORG, LOC and MISC. It has never seen a PNR and will call one an ORG with total confidence, which is the useful half of this demonstration.
first documents
Zero-shot: label it with no training at all
DeBERTa-xsmall trained on natural language inference, through transformers.js. About 90 MB, only on click. It asks, for each label, whether "this text is about X" follows from the document. It has never seen your labels or your data. Compare it with section 6's classifier, which had to be shown hundreds of examples first.
labels
first documents
Everything this session worked out
Your corpus with the tokens, the sentiment from both tracks, the dominant topic and the zero-shot label bolted on. Columns VedaVerse added start with an underscore.
Into VedaForge and VedaML
Two seams. The document-term matrix has columns that mean something (a word each), which VedaForge can select and VedaML can explain. The dense vectors do not: column d17 means nothing on its own, and it is worth saying that out loud to whoever asks.
The code that reproduces this pipeline
Your tokenizer and your vectorizer, written out as the scikit-learn that does the same thing. Paste it into a notebook and it runs.