Key takeaway
Choose a language tool for a defined task: matching text for search, extracting fields, classification and open-ended generation are different needs.
NLP processes text and speech for a task
Natural language processing (NLP) covers methods that let computers process human language in text and speech. A system can target a defined task: split text into tokens, recognize entities such as names and IP addresses, classify a topic or produce a summary. An email filter may classify a suspicious message, while incident search may extract an indicator from log text. These tasks do not necessarily amount to full understanding of intent and context; check results against the intended use.
Tokenization
Splits text into processing units such as words or word pieces.
NER
Labels entities in text, such as names, organizations, locations or IP addresses.
Classification
Assigns a message to a defined category, such as spam or legitimate.
How stemming and lemmatization group word forms
Stemming applies rules to strip endings, and sometimes prefixes, to produce a stem that may not be a dictionary word. Lemmatization uses vocabulary and grammar to find a base form in context. A stemmer may group “connect,” “connected” and “connecting,” yet over-stem unrelated words such as “university” and “universe.” It may also fail to group related forms such as “alumnus” and “alumni.” Word formation differs by language, so choose a suitable stemmer and check real text for both over-stemming and under-stemming.
| Approach | Stemming | Lemmatization |
|---|---|---|
| Output | May produce a form that is not a word | Dictionary form, or lemma |
| Processing method | Lightweight, approximate matching | Matching based on vocabulary and grammar |
A generative LLM builds a response token by token
A large language model (LLM) learns language patterns from large text collections. A generative LLM uses those patterns to produce text. It splits input into tokens and maps each token to a numeric representation called an embedding; in a Transformer-based model, self-attention computes context-dependent relationships among those representations. Using learned parameters, the model estimates the next token and builds a response step by step. Training fits the parameters; inference applies the trained model to a new prompt. Fluent output does not guarantee accuracy: the prompt and connected data also shape the response.
Training
Text examples and a chosen objective shape the model parameters.
Inference
The model generates a probabilistic text response to a new prompt.
Check inputs and outputs in language workflows
NLP can help filter phishing messages, extract indicators from threat reports and summarize incident records. The same capabilities can also produce convincing phishing text. An LLM may treat text from a webpage or document as instructions, state a false fact confidently or include sensitive context in its answer. Check outputs against sources and human review; limit an action-taking integration to the permissions its task requires.