> For the complete documentation index, see [llms.txt](https://edward-stevinson.gitbook.io/tech-notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://edward-stevinson.gitbook.io/tech-notes/machine-learning/nlp.md).

# Natural Language Processing

## Functionality

Some of the most common functionality involved in NLP:

### Tokenisation

### Part-of-Speach tagging

Assigning wod types to tokens, e.g. word or noun.

### Dependency Parsing

Assigning syntactic dependency labels; describing the relations between individual tokens, such as subject or object.

### Lemmatization

Assigning the base forms of words, e.g. the lemma of 'was' is 'be'.

### Sentence Boundary Detection

Segementing sentences.

### Named Entity Recognition

### Similarity

### Text Classification

## Methodology

Many NLP libraries encode all strings to hash values to reduce memory usage and improve efficiency.

## Eigen Terminology

* **Selector** - a function that defines spans.
