Loading the lesson
Loading the lesson
Search AllRounder.ai
Search the courses, subjects, tracks, games and features, or jump straight to a page.
Search the courses, subjects, tracks, games and features, or jump straight to a page.
3 cards from this section. Quick, good the night before a test.
What is tokenization?
Think about what smaller pieces text can be divided into.
Can you name a type of tokenization?
One focuses on words and the other on sentences.
What does tokenization do?
Focus on the definition of tokenization.
True or False: Tokenization is the first step in NLP.
Consider the order of operations in NLP.
Analyze the impact of poor tokenization on sentiment analysis results.
Consider how meanings might change when split incorrectly.
Design a simple program that implements word and sentence tokenization on a given text.
Think about the libraries available in Python for this task.
5 more questions in this section. Enroll free to practise them all.
Enroll free