Practice Tokenization (15.2.1.a) - Natural Language Processing (NLP)

Practise this section

9 questions

Flashcard drill
3 cards from this section. Quick, good the night before a test.
Exercise
6 questions · about 4 min · pass mark 70%. Hints available.
Sign up to start the exercise
Quiz
3 questions · about 2 min · pass mark 70%. Hints available.
Sign up to start the quiz

6 sample questions

Question 1.
What is the purpose of tokenization in NLP?
Show hintHide hint
Think about how text is structured.
Question 2.
List a type of token that can result from tokenization.
Show hintHide hint
What smaller components do texts consist of?
3Question 3.
What does tokenization do?
- Convert text to images
- Breaks text into smaller units
- Translates texts to other languages
Show hintHide hint
Think about what happens to sentences and how they're analyzed.
4Question 4.
True or False: Tokenization is the last step in the NLP preprocessing pipeline.
- True
- False
Show hintHide hint
Recall the order of steps involved in text processing.
Question 5.
Given the text 'The quick brown fox jumps over the lazy dog', implement a tokenization function that returns the tokens without punctuation.
Show hintHide hint
How can you use string methods for splitting and regex for cleaning?
Question 6.
Consider a text from social media filled with emojis and slang. Discuss how you would approach tokenization in this scenario.
Show hintHide hint
What special characters or words should we seek to define as separate tokens?

5 more questions in this section. Enroll free to practise them all.