root
/
Branch-AI
mirror of https://github.com/01-edu/Branch-AI.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
82 Commits
1 Branch
0 Tags
62 MiB
Tree: 32ba30b2a8
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '32ba30b2a8'
${ noResults }
Branch-AI/one_exercise_per_file/week03/day04/ex03/readme.md

648 B
Raw Blame History

Exercise 3 Tokenization

The goal of this exercise is to learn to tokenize as text. This step is important because it splits the text into token. A token could be a sentence or a word.

text = """Bitcoin is a cryptocurrency invented in 2008 by an unknown person or group of people using the name Satoshi Nakamoto. The currency began use in 2009 when its implementation was released as open-source software."""

  1. Tokenize this text using sent_tokenize from NLTK.

  2. Tokenize this text using word_tokenize from NLTK.

Ressource: https://www.analyticsvidhya.com/blog/2019/07/how-get-started-nlp-6-unique-ways-perform-tokenization/