Problem 1 [30 points]. Write a (Python) program that preprocesses a
collection of documents using the recommendations given in the
Text Operations lecture. The input to the program will be a directory
containing a list of text files. Use the files from assignment #3 as
test data as well as 10 documents (manually) collected from news.yahoo.com .
The yahoo documents must be converted to text before using them.
Remove the following during the preprocessing:
– digits
– punctuation
– stop words (use the generic list available at …ir-websearch/papers/english.stopwords.txt)
– urls and other html-like strings
– uppercases
– morphological variations
Need help with a similar or different Task?
We have the best writers to help you. Hire Writer Now
