Web scraping and data scraping are particularly popular on Wall Street. Web scraping is the practice to extract data from a web site automatically and feed them to a data base for further data analysis.
Investment firms and particularly hedge funds make a variety of use of web-scraping. A few years ago the idea of social media hedge fund was born. A scientific paper published at the university of Indiana demonstrated a level of correlation between sentiment in twitters short messages and the next day appreciation or depreciation of the NYSE index.
However web scraping has been utilized in wall street to mine data in connection with events, macro economics trends, real estates prices and more. Ultimately web scraping and data scraping is used to predict directional changes and price variability of equities and derivatives.
Google to some accounts, is poised to enter the investment management industry. Both the searches on its search engine and the scraping performed by robots could be used to make macro economic prediction as well as event predictions on equities.
Typically the professional utilization of automated web-scraping is confined to the realm of automated quantitative investments. The more down to the heart investors utilize investment strategies resting on fundamental analysis.