One of the most frequently asked questions we get is how customers can monitor particular topics of interest using open source tools such as Elasticsearch or Hadoop. In this write up we will talk about some methods in which you can setup alerting and collection of information using open source tools.
Getting the News Content
For this exercise to work properly we have to get some news articles or data that is of interest to us. A good source of news may be news.google.com or possibly RSS feeds that contain information that may be of interest. In this example below we will use RSS as the source and we will feed Elasticsearch. In reality you can use this method to really pull in pretty much any source of data.
Using Logstash to read RSS locations
Since were interested in the ingestion of RSS data into Elasticsearch we can use Logstash from the ELK stack to read RSS configuration. Below is an example of reading an RSS feed with logstash.
Logstash Input
rss { url => "https://www.yahoo.com/news/rss/mostviewed" interval => 7200 tags => ["en", "rss", "yahoonews"]
In this example the Interval will tell the running logstash process how often to read this source for new data. It really is just that simple.