Background on Activity Monitor Agent¶
Sentilo is a publication-subscription platform. The amount of data held in the system is proportional to Redis deployment and directly depends on the amount of physical memory available for the Redis server. In another words, the data has to be probably deleted after a certain amount of time to free the Redis memory. For example, in the Barcelona deployment, the data is deleted after approximately one week.
Additionally to data expiration, Sentilo does not provide many dashboards and those dashboards are not customizable.
In order to fill the gap of historization and dashboards, we use Elasticsearch and Kibana. Elasticsearch is a powerful Java-based fulltext search database with REST API. It is frequently used together with it’s modules, Kibana for dashboards and Logstash for collecting of logs. The combination of Elasticsearch, Logstash and Kibana is often called the ELK stack. ELK provides a comfortable way to store and exploit historical information, and also a near-realtime monitoring of the platform. Note that Elasticsearch behaves excellently in cluster mode.
Sentilo events are uploaded to Elasticsearch through a Sentilo agent called Activity Monitor Agent. The configuration of this agent is described further in this chapter.
The following image illustrates a possible setup of Sentilo with ELK stack. Logstash is optional and can be used e.g. for monitoring of Sentilo logs (like login errors, invalid messages etc.), as well as monitoring of system resources.
The setup of the ELK stack is well documented and beyond the scope of this page.
Activity Monitor Agent is configured with a set of .properties files in sentilo/sentilo-agent-activity-monitor/src/main/resources/properties.
|topics-to-index||Regexp pattern on event name that enables including/excluding events||
Examples of configuration:
Subscribes to all events
Subsribe only to data of 2 providers
|elasticsearch.url||URL of the ES instance|
|batch.size||How many evens are sent to ES at once.||Every HTTP request consumes certain amount of resources, thus is convenient to use a ES bulk API. The agent won’t send events to ES until batch.size events occurred.|
|batch.workers.size||Number of threads the agent||Determines how many parallel threads communicate with ES.|
|batch.max.retries||Number of retries when ES is unavailable||Number of intents for upload to ES instance.|
The agent will create index(es) called sentilo-YYYY-MM.
Configuration of Elasticsearch, Logstash and Kibana is beyond the scope of this document and can be easily followed on their respective web pages.
Sentilo has been successfully used in with these versions of ELK (which does not mean other versions shouldn’t work as well):
- Elasticsearch 2.3.1
- Kibana 4.5.3
- Logstash 2.3.2