Agent is used for streaming data into BangDB. It runs as a service on the server from where data is supposed to be streamed. Data could be streamed in continuous basis or in batch manner. The agent can also parse and structure data before sending to BangDB if configured. It can parse csv, key-val data and structure them into json or it can also use regex if supplied to structure the data.

Further, it also uses inbuilt default logic to parse and structure known data entities. And finally, we can also use IE (information extraction) to do NER (name entity recognition) to structure the data in probabilistic manner. However, the goal is to send data to BangDB structured or otherwise.

Agent could read data from a log file in continuous manner. Or it can read from other services using API or otherwise. It can also read using protocol like SMTP etc. Idea is to continuously keep reading data, in an eventing manner.

Agent can be queried or instructed remotely using cli. Following are the basic things that can be done by the agent.

// agent related commands
register collector /home/sachin/collector.conf where server = IP:PORT
activate agent collector_name where server = IP:PORT
pause agent collector_name where server = IP:PORT
describe agent all where server = IP:PORT
describe agent collector_name where server = IP:PORT
stats agent all where server = IP:PORT
stats agent collector_name where server = IP:PORT

Register a new collector with the agent on a remote server ( Let's say that agent is running on a server and we wish to instruct it to start collecting data from new file or source with a given configuration. We can do this by callingregister command).

register collector /home/sachin/collector.conf where server = 192.168.1.25:10102

Activate the collection from agent for given collector

activate agent collector_name where server = 192.168.1.25:10102

Pause the collection from agent for given collector

pause agent collector_name where server = 192.168.1.25:10102

To see the details of any agent, including what all collectors are registered, activated, paused etc.

describe agent all where server = 192.168.1.25:10102

To describe just a given collector

describe agent collector_name where server = 192.168.1.25:10102

To see the stats of the data collection so far

stats agent all where server = 192.168.1.25:10102

To see the stats of the data collection for a given collector so far

stats agent collector_name where server = 192.168.1.25:10102