Agent signaling from the outside

Rationale

RHQ is currently very reactive on things. For example we run auto discovery on things after some time, but we are not able to take "hints" from the outside. To illustrate this: Users can insert removable media into a managed platform. If auto discovery happens to kick in while the media is mounted, we will see it in RHQ. When the media is ejected on purpose by the user, it will be dangling in inventory as unavailable.
The same could happen with e.g. network interfaces, or a virtual apache host that is added by some robot scripts outside of RHQ. Or when the platform gets a new IP-address assigned.

This has downsides:

we can only take monitoring data or other stuff from the point in time we auto discover the new resource (and for metrics on services this can up to 20 mins with default settings (and more for traits)).
Through auto discovery, we have no idea if a change in system state was on purpose or by chance

In some systems, there is a notification mechanism where such system events can generate a notification. Otherwise scripts can "ping" other subsystems about this.

Distinction to the Events we have

In RHQ we have Events coming into the system. Currently those are only lines in log files, but in theory this could also be SNMP traps or other means.
Those events are nice to display in the GUI, but we have no way of feeding that information into other subsystems like inventory and discovery.

Two possible solutions

I currently see two possible solutions here - that don't need to close out each other.
The described solution here does not depend on how the external "ping" comes into the system (see below Interfacing)

Agent based signaling

The agent gets another input listening on events (socket, snmp traps, logfiles, ...). When one comes in, it parses it and takes appropriate action. Most of the stuff needed here could be covered by the existing prompt commands. So when a new network interface or apache vhost comes into existence, we could trigger an auto discovery run in the agent itself. If the external source provides us with the type of resource, this could scan could be limited to the appropriate plugin.

Correlation Unit based signaling (server side)

In this scenario we are using the existing events (again with various input listeners, as in the other scenario). With the help of correlation units we can filter matching events and run operations on the agent accordingly (like we do with alerts today).

Interfacing

We need to support different ways of signaling - we currently have log file parsing. But this is a bit hairy to set up, especially when the agent should get signaled.

I definitively see a text based interface with simple commands, as system init scripts could simply use tools like "netcat" to signal when a resource is brought up and down.