Support continuous ingest of message data and parallel metric computation.

[manthey]

So far, this will ingest instagram as pulled from elasticsearch (such as that supplied with the July data), and twitter in either GNIP or firehose JSON (as supplied with the July data). It will not work on the original instagram data that Roni pulled fro parquet files and saved as CSV. It should be easy to extent it to that format (though it has less data for comments and likes, so won't be as rich).

Links are create from mentions, comments (instagram only), likes (instagram only), and replies (twitter only). Because of the comments and likes, Instagram refers to a vastly larger network of entities than twitter. It also has fewer messages per entity. The Instagram network is sparser than the Twitter network.

Ingest and metric computation is fully parallelizable - multiple ingests can run concurrently. Each metric can be computed in a separate process. A possibly useful extension would be to subdivide the entity space so that a single metric can be parallelized with multiple processes as well.

I've implemented four metrics so far. I have not yet implemented the metric based linking between twitter and instagram.

The implemented metrics are:

1hop - This is quick to compute, as I maintain a list of neighbors within the entity records. If the entity is updated, it will be recomputed. The stored value is the number of 1-hop neighbors EXCLUSIVE of the root entity itself.

2hop - This requires visiting all the 1-hop neighbors. Since we don't know which of those neighbors has been updated, any entity update requires that we recheck all 2-hop values. I store both the number of 2-hop-only neighbors and the number of 2-hop and 1-hop neighbors. These counts never include the root entity itself.

substring - This does a longest substring match between ALL entities, keeping the top-k for each service and subset. The substring calculation has been refactored to be about twice as fast as Curt's original implementation. When an entity is added or updated, only the modified entities need to be rechecked to maintain the top-k lists. This is still slow on a large collection.

levenshtein - This computes the levenshtein metric between ALL entities, just like substring. It is even slower than substring matching.

The substring and levenshtein metrics would be helped by more CPU resources, but would be helped even more by excluding entities from the candidate match. For instance, if the substring match is 0 (or below some other threshold), we could skip computing the levenshtein metric.

[jeffbaumes]

Nice @manthey. It would be great to make a README with some of this info in it, and/or set up a Sphinx RTD site for this and include the nice function documentation you have in the code.