Probabilistic Counting algorithm with stochastic averaging (Flajolet-Martin algorithm) was proposed by Philippe Flajolet and G. Nigel Martin in 1985.
It’s a hash-based probabilistic algorithm for counting the number of distinct values in the presence of duplicates.
This implementation stores number of 32-bit single counters (FM Sketches) consequently in a single bitvector.
from pdsa.cardinality.probabilistic_counter import ProbabilisticCounter pc = ProbabilisticCounter(256) pc.add("hello") print(pc.count())
Build a counter¶
To build a counter, specify its length.
from pdsa.cardinality.probabilistic_counter import ProbabilisticCounter pc = ProbabilisticCounter(number_of_counters=256)
Memory for the counter is assigned by chunks, therefore the length of the counter can be rounded up to use it in full.
This implementation uses MurmurHash3 family of hash functions which yields a 32-bit hash value that implies the maximal length of the counter.
The Algorithm has been developed for large cardinalities when
card()/num_of_counters > 10-20, therefore a special correction
required if low cardinality cases has to be supported. In this implementation
we use correction proposed by Scheuermann and Mauve (2007).
from pdsa.cardinality.probabilistic_counter import ProbabilisticCounter pc = ProbabilisticCounter( number_of_counters=256, with_small_cardinality_correction=True)
Index element into the counter¶
It is possible to index into the counter any elements (internally it uses repr() of the python object to calculate hash values for elements that are not integers, strings or bytes.
Size of the counter in bytes¶
Length of the counter¶
Count of unique elements in the counter¶
It is only an approximation of the exact cardinality.