Probability Algorithms: Mathematical Algorithm Deep Dive

1. Introduction and Core Concepts

Probability algorithms, more formally known as randomized or probabilistic algorithms, represent a fundamental class of mathematical algorithms that deliberately incorporate randomness as a core part of their logic. Unlike deterministic algorithms that produce the same output for the same input every time, probability algorithms make random choices during execution, leading to behavior that is non-deterministic but statistically analyzable. This integration of probability theory into algorithm design is not a concession to uncertainty, but a powerful technique for solving complex problems more efficiently, more simply, or in cases where deterministic solutions are infeasible.

At its heart, the power of probability algorithms lies in trading absolute certainty for dramatic gains in speed, simplicity, and scalability. For many computational problems, finding an exact deterministic solution requires exponential time or prohibitive memory. Probability algorithms offer a pragmatic alternative: they can provide an answer that is correct with very high probability, or an answer that is always correct but with a runtime that is probabilistic. This paradigm shift is essential for modern computing, where datasets are massive, real-time constraints are strict, and perfect accuracy can be relaxed in favor of practical performance.

Understanding probability algorithms requires familiarity with a few core concepts. The first is the source of randomness itself, typically a pseudo-random number generator (PRNG) that simulates true randomness. The second is the distinction between different classes of randomized algorithms. Las Vegas algorithms always produce a correct result, but their running time is a random variable. Monte Carlo algorithms have a deterministic running time but may produce an incorrect result with a small, bounded probability. A third, less common class, Atlantic City algorithms, may be incorrect with bounded probability in both directions and have probabilistic runtime. Grasping these distinctions is crucial for selecting the right probabilistic approach for a given problem and for analyzing its guarantees.

Why Probability Algorithms Matter

Probability algorithms matter because they unlock solutions to problems that are otherwise intractable. In cryptography, randomized algorithms are essential for generating secure keys and for primality testing of large numbers, which underpins modern internet security. In machine learning and data science, virtually every model training process, from stochastic gradient descent to Bayesian inference, relies on probabilistic sampling. In large-scale systems, they enable efficient data processing on streams too large to store in memory.

Furthermore, probability algorithms often provide the simplest and most elegant solution to a problem. Randomized QuickSort, for example, avoids the worst-case performance pitfalls of its deterministic counterpart with a single line of randomization, making it more robust in practice. This simplicity translates to less code, fewer bugs, and easier maintenance, which are significant advantages in production software engineering. As data volumes continue to grow exponentially, the ability to approximate answers quickly and with quantifiable confidence is no longer optional but a necessity.

2. How It Works — Detailed Explanation of the Mechanism and Theory

The theoretical foundation of probability algorithms rests on probability theory and the analysis of random variables. Instead of analyzing a single execution path, we analyze the distribution of all possible execution paths weighted by their probabilities. The key metrics become the expected running time and the probability of error, rather than absolute worst-case guarantees. This allows designers to prove that an algorithm will be fast on average or will be correct with overwhelming probability, even if a pathological sequence of random choices could theoretically lead to poor performance.

Monte Carlo algorithms exemplify this theory. They run for a fixed number of steps and then output an answer. The correctness is probabilistic. A classic example is the Monte Carlo method for estimating Pi. By randomly sampling points in a square and counting the fraction that falls inside an inscribed quarter-circle, the ratio converges to Pi/4. The Law of Large Numbers guarantees convergence, and the Central Limit Theorem allows us to quantify the error: the error decreases proportionally to 1 / sqrt(n), where n is the number of samples. By increasing n, we can make the probability of a large error arbitrarily small.

Las Vegas algorithms operate on a different principle. They never return an incorrect answer. Instead, they use randomness to guide the search for a correct answer, and they repeat until they succeed. Randomized QuickSort is a Las Vegas algorithm: it randomly chooses a pivot to partition the array. If it unluckily picks a bad pivot repeatedly, it will be slow, but it will always eventually sort the array correctly. The analysis shows that the expected number of comparisons is O(n log n), and the probability that it degrades to its O(n^2) worst case is astronomically small for large n. This guarantee of correctness with excellent expected performance makes Las Vegas algorithms highly desirable.

The Role of Random Sampling and Amplification

Two core mechanisms power most probability algorithms: random sampling and probability amplification. Random sampling involves selecting a subset of data or a path through a computation according to a probability distribution. This is used in reservoir sampling for streaming data, in Markov Chain Monte Carlo (MCMC) for sampling from complex distributions, and in randomized load balancing.

Probability amplification is a technique to boost the confidence of a Monte Carlo algorithm. If an algorithm has a success probability of p > 0.5, we can run it k times independently and take the majority vote. Using Chernoff bounds, we can prove that the error probability decreases exponentially with k. This means we can transform an algorithm that is correct 75% of the time into one that is correct 99.9999% of the time with only a few dozen repetitions, making probabilistic correctness practically as good as deterministic correctness.

3. Implementation — Python Code Examples with Detailed Comments

Implementing probability algorithms in Python is straightforward thanks to the powerful random module and libraries like numpy. The key is to ensure proper seeding, understand the distribution being sampled, and correctly handle the probabilistic logic. Below are four canonical implementations that demonstrate different classes of probability algorithms, from simple Monte Carlo estimation to more advanced Markov Chain sampling.

The following examples emphasize clarity and correctness. In production, you would use numpy.random or secrets for performance or cryptographic security, and you would add careful handling of edge cases and reproducibility via explicit random seeds.

Example 1: Monte Carlo Estimation of Pi

This is the quintessential Monte Carlo probability algorithm. It demonstrates how randomness can be used to approximate a deterministic mathematical constant. The accuracy improves with more samples, illustrating the trade-off between computation and precision.

import random
import math

def estimate_pi_monte_carlo(num_samples, seed=None):
    """
    Estimate Pi using the Monte Carlo method.
    Concept: Randomly sample points in a 1x1 square. The probability
    a point falls inside the quarter circle (x^2 + y^2 < 1) is Pi/4.
    """
    if seed is not None:
        random.seed(seed)
    
    inside_circle = 0
    
    for _ in range(num_samples):
        # Generate a random point (x, y) in [0, 1) x [0, 1)
        x = random.random()
        y = random.random()
        
        # Check if point is inside the quarter circle
        # Escape < is critical for HTML safety
        if x*x + y*y < 1.0:
            inside_circle += 1
            
    # Pi is approximately 4 * (points inside / total points)
    pi_estimate = 4 * inside_circle / num_samples
    # Standard error is approx 1 / sqrt(num_samples)
    return pi_estimate

# Example usage: Increase samples for higher accuracy
for n in [1000, 10000, 100000]:
    print(f"Samples: {n}, Pi Estimate: {estimate_pi_monte_carlo(n):.5f}")

Example 2: Las Vegas Randomized QuickSort

This Las Vegas algorithm always sorts correctly but uses a random pivot to achieve excellent expected performance. It avoids the O(n^2) worst-case of deterministic QuickSort that occurs on already-sorted data.

import random

def randomized_quicksort(arr):
    """
    Las Vegas Randomized QuickSort.
    Guarantees correct sorting. Expected time O(n log n).
    Random pivot selection avoids worst-case behavior on sorted inputs.
    """
    # Base case: arrays of size 0 or 1 are already sorted
    if len(arr) <= 1:
        return arr
    
    # Randomly choose a pivot element to avoid adversarial inputs
    pivot = random.choice(arr)
    
    # Partition into three groups using list comprehensions
    # Note: < and > are escaped as < and >
    less = [x for x in arr if x < pivot]
    equal = [x for x in arr if x == pivot]
    greater = [x for x in arr if x > pivot]
    
    # Recursively sort partitions and concatenate
    return randomized_quicksort(less) + equal + randomized_quicksort(greater)

# Example: Worst case for deterministic quicksort (sorted array) is average case here
data = list(range(1000, 0, -1))
sorted_data = randomized_quicksort(data)
print(f"Sorted correctly: {sorted_data[:5]} ... {sorted_data[-5:]}")

Example 3: Reservoir Sampling for Streaming Data

Reservoir sampling is a crucial probability algorithm for big data. It maintains a uniform random sample of k items from a stream of unknown length n, using only O(k) memory. Each item in the stream has an equal k/n probability of being in the final sample.

import random

def reservoir_sampling(stream_iterator, k, seed=None):
    """
    Algorithm R: Reservoir Sampling.
    Maintains a uniform random sample of k elements from a stream
    of unknown size in a single pass.
    """
    if seed is not None:
        random.seed(seed)
        
    reservoir = []
    
    for i, item in enumerate(stream_iterator):
        # First k items fill the reservoir
        if i < k:
            reservoir.append(item)
        else:
            # Generate random index j in [0, i]
            j = random.randint(0, i)
            # Replace with decreasing probability k / (i+1)
            # This ensures uniform probability for all items
            if j < k:
                reservoir[j] = item
                
    return reservoir

# Simulate a large stream (e.g., 1 million log lines) without storing it
def large_stream(n):
    for i in range(n):
        yield f"log_entry_{i}"

sample = reservoir_sampling(large_stream(100000), k=5)
print(f"Random sample from stream: {sample}")

Example 4: Metropolis-Hastings Markov Chain Monte Carlo (MCMC)

MCMC is a powerful class of probability algorithms for sampling from complex probability distributions where direct sampling is impossible. It is the workhorse of Bayesian inference. This implementation samples from a target distribution using a random walk proposal.

import random
import math

def metropolis_hastings(target_log_prob, proposal_std, iterations, init=0.0, seed=None):
    """
    Metropolis-Hastings MCMC sampler.
    Samples from a target distribution proportional to exp(target_log_prob).
    Uses a Gaussian random walk proposal.
    """
    if seed is not None:
        random.seed(seed)
        
    samples = []
    current = init
    current_log_prob = target_log_prob(current)
    
    for _ in range(iterations):
        # Propose a new state by adding Gaussian noise
        proposal = random.gauss(current, proposal_std)
        proposal_log_prob = target_log_prob(proposal)
        
        # Compute acceptance probability alpha = min(1, p(proposal)/p(current))
        # Use log probabilities for numerical stability
        log_alpha = proposal_log_prob - current_log_prob
        
        # Accept or reject the proposal
        # Condition: log_alpha >= 0 or random < exp(log_alpha)
        if log_alpha >= 0 or random.random() < math.exp(log_alpha):
            current = proposal
            current_log_prob = proposal_log_prob
        # else, stay at current state (implicit)
        
        samples.append(current)
        
    return samples

# Target: Standard Normal distribution log prob ~ -0.5 * x^2
def standard_normal_log_prob(x):
    return -0.5 * x * x

samples = metropolis_hastings(standard_normal_log_prob, proposal_std=1.0, iterations=10000)
print(f"MCMC Mean: {sum(samples)/len(samples):.3f}, Expected: 0.0")

4. Step-by-Step Example — Walk Through a Concrete Example

To solidify the concepts, let us walk through a complete execution of the Monte Carlo Pi estimation algorithm with a small, traceable number of samples. This will illustrate how randomness converges to a deterministic answer and how error is quantified. We will use num_samples = 8 for manual traceability.

Assume we have a random number generator that produces the following sequence of (x, y) pairs: (0.1, 0.2), (0.9, 0.9), (0.4, 0.3), (0.6, 0.8), (0.2, 0.9), (0.7, 0.1), (0.5, 0.5), (0.85, 0.2). Our task is to count how many points satisfy x^2 + y^2 < 1.

We evaluate each point step by step. This manual walkthrough reveals the core loop of any Monte Carlo probability algorithm: generate, test, and aggregate. Even with a tiny sample, we can see the statistical principle in action.

1. Point 1 (0.1, 0.2): 0.1^2 + 0.2^2 = 0.01 + 0.04 = 0.05 < 1.0 → Inside. Count = 1.
2. Point 2 (0.9, 0.9): 0.81 + 0.81 = 1.62 > 1.0 → Outside. Count = 1.
3. Point 3 (0.4, 0.3): 0.16 + 0.09 = 0.25 < 1.0 → Inside. Count = 2.
4. Point 4 (0.6, 0.8): 0.36 + 0.64 = 1.00 → Not < 1.0 → Outside (on boundary). Count = 2.
5. Point 5 (0.2, 0.9): 0.04 + 0.81 = 0.85 < 1.0 → Inside. Count = 3.
6. Point 6 (0.7, 0.1): 0.49 + 0.01 = 0.50 < 1.0 → Inside. Count = 4.
7. Point 7 (0.5, 0.5): 0.25 + 0.25 = 0.50 < 1.0 → Inside. Count = 5.
8. Point 8 (0.85, 0.2): 0.7225 + 0.04 = 0.7625 < 1.0 → Inside. Count = 6.

After 8 samples, we have 6 points inside. Our estimate is Pi ≈ 4 * (6 / 8) = 4 * 0.75 = 3.0. The true value of Pi is approximately 3.14159, so our error with only 8 samples is about 0.14159, or 4.5%. This is expected with such a small n.

If we were to increase to 10,000 samples, the Law of Large Numbers dictates that the fraction inside would converge to Pi/4 ≈ 0.7854, and the standard deviation of our estimate would be sqrt(p*(1-p)/n) * 4, where p = Pi/4. For n=10000, the standard error is approximately 0.016, meaning we would typically be within ±0.03 of the true Pi. This quantifiable error bound is what makes probability algorithms reliable despite their randomness: we can compute exactly how many samples are needed to achieve a desired confidence interval, such as 95% confidence that the error is less than 0.01.

5. Complexity Analysis — Time and Space Complexity

Analyzing probability algorithms requires a shift from worst-case analysis to expected and probabilistic analysis. We must consider three dimensions: expected time, worst-case time (which may have low probability), and the probability of error. Space complexity is typically deterministic but can also be probabilistic in some streaming algorithms.

For Monte Carlo algorithms, time complexity is deterministic and easy to analyze, but correctness is probabilistic. For Las Vegas algorithms, correctness is deterministic, but time complexity is a random variable. We therefore report Expected Time, which is the average over all possible random choices, and we also bound the probability that the runtime significantly exceeds the expectation using tail bounds like Markov's or Chernoff's inequalities.

The table below summarizes the complexity for the most common probability algorithms. Note that for many of these, the average and expected case is what matters in practice, as the worst case is often vanishingly improbable or requires an adversarial random seed.

A critical insight is that for Randomized QuickSort, the worst-case O(n²) is not just unlikely; its probability decreases exponentially with n. For n=1000, the chance of hitting worst-case behavior is far lower than the chance of a hardware failure. Therefore, expected O(n log n) is the practical complexity. Similarly, for Miller-Rabin primality testing, the error probability after k rounds is at most (1/4)^k. With k=20, the error is less than one in a trillion, making it effectively deterministic for all practical purposes while being vastly faster than deterministic primality proofs.

When analyzing MCMC, we also consider mixing time and convergence rate, which are not traditional Big O metrics but are crucial. The algorithm runs in O(n) time for n iterations, but the quality of the samples depends on how quickly the Markov chain converges to its stationary distribution. This is measured by concepts like effective sample size (ESS) and autocorrelation, which determine how many iterations are needed for a reliable estimate.

6. Variations and Optimizations — Common Improvements and Variants

Basic probability algorithms can be significantly improved through variance reduction, derandomization, and hybridization techniques. These optimizations aim to reduce the number of samples needed for a given accuracy, improve the expected runtime, or provide stronger probabilistic guarantees without increasing computational cost.

One of the most important optimizations is Variance Reduction for Monte Carlo methods. Naive Monte Carlo can require many samples to achieve high accuracy. Techniques like importance sampling, stratified sampling, and antithetic variates reduce variance. Importance sampling, for example, samples more frequently from regions that contribute most to the result, rather than uniformly. In Pi estimation, this is not very useful, but in financial risk modeling where rare events dominate the outcome, it can reduce the required samples by orders of magnitude. Control variates use a known related quantity to reduce the variance of the estimator.

For Las Vegas algorithms, the primary optimization is Randomized Pivot Selection Strategies. While picking a single random pivot is good, picking the median of three random elements as the pivot further reduces the probability of a bad partition. This "median-of-three" variant lowers the constant factor in the O(n log n) expected time and makes the algorithm even more robust. Another optimization is to switch to Insertion Sort for small subarrays (typically < 10 elements), which is a standard hybrid optimization that benefits both deterministic and randomized versions.

Advanced Variants

Beyond basic improvements, several advanced variants extend the power of probability algorithms. Amplification and Boosting is a generic technique to turn a weak probabilistic algorithm into a strong one. By running a Monte Carlo algorithm with success probability 2/3 multiple times and taking the majority vote, the Chernoff bound guarantees the error drops exponentially. This is how we achieve cryptographic levels of certainty from algorithms with modest per-run guarantees.

Derandomization is the theoretical pursuit of converting a randomized algorithm into a deterministic one with similar performance, often using pseudo-random generators with small seeds or conditional expectation methods. While not always practical, it shows that randomness is not always essential and helps understand the true complexity of a problem.

Other notable variants include Approximate Counting algorithms like Morris Counter and HyperLogLog, which use randomization to count up to billions using only a few bytes of memory with small relative error, and Sketching algorithms like Count-Min Sketch, which provide probabilistic frequency estimates for streaming data. These are essential for network monitoring and database query optimization where exact counting is impossible due to memory constraints.

7. Real-World Applications — Where This Is Used in Practice

Probability algorithms are not academic curiosities; they are embedded in the core infrastructure of modern technology. From web servers to databases to network routing, their ability to handle scale and uncertainty makes them indispensable. Their applications span any domain where data is large, time is limited, or exact answers are unnecessary.

In Web Servers and Load Balancing, randomized algorithms like "power of two choices" are used to distribute incoming requests across servers. Instead of tracking exact loads, a request is assigned to the less-loaded of two randomly chosen servers. This simple probabilistic rule achieves near-perfect load balancing with minimal coordination overhead, which is critical for handling millions of requests per second at companies like Google and Amazon.

In Databases and Big Data Systems, probability algorithms enable approximate query processing. When a user runs a COUNT DISTINCT or percentile query on a petabyte-scale table, systems like BigQuery, Redis, and Apache Spark use HyperLogLog and reservoir sampling to return an answer in seconds with 1-2% error, rather than scanning the entire dataset for minutes. Bloom filters, a probabilistic data structure, are used in databases like Cassandra and Chrome's Safe Browsing to test set membership with minimal memory, avoiding expensive disk lookups.

In Networks and Cryptography, randomized algorithms are foundational. The Miller-Rabin primality test, a Monte Carlo algorithm, is used to generate the large prime numbers required for RSA encryption, securing HTTPS traffic. Randomized routing algorithms help avoid congestion and are resilient to failures. In blockchain, proof-of-work is essentially a probabilistic search for a hash with certain properties.

The unifying theme across these applications is the management of trade-offs. In each case, a deterministic exact solution exists but is too slow, too memory-intensive, or too complex to deploy. Probability algorithms provide a tunable knob between resources and accuracy, allowing engineers to meet strict service-level agreements while keeping costs under control. This makes them a cornerstone of scalable system design.

8. Comparison with Alternatives — Use a Table to Compare with Related Approaches

When faced with a computational problem, engineers can choose from several algorithmic paradigms. Probability algorithms are one option among deterministic exact algorithms, approximation algorithms, and heuristics. Understanding the trade-offs between these approaches is essential for making informed architectural decisions. The choice depends on requirements for correctness, runtime guarantees, implementation complexity, and tolerance for error.

Deterministic exact algorithms guarantee a correct answer and have predictable runtime, but they may be exponentially slow for NP-hard problems or require excessive memory for large data. Approximation algorithms guarantee a solution within a provable factor of the optimal (e.g., within 2x of optimal) with deterministic runtime, but they are often complex to design. Heuristics are fast and simple but provide no formal guarantees on either correctness or runtime; they work well empirically but can fail unpredictably.

Probability algorithms occupy a unique middle ground. They provide formal, mathematically provable guarantees, but those guarantees are probabilistic rather than absolute. This allows them to be both fast and reliable in a quantifiable way. For example, a Monte Carlo algorithm might guarantee that its answer is within 1% of the correct answer with 99.9% probability, which is a stronger and more useful guarantee than a heuristic that "usually works well."

In practice, the boundaries between these categories are fluid. Many modern systems are hybrids. For example, a database might use a deterministic algorithm for small tables but automatically switch to a probabilistic HyperLogLog sketch when the data exceeds a threshold. Similarly, machine learning pipelines often combine randomized optimization (SGD) with deterministic validation. The key takeaway is that probability algorithms should be considered whenever the problem involves large scale, high dimensionality, or a need for robustness against worst-case inputs, and where a small, quantifiable probability of error is an acceptable price for massive performance gains.

9. Common Mistakes and Best Practices — Pitfalls to Avoid

Despite their power, probability algorithms are easy to misuse. Because their behavior is non-deterministic, bugs can be subtle and difficult to reproduce, and incorrect analysis of error probabilities can lead to systems that fail more often than expected. Adhering to best practices is crucial for building reliable systems on top of probabilistic foundations.

One of the most common mistakes is Using a Weak or Biased Random Number Generator. Many developers use the default random module without considering its quality. For simulations, Python's Mersenne Twister is sufficient, but for cryptography or large-scale sampling, it is predictable and not cryptographically secure. Using random for security-sensitive tasks like generating tokens or primes can lead to catastrophic vulnerabilities. Best practice is to use secrets or numpy.random.Generator with a well-chosen entropy source for security, and to explicitly seed generators for reproducibility in experiments.

Another frequent pitfall is Misinterpreting the Probability Guarantee. A common error is to assume that an algorithm with a 1% error rate will be wrong exactly 1% of the time in any specific run, or to ignore the need for amplification. For instance, running a Miller-Rabin test with only one round gives a 25% error rate, which is unacceptably high. Developers must calculate the required number of repetitions to achieve their target confidence level using formulas like k = log(error_tolerance) / log(error_per_run). Failing to do so results in systems that are far less reliable than assumed.

A third major mistake is Ignoring the Need for Reproducibility and Testing. Because the output is random, naive unit tests that check for an exact value will be flaky. Best practice is to test probabilistic properties, not exact outputs. For example, test that the Pi estimator is within 0.1 of the true value for large n, or that reservoir sampling produces a uniform distribution over many trials using a chi-squared test. Always allow the random seed to be injected as a parameter so tests can be made deterministic. Log the seed used in production runs to allow reproduction of rare failures.

• Do not use < and > comparisons incorrectly on floating point boundaries: In Monte Carlo tests like x*x + y*y < 1, be explicit about whether the boundary counts. For continuous distributions this rarely matters, but for discrete simulations it can introduce bias.
• Do not underestimate variance: Always compute confidence intervals. Reporting a Monte Carlo estimate without its standard error is misleading. Use the Central Limit Theorem to report estimate ± 1.96 * std_err for 95% confidence.
• Do not forget to handle the Las Vegas infinite loop case: While the expected time is finite, a Las Vegas algorithm could theoretically run for a very long time. In production, always implement a timeout or a fallback to a deterministic algorithm to guarantee liveness.
• Do profile before optimizing: The overhead of generating high-quality random numbers can be significant. Use vectorized operations with numpy for large-scale Monte Carlo simulations rather than Python loops where performance is critical.

Finally, document the probabilistic guarantees clearly. Any function that is probabilistic should have its error probability, expected runtime, and required assumptions (e.g., "assumes uniform random bits") stated explicitly in its docstring. This transparency prevents future maintainers from misusing the algorithm in contexts where its guarantees are insufficient.

10. Interview Questions — 3-5 Common Interview Questions About This Topic

Probability algorithms are a favorite topic in technical interviews for top software engineering and data science roles, because they test both mathematical reasoning and practical system design skills. Interviewers use them to assess whether a candidate understands trade-offs between accuracy, speed, and memory, and can reason about randomness formally. Below are common questions with guidance on what interviewers look for.

Question 1: Explain the difference between Monte Carlo and Las Vegas algorithms. Give an example of each.

What interviewers look for: A clear definition of the correctness vs. runtime trade-off. A strong answer will state that Monte Carlo algorithms have deterministic runtime but probabilistic correctness (e.g., Miller-Rabin primality test, Monte Carlo Pi estimation), while Las Vegas algorithms always return the correct answer but have probabilistic runtime (e.g., Randomized QuickSort, Randomized QuickSelect). The best answers will mention that a Las Vegas algorithm can be converted into a Monte Carlo algorithm by imposing a time limit and returning a failure or approximate answer if it exceeds the limit.

Question 2: How would you efficiently select k random elements from a stream of unknown size n in one pass with O(k) memory?

What interviewers look for: Recognition of the Reservoir Sampling problem. The expected answer is Algorithm R as implemented in Section 3. Candidates should explain the logic: keep the first k elements, then for the i-th element (i >= k), replace a random element in the reservoir with probability k/(i+1). They should be able to prove why this yields a uniform sample (each element has k/n probability) and discuss the time complexity O(n) and space O(k). A follow-up is often to handle weighted sampling or to discuss what happens if k is large.

Question 3: Design a Bloom Filter. What are its guarantees, and when would you use it instead of a hash set?

What interviewers look for: Understanding of a practical probabilistic data structure. A good answer will describe a bit array of size m and k hash functions, explain insertion (set k bits) and query (check if all k bits are set). Crucially, candidates must state the guarantees: no false negatives, but false positives with probability approximately (1 - e^(-kn/m))^k. They should discuss tuning m and k to achieve a target false positive rate and give real-world use cases: checking for existence in a cache or database before an expensive lookup, spell checkers, or network routers. The trade-off is massive memory savings (e.g., 10 bits per element for 1% false positive) versus exactness.

Question 4: You have a randomized algorithm that is correct with probability 0.6. How can you boost its success probability to 0.99?

What interviewers look for: Knowledge of probability amplification via repetition and majority voting. The answer should describe running the algorithm t times independently and taking the majority answer. Using Chernoff bounds or the binomial distribution, candidates should estimate t. For p=0.6, to get to 0.99, t is on the order of a few dozen (e.g., t=50 gives very high confidence). Strong candidates will mention that this only works if p > 0.5 and that the runs must be independent, and they will discuss the cost: t times the runtime. This tests the ability to quantify the trade-off between computation and reliability.

Question 5: Estimate the value of Pi using only a random number generator. How many samples do you need for 3 decimal places of accuracy with 95% confidence?

What interviewers look for: Ability to design a Monte Carlo estimator and analyze its error. Candidates should propose the quarter-circle method and then use the Central Limit Theorem to compute the required n. The variance of the Bernoulli trial is p(1-p) where p=Pi/4 ≈ 0.785. The standard error of the Pi estimate is 4 * sqrt(p(1-p)/n). For 95% confidence, need 1.96 * SE < 0.0005 (for 3 decimal places). Solving gives n ≈ (1.96*4 / 0.0005)^2 * p(1-p) ≈ 10 million samples. This question tests both coding and statistical reasoning, and whether the candidate can translate a theoretical error bound into a concrete resource requirement.