Drawbacks of Colocated Indexing and Search

Although the proposed design seems reasonable, there are a couple of serious drawbacks that we discuss below.

1. Colocation limiting scalability: In a practical system, searching and indexing scale differently with time. That is, depending on a partition, the indexing performed on one group of replicas may be resource-intensive. On the other hand, it is possible that some search queries are frequently performed which will result in a greater load on a group of replicas. Colocating both these operations on the same machine can lead to an imbalance in a cluster and thus result in scalability issues.
2. Index recomputation: We assumed that each replica will recompute the index which leads to inefficient usage of resources. Furthermore, index computation is a resource-intensive task with possibly hundreds of stages of pipelined operations. Thus, recomputing the same index over different replicas requires powerful machines. Instead, it is logical to compute the index once and replicate it across availability zones.

We've created a system that colocates indexing and searching on the same node. Although it seems like efficient usage of resources, it has its downsides. Firstly, searching and indexing are both resource-intensive operations and one will impact the performance of the other. This colocated system doesn't scale efficiently with varying indexing and search scaling over time.