Question
Secondary index update lag during concurrent schema change in Phoenix
Neel Finch
0 reputation · 26 Jun 2022, 21:34 UTC
78.6K views1
Loading…
0 reputation · 26 Jun 2022, 21:34 UTC
When a secondary index is defined on a Phoenix table, writes are normally reflected in the index after a short background update.
Under a workload of concurrent inserts while an ALTER TABLE ADD COLUMN statement is executing, the index update lag can increase, and the index may temporarily miss some newly added column values.
The goal is to understand the expected lag duration and consistency guarantees given the configuration of phoenix.index.update.thread.count, the HBase region server flush interval, and the phoenix.index.update.sync flag in Phoenix 5.x with HBase 2.x.
What is the typical index update latency observed during the schema change? How does varying the thread pool size affect the lag and the risk of stale index reads? Does disabling synchronous updates guarantee bounded staleness, or can it lead to unbounded inconsistency?
34275 reputation · 27 Jun 2022, 00:05 UTC
In Apache Phoenix 5.x on HBase 2.x secondary indexes are maintained asynchronously by default. With phoenix.index.update.sync false, index mutations are queued and applied by background threads controlled by phoenix.index.update.thread.count. There is no guaranteed maximum lag. Typical observed latency under steady load is seconds to tens of seconds, and it can grow unbounded if the backlog drains slower than it accumulates. phoenix.index.update.sync true forces index writes in the same client transaction as data writes, reducing staleness at the cost of higher write latency.
During concurrent inserts while ALTER TABLE executes, catalog refresh and increased write volume can increase the index update queue depth. This raises observed lag and widens the window where an index scan can miss recently inserted rows or show null for the newly added column until the index entry is applied. HBase region server flush interval can add write pressure that slows workers.
| Configuration | Effect on lag | Risk or trade off |
|---|---|---|
| phoenix.index.update.thread.count | Increasing can reduce average lag by processing the queue faster under steady load. | Higher HBase write pressure and possible contention with region server flushes. |
| phoenix.index.update.sync true | Eliminates index lag by forcing index writes into the same transaction as data write. | Significant increase in client write latency. |
| phoenix.index.update.sync false | Lowers write latency via asynchronous background updates. | Risk of unbounded staleness if update rate is lower than insert rate. |
Missing diagnostic detail: Are you seeing lag only on indexes that reference the newly added column, or on all secondary indexes for the table? The answer changes whether the issue is catalog refresh overhead versus general backlog.
Use comments to ask for clarification. Post a solution as an answer.
34,275 reputation · 27 Jun 2022, 02:21 UTC
To clarify the consistency trade-off: when phoenix.index.update.sync is set to false (the default), Phoenix provides eventual consistency. During an ALTER TABLE operation, the metadata refresh across RegionServers can exacerbate the lag of the background update queue, meaning staleness is not bounded by a fixed time limit but is instead dependent on the phoenix.index.update.thread.count and current HBase load.
To quantify the actual lag during your schema changes in Phoenix 5.x, you can use the following approach:
phoenixphoenix.index.update.queue size to identify if the bottleneck is thread exhaustion or HBase region server flush intervals.