Question
Compression property fails when moving Phoenix 4.x from HBase 1.x to 2.x
Mira Moss
0 reputation · 17 Dec 2024, 15:12 UTC
14.8K views1
Loading…
0 reputation · 17 Dec 2024, 15:12 UTC
The goal is to maintain the same column‑family compression settings when a Phoenix 4.x deployment is shifted from a local HBase 1.x cluster to a production HBase 2.x cluster.
In the local environment, the CREATE TABLE … WITH COLUMN FAMILIES … COMPRESSION=<type> clause is accepted and the table functions normally. In the production environment, the same clause results in table creation errors or runtime failures, with Phoenix logs indicating unsupported or invalid compression properties. The Phoenix 4.x documentation explicitly limits compression support to HBase 1.x, and no migration path is provided for HBase 2.x.
Unresolved decisions remain regarding whether future Phoenix releases will reintroduce compression support for HBase 2.x or prescribe an alternative approach.
35275 reputation · 17 Dec 2024, 17:49 UTC
Phoenix 4.x is the release line built for HBase 1.x. HBase 2.x clusters are supported by the Phoenix 5.x line. When a Phoenix 4.x client and its server-side coprocessors run against HBase 2.x, DDL such as CREATE TABLE ... COMPRESSION='SNAPPY' can fail before compression semantics are even evaluated — the coprocessor or system-table layer breaks first. So the fix is not a different compression property; it is moving to the Phoenix release that matches HBase 2.x, then re-applying compression at the HBase column-family level.
Likely explanation: the failure appeared only after the cluster move, which points to a Phoenix/HBase version mismatch rather than a syntax problem. Phoenix does not implement its own compression — it translates COMPRESSION into the HBase column-family attribute. If the Phoenix layer can't load correctly on HBase 2.x, the DDL never reaches that translation cleanly.
Confirmed mechanics (well-established behavior):
phoenixCOMPRESSION affects new writes. Existing store files keep their old encoding until a major compaction rewrites them.# HBase shell
disable 'MY_SCHEMA.MY_TABLE'
alter 'MY_SCHEMA.MY_TABLE', {NAME => '0', COMPRESSION => 'SNAPPY'}
enable 'MY_SCHEMA.MY_TABLE'
describe 'MY_SCHEMA.MY_TABLE' # confirm the CF shows the codecmajor_compact 'MY_SCHEMA.MY_TABLE'Then check RegionServer logs and store-file listings to confirm new HFiles actually use the codec. Without this step, describe showing the new value proves nothing about on-disk data.
Whether a given Phoenix 5.x release re-enables or changes specific compression handling for HBase 2.x is version- and vendor-specific — verify against the release notes of the exact build you deploy rather than assuming parity with 4.x behavior. Setting compression via the HBase shell after Phoenix table creation is a common workaround and is generally safe because Phoenix reads whatever the column family provides, but confirm read/write behavior in a test cluster before relying on it in production.
One missing detail would sharpen this: the exact exception and stack trace, and whether the failure occurs at CREATE/ALTER time or later during compaction or reads. A DDL-time coprocessor error confirms the version-mismatch path; a compaction-time codec error points to missing native libraries instead.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 17 Dec 2024, 17:18 UTC
When Phoenix 4.x runs against HBase 2.x, the DDL clause COMPRESSION=... may be rejected before the property reaches HBase. However, once the table exists (even if created without compression), you can still set or change the column‑family compression using the HBase shell:
alter 'MY_TABLE', {NAME => 'cf1', COMPRESSION => 'SNAPPY'}This bypasses the Phoenix coprocessor limitation because the change is made directly in HBase metadata. After altering, new writes will use the selected codec, while existing store files retain their original encoding until a major compaction rewrites them. Verify the change with:
hbase> describe 'MY_TABLE'
and check the output for COMPRESSION => 'SNAPPY' (or LZ4/ZSTD as appropriate). This approach lets you keep Phoenix 4.x operational on HBase 2.x while still achieving the desired compression settings.