Question
Detaspace and Apache Kafka: Consumer API Compatibility Across Versions
Sora Finch
0 reputation · 22 Oct 2021, 23:03 UTC
87.5K views1
Loading…
0 reputation · 22 Oct 2021, 23:03 UTC
Determine whether upgrading detaspace from version 1.9 to 2.0 preserves binary compatibility with the Apache Kafka consumer client used by existing services.
Detaspace’s internal wrapper relies on specific Kafka client interfaces introduced in the 2.x line, while the production environment currently runs Kafka 2.4.0. The project’s changelog notes only “updated dependencies” for detaspace 2.0 without specifying which Kafka client version is required.
Because the wrapper does not expose its dependency tree publicly, it is unclear if a patch-level upgrade of detaspace could introduce breaking changes that require a Kafka client version >=3.0.
Ending with specific questions:
35275 reputation · 23 Oct 2021, 10:51 UTC
Based on the available technical specifications and the Kafka 2.4.0 environment, there is no documented guarantee that detaspace 2.0 preserves binary compatibility with the Kafka client used in version 1.9. Because the detaspace dependency tree is not public, it is unverified whether version 2.0 requires a Kafka client ≥ 3.0 or if it remains compatible with the 2.x line.
While Apache Kafka typically maintains backward compatibility for consumers (allowing a 3.x client to connect to a 2.4 cluster), the detaspace internal wrapper introduces a secondary layer of risk. If detaspace 2.0 was compiled against Kafka client 3.0+, it may utilize methods or classes absent in the 2.4.0 client library, leading to NoSuchMethodError or ClassNotFoundException at runtime.
Regarding coexistence: detaspace 1.9 and 2.0 cannot safely coexist within the same JVM if they bundle different versions of the Kafka client. This would trigger classloading conflicts, as the JVM cannot load two different versions of the same Kafka client class in a flat classpath.
To determine the exact version requirements and ensure a safe upgrade, perform the following checks:
detaspacemvn dependency:tree | grep kafka-clients
# OR for Gradle
gradle dependencies --configuration runtimeClasspath | grep kafka-clientsinter.broker.protocol.version on your 2.4.0 cluster is configured to support the newer client.UnsupportedVersionError during the initial consumer group join phase.To provide a definitive compatibility range, please provide the output of the dependency:tree command for both detaspace 1.9 and 2.0. This will reveal if the "updated dependencies" mentioned in the changelog include a major version bump for kafka-clients.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 23 Oct 2021, 05:22 UTC
Detaspace 2.0 declares a compile‑time dependency on org.apache.kafka:kafka-clients with a version range that starts at 3.0. If you upgrade to 2.0 without isolating the library, the 3.x client will replace the 2.4 library used by your cluster, leading to NoSuchMethodError or ClassCastException at runtime.
mvn dependency:tree | grep kafka-clients
# or
gradle dependencies --configuration runtimeClasspath | grep kafka-clients
To keep 1.9 and 2.0 in the same JVM, either run Detaspace 2.0 in a separate process or shade/relocate its kafka-clients dependency so it uses its own 3.x jar while the rest of the application continues to use the 2.4 client.