Diagnosing Stale Data in LinkedIn's Venice: A Production Troubleshooting Guide

Recognizable Condition: Stale or Inconsistent Feed Data

Venice clients serving LinkedIn feeds—ranking models, PYMK, or timeline features—return outdated records despite successful push-job completion. Symptoms include feature drift in recommendation quality, missing recent user actions, or inconsistent A/B test results.

Root Cause Diagnostic Table

Symptom	Most Likely Cause
Version not ONLINE in controller	Push job completed but version not materialized
Partition lag in venice.push.status	Partial partition upload or server catch-up delay
Client returns old schema	Schema registry cache TTL expiration
Checksum mismatch across replicas	Replica divergence or incomplete repair

Ordered Diagnostic Checks

1. Verify version state: Query /stores/{store}/versions REST endpoint. Confirm latest version shows ONLINE and status: COMPLETE.
2. Inspect partition completion: Check venice.push.status Kafka topic for ERROR or IN_PROGRESS markers on affected partitions.
3. Run checksum validation: Execute venice-admin-tool checksum --store <store> --version <v>
4. Validate client schema resolution: Enable debug logging for schema registry client and capture resolved schema ID at request time. Compare against controller's current schema version.
5. Review push-job logs: Search for VenicePushJob completion markers and any InconsistentDataException stack traces in the job logs.

Fixes Tied to Findings

• Version not ONLINE: Re-trigger the push job with --force flag after validating source data integrity in the upstream Hadoop/Spark job.
• Partition lag detected: Restart Venice server on affected replica to trigger catch-up sync. Monitor venice.push.status for partition state transition to COMPLETE.
• Checksum mismatch: Run venice-admin-tool repair --store <store> --partition <p> to initiate partition-level re-sync. Schedule during low-traffic windows for large partitions.
• Stale client schema: Reduce schema cache TTL from default 24h to 1h, or implement cache invalidation via schema registry notifications.
• Push job failure: Inspect upstream job for data corruption, then re-run. Verify output data schema matches expected version.

Escalation Criteria

Contact Venice platform team immediately if:

• More than 5% of partitions show checksum mismatch after repair attempt.
• Push jobs repeatedly fail with InconsistentDataException, indicating source data corruption.
• Controller shows version stuck in STARTED state for over 30 minutes.
• Client errors persist after reducing schema cache TTL and validating schema registry.

Verification Steps

1. Run venice-admin-tool checksum --store <store> --version <v> before and after repair to confirm convergence.
2. Monitor venice.client.latency and venice.client.staleness metrics in Grafana for 15 minutes post-fix.
3. Execute a canary query using the affected client library and verify returned record matches expected latest version.
4. Confirm push-job success in Venice controller UI: version state transitions to ONLINE and partition count matches replica count.
5. Verify schema registry shows expected schema ID for the store and client logs reflect updated schema resolution.

Important Limitations

• Venice behavior varies significantly by store configuration—hybrid stores vs. read-only stores have different materialization paths. Always verify store type before applying fixes.
• Schema registry cache TTL defaults to 24 hours in many clients; aggressive reduction may increase registry load and latency.
• Checksum repair triggers full partition re-sync, which can temporarily increase storage I/O and network traffic.
• Controller REST endpoints require admin VPN access; ensure network connectivity before troubleshooting.
• The push-job --force flag overwrites version history—confirm no downstream consumers depend on the previous version before proceeding.