ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Frameworks & web · #hadoop
A small Hadoop application needs to move to a new cluster without a service outage, and the team is weighing two documented migration paths. HDFS Router-based Federation preserves a single filesystem view while underlying NameNodes are replaced, avoiding a hard client cutover. DistCp can replicate the dataset to a target cluster while the source stays live,
Context A team wants a repeatable Hadoop development environment that balances fast feedback loops with confidence that timing-sensitive HDFS operations—NameNode failover, RPC retry storms, and DataNode block reports—behave like production. Constraint CI pipelines must complete in under ten minutes, but the test suite exercises HA failover and block-replicat
Migrating data between Hadoop clusters without downtime often relies on HDFS snapshots and incremental copies via distcopy. While this approach aims to maintain point-in-time consistency without locking source directories, the performance of the metadata comparison remains a concern for specific workloads. When the source directory contains millions of small
When a Kerberos ticket used by a long-running Hadoop job expires, the client throws an AuthenticationFailedException with the message “Failed to obtain/user credentials”, causing the operation to fail unless the ticket is renewed manually or the job is restarted. The goal is to allow such jobs to continue running without manual re‑login by enabling automatic
I am trying to start a docker container with the below command. docker run -it -p 50070:50070 -p 8088:8088 -p 8080:8080 suhothayan/hadoop-spark-pig-hive:2.9.2 bash It ended up with the following error. docker: error response from daemon: Ports are not available: listen tcp 0.0.0.0/50070: bind: An attempt was made to access a socket in a way forbidden by its
Reducing infrastructure costs for low-traffic Hadoop workloads often involves transitioning from traditional 3x replication to Erasure Coding (EC) to minimize physical storage footprints. While EC significantly reduces the storage overhead, it introduces a higher CPU burden during the recovery of lost blocks and during initial write operations. In environmen
When a file stored in HDFS is truncated, overwritten, or replaced, the DataNode block replicas that have been loaded into the off‑heap cache via hdfs cacheadmin remain unchanged unless the cached path is explicitly removed or uncached. Consequently, read requests that hit the cached block may return stale data, and the cache does not self‑heal even though th
docker