MultiThreadedExecutor vs. tracing-first: which lever reduces callback-chain latency when MutuallyExclusiveCallbackGroup serializes work?

Question

ContextA ROS 2 node runs several callbacks registered in a MutuallyExclusiveCallbackGroup under a SingleThreadedExecutor. End-to-end latency exceeds the target. The team is debating whether to switch to a MultiThreadedExecutor or first instrument the pipeline with ros2_tracing (tracetools/LTTng) to locate the bottleneck.ConstraintsMutuallyExclusiveCallbackGroup prevents concurrent execution of its callbacks even under a MultiThreadedExecutor, so adding threads may not help a serialized chain.The real latency source could be executor scheduling, DDS transport queuing, serialization overhead, or QoS history depth — each requires a different fix.ros2_tracing exposes callback start/end, executor wait, and RMW publish/subscribe tracepoints, but tracepoint names and availability vary by distribution (Humble, Jazzy, Rolling).LTTng-based tracing requires Linux, adds overhead, and can perturb sub-millisecond measurements.Open questionsDo the ros2_tracing callback-start/end and executor-wait tracepoints reliably distinguish scheduler latency from middleware queuing on Jazzy?If the callback group already serializes the chain, what measurable signal in a trace would justify adding threads versus tuning QoS history depth?How much tracing overhead is acceptable before it masks the sub-millisecond differences we are trying to resolve?

Tasadduq Burney · Answer

Direct ResolutionSwitching to a MultiThreadedExecutor is unlikely to reduce callback-chain latency when callbacks are managed by a MutuallyExclusiveCallbackGroup because the group enforces serialization regardless of the underlying executor type.Likely Explanation vs. Confirmed FactsThe bottleneck is most likely related to internal callback logic, DDS queuing, or serialization overhead, not thread availability. The confirmed fact is that the MutuallyExclusiveCallbackGroup acts as a serialization barrier.Necessary StepsInstrument the pipeline using ros2_tracing to capture callback start/end and executor wait events.Analyze the trace to determine if time is spent waiting for the executor or executing the callback logic.Diagnostic RequirementTo refine the recommendation, what is the measured percentage of total latency spent in the 'executor wait' tracepoint versus the 'callback execution' tracepoint?

MultiThreadedExecutor vs. tracing-first: which lever reduces callback-chain latency when MutuallyExclusiveCallbackGroup serializes work?

Context

Constraints

Open questions

1 answer

Direct Resolution

Likely Explanation vs. Confirmed Facts

Necessary Steps

Diagnostic Requirement

Your perspective belongs here.

0 question comments