You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 15:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Mon Nov 17, 2025 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2459087995034,12cyclictest121ktimers/023:55:150
245908799471,12cyclictest24911012/usr/sbin/munin20:40:170
2459092994638,5cyclictest2487327-21cron20:30:011
2459087994637,5cyclictest2463037-21idleruntime-cro19:20:000
2459101994538,4cyclictest2573472-21munin-plugin-st00:40:003
2459087994533,9cyclictest121ktimers/000:30:180
2459101994332,7cyclictest2509677-21grep21:30:273
2459096994335,4cyclictest2466682-21latency_hist19:30:002
2459092994334,6cyclictest299-21systemd-logind22:52:321
245908799431,9cyclictest2475621-21/usr/sbin/munin19:55:130
2459087994318,20cyclictest2545970-21ls23:20:000
245908799431,13cyclictest2473683-21cron19:50:010
2459101994232,6cyclictest2487345-21basename20:30:013
2459092994233,6cyclictest2490739-21sh20:40:011
2459092994233,5cyclictest2552997-21grep23:40:001
2459092994222,7cyclictest2509832-21kworker/u8:0+events_unbound21:40:171
2459087994232,6cyclictest2508044-21grep21:25:280
245908799421,9cyclictest2532835-21/usr/sbin/munin22:40:170
245908799421,8cyclictest2469496-21/usr/sbin/munin19:35:250
245908799421,11cyclictest2540439-21unixbench_singl23:00:280
245908799421,11cyclictest2537666-21/usr/sbin/munin22:55:140
245908799413,6cyclictest2528920-21munin-plugin-st22:30:000
245908799412,34cyclictest2475396-21awk19:55:010
245908799411,9cyclictest2524933-21/usr/sbin/munin22:15:270
245908799411,9cyclictest2518283-21/usr/sbin/munin21:55:300
245908799411,9cyclictest2460805-21cat19:15:010
245908799411,8cyclictest2543076-21/usr/sbin/munin23:10:200
245908799411,8cyclictest2509403-21/usr/sbin/munin21:30:250
245908799411,8cyclictest2498709-21/usr/sbin/munin21:00:250
245908799411,6cyclictest2554283-21grep23:40:260
245908799411,11cyclictest2486799-21sed20:25:280
245908799410,10cyclictest2565165-21apt-get00:15:150
2459092994028,8cyclictest2496070-21auditctl20:55:011
2459087994028,8cyclictest2548929-21grep23:25:270
2459087994028,8cyclictest2548929-21grep23:25:270
245908799402,7cyclictest2572733-21runrttasks00:35:290
245908799402,6cyclictest2569901-21sshd00:28:160
245908799402,6cyclictest2569901-21sshd00:28:160
245908799402,15cyclictest2483908-21cron20:20:010
245908799401,9cyclictest2505057-21apt-get21:20:070
245908799401,7cyclictest2499726-21/usr/sbin/munin21:05:130
245908799401,6cyclictest2526020-21/usr/sbin/munin22:20:170
245908799401,6cyclictest2526020-21/usr/sbin/munin22:20:170
245908799401,6cyclictest2488691-21cron20:31:000
245908799400,4cyclictest2496307-21taskset20:55:120
2459092993931,4cyclictest2542564-21idleruntime-cro23:10:001
2459092993929,6cyclictest2567951-21grep00:20:291
245908799397,27cyclictest2520627-21apt-get22:05:170
245908799393,6cyclictest2561930-21cut00:05:180
245908799391,9cyclictest2481821-21users20:10:280
245908799391,9cyclictest2477435-21/usr/sbin/munin20:00:180
245908799391,8cyclictest2555702-21/usr/sbin/munin23:45:280
245908799391,8cyclictest2467216-21diskstats19:30:180
245908799391,8cyclictest2467216-21diskstats19:30:180
245908799391,7cyclictest2541175-21/usr/sbin/munin23:05:180
245908799391,6cyclictest2512944-21/usr/sbin/munin21:40:270
245908799391,4cyclictest2510503-21/usr/sbin/munin21:35:190
245908799391,4cyclictest2501433-21/usr/sbin/munin21:10:140
245908799391,3cyclictest2534785-21/usr/sbin/munin22:45:200
2459101993832,4cyclictest2462691-21kworker/u8:1+events_unbound19:31:513
2459101993832,4cyclictest2462691-21kworker/u8:1+events_unbound19:31:513
2459101993831,4cyclictest2559839-21sendmail00:00:003
2459101993830,5cyclictest2545963-21cron23:20:003
2459101993826,9cyclictest2504615-21idleruntime-cro21:20:013
2459096993830,4cyclictest0-21swapper/223:45:202
2459092993830,6cyclictest2511548-21kworker/u8:1+events_unbound22:12:251
245908799382,4cyclictest2494158-21apt-get20:50:030
245908799382,3cyclictest2564498-21/usr/sbin/munin00:10:290
245908799381,9cyclictest2484279-21df_inode20:20:160
245908799381,8cyclictest2465297-21/usr/sbin/munin19:25:150
245908799381,7cyclictest2536177-21fschecks_time22:50:180
245908799381,7cyclictest2503255-21diskstats21:15:180
245908799381,6cyclictest2532432-21latency_hist22:40:010
245908799381,6cyclictest2464347-21sed19:20:240
245908799381,5cyclictest2522660-21/usr/sbin/munin22:10:190
245908799381,4cyclictest2566890-21munin-node00:20:160
245908799381,4cyclictest2546317-21/usr/sbin/munin23:20:150
245908799381,4cyclictest2546317-21/usr/sbin/munin23:20:150
245908799381,4cyclictest2490338-21/usr/sbin/munin20:35:280
245908799381,11cyclictest2492816-21/usr/sbin/munin20:45:160
2459096993728,5cyclictest0-21swapper/223:20:002
2459096993725,8cyclictest0-21swapper/219:20:012
2459092993730,5cyclictest2497657-21kworker/u8:1+events_unbound21:02:131
2459092993718,6cyclictest2484156-21kworker/u8:2+events_unbound20:30:221
245908799373,8cyclictest2560355-21/usr/sbin/munin00:00:180
245908799373,5cyclictest2520001-21users22:00:300
245908799371,9cyclictest2515276-21munin-run21:50:000
245908799371,5cyclictest2529485-21irqstats22:30:210
245908799371,5cyclictest2516577-21users21:50:270
245908799371,4cyclictest2557956-21sshd23:53:170
245910199367,4cyclictest131rcu_preempt23:50:133
2459101993630,4cyclictest2492152-21kworker/u8:0+events_unbound20:57:123
2459101993629,4cyclictest0-21swapper/322:30:003
2459101993626,6cyclictest0-21swapper/323:12:363
2459092993629,5cyclictest2520156-21kworker/u8:3+events_unbound22:09:321
2459092993624,8cyclictest2506120-21grep21:20:301
2459092993621,3cyclictest131rcu_preempt21:55:221
245908799361,5cyclictest2551733-21grep23:35:170
245908799361,4cyclictest2470630-21df19:40:190
245908799361,3cyclictest2550844-21/usr/sbin/munin23:30:280
2459101993528,4cyclictest2552531-21unixbench_multi23:35:263
2459101993525,7cyclictest2532332-21sendmail22:40:013
2459101993525,6cyclictest0-21swapper/322:25:003
2459101993525,6cyclictest0-21swapper/322:24:593
2459092993524,7cyclictest2569633-21grep00:25:261
2459092993524,7cyclictest2569633-21grep00:25:261
2459092993515,3cyclictest131rcu_preempt22:45:011
2459092993511,10cyclictest0-21swapper/122:25:111
245908799351,8cyclictest2479204-21/usr/sbin/munin20:05:200
2459101993423,7cyclictest0-21swapper/320:00:303
2459096993425,6cyclictest0-21swapper/219:45:002
2459096993425,5cyclictest2570460-21fschecks_count00:30:202
2459096993425,5cyclictest0-21swapper/221:20:012
2459096993424,7cyclictest0-21swapper/220:45:172
2459096993424,6cyclictest0-21swapper/221:45:002
2459096993423,7cyclictest0-21swapper/222:00:002
2459092993428,4cyclictest2538924-21kworker/u8:0+events_unbound23:02:351
2459101993326,4cyclictest2568357-21kthreadd00:25:003
2459101993324,6cyclictest2564921-21cron00:15:003
2459101993323,7cyclictest2500766-21grep21:05:283
2459101993323,6cyclictest0-21swapper/321:00:273
2459101993323,6cyclictest0-21swapper/320:45:263
2459101993323,6cyclictest0-21swapper/319:15:273
245909299339,9cyclictest24-21ksoftirqd/119:55:011
2459092993326,3cyclictest0-21swapper/121:35:301
2459092993325,4cyclictest0-21swapper/123:20:271
2459092993325,4cyclictest0-21swapper/123:20:271
2459092993325,4cyclictest0-21swapper/119:45:271
2459092993325,4cyclictest0-21swapper/119:20:011
2459092993324,4cyclictest0-21swapper/119:35:001
2459092993324,4cyclictest0-21swapper/119:35:001
2459092993323,7cyclictest2504613-21cut21:20:001
2459092993321,7cyclictest2570486-21uniq00:30:201
2459101993224,4cyclictest0-21swapper/300:20:003
245910199321,4cyclictest131rcu_preempt20:20:143
2459101993214,4cyclictest131rcu_preempt20:06:133
2459096993222,6cyclictest2477083-21cron20:00:012
2459096993222,6cyclictest0-21swapper/219:21:492
2459096993222,6cyclictest0-21swapper/200:00:002
2459092993226,4cyclictest2550910-21kworker/u8:0+events_unbound23:42:431
2459092993225,5cyclictest2487217-21kworker/u8:3+events_unbound20:43:421
2459092993224,4cyclictest0-21swapper/122:20:001
2459092993221,7cyclictest2460293-21grep19:10:301
2459101993123,4cyclictest0-21swapper/322:10:303
2459101993121,6cyclictest0-21swapper/322:50:263
2459101993121,6cyclictest0-21swapper/321:50:263
2459101993120,7cyclictest0-21swapper/300:25:263
2459101993120,7cyclictest0-21swapper/300:25:263
245910199311,4cyclictest131rcu_preempt21:10:163
2459096993125,4cyclictest2526992-21kworker/u8:2+events_unbound23:27:392
2459096993125,4cyclictest2526992-21kworker/u8:2+events_unbound23:27:382
2459096993123,4cyclictest0-21swapper/222:10:302
2459096993122,6cyclictest2502900-21auditctl21:15:012
2459096993122,5cyclictest0-21swapper/222:10:002
2459096993121,6cyclictest0-21swapper/200:20:302
2459096993120,7cyclictest0-21swapper/200:10:292
2459096993120,7cyclictest0-21swapper/200:05:002
2459092993125,4cyclictest2556376-21kworker/u8:2+events_unbound23:58:191
2459092993123,4cyclictest0-21swapper/120:20:011
2459092993122,6cyclictest2564927-21sh00:15:001
2459092993119,7cyclictest2562817-21grep00:05:271
2459092993118,5cyclictest131rcu_preempt21:35:011
2459092993117,4cyclictest131rcu_preempt20:05:161
2459092993113,7cyclictest24-21ksoftirqd/122:40:011
2459092993110,11cyclictest131rcu_preempt23:50:251
2459101993023,4cyclictest0-21swapper/321:25:183
2459101993022,4cyclictest0-21swapper/320:40:293
2459101993021,4cyclictest0-21swapper/319:55:283
2459101993013,5cyclictest2487217-21kworker/u8:3+events_unbound20:52:483
2459096993023,5cyclictest2538924-21kworker/u8:0+events_unbound23:04:452
2459096993023,4cyclictest0-21swapper/222:45:012
2459096993022,4cyclictest0-21swapper/222:20:002
2459096993017,9cyclictest311ktimers/219:45:142
245909699300,23cyclictest0-21swapper/219:35:152
245909299308,14cyclictest24-21ksoftirqd/119:35:281
245909299308,12cyclictest24-21ksoftirqd/122:25:011
245909299308,12cyclictest24-21ksoftirqd/122:25:011
245909299307,6cyclictest24-21ksoftirqd/122:55:281
2459092993022,4cyclictest0-21swapper/100:15:271
2459092993011,4cyclictest131rcu_preempt21:50:261
2459092993011,11cyclictest131rcu_preempt19:55:231
2459101992921,4cyclictest0-21swapper/323:45:303
2459101992920,5cyclictest0-21swapper/322:15:163
2459101992919,6cyclictest0-21swapper/322:05:293
2459101992918,7cyclictest2483466-21unixbench_multi20:15:273
2459101992917,5cyclictest2526992-21kworker/u8:2+events_unbound22:55:283
2459101992914,7cyclictest131rcu_preempt20:35:013
2459096992922,5cyclictest2526992-21kworker/u8:2+events_unbound22:57:352
2459092992923,4cyclictest2558209-21kworker/u8:1+events_unbound00:03:201
2459092992921,4cyclictest0-21swapper/122:45:261
2459101992821,5cyclictest2532156-21kworker/u8:3+events_unbound22:42:313
2459101992819,6cyclictest0-21swapper/319:50:013
2459101992818,6cyclictest0-21swapper/323:30:003
2459101992818,6cyclictest0-21swapper/323:30:003
245909699287,13cyclictest131rcu_preempt21:20:202
2459096992822,4cyclictest2526992-21kworker/u8:2+events_unbound22:32:292
2459096992820,6cyclictest2497657-21kworker/u8:1+events_unbound21:07:142
2459096992816,7cyclictest32-21ksoftirqd/221:30:252
245909699280,14cyclictest2511617-21sshd21:37:172
245909299288,13cyclictest24-21ksoftirqd/121:10:011
2459092992820,4cyclictest0-21swapper/119:25:261
2459101992720,4cyclictest2468337-21kworker/u8:3+events_unbound19:41:533
2459101992720,4cyclictest0-21swapper/321:42:193
2459101992719,5cyclictest2570485-21fschecks_time00:30:203
2459101992719,4cyclictest0-21swapper/319:50:273
2459101992711,10cyclictest40-21ksoftirqd/319:25:163
2459096992720,4cyclictest2497797-21sendmail21:00:012
2459096992720,3cyclictest0-21swapper/222:30:002
2459096992719,5cyclictest0-21swapper/223:50:262
2459096992719,5cyclictest0-21swapper/220:40:012
2459096992719,5cyclictest0-21swapper/200:25:262
2459096992719,5cyclictest0-21swapper/200:25:262
2459096992718,5cyclictest0-21swapper/220:15:262
2459096992717,8cyclictest2460866-21latency_hist19:15:012
2459096992717,7cyclictest0-21swapper/219:30:282
2459096992717,7cyclictest0-21swapper/219:30:282
245909299278,11cyclictest24-21ksoftirqd/120:50:011
245909299277,9cyclictest24-21ksoftirqd/120:55:191
2459092992720,5cyclictest2476906-21kworker/u8:2+events_unbound20:02:411
2459092992720,4cyclictest0-21swapper/119:20:251
2459092992719,6cyclictest2542722-21kworker/u8:3+events_unbound23:25:131
2459092992719,6cyclictest2542722-21kworker/u8:3+events_unbound23:25:131
2459092992716,8cyclictest231ktimers/120:20:131
2459092992712,4cyclictest131rcu_preempt21:25:021
2459101992618,6cyclictest2542722-21kworker/u8:3+events_unbound23:32:413
2459101992618,4cyclictest0-21swapper/323:05:303
2459101992618,4cyclictest0-21swapper/319:35:183
2459101992617,6cyclictest0-21swapper/300:05:263
245909699268,12cyclictest0-21swapper/200:05:132
245909699267,13cyclictest131rcu_preempt23:30:172
245909699267,12cyclictest301rcuc/221:00:282
2459096992619,5cyclictest2485924-21kworker/u8:1+events_unbound20:28:412
2459096992618,4cyclictest0-21swapper/222:00:292
2459096992616,6cyclictest0-21swapper/223:10:282
2459096992614,9cyclictest2479239-21cut20:05:202
2459096992614,6cyclictest0-21swapper/221:25:182
245909299268,11cyclictest24-21ksoftirqd/121:45:171
245909299267,11cyclictest24-21ksoftirqd/100:35:191
245909299266,7cyclictest24-21ksoftirqd/123:15:291
245909299266,4cyclictest131rcu_preempt22:05:001
245909299264,3cyclictest131rcu_preempt19:40:291
245909299262,7cyclictest24-21ksoftirqd/123:45:301
2459101992519,4cyclictest2480457-21kworker/u8:2+events_unbound20:14:273
245909699254,15cyclictest32-21ksoftirqd/222:50:152
2459096992519,3cyclictest0-21swapper/220:00:232
2459096992519,3cyclictest0-21swapper/219:50:182
2459096992518,5cyclictest2538924-21kworker/u8:0+events_unbound23:05:192
2459096992518,4cyclictest0-21swapper/220:40:272
2459096992518,3cyclictest0-21swapper/220:35:012
245909699251,6cyclictest2514862-21grep21:45:292
2459096992514,5cyclictest2542722-21kworker/u8:3+events_unbound23:20:552
2459096992514,5cyclictest2542722-21kworker/u8:3+events_unbound23:20:552
245909699251,14cyclictest2552995-21sendmail23:40:002
2459092992511,10cyclictest215-21dbus-daemon20:12:001
245910199247,9cyclictest40-21ksoftirqd/321:20:233
2459101992418,3cyclictest0-21swapper/322:35:003
2459101992417,3cyclictest0-21swapper/323:40:253
2459101992416,5cyclictest0-21swapper/322:45:263
2459096992418,3cyclictest0-21swapper/222:40:012
2459096992418,3cyclictest0-21swapper/200:40:002
2459096992416,5cyclictest311ktimers/220:20:152
245909299246,11cyclictest24-21ksoftirqd/121:15:011
245909299245,3cyclictest131rcu_preempt22:30:241
2459092992412,4cyclictest131rcu_preempt23:10:181
2459101992316,4cyclictest0-21swapper/319:15:023
2459101992311,6cyclictest40-21ksoftirqd/321:55:283
245909699232,4cyclictest131rcu_preempt21:50:132
2459096992317,4cyclictest2473632-21kworker/u8:0+events_unbound20:12:012
2459096992316,4cyclictest0-21swapper/222:45:262
2459096992313,7cyclictest32-21ksoftirqd/200:15:172
2459096992313,7cyclictest2495499-21cat20:50:272
245910199228,5cyclictest2505056-21kworker/u8:3+events_unbound21:35:593
2459101992215,4cyclictest2490756-21awk20:40:003
245909699227,6cyclictest2526992-21kworker/u8:2+events_unbound22:23:062
245909699227,6cyclictest2526992-21kworker/u8:2+events_unbound22:23:062
245909699224,11cyclictest0-21swapper/223:40:252
2459101992114,5cyclictest0-21swapper/319:20:163
2459101992110,8cyclictest2539387-21apt-get23:00:133
2459101992014,3cyclictest0-21swapper/322:00:163
2459101992013,5cyclictest2513319-21kworker/u8:2+events_unbound21:45:463
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional