You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 13:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Wed Nov 19, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3245002994135,2cyclictest3323470-21kworker/u8:2+events_unbound22:20:171
3245002993937,2cyclictest3234575-21kworker/u8:0+events_unbound19:10:161
3245002993933,4cyclictest3421530-21latency_hist22:00:001
3245002993734,2cyclictest3509780-21kworker/u8:0+events_unbound00:15:181
3245002993733,3cyclictest3405823-21kworker/u8:1+events_unbound21:45:181
3245002993632,3cyclictest3509780-21kworker/u8:0+flush-259:000:10:181
3245010993532,2cyclictest3447340-21kworker/u8:1+events_unbound22:50:253
3245002993532,2cyclictest3281828-21kworker/u8:0+events_unbound19:50:251
3245010993428,4cyclictest3359695-21latency_hist21:00:013
3244998993428,4cyclictest3291876-21latency_hist19:55:010
3245010993330,2cyclictest3555787-21kworker/u8:1+events_unbound00:10:013
3245002993330,2cyclictest3333610-21kworker/u8:1+flush-259:020:50:181
3245002993330,2cyclictest3281828-21kworker/u8:0+flush-259:019:55:241
3245010993229,2cyclictest3323470-21kworker/u8:2+events_unbound20:37:383
3245002993226,3cyclictest3297275-21kworker/u8:4+events_unbound20:00:191
3245002993128,2cyclictest3452869-21kworker/u8:3+events_unbound23:40:201
3245010993027,2cyclictest3509780-21kworker/u8:0+events_unbound23:52:363
3245002993027,2cyclictest3260273-21kworker/u8:1+events_unbound19:25:181
3245002993023,5cyclictest3338737-21latency_hist20:40:001
3245010992926,2cyclictest3509780-21kworker/u8:0+events_unbound23:35:163
3245010992926,2cyclictest3405823-21kworker/u8:1+events_unbound22:01:383
3245002992926,2cyclictest3260273-21kworker/u8:1+flush-259:019:45:251
3245010992826,2cyclictest3452869-21kworker/u8:3+events_unbound00:23:383
3245010992826,2cyclictest3260273-21kworker/u8:1+events_unbound19:45:193
3245010992825,2cyclictest3333610-21kworker/u8:1+events_unbound21:20:283
3245010992825,2cyclictest3323470-21kworker/u8:2+events_unbound22:27:213
3245010992825,2cyclictest3323470-21kworker/u8:2+events_unbound20:45:243
3245006992822,5cyclictest3349437-21diskstats20:50:162
3245002992825,2cyclictest3452869-21kworker/u8:3+flush-259:000:30:211
3245002992825,2cyclictest3421770-21kworker/u8:0+events_unbound22:00:191
3245002992825,2cyclictest3396057-21kworker/u8:3+events_unbound21:40:161
3245002992825,2cyclictest3323470-21kworker/u8:2+flush-259:021:25:261
3245002992825,2cyclictest3323470-21kworker/u8:2+flush-259:020:30:261
3245002992824,3cyclictest3291905-21kworker/u8:3+flush-259:020:05:261
3245002992822,4cyclictest2698871-21systemd-journal21:01:371
3245002992822,3cyclictest3452869-21kworker/u8:3+events_unbound00:25:221
3244998992823,4cyclictest2698871-21systemd-journal21:25:010
3245010992725,2cyclictest3447340-21kworker/u8:1+events_unbound22:32:203
3245010992725,2cyclictest3447340-21kworker/u8:1+events_unbound22:32:203
3245010992725,2cyclictest3291905-21kworker/u8:3+events_unbound20:10:273
3245006992721,4cyclictest3452845-21latency_hist22:30:002
3245002992724,2cyclictest3452869-21kworker/u8:3+flush-259:022:40:191
3245002992724,2cyclictest3385572-21kworker/u8:0+flush-259:021:30:261
3245010992624,2cyclictest3244623-21kworker/u8:3+events_unbound19:11:463
3245010992623,2cyclictest3452869-21kworker/u8:3+events_unbound00:15:293
3245010992623,2cyclictest3405823-21kworker/u8:1+events_unbound21:50:223
3245010992623,2cyclictest3323470-21kworker/u8:2+events_unbound21:05:163
3245010992623,2cyclictest3234575-21kworker/u8:0+events_unbound19:15:193
3245006992620,4cyclictest3390686-21latency_hist21:30:012
3245002992623,2cyclictest3323470-21kworker/u8:2+events_unbound21:20:201
3245002992623,2cyclictest3244623-21kworker/u8:3+events_unbound19:35:201
3245002992621,4cyclictest2698871-21systemd-journal19:34:591
3244998992622,3cyclictest3406378-21apt-get21:45:140
3244998992619,5cyclictest2698871-21systemd-journal19:40:000
3245010992522,2cyclictest3260273-21kworker/u8:1+flush-259:019:35:003
3245006992522,2cyclictest3234575-21kworker/u8:0+flush-259:019:20:012
3245006992521,3cyclictest3432123-21apt-get22:10:142
3245002992522,2cyclictest3260273-21kworker/u8:1+flush-259:019:40:241
3244998992521,3cyclictest3385572-21kworker/u8:0+events_unbound21:30:000
3245010992421,2cyclictest3328315-21kworker/u8:4+events_unbound20:30:223
3245006992417,5cyclictest2698871-21systemd-journal22:45:002
3245002992422,2cyclictest3447340-21kworker/u8:1+flush-259:022:25:251
3245002992421,2cyclictest3447340-21kworker/u8:1+events_unbound22:55:251
3245002992421,2cyclictest3431880-21kworker/u8:3+events_unbound22:15:221
3245002992421,2cyclictest3323470-21kworker/u8:2+flush-259:022:10:181
3245002992419,4cyclictest3545841-21gzip00:00:011
3245002992419,2cyclictest3509780-21kworker/u8:0+events_unbound23:35:201
3245010992321,2cyclictest3452869-21kworker/u8:3+events_unbound23:20:253
3245010992321,2cyclictest3452869-21kworker/u8:3+events_unbound22:55:373
3245010992321,2cyclictest3385572-21kworker/u8:0+events_unbound21:39:373
3245010992321,2cyclictest3323470-21kworker/u8:2+events_unbound22:10:243
3245010992321,2cyclictest3260273-21kworker/u8:1+events_unbound19:40:273
3245010992321,2cyclictest3244623-21kworker/u8:3+events_unbound19:35:013
3245010992319,3cyclictest3255740-21/usr/sbin/munin19:20:273
3245010992316,5cyclictest3499902-21diskstats23:15:183
3245006992319,3cyclictest3244623-21kworker/u8:3+events_unbound19:10:022
3245006992318,3cyclictest3302425-21/usr/sbin/munin20:05:162
3245002992321,2cyclictest3452869-21kworker/u8:3+events_unbound23:20:171
3245002992320,2cyclictest3462915-21kworker/u8:0+events_unbound22:45:281
3245002992319,3cyclictest3535543-21kworker/u8:2+flush-259:000:05:291
3245010992220,2cyclictest3462915-21kworker/u8:0+events_unbound22:45:263
3245010992219,2cyclictest3571998-21kworker/u8:1+events_unbound00:34:383
3245006992219,2cyclictest3452869-21kworker/u8:3+flush-259:023:55:192
3245002992219,2cyclictest3323470-21kworker/u8:2+flush-259:021:09:571
3245002992218,2cyclictest3509780-21kworker/u8:0+events_unbound23:25:161
3245002992216,5cyclictest3462203-21ntpq22:35:231
3244998992218,3cyclictest3281828-21kworker/u8:0+flush-259:020:05:000
3244998992216,4cyclictest3322818-21latency_hist20:25:010
3245010992119,2cyclictest3333610-21kworker/u8:1+events_unbound21:19:563
3245010992119,2cyclictest3323470-21kworker/u8:2+events_unbound21:55:273
3245010992119,2cyclictest3312557-21kworker/u8:0+events_unbound20:21:543
3245010992118,2cyclictest3385572-21kworker/u8:0+events_unbound21:27:383
3245010992118,2cyclictest3343914-21kworker/u8:3+events_unbound21:12:383
3245006992118,2cyclictest3323470-21kworker/u8:2+flush-259:021:15:002
3245006992117,3cyclictest3365142-21/usr/sbin/munin21:05:172
3245002992118,2cyclictest3447340-21kworker/u8:1+flush-259:022:50:201
3245002992118,2cyclictest3333610-21kworker/u8:1+flush-259:021:15:211
3245002992118,2cyclictest3244623-21kworker/u8:3+events_unbound19:20:241
3245002992116,4cyclictest3349007-21latency_hist20:50:011
3244998992118,2cyclictest3462915-21kworker/u8:0+flush-259:022:45:220
3244998992118,2cyclictest3452869-21kworker/u8:3+flush-259:000:00:000
3244998992117,3cyclictest3323470-21kworker/u8:2+events_unbound21:45:000
3244998992116,4cyclictest3566840-21latency_hist00:20:010
3245010992015,2cyclictest3509780-21kworker/u8:0+events_unbound23:40:233
3245002992017,2cyclictest3452869-21kworker/u8:3+events_unbound23:10:351
3245002992017,2cyclictest3396057-21kworker/u8:3+flush-259:021:35:261
3245002992017,2cyclictest3333610-21kworker/u8:1+flush-259:020:40:261
3245002992016,3cyclictest3234575-21kworker/u8:0+events_unbound19:15:251
3244998992017,2cyclictest3343914-21kworker/u8:3+flush-259:021:00:180
3244998992017,2cyclictest3291905-21kworker/u8:3+flush-259:020:15:210
3244998992017,2cyclictest3291905-21kworker/u8:3+flush-259:020:15:210
3244998992017,2cyclictest2698871-21systemd-journal00:15:000
3244998992016,3cyclictest3582299-21apt-get00:34:590
3244998992015,4cyclictest3468273-21apt-get22:45:000
3245010991917,2cyclictest3535543-21kworker/u8:2+events_unbound00:05:243
3245010991917,2cyclictest3452869-21kworker/u8:3+events_unbound22:38:553
3245010991917,2cyclictest3323470-21kworker/u8:2+events_unbound20:25:253
3245010991916,2cyclictest3452869-21kworker/u8:3+events_unbound23:12:313
3245010991916,2cyclictest3323470-21kworker/u8:2+events_unbound20:41:383
3245010991911,1cyclictest191rcu_preempt23:30:143
3245006991916,2cyclictest3452869-21kworker/u8:3+events_unbound00:00:142
3245006991915,3cyclictest3260273-21kworker/u8:1+events_unbound19:45:012
3245006991914,4cyclictest3468273-21apt-get22:45:022
3245006991914,4cyclictest3281179-21apt-get19:45:002
324500699191,16cyclictest1-21systemd20:45:002
3245002991916,2cyclictest3291905-21kworker/u8:3+events_unbound20:10:251
3245002991913,4cyclictest3411489-21/usr/sbin/munin21:50:171
3244998991916,2cyclictest3509780-21kworker/u8:0+flush-259:023:30:000
3244998991916,2cyclictest3307640-21apt-get20:10:150
3244998991916,2cyclictest3260273-21kworker/u8:1+flush-259:019:25:180
3245010991816,2cyclictest3281828-21kworker/u8:0+events_unbound19:58:373
3245010991816,1cyclictest3281828-21kworker/u8:0+events_unbound19:54:553
3245010991815,2cyclictest3494369-21cron23:10:003
3245010991815,2cyclictest3462915-21kworker/u8:0+events_unbound22:42:373
3245010991815,2cyclictest3291905-21kworker/u8:3+events_unbound20:16:383
3245010991815,2cyclictest3291905-21kworker/u8:3+events_unbound20:16:383
3245010991812,4cyclictest3000063-21rs:main3
3245006991815,2cyclictest3478972-21kworker/u8:2+flush-259:023:05:182
3245002991816,2cyclictest3291905-21kworker/u8:3+events_unbound20:15:231
3245002991816,2cyclictest3291905-21kworker/u8:3+events_unbound20:15:231
3245002991815,2cyclictest3509780-21kworker/u8:0+flush-259:023:49:571
3245002991815,2cyclictest3478972-21kworker/u8:2+flush-259:023:05:261
3245002991815,2cyclictest3323470-21kworker/u8:2+events_unbound20:26:351
3244998991816,1cyclictest3244992-21cyclictest22:40:000
3244998991816,1cyclictest0-21swapper/020:55:000
3244998991815,2cyclictest3260273-21kworker/u8:1+flush-259:019:45:190
3244998991815,2cyclictest0-21swapper/020:10:000
3244998991814,3cyclictest3281828-21kworker/u8:0+flush-259:019:55:260
324501099176,6cyclictest3354141-21munin-run20:54:593
3245010991715,2cyclictest3452869-21kworker/u8:3+events_unbound00:02:153
3245010991714,2cyclictest3385572-21kworker/u8:0+events_unbound21:34:373
3245006991715,1cyclictest3509780-21kworker/u8:0+flush-259:000:25:012
3245006991714,2cyclictest3516205-21kthreadcore23:30:182
3245006991714,2cyclictest3452869-21kworker/u8:3+flush-259:000:30:172
3245006991714,2cyclictest3421770-21kworker/u8:0+flush-259:022:00:202
3245006991714,2cyclictest3385572-21kworker/u8:0+events_unbound21:35:012
3245006991714,2cyclictest3323470-21kworker/u8:2+events_unbound20:25:182
3245006991713,3cyclictest3000063-21rs:main2
3245006991713,3cyclictest2698871-21systemd-journal20:18:382
3245006991713,3cyclictest2698871-21systemd-journal20:18:372
3244998991714,2cyclictest3478972-21kworker/u8:2+flush-259:023:05:200
3244998991714,2cyclictest3478972-21kworker/u8:2+flush-259:023:00:010
3244998991714,2cyclictest3328315-21kworker/u8:4+events_unbound20:30:000
3244998991714,2cyclictest3244623-21kworker/u8:3+flush-259:019:15:010
3244998991712,4cyclictest2698871-21systemd-journal21:07:380
3245010991614,2cyclictest3452869-21kworker/u8:3+events_unbound00:25:283
3245010991614,2cyclictest3291905-21kworker/u8:3+events_unbound20:08:373
3245010991614,2cyclictest3244623-21kworker/u8:3+events_unbound19:27:383
3245010991613,2cyclictest3509780-21kworker/u8:0+events_unbound23:28:553
3245010991611,2cyclictest3291905-21kworker/u8:3+events_unbound20:04:323
3245006991614,1cyclictest3510445-21irqcore23:25:192
3245006991613,2cyclictest3333610-21kworker/u8:1+flush-259:020:35:152
3245006991613,2cyclictest3323470-21kworker/u8:2+flush-259:022:20:272
3245006991613,2cyclictest3323470-21kworker/u8:2+flush-259:021:20:182
3245006991612,3cyclictest3452869-21kworker/u8:3+flush-259:023:20:002
3245006991611,3cyclictest3000063-21rs:main2
3245002991614,1cyclictest3333610-21kworker/u8:1+flush-259:020:55:211
3245002991613,2cyclictest3535543-21kworker/u8:2+events_unbound00:00:181
3245002991613,2cyclictest3478972-21kworker/u8:2+events_unbound23:01:141
3245002991613,2cyclictest3452869-21kworker/u8:3+events_unbound23:30:301
3245002991613,2cyclictest3447340-21kworker/u8:1+events_unbound22:33:401
3245002991613,2cyclictest3447340-21kworker/u8:1+events_unbound22:33:391
3244998991614,1cyclictest60650irq/133-enp0s29f1:rx-023:40:180
3244998991614,1cyclictest60650irq/133-enp0s29f1:rx-022:00:240
3244998991613,2cyclictest60650irq/133-enp0s29f1:rx-021:10:190
3244998991613,2cyclictest60650irq/133-enp0s29f1:rx-019:20:150
3244998991613,2cyclictest3509780-21kworker/u8:0+flush-259:000:20:230
3244998991613,2cyclictest3447340-21kworker/u8:1+flush-259:022:50:180
3244998991610,4cyclictest3457989-21latency_hist22:35:000
3244998991610,4cyclictest3457989-21latency_hist22:35:000
324501099155,1cyclictest0-21swapper/323:45:143
324501099154,2cyclictest0-21swapper/322:05:173
324501099153,8cyclictest50-21ksoftirqd/321:45:183
3245010991512,2cyclictest3582188-21kworker/u8:2+events_unbound00:36:363
3245010991512,2cyclictest3431880-21kworker/u8:3+events_unbound22:18:223
3245010991512,2cyclictest3333610-21kworker/u8:1+events_unbound21:02:183
3245006991512,2cyclictest3561232-21idleruntime-cro00:14:592
3245006991512,2cyclictest3405823-21kworker/u8:1+flush-259:021:50:192
3245006991511,3cyclictest3421770-21kworker/u8:0+events_unbound22:05:172
3245006991510,4cyclictest3587885-21latency_hist00:40:002
3245006991510,4cyclictest3312539-21latency_hist20:15:012
324500299157,1cyclictest191rcu_preempt00:35:171
3245002991512,2cyclictest61550irq/136-enp0s29f1:tx-123:52:471
3245002991512,2cyclictest3421770-21kworker/u8:0+events_unbound22:08:391
3244998991513,1cyclictest60650irq/133-enp0s29f1:rx-023:45:240
3244998991513,1cyclictest60650irq/133-enp0s29f1:rx-022:25:150
3244998991513,1cyclictest60650irq/133-enp0s29f1:rx-022:05:170
3244998991513,1cyclictest60650irq/133-enp0s29f1:rx-020:37:510
3244998991513,1cyclictest60650irq/133-enp0s29f1:rx-000:25:150
3244998991513,1cyclictest3452869-21kworker/u8:3+events_unbound23:20:150
3244998991512,2cyclictest3396057-21kworker/u8:3+flush-259:021:35:160
324501099147,1cyclictest191rcu_preempt23:55:263
324501099143,3cyclictest50-21ksoftirqd/321:40:143
324500699149,4cyclictest2698871-21systemd-journal21:57:372
324500699147,6cyclictest3291863-21cron19:55:002
3245006991412,1cyclictest3484766-21irqcore23:00:192
3245006991412,1cyclictest0-21swapper/223:10:272
3245006991411,2cyclictest3323470-21kworker/u8:2+flush-259:020:30:262
3245006991411,2cyclictest3271589-21irqcore19:35:182
3245006991411,1cyclictest0-21swapper/220:25:012
3245002991412,2cyclictest3343914-21kworker/u8:3+events_unbound21:12:061
324499899149,4cyclictest3276298-21apt-get19:40:130
3244998991413,1cyclictest60650irq/133-enp0s29f1:rx-000:08:540
3244998991412,1cyclictest60650irq/133-enp0s29f1:rx-023:51:140
3244998991412,1cyclictest60650irq/133-enp0s29f1:rx-023:38:490
3244998991412,1cyclictest60650irq/133-enp0s29f1:rx-022:24:270
3244998991412,1cyclictest60650irq/133-enp0s29f1:rx-021:55:150
3244998991412,1cyclictest60650irq/133-enp0s29f1:rx-020:45:590
3244998991412,1cyclictest60650irq/133-enp0s29f1:rx-019:31:460
3244998991412,1cyclictest0-21swapper/023:35:000
3244998991411,2cyclictest60650irq/133-enp0s29f1:rx-023:00:240
3244998991411,2cyclictest60650irq/133-enp0s29f1:rx-021:30:300
3244998991411,2cyclictest60650irq/133-enp0s29f1:rx-020:40:180
3244998991411,2cyclictest3333610-21kworker/u8:1+flush-259:020:55:160
3244998991410,3cyclictest3405823-21kworker/u8:1+events_unbound21:50:180
3245006991310,2cyclictest3447340-21kworker/u8:1+events_unbound22:30:172
3245006991310,2cyclictest3447340-21kworker/u8:1+events_unbound22:30:172
3245006991310,2cyclictest3323470-21kworker/u8:2+flush-259:020:45:242
3245006991310,2cyclictest3260175-21unixbench_multi19:20:272
324499899137,4cyclictest3499477-21latency_hist23:15:010
3244998991312,1cyclictest60650irq/133-enp0s29f1:rx-000:35:130
3244998991311,1cyclictest60650irq/133-enp0s29f1:rx-022:19:540
3244998991311,1cyclictest60650irq/133-enp0s29f1:rx-022:12:390
3244998991311,1cyclictest60650irq/133-enp0s29f1:rx-019:15:260
3244998991310,2cyclictest0-21swapper/021:15:150
324500699129,2cyclictest3483079-21needreboot22:55:212
324500699129,2cyclictest3360215-21fschecks_time21:00:162
324500699129,2cyclictest3261079-21fschecks_count19:25:162
3245006991210,1cyclictest3266470-21irqcore19:30:212
324500299129,2cyclictest3452869-21kworker/u8:3+events_unbound23:15:191
3244998991210,1cyclictest60650irq/133-enp0s29f1:rx-023:18:260
324500699118,2cyclictest3504970-21cpuspeed_turbos23:20:152
324500699115,5cyclictest3437598-21rred22:15:012
324499899108,1cyclictest60650irq/133-enp0s29f1:rx-000:00:150
32450069997,1cyclictest0-21swapper/223:35:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional