You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-10 - 02:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Sep 09, 2026 12:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3637269999591,2cyclictest0-21swapper/010:35:010
3637269997934,21cyclictest131rcu_preempt09:30:010
3637273997129,38cyclictest3639763-21sshd07:55:191
3637269996932,16cyclictest131rcu_preempt10:10:010
363727399682,14cyclictest3735039-21ssh12:25:061
3637269996864,2cyclictest0-21swapper/009:45:020
363727399671,15cyclictest3697162-21ssh11:04:431
363727399671,15cyclictest3697162-21ssh11:04:431
363727399671,12cyclictest3642139-21sshd08:43:191
363727399662,17cyclictest3688000-21systemd-run10:45:011
3637273996617,37cyclictest0-21swapper/109:50:011
3637269996618,5cyclictest131rcu_preempt12:32:450
363727399652,16cyclictest3715928-21ssh11:45:021
363727399651,14cyclictest3656004-21ssh09:36:301
363727399651,14cyclictest3656004-21ssh09:36:291
363727399651,13cyclictest3639059-21pluginstate07:45:011
363727399651,10cyclictest3684230-21ssh10:36:481
363727399642,15cyclictest3642928-21munin-run09:00:011
363727399641,12cyclictest3701923-21ssh11:14:541
363727399641,12cyclictest3467602-21systemd-journal10:06:011
363727399641,11cyclictest3731129-21ssh12:17:041
363727399641,11cyclictest3647912-21ssh09:19:341
363727399641,11cyclictest3647912-21ssh09:19:331
363727399641,10cyclictest3640831-21sshd08:19:181
3637273996344,14cyclictest275-21dbus-daemon12:00:011
363727399632,13cyclictest3654250-21ssh09:32:531
363727399631,13cyclictest3702930-21ssh11:16:551
3637269996327,5cyclictest131rcu_preempt10:50:000
363727399622,16cyclictest3729176-21ssh12:12:531
363727399622,16cyclictest3729176-21ssh12:12:521
363727399622,14cyclictest3639562-21munin-run07:55:001
363727399622,12cyclictest3694826-21ssh10:59:371
363727399621,12cyclictest3679180-21ssh10:25:531
363727399621,11cyclictest48-21kcompactd009:50:251
363727399621,10cyclictest3644403-21ssh09:11:451
3637269996220,7cyclictest16850irq/134-800000010:27:450
3637269996218,40cyclictest12-21ksoftirqd/012:21:430
3637273996133,24cyclictest3637730-21cron07:20:011
3637273996121,36cyclictest3705649-21sh11:22:341
363727399612,12cyclictest3674467-21ssh10:16:001
363727399611,13cyclictest3648450-21ssh09:20:161
363727399611,12cyclictest3726181-21ssh12:06:201
363727399611,10cyclictest3657822-21ssh09:40:181
3637269996130,14cyclictest3699761-21ssh11:10:010
3637269996116,40cyclictest12-21ksoftirqd/009:47:230
363727399601,9cyclictest3723979-21ssh12:01:451
3637269996030,15cyclictest3644892-21ssh09:12:590
3637269996010,42cyclictest12-21ksoftirqd/009:39:070
3637269996010,42cyclictest12-21ksoftirqd/009:39:070
3637282995938,16cyclictest3467602-21systemd-journal08:19:593
363727399592,15cyclictest3638515-21munin-run07:35:021
363727399592,14cyclictest3640588-21munin-run08:15:011
363727399591,9cyclictest3467602-21systemd-journal10:13:031
3637273995912,40cyclictest275-21dbus-daemon11:55:001
363727399591,16cyclictest3689552-21ssh10:48:111
3637273995911,43cyclictest0-21swapper/112:20:411
363727399591,10cyclictest3711719-21ssh11:35:391
363727399591,10cyclictest364-21snmpd09:09:361
363727399591,10cyclictest364-21snmpd09:09:351
363726999598,38cyclictest131rcu_preempt10:05:000
363726999591,43cyclictest3715030-21ssh11:42:490
3637278995842,7cyclictest3680917-21kworker/2:311:34:062
363727399582,15cyclictest3690964-21ssh10:51:011
363727399582,11cyclictest364-21snmpd11:10:001
363727399581,16cyclictest3638274-21systemd-run07:30:011
363727399581,15cyclictest3641614-21sshd08:34:171
363727399581,13cyclictest3641394-21pluginstate08:30:021
363727399581,13cyclictest3641394-21pluginstate08:30:011
363727399581,12cyclictest364-21snmpd11:34:061
363727399581,10cyclictest3664926-21sh09:55:411
3637269995835,19cyclictest16850irq/134-800000010:55:020
3637269995821,26cyclictest151rcuc/007:40:020
3637273995718,35cyclictest23-21ksoftirqd/108:37:111
363727399571,11cyclictest48-21kcompactd009:26:311
363726999571,40cyclictest37117961ssh11:35:470
363727399564,8cyclictest3467602-21systemd-journal08:50:011
363727399562,5cyclictest1-21systemd07:15:001
3637273995620,32cyclictest3643454-21diskmemload12:36:121
3637273995616,35cyclictest3643454-21diskmemload12:30:521
363727399561,17cyclictest3716352-21ssh11:45:291
363727399561,11cyclictest3643204-21systemd-run09:05:001
3637269995630,7cyclictest3676272-21modprobe10:20:000
3637269995629,12cyclictest3691189-21ssh10:51:310
3637269995628,24cyclictest36469981ssh09:17:280
3637269995628,24cyclictest36469981ssh09:17:280
3637269995628,13cyclictest3726441-21ssh12:07:060
3637269995615,4cyclictest12-21ksoftirqd/010:42:260
3637282995515,19cyclictest3716096-21ssh11:45:033
3637278995547,5cyclictest3638289-21kworker/2:309:33:202
363727399557,5cyclictest3467602-21systemd-journal08:08:541
363727399552,14cyclictest3642638-21sshd08:52:191
363727399551,15cyclictest3637275-21latency_hist07:10:011
363727399551,12cyclictest3669165-21munin-run10:05:011
3637269995527,13cyclictest3701581-21ssh11:14:120
3637269995523,15cyclictest3705543-21ssh11:22:200
3637269995518,32cyclictest16850irq/134-800000009:33:310
3637269995517,18cyclictest3725418-21ssh12:05:010
3637269995511,39cyclictest0-21swapper/008:00:000
3637278995445,6cyclictest3680917-21kworker/2:311:08:472
3637278995443,7cyclictest3638289-21kworker/2:309:14:532
363727399542,6cyclictest3638752-21cron07:39:591
363727399542,13cyclictest3707277-21sh11:26:081
363727399542,13cyclictest3707277-21sh11:26:081
363727399541,13cyclictest3677450-21ssh10:22:201
3637269995434,4cyclictest16850irq/134-800000012:15:010
3637269995434,4cyclictest16850irq/134-800000012:15:000
3637269995429,19cyclictest131rcu_preempt09:20:150
3637269995428,10cyclictest3676666-21ssh10:20:320
3637269995426,12cyclictest3717170-21ssh11:47:200
363726999542,45cyclictest3685791-21latency_hist10:40:030
3637269995417,32cyclictest16850irq/134-800000011:26:470
3637269995417,32cyclictest16850irq/134-800000011:26:460
3637269995417,32cyclictest0-21swapper/011:55:000
3637278995345,5cyclictest3680917-21kworker/2:312:16:492
3637278995345,5cyclictest3680917-21kworker/2:311:39:282
3637278995345,5cyclictest3660119-21kworker/2:012:35:562
3637278995344,6cyclictest3680917-21kworker/2:310:30:232
3637278995344,6cyclictest3660119-21kworker/2:010:10:062
3637278995344,5cyclictest3652853-21kworker/2:209:44:232
3637278995343,6cyclictest3680917-21kworker/2:311:01:012
3637278995343,6cyclictest3680917-21kworker/2:311:01:002
3637278995343,6cyclictest3660119-21kworker/2:009:54:452
3637278995342,6cyclictest3680917-21kworker/2:311:27:072
3637278995342,6cyclictest3680917-21kworker/2:311:27:062
3637278995341,8cyclictest3680917-21kworker/2:311:11:442
3637278995341,8cyclictest3680917-21kworker/2:310:40:322
363727899531,46cyclictest3652900-21ssh09:30:022
363727399532,15cyclictest3639291-21cron07:50:001
363727399531,11cyclictest3681628-21ssh10:31:061
3637269995318,31cyclictest16850irq/134-800000009:53:160
3637269995317,32cyclictest16850irq/134-800000010:11:240
3637269995317,31cyclictest16850irq/134-800000011:18:310
3637269995316,6cyclictest16850irq/134-800000012:39:290
3637278995244,5cyclictest3680917-21kworker/2:312:02:112
3637278995244,5cyclictest3660119-21kworker/2:010:01:192
3637278995244,5cyclictest3638289-21kworker/2:309:22:202
3637278995243,5cyclictest3660119-21kworker/2:010:22:202
3637278995242,6cyclictest3680917-21kworker/2:310:51:232
3637278995242,6cyclictest3660119-21kworker/2:012:32:172
3637269995226,11cyclictest3664829-21ssh09:55:200
3637269995217,6cyclictest16850irq/134-800000011:00:540
3637269995217,6cyclictest16850irq/134-800000011:00:540
3637269995217,31cyclictest16850irq/134-800000012:26:160
3637269995217,31cyclictest16850irq/134-800000011:34:160
3637269995216,3cyclictest12-21ksoftirqd/008:49:310
363726999521,36cyclictest3730760-21ssh12:16:030
3637282995134,12cyclictest3467602-21systemd-journal11:00:003
3637278995144,4cyclictest3680917-21kworker/2:310:50:002
3637278995143,5cyclictest3680917-21kworker/2:312:11:222
3637278995143,5cyclictest3680917-21kworker/2:312:11:212
3637278995143,5cyclictest3680917-21kworker/2:311:16:572
3637278995143,5cyclictest3680917-21kworker/2:310:38:572
3637278995143,5cyclictest3660119-21kworker/2:010:07:212
3637278995143,5cyclictest3638289-21kworker/2:309:15:352
3637278995143,5cyclictest3638289-21kworker/2:309:15:342
3637278995142,6cyclictest3660119-21kworker/2:011:59:132
3637278995142,5cyclictest3680917-21kworker/2:311:49:562
3637278995142,5cyclictest3660119-21kworker/2:010:16:342
3637278995141,6cyclictest3652853-21kworker/2:209:45:522
363727399511,10cyclictest364-21snmpd08:24:261
363726999511,35cyclictest3721799-21ssh11:57:140
3637282995033,12cyclictest309-21in:imuxsock09:51:193
3637278995042,5cyclictest3680917-21kworker/2:312:08:322
3637278995042,5cyclictest3680917-21kworker/2:311:23:152
3637278995042,5cyclictest3680917-21kworker/2:310:55:282
3637278995042,5cyclictest3660119-21kworker/2:012:26:272
3637278995042,5cyclictest3660119-21kworker/2:012:21:572
3637278995042,5cyclictest3660119-21kworker/2:010:26:162
3637278995042,5cyclictest3660119-21kworker/2:009:57:172
3637278995042,5cyclictest3638289-21kworker/2:309:36:282
3637278995042,5cyclictest3638289-21kworker/2:309:36:272
3637278995042,5cyclictest3628098-21kworker/2:007:23:192
3637278995040,6cyclictest3680917-21kworker/2:311:40:512
363728299495,3cyclictest131rcu_preempt10:37:223
3637282994934,11cyclictest1-21systemd08:05:013
3637278994941,5cyclictest3628098-21kworker/2:007:45:022
3637278994931,14cyclictest3638770-21cat07:40:012
363726999494,8cyclictest3643479-21ssh09:10:010
363726999494,8cyclictest3643479-21ssh09:10:010
3637278994841,4cyclictest3680917-21kworker/2:311:50:042
3637278994840,5cyclictest3638289-21kworker/2:308:52:592
3637278994840,5cyclictest3628098-21kworker/2:008:05:022
3637278994836,9cyclictest3642887-21sshd08:56:202
363727399481,10cyclictest3637982-21sshd07:22:171
3637269994829,4cyclictest0-21swapper/008:30:040
363726999482,14cyclictest3640079-21munin-plugin-st08:05:000
3637278994739,5cyclictest3638289-21kworker/2:308:30:022
3637278994739,5cyclictest3638289-21kworker/2:308:30:012
3637278994739,5cyclictest3628098-21kworker/2:007:50:012
3637278994731,11cyclictest275-21dbus-daemon08:50:012
3637269994715,28cyclictest121ksoftirqd/008:59:190
3637282994628,13cyclictest275-21dbus-daemon10:45:023
3637278994639,4cyclictest3638289-21kworker/2:308:33:142
3637278994638,5cyclictest3628098-21kworker/2:008:05:522
3637278994638,5cyclictest3628098-21kworker/2:007:27:192
3637269994629,13cyclictest3641393-21systemd-run08:30:020
3637269994629,13cyclictest3641393-21systemd-run08:30:010
363726999461,11cyclictest3641886-21cron08:40:000
3637278994537,5cyclictest3638289-21kworker/2:308:43:142
3637278994537,5cyclictest3638289-21kworker/2:308:18:372
363727399454,4cyclictest3640081-21modprobe08:05:001
3637269994538,4cyclictest16950irq/224-800000008:55:010
3637269994532,9cyclictest12-21ksoftirqd/008:25:010
3637282994432,9cyclictest37-21ksoftirqd/311:00:493
3637282994432,9cyclictest37-21ksoftirqd/311:00:483
3637282994428,12cyclictest275-21dbus-daemon10:20:013
363728299441,4cyclictest131rcu_preempt11:20:133
363728299441,3cyclictest131rcu_preempt10:50:143
3637278994436,5cyclictest3628098-21kworker/2:007:10:542
3637278994429,12cyclictest3639755-21sed07:55:022
3637269994436,5cyclictest16950irq/224-800000007:20:000
3637269994424,14cyclictest275-21dbus-daemon07:55:010
3637269994420,6cyclictest131rcu_preempt08:20:000
363728299435,4cyclictest131rcu_preempt09:13:073
363728299433,3cyclictest131rcu_preempt10:23:313
3637282994333,5cyclictest3637738-21munin-plugin-st07:20:003
3637282994327,11cyclictest1-21systemd07:25:003
363728299432,3cyclictest131rcu_preempt12:15:533
363728299431,3cyclictest131rcu_preempt09:40:023
363728299431,3cyclictest131rcu_preempt09:40:013
3637278994336,5cyclictest3628098-21kworker/2:007:57:362
3637278994335,5cyclictest3638289-21kworker/2:308:22:352
3637278994335,5cyclictest3628098-21kworker/2:008:14:242
3637278994335,5cyclictest3628098-21kworker/2:007:15:072
363726999431,7cyclictest3638766-21sh07:40:010
363728299424,3cyclictest131rcu_preempt12:21:353
363728299423,4cyclictest131rcu_preempt11:39:273
3637282994230,7cyclictest309-21in:imuxsock09:10:003
3637282994230,7cyclictest309-21in:imuxsock09:10:003
3637282994226,12cyclictest37-21ksoftirqd/311:20:033
363728299422,3cyclictest131rcu_preempt09:23:043
363728299421,3cyclictest131rcu_preempt12:11:373
363728299421,3cyclictest131rcu_preempt12:11:373
363728299421,3cyclictest131rcu_preempt10:28:133
363728299421,37cyclictest37-21ksoftirqd/309:16:263
363728299421,37cyclictest37-21ksoftirqd/309:16:263
3637278994234,5cyclictest3638289-21kworker/2:308:36:182
3637278994234,5cyclictest3628098-21kworker/2:007:32:312
363726999423,7cyclictest1-21systemd07:25:000
3637269994236,3cyclictest16950irq/224-800000007:15:020
3637269994226,12cyclictest3467718-21lldpd08:09:240
3637282994133,5cyclictest37-21ksoftirqd/311:30:023
3637282994133,5cyclictest37-21ksoftirqd/311:30:013
3637282994131,7cyclictest1-21systemd11:55:013
3637282994129,7cyclictest37-21ksoftirqd/310:35:013
3637282994127,12cyclictest37-21ksoftirqd/311:55:123
3637282994126,12cyclictest3467602-21systemd-journal07:35:003
3637282994126,11cyclictest3727235-21md.daily12:09:003
3637282994126,11cyclictest37-21ksoftirqd/312:27:463
3637282994125,12cyclictest37-21ksoftirqd/312:31:313
363728299411,3cyclictest131rcu_preempt11:13:053
3637269994121,5cyclictest12-21ksoftirqd/007:50:000
3637282994031,5cyclictest3641400-21latency_hist08:30:013
3637282994031,5cyclictest3641400-21latency_hist08:30:013
3637282994029,7cyclictest309-21in:imuxsock09:49:253
363728299402,3cyclictest131rcu_preempt10:49:123
363728299401,4cyclictest131rcu_preempt12:04:373
363728299401,3cyclictest131rcu_preempt11:30:153
363728299401,3cyclictest131rcu_preempt09:43:563
3637282994010,6cyclictest131rcu_preempt08:55:013
3637278994032,5cyclictest3638289-21kworker/2:309:05:452
3637278994032,5cyclictest3638289-21kworker/2:309:05:442
3637269994031,5cyclictest3640583-21cron08:15:000
3637282993930,7cyclictest37-21ksoftirqd/309:34:183
3637282993928,8cyclictest37-21ksoftirqd/311:45:513
3637282993922,13cyclictest3642906-21cron09:00:003
363728299392,11cyclictest3671332-21ssh10:09:363
3637282993911,5cyclictest131rcu_preempt08:12:183
363726999396,5cyclictest131rcu_preempt07:35:010
3637269993924,11cyclictest12-21ksoftirqd/008:45:000
3637282993824,11cyclictest0-21swapper/310:05:013
3637282993824,10cyclictest37-21ksoftirqd/309:29:173
363728299382,3cyclictest131rcu_preempt07:30:013
3637282993823,11cyclictest37-21ksoftirqd/310:00:013
3637278993830,5cyclictest3638289-21kworker/2:309:04:332
3637282993724,9cyclictest275-21dbus-daemon08:25:013
363728299372,3cyclictest131rcu_preempt11:09:223
363728299371,4cyclictest131rcu_preempt12:39:313
363728299371,32cyclictest3467602-21systemd-journal08:40:003
3637269993723,11cyclictest12-21ksoftirqd/007:25:330
363726999371,7cyclictest151rcuc/009:00:190
3637282993627,6cyclictest37-21ksoftirqd/310:13:303
3637269993610,5cyclictest131rcu_preempt07:10:010
3637282993525,8cyclictest37-21ksoftirqd/307:50:013
3637282993525,7cyclictest37-21ksoftirqd/309:05:013
3637282993523,8cyclictest37-21ksoftirqd/307:40:013
363728299351,4cyclictest131rcu_preempt08:35:003
3637282993421,10cyclictest37-21ksoftirqd/308:10:003
363728299341,29cyclictest48-21kcompactd007:55:303
363728299337,22cyclictest3639725-21cat07:55:023
3637282993320,10cyclictest37-21ksoftirqd/307:10:023
3637282993223,7cyclictest37-21ksoftirqd/308:50:023
3637282993223,7cyclictest37-21ksoftirqd/307:41:193
3637282993120,9cyclictest37-21ksoftirqd/307:15:013
3637282993023,4cyclictest3642175-21sh08:45:003
3637278993021,7cyclictest30-21ksoftirqd/207:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional