You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-22 - 09:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Aug 22, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
319059099121111,8cyclictest23-21ksoftirqd/120:55:021
31905879910399,2cyclictest0-21swapper/019:10:020
3190587999711,16cyclictest16850irq/134-800000022:00:030
3190587997310,45cyclictest16950irq/224-800000022:30:020
3190600997221,15cyclictest3251355-21ssh23:05:373
3190600997121,15cyclictest3203416-21ssh21:23:593
3190600997033,9cyclictest3291966-21ssh00:32:033
3190600996950,15cyclictest3213157-21ssh21:44:483
3190600996920,15cyclictest3193312-21cat20:00:023
3190594996830,34cyclictest3283947-21sed00:15:022
3190590996825,5cyclictest3229684-21sendmail22:20:001
3190590996822,12cyclictest3467602-21systemd-journal23:05:001
3190590996727,8cyclictest3235233-21sshd22:31:191
3190590996722,5cyclictest309-21in:imuxsock22:54:551
319058799668,6cyclictest131rcu_preempt00:40:000
319058799668,6cyclictest131rcu_preempt00:40:000
3190600996552,10cyclictest3222328-21ssh22:04:183
3190600996525,13cyclictest3196815-21diskmemload23:21:373
3190600996524,14cyclictest3271682-21kworker/u8:123:52:433
3190594996534,26cyclictest3248550-21ssh23:00:002
319059499652,6cyclictest0-21swapper/223:14:292
3190594996521,38cyclictest309-21in:imuxsock19:35:002
3190590996525,7cyclictest3218009-21cat21:55:021
319059499642,19cyclictest3239115-21cron22:40:002
3190590996431,29cyclictest3467602-21systemd-journal20:50:001
3190590996424,7cyclictest23-21ksoftirqd/122:27:031
3190600996345,14cyclictest3264647-21ssh23:34:173
3190600996343,16cyclictest3193931-21sshd20:13:563
3190600996340,18cyclictest3237320-21ssh22:35:463
3190600996324,11cyclictest0-21swapper/323:59:013
319059499632,23cyclictest3269855-21sed23:45:022
319059499631,9cyclictest3278486-21ssh00:03:192
319059499631,9cyclictest3207718-21ssh21:33:082
319059499631,10cyclictest3284579-21ssh00:16:092
3190590996318,13cyclictest3467602-21systemd-journal19:19:591
319058799635,33cyclictest16950irq/224-800000023:25:010
3190587996327,7cyclictest12-21ksoftirqd/022:04:060
3190600996251,9cyclictest3257076-21ssh23:18:063
3190600996251,9cyclictest3257076-21ssh23:18:063
3190600996250,9cyclictest3223817-21ssh22:07:213
3190600996223,12cyclictest3213700-21ssh21:45:383
3190594996237,5cyclictest3229746-21kworker/2:022:48:152
319059499622,17cyclictest3239537-21ssh22:40:202
319059499622,15cyclictest3204525-21sshd21:26:032
319059499621,15cyclictest48-21kcompactd021:35:182
319059499621,10cyclictest3194979-21sshd20:34:562
3190590996219,11cyclictest23-21ksoftirqd/122:09:491
3190600996150,9cyclictest3253915-21ssh23:11:103
3190600996148,10cyclictest3242182-21ssh22:46:043
3190600996145,13cyclictest3293738-21ssh00:35:423
3190600996145,13cyclictest3293738-21ssh00:35:423
3190600996145,13cyclictest3270179-21ssh23:45:303
3190600996145,13cyclictest3270179-21ssh23:45:303
3190600996144,14cyclictest3219405-21ssh21:58:063
3190600996143,14cyclictest3287402-21ssh00:22:073
3190600996143,14cyclictest3261412-21ssh23:27:173
3190600996143,14cyclictest3261412-21ssh23:27:173
3190600996123,13cyclictest3282888-21ssh00:12:433
3190600996119,14cyclictest364-21snmpd21:34:223
319059499612,7cyclictest3249156-21sshd23:00:552
3190594996123,33cyclictest30-21ksoftirqd/222:30:292
319059499612,12cyclictest3194178-21cron20:17:012
319059499611,9cyclictest3227791-21ssh22:15:362
319059499611,9cyclictest3217228-21ssh21:53:262
319059499611,12cyclictest3214815-21sh21:48:182
319059499611,10cyclictest3276261-21ssh23:58:482
319059499611,10cyclictest3274313-21ssh23:54:432
319059099614,18cyclictest3193093-21latency_hist19:55:021
3190600996047,10cyclictest3267595-21ssh23:40:043
3190600996045,11cyclictest3245840-21ssh22:54:123
3190600996045,11cyclictest3230276-21ssh22:20:453
3190600996044,13cyclictest3265271-21ssh23:35:173
3190600996044,13cyclictest3265271-21ssh23:35:173
3190600996043,13cyclictest3248225-21ssh22:59:193
3190600996043,13cyclictest3233403-21ssh22:27:373
3190600996043,13cyclictest3209421-21ssh21:36:343
3190600996043,13cyclictest3198418-21ssh21:13:163
3190600996042,14cyclictest3277372-21ssh00:00:483
3190600996042,14cyclictest3226309-21ssh22:12:383
3190600996041,15cyclictest3284863-21ssh00:16:553
3190600996040,16cyclictest3194195-21sshd20:19:543
319059499603,24cyclictest3262635-21latency_hist23:30:002
319059499603,24cyclictest3262635-21latency_hist23:30:002
3190594996031,24cyclictest3281230-21sh00:09:212
319059499601,9cyclictest3230857-21ssh22:22:112
319059499601,8cyclictest3257262-21ssh23:18:292
319059499601,8cyclictest3257262-21ssh23:18:292
319059499601,8cyclictest3233045-21ssh22:26:452
319059499601,12cyclictest3223414-21ssh22:06:192
319059499601,10cyclictest3264527-21ssh23:34:022
3190590996019,12cyclictest3255557-21cron23:15:011
3190587996017,7cyclictest12-21ksoftirqd/021:20:010
3190587996012,33cyclictest16950irq/224-800000020:45:030
3190600995947,9cyclictest3249946-21ssh23:02:473
3190600995944,12cyclictest3289467-21ssh00:26:313
3190600995943,13cyclictest3215976-21ssh21:50:273
3190600995943,12cyclictest3235394-21ssh22:31:403
3190600995943,12cyclictest3200479-21ssh21:17:403
3190600995942,14cyclictest3228075-21ssh22:16:123
319059499591,8cyclictest3192048-21sshd19:35:442
3190594995917,38cyclictest30-21ksoftirqd/221:41:112
319059499591,13cyclictest3295550-21systemd-run00:40:012
319059499591,13cyclictest3295550-21systemd-run00:40:012
319059499591,11cyclictest364-21snmpd21:19:232
319059499591,11cyclictest3265811-21ssh23:36:332
319059499591,11cyclictest3265811-21ssh23:36:332
319059099594,20cyclictest3210895-21cron21:40:011
319059099593,19cyclictest3279293-21latency_hist00:05:021
3190590995925,10cyclictest23-21ksoftirqd/121:42:311
3190590995917,10cyclictest23-21ksoftirqd/121:34:551
319058799598,11cyclictest12-21ksoftirqd/022:10:480
3190587995932,12cyclictest3236665-21ssh22:34:410
3190587995919,32cyclictest12-21ksoftirqd/023:03:160
3190600995845,10cyclictest3240190-21ssh22:41:503
3190600995845,10cyclictest3204677-21ssh21:26:203
3190600995841,13cyclictest3280332-21ssh00:07:163
3190600995822,26cyclictest0-21swapper/320:40:003
3190594995828,27cyclictest3197777-21sh21:11:462
3190594995828,26cyclictest30-21ksoftirqd/223:08:272
319059499582,4cyclictest3195541-21munin-run20:45:022
319059499581,7cyclictest3193683-21sshd20:09:562
319059099582,19cyclictest3283827-21munin-run00:15:011
3190590995819,10cyclictest23-21ksoftirqd/120:25:001
3190590995818,9cyclictest3196336-21sort21:00:021
3190600995742,11cyclictest3192929-21munin-run19:55:013
3190600995737,15cyclictest3192617-21munin-run19:50:013
3190600995731,21cyclictest3192416-21latency_hist19:45:003
3190594995730,23cyclictest3225352-21sh22:10:272
319059499572,19cyclictest3288953-21ssh00:25:152
319059499572,15cyclictest3219584-21ssh21:58:282
319059499572,14cyclictest3258738-21ssh23:21:252
319059499571,9cyclictest48-21kcompactd021:20:142
3190594995712,40cyclictest284-21systemd-logind21:00:012
3190594995711,42cyclictest3193552-21latency_hist20:05:022
3190594995711,42cyclictest3193552-21latency_hist20:05:012
3190590995723,9cyclictest23-21ksoftirqd/121:21:411
3190590995720,10cyclictest23-21ksoftirqd/100:30:151
3190587995724,16cyclictest3276894-21ssh00:00:010
3190600995642,10cyclictest3196247-21sshd20:55:543
3190600995642,10cyclictest3191314-21sshd19:24:553
3190600995642,10cyclictest3191314-21sshd19:24:553
3190600995639,12cyclictest3194967-21sshd20:31:553
319059499562,13cyclictest3191594-21munin-run19:30:012
319059499561,9cyclictest3196541-21sshd21:03:552
319059499561,18cyclictest3291922-21ssh00:31:572
319059499561,15cyclictest3244309-21ssh22:50:352
319059099564,9cyclictest3257909-21modprobe23:20:001
319059099564,9cyclictest3257909-21modprobe23:20:001
3190590995636,15cyclictest275-21dbus-daemon20:20:011
3190590995622,8cyclictest23-21ksoftirqd/123:50:011
3190590995622,8cyclictest23-21ksoftirqd/123:50:011
3190587995627,14cyclictest3264557-21ssh23:34:050
3190587995625,15cyclictest3198608-21ssh21:13:410
3190587995625,14cyclictest3237101-21ssh22:35:190
3190600995542,10cyclictest3195998-21sshd20:50:563
3190600995542,10cyclictest3195497-21sshd20:41:553
3190600995542,10cyclictest3192067-21sshd19:39:553
319059499553,8cyclictest3195767-21cron20:50:002
319059499552,7cyclictest3194743-21munin-run20:30:012
319059499551,9cyclictest364-21snmpd19:12:002
319059499551,9cyclictest364-21snmpd19:12:002
3190594995514,37cyclictest3287839-21rm00:23:142
319059499551,14cyclictest3221273-21ssh22:01:472
3190590995521,10cyclictest23-21ksoftirqd/122:44:381
3190587995548,5cyclictest12-21ksoftirqd/023:45:010
3190587995540,11cyclictest12-21ksoftirqd/022:20:000
3190587995526,13cyclictest3277422-21ssh00:00:530
3190587995518,33cyclictest3245762-21kworker/u8:123:20:000
3190587995518,33cyclictest3245762-21kworker/u8:123:20:000
3190600995440,10cyclictest48-21kcompactd020:00:183
3190600995440,10cyclictest48-21kcompactd020:00:173
319059499541,10cyclictest3271937-21ssh23:49:442
319059499541,10cyclictest3271937-21ssh23:49:442
3190590995426,9cyclictest23-21ksoftirqd/122:01:491
3190590995426,5cyclictest3264638-21sh23:34:161
3190587995444,7cyclictest12-21ksoftirqd/023:36:310
3190587995444,7cyclictest12-21ksoftirqd/023:36:310
3190587995426,12cyclictest3272913-21ssh23:51:260
3190587995424,26cyclictest12-21ksoftirqd/000:35:030
3190587995419,12cyclictest12-21ksoftirqd/023:10:010
3190587995416,12cyclictest12-21ksoftirqd/021:21:330
319058799541,38cyclictest3247727-21ssh22:58:040
3190600995334,14cyclictest364-21snmpd19:27:163
3190600995328,21cyclictest3196791-21sshd21:07:553
319059499537,28cyclictest30-21ksoftirqd/219:40:032
319059499532,12cyclictest3192847-21t-helper19:52:552
319059499531,13cyclictest3196980-21ls21:10:022
319059499531,12cyclictest364-21snmpd19:18:222
3190590995318,8cyclictest3269797-21wc23:45:011
3190587995328,10cyclictest3244682-21ssh22:51:250
3190587995326,12cyclictest3216830-21ssh21:52:250
3190600995237,11cyclictest364-21snmpd20:24:593
3190600995235,13cyclictest364-21snmpd20:49:123
3190600995229,19cyclictest3196579-21latency_hist21:05:023
3190594995232,17cyclictest30-21ksoftirqd/220:15:022
319059499522,7cyclictest364-21snmpd19:47:312
319059499522,7cyclictest3191342-21systemd-run19:25:012
319059499522,7cyclictest3191342-21systemd-run19:25:002
319059499522,6cyclictest3194507-21munin-run20:25:012
3190590995231,16cyclictest275-21dbus-daemon00:10:011
3190590995223,10cyclictest23-21ksoftirqd/100:38:431
3190590995223,10cyclictest23-21ksoftirqd/100:38:431
3190587995243,7cyclictest12-21ksoftirqd/021:46:110
3190587995225,12cyclictest3288579-21ssh00:24:560
3190587995225,12cyclictest3212577-21ssh21:43:290
3190587995223,14cyclictest3240791-21ssh22:43:160
3190587995219,29cyclictest12-21ksoftirqd/021:30:300
319058799521,48cyclictest0-21swapper/019:20:020
3190587995214,19cyclictest16850irq/134-800000000:05:300
3190600995139,9cyclictest364-21snmpd20:28:053
3190600995139,9cyclictest364-21snmpd19:34:373
3190600995138,9cyclictest364-21snmpd20:06:473
3190600995136,11cyclictest3190803-21munin-run19:15:023
3190600995136,11cyclictest3190803-21munin-run19:15:023
319059499518,3cyclictest0-21swapper/220:39:262
319059099514,14cyclictest3196806-21modprobe21:10:011
3190590995125,5cyclictest23-21ksoftirqd/121:16:351
3190590995124,4cyclictest3246502-21sh22:55:171
319058799512,8cyclictest0-21swapper/021:40:020
3190587995123,13cyclictest3282344-21ssh00:11:240
3190587995123,12cyclictest3232039-21ssh22:24:560
3190587995122,14cyclictest3289685-21ssh00:26:570
3190590995024,9cyclictest23-21ksoftirqd/122:47:131
319058799509,37cyclictest12-21ksoftirqd/021:30:010
3190587995023,13cyclictest3255216-21ssh23:14:080
3190587995022,13cyclictest3262259-21ssh23:29:130
3190587995022,13cyclictest3262259-21ssh23:29:130
319059499492,6cyclictest275-21dbus-daemon20:55:002
3190590994934,10cyclictest3467602-21systemd-journal22:35:031
3190587994936,7cyclictest309-21in:imuxsock20:10:010
3190587994934,11cyclictest311-21rs:main0
3190587994916,29cyclictest3196850-21latency_hist21:10:010
319058799491,34cyclictest151rcuc/022:06:230
3190600994834,10cyclictest364-21snmpd19:17:063
3190600994827,16cyclictest3190677-21wc19:10:013
319059099482,10cyclictest3193712-21unin-run20:10:011
3190587994834,8cyclictest3192087-21modprobe19:40:000
319059499474,7cyclictest3190637-21latency_hist19:10:022
319059499471,9cyclictest3467602-21systemd-journal19:59:172
3190590994735,10cyclictest23-21ksoftirqd/120:15:001
3190590994723,7cyclictest23-21ksoftirqd/122:10:261
3190587994728,6cyclictest12-21ksoftirqd/023:46:480
3190587994728,6cyclictest12-21ksoftirqd/023:46:480
3190587994713,8cyclictest12-21ksoftirqd/022:47:340
3190590994634,8cyclictest23-21ksoftirqd/100:00:021
3190587994621,20cyclictest16850irq/134-800000000:20:020
3190590994532,10cyclictest23-21ksoftirqd/122:20:021
3190587994535,6cyclictest12-21ksoftirqd/020:30:000
3190587994530,11cyclictest12-21ksoftirqd/020:00:020
3190590994432,9cyclictest3192195-21sed19:40:021
3190590994427,12cyclictest3267332-21systemctl23:40:001
3190590994427,12cyclictest3267332-21systemctl23:40:001
3190587994425,13cyclictest0-21swapper/019:30:000
3190587994424,14cyclictest275-21dbus-daemon20:15:010
3190590994325,14cyclictest275-21dbus-daemon19:35:011
3190590994323,14cyclictest1-21systemd19:45:001
319059099432,20cyclictest0-21swapper/119:10:401
319059099432,20cyclictest0-21swapper/119:10:401
319059099432,14cyclictest3195254-21systemctl20:40:001
3190587994313,6cyclictest131rcu_preempt20:20:010
3190590994230,9cyclictest3467602-21systemd-journal20:34:591
3190590994230,9cyclictest3220306-21systemctl22:00:011
3190590994227,12cyclictest3467602-21systemd-journal20:05:011
3190590994227,12cyclictest3467602-21systemd-journal20:05:011
319058799421,4cyclictest0-21swapper/019:47:060
3190587994210,29cyclictest3194502-21systemd-run20:25:010
319059099412,3cyclictest131rcu_preempt23:25:011
3190590994119,16cyclictest275-21dbus-daemon00:20:011
3190587994130,8cyclictest3467602-21systemd-journal19:44:020
319058799411,36cyclictest0-21swapper/020:55:010
3190590994028,10cyclictest23-21ksoftirqd/100:25:151
3190590994024,12cyclictest275-21dbus-daemon19:25:011
3190590994024,12cyclictest275-21dbus-daemon19:25:011
3190587994031,6cyclictest19350irq/34-mmc120:55:230
319059099399,6cyclictest131rcu_preempt19:29:541
3190590993926,10cyclictest23-21ksoftirqd/121:50:021
3190590993924,11cyclictest23-21ksoftirqd/123:51:131
3190587993929,7cyclictest12-21ksoftirqd/019:34:590
3190587993929,7cyclictest12-21ksoftirqd/019:25:020
3190587993929,7cyclictest12-21ksoftirqd/019:25:010
319058799391,7cyclictest3196528-21sshd21:00:550
319058799391,7cyclictest3190759-21sendmail-mta19:10:570
319058799391,7cyclictest3190759-21sendmail-mta19:10:560
3190587993910,5cyclictest131rcu_preempt20:35:010
3190590993823,12cyclictest23-21ksoftirqd/100:24:041
3190587993832,4cyclictest16950irq/224-800000019:52:560
3190590993724,11cyclictest23-21ksoftirqd/123:05:201
3190587993726,9cyclictest0-21swapper/020:50:020
319058799371,6cyclictest151rcuc/020:05:010
319058799371,6cyclictest151rcuc/020:05:010
3190590993623,8cyclictest3193167-21sh19:59:591
319059099361,5cyclictest0-21swapper/121:28:251
319059099359,5cyclictest131rcu_preempt19:45:591
3190590993527,5cyclictest3195524-21latency_hist20:45:031
3190590993526,7cyclictest3261931-21ssh23:28:221
3190590993526,7cyclictest3261931-21ssh23:28:221
3190590993523,9cyclictest23-21ksoftirqd/121:13:421
3190590993523,10cyclictest23-21ksoftirqd/120:30:011
3190590993224,5cyclictest23-21ksoftirqd/119:10:011
3190590993117,10cyclictest0-21swapper/121:05:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional