You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 06:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Mar 16, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
302213099853,12cyclictest3467602-21systemd-journal23:35:000
3022134997719,19cyclictest3049693-21ssh21:57:451
3022134997624,16cyclictest3081537-21sh23:09:201
302214099752,8cyclictest131rcu_preempt22:00:002
3022134997528,14cyclictest3085580-21sh23:18:141
3022134997432,12cyclictest3035309-21ssh21:25:201
3022134997223,16cyclictest3063941-21ssh22:29:581
3022134997220,16cyclictest3050852-21rm22:00:051
3022134997220,16cyclictest3050852-21rm22:00:051
3022130997228,13cyclictest3088951-21ssh23:25:230
3022134997130,16cyclictest3119019-21ssh00:33:161
3022134997130,16cyclictest3119019-21ssh00:33:161
3022134997126,19cyclictest3056499-21ssh22:13:011
3022134997126,19cyclictest3056499-21ssh22:13:011
3022134997119,15cyclictest3039111-21ssh21:34:131
3022130997138,6cyclictest16950irq/224-800000022:15:010
3022130997138,6cyclictest16950irq/224-800000022:15:000
3022134997029,13cyclictest3041659-21munin-run21:40:001
3022134997019,15cyclictest3022827-21cat19:20:021
3022134996931,13cyclictest3042398-21ssh21:41:131
3022134996931,13cyclictest3042398-21ssh21:41:131
3022134996918,23cyclictest3111656-21rm00:16:371
3022144996814,12cyclictest37-21ksoftirqd/300:25:023
3022134996727,15cyclictest3071244-21ssh22:45:501
3022134996723,7cyclictest3115595-21ssh00:25:141
3022134996721,15cyclictest3095169-21ssh23:39:511
3022134996717,13cyclictest3088800-21modprobe23:25:021
3022134996627,13cyclictest3029546-21ssh21:12:441
3022134996618,14cyclictest3088569-21munin-run23:25:001
3022134996519,15cyclictest3054923-21ssh22:09:401
3022144996416,12cyclictest37-21ksoftirqd/323:26:323
302214099645,47cyclictest3081935-21ssh23:10:022
3022134996418,13cyclictest3105440-21ssh00:02:401
3022134996418,13cyclictest3105440-21ssh00:02:391
3022144996319,9cyclictest37-21ksoftirqd/321:47:393
3022144996318,10cyclictest37-21ksoftirqd/319:20:003
3022134996326,13cyclictest3114808-21ssh00:23:461
302213499632,47cyclictest3071934-21kworker/u8:322:55:021
302213099636,14cyclictest3467602-21systemd-journal23:00:020
302213099636,14cyclictest3467602-21systemd-journal23:00:020
3022144996212,8cyclictest37-21ksoftirqd/320:45:023
3022134996221,6cyclictest3120315-21ssh00:35:491
3022134996218,15cyclictest3097406-21ssh23:44:451
3022144996122,18cyclictest37-21ksoftirqd/300:36:453
3022144996116,11cyclictest37-21ksoftirqd/322:54:403
3022134996144,14cyclictest3059024-21ssh22:18:481
3022134996142,15cyclictest3082995-21ssh23:12:231
302214499605,16cyclictest3097518-21munin-run23:45:013
3022140996040,15cyclictest1-21systemd23:50:002
3022134996046,11cyclictest3078839-21ssh23:03:091
3022134996041,15cyclictest3031068-21ssh21:15:551
3022134996040,16cyclictest3093001-21ssh23:34:571
3022134996025,11cyclictest150-21pr/ttyS219:46:591
3022130996029,27cyclictest3064638-21sh22:31:050
302213099602,51cyclictest3073020-21latency_hist22:50:020
3022130996014,28cyclictest16950irq/224-800000023:20:040
3022144995915,9cyclictest37-21ksoftirqd/323:55:433
3022144995915,11cyclictest37-21ksoftirqd/300:18:303
3022134995943,13cyclictest3068810-21ssh22:40:201
3022134995942,13cyclictest3100621-21ssh23:51:561
3022134995942,13cyclictest3100621-21ssh23:51:561
3022134995941,14cyclictest3066645-21ssh22:35:321
3022134995941,14cyclictest3066645-21ssh22:35:321
3022130995938,18cyclictest16850irq/134-800000021:30:000
3022144995817,7cyclictest37-21ksoftirqd/323:02:033
3022140995849,6cyclictest3066939-21kworker/2:123:23:152
3022134995843,12cyclictest3103235-21ssh23:57:531
3022134995841,15cyclictest3033182-21ssh21:20:371
3022134995840,14cyclictest3107552-21ssh00:07:301
3022134995840,14cyclictest3098378-21ssh23:46:441
3022134995840,14cyclictest3064600-21ssh22:31:001
302213099585,45cyclictest3046245-21grep21:50:010
3022130995828,14cyclictest3084595-21ssh23:15:460
302214499573,19cyclictest3088782-21ssh23:25:023
3022144995720,17cyclictest37-21ksoftirqd/323:36:513
3022140995715,27cyclictest3027232-21kworker/2:200:00:032
3022134995745,9cyclictest3059952-21ssh22:20:311
3022134995740,14cyclictest3109855-21ssh00:12:481
3022134995740,14cyclictest3075848-21ssh22:56:101
3022134995740,14cyclictest3075848-21ssh22:56:101
3022134995740,14cyclictest3047921-21ssh21:54:001
3022134995740,14cyclictest3047921-21ssh21:54:001
3022130995735,16cyclictest275-21dbus-daemon22:20:010
3022130995731,21cyclictest16850irq/134-800000022:25:010
3022130995730,21cyclictest3081837-21ssh23:10:010
3022130995729,12cyclictest3120980-21ssh00:37:310
3022130995724,29cyclictest3101868-21latency_hist23:55:030
3022130995724,29cyclictest3101868-21latency_hist23:55:020
3022144995616,9cyclictest37-21ksoftirqd/300:31:083
3022144995616,9cyclictest37-21ksoftirqd/300:31:083
3022144995615,9cyclictest37-21ksoftirqd/321:40:353
3022144995615,9cyclictest37-21ksoftirqd/321:40:353
3022134995643,10cyclictest3044176-21ssh21:45:111
3022144995538,13cyclictest3467602-21systemd-journal20:20:013
3022144995516,9cyclictest37-21ksoftirqd/323:10:103
3022144995511,9cyclictest3046751-21ssh21:51:023
3022144995511,9cyclictest3046751-21ssh21:51:023
3022140995546,6cyclictest3027232-21kworker/2:200:18:342
3022140995546,6cyclictest3022879-21kworker/2:022:32:472
3022140995540,11cyclictest1-21systemd20:35:012
3022134995532,18cyclictest3022857-21cron19:25:001
302213099553,35cyclictest3077599-21ssh23:00:030
3022130995527,13cyclictest3099215-21ssh23:48:550
3022130995527,12cyclictest3047931-21ssh21:54:020
3022130995527,12cyclictest3047931-21ssh21:54:010
3022130995526,12cyclictest3033198-21ssh21:20:390
3022130995519,8cyclictest12-21ksoftirqd/000:01:310
3022130995519,8cyclictest12-21ksoftirqd/000:01:300
302214499547,15cyclictest3057453-21cat22:15:023
302214499547,15cyclictest3057453-21cat22:15:023
302214499545,16cyclictest3092450-21ssh23:33:393
302214499544,15cyclictest3027664-21sed20:55:013
302214499544,13cyclictest3109045-21ssh00:10:423
3022144995439,12cyclictest3077353-21systemctl23:00:013
3022144995439,12cyclictest3077353-21systemctl23:00:003
3022144995412,10cyclictest37-21ksoftirqd/320:25:023
3022140995434,15cyclictest1-21systemd00:40:002
3022130995443,7cyclictest16850irq/134-800000023:59:590
3022130995439,10cyclictest309-21in:imuxsock00:30:000
3022130995436,14cyclictest1-21systemd00:20:000
3022130995435,13cyclictest1-21systemd20:30:010
3022144995310,7cyclictest37-21ksoftirqd/300:28:443
3022144995310,12cyclictest37-21ksoftirqd/321:20:023
3022140995344,6cyclictest3027232-21kworker/2:222:47:312
3022140995343,6cyclictest3027232-21kworker/2:200:27:312
3022134995340,9cyclictest48-21kcompactd019:11:211
3022134995339,11cyclictest3023341-21sshd19:30:411
3022134995339,10cyclictest3023135-21munin-run19:30:021
3022134995336,14cyclictest364-21snmpd21:05:431
3022134995334,15cyclictest1-21systemd20:30:011
3022134995325,23cyclictest3027059-21date20:45:011
3022144995236,12cyclictest1-21systemd20:50:013
3022144995236,12cyclictest1-21systemd20:50:013
3022144995232,15cyclictest275-21dbus-daemon23:20:003
302214099526,6cyclictest131rcu_preempt00:35:002
302214099526,6cyclictest131rcu_preempt00:35:002
3022140995244,5cyclictest3027232-21kworker/2:222:06:322
3022140995244,5cyclictest3022879-21kworker/2:021:35:062
3022140995243,6cyclictest3022879-21kworker/2:022:15:052
3022140995243,6cyclictest3022879-21kworker/2:021:43:122
3022140995243,6cyclictest3022879-21kworker/2:021:43:122
3022134995238,10cyclictest3026454-21sshd20:31:251
3022134995235,12cyclictest1-21systemd20:05:001
3022134995227,20cyclictest3023672-21cat19:40:011
3022134995224,23cyclictest3023907-21cut19:45:001
3022134995224,23cyclictest3023907-21cut19:45:001
3022134995223,24cyclictest275-21dbus-daemon21:00:001
3022130995242,5cyclictest16850irq/134-800000022:55:010
3022130995225,12cyclictest3049440-21ssh21:57:100
3022130995220,29cyclictest12-21ksoftirqd/023:37:480
3022140995143,5cyclictest3066939-21kworker/2:122:39:512
3022140995143,5cyclictest3066939-21kworker/2:122:39:512
3022140995143,5cyclictest3066939-21kworker/2:100:10:432
3022140995143,5cyclictest3027232-21kworker/2:223:54:492
3022140995143,5cyclictest3027232-21kworker/2:223:54:492
3022140995143,5cyclictest3027232-21kworker/2:222:26:412
3022140995143,5cyclictest3022879-21kworker/2:022:04:092
3022140995143,5cyclictest3022879-21kworker/2:022:04:082
3022140995143,5cyclictest3022879-21kworker/2:021:13:492
3022140995142,6cyclictest3027232-21kworker/2:222:10:302
3022140995142,6cyclictest3027232-21kworker/2:222:10:302
3022140995142,6cyclictest3022879-21kworker/2:021:50:082
3022140995142,6cyclictest3022879-21kworker/2:021:50:082
3022140995142,6cyclictest3021309-21kworker/2:319:25:022
3022134995137,10cyclictest3467602-21systemd-journal19:57:321
302213099513,42cyclictest3070780-21grep22:45:010
3022130995133,13cyclictest275-21dbus-daemon20:15:010
3022130995122,27cyclictest19350irq/34-mmc122:30:010
302214499508,10cyclictest37-21ksoftirqd/323:47:263
302214499505,14cyclictest3071645-21ssh22:46:473
3022144995011,7cyclictest37-21ksoftirqd/322:05:023
3022144995011,7cyclictest37-21ksoftirqd/322:05:013
3022140995043,4cyclictest3066939-21kworker/2:123:10:042
3022140995042,5cyclictest3066939-21kworker/2:122:50:372
3022140995042,5cyclictest3027232-21kworker/2:222:58:362
3022140995042,5cyclictest3027232-21kworker/2:222:58:352
3022140995042,5cyclictest3027232-21kworker/2:221:25:082
3022140995042,5cyclictest3022879-21kworker/2:022:23:432
3022140995042,5cyclictest3022879-21kworker/2:021:31:462
3022140995041,6cyclictest3027232-21kworker/2:221:22:222
3022140995041,5cyclictest3027232-21kworker/2:200:20:092
3022140995040,6cyclictest3022879-21kworker/2:020:42:302
3022134995038,9cyclictest3024374-21sshd19:50:441
3022134995036,11cyclictest3025171-21sshd20:09:251
3022134995036,10cyclictest3028048-21munin-run21:05:021
3022134995036,10cyclictest3028048-21munin-run21:05:011
3022134995036,10cyclictest3025982-21munin-run20:25:001
3022130995030,16cyclictest3082888-21ssh23:12:070
3022130995026,19cyclictest3041632-21ssh21:40:000
3022130995024,11cyclictest3032211-21ssh21:18:460
302214499491,23cyclictest3022629-21kworker/3:020:15:023
3022140994941,5cyclictest3066939-21kworker/2:123:35:092
3022140994941,5cyclictest3066939-21kworker/2:123:34:432
3022140994941,5cyclictest3066939-21kworker/2:123:15:372
3022140994941,5cyclictest3066939-21kworker/2:100:09:352
3022140994941,5cyclictest3022879-21kworker/2:021:48:462
3022140994941,5cyclictest3022879-21kworker/2:021:15:112
3022140994940,6cyclictest3066939-21kworker/2:123:00:112
3022140994940,6cyclictest3021309-21kworker/2:319:19:302
302213099495,9cyclictest0-21swapper/022:05:000
302213099495,9cyclictest0-21swapper/022:05:000
3022130994942,4cyclictest16850irq/134-800000021:15:000
3022130994938,7cyclictest16850irq/134-800000021:35:000
3022130994934,9cyclictest309-21in:imuxsock20:10:010
302213099492,3cyclictest131rcu_preempt00:10:000
3022144994835,10cyclictest1-21systemd22:20:003
302214499481,33cyclictest3037308-21ssh21:30:023
3022140994840,5cyclictest3066939-21kworker/2:123:40:492
3022140994840,5cyclictest3066939-21kworker/2:123:25:132
3022140994840,5cyclictest3066939-21kworker/2:100:01:072
3022140994840,5cyclictest3066939-21kworker/2:100:01:072
3022140994840,5cyclictest3027232-21kworker/2:222:42:352
3022140994840,5cyclictest3022879-21kworker/2:021:00:022
3022140994840,5cyclictest3021309-21kworker/2:319:37:452
3022140994840,5cyclictest3021309-21kworker/2:319:14:462
3022134994835,9cyclictest364-21snmpd20:51:151
3022134994833,10cyclictest1-21systemd20:15:011
302213099482,37cyclictest0-21swapper/020:20:020
302214499475,8cyclictest37-21ksoftirqd/322:31:253
3022144994712,9cyclictest37-21ksoftirqd/321:38:543
3022140994740,4cyclictest3021309-21kworker/2:319:45:052
3022140994739,5cyclictest3022879-21kworker/2:020:48:012
3022140994739,5cyclictest3022879-21kworker/2:020:48:002
3022140994739,5cyclictest3021309-21kworker/2:319:56:332
3022140994739,5cyclictest3021309-21kworker/2:319:52:302
3022140994739,5cyclictest3021309-21kworker/2:319:28:202
3022130994730,13cyclictest3053486-21ssh22:06:040
3022130994729,4cyclictest16850irq/134-800000000:11:250
3022130994726,16cyclictest275-21dbus-daemon21:45:010
3022130994726,16cyclictest275-21dbus-daemon21:45:010
3022130994725,15cyclictest275-21dbus-daemon20:40:000
3022144994633,10cyclictest275-21dbus-daemon00:10:003
3022140994639,4cyclictest3021309-21kworker/2:319:42:382
3022140994639,4cyclictest3021309-21kworker/2:319:42:382
3022140994638,5cyclictest3022879-21kworker/2:020:52:272
3022140994638,5cyclictest3021309-21kworker/2:320:05:312
3022140994638,5cyclictest3021309-21kworker/2:320:01:172
3022140994637,5cyclictest3021309-21kworker/2:320:28:162
3022140994635,7cyclictest30-21ksoftirqd/220:40:022
3022140994635,7cyclictest3021309-21kworker/2:320:24:332
3022134994631,11cyclictest3025709-21/usr/sbin/munin20:20:021
3022130994636,5cyclictest16950irq/224-800000019:20:010
3022130994628,15cyclictest3068381-21ssh22:39:560
3022130994628,15cyclictest3068381-21ssh22:39:550
302214499458,4cyclictest131rcu_preempt23:07:213
3022144994531,11cyclictest37-21ksoftirqd/321:30:493
302214499451,10cyclictest37-21ksoftirqd/323:51:123
302214499451,10cyclictest37-21ksoftirqd/323:51:113
3022140994536,6cyclictest3021309-21kworker/2:320:18:582
3022134994532,9cyclictest364-21snmpd20:38:111
3022134994529,13cyclictest364-21snmpd20:47:321
3022134994529,13cyclictest364-21snmpd20:47:321
3022130994538,4cyclictest16950irq/224-800000021:00:000
3022130994536,6cyclictest16950irq/224-800000020:00:000
3022130994536,5cyclictest16950irq/224-800000020:25:010
302213099452,14cyclictest3467602-21systemd-journal00:31:040
302213099452,14cyclictest3467602-21systemd-journal00:31:030
302214499449,7cyclictest37-21ksoftirqd/300:04:213
302214499449,7cyclictest37-21ksoftirqd/300:04:213
302214499448,6cyclictest0-21swapper/322:28:013
3022144994430,11cyclictest3024681-21tr20:00:003
302213099442,35cyclictest3028309-21cut21:10:000
3022130994421,20cyclictest3027261-21latency_hist20:50:020
3022130994421,20cyclictest3027261-21latency_hist20:50:020
3022130994417,12cyclictest3114102-21ssh00:22:060
302213099441,38cyclictest3095799-21ssh23:40:450
302214499435,7cyclictest37-21ksoftirqd/322:40:503
3022144994327,12cyclictest3023869-21sshd19:40:253
3022144994327,12cyclictest3023869-21sshd19:40:253
3022144994326,13cyclictest3029566-21ssh21:12:463
3022144994325,14cyclictest3053978-21ssh22:07:153
3022140994335,5cyclictest3021309-21kworker/2:320:13:302
3022130994336,4cyclictest16950irq/224-800000019:55:010
302213099432,4cyclictest0-21swapper/019:30:050
302213099432,15cyclictest3023075-21latency_hist19:25:020
302214499425,10cyclictest37-21ksoftirqd/322:24:043
3022144994225,13cyclictest3033242-21ssh21:20:463
3022144994224,14cyclictest3066532-21ssh22:35:133
3022144994224,14cyclictest3066532-21ssh22:35:133
3022140994236,4cyclictest3022879-21kworker/2:021:01:312
3022140994236,4cyclictest3022879-21kworker/2:021:01:302
3022140994235,5cyclictest3022879-21kworker/2:021:08:152
3022140994234,5cyclictest3021309-21kworker/2:319:31:442
3022130994224,13cyclictest1-21systemd19:50:000
302213099422,33cyclictest3027518-21munin-run20:55:000
3022130994212,20cyclictest3022141-21latency_hist19:10:020
3022144994126,12cyclictest3048919-21ssh21:55:493
3022144994124,12cyclictest3023385-21munin-run19:35:013
3022130994024,12cyclictest12-21ksoftirqd/019:30:010
302213099401,4cyclictest0-21swapper/019:35:090
3022144993929,8cyclictest37-21ksoftirqd/321:00:023
3022144993925,10cyclictest364-21snmpd20:00:313
3022130993925,9cyclictest12-21ksoftirqd/019:45:010
3022130993925,9cyclictest12-21ksoftirqd/019:45:000
3022144993825,9cyclictest3467602-21systemd-journal19:24:273
3022144993823,10cyclictest48-21kcompactd019:12:283
3022130993829,7cyclictest12-21ksoftirqd/019:15:030
3022144993725,10cyclictest37-21ksoftirqd/320:10:013
3022144993724,9cyclictest48-21kcompactd021:07:523
3022144993724,10cyclictest3025946-21sshd20:20:253
3022144993720,13cyclictest275-21dbus-daemon20:35:013
3022140993713,6cyclictest131rcu_preempt19:10:022
3022134993723,11cyclictest3022136-21latency_hist19:10:021
302214499363,7cyclictest37-21ksoftirqd/320:35:033
3022144993624,9cyclictest37-21ksoftirqd/319:30:023
3022144993623,9cyclictest3023603-21sshd19:38:253
3022130993624,8cyclictest1-21systemd20:41:250
3022144993523,9cyclictest48-21kcompactd021:02:193
3022144993523,9cyclictest48-21kcompactd021:02:193
3022130993518,14cyclictest3028029-21sshd21:03:250
3022130993518,14cyclictest3028029-21sshd21:03:250
302213099351,30cyclictest0-21swapper/020:31:590
3022130993419,11cyclictest16850irq/134-800000020:00:470
3022144993320,9cyclictest364-21snmpd19:50:553
3022144993210,17cyclictest3022257-21latency_hist19:10:023
3022144993118,9cyclictest364-21snmpd19:47:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional