You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-31 - 04:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Tue Mar 31, 2026 00:43:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1185093995746,7cyclictest3424687-21cups-browsed23:05:360
1185093995546,5cyclictest1275843-21ntpq22:15:320
1185093995546,5cyclictest1275843-21ntpq22:15:320
1185093995530,16cyclictest12350irq/32-eth023:10:310
1185095995443,7cyclictest0-21swapper/200:00:012
1185094995344,6cyclictest1221072-21idleruntime-cro21:10:001
1185095995040,6cyclictest1266540-21latency_hist22:05:022
1185094995043,3cyclictest0-21swapper/123:05:361
1185094995042,5cyclictest1054-21mta-sts-daemon19:40:301
1185093995035,10cyclictest151ktimers/019:50:290
1185095994734,10cyclictest1-21systemd23:30:012
1185094994739,5cyclictest1368367-21latency_hist00:10:021
1185094994736,7cyclictest1320524-21kernelversion23:10:251
1185093994731,10cyclictest151ktimers/000:25:330
1185093994723,18cyclictest151ktimers/022:00:280
1185094994638,5cyclictest1221071-21diskmemload23:55:291
1185094994636,5cyclictest0-21swapper/122:00:291
1185093994640,3cyclictest1055-21mta-sts-daemon20:55:010
1185093994631,12cyclictest12350irq/32-eth021:50:330
1185095994535,6cyclictest1284303-21mailstats22:25:372
1185095994535,6cyclictest1284303-21mailstats22:25:372
118509599451,34cyclictest1329064-21rm23:20:292
1185094994438,4cyclictest0-21swapper/123:50:011
1185094994435,6cyclictest1341508-21find23:35:371
1185094994435,6cyclictest1341508-21find23:35:371
118509499441,7cyclictest171rcu_preempt23:20:311
1185093994435,5cyclictest151ktimers/020:15:310
1185093994435,5cyclictest151ktimers/020:15:310
1185093994433,7cyclictest12350irq/32-eth019:35:330
1185095994334,7cyclictest0-21swapper/222:25:012
1185095994334,7cyclictest0-21swapper/222:25:012
1185094994334,5cyclictest1392471-21munin-plugin-st00:40:001
118509499433,32cyclictest1377451-21ssh00:20:271
118509499433,32cyclictest1377451-21ssh00:20:271
118509499432,34cyclictest1308665-21ssh22:55:341
1185093994336,5cyclictest1-21systemd21:35:020
1185093994235,4cyclictest421-21dbus-daemon21:05:010
1185095994132,5cyclictest1-21systemd00:10:022
1185095994131,6cyclictest420-21avahi-daemon22:07:472
1185095994131,3cyclictest0-21swapper/223:05:302
1185094994132,5cyclictest1296935-21unixbench_singl22:40:421
1185094994132,5cyclictest1229224-21idleruntime-cro21:20:011
1185094994132,4cyclictest0-21swapper/123:30:311
1185093994133,5cyclictest1344135-21sendmail23:40:010
1185093994133,5cyclictest1344135-21sendmail23:40:010
1185093994132,5cyclictest1258794-21fschecks_count21:55:230
118509599404,30cyclictest1341301-21ssh23:35:312
118509599404,30cyclictest1341301-21ssh23:35:312
1185095994032,4cyclictest0-21swapper/223:10:332
1185095994028,8cyclictest1369986-21timerwakeupswit00:10:422
118509599402,33cyclictest1361329-21ssh00:00:272
118509599401,33cyclictest1195041-21sshd19:40:322
1185094994034,3cyclictest0-21swapper/120:05:011
1185094994032,5cyclictest1233386-21latency_hist21:25:021
1185094994030,7cyclictest1219563-21cron21:05:011
1185094994030,4cyclictest0-21swapper/100:20:011
1185094994030,4cyclictest0-21swapper/100:20:011
118509499402,8cyclictest171rcu_preempt22:10:331
1185093994034,4cyclictest1311531-21ssh23:00:010
1185093994033,4cyclictest1292094-21needreboot22:35:270
1185093994026,3cyclictest0-21swapper/023:00:280
1185093994025,5cyclictest421-21dbus-daemon23:30:010
1185093994014,19cyclictest12350irq/32-eth022:30:320
118509399400,9cyclictest14-21ksoftirqd/020:20:280
1185095993931,5cyclictest1204586-21systemd-run20:15:012
1185095993931,5cyclictest1204586-21systemd-run20:15:012
1185095993927,9cyclictest1251319-21switchtime21:45:392
118509599392,32cyclictest1337129-21ssh23:30:272
118509499393,28cyclictest1234514-21ssh21:25:311
1185094993932,4cyclictest0-21swapper/120:40:011
1185094993930,4cyclictest0-21swapper/122:20:011
1185094993930,4cyclictest0-21swapper/122:20:011
118509499392,30cyclictest1250892-21ssh21:45:271
1185093993933,4cyclictest1245676-21idleruntime-cro21:40:010
1185093993933,4cyclictest1229236-21idleruntime-cro21:20:010
1185093993932,4cyclictest1376367-21sendmail00:20:010
1185093993932,4cyclictest1376367-21sendmail00:20:010
1185093993930,6cyclictest3424687-21cups-browsed22:47:470
1185093993929,4cyclictest0-21swapper/023:15:310
1185095993830,5cyclictest1249819-21cron21:45:012
1185095993830,5cyclictest1249819-21cron21:45:012
1185095993821,9cyclictest261-21systemd-journal20:25:332
1185094993832,3cyclictest0-21swapper/122:10:011
1185094993830,5cyclictest1188120-21systemd-run19:20:011
1185094993828,8cyclictest421-21dbus-daemon19:50:011
1185094993827,7cyclictest420-21avahi-daemon21:34:141
1185094993825,9cyclictest0-21swapper/100:25:331
1185093993830,5cyclictest1-21systemd19:10:310
1185093993826,6cyclictest12450irq/33-eth023:55:310
1185095993730,3cyclictest0-21swapper/221:25:292
1185095993729,5cyclictest1278793-21munin-run22:20:012
1185095993729,5cyclictest1278793-21munin-run22:20:012
1185095993729,5cyclictest1221071-21diskmemload23:50:282
1185095993729,5cyclictest1-21systemd19:15:012
1185095993729,5cyclictest1201604-21systemd-run20:05:012
1185094993728,6cyclictest0-21swapper/121:10:341
1185094993727,5cyclictest0-21swapper/100:00:271
118509499371,28cyclictest1292341-21ssh22:35:351
118509399377,21cyclictest261-21systemd-journal20:40:340
1185093993730,4cyclictest0-21swapper/021:45:340
1185093993719,15cyclictest1257306-21kworker/u8:1+rpciod22:20:330
1185093993719,15cyclictest1257306-21kworker/u8:1+rpciod22:20:330
1185093993719,12cyclictest1349328-21sh23:45:310
1185095993628,5cyclictest1296818-21timerwakeupswit22:40:422
1185095993628,5cyclictest1212595-21fschecks_count20:40:222
1185095993628,4cyclictest0-21swapper/200:25:412
1185095993626,7cyclictest3424680-21cupsd22:32:232
1185094993629,3cyclictest0-21swapper/123:20:011
1185094993628,5cyclictest1315638-21munin-plugin-st23:05:011
1185093993622,9cyclictest12350irq/32-eth023:50:270
118509399362,26cyclictest1296397-21ssh22:40:340
1185093993616,11cyclictest1230359-21ssh21:20:290
1185095993528,3cyclictest0-21swapper/222:45:342
1185095993527,5cyclictest1229226-21munin-run21:20:012
1185095993526,6cyclictest1305040-21grep22:50:412
1185094993528,3cyclictest0-21swapper/120:40:341
1185094993527,5cyclictest3424687-21cups-browsed20:55:201
1185094993527,5cyclictest1279814-21needreboot22:20:241
1185094993527,5cyclictest1279814-21needreboot22:20:241
1185094993527,5cyclictest1191605-21fschecks_time19:30:221
1185094993523,9cyclictest1385286-21latency00:30:241
1185093993528,4cyclictest0-21swapper/022:05:330
1185093993527,5cyclictest1336819-21sh23:30:230
1185093993524,9cyclictest1388810-21/usr/sbin/munin00:35:250
1185095993427,4cyclictest1271323-21if_eth022:10:252
1185095993426,5cyclictest1392536-21latency_hist00:40:022
1185095993426,5cyclictest0-21swapper/219:20:002
1185095993426,3cyclictest0-21swapper/221:20:352
118509599342,23cyclictest1325040-21ssh23:15:342
118509399343,25cyclictest1329020-21ssh23:20:280
1185093993425,6cyclictest1222388-21sh21:10:360
1185093993422,8cyclictest0-21swapper/000:05:310
1185095993327,3cyclictest0-21swapper/223:45:002
1185095993327,3cyclictest0-21swapper/221:30:352
1185095993326,4cyclictest1245672-21ls21:40:012
1185095993325,5cyclictest1192599-21systemd-run19:35:012
1185095993324,5cyclictest1222552-21switchtime21:10:402
1185094993326,4cyclictest1262212-21cron22:00:001
1185094993326,4cyclictest1194073-21munin-run19:40:011
118509499332,22cyclictest1333099-21ssh23:25:301
118509399334,9cyclictest14-21ksoftirqd/020:00:320
1185093993327,3cyclictest1054-21mta-sts-daemon00:00:290
1185093993324,7cyclictest1196449-21ntp_kernel_pll_19:45:290
1185093993324,6cyclictest420-21avahi-daemon00:20:320
1185093993324,6cyclictest420-21avahi-daemon00:20:320
118509599324,19cyclictest1349220-21ssh23:45:272
1185095993226,3cyclictest0-21swapper/220:40:012
1185095993225,4cyclictest1054-21mta-sts-daemon21:55:452
1185095993219,9cyclictest351ktimers/220:30:332
1185094993224,5cyclictest1370003-21unixbench-2d00:10:421
1185094993224,5cyclictest1204552-21cron20:15:001
1185094993224,5cyclictest1204552-21cron20:15:001
118509499322,25cyclictest1255191-21ssh21:50:321
1185094993221,7cyclictest1188906-21needreboot19:20:251
118509499320,7cyclictest171rcu_preempt20:45:291
118509399327,4cyclictest14-21ksoftirqd/022:50:280
118509399323,22cyclictest1385579-21rm00:30:310
1185093993224,5cyclictest1209003-21unixbench_singl20:25:420
1185093993223,6cyclictest3424680-21cupsd22:25:130
1185093993223,6cyclictest3424680-21cupsd22:25:130
118509399321,26cyclictest1234429-21sh21:25:270
1185095993123,5cyclictest1207475-21unixbench_multi20:20:382
118509599312,23cyclictest1255048-21ssh21:50:272
1185095993121,7cyclictest0-21swapper/223:00:012
118509499314,19cyclictest1288237-21ssh22:30:341
1185094993124,4cyclictest1246529-21kernelversion21:40:241
1185094993124,4cyclictest1246529-21kernelversion21:40:241
1185094993122,5cyclictest1300967-21unixbench_multi22:45:401
118509499311,23cyclictest1353369-21ssh23:50:311
1185093993124,5cyclictest1-21systemd22:15:020
118509399311,25cyclictest1345241-21ssh23:40:290
118509399311,15cyclictest1278841-21kworker/u8:14+rpciod00:10:350
1185095993023,3cyclictest0-21swapper/200:25:012
1185095993023,3cyclictest0-21swapper/200:25:012
1185095993021,6cyclictest1186554-21kworker/u8:3-xprtiod20:05:212
1185095993021,4cyclictest0-21swapper/219:55:312
1185095993020,8cyclictest1373300-21missed_timers00:15:252
1185095993020,8cyclictest1373300-21missed_timers00:15:252
118509599301,6cyclictest171rcu_preempt23:00:262
118509499303,21cyclictest1345227-21ssh23:40:281
118509499303,20cyclictest1242735-21ssh21:35:311
1185094993023,4cyclictest0-21swapper/122:50:221
1185094993022,4cyclictest1181096-21kworker/u8:5-xprtiod20:00:001
1185094993011,14cyclictest271ktimers/119:10:291
118509399309,14cyclictest1246858-21sh21:40:310
118509399309,14cyclictest1246858-21sh21:40:310
1185093993024,4cyclictest1195150-21sendmail_mailqu19:40:350
1185093993023,5cyclictest421-21dbus-daemon20:40:010
1185093993021,7cyclictest0-21swapper/021:10:010
1185095992921,5cyclictest518-21rs:main2
1185095992921,5cyclictest1219572-21idleruntime-cro21:05:012
1185095992918,8cyclictest1292713-21timerwakeupswit22:35:412
118509499294,20cyclictest1051-21mta-sts-daemon20:15:321
118509499294,20cyclictest1051-21mta-sts-daemon20:15:321
118509499292,18cyclictest1284001-21ssh22:25:281
118509499292,18cyclictest1284001-21ssh22:25:281
1185094992921,5cyclictest1207482-21unixbench_multi20:20:381
118509399298,11cyclictest14-21ksoftirqd/019:15:320
118509399296,17cyclictest1055-21mta-sts-daemon19:55:330
118509399295,5cyclictest14-21ksoftirqd/019:30:400
1185095992821,5cyclictest1196094-21fschecks_count19:45:222
1185095992821,4cyclictest0-21swapper/220:20:002
1185095992821,4cyclictest0-21swapper/220:20:002
1185095992820,5cyclictest1198560-21cron19:55:012
118509599281,21cyclictest3424687-21cups-browsed19:25:272
1185094992821,4cyclictest1188122-21kworker/u8:0-xprtiod19:50:181
118509399287,5cyclictest14-21ksoftirqd/020:55:270
1185093992814,5cyclictest171rcu_preempt20:45:270
1185093992719,3cyclictest0-21swapper/020:05:290
118509599264,18cyclictest1194080-21awk19:40:012
1185095992621,2cyclictest0-21swapper/220:55:012
1185095992619,4cyclictest0-21swapper/221:05:342
1185094992616,8cyclictest1210310-21timerandwakeup20:30:391
1185094992614,7cyclictest1202108-21grep20:05:211
118509399266,4cyclictest14-21ksoftirqd/020:30:290
1185093992618,2cyclictest171rcu_preempt19:26:310
1185093992614,3cyclictest171rcu_preempt19:20:240
1185093992610,10cyclictest151ktimers/020:10:280
1185093992610,10cyclictest151ktimers/020:10:280
1185095992519,3cyclictest0-21swapper/220:55:332
1185095992414,7cyclictest1188641-21idleruntime19:20:212
1185094992419,3cyclictest1055-21mta-sts-daemon20:53:411
1185094992418,3cyclictest0-21swapper/119:25:391
118509399244,6cyclictest1185207-21cut19:10:030
118509399244,6cyclictest1185207-21cut19:10:030
118509499232,15cyclictest1208598-21ps20:25:371
1185095992215,4cyclictest1055-21mta-sts-daemon19:10:022
1185095992215,4cyclictest1055-21mta-sts-daemon19:10:022
1185095992214,4cyclictest0-21swapper/220:48:332
118509499218,9cyclictest421-21dbus-daemon19:10:021
118509499218,9cyclictest421-21dbus-daemon19:10:021
118509699203,8cyclictest0-21swapper/320:55:313
118509699192,7cyclictest0-21swapper/319:45:293
118509699181,8cyclictest0-21swapper/323:30:283
118509699174,6cyclictest0-21swapper/322:30:323
118509699172,8cyclictest0-21swapper/323:40:293
118509699172,8cyclictest0-21swapper/323:00:333
118509699172,8cyclictest0-21swapper/320:10:313
118509699172,8cyclictest0-21swapper/320:10:313
118509699172,6cyclictest0-21swapper/320:30:303
118509699172,6cyclictest0-21swapper/320:00:263
118509699163,7cyclictest0-21swapper/323:05:343
118509699163,7cyclictest0-21swapper/322:40:313
118509699163,7cyclictest0-21swapper/321:35:363
118509699163,6cyclictest0-21swapper/300:05:343
118509699162,8cyclictest0-21swapper/323:10:283
118509699161,8cyclictest0-21swapper/321:55:303
118509699161,7cyclictest0-21swapper/319:10:273
118509699154,6cyclictest0-21swapper/322:10:333
118509699153,7cyclictest0-21swapper/323:55:343
118509699153,7cyclictest0-21swapper/322:35:343
118509699153,4cyclictest0-21swapper/320:45:293
118509699152,8cyclictest0-21swapper/322:05:313
118509699152,8cyclictest0-21swapper/321:15:193
118509699152,7cyclictest0-21swapper/322:15:273
118509699152,7cyclictest0-21swapper/322:15:273
118509699151,9cyclictest0-21swapper/322:25:423
118509699151,9cyclictest0-21swapper/322:25:423
118509699151,8cyclictest0-21swapper/323:45:323
118509699151,7cyclictest0-21swapper/322:00:193
118509699151,7cyclictest0-21swapper/321:30:303
118509699151,6cyclictest0-21swapper/321:20:273
118509699142,7cyclictest0-21swapper/300:35:203
118509699142,6cyclictest0-21swapper/319:40:013
118509699141,8cyclictest0-21swapper/322:50:263
118509699141,7cyclictest0-21swapper/323:25:343
118509699141,7cyclictest0-21swapper/322:20:293
118509699141,7cyclictest0-21swapper/322:20:293
118509699141,7cyclictest0-21swapper/321:40:283
118509699141,7cyclictest0-21swapper/321:40:283
118509699133,6cyclictest0-21swapper/300:30:303
118509699132,7cyclictest0-21swapper/300:25:303
118509699132,6cyclictest0-21swapper/300:15:273
118509699132,6cyclictest0-21swapper/300:15:273
118509699132,6cyclictest0-21swapper/300:00:253
118509699131,8cyclictest0-21swapper/323:35:203
118509699131,8cyclictest0-21swapper/323:35:203
118509699131,7cyclictest0-21swapper/323:50:293
118509699131,7cyclictest0-21swapper/322:45:263
118509699131,7cyclictest0-21swapper/300:20:273
118509699131,7cyclictest0-21swapper/300:20:273
118509699131,6cyclictest0-21swapper/320:25:323
118509699131,3cyclictest0-21swapper/319:55:333
118509699125,5cyclictest0-21swapper/322:55:293
118509699123,6cyclictest0-21swapper/321:50:213
118509699122,6cyclictest0-21swapper/321:25:223
118509699122,6cyclictest0-21swapper/321:10:203
118509699122,5cyclictest0-21swapper/320:05:283
118509699121,7cyclictest0-21swapper/323:20:283
118509699121,7cyclictest0-21swapper/323:15:403
118509699121,7cyclictest0-21swapper/321:45:263
118509699121,7cyclictest0-21swapper/300:10:323
118509699121,6cyclictest0-21swapper/319:40:333
118509699121,5cyclictest0-21swapper/321:00:283
118509699121,5cyclictest0-21swapper/320:20:263
118509699120,5cyclictest0-21swapper/320:15:263
118509699120,5cyclictest0-21swapper/320:15:263
118509699112,5cyclictest0-21swapper/321:05:293
118509699111,6cyclictest0-21swapper/320:40:183
118509699111,6cyclictest0-21swapper/320:40:013
118509699111,6cyclictest0-21swapper/319:25:273
118509699111,5cyclictest0-21swapper/319:20:183
118509699101,6cyclictest0-21swapper/319:35:013
11850969991,5cyclictest0-21swapper/319:50:233
11850969991,5cyclictest0-21swapper/319:15:213
11850969983,4cyclictest0-21swapper/320:50:283
11850969981,5cyclictest0-21swapper/319:10:023
11850969981,5cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional