You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 09:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Sun Apr 19, 2026 00:43:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2912769997761,5cyclictest439-21systemd-logind22:50:010
291276999573,49cyclictest1128531-21cups-browsed23:40:490
2912770995544,7cyclictest3091888-21systemctl00:20:011
2912769995448,4cyclictest2947251-21cron21:05:010
2912770995241,7cyclictest3078577-21/usr/sbin/munin00:00:531
2912770995241,7cyclictest3078577-21/usr/sbin/munin00:00:531
2912769995243,5cyclictest151ktimers/019:15:370
2912769995145,4cyclictest2945764-21sendmail21:00:010
2912771995040,8cyclictest3022269-21latency22:45:342
2912770995042,5cyclictest3013610-21cut22:35:011
2912770995042,5cyclictest3013610-21cut22:35:011
2912770995040,7cyclictest3017440-21/usr/sbin/munin22:40:011
2912771994939,6cyclictest421-21dbus-daemon00:40:012
2912771994939,6cyclictest421-21dbus-daemon00:40:012
2912770994941,5cyclictest3088075-21latency_hist00:15:001
2912769994840,5cyclictest151ktimers/020:05:400
2912771994537,5cyclictest2971093-21systemd-run21:40:012
2912771994537,5cyclictest1056-21mta-sts-daemon21:53:212
2912771994536,5cyclictest1-21systemd00:30:012
2912769994539,4cyclictest0-21swapper/022:25:010
2912769994530,12cyclictest3011476-21tail22:30:490
2912769994530,12cyclictest3011476-21tail22:30:490
2912771994435,6cyclictest421-21dbus-daemon22:25:022
2912771994432,7cyclictest3076692-21cups-browsed00:00:532
2912771994432,7cyclictest3076692-21cups-browsed00:00:532
2912769994336,4cyclictest1-21systemd23:05:020
2912769994331,9cyclictest2969038-21head21:35:500
291277199424,29cyclictest3051825-21ssh23:25:412
2912770994235,3cyclictest0-21swapper/123:10:391
2912770994234,5cyclictest3082303-21head00:05:501
2912770994233,4cyclictest0-21swapper/122:50:431
2912769994236,4cyclictest3088834-21fschecks_time00:15:310
2912769994233,4cyclictest0-21swapper/021:25:010
2912769994231,5cyclictest0-21swapper/022:55:430
291277199413,29cyclictest3030106-21ssh22:55:382
2912771994132,6cyclictest1128527-21cupsd23:40:272
2912771994132,5cyclictest421-21dbus-daemon22:35:022
2912771994132,5cyclictest421-21dbus-daemon22:35:022
2912770994132,6cyclictest421-21dbus-daemon22:30:011
2912769994125,12cyclictest0-21swapper/000:25:410
2912771994033,3cyclictest0-21swapper/222:50:322
2912770994032,5cyclictest3021314-21cron22:45:001
2912770994032,5cyclictest2933786-21systemd-run20:20:011
2912770994032,5cyclictest2930771-21systemd-run20:10:011
2912770994030,6cyclictest2948743-21cron21:10:011
2912770994028,6cyclictest0-21swapper/122:00:421
2912769994034,4cyclictest3095925-21latency_hist00:25:030
2912769994034,3cyclictest0-21swapper/021:10:030
291276999403,24cyclictest3048166-21ssh23:20:400
2912771993933,3cyclictest0-21swapper/221:25:022
291277199393,28cyclictest3081788-21ssh00:05:422
2912771993932,4cyclictest3017437-21sendmail22:40:002
2912771993929,6cyclictest2976194-21sendmail-mta21:45:402
2912770993933,3cyclictest0-21swapper/119:30:011
2912770993933,3cyclictest0-21swapper/119:30:011
2912770993931,5cyclictest421-21dbus-daemon00:40:021
2912770993931,5cyclictest421-21dbus-daemon00:40:021
2912770993931,5cyclictest2920773-21fschecks_count19:35:311
2912770993931,4cyclictest0-21swapper/121:50:381
2912770993931,3cyclictest0-21swapper/120:15:011
2912770993930,6cyclictest2926262-21systemd-run19:55:011
2912770993930,5cyclictest3066940-21switchtime23:45:491
2912769993933,3cyclictest0-21swapper/000:00:010
2912769993932,4cyclictest2935288-21kworker/u8:10+xprtiod20:40:000
2912771993832,3cyclictest0-21swapper/223:05:012
2912771993830,5cyclictest0-21swapper/221:20:012
2912771993828,8cyclictest2940602-21ntp_kernel_err20:40:362
2912771993828,8cyclictest2940602-21ntp_kernel_err20:40:362
2912770993831,3cyclictest0-21swapper/100:35:011
2912770993830,5cyclictest0-21swapper/123:40:011
2912770993828,6cyclictest0-21swapper/122:20:391
291277099381,33cyclictest2953626-21ssh21:15:361
2912769993831,4cyclictest3103329-21latency_hist00:35:020
2912769993828,7cyclictest1128531-21cups-browsed21:15:270
2912771993731,3cyclictest0-21swapper/221:10:522
2912771993730,4cyclictest0-21swapper/223:20:002
2912771993726,7cyclictest420-21avahi-daemon21:27:472
291277199372,30cyclictest2999311-21ssh22:15:402
2912770993731,3cyclictest0-21swapper/123:15:301
2912770993729,5cyclictest1128531-21cups-browsed22:45:351
2912770993728,6cyclictest2994330-21latency_hist22:10:021
2912770993728,5cyclictest0-21swapper/123:20:431
2912769993726,8cyclictest2972725-21timerandwakeup21:40:500
291276999372,27cyclictest3044542-21sh23:15:380
291276999372,27cyclictest3026462-21ssh22:50:430
2912769993720,11cyclictest12350irq/32-eth023:45:410
2912771993629,3cyclictest0-21swapper/200:00:012
2912771993627,6cyclictest421-21dbus-daemon19:15:012
2912771993627,6cyclictest2932172-21head20:10:492
2912771993625,8cyclictest2927084-21ntp_kernel_err19:55:362
291277199362,26cyclictest2972231-21ssh21:40:382
2912770993631,2cyclictest0-21swapper/119:15:011
2912770993626,8cyclictest3032694-21munin-plugin-st23:00:011
2912770993626,7cyclictest1051-21mta-sts-daemon21:43:001
2912770993625,8cyclictest2927759-21sendmail20:00:011
291277099361,27cyclictest2964705-21ssh21:30:391
2912769993629,4cyclictest2921770-21sendmail19:40:010
291276999361,28cyclictest2976126-21ssh21:45:380
291276999361,18cyclictest659-21snmpd19:55:370
2912771993528,4cyclictest1054-21mta-sts-daemon21:55:182
2912771993528,4cyclictest1054-21mta-sts-daemon21:55:182
2912771993528,4cyclictest0-21swapper/200:15:502
2912771993527,5cyclictest2998197-21latency_hist22:15:022
2912771993525,7cyclictest3076692-21cups-browsed00:20:462
2912771993525,6cyclictest1128531-21cups-browsed22:40:262
2912771993524,7cyclictest420-21avahi-daemon22:27:462
291277199352,26cyclictest3059251-21ssh23:35:432
2912770993527,5cyclictest0-21swapper/100:24:441
2912770993525,6cyclictest2969050-21head21:35:511
2912770993524,7cyclictest1-21systemd21:45:461
291277099351,25cyclictest3074286-21ssh23:55:471
291276999354,24cyclictest3040943-21ssh23:10:370
2912769993527,5cyclictest3059092-21ntp_kernel_err23:35:370
291276999352,25cyclictest2995379-21sh22:10:360
2912769993515,11cyclictest0-21swapper/019:20:350
2912769993513,5cyclictest14-21ksoftirqd/021:55:440
2912769993513,5cyclictest14-21ksoftirqd/021:55:440
2912771993427,4cyclictest0-21swapper/223:05:382
2912771993427,3cyclictest0-21swapper/223:50:412
2912771993427,3cyclictest0-21swapper/220:20:012
2912771993421,10cyclictest2928600-21ntp_kernel_err20:00:382
2912770993427,3cyclictest0-21swapper/123:00:361
2912770993426,5cyclictest2959855-21cron21:25:011
2912770993425,6cyclictest1054-21mta-sts-daemon23:42:451
2912770993424,8cyclictest2916921-21switchtime19:20:491
2912770993422,9cyclictest2961431-21timerandwakeup21:25:491
291277099342,22cyclictest3055554-21ssh23:30:431
291276999343,26cyclictest3014824-21ssh22:35:410
2912769993425,6cyclictest3076692-21cups-browsed00:00:420
2912769993425,6cyclictest3076692-21cups-browsed00:00:420
2912771993329,2cyclictest0-21swapper/221:30:322
2912771993326,4cyclictest0-21swapper/219:20:012
2912771993323,6cyclictest3084891-21aten_r2power_po00:10:292
2912770993327,3cyclictest0-21swapper/123:55:011
2912770993327,3cyclictest0-21swapper/122:20:021
2912770993325,5cyclictest2938766-21fschecks_count20:35:341
291276999334,23cyclictest2987704-21ssh22:00:380
291276999333,26cyclictest3055494-21awk23:30:400
291276999333,22cyclictest3007120-21ssh22:25:410
2912769993327,3cyclictest0-21swapper/020:15:010
2912769993321,7cyclictest3085706-21sh00:10:450
2912769993320,9cyclictest1-21systemd00:35:270
2912769993320,9cyclictest1-21systemd00:35:270
2912769993320,10cyclictest3032746-21kworker/u8:1-xprtiod23:10:010
291276999331,26cyclictest3051888-21rm23:25:430
2912771993223,6cyclictest2918782-21systemd-run19:30:012
2912771993223,6cyclictest2918782-21systemd-run19:30:012
2912771993222,6cyclictest2934069-21aten_r2power_en20:20:262
2912770993226,3cyclictest0-21swapper/120:50:011
2912770993226,3cyclictest0-21swapper/120:25:011
2912770993225,3cyclictest0-21swapper/122:10:421
2912770993224,4cyclictest0-21swapper/121:55:401
2912770993224,4cyclictest0-21swapper/121:55:401
291277099322,24cyclictest3096949-21ssh00:25:421
291277099322,22cyclictest3051721-21ssh23:25:371
291277099322,22cyclictest3037422-21ssh23:05:381
2912769993225,3cyclictest0-21swapper/022:15:450
2912769993224,4cyclictest2950563-21grep21:10:510
2912769993223,6cyclictest1-21systemd21:55:010
2912769993210,5cyclictest14-21ksoftirqd/000:08:400
2912771993125,3cyclictest0-21swapper/221:00:412
2912771993124,4cyclictest1054-21mta-sts-daemon00:31:222
2912771993122,4cyclictest0-21swapper/221:10:022
291277199312,23cyclictest3066622-21ssh23:45:402
2912771993120,8cyclictest3055825-21switchtime23:30:492
2912770993125,3cyclictest0-21swapper/121:15:011
2912770993125,3cyclictest0-21swapper/121:00:011
291276999319,4cyclictest14-21ksoftirqd/022:05:400
2912769993119,9cyclictest0-21swapper/022:40:480
2912769993118,9cyclictest2919704-21ntpq19:30:420
2912769993110,4cyclictest14-21ksoftirqd/020:45:370
2912771993025,2cyclictest0-21swapper/220:40:002
2912771993022,5cyclictest421-21dbus-daemon21:00:022
2912770993024,3cyclictest0-21swapper/119:20:011
2912770993022,5cyclictest2946818-21sendmail_mailqu21:00:471
291276999307,6cyclictest2913706-21modprobe19:10:410
2912769993023,4cyclictest2929295-21latency_hist20:05:020
2912769993021,7cyclictest2936159-21ntp_kernel_pll_20:25:390
2912769993019,7cyclictest2964697-21sh21:30:380
2912771992922,4cyclictest2915790-21kworker/u8:7-xprtiod19:25:012
2912771992921,5cyclictest2920254-21idleruntime-cro19:35:002
291277099292,21cyclictest2936196-21sendmail-mta20:25:401
291276999299,5cyclictest14-21ksoftirqd/019:40:380
2912769992924,3cyclictest2912697-21kworker/u8:8+xprtiod19:25:300
2912769992924,3cyclictest2912697-21kworker/u8:8+xprtiod19:25:300
2912769992923,3cyclictest0-21swapper/021:30:020
2912769992919,6cyclictest12350irq/32-eth023:50:450
291277199282,21cyclictest3048132-21ssh23:20:392
2912771992821,4cyclictest2930759-21munin-plugin-st20:10:012
2912771992821,3cyclictest0-21swapper/222:05:342
291276999289,4cyclictest14-21ksoftirqd/020:15:420
291276999289,3cyclictest14-21ksoftirqd/019:45:340
291276999288,4cyclictest14-21ksoftirqd/020:50:390
2912769992815,9cyclictest2937724-21ntpq20:30:410
2912771992720,4cyclictest2925017-21head19:50:282
291277199271,19cyclictest2987763-21awk22:00:402
291276999279,4cyclictest14-21ksoftirqd/020:20:410
291276999278,3cyclictest14-21ksoftirqd/019:50:300
2912771992615,5cyclictest0-21swapper/219:35:372
2912771992612,11cyclictest1051-21mta-sts-daemon20:45:422
291277199261,20cyclictest3041032-21ssh23:10:412
2912770992618,4cyclictest1051-21mta-sts-daemon20:30:371
291276999269,4cyclictest14-21ksoftirqd/020:40:300
291276999269,4cyclictest14-21ksoftirqd/020:40:300
291276999268,4cyclictest14-21ksoftirqd/019:10:020
291277199252,16cyclictest1128531-21cups-browsed20:25:432
2912771992520,2cyclictest0-21swapper/220:50:322
2912771992514,8cyclictest2938263-21sh20:35:012
2912770992520,2cyclictest0-21swapper/119:35:011
2912770992419,2cyclictest0-21swapper/119:10:011
2912770992418,3cyclictest1055-21mta-sts-daemon20:50:191
2912770992418,3cyclictest0-21swapper/119:40:341
2912770992416,5cyclictest2941156-21unixbench_multi20:40:481
2912770992416,5cyclictest2941156-21unixbench_multi20:40:481
2912771992314,6cyclictest2922408-21kernelversion19:40:332
2912771992313,6cyclictest2923784-21fschecks_time19:45:322
2912770992316,3cyclictest0-21swapper/120:00:331
291277099222,15cyclictest2924211-21ntp_states19:45:421
2912771992116,2cyclictest0-21swapper/219:10:022
291277299191,9cyclictest0-21swapper/321:40:393
291277299181,7cyclictest0-21swapper/323:10:373
291277299172,8cyclictest0-21swapper/323:30:363
291277299172,7cyclictest0-21swapper/323:45:413
291277299164,5cyclictest0-21swapper/321:45:383
291277299162,9cyclictest0-21swapper/321:50:363
291277299162,7cyclictest0-21swapper/322:30:413
291277299162,7cyclictest0-21swapper/322:30:413
291277299161,8cyclictest0-21swapper/300:35:413
291277299161,8cyclictest0-21swapper/300:35:413
291277299161,5cyclictest0-21swapper/319:25:383
291277299161,5cyclictest0-21swapper/319:25:383
291277299152,8cyclictest0-21swapper/322:55:433
291277299152,7cyclictest0-21swapper/323:25:433
291277299152,7cyclictest0-21swapper/323:15:373
291277299151,8cyclictest0-21swapper/322:05:433
291277299144,7cyclictest0-21swapper/322:20:373
291277299143,7cyclictest0-21swapper/322:35:393
291277299142,7cyclictest0-21swapper/323:55:463
291277299142,7cyclictest0-21swapper/322:10:263
291277299142,7cyclictest0-21swapper/321:10:393
291277299141,8cyclictest0-21swapper/300:15:363
291277299141,7cyclictest0-21swapper/321:35:373
291277299141,7cyclictest0-21swapper/321:25:373
291277299141,6cyclictest0-21swapper/320:35:443
291277299140,6cyclictest0-21swapper/319:35:393
291277299133,6cyclictest0-21swapper/323:05:383
291277299133,6cyclictest0-21swapper/322:25:513
291277299131,7cyclictest0-21swapper/323:50:423
291277299131,7cyclictest0-21swapper/322:15:383
291277299131,7cyclictest0-21swapper/321:55:393
291277299131,7cyclictest0-21swapper/321:55:393
291277299131,7cyclictest0-21swapper/321:20:403
291277299131,7cyclictest0-21swapper/300:20:433
291277299131,7cyclictest0-21swapper/300:05:383
291277299131,6cyclictest0-21swapper/320:30:393
291277299131,6cyclictest0-21swapper/320:25:403
291277299131,6cyclictest0-21swapper/320:05:393
291277299131,6cyclictest0-21swapper/319:55:393
291277299131,6cyclictest0-21swapper/319:50:393
291277299131,6cyclictest0-21swapper/319:15:413
291277299131,5cyclictest0-21swapper/320:40:363
291277299131,5cyclictest0-21swapper/320:40:363
291277299131,5cyclictest0-21swapper/319:45:393
291277299130,6cyclictest0-21swapper/319:10:393
291277299124,5cyclictest0-21swapper/321:15:373
291277299123,7cyclictest0-21swapper/323:20:333
291277299123,6cyclictest0-21swapper/320:15:363
291277299123,4cyclictest0-21swapper/320:55:453
291277299122,6cyclictest0-21swapper/323:35:273
291277299122,6cyclictest0-21swapper/323:00:383
291277299122,5cyclictest0-21swapper/321:05:013
291277299121,7cyclictest0-21swapper/322:50:353
291277299121,7cyclictest0-21swapper/322:45:323
291277299121,7cyclictest0-21swapper/322:40:313
291277299121,7cyclictest0-21swapper/300:25:443
291277299121,6cyclictest0-21swapper/323:40:293
291277299121,6cyclictest0-21swapper/320:20:423
291277299121,6cyclictest0-21swapper/319:20:393
291277299121,6cyclictest0-21swapper/300:00:323
291277299121,6cyclictest0-21swapper/300:00:323
291277299121,5cyclictest0-21swapper/321:05:383
291277299121,3cyclictest0-21swapper/300:10:373
291277299113,4cyclictest0-21swapper/320:10:363
291277299111,6cyclictest0-21swapper/322:00:323
291277299111,6cyclictest0-21swapper/321:30:373
291277299111,6cyclictest0-21swapper/319:40:373
291277299111,6cyclictest0-21swapper/300:30:293
291277299111,5cyclictest0-21swapper/320:45:373
291277299102,5cyclictest0-21swapper/320:50:363
291277299101,5cyclictest0-21swapper/319:30:293
29127729991,5cyclictest0-21swapper/320:00:333
29127729981,5cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional