You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 22:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu May 21, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
285934399911,87cyclictest0-21swapper/008:40:010
2859343998779,6cyclictest12-21ksoftirqd/010:50:010
285934399853,71cyclictest0-21swapper/011:45:020
2859353998031,45cyclictest2859559-21wc07:15:022
285935399781,54cyclictest0-21swapper/211:00:012
2859343997773,2cyclictest0-21swapper/007:15:020
2859343997670,3cyclictest0-21swapper/011:50:020
285934999746,52cyclictest221rcuc/107:45:021
2859343997141,16cyclictest16950irq/224-800000011:55:020
285935399701,65cyclictest2907644-21ssh10:37:032
285935399691,64cyclictest2914522-21ssh10:51:162
285935399691,64cyclictest2914522-21ssh10:51:152
285935399691,64cyclictest2886974-21ssh09:54:252
285935899684,17cyclictest2863689-21modprobe08:35:003
285935899684,17cyclictest2863689-21modprobe08:35:003
2859358996814,8cyclictest2929854-21sshd11:23:073
285935399681,63cyclictest2939457-21ssh11:43:132
285935399672,61cyclictest2952750-21ssh12:10:212
285935399671,62cyclictest2949484-21ssh12:03:532
285935399671,62cyclictest2889591-21ssh09:59:512
285935399671,61cyclictest2924927-21ssh11:13:002
2859358996626,36cyclictest2928221-21cron11:20:003
285935399661,62cyclictest2905855-21ssh10:33:252
285935399661,61cyclictest2957407-21ssh12:20:032
285935399661,61cyclictest2951435-21ssh12:07:582
285935399661,61cyclictest2951435-21ssh12:07:572
285935399661,61cyclictest2901969-21ssh10:25:052
285935399661,61cyclictest2901379-21ssh10:24:112
285935399661,61cyclictest2897972-21ssh10:17:012
285935399661,61cyclictest2883999-21ssh09:48:022
285935399661,60cyclictest2909251-21ssh10:40:112
2859343996642,19cyclictest16850irq/134-800000010:20:000
2859343996611,9cyclictest2916927-21ssh10:56:110
285935899654,13cyclictest2864724-21modprobe08:55:003
2859353996527,35cyclictest2940845-21sh11:45:482
285935399652,59cyclictest2936113-21ssh11:36:062
285935399651,61cyclictest2931439-21ssh11:26:162
285935399651,60cyclictest2962797-21ssh12:31:152
285935399651,60cyclictest2946110-21ssh11:56:512
285935399651,60cyclictest2933312-21ssh11:30:052
285935399651,60cyclictest2933312-21ssh11:30:052
285935399651,60cyclictest2923163-21ssh11:09:302
285935399651,59cyclictest2943035-21ssh11:50:252
285935399651,59cyclictest2926447-21ssh11:15:572
2859358996426,34cyclictest2921028-21latency_hist11:05:023
285935899642,17cyclictest2860337-21cut07:30:003
285935899642,15cyclictest2945121-21ssh11:55:013
285935399642,57cyclictest2861126-21munin-run07:45:012
285935399641,61cyclictest2874795-21ssh09:29:172
285935399641,59cyclictest2964934-21ssh12:35:472
285935399641,59cyclictest2960608-21ssh12:26:432
285935399641,59cyclictest2956303-21ssh12:18:042
285935399641,59cyclictest2929300-21ssh11:21:592
285935399641,59cyclictest2912765-21ssh10:47:402
285935399641,59cyclictest2894773-21ssh10:10:072
285935399641,59cyclictest2891234-21ssh10:03:052
285935399641,59cyclictest2881648-21ssh09:43:132
285935399641,59cyclictest2879040-21ssh09:37:442
285935399641,59cyclictest2877023-21ssh09:33:422
285935399641,59cyclictest2877023-21ssh09:33:422
285935399641,59cyclictest2871346-21ssh09:21:442
285935399641,59cyclictest2871346-21ssh09:21:442
285935399641,59cyclictest2869450-21ssh09:17:562
285935399641,59cyclictest2869450-21ssh09:17:562
285935399641,59cyclictest2866082-21ssh09:10:482
2859343996410,38cyclictest12-21ksoftirqd/009:40:030
2859358996319,9cyclictest2963595-21sshd12:33:083
285935399632,55cyclictest2864748-21munin-run08:55:012
285935399631,58cyclictest2919106-21ssh11:00:412
285935399631,58cyclictest2892783-21ssh10:06:072
285935399631,58cyclictest2862362-21sshd08:05:082
2859358996225,33cyclictest37-21ksoftirqd/311:26:393
2859358996213,11cyclictest1-21systemd11:50:003
285934399626,7cyclictest121ksoftirqd/012:19:100
285934399621,53cyclictest2916225-21wc10:55:030
285934399621,53cyclictest2916225-21wc10:55:020
285935899612,18cyclictest2925775-21munin-run11:15:013
285935899612,17cyclictest2935695-21ssh11:35:043
285935399612,54cyclictest2863170-21sshd08:22:082
2859353996123,34cyclictest30-21ksoftirqd/208:05:022
2859353996115,42cyclictest2860875-21sh07:40:012
285934999613,40cyclictest2886827-21kworker/u8:010:30:011
285934399615,40cyclictest12-21ksoftirqd/010:34:410
2859343996115,41cyclictest151rcuc/012:32:320
285935899609,15cyclictest275-21dbus-daemon11:00:013
2859358996024,32cyclictest2914756-21sh10:51:513
2859358996024,32cyclictest2914756-21sh10:51:513
285935899602,15cyclictest2957736-21ssh12:20:453
285935899602,14cyclictest2868050-21latency_hist09:15:023
285935899602,14cyclictest2864471-21cron08:49:593
285935399602,54cyclictest2863699-21munin-run08:35:012
285935399602,54cyclictest2863699-21munin-run08:35:012
285935399602,53cyclictest2860347-21munin-run07:30:012
285935399601,55cyclictest48-21kcompactd007:15:132
285935899592,17cyclictest2896842-21ssh10:14:583
285935899592,15cyclictest2909183-21modprobe10:40:023
285935899592,15cyclictest1-21systemd09:25:003
285935899592,15cyclictest1-21systemd09:25:003
2859358995914,6cyclictest2906567-21sh10:35:023
285935899591,15cyclictest2935505-21ssh11:35:023
285935899591,15cyclictest2935505-21ssh11:35:023
285935399592,52cyclictest364-21snmpd08:41:362
285935399592,52cyclictest364-21snmpd08:41:362
285935399591,54cyclictest2864453-21sshd08:46:072
285935399591,54cyclictest2863913-21sshd08:35:072
2859349995932,19cyclictest1-21systemd09:10:001
285935899583,13cyclictest2899375-21grep10:20:013
2859358995825,29cyclictest37-21ksoftirqd/310:10:023
285935899582,17cyclictest2860083-21munin-plugin-st07:25:003
285935899582,16cyclictest2908935-21latency_hist10:40:013
285935899582,14cyclictest2949071-21ssh12:03:013
2859358995820,34cyclictest2863463-21wc08:30:023
285935399582,52cyclictest2861630-21munin-run07:55:012
285935399581,53cyclictest2864949-21sshd08:55:072
285935399581,53cyclictest2863412-21sshd08:25:302
2859349995840,13cyclictest275-21dbus-daemon10:55:021
2859349995840,13cyclictest275-21dbus-daemon10:55:011
2859343995827,13cyclictest2869065-21ssh09:17:070
2859343995827,13cyclictest2869065-21ssh09:17:070
285935899573,9cyclictest2864997-21munin-run09:00:003
285935899573,11cyclictest2952396-21grep12:10:023
285935899573,11cyclictest2952396-21grep12:10:013
2859358995725,27cyclictest2922329-21sh11:07:343
2859358995723,30cyclictest2886258-21sh09:52:433
285935899572,17cyclictest2879998-21munin-run09:40:003
285935899572,14cyclictest2870584-21ls09:20:033
285935899572,14cyclictest2870584-21ls09:20:023
2859358995717,14cyclictest2863165-21sshd08:21:073
285935399571,52cyclictest2865488-21sshd09:06:082
285935399571,52cyclictest2862866-21sshd08:15:062
285935399571,50cyclictest48-21kcompactd007:57:562
285934399573,20cyclictest0-21swapper/011:59:320
2859343995729,12cyclictest2948518-21ssh12:01:480
2859343995727,15cyclictest2904142-21ssh10:29:580
285934399571,39cyclictest2919250-21ssh11:00:590
285935899562,18cyclictest2888248-21ssh09:56:453
285935899561,17cyclictest2945838-21ssh11:56:173
285935899561,16cyclictest2962094-21sed12:30:013
285935399562,50cyclictest2861341-21sshd07:47:072
285935399562,49cyclictest2865234-21sshd09:00:362
285935899556,20cyclictest2892171-21grep10:05:023
2859358995521,30cyclictest2966099-21sh12:38:263
285935899551,15cyclictest364-21snmpd09:08:463
285935399552,49cyclictest2860581-21latency_hist07:35:022
2859343995527,24cyclictest16850irq/134-800000011:27:300
2859343995527,12cyclictest2936196-21ssh11:36:170
2859343995521,30cyclictest12-21ksoftirqd/012:25:480
285934399551,38cyclictest2933887-21ssh11:31:240
285934399551,38cyclictest2933887-21ssh11:31:240
285935899543,16cyclictest2882432-21modprobe09:45:013
2859358995422,29cyclictest37-21ksoftirqd/307:50:013
285935899542,16cyclictest2901730-21munin-plugin-st10:25:013
285935899542,16cyclictest2884903-21munin-run09:50:013
285935899542,12cyclictest2861856-21cron08:00:003
2859358995413,12cyclictest3467602-21systemd-journal10:30:003
285935399541,49cyclictest48-21kcompactd007:23:332
285935399541,47cyclictest364-21snmpd08:10:082
2859349995433,14cyclictest3467602-21systemd-journal09:40:001
285934399543,5cyclictest121ksoftirqd/010:35:360
2859343995425,13cyclictest2887565-21ssh09:55:110
2859343995417,27cyclictest16850irq/134-800000010:10:140
2859343995412,11cyclictest0-21swapper/009:46:270
285935899533,17cyclictest2860828-21modprobe07:40:003
285935899533,15cyclictest2940251-21munin-run11:45:013
2859358995325,24cyclictest37-21ksoftirqd/312:19:433
2859358995323,26cyclictest2861165-21latency_hist07:45:013
2859343995326,12cyclictest2958092-21ssh12:21:380
2859343995319,16cyclictest2873050-21ssh09:25:070
2859358995215,9cyclictest37-21ksoftirqd/310:45:223
285935899521,16cyclictest2953743-21ssh12:12:413
285935899521,16cyclictest2876060-21ssh09:31:333
285935899521,16cyclictest2876060-21ssh09:31:333
2859349995223,13cyclictest2885115-21ssh09:50:021
2859343995235,14cyclictest19350irq/34-mmc111:17:160
2859343995226,11cyclictest2925574-21ssh11:14:220
2859343995226,11cyclictest2894368-21ssh10:09:460
2859343995225,12cyclictest2866156-21ssh09:11:040
285935899519,11cyclictest2863931-21sshd08:38:083
2859343995117,17cyclictest292341150kworker/u8:411:10:020
2859343995112,28cyclictest0-21swapper/012:05:030
2859343995112,28cyclictest0-21swapper/012:05:020
285934399511,10cyclictest12-21ksoftirqd/008:15:000
285935899503,14cyclictest2861737-21latency_hist07:55:023
285935899502,14cyclictest2863080-21latency_hist08:20:023
2859358995020,27cyclictest37-21ksoftirqd/309:25:483
2859349995023,15cyclictest0-21swapper/107:40:001
285934399501,34cyclictest2909958-21ssh10:41:480
285935899492,15cyclictest2860597-21munin-run07:35:023
285935899492,14cyclictest2862392-21modprobe08:10:003
285935899491,15cyclictest2865366-21grep09:05:023
2859343994942,5cyclictest19350irq/34-mmc108:20:040
2859343994925,9cyclictest2952715-21ssh12:10:170
2859343994922,11cyclictest2885923-21ssh09:51:520
2859358994814,7cyclictest37-21ksoftirqd/308:13:593
2859349994839,6cyclictest2937830-21modprobe11:39:591
2859349994833,11cyclictest1-21systemd09:50:001
2859349994832,11cyclictest3467602-21systemd-journal10:44:071
2859349994822,19cyclictest2861877-21systemctl08:00:001
2859343994837,7cyclictest12-21ksoftirqd/010:22:240
285934399482,5cyclictest2864733-21munin-plugin-st08:55:000
2859343994811,5cyclictest0-21swapper/009:33:070
2859343994811,5cyclictest0-21swapper/009:33:070
285935899473,8cyclictest2859810-21munin-run07:20:003
2859349994726,13cyclictest275-21dbus-daemon08:05:021
2859343994731,3cyclictest16850irq/134-800000012:36:040
2859343994724,20cyclictest2881524-21ssh09:42:500
2859343994721,11cyclictest2871605-21ssh09:22:250
2859343994721,11cyclictest2871605-21ssh09:22:240
2859343994714,29cyclictest151rcuc/008:50:000
285935899469,9cyclictest3467602-21systemd-journal07:10:023
2859358994611,5cyclictest0-21swapper/308:45:013
2859358994611,5cyclictest0-21swapper/308:45:013
2859349994629,12cyclictest1-21systemd09:15:011
2859343994639,3cyclictest16950irq/224-800000007:35:000
2859343994629,13cyclictest3467602-21systemd-journal08:45:020
2859343994629,13cyclictest3467602-21systemd-journal08:45:010
2859343994627,15cyclictest2891609-21ssh10:03:510
2859343994617,14cyclictest2928766-21ssh11:20:400
2859349994525,14cyclictest2962066-21tmpfiles12:30:021
285934399457,34cyclictest151rcuc/007:40:020
285934399452,36cyclictest2859799-21modprobe07:20:000
285935899443,9cyclictest2862207-21cat08:05:023
285934999446,18cyclictest0-21swapper/109:30:011
2859349994427,13cyclictest2947463-21cron12:00:001
2859349994425,15cyclictest1-21systemd12:25:011
2859349994425,14cyclictest1-21systemd07:55:011
2859343994429,11cyclictest1-21systemd09:00:010
285934399441,35cyclictest2861354-21modprobe07:50:000
285935399431,38cyclictest2859403-21latency_hist07:10:022
285934999437,13cyclictest131rcu_preempt10:10:021
2859349994330,10cyclictest23-21ksoftirqd/112:12:391
285934399431,7cyclictest2865537-21true09:10:000
2859349994227,11cyclictest309-21in:imuxsock08:20:001
2859349994226,12cyclictest23-21ksoftirqd/112:01:121
285934999422,4cyclictest131rcu_preempt12:35:561
285934999421,4cyclictest131rcu_preempt10:01:531
2859343994234,5cyclictest151rcuc/007:40:080
2859343994227,11cyclictest12-21ksoftirqd/007:26:390
2859343994222,16cyclictest2862876-21sshd08:16:070
285934999413,3cyclictest131rcu_preempt11:04:561
2859349994132,6cyclictest23-21ksoftirqd/108:45:021
2859349994132,6cyclictest23-21ksoftirqd/108:45:021
2859349994129,8cyclictest2935440-21latency_hist11:35:031
2859349994129,8cyclictest2935440-21latency_hist11:35:021
2859349994128,10cyclictest23-21ksoftirqd/110:35:081
2859349994126,7cyclictest23-21ksoftirqd/110:20:001
2859349994125,12cyclictest1-21systemd10:50:001
285934999412,4cyclictest131rcu_preempt09:24:291
285934999412,4cyclictest131rcu_preempt09:24:281
285934999411,3cyclictest0-21swapper/112:17:421
2859343994134,5cyclictest12-21ksoftirqd/009:05:000
2859343994134,4cyclictest16950irq/224-800000008:30:010
285934999404,22cyclictest2918601-21ssh11:00:001
2859349994030,6cyclictest2865521-21diskmemload11:25:011
2859349994030,6cyclictest2862395-21sh08:10:001
2859349994029,9cyclictest23-21ksoftirqd/112:08:311
2859349994029,9cyclictest23-21ksoftirqd/112:08:301
2859349994029,9cyclictest23-21ksoftirqd/111:44:071
2859349994029,9cyclictest23-21ksoftirqd/109:20:011
2859349994029,9cyclictest23-21ksoftirqd/109:20:011
2859349994029,7cyclictest2860081-21munin-run07:25:001
2859349994028,9cyclictest23-21ksoftirqd/111:48:551
2859349994025,11cyclictest3467602-21systemd-journal11:20:011
2859349994024,12cyclictest3467602-21systemd-journal10:10:081
285934999402,3cyclictest131rcu_preempt11:11:291
285934999402,3cyclictest131rcu_preempt09:33:111
285934999402,3cyclictest131rcu_preempt09:33:111
2859343994031,6cyclictest12-21ksoftirqd/007:25:020
2859343994026,10cyclictest12-21ksoftirqd/007:55:000
285934999394,25cyclictest0-21swapper/107:25:031
2859349993931,5cyclictest2859803-21sh07:20:001
2859349993930,5cyclictest2906550-21sh10:35:001
2859349993928,9cyclictest23-21ksoftirqd/111:51:351
2859349993926,10cyclictest23-21ksoftirqd/109:43:341
2859349993925,11cyclictest23-21ksoftirqd/112:31:041
2859349993925,10cyclictest3467602-21systemd-journal08:27:071
2859349993925,10cyclictest275-21dbus-daemon07:50:011
2859343993928,8cyclictest12-21ksoftirqd/008:05:020
2859343993823,11cyclictest3467602-21systemd-journal08:10:000
2859349993726,9cyclictest23-21ksoftirqd/111:08:241
2859349993725,8cyclictest23-21ksoftirqd/111:26:261
2859349993718,14cyclictest275-21dbus-daemon08:25:001
285934999371,3cyclictest131rcu_preempt10:20:181
2859343993710,20cyclictest2859338-21cyclictest07:10:020
2859349993627,7cyclictest23-21ksoftirqd/109:57:521
2859349993626,8cyclictest23-21ksoftirqd/108:35:021
2859349993626,8cyclictest23-21ksoftirqd/108:35:021
2859349993623,8cyclictest3467602-21systemd-journal09:00:001
285934399361,31cyclictest0-21swapper/007:59:450
2859358993527,6cyclictest37-21ksoftirqd/307:10:023
2859349993524,8cyclictest23-21ksoftirqd/108:55:011
2859349993521,11cyclictest23-21ksoftirqd/109:05:021
285934399351,30cyclictest0-21swapper/008:32:400
285934399351,30cyclictest0-21swapper/008:32:400
2859349993427,4cyclictest2859507-21sshd07:10:101
2859349993424,8cyclictest23-21ksoftirqd/108:50:011
2859349993424,7cyclictest2862720-21sed08:15:011
2859349993325,5cyclictest2863959-21munin-plugin-st08:40:001
2859349993323,7cyclictest23-21ksoftirqd/107:35:021
285934999278,13cyclictest0-21swapper/107:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional