You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-08 - 20:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Jan 08, 2026 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
283271324390,4sleep30-21swapper/307:05:183
283374722250,7sleep20-21swapper/207:09:552
286341021080,5sleep30-21swapper/308:35:133
197399900,8rtkit-daemon1972-21rtkit-daemon11:46:580
28556442840,6sleep3411rcuc/308:10:263
197399840,27rtkit-daemon0-21swapper/207:11:582
28452822800,9sleep14497-21PK-Backend07:40:131
283228227946,23sleep00-21swapper/007:05:120
29076892670,5sleep2331rcuc/210:40:362
283389799651,27cyclictest2856763-21idleruntime08:15:202
283389799651,27cyclictest2856763-21idleruntime08:15:192
283371926344,13sleep10-21swapper/107:09:361
283389799612,32cyclictest2889008-21unin-run09:50:012
283389799612,32cyclictest2889008-21unin-run09:50:012
283389799611,37cyclictest2876631-21taskset09:12:262
28604192600,5chrt0-21swapper/208:25:222
283390299601,32cyclictest2893289-21taskset10:00:253
283389799601,33cyclictest2947409-21latency_hist12:40:022
283390299597,33cyclictest0-21swapper/312:05:013
283389799593,52cyclictest2929000-21wc11:45:282
283389799591,36cyclictest2846182-21taskset07:41:102
29322932580,9sleep00-21swapper/011:55:220
29322932580,9sleep00-21swapper/011:55:220
283389799571,32cyclictest2848742-21date07:50:022
283389799571,22cyclictest0-21swapper/209:30:142
283389799571,22cyclictest0-21swapper/209:30:132
283390299561,50cyclictest2923584-21needreboot11:30:213
2833902995518,33cyclictest2917448-21latency_hist11:15:013
283390299551,49cyclictest2931699-21sort11:55:123
283390299551,49cyclictest2931699-21sort11:55:123
283389799551,34cyclictest2938187-21taskset12:14:392
283389799551,32cyclictest2862573-21sed08:30:302
283390299541,48cyclictest2924704-21sort11:35:133
2833902995414,36cyclictest1137-21runrttasks11:42:373
2833902995413,37cyclictest1137-21runrttasks08:20:553
283389799541,27cyclictest2924470-21cut11:35:032
197399540,38rtkit-daemon2838907-21gunzip07:20:282
283390299531,29cyclictest2879330-21taskset09:20:193
283389799532,29cyclictest0-21swapper/208:36:302
283389799532,28cyclictest2938862-21df12:15:142
283389799531,30cyclictest2882178-21latency_hist09:30:022
197399530,8rtkit-daemon1972-21rtkit-daemon09:41:331
197399530,7rtkit-daemon1972-21rtkit-daemon08:05:130
283390299522,45cyclictest2890803-40lsb_release09:53:033
283390299522,45cyclictest2890803-40lsb_release09:53:023
2833902995119,27cyclictest2936016-21cat12:05:283
2833902995118,28cyclictest2860168-21expr08:25:133
283390299511,45cyclictest2873592-21latency_hist09:05:033
283390299511,45cyclictest2850267-21latency_hist07:55:033
283389799514,42cyclictest2850426-21latency_hist07:55:032
2833897995125,21cyclictest2942998-21strings12:25:212
2833897995116,29cyclictest2906312-21wc10:40:022
283389799511,46cyclictest2919212-21systemd-run11:20:002
28908122500,5chrt0-21swapper/009:53:040
28908122500,5chrt0-21swapper/009:53:040
2833902995021,24cyclictest2862216-21needreboot08:30:233
283390299501,45cyclictest1137-21runrttasks07:12:433
283390299501,44cyclictest2942847-21wc12:25:203
283390299501,44cyclictest2888429-21cat09:45:283
283390299501,44cyclictest2888429-21cat09:45:273
283390299501,44cyclictest2885999-21expr09:40:133
283389799502,43cyclictest2854432-21sort08:10:012
2833897995022,24cyclictest2900080-21cat10:20:292
2833897995021,25cyclictest2193-21gdbus11:29:562
283390299492,42cyclictest2940937-21df_inode12:20:233
283390299492,42cyclictest2940937-21df_inode12:20:233
283390299492,42cyclictest0-21swapper/310:30:273
2833902994921,24cyclictest2841263-21basename07:30:003
2833902994918,26cyclictest2939264-21awk12:15:263
2833902994917,27cyclictest2928445-21expr11:45:143
2833902994916,28cyclictest2901620-21cut10:25:203
2833902994915,30cyclictest2845743-21ntp_kernel_pll_07:40:223
283390299491,44cyclictest2944385-21cstates12:30:133
283390299491,44cyclictest2852354-21cpuspeed_turbos08:00:153
283390299491,44cyclictest0-21swapper/312:35:153
283390299491,43cyclictest2882048-21latency_hist09:30:033
283390299491,43cyclictest2848723-21cat07:50:033
283390299491,43cyclictest2836158-21cstates07:15:053
2833902994912,32cyclictest2910219-21ls10:50:113
283389799493,41cyclictest2889355-21apt-key09:50:032
283389799493,41cyclictest2889355-21apt-key09:50:032
2833897994914,30cyclictest2914656-21hddtemp_smartct11:05:192
2833897994914,30cyclictest2914656-21hddtemp_smartct11:05:182
28791642480,7sleep14497-21PK-Backend09:20:141
283390299482,42cyclictest2838920-21python307:20:293
283390299482,41cyclictest2876334-21find09:10:293
2833902994820,23cyclictest2905334-21ntpq10:35:233
2833902994818,25cyclictest2914809-21cat11:05:223
2833902994818,25cyclictest2914809-21cat11:05:213
2833902994815,28cyclictest2898437-21cat10:15:263
283390299481,43cyclictest2930788-21threads11:50:283
283390299481,43cyclictest2850925-21/usr/sbin/munin07:55:183
283390299481,43cyclictest0-21swapper/312:10:143
283390299481,42cyclictest2843463-21expr07:35:113
283389799485,40cyclictest1137-21runrttasks10:28:092
283389799483,41cyclictest2885936-21cpuspeed_turbos09:40:132
283389799482,42cyclictest2880296-21cron09:25:002
283389799481,21cyclictest0-21swapper/211:10:212
29336842470,14chrt34-21ksoftirqd/212:00:152
2833902994721,21cyclictest2899993-21python310:20:273
2833902994715,28cyclictest2913743-21seq11:02:023
2833902994715,28cyclictest2857816-21latency_hist08:20:023
2833902994715,28cyclictest2857816-21latency_hist08:20:023
2833902994714,28cyclictest2921415-21expr11:25:143
283390299471,41cyclictest2919455-21apt11:20:113
283390299471,41cyclictest2906580-21/usr/sbin/munin10:40:183
283390299471,41cyclictest2869112-21grep08:50:213
283389799472,41cyclictest2908603-21cpuspeed_turbos10:45:112
2833897994718,24cyclictest2944825-21awk12:30:242
2833897994712,30cyclictest438-20systemd-journal11:04:532
2833892994729,13cyclictest2345-21gdbus10:39:570
283389299471,41cyclictest1972-21rtkit-daemon08:31:580
29432862460,7sleep04497-21PK-Backend12:25:280
28569792460,8sleep14497-21PK-Backend08:15:221
28569792460,8sleep14497-21PK-Backend08:15:221
2833902994619,23cyclictest2873818-21grep09:05:133
2833902994619,23cyclictest2873818-21grep09:05:133
2833902994615,27cyclictest2883263-21timerandwakeup09:30:293
2833902994615,27cyclictest2883263-21timerandwakeup09:30:293
283390299461,41cyclictest2871095-21switchtime08:55:293
283390299461,41cyclictest2866039-21sed08:40:303
283390299461,41cyclictest2842501-21cat07:30:303
283390299461,40cyclictest2853994-21taskset08:05:393
283389799463,39cyclictest2836524-21cstates07:15:142
2833897994621,20cyclictest3693053-21snapd12:05:522
2833897994619,23cyclictest2854680-21cpuspeed08:10:082
2833897994618,24cyclictest2850733-21cstates07:55:142
2833897994613,27cyclictest2870307-21cpuspeed_turbos08:55:142
283389299461,40cyclictest1972-21rtkit-daemon12:05:230
29438162450,15sleep1251rcuc/112:30:011
29235362450,15sleep126-21ksoftirqd/111:30:201
283390299451,41cyclictest2912272-21awk11:00:003
283390299451,40cyclictest0-21swapper/310:45:263
283390299451,39cyclictest2884834-21/usr/sbin/munin09:35:253
283389799456,35cyclictest2893749-21cut10:05:012
283389799454,36cyclictest1137-21runrttasks10:55:302
2833897994520,22cyclictest1137-21runrttasks11:58:332
2833897994520,22cyclictest1137-21runrttasks11:58:322
2833897994519,22cyclictest2877323-21cstates09:15:132
2833897994518,23cyclictest2910371-21cstates10:50:122
2833897994518,23cyclictest2858290-21cstates08:20:132
2833897994514,27cyclictest2904406-21date10:35:012
283389799451,40cyclictest2942089-21unin-run12:25:012
283389799451,40cyclictest2942089-21unin-run12:25:002
283390299442,38cyclictest2891604-21pkexec09:56:073
283390299442,38cyclictest2868181-21cron08:49:593
283390299441,39cyclictest2918033-21cstates11:15:123
283390299441,39cyclictest2894234-21expr10:05:153
2833897994419,21cyclictest2919664-21cstates11:20:142
2833897994417,22cyclictest2927044-21head11:40:212
2833897994413,26cyclictest2895928-21expr10:10:142
2833892994420,10cyclictest2868459-21seq08:50:100
283390299431,38cyclictest2877490-21/usr/sbin/munin09:15:153
2833897994319,21cyclictest2844758-21latency_hist07:40:012
2833897994318,22cyclictest1137-21runrttasks08:01:012
2833897994318,21cyclictest2897595-21cstates10:15:122
2833897994316,23cyclictest1137-21runrttasks07:26:162
283389799431,36cyclictest2872088-21grep09:00:132
29401902420,15sleep12940192-21latency_hist12:20:021
29376432420,8sleep04497-21PK-Backend12:10:280
28940852420,7sleep234-21ksoftirqd/210:05:132
2833902994214,23cyclictest1081-21ntpd10:10:243
283389799429,28cyclictest2884645-21cat09:35:222
283389799424,33cyclictest2865162-21grep08:40:132
283389799421,37cyclictest2925453-21grep11:35:272
283389799421,36cyclictest2929990-21grep11:50:122
283389799421,36cyclictest2874452-21wc09:05:262
283389799421,36cyclictest2874452-21wc09:05:252
283389799421,36cyclictest2868594-21cut08:50:132
283389699421,13cyclictest2910824-21needreboot10:50:211
2833892994224,14cyclictest2917872-21tr11:15:110
2833892994218,11cyclictest2939767-21systemd-udevd12:17:330
28960602410,8sleep00-21swapper/010:10:190
283389799412,35cyclictest2842972-21sh07:35:002
2833897994115,22cyclictest1137-21runrttasks09:58:552
283389799411,35cyclictest2866897-21tr08:45:122
2833892994122,14cyclictest2919816-21df11:20:150
283389699381,10cyclictest141rcu_preempt12:25:001
283389699381,10cyclictest141rcu_preempt12:24:591
283389299389,11cyclictest13-21ksoftirqd/007:19:560
283389299388,11cyclictest13-21ksoftirqd/011:41:540
2833892993827,7cyclictest2915725-21turbostat11:10:000
2833892993827,7cyclictest2915725-21turbostat11:10:000
2833892993821,13cyclictest2855361-21ntp_states08:10:220
2833892993819,13cyclictest2856710-21df08:15:150
2833892993819,13cyclictest2856710-21df08:15:140
283389299381,8cyclictest2898863-21turbostat10:19:590
283389299381,34cyclictest1972-21rtkit-daemon08:45:510
283389299381,33cyclictest1972-21rtkit-daemon09:05:330
283389299381,33cyclictest1972-21rtkit-daemon09:05:320
28418102370,18sleep126-21ksoftirqd/107:30:151
2833896993721,12cyclictest2871643-21cron09:00:011
283389699371,9cyclictest4497-21PK-Backend09:28:141
2833896993718,9cyclictest2834540-21awk07:10:191
2833892993720,13cyclictest1-21systemd10:55:000
2833892993715,9cyclictest13-21ksoftirqd/012:02:580
2833896993621,11cyclictest2926151-21grep11:40:011
283389699361,11cyclictest4497-21PK-Backend07:19:551
283389699361,10cyclictest4497-21PK-Backend12:08:091
283389299368,11cyclictest13-21ksoftirqd/011:38:230
283389299368,11cyclictest13-21ksoftirqd/008:57:340
283389299368,11cyclictest13-21ksoftirqd/008:21:000
283389299368,11cyclictest13-21ksoftirqd/007:28:160
2833892993623,9cyclictest2877786-21grep09:15:200
2833892993619,12cyclictest438-20systemd-journal07:39:590
2833896993521,9cyclictest2883784-21cut09:34:591
2833896993521,9cyclictest2883784-21cut09:34:591
2833896993521,10cyclictest2889463-21apt-get09:50:111
2833896993521,10cyclictest2889463-21apt-get09:50:101
283389699351,9cyclictest4497-21PK-Backend10:20:091
283389699351,8cyclictest4497-21PK-Backend11:58:201
283389699351,8cyclictest4497-21PK-Backend11:58:201
283389699351,8cyclictest4497-21PK-Backend07:28:171
283389699351,30cyclictest4497-21PK-Backend08:13:161
283389299359,7cyclictest0-21swapper/007:13:560
283389299357,11cyclictest13-21ksoftirqd/010:20:080
283389299356,12cyclictest13-21ksoftirqd/011:25:040
283389299351,7cyclictest2904357-21turbostat10:35:010
283389299351,31cyclictest1972-21rtkit-daemon09:37:320
283389299351,30cyclictest2876856-21turbostat09:14:590
283389699343,6cyclictest20691gnome-shell08:30:591
283389699341,9cyclictest4497-21PK-Backend11:47:371
2833896993419,10cyclictest2894737-21cut10:05:261
283389699341,8cyclictest4497-21PK-Backend11:41:551
2833896993417,12cyclictest2946778-21grep12:35:291
283389699341,30cyclictest0-21swapper/107:50:071
283389699341,29cyclictest4497-21PK-Backend12:34:411
283389699341,29cyclictest4497-21PK-Backend10:18:581
283389699341,29cyclictest4497-21PK-Backend09:45:121
283389699341,29cyclictest4497-21PK-Backend09:45:111
283389699341,29cyclictest4497-21PK-Backend09:02:211
283389699341,29cyclictest0-21swapper/110:00:131
283389699341,28cyclictest4497-21PK-Backend09:08:171
283389699341,28cyclictest4497-21PK-Backend09:08:171
283389699341,10cyclictest4497-21PK-Backend08:21:011
283389299348,7cyclictest0-21swapper/008:35:100
283389299348,11cyclictest13-21ksoftirqd/007:54:250
2833892993424,6cyclictest2872113-21tr09:00:140
2833892993418,11cyclictest466-21systemd-udevd10:00:410
2833892993417,13cyclictest2838956-21sed07:20:290
283389699338,5cyclictest141rcu_preempt10:15:001
2833896993326,4cyclictest4497-21PK-Backend10:43:401
2833896993322,8cyclictest2902236-40apt-check10:26:021
283389699331,9cyclictest4497-21PK-Backend08:00:211
2833896993318,11cyclictest2843393-21tr07:35:121
2833896993318,10cyclictest2912688-21grep11:00:091
283389699331,5cyclictest4497-21PK-Backend11:21:311
283389699331,5cyclictest4497-21PK-Backend08:25:471
283389699331,5cyclictest4497-21PK-Backend07:21:391
283389699331,29cyclictest4497-21PK-Backend12:12:481
283389699331,29cyclictest4497-21PK-Backend10:59:181
283389699331,28cyclictest4497-21PK-Backend10:34:551
283389299337,10cyclictest13-21ksoftirqd/008:29:530
283389299335,11cyclictest13-21ksoftirqd/011:01:100
2833892993325,5cyclictest0-21swapper/011:50:310
2833892993323,6cyclictest2912263-21turbostat11:00:010
2833892993321,8cyclictest2841702-21tr07:30:140
2833892993318,11cyclictest2886692-21grep09:40:280
2833892993318,11cyclictest2846480-21cron07:45:010
283389299331,28cyclictest4497-21PK-Backend08:40:040
28482462320,7sleep04497-21PK-Backend07:47:000
283389699327,5cyclictest141rcu_preempt11:10:131
2833896993225,5cyclictest0-21swapper/108:46:241
2833896993221,7cyclictest2914409-21tr11:05:141
2833896993221,7cyclictest2914409-21tr11:05:141
2833896993218,10cyclictest2863463-21tr08:35:121
2833896993216,12cyclictest923-21dbus-daemon07:47:061
283389699321,5cyclictest4497-21PK-Backend10:47:081
283389699321,4cyclictest4497-21PK-Backend07:57:181
283389699321,28cyclictest4497-21PK-Backend09:38:191
283389699321,10cyclictest4497-21PK-Backend08:40:121
283389299325,11cyclictest13-21ksoftirqd/009:27:090
2833892993225,4cyclictest4497-21PK-Backend12:35:140
2833892993221,8cyclictest2916144-21cpuspeed_turbos11:10:120
2833892993219,9cyclictest2069-21gnome-shell11:32:580
2833892993216,12cyclictest2900588-21apt-get10:25:080
2833892993215,13cyclictest2908793-21df10:45:120
2833892993215,13cyclictest2907733-40apt-check10:40:580
2833892993215,13cyclictest2891549-40apt-check09:55:580
283389299321,4cyclictest0-21swapper/012:30:510
283389299321,28cyclictest0-21swapper/012:24:540
283389299321,28cyclictest0-21swapper/012:24:540
283389299321,11cyclictest2853715-21idleruntime-cro08:04:590
2833896993117,10cyclictest2875907-21grep09:10:201
283389699311,4cyclictest4497-21PK-Backend12:03:451
283389699311,28cyclictest0-21swapper/111:54:391
283389699311,28cyclictest0-21swapper/109:18:381
283389699311,28cyclictest0-21swapper/108:09:311
283389299314,9cyclictest0-21swapper/007:55:190
283389299311,4cyclictest0-21swapper/009:30:250
283389299311,4cyclictest0-21swapper/009:30:240
283389299311,27cyclictest4497-21PK-Backend09:46:590
283389299311,27cyclictest4497-21PK-Backend09:46:590
283389699306,9cyclictest141rcu_preempt08:50:241
2833896993013,12cyclictest438-20systemd-journal09:55:531
283389699301,11cyclictest4497-21PK-Backend10:36:571
283389299301,5cyclictest0-21swapper/010:05:140
2833896992918,6cyclictest26-21ksoftirqd/111:15:321
2833896992813,10cyclictest2746-21gmain11:27:131
283389299281,5cyclictest0-21swapper/009:20:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional