You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 23:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Thu Jan 01, 2026 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
197399960,9rtkit-daemon1972-21rtkit-daemon07:29:041
197399920,9rtkit-daemon1972-21rtkit-daemon12:06:412
197399900,8rtkit-daemon1972-21rtkit-daemon08:45:562
197399900,8rtkit-daemon1972-21rtkit-daemon07:21:371
30841632880,6sleep04497-21PK-Backend11:25:140
197399880,9rtkit-daemon1972-21rtkit-daemon11:51:102
197399880,9rtkit-daemon1972-21rtkit-daemon11:48:362
197399820,10rtkit-daemon3021225-21irqstats08:25:213
197399810,71rtkit-daemon0-21swapper/107:07:251
30339442790,5sleep20-21swapper/209:00:332
299337527941,31sleep30-21swapper/307:07:103
30441422760,12sleep2141rcu_preempt09:30:312
30639132740,4chrt3063914-21/usr/sbin/munin10:25:300
299288227244,20sleep00-21swapper/007:05:230
299334426847,14sleep20-21swapper/207:06:452
299374899603,15cyclictest3002537-21chrt07:31:370
197399600,9rtkit-daemon1972-21rtkit-daemon10:25:153
197399600,9rtkit-daemon1972-21rtkit-daemon08:54:152
197399600,9rtkit-daemon1972-21rtkit-daemon08:54:142
299374899593,13cyclictest2997273-21chrt07:15:430
197399590,9rtkit-daemon1972-21rtkit-daemon11:02:243
299374899588,8cyclictest2999354-21munin-run07:25:000
299374899583,13cyclictest3095991-21chrt11:57:420
197399580,9rtkit-daemon1972-21rtkit-daemon08:56:363
299374899575,10cyclictest3072911-21chrt10:53:510
299374899574,32cyclictest3055439-21taskset10:02:030
299374899574,10cyclictest3037606-21chrt09:12:500
197399570,9rtkit-daemon1972-21rtkit-daemon11:35:423
197399570,7rtkit-daemon1972-21rtkit-daemon07:51:470
299374899565,11cyclictest3099354-21chrt12:06:270
299374899563,10cyclictest3040986-21chrt09:21:440
299374899562,11cyclictest3011331-21chrt07:57:450
197399560,9rtkit-daemon1972-21rtkit-daemon11:47:273
197399560,9rtkit-daemon1972-21rtkit-daemon10:24:433
299374899553,10cyclictest3007939-21chrt07:48:390
197399550,9rtkit-daemon1972-21rtkit-daemon11:53:263
197399550,9rtkit-daemon1972-21rtkit-daemon10:35:573
197399550,9rtkit-daemon1972-21rtkit-daemon09:05:363
197399550,8rtkit-daemon1972-21rtkit-daemon08:38:113
197399550,8rtkit-daemon1972-21rtkit-daemon08:31:463
299374899543,10cyclictest3012971-21chrt08:01:350
197399540,9rtkit-daemon1972-21rtkit-daemon11:29:582
197399540,8rtkit-daemon1972-21rtkit-daemon12:03:482
197399540,8rtkit-daemon1972-21rtkit-daemon07:32:421
299374899538,27cyclictest3076785-21munin-run11:05:000
299374899534,31cyclictest3082169-21latency_hist11:20:010
299374899534,31cyclictest3082169-21latency_hist11:20:010
299374899532,9cyclictest3032360-21chrt08:57:090
2993748995314,13cyclictest3034329-21latency_hist09:05:010
299374899530,10cyclictest3089177-21chrt11:39:550
2993753995226,10cyclictest26-21ksoftirqd/111:50:011
299374899521,11cyclictest3051549-21chrt09:50:410
299374899521,10cyclictest3023845-21chrt08:33:550
197399520,8rtkit-daemon1972-21rtkit-daemon11:44:093
30969402510,19sleep342-21ksoftirqd/312:00:223
30610672510,13sleep3411rcuc/310:20:023
2993758995127,19cyclictest3033346-21irqstats09:00:213
299374899512,10cyclictest3020208-21chrt08:21:430
197399510,8rtkit-daemon1972-21rtkit-daemon10:34:083
2993753994918,9cyclictest26-21ksoftirqd/112:06:271
2993753994912,15cyclictest26-21ksoftirqd/108:20:201
299374899491,13cyclictest3094318-21chrt11:53:270
299374899488,12cyclictest3070196-21df_inode10:45:170
299374899482,18cyclictest3091117-21cut11:45:020
299374899481,14cyclictest3074909-21cat11:00:010
2993753994615,16cyclictest26-21ksoftirqd/107:48:391
2993753994611,20cyclictest26-21ksoftirqd/109:15:001
299374899469,10cyclictest3029492-21ls08:50:140
299374899469,10cyclictest3029492-21ls08:50:140
299374899468,16cyclictest3039291-21chrt09:17:130
299374899468,16cyclictest3039291-21chrt09:17:130
299374899468,14cyclictest3060340-21sed10:15:290
299374899467,14cyclictest3103471-21ls12:20:140
299374899466,15cyclictest3049617-21chrt09:46:270
2993755994532,9cyclictest2080-21gdbus11:38:082
2993753994519,13cyclictest26-21ksoftirqd/107:15:431
2993753994515,16cyclictest26-21ksoftirqd/109:50:411
299374899456,15cyclictest0-21swapper/009:25:300
299374899456,13cyclictest3110061-21unin-run12:40:010
197399450,4rtkit-daemon4497-21PK-Backend09:57:270
2993753994419,13cyclictest26-21ksoftirqd/111:57:421
2993753994410,20cyclictest26-21ksoftirqd/107:15:021
299374899449,16cyclictest3014388-21python308:05:270
299374899446,16cyclictest3083378-21chrt11:21:000
2993748994415,9cyclictest0-21swapper/012:15:250
299374899441,12cyclictest2746-21gmain08:47:570
299375399438,20cyclictest26-21ksoftirqd/110:25:001
299375399437,19cyclictest26-21ksoftirqd/111:05:001
2993753994319,8cyclictest26-21ksoftirqd/111:22:411
2993753994319,10cyclictest26-21ksoftirqd/110:53:511
2993753994316,14cyclictest26-21ksoftirqd/110:05:101
299375399431,15cyclictest4497-21PK-Backend11:00:001
2993753994310,18cyclictest26-21ksoftirqd/112:05:011
299374899437,12cyclictest3092457-21chrt11:47:360
299374899437,12cyclictest3062373-21chrt10:21:520
299374899437,11cyclictest3086597-21grep11:30:290
299374899437,11cyclictest3016746-21munin-run08:15:010
299374899437,11cyclictest3006206-21chrt07:43:370
299374899437,11cyclictest3006206-21chrt07:43:360
299374899436,12cyclictest3106485-21chrt12:29:090
30966992420,7sleep00-21swapper/012:00:140
30736502420,6sleep342-21ksoftirqd/310:55:153
299375399429,19cyclictest26-21ksoftirqd/108:35:011
299375399428,19cyclictest26-21ksoftirqd/111:45:001
299375399428,19cyclictest26-21ksoftirqd/109:50:011
2993753994216,13cyclictest26-21ksoftirqd/109:21:441
299374899428,12cyclictest3079260-21irqstats11:10:200
299374899427,19cyclictest3018914-21latency_hist08:20:020
299374899426,11cyclictest3026703-21awk08:40:260
299374899421,10cyclictest13-21ksoftirqd/012:10:230
299374899421,10cyclictest13-21ksoftirqd/010:30:240
2993758994119,16cyclictest42-21ksoftirqd/311:15:233
2993758994119,16cyclictest42-21ksoftirqd/311:15:223
2993755994123,14cyclictest3044986-21apt09:35:132
299375599411,9cyclictest924-21NetworkManager07:20:002
2993755994111,26cyclictest3048166-21cron09:45:002
299375399418,11cyclictest3034485-21cat09:05:021
299375399417,17cyclictest3093152-21tr11:50:161
2993753994116,12cyclictest26-21ksoftirqd/107:57:451
2993753994113,13cyclictest141rcu_preempt12:32:391
299374899417,11cyclictest3068615-21latency10:40:230
299374899414,15cyclictest3058865-21taskset10:11:200
2993748994111,16cyclictest13-21ksoftirqd/012:30:030
2993748994110,16cyclictest13-21ksoftirqd/007:12:480
2993758994018,17cyclictest42-21ksoftirqd/312:05:233
2993755994015,9cyclictest34-21ksoftirqd/212:13:172
2993753994028,9cyclictest26-21ksoftirqd/108:10:001
2993753994015,8cyclictest26-21ksoftirqd/108:15:001
2993748994012,13cyclictest4497-21PK-Backend07:38:170
30872152390,5sleep1251rcuc/111:35:001
30435232390,6sleep3411rcuc/309:30:163
2993758993923,11cyclictest3002540-21sshd07:31:393
2993758993922,5cyclictest42-21ksoftirqd/309:20:003
2993758993922,5cyclictest42-21ksoftirqd/309:20:003
2993758993918,16cyclictest42-21ksoftirqd/307:15:233
299375899391,33cyclictest2995439-21munin-node07:15:023
299375599397,11cyclictest141rcu_preempt10:00:002
299375399395,19cyclictest26-21ksoftirqd/111:15:011
299375399393,20cyclictest26-21ksoftirqd/110:15:001
299375399393,19cyclictest26-21ksoftirqd/110:05:011
2993753993916,9cyclictest26-21ksoftirqd/108:01:361
2993753993914,12cyclictest26-21ksoftirqd/111:39:551
2993753993910,16cyclictest26-21ksoftirqd/108:38:481
2993748993916,19cyclictest3057051-21users10:05:300
2993758993820,6cyclictest42-21ksoftirqd/310:45:003
2993758993820,14cyclictest3101405-21pluginstate12:15:013
299375399385,10cyclictest26-21ksoftirqd/107:50:181
299375399384,19cyclictest26-21ksoftirqd/109:35:291
2993753993821,13cyclictest3047798-21users09:40:311
2993753993821,13cyclictest3035374-21pluginstate09:05:251
2993753993819,5cyclictest141rcu_preempt10:30:431
2993753993817,6cyclictest141rcu_preempt08:55:011
2993753993817,6cyclictest141rcu_preempt08:55:011
2993753993811,15cyclictest26-21ksoftirqd/112:19:391
2993753993810,9cyclictest26-21ksoftirqd/109:27:511
299374899382,14cyclictest13-21ksoftirqd/009:42:120
2993748993810,15cyclictest13-21ksoftirqd/009:39:220
2993758993723,11cyclictest42-21ksoftirqd/312:35:283
2993758993722,11cyclictest42-21ksoftirqd/309:10:263
2993758993721,12cyclictest3012900-21sshd08:00:393
2993758993716,16cyclictest42-21ksoftirqd/310:00:293
299375899371,12cyclictest141rcu_preempt09:50:023
299375599372,7cyclictest3056377-21smartctl10:05:182
299375599372,7cyclictest3031500-21df08:55:152
2993755993727,6cyclictest3060899-21latency_hist10:20:002
299375599372,13cyclictest3107216-21diskstats12:30:182
2993755993720,13cyclictest3082756-21cut11:20:212
2993753993721,13cyclictest26-21ksoftirqd/111:16:481
2993753993721,13cyclictest26-21ksoftirqd/111:16:481
2993753993715,9cyclictest26-21ksoftirqd/108:57:091
2993753993711,8cyclictest26-21ksoftirqd/108:25:141
299374899376,10cyclictest13-21ksoftirqd/008:27:230
299374899371,12cyclictest13-21ksoftirqd/009:32:130
30187692361,11sleep23018787-21sort08:20:012
299375899369,5cyclictest141rcu_preempt08:05:293
299375899368,24cyclictest3087171-21sh11:35:003
2993758993621,11cyclictest42-21ksoftirqd/312:25:163
2993758993620,12cyclictest3003817-21ntpq07:35:243
2993758993620,11cyclictest438-20systemd-journal09:21:393
2993758993619,13cyclictest2997871-21basename07:20:113
2993758993617,14cyclictest3027673-21apt08:45:123
299375599369,5cyclictest141rcu_preempt11:45:002
299375599366,15cyclictest34-21ksoftirqd/208:00:002
2993755993625,7cyclictest3102355-21ntp_kernel_err12:15:232
2993755993622,10cyclictest2069-21gnome-shell10:02:092
2993755993619,12cyclictest3065177-21sed10:30:222
2993755993612,6cyclictest141rcu_preempt11:15:272
2993755993612,6cyclictest141rcu_preempt11:15:272
299375599361,11cyclictest924-21NetworkManager08:14:552
299375399365,9cyclictest26-21ksoftirqd/112:40:011
2993753993621,11cyclictest3044341-21vmstat09:30:331
2993753993620,5cyclictest26-21ksoftirqd/107:35:161
2993753993611,9cyclictest26-21ksoftirqd/112:25:131
299375399361,13cyclictest4497-21PK-Backend12:10:221
2993753993610,8cyclictest26-21ksoftirqd/110:37:571
299374899366,9cyclictest0-21swapper/011:05:510
299374899362,12cyclictest13-21ksoftirqd/010:39:190
299374899362,12cyclictest13-21ksoftirqd/008:38:050
299374899361,13cyclictest13-21ksoftirqd/009:07:430
30748172352,8sleep234-21ksoftirqd/211:00:012
30052202350,6sleep30-21swapper/307:40:133
30052202350,6sleep30-21swapper/307:40:133
2993758993524,7cyclictest3082433-21grep11:20:143
2993758993522,9cyclictest2069-21gnome-shell09:43:093
2993758993522,8cyclictest3029887-21irqstats08:50:213
2993758993522,8cyclictest3029887-21irqstats08:50:213
2993758993522,10cyclictest42-21ksoftirqd/312:30:253
2993758993520,11cyclictest3008807-21awk07:50:193
2993758993517,13cyclictest3045735-21ntpq09:35:253
299375599359,6cyclictest141rcu_preempt10:55:012
299375599357,5cyclictest141rcu_preempt11:13:062
299375599354,27cyclictest3104107-21grep12:20:252
2993755993520,11cyclictest3002084-21ntp_kernel_pll_07:30:272
2993755993519,12cyclictest3074150-21python310:55:282
299375599351,8cyclictest0-21swapper/210:12:252
299375399358,6cyclictest141rcu_preempt07:43:521
299375399358,6cyclictest141rcu_preempt07:43:521
299375399356,14cyclictest26-21ksoftirqd/110:20:001
299375399353,17cyclictest3063399-21tr10:25:211
299375399353,15cyclictest0-21swapper/110:47:071
299375399353,15cyclictest0-21swapper/108:18:291
2993753993518,13cyclictest26-21ksoftirqd/108:49:581
2993753993514,8cyclictest26-21ksoftirqd/111:06:501
299375399351,14cyclictest4497-21PK-Backend09:20:001
299375399351,14cyclictest4497-21PK-Backend09:20:001
299375399351,11cyclictest0-21swapper/109:55:241
299374899356,8cyclictest0-21swapper/007:27:220
197399350,28rtkit-daemon0-21swapper/308:41:423
299375899342,9cyclictest3019412-21hddtemp_smartct08:20:173
2993758993419,11cyclictest42-21ksoftirqd/308:00:003
2993758993419,11cyclictest3102516-21seq12:15:293
2993758993419,11cyclictest3095669-21cat11:55:303
2993758993419,11cyclictest3079348-21sed11:10:213
2993758993419,10cyclictest3052742-21sed09:55:213
2993758993418,12cyclictest3085047-21cat11:25:313
299375899341,30cyclictest4497-21PK-Backend09:54:293
299375899341,29cyclictest4497-21PK-Backend07:28:463
299375599348,5cyclictest141rcu_preempt10:40:002
299375599342,8cyclictest924-21NetworkManager08:20:152
2993755993426,5cyclictest0-21swapper/207:41:272
2993755993426,5cyclictest0-21swapper/207:41:272
2993755993421,10cyclictest34-21ksoftirqd/212:35:132
2993755993418,12cyclictest3003647-21cat07:35:212
2993755993416,14cyclictest3063613-21ntpq10:25:242
299375599341,17cyclictest3086189-21kernelversion11:30:202
299375599341,17cyclictest3062118-21cat10:20:332
299375599341,14cyclictest3071274-21cron10:50:002
299375399341,18cyclictest3104407-21timerwakeupswit12:20:311
299375399341,18cyclictest3069430-21seq10:43:371
2993753993410,8cyclictest26-21ksoftirqd/108:42:081
299375899336,23cyclictest3103894-21/usr/sbin/munin12:20:223
2993758993322,7cyclictest3078515-21idleruntime-cro11:10:003
299375899331,28cyclictest0-21swapper/310:50:093
299375599338,9cyclictest2994749-21cut07:10:272
299375599337,13cyclictest141rcu_preempt08:40:182
299375599336,5cyclictest141rcu_preempt08:00:272
299375599332,8cyclictest924-21NetworkManager09:20:242
2993755993320,9cyclictest34-21ksoftirqd/210:40:122
2993755993318,11cyclictest3049611-21seq09:46:222
2993755993316,12cyclictest3105227-21sort12:25:142
2993755993311,6cyclictest141rcu_preempt07:50:002
2993758993222,6cyclictest42-21ksoftirqd/310:10:113
299375899321,4cyclictest0-21swapper/308:10:213
299375899321,27cyclictest0-21swapper/307:50:023
299375899321,10cyclictest3041526-21/usr/sbin/munin09:25:133
299375599328,9cyclictest141rcu_preempt11:55:242
2993755993224,5cyclictest4497-21PK-Backend09:05:532
299375599321,9cyclictest4497-21PK-Backend07:22:502
2993755993218,11cyclictest3014206-21netstat08:05:222
2993755993217,11cyclictest3077653-21cat11:05:222
2993755993217,11cyclictest3023472-21grep08:30:312
2993755993215,13cyclictest1-21systemd08:28:312
299375599321,4cyclictest4497-21PK-Backend09:50:362
299375599321,4cyclictest4497-21PK-Backend09:10:162
299375599321,16cyclictest3001088-21turbostat07:30:002
2993755993210,6cyclictest141rcu_preempt07:50:242
299375399324,9cyclictest141rcu_preempt11:26:231
299375899312,24cyclictest3056339-21df_inode10:05:143
2993758993117,10cyclictest3017762-21grep08:15:223
299375899311,27cyclictest0-21swapper/310:45:093
299375599311,6cyclictest4497-21PK-Backend09:25:282
2993755993114,13cyclictest3025608-21seq08:39:222
299375599311,28cyclictest0-21swapper/209:19:012
299375599311,28cyclictest0-21swapper/209:19:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional