You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 17:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Fri Aug 07, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
303329178,9sleep20-21swapper/207:05:382
303128969,9sleep00-21swapper/007:05:370
326628471,9sleep30-21swapper/307:08:403
308928366,13sleep10-21swapper/107:06:211
93152770,0sleep20-21swapper/209:17:132
301332600,0sleep00-21swapper/010:47:070
345599301,14cyclictest0-21swapper/009:55:120
346199280,27cyclictest0-21swapper/212:03:492
346399278,14cyclictest39-21ksoftirqd/307:35:143
346199260,26cyclictest0-21swapper/210:22:392
3463992421,3cyclictest39-21ksoftirqd/311:25:363
346199241,22cyclictest0-21swapper/209:10:142
346199240,20cyclictest0-21swapper/210:11:582
346199240,20cyclictest0-21swapper/210:11:572
3463992322,1cyclictest0-21swapper/309:24:013
3463992311,12cyclictest39-21ksoftirqd/309:55:143
3457992320,2cyclictest24228-21chrt10:24:561
345599234,15cyclictest6147-21munin-run12:10:010
3463992221,1cyclictest39-21ksoftirqd/310:30:183
3463992220,1cyclictest319272sleep311:45:493
3463992220,1cyclictest24103-21taskset12:39:553
3461992220,1cyclictest300372sleep211:25:242
3461992219,2cyclictest30630-21wc11:45:202
346199220,21cyclictest0-21swapper/209:34:472
3457992219,2cyclictest17348-21chrt09:21:571
345799220,22cyclictest0-21swapper/111:58:091
345799220,20cyclictest0-21swapper/110:45:561
345599224,17cyclictest0-21swapper/008:45:130
3463992119,1cyclictest17861-21head10:20:203
3463992118,2cyclictest6153-21basename12:10:003
3457992118,2cyclictest30336-21chrt10:09:011
3455992119,1cyclictest3406-21cut11:10:120
3455992119,1cyclictest276902sleep012:22:130
3455992118,2cyclictest20815-21chrt10:22:160
3463992018,1cyclictest31598-21apt10:10:003
346399201,18cyclictest14018-21ssh11:55:153
346199200,19cyclictest0-21swapper/212:10:022
345799202,18cyclictest17458-21ssh11:18:231
3457992019,1cyclictest0-21swapper/110:43:491
345799200,19cyclictest0-21swapper/110:34:421
3455992018,1cyclictest26701-21chrt11:04:540
3455992017,2cyclictest16046-21dnf10:00:170
346399196,11cyclictest0-21swapper/309:53:413
3463991917,1cyclictest0-21swapper/309:30:173
3463991913,5cyclictest28665-21diskmemload11:09:483
3463991911,5cyclictest0-21swapper/310:35:153
3461991918,1cyclictest0-21swapper/212:38:062
346199190,17cyclictest0-21swapper/209:59:442
345799192,16cyclictest19418-21apt-get08:50:131
345799190,18cyclictest0-21swapper/111:40:131
3455991916,2cyclictest18350-21chrt12:35:270
3463991816,2cyclictest39-21ksoftirqd/310:45:323
3463991815,2cyclictest955-21cut12:25:223
3463991815,2cyclictest23765-21ps12:00:213
3463991813,4cyclictest381rcuc/309:49:133
346399180,17cyclictest0-21swapper/309:44:563
346199182,16cyclictest0-21swapper/211:50:352
3461991815,2cyclictest5646-21awk11:30:222
3461991815,2cyclictest10732-21tr10:55:142
346199180,18cyclictest0-21swapper/212:17:142
346199180,18cyclictest0-21swapper/210:34:362
345799180,18cyclictest0-21swapper/110:01:321
345799180,17cyclictest0-21swapper/109:41:161
47332170,0sleep10-21swapper/109:14:591
346399179,7cyclictest0-21swapper/309:10:183
3463991716,1cyclictest0-21swapper/310:12:533
3463991716,1cyclictest0-21swapper/310:12:523
3463991715,1cyclictest263112sleep312:21:073
3463991715,1cyclictest17654-21apt-get10:40:153
3463991714,2cyclictest28665-21diskmemload10:54:143
3463991714,2cyclictest28665-21diskmemload09:26:173
3463991714,2cyclictest16593-21awk12:15:253
3463991711,3cyclictest7954-21ssh12:30:133
346199170,17cyclictest0-21swapper/210:02:542
346199170,17cyclictest0-21swapper/209:43:532
3457991716,1cyclictest0-21swapper/111:13:251
345799170,6cyclictest12378-21grep10:55:261
345799170,2cyclictest141rcu_preempt09:52:471
3463991613,3cyclictest0-21swapper/311:44:003
3463991613,2cyclictest28665-21diskmemload09:19:153
3463991613,2cyclictest21528-21munin-run08:55:013
3463991612,3cyclictest28399-21ssh10:26:403
3463991611,4cyclictest39-21ksoftirqd/308:35:203
3461991615,1cyclictest32-21ksoftirqd/212:26:382
3461991615,1cyclictest0-21swapper/212:34:572
3461991613,2cyclictest3166-21grep10:50:202
346199160,16cyclictest0-21swapper/210:44:462
346199160,16cyclictest0-21swapper/209:50:532
346199160,15cyclictest0-21swapper/210:37:182
3457991614,2cyclictest0-21swapper/109:46:141
345799160,2cyclictest0-21swapper/112:26:401
345599164,6cyclictest5440-21munin-run09:34:590
3455991613,2cyclictest26136-21grep10:25:210
3455991611,0cyclictest0-21swapper/010:13:070
3455991611,0cyclictest0-21swapper/010:13:060
3463991512,2cyclictest0-21swapper/311:14:583
3463991510,5cyclictest0-21swapper/307:10:133
3463991510,4cyclictest475-21in:imuxsock10:20:013
3461991513,2cyclictest32-21ksoftirqd/210:46:182
346199150,1cyclictest0-21swapper/209:48:472
346199150,15cyclictest0-21swapper/212:23:282
346199150,15cyclictest0-21swapper/209:28:042
345799157,6cyclictest15110-21tr12:15:141
345799150,0cyclictest0-21swapper/109:55:121
345599152,1cyclictest141rcu_preempt11:39:160
346399145,8cyclictest0-21swapper/311:03:143
346399142,4cyclictest381rcuc/310:59:053
3463991413,1cyclictest28665-21diskmemload11:35:263
346399141,2cyclictest13311-21apt-config11:55:013
346199144,5cyclictest225-21systemd-journal08:50:002
3461991413,1cyclictest0-21swapper/211:14:512
3461991413,1cyclictest0-21swapper/209:37:552
3461991410,3cyclictest8355-21grep08:25:212
346199140,14cyclictest0-21swapper/212:05:312
345799145,7cyclictest28665-21diskmemload10:20:011
345799144,7cyclictest17744-21cron07:40:001
345799144,6cyclictest4371-21apt-get11:30:141
3457991411,2cyclictest15960-21wc08:40:241
3457991410,3cyclictest26286-21wc10:25:221
345799140,1cyclictest0-21swapper/111:20:031
345799140,14cyclictest0-21swapper/110:12:251
345799140,14cyclictest0-21swapper/110:12:241
345799140,14cyclictest0-21swapper/109:33:561
3455991410,3cyclictest14117-21grep09:20:160
3455991410,2cyclictest8493-21grep12:10:240
345599140,8cyclictest9385-21basename08:30:010
346399139,3cyclictest22014-21grep08:55:133
346399133,9cyclictest0-21swapper/309:38:023
3463991310,2cyclictest7059-21cat08:25:003
3463991310,2cyclictest7059-21cat08:25:003
3463991310,2cyclictest25286-21grep09:00:203
3463991310,2cyclictest20549-21grep07:45:143
346399130,3cyclictest0-21swapper/311:30:553
346399130,12cyclictest23470-21munin-node11:15:133
3461991313,0cyclictest0-21swapper/211:20:132
346199130,11cyclictest0-21swapper/210:19:222
345799139,3cyclictest24286-21grep12:20:181
3457991311,1cyclictest27353-21apache_volume11:05:141
3457991310,2cyclictest7214-21grep11:50:271
3457991310,2cyclictest4765-21wc08:20:001
3457991310,2cyclictest23568-21grep07:50:151
345799130,13cyclictest0-21swapper/110:38:481
345599139,3cyclictest25288-21cut09:00:190
345599139,2cyclictest5650-21sed08:20:170
345599139,2cyclictest5650-21sed08:20:160
345599139,2cyclictest32439-21ls09:50:240
345599139,2cyclictest26338-21cat07:55:210
345599135,1cyclictest0-21swapper/010:52:230
345599132,6cyclictest23903-21sendmail08:59:590
3455991311,1cyclictest8896-21chrt10:35:010
3455991310,2cyclictest7534-21cat12:30:000
3455991310,2cyclictest28548-21cut11:25:140
3455991310,2cyclictest16062-21grep08:40:240
3455991310,2cyclictest13162-21grep08:35:220
345599130,9cyclictest22914-21dump-pmu-power12:20:010
346399129,2cyclictest8718-21tr07:20:153
346399129,2cyclictest6296-21grep07:15:143
346399129,2cyclictest16167-21grep08:40:243
346399129,2cyclictest16153-21grep10:00:163
346399129,2cyclictest1520-21cat08:10:253
346399128,2cyclictest0-21swapper/308:00:253
346399121,8cyclictest6602-21apache_accesses12:10:103
346199129,2cyclictest9723-21wc07:20:222
346199129,2cyclictest3105-21sort08:15:182
346199129,2cyclictest26309-21perf09:05:002
346199129,2cyclictest22263-21cat11:40:202
346199129,2cyclictest1750-21tail08:10:252
346199129,2cyclictest15226-21grep08:40:152
346199129,2cyclictest11346-21sed08:30:272
346199128,2cyclictest28527-21cut08:00:152
346199120,12cyclictest0-21swapper/211:56:152
345799129,2cyclictest8239-21tr12:30:151
345799129,2cyclictest6591-21ssh09:15:211
345799129,2cyclictest32157-21ssh12:05:221
345799129,2cyclictest19456-21sh11:00:141
345799128,3cyclictest17173-21grep12:35:161
345799127,4cyclictest14146-21taskset08:39:521
345799127,2cyclictest26252-21taskset09:04:381
345799124,4cyclictest225-21systemd-journal08:10:011
345799123,3cyclictest225-21systemd-journal10:55:001
3457991211,1cyclictest38150irq/131-snd_hda09:38:301
3457991210,1cyclictest28665-21diskmemload12:13:221
345599129,2cyclictest9665-21cut08:30:150
345599129,2cyclictest307-21grep08:10:150
345599129,2cyclictest29869-21grep09:10:160
345599129,2cyclictest2720-21tr08:15:150
345599129,2cyclictest20761-21cut11:20:170
345599129,2cyclictest20138-21cron07:44:590
345599129,2cyclictest18228-21cron11:00:000
345599128,3cyclictest2766-21latency_hist11:10:000
345599128,3cyclictest10129-21sed07:20:260
345599121,5cyclictest15404-21perf07:34:590
3455991210,1cyclictest274282sleep011:43:490
346399119,1cyclictest18188-21cat08:45:223
346399118,2cyclictest30248-21basename08:05:183
346399118,2cyclictest28257-21head09:05:243
346399118,2cyclictest19865-21ssh11:20:043
346399118,2cyclictest14052-21head07:30:183
346399118,2cyclictest10344-21head08:30:173
346399111,7cyclictest39-21ksoftirqd/307:55:213
3463991110,1cyclictest0-21swapper/307:40:123
346199118,2cyclictest27318-21chrt07:58:002
346199118,2cyclictest26883-21ssh10:25:272
346199118,2cyclictest23570-21ssh10:05:052
346199118,2cyclictest22496-21head08:55:152
346199118,2cyclictest21064-21sed08:50:262
346199118,2cyclictest18709-21ssh11:00:012
346199118,2cyclictest17287-21awk07:35:272
346199118,2cyclictest15278-21ssh11:35:502
346199118,2cyclictest15129-21sh09:20:262
346199118,2cyclictest14088-21ssh11:15:452
346199111,9cyclictest4747-21missed_timers07:10:192
345799118,2cyclictest8040-21cut08:25:181
345799118,2cyclictest7713-21tail07:15:261
345799118,2cyclictest3541-21ssh11:29:471
345799118,2cyclictest30796-21ssh11:45:211
345799118,2cyclictest29877-21cron08:05:001
345799118,2cyclictest28738-21ssh09:29:241
345799118,2cyclictest22169-21ssh12:00:071
345799118,2cyclictest21030-21cut07:45:161
345799118,2cyclictest18141-21grep07:40:121
345799118,2cyclictest18054-21awk08:45:211
345799118,2cyclictest1718-21sed08:10:251
345799118,2cyclictest15668-21ssh11:36:081
345799118,2cyclictest12833-21chrt07:27:581
345799118,2cyclictest10365-21grep08:30:171
345799117,3cyclictest23912-21run-parts08:59:591
345799117,3cyclictest13537-21grep07:30:151
345599118,2cyclictest9029-21ls12:30:200
345599118,2cyclictest6233-21tr09:35:160
345599118,2cyclictest6106-21grep11:50:170
345599118,2cyclictest4936-21rm09:15:090
345599118,2cyclictest4179-21ssh11:30:070
345599118,2cyclictest31362-21grep11:45:270
345599118,2cyclictest29901-21sleep08:05:000
345599118,2cyclictest28568-21ssh10:07:380
345599118,2cyclictest28097-21ssh12:03:220
345599118,2cyclictest24910-21ls07:54:590
345599118,2cyclictest24636-21seq09:26:090
345599118,2cyclictest20167-21grep08:50:170
345599118,2cyclictest20138-21ssh10:40:510
345599118,2cyclictest19070-21ssh09:42:360
345599118,2cyclictest15354-21ls11:55:240
345599118,2cyclictest13404-21ls11:15:250
345599118,2cyclictest1027-21tr10:30:140
345599118,2cyclictest10005-21awk10:15:280
345599117,3cyclictest11516-21sort07:25:170
345599111,7cyclictest30616-21latency_hist09:50:010
346399108,1cyclictest92542sleep308:28:463
346399108,1cyclictest2789-21cat08:15:163
346399107,2cyclictest23223-21cat07:50:133
346199108,1cyclictest7306-21awk07:15:212
346199108,1cyclictest32014-21chrt08:07:022
346199108,1cyclictest285502sleep209:08:392
346199107,2cyclictest6475-21date08:20:262
346199107,2cyclictest6475-21date08:20:252
346199107,2cyclictest27076-21ssh11:05:022
346199107,2cyclictest22784-21apache_processe07:50:102
346199107,2cyclictest12659-21fschecks_count08:35:202
346199106,3cyclictest21117-21grep07:45:172
345799108,1cyclictest8420-21chrt07:20:011
345799108,1cyclictest5166-21wc07:10:251
345799108,1cyclictest26870-21wc09:05:121
345799107,2cyclictest6751-21taskset08:22:051
345799107,2cyclictest6751-21taskset08:22:041
345799106,3cyclictest25414-21grep07:55:151
345599108,1cyclictest5779-21chrt07:14:540
345599108,1cyclictest31500-21basename08:05:260
345599108,1cyclictest20389-21basename07:45:110
345599108,1cyclictest15948-21cat07:35:150
345599107,2cyclictest7298-21grep07:15:220
34639991,3cyclictest10979-21tr07:25:143
34619997,1cyclictest13373-21cut07:30:142
34619997,1cyclictest12243-21grep07:25:222
34619996,2cyclictest18290-21tr07:40:132
34559994,2cyclictest225-21systemd-journal09:05:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional