You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-27 - 22:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Aug 27, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1134729683,9sleep00-21swapper/007:07:060
1129929061,9sleep20-21swapper/207:06:262
1006028673,9sleep30-21swapper/307:05:163
1141528364,15sleep10-21swapper/107:07:591
131912550,0sleep00-21swapper/012:09:150
1166199270,24cyclictest0-21swapper/211:19:492
1166399251,19cyclictest0-21swapper/311:20:133
1166199240,24cyclictest0-21swapper/212:10:572
1165799240,21cyclictest0-21swapper/109:17:471
11663992320,3cyclictest0-21swapper/310:37:043
11661992322,1cyclictest0-21swapper/207:45:142
1165799230,23cyclictest0-21swapper/109:22:031
1165799221,19cyclictest13569-21df_abs08:20:141
1166399210,0cyclictest0-21swapper/310:18:523
1166199210,21cyclictest0-21swapper/212:20:042
1166199210,21cyclictest0-21swapper/211:02:152
11657992120,1cyclictest0-21swapper/111:45:141
1166199200,18cyclictest0-21swapper/209:26:212
11657992016,3cyclictest25-21ksoftirqd/112:32:511
11663991918,1cyclictest0-21swapper/312:37:273
1166199190,19cyclictest0-21swapper/211:39:032
1165799190,16cyclictest0-21swapper/109:56:251
1165499195,7cyclictest16612-21munin-run07:20:000
1166399180,2cyclictest0-21swapper/311:44:073
1166399180,18cyclictest28867-21kworker/3:011:26:333
1165499189,2cyclictest225-21systemd-journal10:40:000
11663991715,2cyclictest0-21swapper/310:54:193
1166399170,17cyclictest28867-21kworker/3:011:07:033
1166199170,2cyclictest0-21swapper/209:50:142
1166199170,17cyclictest0-21swapper/209:12:422
1165799170,17cyclictest0-21swapper/111:18:201
24072160,1sleep20-21swapper/211:43:212
1166399160,4cyclictest0-21swapper/310:10:013
1166399160,4cyclictest0-21swapper/310:10:013
11657991613,2cyclictest1859-21cron09:05:001
1165799160,16cyclictest0-21swapper/112:03:381
11654991611,4cyclictest4257-21diskmemload12:30:210
11663991510,3cyclictest8033-21wc12:05:243
1166199157,5cyclictest537-21snmpd10:38:542
1166199156,5cyclictest2630-21tr11:05:132
1166199155,1cyclictest24911-21apt-get10:40:112
11661991510,4cyclictest25330-21apache210:27:362
1166199150,1cyclictest0-21swapper/212:01:222
1166199150,13cyclictest0-21swapper/212:35:122
1165799155,10cyclictest4257-21diskmemload10:52:181
1165799152,6cyclictest0-21swapper/109:43:561
11657991513,1cyclictest29329-21open_inodes09:25:221
1165799150,1cyclictest0-21swapper/112:06:121
11654991514,1cyclictest0-21swapper/010:55:310
11654991511,3cyclictest24865-21mailstats12:15:220
253512140,1sleep30-21swapper/310:20:263
1166399147,5cyclictest39-21ksoftirqd/309:28:293
11663991411,2cyclictest5424-21wc09:30:223
11663991410,2cyclictest9923-21sed08:10:233
1166199148,2cyclictest4257-21diskmemload09:15:132
1166199141,1cyclictest141rcu_preempt10:12:512
11661991410,3cyclictest7663-21grep12:05:212
11661991410,3cyclictest15198-21wc11:50:262
1166199140,1cyclictest0-21swapper/209:59:512
1165799142,11cyclictest13563-21/usr/sbin/munin11:50:131
11657991414,0cyclictest0-21swapper/112:18:031
11657991410,3cyclictest22618-21wc07:30:181
1165799140,3cyclictest0-21swapper/109:13:131
1165799140,2cyclictest0-21swapper/112:39:151
1165499144,9cyclictest20599-21apt-get11:35:120
11654991411,2cyclictest20034-21sed07:25:170
1165499140,13cyclictest25955-21taskset10:40:160
1166399139,3cyclictest21353-21cron07:29:593
1166399139,3cyclictest1114-21wc10:25:223
1166399138,3cyclictest20912-21grep09:20:003
11663991310,2cyclictest7289-21latency_hist12:25:013
11663991310,2cyclictest6627-21tr09:50:223
11663991310,2cyclictest6182-21tr09:10:233
11663991310,2cyclictest3291-21perf08:00:003
11663991310,2cyclictest27221-21grep07:40:163
11663991310,2cyclictest25262-21head12:15:263
1166199139,3cyclictest23616-21wc11:55:262
1166199139,3cyclictest23616-21wc11:55:252
1166199139,3cyclictest17177-21grep08:25:262
1166199139,3cyclictest12109-21wc08:15:262
1166199139,3cyclictest12109-21wc08:15:262
1166199137,5cyclictest32-21ksoftirqd/209:21:272
11661991313,0cyclictest0-21swapper/211:25:132
11661991311,1cyclictest0-21swapper/210:01:552
11661991310,2cyclictest30769-21wc08:55:182
11661991310,2cyclictest25526-21wc10:20:262
1166199130,13cyclictest0-21swapper/210:18:162
1166199130,13cyclictest0-21swapper/209:45:452
1165799139,3cyclictest8493-21ssh12:25:151
1165799139,3cyclictest18955-21ssh10:55:181
1165799139,1cyclictest18710-21cron11:14:591
1165799138,4cyclictest25348-21apache211:36:251
1165799138,4cyclictest1831-21ssh11:04:591
1165799133,7cyclictest38150irq/131-snd_hda08:40:131
1165799131,2cyclictest20223-21latency_hist11:35:001
1165799131,1cyclictest0-21swapper/112:20:131
1165799131,12cyclictest4257-21diskmemload11:44:251
11657991310,2cyclictest13273-21cat09:55:011
1165799130,1cyclictest0-21swapper/108:45:131
1165799130,13cyclictest0-21swapper/110:25:161
1165499139,3cyclictest8245-21ps10:10:230
1165499139,3cyclictest5843-21grep09:30:260
1165499132,6cyclictest26209-21sendmail07:40:000
11654991311,1cyclictest30380-21ls07:45:260
11654991310,2cyclictest29620-21cat08:55:000
11654991310,2cyclictest28558-21cron07:44:590
11654991310,2cyclictest20992-21head08:35:170
11654991310,2cyclictest14785-21sed08:20:250
11654991310,2cyclictest0-21swapper/011:25:260
1165499130,2cyclictest0-21swapper/010:01:560
1165499130,1cyclictest141rcu_preempt10:33:240
1166399129,2cyclictest9003-21cut12:25:183
1166399129,2cyclictest5559-21expr11:45:153
1166399129,2cyclictest31879-21cron09:00:003
1166399129,2cyclictest30065-21grep09:45:183
1166399129,2cyclictest30061-21cat10:05:003
1166399129,2cyclictest26858-21grep10:40:233
1166399129,2cyclictest22919-21cat12:15:013
1166399129,2cyclictest21850-21grep08:35:263
1166399128,3cyclictest23817-21ssh11:55:273
1166399128,3cyclictest23817-21ssh11:55:263
1166399128,3cyclictest20647-21grep11:35:143
1166399128,2cyclictest31495-21grep07:50:153
1166399128,2cyclictest18155-21grep10:55:143
1166399126,2cyclictest225-21systemd-journal08:27:543
11663991212,0cyclictest0-21swapper/310:48:183
1166399120,11cyclictest14750-21sed08:20:263
1166199129,2cyclictest8041-21sh08:10:002
1166199129,2cyclictest29107-21grep08:50:262
1166199129,2cyclictest28568-21sh07:45:002
1166199129,2cyclictest22354-21grep07:30:162
1166199129,2cyclictest19729-21grep10:55:252
1166199129,2cyclictest17598-21grep07:20:172
1166199129,2cyclictest16817-21df12:30:152
1166199129,2cyclictest13530-21sed07:10:252
1166199128,2cyclictest15164-21sort07:15:162
1166199123,4cyclictest537-21snmpd09:35:372
1166199122,3cyclictest32682-21grep10:05:282
1166199122,3cyclictest32682-21grep10:05:282
1166199120,2cyclictest0-21swapper/210:52:562
1166199120,1cyclictest0-21swapper/212:29:432
1166199120,11cyclictest24043-21wc08:40:252
1165799129,2cyclictest4655-21cat08:00:201
1165799129,2cyclictest3444-21cat11:05:181
1165799129,2cyclictest26839-21grep10:40:231
1165799129,2cyclictest23133-21chrt11:55:211
1165799129,2cyclictest23133-21chrt11:55:201
1165799129,2cyclictest20239-21cut08:35:121
1165799129,2cyclictest19470-21sed08:30:241
1165799129,2cyclictest11332-21uniq08:15:171
1165799129,2cyclictest11332-21uniq08:15:171
1165799128,3cyclictest19361-21grep07:25:141
1165799128,3cyclictest14905-21sh10:15:021
1165799128,3cyclictest14241-21sed09:35:251
1165799127,4cyclictest2695-21ssh10:07:161
1165799127,4cyclictest2695-21ssh10:07:151
1165799123,7cyclictest0-21swapper/108:25:131
1165799122,8cyclictest8021-21turbostat.cron08:09:551
11657991210,1cyclictest24461-21grep10:20:181
1165499129,2cyclictest5181-21tr11:45:150
1165499129,2cyclictest30920-21tr08:55:210
1165499129,2cyclictest30752-21ssh11:02:160
1165499129,2cyclictest29400-21tr11:40:140
1165499129,2cyclictest29325-21basename09:45:150
1165499129,2cyclictest28712-21turbostat.cron09:45:000
1165499129,2cyclictest27974-21sh09:25:130
1165499129,2cyclictest27883-21basename11:20:150
1165499129,2cyclictest22876-21tr08:40:140
1165499129,2cyclictest14851-21cut09:55:190
1165499129,2cyclictest14019-21cat09:35:250
1165499128,3cyclictest6166-21sed09:10:220
1165499128,3cyclictest459-21grep10:25:160
1165499128,3cyclictest26646-21grep08:45:250
1165499128,3cyclictest0-21swapper/010:22:540
1165499123,8cyclictest25504-21runrttasks11:56:430
1165499123,8cyclictest25504-21runrttasks11:56:430
1165499123,8cyclictest12166-21/usr/sbin/munin11:10:210
1165499120,4cyclictest0-21swapper/012:39:230
1165499120,4cyclictest0-21swapper/009:23:450
1165499120,1cyclictest0-21swapper/011:50:370
1165499120,11cyclictest16167-21tail08:25:170
1165499120,0cyclictest0-21swapper/012:27:330
229722110,0sleep30-21swapper/309:40:353
1166399118,2cyclictest7991-21cat10:10:203
1166399118,2cyclictest7798-21chrt08:07:033
1166399118,2cyclictest763-21ssh12:01:083
1166399118,2cyclictest4081-21taskset09:08:053
1166399118,2cyclictest27848-21expr08:50:143
1166399118,2cyclictest26520-21grep11:00:143
1166399118,2cyclictest25728-21tune2fs08:45:173
1166399118,2cyclictest24785-21sed07:35:173
1166399118,2cyclictest21385-21tr09:21:243
1166399118,2cyclictest20550-21ssh11:15:213
1166399118,2cyclictest19923-21ssh11:53:563
1166399118,2cyclictest17046-21cut12:30:163
1166399118,2cyclictest13306-21cat11:30:193
1166399118,2cyclictest12900-21ssh09:35:143
1166399118,2cyclictest12866-21ssh10:32:353
1166399118,2cyclictest12343-21sed08:15:273
1166399118,2cyclictest12343-21sed08:15:273
1166399117,3cyclictest22567-21tr07:30:183
1166399117,2cyclictest15202-21sensors09:55:223
1166399114,6cyclictest28802-21/usr/sbin/munin07:45:213
1166399110,11cyclictest0-21swapper/311:12:303
1166399110,0cyclictest0-21swapper/307:15:143
1166199118,2cyclictest9016-21ssh11:46:592
1166199118,2cyclictest8567-21tr08:10:122
1166199118,2cyclictest8382-21wc10:30:142
1166199118,2cyclictest3899-21ssh09:30:102
1166199118,2cyclictest32-21ksoftirqd/207:55:232
1166199118,2cyclictest28058-21grep11:20:152
1166199118,2cyclictest25193-21wc08:45:142
1166199118,2cyclictest24955-21ssh12:15:252
1166199118,2cyclictest22519-21wc09:40:252
1166199118,2cyclictest19292-21grep07:25:132
1166199118,2cyclictest1241-21ssh10:45:142
1166199118,2cyclictest10634-21ssh11:10:052
1166199117,3cyclictest4257-21diskmemload11:30:252
1165799119,1cyclictest3883-21cut11:25:141
1165799118,2cyclictest31925-21head07:50:161
1165799118,2cyclictest29884-21wget08:55:131
1165799118,2cyclictest29576-21wc11:20:271
1165799118,2cyclictest29189-21cat07:45:151
1165799118,2cyclictest29114-21ssh09:45:081
1165799118,2cyclictest21951-21ssh10:00:091
1165799118,2cyclictest16355-21ssh10:35:061
1165799118,2cyclictest11316-21ssh10:12:251
1165799117,3cyclictest8864-21tail10:30:181
11657991110,1cyclictest0-21swapper/109:30:351
1165799110,1cyclictest0-21swapper/112:14:121
1165499118,2cyclictest6669-21grep08:05:160
1165499118,2cyclictest6103-21tail09:50:180
1165499118,2cyclictest5504-21ssh11:06:190
1165499118,2cyclictest5064-21ssh10:47:030
1165499118,2cyclictest4305-21grep08:00:170
1165499118,2cyclictest3350-21ssh12:21:570
1165499118,2cyclictest31120-21cstates10:05:160
1165499118,2cyclictest31120-21cstates10:05:150
1165499118,2cyclictest22393-21ssh11:16:200
1165499118,2cyclictest2209-21ssh12:02:150
1165499118,2cyclictest21048-21ssh10:18:260
1165499118,2cyclictest19907-21chrt08:34:440
1165499118,2cyclictest16566-21ssh11:32:170
1165499118,2cyclictest15387-21expr12:10:130
1165499118,2cyclictest11387-21grep08:15:180
1165499118,2cyclictest11387-21grep08:15:170
1165499117,3cyclictest4453-21ssh09:10:000
1165499113,3cyclictest0-21swapper/010:50:140
1165499112,2cyclictest225-21systemd-journal07:52:080
1166399108,1cyclictest4565-21sed08:00:183
1166399108,1cyclictest24592-21seq08:43:373
1166399108,1cyclictest187282sleep307:21:473
1166399108,1cyclictest18557-21tail08:30:163
1166399107,2cyclictest32597-21expr09:00:143
1166399107,2cyclictest13817-21chrt07:12:493
1166199108,1cyclictest445-21grep07:50:252
1166199108,1cyclictest221232sleep208:37:432
1166199108,1cyclictest19737-21taskset08:32:312
1166199107,2cyclictest4313-21sed08:00:162
1166199107,2cyclictest24006-21chrt07:35:122
1166199104,3cyclictest1873-21latency_hist09:05:002
1165799108,2cyclictest8918-21grep08:10:141
1165799107,2cyclictest17612-21head07:20:171
1165799107,2cyclictest15743-21ps07:15:211
1165799106,3cyclictest2241-21tr09:05:141
1165799100,9cyclictest29382sleep107:55:511
1165799100,9cyclictest1739-21tune2fs10:45:171
1165499108,1cyclictest2367-21cut07:55:220
1165499108,1cyclictest22022-21expr07:30:140
1165499107,2cyclictest17286-21expr07:20:150
1165499107,2cyclictest13094-21basename09:15:110
1165499107,2cyclictest10227-21chrt08:12:450
116619997,1cyclictest20922sleep209:05:042
116619996,2cyclictest14173-21grep08:20:202
116579997,1cyclictest274102sleep108:50:121
116579997,1cyclictest26673-21ls07:40:131
116579997,1cyclictest12298-21expr07:10:141
116579996,2cyclictest24370-21expr07:35:151
116549997,1cyclictest11847-21chrt07:10:020
116549996,2cyclictest32587-21cat09:00:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional