You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-17 - 06:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Mar 17, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
124129267,9sleep00-21swapper/019:08:340
111029063,9sleep20-21swapper/219:06:592
116028661,9sleep30-21swapper/319:07:323
117827958,6sleep10-21swapper/119:07:461
154022700,1sleep015380-21timerandwakeup00:15:270
1443993732,4cyclictest0-21swapper/222:05:542
144399340,33cyclictest0-21swapper/223:46:042
144399340,33cyclictest0-21swapper/223:46:032
1443993027,3cyclictest0-21swapper/223:44:232
1443992925,1cyclictest0-21swapper/200:25:232
1443992924,5cyclictest0-21swapper/221:35:232
144399280,27cyclictest0-21swapper/223:33:022
1443992724,3cyclictest0-21swapper/223:23:422
1443992724,3cyclictest0-21swapper/223:01:432
1443992724,3cyclictest0-21swapper/221:21:242
144399270,27cyclictest0-21swapper/221:41:282
144399270,27cyclictest0-21swapper/221:28:102
143799270,26cyclictest0-21swapper/023:31:080
1443992622,3cyclictest0-21swapper/200:15:232
1443992522,3cyclictest0-21swapper/200:36:532
144399250,24cyclictest0-21swapper/222:29:132
143999252,1cyclictest26658-21diskmemload23:04:511
1443992421,3cyclictest0-21swapper/223:09:332
144399240,23cyclictest26658-21diskmemload23:17:532
144399240,23cyclictest0-21swapper/200:02:232
143999240,24cyclictest0-21swapper/123:20:221
1443992318,5cyclictest0-21swapper/222:24:442
144399230,19cyclictest0-21swapper/222:30:172
144399230,0cyclictest0-21swapper/222:37:242
1439992321,1cyclictest0-21swapper/122:45:211
1437992321,1cyclictest13-21ksoftirqd/022:24:180
1443992218,4cyclictest0-21swapper/223:54:132
1443992218,4cyclictest0-21swapper/223:29:242
1443992218,4cyclictest0-21swapper/221:59:322
144399220,22cyclictest0-21swapper/222:12:462
144399220,22cyclictest0-21swapper/222:12:452
144399220,1cyclictest0-21swapper/221:19:412
63722210,0sleep20-21swapper/222:15:072
1443992118,3cyclictest0-21swapper/221:48:122
1443992118,3cyclictest0-21swapper/221:12:232
1443992118,3cyclictest0-21swapper/200:30:442
1443992117,4cyclictest0-21swapper/200:06:442
143999212,18cyclictest26658-21diskmemload23:27:021
1439992119,1cyclictest0-21swapper/122:09:551
143799211,19cyclictest0-21swapper/000:05:110
1443992017,3cyclictest0-21swapper/223:38:242
1443992017,3cyclictest0-21swapper/222:01:322
1443992017,3cyclictest0-21swapper/221:53:542
144399200,14cyclictest0-21swapper/221:30:122
1439992019,1cyclictest0-21swapper/100:04:221
1437992017,2cyclictest13-21ksoftirqd/021:19:280
144599190,0cyclictest0-21swapper/322:54:523
1443991918,1cyclictest0-21swapper/219:53:082
1443991916,3cyclictest0-21swapper/223:13:432
144399190,17cyclictest0-21swapper/200:12:112
1439991918,1cyclictest0-21swapper/121:44:511
143999190,19cyclictest0-21swapper/122:03:241
144599182,0cyclictest0-21swapper/322:19:133
144599180,0cyclictest0-21swapper/300:28:513
143999184,6cyclictest0-21swapper/100:05:111
1439991817,1cyclictest0-21swapper/123:34:331
144599175,10cyclictest13466-21/usr/sbin/munin23:55:213
1445991715,1cyclictest0-21swapper/300:16:423
1445991714,1cyclictest8754-21kworker/3:023:24:443
144399171,1cyclictest10997-21ssh22:55:332
143999172,15cyclictest0-21swapper/100:15:541
143999170,1cyclictest0-21swapper/121:26:331
1445991616,0cyclictest0-21swapper/323:48:433
1445991616,0cyclictest0-21swapper/323:48:423
1445991613,1cyclictest39-21ksoftirqd/323:19:423
1443991613,2cyclictest32-21ksoftirqd/219:35:132
144399160,15cyclictest0-21swapper/220:41:402
143999161,9cyclictest38150irq/131-snd_hda21:59:391
1439991612,3cyclictest0-21swapper/119:40:011
143799160,3cyclictest0-21swapper/019:30:130
1445991515,0cyclictest0-21swapper/323:43:533
144599151,13cyclictest0-21swapper/323:06:363
1445991511,2cyclictest16621-21wc22:20:253
144599150,2cyclictest0-21swapper/323:36:563
1443991514,1cyclictest0-21swapper/222:41:422
1443991510,5cyclictest0-21swapper/221:06:062
143999151,1cyclictest26658-21diskmemload21:52:121
143999151,13cyclictest0-21swapper/100:30:261
143799154,5cyclictest0-21swapper/021:35:180
144599147,6cyclictest14845-21apache223:33:573
1443991411,2cyclictest7140-21apt20:30:002
144399140,12cyclictest22842-21grep21:00:152
143999141,1cyclictest141rcu_preempt00:11:381
1439991411,2cyclictest7931-21wc19:20:241
143999140,0cyclictest0-21swapper/100:37:181
143799147,6cyclictest1142-21runrttasks23:15:480
143799142,2cyclictest11662-21perf23:55:000
1437991410,3cyclictest13485-21wc21:20:260
143799140,8cyclictest24302-21perf21:05:000
143799140,1cyclictest141rcu_preempt23:38:130
1445991312,1cyclictest0-21swapper/321:35:183
1445991310,2cyclictest834-21sh23:29:003
1445991310,2cyclictest5080-21wc19:15:193
1445991310,2cyclictest28046-21sh21:30:003
1445991310,2cyclictest21879-21cron21:00:013
1445991310,2cyclictest16252-21tr19:40:143
144399139,3cyclictest21317-21wc20:55:262
1443991310,2cyclictest6155-21cron19:19:592
1443991310,2cyclictest31247-21cat20:10:192
1443991310,2cyclictest28516-21uniq20:05:142
144399130,2cyclictest0-21swapper/200:24:322
144399130,12cyclictest0-21swapper/222:50:582
143999139,3cyclictest32115-21grep22:30:141
143999139,3cyclictest11765-21cat23:55:001
143999139,3cyclictest11484-21head20:35:271
143999139,2cyclictest25769-21grep20:00:141
143999138,4cyclictest38150irq/131-snd_hda23:57:311
143999136,2cyclictest11070-21memory23:15:201
1439991313,0cyclictest0-21swapper/123:38:381
1439991312,1cyclictest0-21swapper/122:20:081
1439991310,3cyclictest14845-21apache222:28:111
1439991310,2cyclictest7328-21head22:15:171
1439991310,2cyclictest5141-21tail21:35:161
1439991310,2cyclictest2460-21wc23:10:171
1439991310,2cyclictest17894-21tail20:50:161
143999130,12cyclictest7893-21cut22:35:151
143799139,3cyclictest6002-21tail00:10:180
143799139,3cyclictest26797-21grep23:25:140
143799139,2cyclictest29353-21ssh21:50:100
143799134,5cyclictest31760-21/usr/sbin/munin22:30:210
143799134,5cyclictest225-21systemd-journal19:49:590
143799133,8cyclictest21775-21munin-run22:05:000
143799133,6cyclictest12123-21apache_accesses23:55:110
143799132,6cyclictest10937-21munin-run19:30:000
1437991311,2cyclictest0-21swapper/023:10:140
1437991310,2cyclictest5289-21tr20:25:140
1437991310,2cyclictest23286-21tr19:55:140
143799130,12cyclictest3902-21seq20:20:220
143799130,12cyclictest3902-21seq20:20:210
144599129,2cyclictest6652-21head21:55:203
144599129,2cyclictest4559-21grep23:50:173
144599129,2cyclictest3495-21sed23:10:273
144599129,2cyclictest32600-21ssh00:06:363
144599129,2cyclictest30076-21sed21:30:243
144599129,2cyclictest22685-21cut00:20:153
144599129,2cyclictest21787-21awk22:05:013
144599129,2cyclictest17590-21wc22:40:233
144599129,2cyclictest15479-21tune2fs20:45:183
144599129,2cyclictest10961-21tr21:19:593
144599128,3cyclictest9910-21tail19:25:183
144599128,3cyclictest10713-21sed22:55:263
144599122,6cyclictest0-21swapper/300:00:133
144599120,0cyclictest0-21swapper/321:50:073
144399129,2cyclictest18477-21basename19:45:142
144399129,2cyclictest16248-21cut19:40:142
144399128,3cyclictest32-21ksoftirqd/219:10:212
144399128,3cyclictest12739-21df23:55:152
143999129,2cyclictest728-21cut22:50:151
143999129,2cyclictest32461-21grep22:10:261
143999129,2cyclictest32461-21grep22:10:251
143999129,2cyclictest28402-21grep23:45:171
143999129,2cyclictest28402-21grep23:45:161
143999129,2cyclictest26058-21wc21:05:251
143999129,2cyclictest18433-21cut19:45:141
143999129,2cyclictest17013-21grep22:40:181
143999129,2cyclictest13414-21grep19:34:591
143999129,2cyclictest11113-21cat21:19:591
143999129,2cyclictest10230-21grep22:55:211
143999128,2cyclictest30561-21tr20:10:141
143999127,3cyclictest8925-21grep19:25:131
1439991210,1cyclictest4032-21basename19:15:111
143999120,12cyclictest0-21swapper/121:20:591
143999120,12cyclictest0-21swapper/121:10:171
143999120,11cyclictest26109-21df23:05:141
143799129,2cyclictest8550-21cron19:24:590
143799129,2cyclictest8409-21sed00:30:250
143799129,2cyclictest6403-21ssh21:55:180
143799129,2cyclictest31989-21head20:10:260
143799129,2cyclictest29090-21grep21:10:250
143799129,2cyclictest28875-21wc20:05:180
143799129,2cyclictest26402-21sed23:05:160
143799129,2cyclictest21901-21cat21:00:000
143799129,2cyclictest19529-21sed20:55:000
143799129,2cyclictest18653-21tr23:20:130
143799129,2cyclictest17288-21cut19:40:210
143799129,2cyclictest13912-21grep21:40:190
143799129,2cyclictest10762-21tail22:55:260
143799129,2cyclictest10480-21tail20:35:170
143799128,3cyclictest17362-21basename23:00:150
143799128,2cyclictest29002-21wc21:50:010
143799128,2cyclictest2448-21cut19:10:170
143799128,2cyclictest22534-21cut22:05:140
143799124,4cyclictest225-21systemd-journal23:50:010
143799124,4cyclictest225-21systemd-journal23:50:000
143799120,2cyclictest0-21swapper/022:12:300
143799120,2cyclictest0-21swapper/022:12:300
144599118,2cyclictest9961-21ssh00:31:353
144599118,2cyclictest7826-21ssh22:35:143
144599118,2cyclictest6519-21tr19:20:143
144599118,2cyclictest6009-21cat20:25:173
144599118,2cyclictest5592-21expr00:10:143
144599118,2cyclictest4234-21ssh22:13:393
144599118,2cyclictest4234-21ssh22:13:383
144599118,2cyclictest32140-21tr22:30:143
144599118,2cyclictest27929-21head21:10:143
144599118,2cyclictest27326-21taskset20:00:593
144599118,2cyclictest24457-21grep19:55:223
144599118,2cyclictest24273-21rm22:45:053
144599118,2cyclictest23221-21ssh22:25:023
144599118,2cyclictest22123-21rm21:45:193
144599118,2cyclictest19972-21ssh23:01:023
144599118,2cyclictest18655-21grep19:45:153
144599118,2cyclictest16304-21ssh21:41:413
144599118,2cyclictest14632-21ssh00:35:023
144599118,2cyclictest14344-21expr19:35:183
144599118,2cyclictest11535-21ssh21:20:073
144599111,1cyclictest0-21swapper/322:07:193
144399118,2cyclictest652-21expr20:15:172
144399118,2cyclictest30134-21ssh22:48:182
144399118,2cyclictest26771-21head20:00:212
144399118,2cyclictest25210-21taskset19:59:352
144399118,2cyclictest19036-21chrt20:51:482
144399118,2cyclictest11948-21head19:30:172
144399110,10cyclictest9568-21grep19:25:162
143999118,2cyclictest9300-21taskset20:32:271
143999118,2cyclictest3398-21grep20:20:171
143999118,2cyclictest3398-21grep20:20:161
143999118,2cyclictest31284-21ssh21:31:091
143999118,2cyclictest30106-21ssh00:25:031
143999118,2cyclictest28582-21cut20:05:141
143999118,2cyclictest25761-21ssh00:21:511
143999118,2cyclictest24137-21sort19:55:191
143999118,2cyclictest22730-21taskset19:53:131
143999118,2cyclictest2216-21chrt20:18:151
143999118,2cyclictest19790-21tr23:40:151
143999118,2cyclictest15916-21strings20:45:211
143999118,2cyclictest13284-21ntp_kernel_err20:40:201
143999113,4cyclictest225-21systemd-journal21:00:221
143999110,10cyclictest27337-21sh21:48:471
143799118,2cyclictest5426-21ssh00:29:420
143799118,2cyclictest28180-21ssh21:30:030
143799118,2cyclictest26817-21ssh22:26:290
143799118,2cyclictest24763-21tr22:45:130
143799118,2cyclictest24654-21ssh21:27:270
143799118,2cyclictest23539-21ssh00:01:080
143799118,2cyclictest22477-21sed19:50:270
143799118,2cyclictest20171-21sort23:40:180
143799118,2cyclictest17798-21ssh22:40:260
143799118,2cyclictest15634-21cut00:35:170
143799118,2cyclictest15074-21tr20:45:160
143799118,2cyclictest14302-21grep19:35:180
143799118,2cyclictest11210-21ssh22:36:360
143799117,2cyclictest7500-21grep20:30:120
143799117,2cyclictest14291-21perf20:45:000
143799116,4cyclictest8272-21wc22:15:260
143799113,7cyclictest29928-21latency_hist00:25:000
144599108,1cyclictest30651-21cat20:10:153
144599108,1cyclictest2687-21cut20:20:143
144599108,1cyclictest2687-21cut20:20:133
144599108,1cyclictest25486-21cut21:05:193
144599108,1cyclictest2220-21taskset20:18:173
144599108,1cyclictest212202sleep319:50:153
144599108,1cyclictest14131-21taskset20:43:183
144599107,2cyclictest9960-21egrep20:35:143
144599107,2cyclictest7626-21tr20:30:143
144599107,2cyclictest3989-21cat19:15:003
144599107,2cyclictest29054-21ntpq20:05:183
144599107,2cyclictest23304-21grep21:00:193
144599107,2cyclictest17266-21basename20:50:133
144399108,1cyclictest6910-21expr19:20:152
144399108,1cyclictest4425-21taskset20:20:512
144399108,1cyclictest4425-21taskset20:20:502
144399108,1cyclictest14956-21cpu20:45:152
144399107,2cyclictest12014-21cat20:40:012
143999108,1cyclictest6602-21cat20:25:241
143999107,2cyclictest1938-21tr19:10:151
143999107,2cyclictest16799-21sed19:40:161
143999100,9cyclictest20122-21expr20:55:151
143799108,1cyclictest5854-21chrt19:16:150
143799108,1cyclictest26013-21expr20:00:150
143799108,1cyclictest20362sleep020:15:570
143799107,2cyclictest26248-21grep21:05:250
143799107,2cyclictest2158-21ssh22:50:260
14439996,2cyclictest7805-21expr20:30:142
14459986,1cyclictest11161-21seq19:30:103
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional