You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-24 - 09:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Sep 24, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8351210183,14sleep20-21swapper/219:07:362
849629279,9sleep30-21swapper/319:09:303
846329279,9sleep00-21swapper/019:09:020
843728865,12sleep10-21swapper/119:08:431
86362620,0sleep10-21swapper/123:10:131
91292550,0sleep30-21swapper/323:48:463
52842470,0sleep30-21swapper/322:10:123
862699312,28cyclictest13184-21turbostat.cron00:30:001
863299250,24cyclictest0-21swapper/322:20:503
8632992322,1cyclictest0-21swapper/321:35:473
8632992320,2cyclictest39-21ksoftirqd/321:52:573
862299230,13cyclictest0-21swapper/021:13:200
862999220,22cyclictest0-21swapper/223:22:582
862999210,21cyclictest0-21swapper/200:13:232
862999210,21cyclictest0-21swapper/200:13:222
862699215,14cyclictest1142-21runrttasks23:58:091
862699210,20cyclictest0-21swapper/121:37:121
862299217,7cyclictest2641-21munin-run20:05:000
863299191,3cyclictest28040-21apt-get00:00:133
862999190,15cyclictest225-21systemd-journal20:53:482
8626991917,2cyclictest25-21ksoftirqd/100:37:121
8632991813,3cyclictest12523-21apt-get00:10:103
8632991813,3cyclictest12523-21apt-get00:10:103
862999182,14cyclictest0-21swapper/223:06:352
862699180,1cyclictest0-21swapper/121:25:321
862299180,2cyclictest0-21swapper/023:43:150
863299170,16cyclictest0-21swapper/322:27:353
863299170,15cyclictest0-21swapper/300:27:293
862999177,2cyclictest225-21systemd-journal23:00:462
862999171,15cyclictest55850irq/129-enp2s0-23:55:092
862699171,2cyclictest0-21swapper/122:00:121
862299171,1cyclictest1231-21diskmemload22:48:550
862999160,16cyclictest0-21swapper/200:06:542
8626991616,0cyclictest0-21swapper/100:00:191
8626991615,1cyclictest0-21swapper/122:07:181
8626991613,2cyclictest20681-21cat19:35:001
8622991612,2cyclictest0-21swapper/023:16:580
862299160,3cyclictest0-21swapper/000:29:160
254252160,0sleep00-21swapper/000:17:240
863299153,10cyclictest0-21swapper/323:26:133
863299153,0cyclictest0-21swapper/322:47:323
863299150,11cyclictest0-21swapper/321:40:133
862999152,6cyclictest0-21swapper/221:49:552
862999151,2cyclictest4011-21ssh23:25:532
862999150,14cyclictest0-21swapper/221:18:502
862699159,6cyclictest24341-21apache200:05:131
862699153,11cyclictest3305-21ssh22:47:051
862699152,11cyclictest26696-21ssh23:40:011
8626991514,1cyclictest0-21swapper/122:30:001
862699150,13cyclictest0-21swapper/123:27:001
8622991514,1cyclictest0-21swapper/000:24:220
8622991510,4cyclictest0-21swapper/020:10:130
862299150,15cyclictest0-21swapper/021:22:320
863299149,2cyclictest6892-21ssh00:06:103
8632991413,1cyclictest0-21swapper/323:36:253
8632991410,1cyclictest0-21swapper/323:06:423
863299140,13cyclictest0-21swapper/322:05:103
862999141,11cyclictest0-21swapper/219:45:162
862999140,1cyclictest0-21swapper/222:47:212
862999140,13cyclictest0-21swapper/222:40:112
862699149,2cyclictest25-21ksoftirqd/121:15:381
862699146,2cyclictest14454-21latency_hist22:35:011
862699144,4cyclictest10058-21apt-get20:20:131
8626991410,1cyclictest19281-21sh20:40:001
862699140,4cyclictest0-21swapper/123:17:591
8622991411,2cyclictest20193-21cut20:40:160
862299140,4cyclictest0-21swapper/022:03:440
862299140,2cyclictest225-21systemd-journal20:58:420
862299140,12cyclictest12859-21grep21:55:130
863299138,4cyclictest0-21swapper/322:15:163
863299138,4cyclictest0-21swapper/322:15:163
863299137,6cyclictest26952-21munin-node22:39:183
863299137,5cyclictest39-21ksoftirqd/300:30:083
863299132,9cyclictest22061-21apache_accesses00:35:143
863299131,4cyclictest1231-21diskmemload23:22:073
863299130,6cyclictest0-21swapper/322:43:423
863299130,2cyclictest0-21swapper/322:52:523
862999139,4cyclictest32-21ksoftirqd/222:30:282
862999139,3cyclictest29822-21tail19:50:262
862999139,3cyclictest18145-21cat19:30:002
8629991310,2cyclictest9838-21sed23:10:232
8629991310,2cyclictest6531-21cat22:10:252
8629991310,2cyclictest22786-21grep22:20:162
8629991310,2cyclictest20299-21grep20:40:162
8629991310,2cyclictest15741-21idleruntime-cro19:25:002
862999130,11cyclictest0-21swapper/222:06:052
862699139,3cyclictest25310-21taskset19:42:151
8626991312,1cyclictest1231-21diskmemload22:15:231
8626991312,1cyclictest1231-21diskmemload22:15:221
8626991312,1cyclictest0-21swapper/123:06:581
862699131,1cyclictest0-21swapper/121:40:131
8626991310,2cyclictest15562-21grep22:35:171
862699130,12cyclictest22813-21ssh21:22:131
862699130,12cyclictest2010-21wc20:00:281
8622991310,2cyclictest5000-21cron20:10:000
8622991310,2cyclictest28369-21wc21:25:220
8622991310,2cyclictest27268-21sed23:20:240
8622991310,2cyclictest23042-21grep20:45:170
8622991310,2cyclictest13184-21head21:35:250
8622991310,2cyclictest12077-21grep21:15:290
862299130,1cyclictest0-21swapper/021:34:050
862299130,12cyclictest18061-21ssh23:54:130
863299129,2cyclictest8782-21ntpq20:15:223
863299129,2cyclictest3148-21tr20:05:173
863299129,2cyclictest31425-21ssh23:42:223
863299129,2cyclictest30860-21grep21:00:263
863299129,2cyclictest30275-21cron19:55:003
863299129,2cyclictest29226-21ssh23:02:413
863299129,2cyclictest27900-21cron19:50:003
863299129,2cyclictest26998-21tr20:55:163
863299129,2cyclictest2635-21ls20:05:003
863299129,2cyclictest22599-21ssh21:22:043
863299129,2cyclictest20597-21cat19:34:593
863299129,2cyclictest15801-21ssh22:55:083
863299129,2cyclictest13132-21sed23:50:263
863299129,2cyclictest11607-21expr19:15:163
863299129,2cyclictest10991-21ssh21:15:183
863299128,3cyclictest29724-21expr00:20:153
863299127,4cyclictest13846-21ssh23:12:573
863299127,2cyclictest39-21ksoftirqd/320:00:003
863299125,5cyclictest5281-21apt-get20:10:133
863299121,1cyclictest8637-21ssh22:30:363
863299120,7cyclictest0-21swapper/323:33:273
863299120,4cyclictest0-21swapper/323:56:453
862999129,2cyclictest9768-21sh20:20:002
862999129,2cyclictest3977-21wc23:45:182
862999129,2cyclictest31668-21tr21:05:162
862999129,2cyclictest30798-21tr22:25:132
862999129,2cyclictest3047-21tr21:10:232
862999129,2cyclictest28867-21grep00:00:172
862999129,2cyclictest23905-21ssh00:35:272
862999129,2cyclictest22175-21grep20:45:132
862999129,2cyclictest18475-21grep23:15:212
862999129,2cyclictest16468-21wc22:35:252
862999129,2cyclictest12824-21tr21:55:132
862999129,2cyclictest11848-21sed23:30:252
862999129,2cyclictest10099-21rm21:53:302
862999128,3cyclictest20778-21sh00:15:072
862999128,2cyclictest30174-21head00:20:172
862999120,1cyclictest0-21swapper/221:39:432
862999120,11cyclictest28436-21find21:25:222
862999120,11cyclictest18812-21sed20:35:242
862699129,2cyclictest6022-21ssh21:31:211
862699129,2cyclictest5524-21cut22:10:191
862699129,2cyclictest4789-21grep23:45:251
862699129,2cyclictest30147-21sed21:45:251
862699129,2cyclictest25770-21awk23:00:221
862699129,2cyclictest25607-21head20:50:231
862699129,2cyclictest23177-21sh22:40:121
862699129,2cyclictest21198-21cut00:15:141
862699129,2cyclictest21101-21sh23:36:031
862699129,2cyclictest20564-21sh22:57:311
862699129,2cyclictest19827-21grep20:40:141
862699129,2cyclictest14374-21tail19:20:171
862699129,1cyclictest1449-21sh21:10:001
862699128,4cyclictest15904-21grep20:30:221
862699124,7cyclictest28863-21sh22:24:091
862299129,2cyclictest9545-21grep23:10:210
862299129,2cyclictest7041-21grep22:30:160
862299129,2cyclictest29059-21grep21:45:170
862299129,2cyclictest28702-21ssh00:00:160
862299129,2cyclictest24633-21tr20:50:160
862299129,2cyclictest24032-21grep00:35:280
862299129,2cyclictest23255-21cut22:40:130
862299129,2cyclictest16777-21sed22:35:260
862299129,2cyclictest16314-21cat20:30:280
862299127,3cyclictest31563-21grep21:05:150
862299124,4cyclictest225-21systemd-journal22:30:000
862299121,9cyclictest15420-21awk22:55:010
862299121,6cyclictest30293-21munin-run19:55:000
8622991211,1cyclictest0-21swapper/023:29:400
863299119,2cyclictest9556-21fw_packets19:10:163
863299119,2cyclictest16040-21grep20:30:243
863299119,1cyclictest50902sleep321:30:363
863299119,1cyclictest28938-21rm21:25:273
863299119,1cyclictest23236-21ssh00:15:413
863299119,1cyclictest216042sleep320:44:133
863299119,1cyclictest20918-21ssh22:00:123
863299119,1cyclictest17819-21fschecks_time20:35:143
863299119,1cyclictest16858-21ssh23:15:073
863299119,1cyclictest16038-21hostname19:25:133
863299119,1cyclictest1493-21ssh21:10:023
863299119,1cyclictest14681-21cat19:20:193
863299119,1cyclictest10943-21cat20:20:183
863299119,1cyclictest10712sleep321:08:253
863299118,2cyclictest24518-21cut20:50:163
863299118,2cyclictest23884-21chrt20:48:023
863299118,2cyclictest15445-21ssh21:56:063
863299116,3cyclictest0-21swapper/319:40:153
863299110,11cyclictest12844-21expr20:25:153
863299110,10cyclictest0-21swapper/321:45:123
862999118,2cyclictest7459-21ssh22:50:082
862999118,2cyclictest6643-21grep00:25:222
862999118,2cyclictest3820-21grep21:30:222
862999118,2cyclictest31322-21head19:55:172
862999118,2cyclictest30476-21which21:00:222
862999118,2cyclictest28528-21fgrep23:40:252
862999118,2cyclictest28017-21wc20:55:222
862999118,2cyclictest25900-21ssh22:02:412
862999118,2cyclictest21528-21ssh21:21:132
862999118,2cyclictest19468-21cat23:35:202
862999118,2cyclictest19078-21cron23:54:592
862999118,2cyclictest15716-21ssh22:55:052
862999118,2cyclictest14891-21sed22:15:222
862999118,2cyclictest14891-21sed22:15:222
862999118,2cyclictest14754-21uname00:30:192
862999111,8cyclictest12172-21idleruntime-cro20:25:002
862999111,4cyclictest225-21systemd-journal20:14:472
862999110,10cyclictest26014-21ssh21:43:502
862699119,2cyclictest29301-21awk19:50:201
862699119,2cyclictest18092-21ssh23:54:151
862699119,1cyclictest7335-21sh22:50:041
862699119,1cyclictest4151-21seq00:24:331
862699119,1cyclictest31817-21tr21:05:161
862699119,1cyclictest306672sleep123:22:591
862699119,1cyclictest22574-21cut20:45:151
862699119,1cyclictest15512-21ssh21:56:101
862699119,1cyclictest13660-21cat00:10:161
862699119,1cyclictest13660-21cat00:10:161
862699118,3cyclictest25-21ksoftirqd/121:51:171
862699118,2cyclictest5365-21grep20:10:131
862699118,2cyclictest31335-21tail19:55:161
862699118,2cyclictest18811-21ssh00:32:531
862699118,2cyclictest14540-21apt20:30:001
862699118,2cyclictest12267-21cat19:15:201
862699118,2cyclictest10640-21uniq23:30:161
862699117,3cyclictest17230-21head19:25:211
862699110,9cyclictest1-21systemd20:58:421
862699110,10cyclictest3026-21hostname20:05:181
862299118,2cyclictest9960-21grep19:10:200
862299118,2cyclictest7491-21ssh22:10:350
862299118,2cyclictest4798-21grep23:45:250
862299118,2cyclictest4127-21sh00:05:140
862299118,2cyclictest30812-21grep21:00:260
862299118,2cyclictest2724-21sshd23:05:450
862299118,2cyclictest26524-21grep19:45:190
862299118,2cyclictest23382-21ssh22:20:210
862299118,2cyclictest22869-21ssh21:41:260
862299118,2cyclictest19366-21ssh00:33:190
862299118,2cyclictest17624-21taskset22:16:580
862299118,2cyclictest17624-21taskset22:16:580
862299118,2cyclictest14066-21tail20:25:260
862299118,2cyclictest13193-21cut00:10:130
862299118,2cyclictest13193-21cut00:10:130
862299118,2cyclictest12848-21sed19:15:270
862299117,3cyclictest31341-21grep22:05:260
862299117,3cyclictest19783-21wc23:35:230
862299116,3cyclictest17625-21grep19:25:250
862299115,4cyclictest25404-21uname23:00:200
862299112,7cyclictest20404-21grep23:55:170
862299111,5cyclictest225-21systemd-journal23:00:000
862299110,10cyclictest4902-21ssh21:50:200
863299107,2cyclictest20724-21seq19:35:103
862999108,1cyclictest253542sleep219:42:492
862999108,1cyclictest20902-21egrep19:35:152
862999108,1cyclictest18670-21tr19:30:172
862999108,1cyclictest1838-21gzip20:00:242
862999107,2cyclictest9085-21grep19:10:142
862999107,2cyclictest3141-21grep20:05:182
862999107,2cyclictest13139-21head20:25:172
862999107,2cyclictest11495-21cut19:15:152
862999102,2cyclictest0-21swapper/220:30:152
862699108,1cyclictest7696-21aten2_r7power_e20:15:151
862699108,1cyclictest28060-21ls19:50:001
862699108,1cyclictest20995-21tr19:35:151
862699100,10cyclictest30907-21chrt21:00:381
862299108,1cyclictest7844-21ls20:15:160
862299108,1cyclictest23913-21df19:40:170
862299108,1cyclictest19107-21chrt20:37:530
862299108,1cyclictest12065-21ssh23:30:310
862299107,2cyclictest22944-21cut19:40:000
862299107,2cyclictest13594-21taskset19:20:050
862299107,2cyclictest12283-21cat20:24:590
862299101,8cyclictest32670-21munin-run20:00:000
862299100,9cyclictest20146-21taskset19:30:420
86269998,1cyclictest10285-21sed19:10:261
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional