You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 06:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 23, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1091229066,8sleep00-21swapper/019:06:050
1117328875,9sleep20-21swapper/219:09:292
1097728861,9sleep30-21swapper/319:06:573
1106727969,6sleep10-21swapper/119:08:071
1130399270,27cyclictest0-21swapper/221:52:012
1130399270,26cyclictest0-21swapper/222:43:242
1130099261,13cyclictest0-21swapper/123:00:141
1130399230,22cyclictest0-21swapper/223:45:052
1130399230,22cyclictest0-21swapper/222:36:352
11296992322,1cyclictest0-21swapper/000:05:150
11296992320,2cyclictest13-21ksoftirqd/023:20:000
56152220,0chrt5616-21aten2_r7power_c21:30:151
1130099220,1cyclictest0-21swapper/122:11:051
11303992116,4cyclictest32-21ksoftirqd/221:31:292
11296992120,1cyclictest0-21swapper/019:53:040
1129699210,1cyclictest0-21swapper/021:20:030
1130699200,3cyclictest0-21swapper/300:22:063
1130399200,20cyclictest0-21swapper/200:26:012
1130099204,8cyclictest15157-21cat22:55:011
1130099200,19cyclictest0-21swapper/122:16:221
1130099200,19cyclictest0-21swapper/100:10:231
1129699200,20cyclictest0-21swapper/022:43:240
1130399196,12cyclictest0-21swapper/223:29:312
1129699193,2cyclictest25915-21hddtemp_smartct22:20:170
1129699190,17cyclictest0-21swapper/022:13:360
57082180,0sleep20-21swapper/223:08:252
1130699180,16cyclictest0-21swapper/323:30:493
11303991817,1cyclictest32-21ksoftirqd/223:20:562
1130399180,1cyclictest3893-21diskmemload00:20:322
1130399180,16cyclictest0-21swapper/223:10:092
1130399180,11cyclictest0-21swapper/219:35:142
1129699180,3cyclictest0-21swapper/022:45:140
1129699180,17cyclictest0-21swapper/021:29:260
11306991716,1cyclictest0-21swapper/321:37:123
11306991715,1cyclictest0-21swapper/321:10:143
1130699170,16cyclictest0-21swapper/300:29:423
1130399171,3cyclictest141rcu_preempt00:10:122
11303991713,1cyclictest32-21ksoftirqd/221:55:302
1130399170,1cyclictest0-21swapper/222:00:502
1130399170,16cyclictest0-21swapper/223:37:452
1130399170,14cyclictest0-21swapper/221:21:292
1130099173,2cyclictest141rcu_preempt23:33:421
1129699170,1cyclictest0-21swapper/000:26:150
1129699170,16cyclictest0-21swapper/022:36:350
11306991615,1cyclictest0-21swapper/322:25:593
1130399161,14cyclictest0-21swapper/200:35:142
1130399160,7cyclictest0-21swapper/221:43:302
1130399160,7cyclictest0-21swapper/221:43:302
1130399160,2cyclictest0-21swapper/221:15:122
1130399160,16cyclictest0-21swapper/220:07:112
1130399160,14cyclictest0-21swapper/222:08:332
1130399160,13cyclictest0-21swapper/221:35:142
11300991615,1cyclictest0-21swapper/122:41:101
11300991614,1cyclictest2218-21sed19:55:271
11306991514,1cyclictest0-21swapper/300:39:193
1130699151,13cyclictest0-21swapper/322:20:133
1130699150,3cyclictest0-21swapper/323:20:133
1130399154,5cyclictest32-21ksoftirqd/222:56:062
1130399153,7cyclictest11246-21apache200:19:132
1130399150,13cyclictest0-21swapper/222:18:352
11300991515,0cyclictest0-21swapper/100:34:331
1130099151,13cyclictest0-21swapper/120:17:281
1130099150,2cyclictest0-21swapper/100:09:371
1130099150,15cyclictest0-21swapper/123:06:511
11296991514,1cyclictest0-21swapper/021:15:120
11306991410,2cyclictest16255-21head22:55:163
1130699140,14cyclictest0-21swapper/321:59:063
1130699140,13cyclictest0-21swapper/322:40:113
1130399149,3cyclictest32-21ksoftirqd/200:30:152
1130399143,4cyclictest0-21swapper/222:21:242
11303991412,1cyclictest10545-21apt-get23:50:142
1130399140,3cyclictest0-21swapper/200:09:382
1130399140,1cyclictest0-21swapper/222:13:332
11300991413,1cyclictest0-21swapper/122:27:531
11300991411,2cyclictest743-21sort22:05:201
1130099140,14cyclictest0-21swapper/122:59:091
1130099140,13cyclictest0-21swapper/123:49:231
1129699145,3cyclictest29150-21cron20:54:590
11296991413,1cyclictest0-21swapper/023:34:000
1129699140,12cyclictest0-21swapper/020:55:140
1130699139,3cyclictest21422-21wc20:35:273
1130699139,2cyclictest32169-21sed21:45:213
1130699134,6cyclictest986-21wc23:05:193
1130699134,3cyclictest10111-21apt-get20:15:143
11306991312,1cyclictest0-21swapper/322:01:143
11306991310,2cyclictest9396-21grep20:10:263
11306991310,2cyclictest8593-21sed21:50:263
11306991310,2cyclictest32493-21ssh22:45:223
11306991310,2cyclictest2837-21basename00:05:003
11306991310,2cyclictest26458-21perf00:20:013
11306991310,2cyclictest22385-21wc21:20:193
11306991310,2cyclictest20219-21grep23:55:203
11306991310,2cyclictest16521-21wc20:25:273
1130699130,12cyclictest5849-21grep21:30:163
1130399138,4cyclictest4515-21grep21:10:142
1130399138,2cyclictest31530-21sh21:00:002
1130399136,6cyclictest32-21ksoftirqd/223:40:252
1130399131,2cyclictest0-21swapper/219:30:132
1130399131,12cyclictest0-21swapper/222:52:342
1130399130,1cyclictest0-21swapper/222:25:472
1130399130,0cyclictest0-21swapper/221:28:302
1130099139,2cyclictest16768-21grep20:25:261
1130099137,2cyclictest22153-21sed21:20:181
11300991311,1cyclictest0-21swapper/100:15:141
11300991310,3cyclictest6985-21kworker/1:123:16:501
11300991310,2cyclictest9986-21cat20:15:001
11300991310,2cyclictest4084-21strings20:00:191
11300991310,2cyclictest26998-21cat00:00:001
11300991310,2cyclictest25939-21grep23:20:181
11300991310,2cyclictest21326-21cat20:35:271
11300991310,2cyclictest19015-21cat20:30:271
1129699138,1cyclictest3893-21diskmemload00:14:430
1129699138,1cyclictest13-21ksoftirqd/022:00:000
11296991312,1cyclictest0-21swapper/022:00:500
11296991310,2cyclictest8288-21basename23:10:130
11296991310,2cyclictest2783-21cat23:25:270
11296991310,2cyclictest11368-21grep23:50:170
1129699130,3cyclictest0-21swapper/023:57:190
1129699130,3cyclictest0-21swapper/019:10:130
1130699129,2cyclictest3219-21ssh23:45:183
1130699129,2cyclictest32116-21wc19:50:253
1130699129,2cyclictest28712-21grep20:50:303
1130699129,2cyclictest27341-21wc23:40:213
1130699129,2cyclictest25916-21grep23:00:283
1130699129,2cyclictest24403-21cron20:45:003
1130699129,2cyclictest2014-21grep21:05:213
1130699129,2cyclictest18301-21ssh23:15:263
1130699129,2cyclictest1502-21cron21:04:593
1130699129,2cyclictest12531-21gpgv20:20:003
1130699129,2cyclictest11350-21grep23:50:183
1130699129,2cyclictest10171-21tail23:10:263
1130699128,2cyclictest26172-21tr19:40:163
1130699122,8cyclictest7076-21/usr/sbin/munin22:50:243
1130699121,4cyclictest0-21swapper/300:11:133
1130699120,11cyclictest0-21swapper/323:39:573
1130699120,11cyclictest0-21swapper/300:32:013
1130399129,2cyclictest4619-21grep20:00:262
1130399129,2cyclictest11212-21cut20:15:202
1130399128,4cyclictest32-21ksoftirqd/221:45:272
1130399126,5cyclictest12146-21diskmemload22:32:222
1130399120,7cyclictest0-21swapper/219:50:122
1130399120,1cyclictest0-21swapper/223:02:262
1130399120,11cyclictest7777-21basename20:10:142
1130399120,11cyclictest5653-21sh22:49:162
1130399120,11cyclictest19384-21chrt20:31:502
1130399120,11cyclictest14175-21tr19:15:162
1130099129,2cyclictest491-21sh22:45:271
1130099129,2cyclictest31023-21grep21:25:221
1130099129,2cyclictest29351-21grep19:45:211
1130099129,2cyclictest25116-21grep19:35:271
1130099129,2cyclictest24402-21cron20:45:001
1130099129,2cyclictest24256-21sed21:40:271
1130099129,2cyclictest24256-21sed21:40:261
1130099129,2cyclictest1887-21grep23:25:181
1130099129,2cyclictest12970-21tr20:20:121
1130099129,2cyclictest11805-21tr19:10:141
1130099128,3cyclictest5935-21ssh00:26:011
1130099127,4cyclictest26878-21ls19:40:201
1130099127,3cyclictest13978-21apt-get21:35:141
1130099124,5cyclictest38150irq/131-snd_hda19:50:151
1129699129,2cyclictest9450-21grep20:10:260
1129699129,2cyclictest3485-21sed21:05:330
1129699129,2cyclictest26393-21ssh23:20:210
1129699129,2cyclictest26034-21mailstats20:45:190
1129699129,2cyclictest22023-21ls20:39:590
1129699129,2cyclictest20405-21tail19:25:250
1129699129,2cyclictest12458-21cat20:20:000
1129699123,6cyclictest0-21swapper/000:34:060
1129699122,9cyclictest32255-21/usr/sbin/munin23:05:270
11296991211,1cyclictest0-21swapper/022:27:440
11296991210,1cyclictest8533-21tr22:50:220
11296991210,1cyclictest30942-21tr21:45:130
11296991210,1cyclictest3087-21tr20:00:130
1129699120,1cyclictest0-21swapper/021:32:160
1130699118,2cyclictest5630-21grep20:05:163
1130699118,2cyclictest4795-21grep00:05:233
1130699118,2cyclictest4080-21grep20:00:193
1130699118,2cyclictest362-21ssh21:26:223
1130699118,2cyclictest24744-21cut20:45:113
1130699118,2cyclictest24213-21tail19:35:183
1130699118,2cyclictest23079-21seq19:33:173
1130699118,2cyclictest22303-21sh21:40:113
1130699118,2cyclictest22303-21sh21:40:103
1130699118,2cyclictest19519-21cut19:25:173
1130699118,2cyclictest1905-21ssh22:05:403
1130699118,2cyclictest18493-21perf19:25:003
1130699118,2cyclictest18342-21ssh21:18:103
1130699118,2cyclictest16340-21ssh22:35:283
1130699118,2cyclictest12422-21ssh22:12:213
1130399119,1cyclictest831-21tr19:55:152
1130399119,1cyclictest23360-21wc20:40:202
1130399118,2cyclictest23932-21sh23:19:322
1130399118,2cyclictest20027-21cut20:35:152
1130399118,2cyclictest18882-21cut19:25:132
1130399118,2cyclictest18490-21munin-run19:25:002
1130399118,2cyclictest11833-21seq19:10:142
1130399110,8cyclictest0-21swapper/200:00:142
1130399110,3cyclictest0-21swapper/223:32:492
1130399110,1cyclictest0-21swapper/223:55:122
1130399110,10cyclictest15774-21fschecks_time20:25:172
1130099118,2cyclictest6337-21sed21:10:271
1130099118,2cyclictest3680-21ssh21:48:011
1130099118,2cyclictest32452-21ssh00:23:071
1130099118,2cyclictest29713-21ssh22:03:371
1130099118,2cyclictest27648-21tr00:00:131
1130099118,2cyclictest26729-21expr23:40:171
1130099118,2cyclictest26681-21chrt20:48:471
1130099118,2cyclictest24598-21ssh22:20:001
1130099118,2cyclictest21718-21ssh23:36:531
1130099118,2cyclictest20891-21ssh00:35:171
1130099118,2cyclictest19373-21rm22:37:451
1130099118,2cyclictest16932-21cat21:55:261
1130099118,2cyclictest14886-21ssh22:34:311
1130099118,2cyclictest14193-21tr19:15:161
1130099118,2cyclictest12662-21taskset21:15:071
1130099118,2cyclictest11801-21ssh23:11:381
1130099118,2cyclictest11214-21fschecks_count23:50:171
1130099118,1cyclictest5579-21tr20:05:161
1130099117,4cyclictest29037-21munin-node20:59:121
1130099116,4cyclictest7720-21if_err_enp0s31f21:50:191
1129699119,1cyclictest27697-21wc23:40:230
1129699119,1cyclictest24214-21head19:35:180
1129699119,1cyclictest20209-21grep00:15:230
1129699119,1cyclictest14612-21sh21:55:010
1129699118,2cyclictest812-21grep19:55:160
1129699118,2cyclictest6093-21head20:05:190
1129699118,2cyclictest5308-21ssh23:46:180
1129699118,2cyclictest18175-21tr23:35:150
1129699118,2cyclictest15335-21rm22:55:080
1129699118,2cyclictest14713-21grep21:35:190
1129699117,3cyclictest23660-21chrt00:36:350
1129699116,4cyclictest16030-21sh19:20:000
1129699115,1cyclictest0-21swapper/021:45:010
1129699115,1cyclictest0-21swapper/021:45:000
1129699110,8cyclictest2049-21ssh22:05:480
1129699110,10cyclictest23367-21sed20:40:190
1130699108,1cyclictest9184-21sh22:30:033
1130699108,1cyclictest30775-21cut20:55:223
1130699108,1cyclictest2356-21sh23:25:233
1130699108,1cyclictest19553-21taskset20:34:023
1130699108,1cyclictest17289-21hwlatdetect22:15:173
1130699107,2cyclictest14990-21sed19:15:213
1130699107,2cyclictest1278-21grep19:55:173
1130699105,2cyclictest225-21systemd-journal19:12:253
1130699100,9cyclictest28218-21chrt19:45:023
1130399108,1cyclictest30405-21sed19:50:002
1130399108,1cyclictest26633-21grep19:40:182
1130399108,1cyclictest25175-21sed20:45:122
1130399108,1cyclictest1828-21awk21:05:212
1130399108,1cyclictest14386-21grep20:20:242
1130399107,2cyclictest998-21sed21:00:242
1130399107,2cyclictest27761-21grep20:50:202
1130099108,1cyclictest27425-21expr20:50:181
1130099108,1cyclictest21343-21grep19:30:141
1130099108,1cyclictest18244-21taskset19:22:071
1130099107,2cyclictest3834-21taskset21:09:311
1130099105,4cyclictest20612-21taskset19:27:041
1129699108,2cyclictest16927-21taskset20:25:440
1129699108,1cyclictest4139-21ssh21:10:020
1129699108,1cyclictest31827-21aten2_r7power_c21:00:120
1129699108,1cyclictest27440-21expr00:20:140
1129699108,1cyclictest27110-21sh00:00:010
1129699108,1cyclictest12956-21grep20:20:130
1129699107,2cyclictest26290-21cat19:40:150
1129699107,2cyclictest23977-21sh23:00:130
1129699107,2cyclictest16546-21grep19:20:150
1129699102,3cyclictest23213-21sshd19:34:590
1129699100,9cyclictest10772-21ssh22:31:190
1129699100,8cyclictest0-21swapper/022:15:510
113009996,2cyclictest32571-21grep21:00:161
112969995,1cyclictest0-21swapper/020:31:200
112969993,1cyclictest28927-21fschecks_time19:45:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional