You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 06:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Mar 11, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23722210164,33sleep20-21swapper/219:05:512
2377228875,9sleep00-21swapper/019:06:310
2379128067,9sleep30-21swapper/319:06:453
2398427962,13sleep10-21swapper/119:09:161
2412799330,7cyclictest23566-21ssh23:10:131
2413399271,25cyclictest17902-21kworker/3:221:31:583
24130992718,8cyclictest525-21gdbus00:09:202
2413099270,27cyclictest0-21swapper/200:38:352
2413099270,0cyclictest0-21swapper/200:23:062
2412799272,3cyclictest17127-21diskmemload21:15:121
2412799271,2cyclictest5894-21grep22:20:211
2412799267,18cyclictest28133-21taskset23:31:431
24127992621,4cyclictest25-21ksoftirqd/123:03:081
2413099250,23cyclictest0-21swapper/200:33:562
2412799255,1cyclictest17127-21diskmemload00:26:351
2412799250,3cyclictest0-21swapper/121:25:191
2412799250,1cyclictest0-21swapper/100:23:161
2413399241,9cyclictest0-21swapper/300:25:123
2413099240,24cyclictest0-21swapper/222:46:142
2413099240,24cyclictest0-21swapper/222:46:142
2412799242,1cyclictest3945-21grep20:40:201
2412799241,3cyclictest0-21swapper/121:55:131
2413399233,18cyclictest1083-21sh23:15:253
2413099230,22cyclictest0-21swapper/221:55:202
2412799231,3cyclictest17127-21diskmemload00:36:271
2412799230,6cyclictest15438-21processes22:45:201
2412799230,6cyclictest15438-21processes22:45:201
2413099220,21cyclictest0-21swapper/223:10:132
2413099220,21cyclictest0-21swapper/221:51:372
2412799223,2cyclictest15239-21apt-get23:05:141
24127992220,1cyclictest0-21swapper/119:25:131
2412799220,20cyclictest0-21swapper/122:04:381
24133992117,3cyclictest0-21swapper/323:23:173
2413099210,21cyclictest0-21swapper/221:28:282
2412799218,12cyclictest3575-21sh22:58:041
2412799217,14cyclictest0-21swapper/123:18:151
24127992115,5cyclictest0-21swapper/122:37:431
2412799211,2cyclictest0-21swapper/121:35:241
2412499211,16cyclictest4516-21grep00:15:210
274472200,0sleep30-21swapper/322:52:523
2413399201,9cyclictest225-21systemd-journal19:35:003
2413099200,19cyclictest0-21swapper/222:21:172
2413099200,17cyclictest0-21swapper/221:45:122
2412799200,3cyclictest0-21swapper/123:27:481
2412499204,5cyclictest0-21swapper/022:20:130
2412499200,20cyclictest0-21swapper/023:49:530
2413099190,17cyclictest0-21swapper/221:10:162
2413099190,17cyclictest0-21swapper/219:40:122
2412799195,8cyclictest0-21swapper/122:40:121
2412799191,4cyclictest0-21swapper/122:30:121
2412799190,5cyclictest6278-21ssh21:41:361
2412799190,4cyclictest0-21swapper/121:30:141
2412799190,2cyclictest0-21swapper/123:20:131
24130991818,0cyclictest0-21swapper/221:35:132
24130991817,0cyclictest0-21swapper/219:35:142
2413099180,1cyclictest0-21swapper/223:25:582
2413099180,16cyclictest0-21swapper/223:55:042
2412799188,6cyclictest8510-21sh23:59:011
2412799188,6cyclictest13768-21sh21:45:581
2412799184,2cyclictest29336-21df_inode00:30:151
24127991817,1cyclictest0-21swapper/123:41:371
2412499180,2cyclictest0-21swapper/021:39:280
24133991715,2cyclictest17902-21kworker/3:221:57:023
2413399170,9cyclictest0-21swapper/322:48:063
2413399170,9cyclictest0-21swapper/322:48:063
2413399170,1cyclictest0-21swapper/320:25:113
24130991715,2cyclictest0-21swapper/223:01:222
2413099170,14cyclictest0-21swapper/200:26:252
2413099170,0cyclictest0-21swapper/222:30:442
2412799177,6cyclictest24872-21ssh00:08:431
2412499175,8cyclictest5256-21cups-browsed23:20:010
2412499174,4cyclictest25676-21/usr/sbin/munin23:50:170
2412499171,12cyclictest0-21swapper/019:10:140
24133991614,2cyclictest17902-21kworker/3:222:31:533
2413099165,6cyclictest225-21systemd-journal00:15:002
24130991611,4cyclictest5231-21apache223:31:322
2413099161,12cyclictest0-21swapper/222:07:502
2413099160,14cyclictest0-21swapper/223:35:402
2412799169,6cyclictest22287-21rm22:50:121
2412799168,7cyclictest875-21expr23:35:151
2412799168,7cyclictest28595-21expr20:25:141
2412799168,7cyclictest18272-21rm22:28:011
2412799168,7cyclictest14816-21date21:05:011
2412799168,7cyclictest11476-21head20:55:211
2412799167,8cyclictest8545-21sh00:18:031
24127991610,5cyclictest21236-21sort22:10:171
2412799160,1cyclictest0-21swapper/123:52:451
2412799160,15cyclictest20430-21rm23:46:151
2412799160,15cyclictest15720-21ssh22:06:461
2412499161,14cyclictest550-21lldpd19:33:130
2412499161,13cyclictest0-21swapper/023:58:030
24133991514,1cyclictest55750irq/128-enp2s0-23:40:123
2413399151,13cyclictest17127-21diskmemload23:58:463
2413399151,13cyclictest0-21swapper/323:33:513
2413099151,2cyclictest5560-21grep22:00:252
2413099150,10cyclictest0-21swapper/223:52:522
2412799158,6cyclictest5465-21chrt20:45:071
2412799158,6cyclictest31872-21sh22:16:231
2412799158,6cyclictest16937-21ssh00:04:031
2412799157,7cyclictest3550-21chrt19:34:511
2412799157,7cyclictest21687-21rm21:50:371
2412799157,7cyclictest11156-21basename19:50:141
2412799156,2cyclictest24262-21fgrep20:15:231
2412799156,2cyclictest16634-21basename20:00:181
2412799156,2cyclictest14242-21basename19:55:171
2412799152,4cyclictest225-21systemd-journal19:35:481
2412799150,5cyclictest225-21systemd-journal00:10:521
2412499155,8cyclictest13913-21munin-node22:30:050
2412499150,1cyclictest0-21swapper/022:47:500
2412499150,1cyclictest0-21swapper/022:47:500
2412499150,0cyclictest0-21swapper/021:25:340
2413399149,3cyclictest18846-21sort21:10:203
24133991412,2cyclictest17902-21kworker/3:223:07:023
24133991412,2cyclictest17902-21kworker/3:222:28:323
24133991411,2cyclictest19093-21wc23:45:253
24133991410,3cyclictest6596-21tail22:20:263
24133991410,2cyclictest25249-21wc23:10:253
2413399140,2cyclictest12539-21ssh22:44:153
2413099148,4cyclictest12373-21munin-run21:00:002
2413099141,11cyclictest0-21swapper/222:16:082
2413099140,1cyclictest0-21swapper/220:45:122
2413099140,14cyclictest0-21swapper/222:14:242
2412799148,5cyclictest8253-21taskset19:43:141
2412799148,5cyclictest27453-21head19:15:171
2412799147,6cyclictest7574-21sh21:23:241
2412799147,6cyclictest7574-21sh21:23:241
2412799147,6cyclictest29750-21grep19:20:171
2412799147,6cyclictest27139-21cat20:20:261
2412799147,6cyclictest24777-21expr19:10:151
2412799147,6cyclictest2129-21sed20:35:261
2412799147,6cyclictest167782sleep121:05:391
2412799146,2cyclictest18264-21sh21:10:151
2412799145,1cyclictest10465-21taskset19:46:081
2412799142,3cyclictest225-21systemd-journal20:05:541
2412799140,2cyclictest225-21systemd-journal20:55:011
2412499146,7cyclictest14759-21munin-run21:05:000
24124991411,2cyclictest13-21ksoftirqd/021:11:250
2412499140,1cyclictest0-21swapper/023:35:570
2412499140,1cyclictest0-21swapper/021:19:360
2412499140,0cyclictest0-21swapper/022:12:260
2413399139,3cyclictest10786-21sleep19:50:003
2413399139,2cyclictest10593-21tr00:00:153
2413399139,2cyclictest0-21swapper/321:20:113
2413399139,2cyclictest0-21swapper/321:20:113
2413399134,8cyclictest25787-31find00:10:343
2413399134,6cyclictest30403-21cat22:15:263
24133991310,2cyclictest7001-21head19:40:173
24133991310,2cyclictest5622-21grep22:00:263
24133991310,2cyclictest487-21avahi-daemon23:52:083
24133991310,2cyclictest14747-21cron21:05:003
24133991310,2cyclictest11855-21grep21:45:153
2413399130,2cyclictest0-21swapper/300:05:143
2413399130,1cyclictest0-21swapper/322:06:203
2413399130,12cyclictest1121-21tune2fs23:35:173
2413099139,3cyclictest20281-21cron20:09:592
2413099131,0cyclictest0-21swapper/223:16:542
24130991310,2cyclictest31218-21grep20:30:182
24130991310,2cyclictest30442-21ssh22:35:162
24130991310,2cyclictest22178-21grep20:10:262
24130991310,2cyclictest17097-21grep23:05:272
2413099130,2cyclictest0-21swapper/223:49:392
2413099130,1cyclictest0-21swapper/221:22:162
2413099130,1cyclictest0-21swapper/221:22:152
2412799136,1cyclictest20895-21expr20:10:151
2412499139,3cyclictest23991-21sh23:29:590
2412499135,7cyclictest13913-21munin-node00:36:010
24124991312,1cyclictest13-21ksoftirqd/023:00:270
24124991312,1cyclictest0-21swapper/023:05:130
24124991310,2cyclictest29669-21ssh22:35:120
24124991310,2cyclictest27278-21sleep21:55:000
24124991310,2cyclictest22228-21grep20:10:260
24124991310,2cyclictest14811-21grep22:25:250
2413399129,2cyclictest4279-21ls00:35:003
2413399129,2cyclictest29978-21tr22:35:143
2413399129,2cyclictest2890-21aten2_r7power_v21:40:143
2413399129,2cyclictest235072sleep320:15:163
2413399129,2cyclictest20810-21cut20:10:153
2413399129,2cyclictest17949-21wc20:05:003
2413399128,3cyclictest21687-21sed22:10:213
2413399126,4cyclictest10179-21wget21:25:133
2413399121,1cyclictest17127-21diskmemload00:35:553
24133991210,2cyclictest39-21ksoftirqd/323:00:113
24133991210,2cyclictest17902-21kworker/3:222:56:223
2413099129,2cyclictest9109-21grep23:40:142
2413099129,2cyclictest3937-21grep00:15:172
2413099129,2cyclictest3319-21sh21:40:152
2413099129,2cyclictest31253-21ls19:25:002
2413099129,2cyclictest30948-21tr22:55:162
2413099129,2cyclictest27364-21sort19:15:152
2413099129,2cyclictest26017-21grep19:10:272
2413099129,2cyclictest14868-21grep22:25:252
2413099124,4cyclictest225-21systemd-journal21:05:182
2412799129,2cyclictest32606-21cron20:35:001
2412499129,2cyclictest9665-21grep23:40:160
2412499129,2cyclictest8015-21grep23:20:130
2412499129,2cyclictest4398-21cat20:40:260
2412499129,2cyclictest29851-21head19:20:170
2412499129,2cyclictest29459-21grep22:15:170
2412499129,2cyclictest12361-21dump-pmu-power21:00:000
2412499129,2cyclictest12267-21cut22:05:130
2412499121,5cyclictest26637-21munin-run00:10:000
2412499120,1cyclictest0-21swapper/021:43:460
2413399118,2cyclictest4743-21taskset20:42:183
2413399118,2cyclictest3934-21cut19:35:153
2413399118,2cyclictest3764-21ssh00:15:153
2413399118,2cyclictest31047-21chrt19:22:353
2413399118,2cyclictest28609-21cat21:35:263
2413399118,2cyclictest27741-21sed21:15:263
2413399118,2cyclictest27351-21taskset20:21:123
2413399118,2cyclictest27003-21tr19:15:133
2413399118,2cyclictest19581-21wget21:50:143
2413399118,2cyclictest16423-21ssh23:25:143
2413399118,2cyclictest11961-21expr00:20:153
2413099119,1cyclictest26651-21grep20:20:202
2413099118,2cyclictest3532-21sort20:40:162
2413099118,2cyclictest25142-21sh22:51:032
2413099118,2cyclictest18839-21grep21:30:142
2413099118,2cyclictest15430-21taskset19:58:592
2413099118,2cyclictest13185-21ssh23:23:032
2413099118,2cyclictest13046-21sh00:01:002
2413099117,3cyclictest27597-21wc21:15:262
2413099116,4cyclictest728-21tr20:35:162
2413099115,2cyclictest225-21systemd-journal20:25:182
2413099114,6cyclictest7086-21wc22:40:202
2413099111,5cyclictest225-21systemd-journal19:31:182
2412499119,1cyclictest302592sleep020:30:000
2412499119,1cyclictest264342sleep000:28:470
2412499119,1cyclictest14495-21chrt00:21:090
2412499119,1cyclictest11273-21grep19:50:140
2412499118,2cyclictest8943-21tr19:45:130
2412499118,2cyclictest7522-21head19:40:220
2412499118,2cyclictest717-21grep19:25:250
2412499118,2cyclictest6237-21ssh22:00:510
2412499118,2cyclictest5670-21ssh22:40:110
2412499118,2cyclictest30645-21grep20:30:150
2412499118,2cyclictest30300-21rm22:55:000
2412499118,2cyclictest3006-21grep21:20:180
2412499118,2cyclictest3006-21grep21:20:180
2412499118,2cyclictest29851-21cat00:30:180
2412499118,2cyclictest29769-21apache200:10:540
2412499118,2cyclictest27180-21ssh23:30:580
2412499118,2cyclictest25133-21gpgv20:20:010
2412499118,2cyclictest16703-21ssh21:48:150
2412499118,2cyclictest15394-21ssh00:02:520
2412499118,2cyclictest13412-21apache_volume19:55:110
2412499117,3cyclictest16235-21ls21:05:210
2412499117,1cyclictest26746-21ssh23:11:220
2412499113,7cyclictest30699-21ssh22:55:150
2412499110,11cyclictest0-21swapper/021:56:120
2413399108,1cyclictest9942-21taskset20:54:363
2413399108,1cyclictest8872sleep319:26:173
2413399108,1cyclictest7801-21ls20:50:003
2413399108,1cyclictest24806-21expr19:10:163
2413399108,1cyclictest2324-21taskset20:36:593
2413399108,1cyclictest14189-21cut19:55:163
2413399107,2cyclictest32501-21taskset20:33:403
2413399107,2cyclictest14986-21taskset21:05:103
2413399107,2cyclictest10447-21sort20:55:143
2413399106,3cyclictest11799-21grep19:50:173
2413099108,1cyclictest8189-21wc20:50:152
2413099108,1cyclictest23862-21cut20:15:192
2413099108,1cyclictest14528-21taskset21:02:142
2413099108,1cyclictest11508-21cat19:50:162
2413099108,1cyclictest104362sleep219:45:462
2413099107,2cyclictest15815-21aten2_r7power_c20:00:142
2413099106,3cyclictest31614-21grep19:25:132
2412499108,1cyclictest8287-21expr20:50:150
2412499108,1cyclictest27751-21cat20:25:000
2412499108,1cyclictest20944-21ssh21:30:550
2412499107,2cyclictest704-21tr20:35:150
2412499107,2cyclictest6591-21grep20:45:190
2412499107,2cyclictest27502-21tail19:15:160
2412499107,2cyclictest17029-21grep20:00:220
2412499100,9cyclictest20007-21taskset20:06:290
241339996,2cyclictest18441-21grep20:05:153
241249997,1cyclictest4533-21tune2fs19:35:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional