You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-30 - 22:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 30, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23216210164,33sleep20-21swapper/207:07:062
2317429179,8sleep00-21swapper/007:06:330
2327428774,9sleep30-21swapper/307:07:523
2325728258,13sleep10-21swapper/107:07:371
134372380,0sleep30-21swapper/309:01:243
2353199300,30cyclictest0-21swapper/312:16:393
2352399240,22cyclictest0-21swapper/111:07:331
2352799230,23cyclictest0-21swapper/212:24:402
23523992322,1cyclictest38150irq/131-snd_hda12:06:181
23523992216,5cyclictest1428-21grep11:15:161
2353199213,1cyclictest0-21swapper/310:56:313
23531992120,1cyclictest0-21swapper/310:30:103
2352399212,4cyclictest141rcu_preempt11:54:501
2352399212,4cyclictest141rcu_preempt11:54:491
23523992119,1cyclictest22338-21apt-get08:15:131
23523992114,7cyclictest0-21swapper/111:46:411
2352399210,0cyclictest0-21swapper/112:28:211
2351799210,17cyclictest0-21swapper/012:30:130
23531992015,5cyclictest16120-21diskmemload09:50:443
2353199200,3cyclictest0-21swapper/311:48:153
2352399200,1cyclictest3615-21if_enp2s011:55:181
250002190,0sleep00-21swapper/009:15:020
2353199199,8cyclictest0-21swapper/310:01:313
23531991918,1cyclictest0-21swapper/311:34:173
23531991917,1cyclictest16120-21diskmemload10:19:243
23531991917,1cyclictest16120-21diskmemload09:40:213
23531991917,1cyclictest0-21swapper/312:25:023
2352799190,19cyclictest0-21swapper/209:19:502
2352799190,18cyclictest4960-21kworker/u8:211:38:142
23523991918,1cyclictest38150irq/131-snd_hda10:37:531
2352399190,18cyclictest0-21swapper/109:18:331
2351799194,4cyclictest0-21swapper/009:20:300
23517991918,1cyclictest0-21swapper/012:10:160
2353199187,11cyclictest0-21swapper/308:30:153
23531991816,1cyclictest16120-21diskmemload10:22:593
23531991814,3cyclictest16120-21diskmemload11:27:123
23531991813,5cyclictest0-21swapper/310:12:153
2352799181,3cyclictest16120-21diskmemload12:29:062
23523991817,1cyclictest0-21swapper/112:15:231
2352399180,18cyclictest0-21swapper/112:22:131
2352399180,18cyclictest0-21swapper/110:54:341
2352399180,17cyclictest2903-21cpu09:40:161
2353199178,7cyclictest16120-21diskmemload09:18:153
2352799173,11cyclictest3767-21runrttasks10:00:012
2352799170,16cyclictest0-21swapper/211:16:272
2352799170,0cyclictest0-21swapper/211:57:482
2352399173,13cyclictest16120-21diskmemload11:30:251
23523991716,1cyclictest0-21swapper/110:44:441
2352399170,17cyclictest0-21swapper/110:16:361
2353199168,7cyclictest0-21swapper/307:50:133
2353199167,8cyclictest16120-21diskmemload11:59:413
23531991614,2cyclictest39-21ksoftirqd/312:21:123
23531991614,1cyclictest28217-21sensors_fan09:35:233
23531991614,1cyclictest0-21swapper/309:25:213
23531991613,2cyclictest22061-21cron08:15:003
23531991611,4cyclictest0-21swapper/309:10:153
23531991610,5cyclictest0-21swapper/311:24:123
2352799160,9cyclictest0-21swapper/210:15:122
2352799160,2cyclictest0-21swapper/212:03:322
2352799160,1cyclictest0-21swapper/211:48:042
2352799160,15cyclictest0-21swapper/211:00:482
2352799160,15cyclictest0-21swapper/209:46:062
23523991611,3cyclictest475-21in:imuxsock09:20:311
2351799162,1cyclictest141rcu_preempt10:53:210
2351799161,8cyclictest9472-21idleruntime-cro11:40:000
2353199157,7cyclictest16120-21diskmemload11:53:553
2353199157,7cyclictest16120-21diskmemload11:53:553
2353199153,1cyclictest0-21swapper/310:48:573
2353199152,13cyclictest16120-21diskmemload09:58:473
2352799151,14cyclictest17263-21kworker/u8:211:08:362
2352799150,3cyclictest0-21swapper/210:23:032
2352799150,2cyclictest0-21swapper/210:11:012
2352799150,1cyclictest0-21swapper/209:51:372
23523991514,1cyclictest0-21swapper/112:34:331
23523991514,1cyclictest0-21swapper/110:49:031
23523991511,3cyclictest29380-21apache210:21:361
2352399150,14cyclictest0-21swapper/109:13:271
2351799159,5cyclictest277542sleep009:55:030
2351799157,7cyclictest10189-21munin-run07:50:000
23517991510,3cyclictest19031-21wc12:05:010
2351799150,13cyclictest0-21swapper/012:35:130
299212140,0sleep00-21swapper/011:51:230
299212140,0sleep00-21swapper/011:51:220
2353199148,6cyclictest16120-21diskmemload11:12:433
2353199147,7cyclictest0-21swapper/312:00:393
2353199147,4cyclictest12702-21sendmail12:39:593
2353199146,8cyclictest0-21swapper/310:40:333
2353199145,9cyclictest0-21swapper/307:35:133
2353199143,10cyclictest0-21swapper/309:48:373
23531991413,1cyclictest381rcuc/309:30:343
23531991411,3cyclictest23512-21cyclictest10:05:233
23531991411,2cyclictest26860-21munin-run08:25:013
23531991410,3cyclictest27815-21sed07:15:263
23531991410,3cyclictest0-21swapper/307:44:593
2353199140,1cyclictest0-21swapper/309:05:123
2353199140,0cyclictest0-21swapper/310:25:083
2352799145,7cyclictest537-21snmpd09:43:572
23527991410,3cyclictest6629-21tr07:40:182
23527991410,3cyclictest28271-21systemctl07:20:002
2352799140,14cyclictest0-21swapper/212:06:432
23523991411,2cyclictest38150irq/131-snd_hda11:20:201
23523991411,2cyclictest16120-21diskmemload12:10:141
23523991411,2cyclictest11883-21head09:45:191
2352399141,12cyclictest0-21swapper/110:06:531
23523991410,3cyclictest19294-21grep08:05:251
23517991412,1cyclictest16120-21diskmemload10:11:010
2351799140,8cyclictest8480-21awk11:20:010
2353199139,4cyclictest55750irq/128-enp2s0-11:35:203
2353199139,3cyclictest8828-21head07:45:163
23531991312,1cyclictest29406-21apache211:09:083
23531991312,1cyclictest0-21swapper/310:50:193
23531991312,1cyclictest0-21swapper/309:23:313
23531991311,1cyclictest16120-21diskmemload12:33:143
23531991311,1cyclictest16120-21diskmemload10:38:133
23531991310,2cyclictest5891-21grep08:45:283
23531991310,2cyclictest5891-21grep08:45:283
23531991310,2cyclictest2994-21grep08:40:213
23531991310,2cyclictest28641-21grep07:20:143
23531991310,2cyclictest27881-21sed08:25:143
2352799139,3cyclictest26346-21sed08:20:262
2352799138,3cyclictest16217-21cat09:10:002
2352799136,6cyclictest17895-21df_inode11:25:152
2352799135,6cyclictest29380-21apache210:06:012
23527991310,2cyclictest9205-21grep11:20:122
23527991310,2cyclictest28007-21grep12:10:152
2352799130,13cyclictest0-21swapper/209:29:452
2352399139,2cyclictest20993-21ls08:10:201
23523991310,2cyclictest690-21tr10:55:181
23523991310,2cyclictest32693-21head08:35:171
23523991310,2cyclictest29266-21head07:20:161
23523991310,2cyclictest19614-21tr09:50:141
2352399130,0cyclictest0-21swapper/109:26:091
2351799139,3cyclictest22223-21grep10:30:140
2351799139,2cyclictest2859-21cut09:40:160
2351799139,2cyclictest18115-21sed11:25:170
2351799139,2cyclictest11699-21cat12:20:000
2351799137,4cyclictest29380-21apache210:43:380
2351799132,8cyclictest0-21swapper/010:15:180
2351799131,6cyclictest24472-21munin-run08:20:000
23517991310,2cyclictest26920-21ntpq11:30:200
23517991310,2cyclictest1550-21grep07:30:160
2353199129,2cyclictest28586-21grep12:10:183
2353199129,2cyclictest17310-21cron08:05:003
2353199129,2cyclictest11124-21dump-pmu-power09:00:003
2353199124,8cyclictest0-21swapper/311:15:153
2353199124,4cyclictest225-21systemd-journal07:29:593
23531991211,1cyclictest16120-21diskmemload11:04:253
23531991211,1cyclictest0-21swapper/311:40:143
2352799129,2cyclictest8824-21grep07:45:172
2352799129,2cyclictest29824-21grep08:30:152
2352799129,2cyclictest28539-21grep09:35:262
2352799129,2cyclictest27531-21sed11:50:182
2352799129,2cyclictest27531-21sed11:50:182
2352799129,2cyclictest24205-21cut10:50:192
2352799129,2cyclictest18579-21tr09:30:162
2352799125,5cyclictest23218-21tr10:30:192
2352799121,9cyclictest8754-21apt08:55:012
2352799121,10cyclictest14037-21sh11:42:032
2352799121,10cyclictest12709-21perf12:39:592
2352399129,2cyclictest9735-21grep08:55:171
2352399129,2cyclictest8345-21head08:50:291
2352399129,2cyclictest31591-21cut07:25:161
2352399129,2cyclictest28168-21grep09:55:141
2352399129,2cyclictest26088-21cron09:34:591
2352399129,2cyclictest1887-21grep11:35:141
2352399129,2cyclictest12221-21sort09:00:131
2352399129,2cyclictest1057-21cut07:30:131
2352399128,3cyclictest26309-21wc11:10:251
2352399127,4cyclictest7054-21cat07:40:211
2352399126,5cyclictest11596-21df12:00:151
2352399120,11cyclictest5645-21tail12:35:161
2351799129,2cyclictest8020-21tr11:00:130
2351799129,2cyclictest801-21ls10:55:190
2351799129,2cyclictest7950-21cat07:45:000
2351799129,2cyclictest5838-21ssh10:20:150
2351799129,2cyclictest31505-21taskset07:25:150
2351799129,2cyclictest26819-21sort07:15:170
2351799129,2cyclictest16114-21turbostat09:10:010
2351799129,2cyclictest14934-21systemctl08:00:000
2351799129,2cyclictest12501-21grep10:05:140
2351799126,6cyclictest29380-21apache212:23:160
2351799126,5cyclictest537-21snmpd09:50:000
2351799123,3cyclictest26182-21idleruntime-cro11:50:000
2351799122,8cyclictest31655-21ls08:35:000
23517991211,1cyclictest13-21ksoftirqd/009:25:190
2351799120,12cyclictest0-21swapper/011:57:110
2353199119,1cyclictest26852sleep307:30:403
2353199118,2cyclictest24516-21sed07:10:163
2353199118,2cyclictest21526-21ssh12:05:343
2353199117,3cyclictest14048-21head07:55:223
2353199114,6cyclictest8892-21latency_hist08:55:013
2352799119,1cyclictest4089-21head07:35:182
2352799119,1cyclictest29533-21df12:30:152
2352799118,2cyclictest4897-21ssh12:15:192
2352799118,2cyclictest4008-21ssh09:20:542
2352799118,2cyclictest3514-21sed08:40:262
2352799118,2cyclictest32666-21ssh10:35:282
2352799118,2cyclictest31831-21ssh10:55:092
2352799118,2cyclictest28206-21grep09:55:142
2352799118,2cyclictest26206-21ssh11:30:152
2352799118,2cyclictest25108-21ssh11:10:152
2352799118,2cyclictest21696-21ssh09:12:492
2352799118,2cyclictest21512-21ssh10:49:032
2352799118,2cyclictest15509-21seq08:00:142
2352799118,2cyclictest13978-21sh07:55:222
2352799118,2cyclictest10282-21ssh10:41:472
2352799117,3cyclictest24529-21grep07:10:162
2352799116,2cyclictest225-21systemd-journal08:15:012
2352799115,2cyclictest225-21systemd-journal07:25:112
2352399118,2cyclictest5079-21taskset07:35:571
2352399118,2cyclictest3832-21ssh10:00:041
2352399118,2cyclictest38150irq/131-snd_hda10:10:131
2352399118,2cyclictest28667-21taskset09:35:281
2352399118,2cyclictest26349-21grep08:20:251
2352399118,2cyclictest23494-21cut10:30:221
2352399118,2cyclictest17589-21grep11:25:141
2352399118,2cyclictest16780-21ssh10:26:221
2352399118,2cyclictest15750-21chrt09:05:341
2352399118,2cyclictest15281-21cut08:00:131
2352399118,2cyclictest14119-21ssh11:42:081
2352399118,2cyclictest14052-21grep07:55:211
2352399117,3cyclictest28135-21taskset07:18:401
2352399115,2cyclictest225-21systemd-journal08:29:101
2352399110,10cyclictest1855-21chrt08:40:051
2352399110,10cyclictest13204-21chrt11:03:021
2351799118,2cyclictest9447-21expr08:55:150
2351799118,2cyclictest8946-21rm11:20:110
2351799118,2cyclictest30227-21cat10:35:010
2351799118,2cyclictest29336-21hwlatdetect07:20:170
2351799118,2cyclictest24533-21bc11:10:110
2351799118,2cyclictest2154-21grep08:40:150
2351799118,2cyclictest21404-21cat12:25:140
2351799118,2cyclictest19601-21ssh11:06:320
2351799118,2cyclictest19170-21ssh12:05:050
2351799118,2cyclictest17735-21wc09:10:200
2351799118,2cyclictest16816-21grep10:45:250
2351799118,2cyclictest16180-21ssh10:25:540
2351799118,2cyclictest13491-21ssh11:41:380
2351799118,2cyclictest12937-21fgrep09:00:190
2351799117,3cyclictest20164-21sed09:30:290
2351799116,4cyclictest27495-21tail09:35:170
2351799111,9cyclictest25889-21perf07:15:000
23517991110,1cyclictest13-21ksoftirqd/007:54:190
2351799110,7cyclictest17325-21ls08:05:010
2351799110,10cyclictest11497-21cat09:45:170
2353199108,1cyclictest22710-21expr08:15:143
2353199108,1cyclictest15512sleep308:39:273
2353199107,2cyclictest17886-21cat08:05:143
2352799108,1cyclictest19029-21cat08:05:242
2352799108,1cyclictest13338-21taskset10:25:042
2352799107,2cyclictest9501-21df08:55:162
2352799107,2cyclictest2988-21taskset07:34:352
2352799107,2cyclictest28765-21grep07:20:142
2352799107,2cyclictest28331-21grep08:25:192
2352799107,2cyclictest1600-21cron08:40:002
2352799107,2cyclictest11907-21expr09:00:132
2352799106,3cyclictest22436-21grep08:15:132
2352799100,9cyclictest4389-21cut08:45:162
2352799100,9cyclictest4389-21cut08:45:162
2352399108,1cyclictest24807-21sed07:10:181
2352399107,2cyclictest8657-21entropy07:45:161
2352399107,2cyclictest6029-21taskset08:45:421
2352399107,2cyclictest6029-21taskset08:45:411
2352399107,2cyclictest12345-21taskset07:52:311
2352399100,9cyclictest30038-21grep08:30:161
2351799108,1cyclictest8740-21cron08:55:000
2351799108,1cyclictest6280-21chrt08:48:580
2351799108,1cyclictest6280-21chrt08:48:570
2351799108,1cyclictest6154-21ssh10:00:310
2351799108,1cyclictest32381-21ls08:35:150
2351799108,1cyclictest25105-21cat08:20:140
2351799107,2cyclictest3445-21grep07:35:150
2351799107,2cyclictest28370-21ps08:25:200
2351799107,2cyclictest18320-21expr08:05:160
235279997,1cyclictest11603-21ntpq07:50:192
235179997,1cyclictest21346-21chrt08:10:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional