You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 06:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu May 14, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3022529774,6sleep20-21swapper/219:09:042
3002729566,9sleep30-21swapper/319:06:333
3007329481,9sleep00-21swapper/019:07:060
3026928867,17sleep10-21swapper/119:09:391
196292670,0sleep20-21swapper/222:25:142
30386992727,0cyclictest0-21swapper/122:20:131
30386992626,0cyclictest0-21swapper/123:14:091
3038699240,24cyclictest0-21swapper/100:04:581
30381992423,0cyclictest0-21swapper/023:37:580
3038199233,1cyclictest141rcu_preempt22:11:490
86352220,2sleep20-21swapper/200:14:432
3039199210,20cyclictest0-21swapper/323:34:063
3039199210,1cyclictest0-21swapper/300:28:223
30388992119,1cyclictest0-21swapper/221:35:122
3038699213,1cyclictest0-21swapper/122:16:021
3038699211,1cyclictest141rcu_preempt23:42:091
30381992119,1cyclictest0-21swapper/021:10:150
3038899205,10cyclictest7455-21apt19:30:002
3038899200,1cyclictest0-21swapper/221:42:052
3038699202,17cyclictest0-21swapper/121:25:521
3038699200,5cyclictest141rcu_preempt23:34:491
3038699190,2cyclictest0-21swapper/123:00:511
3038199190,2cyclictest0-21swapper/023:13:340
30391991810,6cyclictest6300-21apt-get20:35:133
3038899181,16cyclictest30109-21irqstats20:15:202
3038899180,16cyclictest0-21swapper/221:45:162
3038899180,14cyclictest0-21swapper/221:29:082
3038699180,1cyclictest0-21swapper/123:54:351
3038199180,0cyclictest0-21swapper/023:34:070
3039199173,6cyclictest225-21systemd-journal22:49:593
3039199171,16cyclictest8038-21ssh00:14:143
3038899170,1cyclictest0-21swapper/222:30:132
3038899170,15cyclictest0-21swapper/200:00:192
3038699170,14cyclictest0-21swapper/100:25:161
30391991615,1cyclictest0-21swapper/323:51:553
30391991614,1cyclictest0-21swapper/323:59:103
3039199160,1cyclictest0-21swapper/323:27:303
3038899164,7cyclictest0-21swapper/200:35:132
3038899160,0cyclictest0-21swapper/219:20:132
3038699164,5cyclictest225-21systemd-journal21:41:461
3038699164,2cyclictest30372-21cyclictest21:24:351
30386991615,1cyclictest0-21swapper/121:31:191
3038699160,2cyclictest0-21swapper/121:52:251
3038699160,2cyclictest0-21swapper/121:52:241
3038699160,2cyclictest0-21swapper/100:24:121
3038699160,1cyclictest0-21swapper/123:20:191
3038699160,16cyclictest0-21swapper/122:09:161
3038199163,11cyclictest24967-21apache_accesses00:05:110
3038199160,2cyclictest0-21swapper/023:46:420
3038199160,1cyclictest0-21swapper/022:38:470
3039199153,3cyclictest225-21systemd-journal00:05:443
30391991514,1cyclictest39-21ksoftirqd/323:05:003
3039199150,13cyclictest0-21swapper/323:17:463
30388991515,0cyclictest0-21swapper/223:51:002
3038899151,14cyclictest0-21swapper/223:59:102
3038899150,15cyclictest0-21swapper/222:47:302
30386991515,0cyclictest0-21swapper/100:38:151
30386991514,1cyclictest0-21swapper/123:48:181
30386991514,1cyclictest0-21swapper/123:05:131
30386991511,4cyclictest38150irq/131-snd_hda21:19:051
3038199150,9cyclictest15862-21munin-run21:25:000
3038199150,14cyclictest0-21swapper/022:52:580
3039199145,2cyclictest225-21systemd-journal21:25:003
30391991410,3cyclictest1607-21wc19:15:183
3039199140,1cyclictest0-21swapper/322:34:463
3038899149,3cyclictest26062-21grep20:05:242
3038899148,2cyclictest12795-21http23:00:002
30388991411,2cyclictest15817-21sh20:54:592
30388991411,2cyclictest15738-21grep23:40:132
30388991410,3cyclictest14762-21grep23:20:142
3038899140,8cyclictest0-21swapper/223:00:222
3038899140,14cyclictest0-21swapper/223:17:192
3038899140,14cyclictest0-21swapper/222:13:382
3038699148,2cyclictest141rcu_preempt00:10:271
30386991414,0cyclictest0-21swapper/122:36:471
30386991410,3cyclictest24482-21basename20:05:121
30381991410,3cyclictest669-21tail23:50:170
30381991410,2cyclictest1057-21grep19:15:150
3039199139,3cyclictest6433-21grep19:25:183
3039199139,2cyclictest24057-21tr21:10:163
30391991312,1cyclictest0-21swapper/321:35:143
30391991310,2cyclictest24136-21idleruntime-cro21:29:593
30391991310,2cyclictest23052-21grep23:05:193
30391991310,2cyclictest17891-21sh22:05:013
30391991310,2cyclictest16815-21apt21:45:003
30391991310,2cyclictest15670-21head19:45:183
3038899139,2cyclictest11419-21wc22:00:212
3038899136,3cyclictest23181-21diskmemload00:06:512
30388991310,2cyclictest377-21wc19:15:002
30388991310,2cyclictest31762-21cron23:50:012
30388991310,2cyclictest30893-21grep23:30:012
30388991310,2cyclictest23685-21sed20:00:272
30388991310,2cyclictest23685-21sed20:00:262
30388991310,2cyclictest18404-21sendmail21:00:002
30388991310,2cyclictest16527-21grep19:45:272
30388991310,2cyclictest1333-21head19:15:162
30388991310,2cyclictest10310-21grep00:15:142
3038899130,1cyclictest0-21swapper/222:53:022
3038899130,13cyclictest0-21swapper/200:32:062
3038699139,3cyclictest12821-21ls23:00:001
3038699138,4cyclictest23015-21taskset21:08:061
30386991310,2cyclictest7967-21tr19:30:131
30386991310,2cyclictest6099-21cat20:35:011
30386991310,2cyclictest3278-21sed00:30:141
30386991310,2cyclictest27907-21wc22:10:181
30386991310,2cyclictest25494-21wc21:10:281
3038699130,11cyclictest31322-21grep19:10:171
3038199137,3cyclictest17024-21cron19:50:000
3038199136,6cyclictest3228-21apt-get22:35:000
3038199131,6cyclictest9493-21sendmail22:00:010
30381991310,2cyclictest23788-21ssh23:25:160
30381991310,2cyclictest228702sleep020:00:190
30381991310,2cyclictest228702sleep020:00:180
30381991310,2cyclictest22511-21grep22:45:260
3038199130,11cyclictest0-21swapper/023:04:410
3038199130,0cyclictest0-21swapper/022:09:380
3039199129,2cyclictest8366-21grep19:30:163
3039199129,2cyclictest8057-21cut23:35:173
3039199129,2cyclictest5149-21cat19:25:003
3039199129,2cyclictest4020-21tr22:35:133
3039199129,2cyclictest3238-21grep20:25:253
3039199129,2cyclictest31869-21tr20:20:143
3039199129,2cyclictest31844-21wc23:50:013
3039199129,2cyclictest2921-21ssh00:30:143
3039199129,2cyclictest25487-21cut21:30:173
3039199129,2cyclictest17532-21grep20:55:203
3039199129,2cyclictest1745-21cut21:55:133
3039199129,2cyclictest17413-21ls00:19:593
3039199129,2cyclictest15425-21sed20:50:243
3039199129,2cyclictest14899-21tr23:20:143
3039199128,2cyclictest14236-21sed19:40:263
3039199127,4cyclictest39-21ksoftirqd/321:45:223
3039199126,2cyclictest455-21dbus-daemon21:00:223
3039199121,10cyclictest25739-21apt-get21:50:133
3039199121,10cyclictest25739-21apt-get21:50:123
3039199120,1cyclictest0-21swapper/323:10:373
3039199120,12cyclictest0-21swapper/322:41:013
3038899129,2cyclictest782-21chrt23:11:092
3038899129,2cyclictest20115-21perf22:45:002
3038899129,2cyclictest18634-21cat22:05:142
3038899129,2cyclictest11761-21ssh22:20:162
3038899129,2cyclictest10806-21turbostat20:45:002
3038899128,3cyclictest29912-21ssh00:26:192
3038899128,3cyclictest26538-21grep21:50:162
3038899128,3cyclictest26538-21grep21:50:152
3038899122,8cyclictest15854-21latency_hist21:25:002
3038899121,11cyclictest0-21swapper/223:09:002
30388991211,1cyclictest0-21swapper/200:22:292
3038899120,11cyclictest21809-21tr20:00:002
3038899120,0cyclictest0-21swapper/221:31:472
3038699129,2cyclictest7002-21grep20:35:171
3038699129,2cyclictest29838-21cut20:15:171
3038699129,2cyclictest21315-21tail19:55:261
3038699129,2cyclictest14962-21wc20:50:201
3038699129,2cyclictest14085-21cron23:20:001
3038699129,2cyclictest12996-21head20:45:281
3038699127,4cyclictest6263-21ssh21:38:201
3038699126,5cyclictest20795-21cron21:05:001
3038699122,7cyclictest28530-21fschecks_time22:30:161
3038699120,1cyclictest0-21swapper/122:51:471
3038199129,2cyclictest4584-21grep19:20:260
3038199129,2cyclictest3413-21sort00:30:150
3038199129,2cyclictest32305-21grep20:20:160
3038199129,2cyclictest32138-21grep21:15:140
3038199129,2cyclictest30218-21cat20:15:220
3038199129,2cyclictest21849-21sed21:05:150
3038199129,2cyclictest20395-21grep19:55:170
3038199129,2cyclictest18489-21grep00:00:250
3038199129,2cyclictest10843-21head22:00:150
3038199129,2cyclictest10304-21ssh22:19:490
3038199129,1cyclictest5946-21sort19:25:160
3038199128,3cyclictest27427-21ls21:50:250
3038199128,3cyclictest27427-21ls21:50:250
3038199128,2cyclictest29108-21cat20:15:010
3038199128,2cyclictest17941-21grep20:55:250
3038199124,7cyclictest16629-21users23:20:280
3038199122,2cyclictest0-21swapper/022:58:070
30381991210,1cyclictest189582sleep021:45:250
3039199118,2cyclictest7827-21ssh22:17:523
3039199118,2cyclictest6024-21cron20:35:013
3039199118,2cyclictest29851-21cat22:50:173
3039199118,2cyclictest26922-21grep22:10:133
3039199118,2cyclictest26311-21chrt20:07:073
3039199118,2cyclictest23455-21sh20:00:263
3039199118,2cyclictest23455-21sh20:00:263
3039199118,2cyclictest21766-21cron20:00:003
3039199118,2cyclictest20688-21ssh22:05:403
3039199118,2cyclictest19178-21taskset19:52:173
3039199118,2cyclictest18966-21rm00:00:423
3039199118,2cyclictest17085-21ssh00:38:303
3039199118,2cyclictest15448-21ssh23:40:063
3039199118,2cyclictest11336-21ssh22:58:563
3039199118,2cyclictest11146-21ssh22:20:133
3039199117,3cyclictest21793-21head21:05:163
3039199117,3cyclictest10692-21sort19:35:163
3039199110,9cyclictest23181-21diskmemload22:25:143
3039199110,10cyclictest20384-21taskset00:20:373
3038899119,1cyclictest19454-21grep21:00:162
3038899118,2cyclictest8395-21sort19:30:162
3038899118,2cyclictest746-21grep23:30:252
3038899118,2cyclictest5415-21ssh21:56:482
3038899118,2cyclictest5205-21cut22:35:202
3038899118,2cyclictest5037-21ssh22:15:412
3038899118,2cyclictest32564-21expr21:15:152
3038899118,2cyclictest30390-21ssh21:14:072
3038899118,2cyclictest14597-21ssh23:39:372
3038899114,6cyclictest7042-21grep20:35:172
3038699119,1cyclictest23181-21diskmemload21:45:091
3038699118,2cyclictest7988-21tr23:35:161
3038699118,2cyclictest7198-21chrt19:26:501
3038699118,2cyclictest28550-21ssh23:28:171
3038699118,2cyclictest28296-21expr20:10:251
3038699118,2cyclictest24925-21ssh00:05:091
3038699118,2cyclictest24478-21ssh22:46:531
3038699118,2cyclictest2321-21ssh21:55:151
3038699118,2cyclictest2266-21grep20:25:171
3038699118,2cyclictest20278-21grep22:25:161
3038699118,2cyclictest15221-21ssh22:02:541
3038699118,2cyclictest12956-21ssh00:16:321
3038699118,2cyclictest12548-21wc22:40:141
3038699118,2cyclictest10114-21sed23:55:261
3038699117,2cyclictest17359-21grep19:50:111
3038699117,2cyclictest10348-21cut19:35:151
3038699116,4cyclictest1288-21dump-pmu-power20:25:011
3038199118,2cyclictest9686-21tr21:40:150
3038199118,2cyclictest9353-21cat23:55:180
3038199118,2cyclictest8953-21head19:30:200
3038199118,2cyclictest889-21rm00:10:070
3038199118,2cyclictest6623-21expr23:15:160
3038199118,2cyclictest3746-21cat20:30:010
3038199118,2cyclictest28949-21ssh00:25:340
3038199118,2cyclictest25293-21ssh21:30:160
3038199118,2cyclictest25231-21ssh23:06:360
3038199118,2cyclictest23684-21ssh22:27:090
3038199118,2cyclictest23616-21ssh00:22:570
3038199118,2cyclictest22510-21ssh21:28:440
3038199118,2cyclictest17558-21sed23:40:260
3038199118,2cyclictest16532-21ssh00:19:200
3038199118,2cyclictest15452-21taskset20:50:280
3038199118,2cyclictest14581-21taskset22:40:410
3038199118,2cyclictest14190-21ssh00:36:140
3038199118,2cyclictest13503-21apache221:36:250
3038199118,2cyclictest13326-21ssh22:20:380
3038199118,2cyclictest12014-21grep20:45:180
3038199118,2cyclictest10861-21sed19:35:170
3038199117,2cyclictest18926-21grep21:00:150
3039199108,1cyclictest808-21ssh21:15:223
3039199108,1cyclictest31040-21ls19:10:153
3039199108,1cyclictest28360-21cat20:10:253
3039199108,1cyclictest11325-21cut20:45:153
3039199107,2cyclictest29185-21apache_accesses20:15:133
3039199107,2cyclictest10116-21cat20:40:263
3038899108,1cyclictest501-21sendmail_mailtr20:20:222
3038899108,1cyclictest229802sleep221:07:402
3038899108,1cyclictest11362-21cut20:45:152
3038899108,1cyclictest10930-21if_enp0s31f619:35:182
3038899107,2cyclictest3197-21grep20:25:252
3038899107,2cyclictest29017-21date20:15:012
3038899107,2cyclictest14601-21chrt19:44:302
3038899105,2cyclictest225-21systemd-journal19:51:232
3038699108,1cyclictest18241-21chrt20:58:021
3038699108,1cyclictest1499-21uname19:15:181
3038699108,1cyclictest12690-21grep19:40:131
3038699108,1cyclictest10003-21chrt20:40:221
3038699107,2cyclictest2952-21cat19:20:111
3038199108,2cyclictest18424-21ntp_kernel_pll_19:50:190
3038199108,1cyclictest10137-21ls20:40:250
3038199107,2cyclictest8211-21taskset20:37:410
3038199107,2cyclictest31105-21df19:10:150
3038199100,9cyclictest24739-21cat20:05:140
303889992,4cyclictest225-21systemd-journal20:31:322
303869997,1cyclictest15003-21hostname19:45:151
303869996,2cyclictest23901-21taskset20:01:431
303869996,2cyclictest23901-21taskset20:01:421
303819997,1cyclictest4821-21cat20:30:160
303819997,1cyclictest12456-21cat19:40:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional