You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-11 - 08:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Apr 11, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2587029072,13sleep30-21swapper/319:05:203
2686828977,8sleep00-21swapper/019:08:470
2676628270,8sleep20-21swapper/219:07:282
2686927659,13sleep10-21swapper/119:08:481
149272490,0sleep00-21swapper/023:20:260
2705599270,24cyclictest0-21swapper/221:26:432
27059992524,1cyclictest0-21swapper/320:01:173
27059992421,2cyclictest5675-21apache_accesses23:35:113
27059992416,8cyclictest17213-21kworker/3:022:48:023
2705399244,19cyclictest4134-21perf19:30:011
27059992219,2cyclictest22631-21ntp_states23:25:213
27053992219,2cyclictest29730-21apache_processe23:30:121
27059992119,2cyclictest17213-21kworker/3:023:41:113
27059992118,2cyclictest30174-21apache_volume21:35:123
27059992118,2cyclictest20532-21apache_processe21:10:113
27059992118,2cyclictest10211-21apache_processe22:40:123
27059992118,2cyclictest10211-21apache_processe22:40:113
27059992116,1cyclictest9028-21ssh22:00:203
2705599210,21cyclictest0-21swapper/222:58:292
27053992118,2cyclictest1299-21ntp_states00:10:221
2705399210,1cyclictest0-21swapper/121:52:201
27049992120,1cyclictest0-21swapper/022:41:280
27049992120,1cyclictest0-21swapper/022:41:280
27059992018,2cyclictest26550-21kworker/3:200:09:433
27059992018,2cyclictest17213-21kworker/3:023:51:223
27059992017,3cyclictest17213-21kworker/3:000:01:123
27059992017,2cyclictest7409-21df23:55:143
27053992017,2cyclictest8177-21ntp_states23:55:201
27053992017,2cyclictest6164-21apache_accesses21:40:121
27053992017,2cyclictest3854-21hddtemp_smartct22:55:151
2705399200,20cyclictest0-21swapper/100:28:291
2705399200,20cyclictest0-21swapper/100:28:281
2705399200,19cyclictest0-21swapper/123:41:471
2704999200,19cyclictest0-21swapper/023:58:500
276742190,0sleep00-21swapper/021:33:280
27059991917,2cyclictest39-21ksoftirqd/321:55:353
27059991917,2cyclictest17213-21kworker/3:022:17:423
27059991916,2cyclictest23196-21df_inode23:45:153
2705999190,16cyclictest6611-21sensors21:20:253
27055991917,2cyclictest0-21swapper/221:16:032
2705599190,2cyclictest0-21swapper/220:20:132
2705399195,2cyclictest141rcu_preempt21:19:021
2705399193,13cyclictest27016-21apt-get22:50:121
2705399190,1cyclictest0-21swapper/122:06:401
2705399190,19cyclictest0-21swapper/121:47:581
2704999195,8cyclictest23805-21apt-get00:05:150
118372190,1sleep20-21swapper/223:38:362
316532180,0sleep1241rcuc/121:55:111
27059991816,2cyclictest26550-21kworker/3:200:31:423
27059991816,2cyclictest17213-21kworker/3:021:30:413
27059991815,2cyclictest26750-21hddtemp_smartct22:30:173
2705599180,18cyclictest0-21swapper/222:41:282
2705599180,18cyclictest0-21swapper/222:41:272
2705599180,17cyclictest0-21swapper/200:28:012
2705599180,17cyclictest0-21swapper/200:28:002
2705399180,17cyclictest0-21swapper/120:20:131
2704999180,4cyclictest141rcu_preempt22:07:580
195612180,1chrt19564-21munin-run23:05:011
27059991714,2cyclictest9638-21ntp_states00:15:213
2705599173,6cyclictest0-21swapper/223:02:042
2705599170,6cyclictest0-21swapper/222:29:192
2705599170,17cyclictest0-21swapper/223:30:202
2705599170,17cyclictest0-21swapper/223:15:272
2705599170,0cyclictest0-21swapper/223:58:492
2705399170,2cyclictest0-21swapper/123:09:101
2705399170,2cyclictest0-21swapper/100:32:281
2704999170,2cyclictest0-21swapper/000:34:070
2704999170,1cyclictest0-21swapper/023:50:010
2704999170,0cyclictest0-21swapper/021:16:030
27055991615,1cyclictest0-21swapper/221:50:452
27055991614,2cyclictest32-21ksoftirqd/221:41:062
27055991614,1cyclictest0-21swapper/223:05:232
2705599160,1cyclictest0-21swapper/200:37:192
2705399160,16cyclictest0-21swapper/121:20:391
2704999165,4cyclictest21785-21taskset23:44:490
2704999162,7cyclictest31164-21munin-run21:54:590
27049991614,1cyclictest20059-21diskmemload22:25:140
38672150,3sleep10-21swapper/121:38:371
27059991512,2cyclictest17420-21memory22:05:183
27059991511,3cyclictest25461-21head22:10:153
2705999150,0cyclictest0-21swapper/322:25:543
2705599153,6cyclictest21323-21apt-get23:25:132
2705599152,12cyclictest15853-21/usr/sbin/munin21:45:182
27055991514,1cyclictest0-21swapper/222:35:162
27053991514,1cyclictest0-21swapper/122:39:251
27053991513,1cyclictest21175-21df_inode21:10:141
2705399150,0cyclictest0-21swapper/121:29:391
2704999156,8cyclictest0-21swapper/021:23:400
27049991514,1cyclictest0-21swapper/019:40:140
2704999150,3cyclictest0-21swapper/023:11:390
2704999150,2cyclictest0-21swapper/023:01:500
2705999142,1cyclictest0-21swapper/323:05:113
27059991411,2cyclictest15065-21cron21:00:003
2705999140,12cyclictest21399-21basename20:05:133
27055991411,1cyclictest0-21swapper/222:08:162
2705599140,12cyclictest0-21swapper/223:44:232
2705399144,5cyclictest225-21systemd-journal22:50:001
27053991411,3cyclictest0-21swapper/100:03:561
27053991411,2cyclictest26144-21cut22:30:151
27053991410,3cyclictest6429-21grep23:35:151
2704999149,1cyclictest13-21ksoftirqd/022:57:260
27049991412,2cyclictest13-21ksoftirqd/020:05:000
2704999140,12cyclictest0-21swapper/022:21:550
2705999139,3cyclictest16217-21wc19:55:003
2705999139,2cyclictest7886-21cron20:45:003
27059991310,2cyclictest5697-21grep23:15:193
27059991310,2cyclictest4174-21head20:35:163
27059991310,2cyclictest16987-21munin-plugin-st22:25:003
27059991310,2cyclictest16797-21grep21:45:253
2705999130,12cyclictest27518-21grep20:15:203
2705999130,11cyclictest32024-21/usr/sbin/munin19:20:213
2705599138,2cyclictest21897-21open_files21:10:212
2705599136,5cyclictest25538-21which22:10:152
2705599135,2cyclictest7914-21basename22:00:152
2705599133,1cyclictest7145-21munin-run21:59:592
27055991312,1cyclictest0-21swapper/222:45:412
27055991310,2cyclictest19613-21sed21:05:252
2705599130,13cyclictest0-21swapper/221:22:352
2705599130,12cyclictest55650irq/127-enp2s0-00:10:092
2705599130,10cyclictest0-21swapper/200:31:412
27053991312,1cyclictest0-21swapper/120:15:201
2705399131,12cyclictest20059-21diskmemload00:16:081
27053991310,2cyclictest723-21grep19:20:181
2705399130,1cyclictest0-21swapper/123:10:241
2705399130,12cyclictest0-21swapper/123:52:271
2704999132,9cyclictest8978-21df_inode00:15:150
27049991311,2cyclictest0-21swapper/022:17:440
27049991310,2cyclictest9666-21wc20:45:240
2704999130,3cyclictest537-21snmpd22:12:160
2704999130,3cyclictest0-21swapper/020:30:140
2705999129,2cyclictest706-21awk00:30:013
2705999129,2cyclictest706-21awk00:30:013
2705999129,2cyclictest3775-21grep22:55:153
2705999129,2cyclictest30337-21sort19:15:173
2705999129,2cyclictest27844-21sed22:50:153
2705999129,2cyclictest26899-21seq21:51:433
2705999129,2cyclictest18442-21sh19:59:593
2705999129,2cyclictest17312-21chrt23:22:173
2705599129,2cyclictest6102-21expr20:40:152
2705599129,2cyclictest5998-21grep19:30:232
2705599129,2cyclictest5658-21munin-run21:39:592
2705599129,2cyclictest3801-21rm22:16:202
2705599129,2cyclictest313-21sh22:34:162
2705599129,2cyclictest3028-21sh23:52:242
2705599129,2cyclictest22483-21ssh23:45:102
2705599121,9cyclictest0-21swapper/222:50:122
27055991210,2cyclictest32-21ksoftirqd/220:20:012
27055991210,1cyclictest23805-21apt-get00:05:152
27055991210,1cyclictest18488-21sendmail-msp19:59:592
2705599120,7cyclictest0-21swapper/200:20:572
2705599120,11cyclictest24580-21grep20:10:162
2705599120,11cyclictest11756-21grep19:45:142
2705599120,11cyclictest0-21swapper/219:40:142
2705599120,10cyclictest0-21swapper/223:20:292
2705599120,10cyclictest0-21swapper/222:24:162
2705399129,2cyclictest9587-21ssh22:00:261
2705399129,2cyclictest9364-21cut20:45:201
2705399129,2cyclictest912-21tr22:15:161
2705399129,2cyclictest3641-21tr20:35:131
2705399129,2cyclictest24638-21grep20:10:161
2705399129,2cyclictest23062-21cat21:30:221
2705399128,3cyclictest22087-21ssh23:25:171
2705399128,3cyclictest18310-21grep22:25:181
2705399128,1cyclictest27389-21ssh23:47:371
27053991210,1cyclictest11283-21grep20:50:161
2704999129,2cyclictest6161-21expr20:40:150
2704999129,2cyclictest5501-21sh20:40:000
2704999129,2cyclictest417-21grep21:55:200
2704999129,2cyclictest31603-21ssh00:28:510
2704999129,2cyclictest31603-21ssh00:28:500
2704999129,2cyclictest30695-21chrt22:32:380
2704999129,2cyclictest28621-21sensors22:50:240
2704999129,2cyclictest19890-21wc22:45:200
2704999129,2cyclictest10813-21ssh23:37:490
27049991211,1cyclictest0-21swapper/021:44:410
2704999120,7cyclictest0-21swapper/000:39:050
2704999120,6cyclictest0-21swapper/023:46:370
2704999120,6cyclictest0-21swapper/023:07:040
2704999120,3cyclictest0-21swapper/023:30:180
2704999120,10cyclictest0-21swapper/023:17:280
2705999119,1cyclictest25065-21ntp_states20:10:213
2705999118,2cyclictest730-21ssh23:31:333
2705999118,2cyclictest6741-21sed21:40:153
2705999118,2cyclictest3236-21ssh00:11:253
2705999118,2cyclictest32092-21ssh23:11:423
2705999118,2cyclictest28893-21cat20:20:143
2705999118,2cyclictest28218-21cut19:10:193
2705999118,2cyclictest2601-21ssh21:18:343
2705999118,2cyclictest2364-21ls19:25:153
2705999118,2cyclictest2125-21tr22:35:133
2705999118,2cyclictest20826-21ssh00:22:013
2705999118,2cyclictest17906-21taskset21:05:013
2705999118,2cyclictest1739-21grep20:30:163
2705999118,2cyclictest14416-21if_err_enp0s31f21:25:193
2705999118,2cyclictest13616-21taskset19:49:093
2705999118,2cyclictest13081-21grep23:00:223
2705999118,2cyclictest11536-21sed00:35:263
2705999118,2cyclictest10113-21taskset20:48:113
2705599119,2cyclictest31262-21unixbench-2d19:15:272
2705599119,2cyclictest18557-21sh00:01:222
2705599119,2cyclictest16026-21taskset19:54:372
2705599119,2cyclictest14990-21taskset20:59:052
2705599119,1cyclictest8134-21ssh00:15:032
2705599119,1cyclictest7210-21expr19:35:162
2705599119,1cyclictest4124-21dump-pmu-power19:30:012
2705599119,1cyclictest3788-21expr20:35:142
2705599119,1cyclictest30854-21rm23:10:442
2705599119,1cyclictest21517-21ssh21:30:002
2705599119,1cyclictest18878-21egrep20:00:152
2705599119,1cyclictest10536-21seq20:50:122
2705399119,1cyclictest27349-21tr19:10:121
2705399118,2cyclictest6279-21sed23:15:261
2705399118,2cyclictest29800-21tr19:15:151
2705399118,2cyclictest19671-21ssh00:21:091
2705399118,2cyclictest15121-21ssh00:38:111
2705399118,2cyclictest14542-21ssh23:20:241
2705399118,2cyclictest13238-21sed19:45:261
2705399117,3cyclictest6081-21grep19:30:231
2705399117,3cyclictest32707-21sed20:25:261
2705399117,1cyclictest20059-21diskmemload22:40:021
2705399117,1cyclictest20059-21diskmemload22:40:011
2705399110,10cyclictest27829-21chrt22:11:181
2704999119,2cyclictest15276-21ssh00:00:090
2704999119,1cyclictest32274-21tr19:20:120
2704999119,1cyclictest30837-21forks21:35:150
2704999119,1cyclictest30725-21sed19:15:210
2704999119,1cyclictest272742sleep019:10:100
2704999119,1cyclictest2519-21cut22:35:150
2704999119,1cyclictest23506-21grep20:10:000
2704999119,1cyclictest23168-21ssh23:25:250
2704999119,1cyclictest17789-21grep00:20:180
2704999119,1cyclictest17542-21cut19:55:210
2704999119,1cyclictest15397-21grep21:00:130
2704999119,1cyclictest14603-21sh21:45:010
2704999119,1cyclictest11342-21ssh22:01:350
2704999118,2cyclictest6439-21chrt19:34:270
2704999118,2cyclictest31021-21chrt20:25:040
2704999118,2cyclictest2070-21sh00:10:310
2704999118,2cyclictest20368-21ssh21:10:040
2704999118,2cyclictest18143-21wc21:05:150
2704999118,2cyclictest14990-21wc19:50:200
2704999110,11cyclictest16005-21ssh21:25:520
2705999108,1cyclictest9365-21cpuspeed_turbos19:40:143
2705999108,1cyclictest11015-21df_abs20:50:143
2705999107,2cyclictest4788-21ls19:30:123
2705999107,2cyclictest31842-21expr20:25:173
2705999106,3cyclictest7508-21sed19:35:183
2705999103,4cyclictest16053-21cut21:00:173
2705599109,1cyclictest21206-21dpkg20:05:002
2705599108,2cyclictest9566-21switchtime20:45:242
2705599108,2cyclictest15761-21expr21:00:152
2705599108,2cyclictest1451-21expr20:30:142
2705599108,1cyclictest27513-21tr19:10:142
2705599107,2cyclictest1336-21sed19:20:242
2705599100,10cyclictest31407-21expr20:25:152
2705399108,1cyclictest8864-21ssh22:20:041
2705399108,1cyclictest8608-21chrt19:36:481
2705399108,1cyclictest19447-21cat21:05:251
2705399108,1cyclictest15785-21taskset19:51:321
2705399108,1cyclictest15674-21ls21:00:151
2705399107,2cyclictest21543-21tr20:05:141
2705399107,2cyclictest15089-21perf21:00:001
2705399107,2cyclictest10869-21grep19:40:271
2705399107,2cyclictest1061-21grep20:30:131
2705399101,3cyclictest0-21swapper/120:01:301
2705399101,1cyclictest0-21swapper/120:40:141
2705399100,9cyclictest24333-21sed00:05:171
2705399100,9cyclictest16474-21tr19:55:131
2704999109,1cyclictest13559-21chrt19:48:240
2704999109,1cyclictest11183-21grep20:50:150
2704999108,1cyclictest6821-21basename19:35:140
2704999108,1cyclictest29788-21grep20:20:190
2704999107,2cyclictest2789-21tail19:25:170
2704999107,2cyclictest25142-21head20:10:210
2704999107,2cyclictest12924-21taskset20:55:060
270499997,1cyclictest26247-21chrt20:15:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional