You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 12:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Mon Mar 30, 2026 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2356399270,26cyclictest16788-21diskmemload22:45:012
23550992724,2cyclictest14616-21cron21:40:010
2356399260,25cyclictest16788-21diskmemload23:53:482
2356399260,25cyclictest16788-21diskmemload23:06:182
2356399260,25cyclictest0-21swapper/200:15:022
2356399260,25cyclictest0-21swapper/200:13:082
2356399260,23cyclictest0-21swapper/221:59:592
2355899260,25cyclictest29677-21ssh00:33:421
2356399250,4cyclictest6275-21ssh21:32:302
2356399250,25cyclictest0-21swapper/200:29:422
2356399250,24cyclictest606-21dbus-daemon19:20:012
2356399250,24cyclictest32411-21ssh23:01:542
2356399250,24cyclictest0-21swapper/222:41:422
2356399250,24cyclictest0-21swapper/222:14:122
2356399250,24cyclictest0-21swapper/222:04:562
2356399250,24cyclictest0-21swapper/221:14:112
2356399250,24cyclictest0-21swapper/219:23:152
23558992523,1cyclictest0-21swapper/122:20:131
23558992523,1cyclictest0-21swapper/122:20:131
2355899250,24cyclictest606-21dbus-daemon19:30:001
2355899250,24cyclictest0-21swapper/121:54:021
2355099250,24cyclictest16788-21diskmemload22:38:510
2355099250,24cyclictest16788-21diskmemload22:20:360
2355099250,24cyclictest16788-21diskmemload22:20:360
23564992421,2cyclictest19203-21diskstats23:53:183
2356499241,22cyclictest0-21swapper/323:32:363
2356499240,3cyclictest10058-21apt-get20:58:183
2356499240,23cyclictest16788-21diskmemload00:32:533
2356499240,23cyclictest0-21swapper/323:03:183
2356499240,23cyclictest0-21swapper/322:47:143
2356499240,23cyclictest0-21swapper/322:12:233
23563992422,2cyclictest26096-21ssh00:30:502
23563992422,2cyclictest0-21swapper/221:16:122
23563992421,3cyclictest28-21ksoftirqd/222:36:392
23563992420,3cyclictest0-21swapper/200:21:542
2356399240,3cyclictest0-21swapper/221:24:532
2356399240,24cyclictest0-21swapper/223:25:242
2356399240,24cyclictest0-21swapper/222:58:262
2356399240,24cyclictest0-21swapper/200:05:552
2356399240,24cyclictest0-21swapper/200:00:342
2356399240,23cyclictest0-21swapper/223:36:192
2356399240,23cyclictest0-21swapper/223:23:012
2356399240,23cyclictest0-21swapper/223:20:002
2356399240,23cyclictest0-21swapper/222:52:102
2356399240,23cyclictest0-21swapper/222:23:092
2356399240,23cyclictest0-21swapper/222:23:092
2356399240,23cyclictest0-21swapper/222:16:592
2356399240,23cyclictest0-21swapper/222:07:282
23558992424,0cyclictest0-21swapper/121:36:181
23558992422,1cyclictest0-21swapper/122:08:491
23558992422,1cyclictest0-21swapper/100:18:431
2355899240,24cyclictest0-21swapper/123:36:041
2355899240,23cyclictest31012-21ls23:00:001
2355899240,23cyclictest16788-21diskmemload22:19:001
2355899240,23cyclictest0-21swapper/123:14:291
2355899240,23cyclictest0-21swapper/100:35:231
2355899240,23cyclictest0-21swapper/100:13:301
2355099240,24cyclictest16788-21diskmemload21:22:100
2355099240,24cyclictest16788-21diskmemload00:01:010
2355099240,24cyclictest0-21swapper/023:08:000
2355099240,23cyclictest19154-21memory22:48:200
23564992323,0cyclictest16788-21diskmemload22:17:583
23564992323,0cyclictest0-21swapper/322:30:463
23564992323,0cyclictest0-21swapper/322:05:553
23564992323,0cyclictest0-21swapper/321:21:333
23564992321,1cyclictest4982-21apt-get22:03:173
23564992320,2cyclictest4785-21rm21:30:053
23564992320,2cyclictest358-21sh00:06:283
2356499231,2cyclictest5516-21munin-run19:40:003
2356499230,23cyclictest16788-21diskmemload21:55:553
2356499230,23cyclictest0-21swapper/321:43:453
2356499230,23cyclictest0-21swapper/300:23:553
2356499230,22cyclictest29146-21ssh22:26:463
2356499230,22cyclictest26786-21sh22:55:473
2356499230,22cyclictest16788-21diskmemload23:45:363
2356499230,22cyclictest16788-21diskmemload23:15:333
2356499230,0cyclictest151rcu_preempt21:28:233
2356499230,0cyclictest0-21swapper/323:43:483
2356499230,0cyclictest0-21swapper/321:45:553
2356499230,0cyclictest0-21swapper/321:37:523
23563992323,0cyclictest0-21swapper/221:46:282
23563992320,3cyclictest0-21swapper/223:58:512
23563992320,2cyclictest10103-21sh23:12:012
2356399230,23cyclictest16788-21diskmemload21:35:342
2356399230,23cyclictest0-21swapper/222:25:382
2356399230,23cyclictest0-21swapper/221:52:382
2356399230,23cyclictest0-21swapper/221:26:102
2356399230,23cyclictest0-21swapper/200:36:152
2356399230,22cyclictest29912-21ssh23:31:262
2356399230,22cyclictest16788-21diskmemload23:47:342
2356399230,22cyclictest16788-21diskmemload21:41:192
2356399230,0cyclictest0-21swapper/222:30:332
23558992323,0cyclictest16788-21diskmemload22:02:511
23558992321,1cyclictest9549-21latency_hist20:55:011
23558992321,1cyclictest32353-21turbostat20:35:001
23558992321,1cyclictest12473-21apt-get21:03:171
23558992320,2cyclictest6463-21ssh23:08:201
2355899230,2cyclictest3733-21rm21:28:421
2355899230,23cyclictest0-21swapper/123:20:041
2355899230,23cyclictest0-21swapper/122:31:201
2355899230,23cyclictest0-21swapper/122:25:361
2355899230,23cyclictest0-21swapper/121:31:031
2355899230,23cyclictest0-21swapper/121:11:081
2355899230,22cyclictest7862-21ssh23:41:591
2355899230,22cyclictest3452-21ssh23:04:421
2355899230,22cyclictest30723-21basename19:25:001
2355899230,22cyclictest0-21swapper/121:48:521
2355899230,1cyclictest24284-21fschecks_count23:58:201
2355899230,1cyclictest0-21swapper/123:30:321
2355899230,1cyclictest0-21swapper/122:53:151
2355899230,0cyclictest0-21swapper/122:36:101
23550992323,0cyclictest16788-21diskmemload23:36:070
23550992323,0cyclictest16788-21diskmemload21:47:070
23550992323,0cyclictest0-21swapper/022:13:150
23550992323,0cyclictest0-21swapper/021:41:340
23550992323,0cyclictest0-21swapper/000:36:000
23550992322,1cyclictest0-21swapper/022:33:110
23550992321,1cyclictest5582-21ssh21:31:250
23550992321,1cyclictest10026-21apt-get00:15:020
23550992321,1cyclictest0-21swapper/023:58:020
23550992320,2cyclictest7345-21sh23:41:030
23550992319,3cyclictest28299-21dump-pmu-power19:20:010
2355099230,2cyclictest17420-21rm21:10:440
2355099230,23cyclictest16788-21diskmemload00:06:330
2355099230,22cyclictest32018-21ssh23:01:090
2355099230,22cyclictest25945-21ssh00:30:360
2355099230,22cyclictest18968-21sh22:16:300
2355099230,22cyclictest18579-21rm23:53:030
2355099230,22cyclictest16788-21diskmemload00:12:570
2355099230,22cyclictest0-21swapper/023:45:070
2355099230,22cyclictest0-21swapper/021:51:030
2355099230,1cyclictest8543-21sh22:05:520
2355099230,1cyclictest25184-21awk20:20:000
2355099230,1cyclictest16788-21diskmemload22:50:410
2355099230,0cyclictest16788-21diskmemload23:19:230
2355099230,0cyclictest0-21swapper/022:44:320
2355099230,0cyclictest0-21swapper/021:26:290
23564992222,0cyclictest16788-21diskmemload22:54:023
23564992222,0cyclictest0-21swapper/300:26:563
23564992222,0cyclictest0-21swapper/300:11:163
23564992221,1cyclictest0-21swapper/323:06:223
23564992220,1cyclictest24041-21ssh23:25:043
23564992220,1cyclictest13085-21turbostat19:55:013
23564992220,1cyclictest10658-21munin-run22:40:013
23564992220,1cyclictest0-21swapper/320:05:013
23564992219,2cyclictest24018-21/usr/sbin/munin21:18:253
2356499220,2cyclictest32225-21rm00:37:283
2356499220,21cyclictest2871-21apt-get20:43:173
2356499220,21cyclictest22483-21ssh23:56:133
2356499220,21cyclictest20856-21apt-get20:13:173
2356499220,21cyclictest19365-21ssh23:20:343
2356499220,21cyclictest16788-21diskmemload23:11:013
2356499220,21cyclictest16788-21diskmemload22:20:123
2356499220,21cyclictest16788-21diskmemload22:20:123
2356499220,21cyclictest16788-21diskmemload21:51:113
2356499220,21cyclictest0-21swapper/300:00:143
2356499220,1cyclictest32374-21munin-run20:35:003
2356499220,1cyclictest16788-21diskmemload23:39:273
2356499220,1cyclictest10416-21ssh00:15:383
2356499220,1cyclictest0-21swapper/322:43:223
2356499220,0cyclictest0-21swapper/321:13:033
23563992221,1cyclictest0-21swapper/220:53:202
23563992220,2cyclictest0-21swapper/223:40:432
2356399220,21cyclictest16786-21cron21:10:002
2356399220,1cyclictest2348-21perf20:40:012
2356399220,1cyclictest17970-21munin-run20:05:012
2356399220,0cyclictest0-21swapper/219:33:192
23558992222,0cyclictest16788-21diskmemload00:05:331
23558992222,0cyclictest0-21swapper/121:55:031
23558992220,1cyclictest23619-21ssh21:17:341
2355899220,22cyclictest0-21swapper/121:23:281
2355899220,21cyclictest15569-21sendmail-msp20:00:011
2355899220,1cyclictest24046-21ssh23:25:051
2355899220,1cyclictest18423-21sh23:52:491
2355899220,1cyclictest17379-21sendmail_mailqu23:18:251
2355899220,1cyclictest15881-21sh21:41:351
2355899220,1cyclictest0-21swapper/122:43:151
2355899220,0cyclictest0-21swapper/122:46:461
2355899220,0cyclictest0-21swapper/100:00:261
23550992222,0cyclictest16788-21diskmemload23:25:420
23550992221,1cyclictest19224-21ssh23:20:210
23550992220,1cyclictest22185-21sh21:15:130
23550992220,1cyclictest0-21swapper/020:50:010
2355099220,21cyclictest15531-21cron20:00:010
2355099220,21cyclictest15190-21sh00:20:120
2355099220,1cyclictest9229-21ssh23:10:320
2355099220,0cyclictest0-21swapper/022:57:180
2356499210,20cyclictest0-21swapper/320:23:183
23563992120,1cyclictest11206-21apt-get19:53:172
23563992116,4cyclictest25156-21awk20:20:002
2356399210,20cyclictest0-21swapper/220:25:002
23558992120,0cyclictest23252-21apt-get20:18:161
23558992119,1cyclictest25885-21awk19:15:011
2355899210,21cyclictest0-21swapper/120:03:181
2355899210,20cyclictest0-21swapper/120:58:181
2355899210,20cyclictest0-21swapper/119:48:151
23550992121,0cyclictest28070-21apt-get20:28:170
23550992121,0cyclictest0-21swapper/022:00:370
23550992121,0cyclictest0-21swapper/000:25:210
23550992120,1cyclictest0-21swapper/023:30:190
23550992119,1cyclictest11973-21munin-run21:00:010
2355099210,21cyclictest0-21swapper/020:55:010
2355099210,21cyclictest0-21swapper/020:13:170
2355099210,20cyclictest5500-21apt-get19:40:020
2355099210,1cyclictest28306-21ssh22:25:210
23564992020,0cyclictest24009-21apt-get19:13:163
2356499200,20cyclictest0-21swapper/320:38:163
2356499200,0cyclictest0-21swapper/319:23:153
23563992020,0cyclictest0-21swapper/220:13:172
2356399200,19cyclictest0-21swapper/221:03:172
2356399200,19cyclictest0-21swapper/220:43:172
2356399200,19cyclictest0-21swapper/220:33:162
23558992020,0cyclictest0-21swapper/120:48:171
2355899200,0cyclictest0-21swapper/120:13:171
23550992020,0cyclictest18463-21apt-get20:08:180
2355099200,0cyclictest0-21swapper/020:43:170
2355099200,0cyclictest0-21swapper/020:33:160
23564991919,0cyclictest0-21swapper/321:08:193
23564991919,0cyclictest0-21swapper/320:00:013
23563991919,0cyclictest0-21swapper/219:28:172
2356399191,15cyclictest0-21swapper/219:48:222
2356399190,18cyclictest3581-21find19:38:172
2356399190,18cyclictest10541-21grep20:58:192
23558991919,0cyclictest0-21swapper/120:43:171
23558991917,1cyclictest5500-21apt-get19:40:001
23550991919,0cyclictest401-21apt-get20:38:160
23550991919,0cyclictest0-21swapper/019:33:190
23564991818,0cyclictest0-21swapper/320:48:173
23564991818,0cyclictest0-21swapper/320:28:173
23564991818,0cyclictest0-21swapper/319:43:163
23564991818,0cyclictest0-21swapper/319:33:183
23564991816,1cyclictest637-21cron19:30:003
2356399180,17cyclictest15513-21cron20:00:012
23558991818,0cyclictest0-21swapper/121:08:191
23550991818,0cyclictest0-21swapper/019:48:150
23564991717,0cyclictest0-21swapper/320:53:203
23563991717,0cyclictest0-21swapper/219:13:162
23550991717,0cyclictest0-21swapper/021:03:170
2355099170,16cyclictest25951-21latency_hist19:15:010
23564991616,0cyclictest0-21swapper/320:08:183
2356499160,15cyclictest783-21snmpd19:48:263
23563991616,0cyclictest0-21swapper/220:08:182
2356399160,16cyclictest0-21swapper/220:47:072
2356399160,16cyclictest0-21swapper/219:43:172
2356399160,15cyclictest0-21swapper/220:28:182
23558991614,1cyclictest0-21swapper/119:45:001
23550991616,0cyclictest0-21swapper/020:03:180
2355099160,15cyclictest29433-21sed19:23:170
2355899155,10cyclictest605-21avahi-daemon19:16:201
23558991515,0cyclictest0-21swapper/120:23:181
23558991515,0cyclictest0-21swapper/119:53:171
2355099150,1cyclictest25629-21chrt20:23:030
23564991414,0cyclictest0-21swapper/319:15:083
2356499140,14cyclictest0-21swapper/320:15:053
2356499140,0cyclictest0-21swapper/321:00:293
23558991412,1cyclictest28490-21df_inode20:28:181
2355899140,14cyclictest0-21swapper/120:35:081
23550991414,0cyclictest0-21swapper/019:25:310
2355099140,14cyclictest0-21swapper/019:35:260
2355099140,0cyclictest0-21swapper/021:05:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional