You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-25 - 06:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jul 25, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2581329077,9sleep00-21swapper/019:06:210
2590628465,15sleep20-21swapper/219:07:322
2603728059,6sleep10-21swapper/119:09:161
2592727966,9sleep30-21swapper/319:07:493
37532680,0sleep30-21swapper/321:57:463
280022550,0sleep30-21swapper/323:48:263
255352480,0sleep10-21swapper/122:30:141
192532290,2sleep119252-21munin-plugin-st23:05:001
2617999270,0cyclictest0-21swapper/000:05:520
2618699251,23cyclictest23798-21sh22:47:572
2618699240,6cyclictest0-21swapper/222:57:292
2618699240,24cyclictest0-21swapper/222:09:342
2618499240,1cyclictest0-21swapper/123:10:231
2617999240,0cyclictest0-21swapper/022:22:530
2618699230,22cyclictest0-21swapper/221:45:532
26179992321,1cyclictest0-21swapper/021:10:270
26186992120,1cyclictest0-21swapper/222:41:242
26184992118,2cyclictest10463-21grep20:50:171
2618699197,2cyclictest7856-21latency_hist00:15:002
2618699190,19cyclictest0-21swapper/221:36:402
2618699190,17cyclictest0-21swapper/221:24:022
26184991917,2cyclictest0-21swapper/121:58:531
2618499190,17cyclictest0-21swapper/122:57:361
2617999190,1cyclictest0-21swapper/022:52:520
2618999180,3cyclictest0-21swapper/322:55:143
2618999180,1cyclictest8602-21ssh22:20:153
2618499184,8cyclictest0-21swapper/100:39:491
26184991811,6cyclictest0-21swapper/100:05:151
26179991817,1cyclictest0-21swapper/022:11:370
2617999180,17cyclictest0-21swapper/023:19:190
2618999170,2cyclictest0-21swapper/300:31:363
26186991716,1cyclictest0-21swapper/223:48:132
2618699170,17cyclictest0-21swapper/223:59:432
2618699170,0cyclictest0-21swapper/221:56:532
26184991712,3cyclictest22631-21ls23:25:251
2618499170,17cyclictest0-21swapper/123:49:031
2617999172,9cyclictest0-21swapper/020:55:140
26179991715,1cyclictest0-21swapper/022:08:480
2618999160,0cyclictest0-21swapper/300:02:233
26186991615,1cyclictest0-21swapper/221:52:402
26186991615,1cyclictest0-21swapper/200:08:332
2618699160,0cyclictest0-21swapper/223:12:302
26184991615,1cyclictest0-21swapper/122:14:431
26184991614,1cyclictest0-21swapper/122:46:461
2617999160,1cyclictest0-21swapper/022:01:030
2618999150,15cyclictest0-21swapper/322:19:363
26186991515,0cyclictest0-21swapper/223:33:092
26186991515,0cyclictest0-21swapper/222:03:152
26186991514,1cyclictest0-21swapper/222:29:032
2618699150,15cyclictest0-21swapper/222:23:292
2618699150,15cyclictest0-21swapper/200:30:432
2618699150,12cyclictest0-21swapper/200:04:522
2618499158,2cyclictest225-21systemd-journal00:01:031
26184991513,1cyclictest19233-21diskmemload22:06:061
2618499151,1cyclictest19233-21diskmemload00:21:311
2618499151,1cyclictest0-21swapper/122:29:431
2618499150,14cyclictest0-21swapper/123:40:121
2617999150,0cyclictest0-21swapper/022:19:010
2618999148,6cyclictest29792-21apache222:37:123
26189991410,3cyclictest31971-21sed20:25:243
2618999140,2cyclictest0-21swapper/321:24:093
26186991413,1cyclictest0-21swapper/223:37:332
26186991413,1cyclictest0-21swapper/222:13:492
26186991411,2cyclictest13555-21apache_processe19:50:112
2618499148,3cyclictest0-21swapper/122:41:531
2618499142,11cyclictest31416-21ssh23:50:161
2618499141,3cyclictest141rcu_preempt21:17:341
2618499141,2cyclictest0-21swapper/119:40:111
26184991410,2cyclictest14976-21tr21:00:161
2618499140,2cyclictest14109-21sed19:50:151
2618499140,13cyclictest27189-21cut19:10:171
2618499140,12cyclictest13794-21ssh21:45:011
2617999144,6cyclictest16301-21sendmail00:20:000
26179991411,2cyclictest30551-21dpkg21:55:010
2618999139,3cyclictest24763-21sed20:10:263
2618999139,3cyclictest15772-21sed23:40:273
2618999132,9cyclictest0-21swapper/322:40:003
26189991310,2cyclictest31237-21grep23:50:153
26189991310,2cyclictest30298-21grep23:30:183
26189991310,2cyclictest2200-21sed19:25:183
26189991310,2cyclictest17629-21cat00:40:003
26189991310,2cyclictest15045-21chrt00:19:033
2618699131,2cyclictest0-21swapper/221:40:592
26186991312,1cyclictest0-21swapper/223:01:432
2618699131,11cyclictest0-21swapper/200:15:122
26186991310,2cyclictest4838-21uniq23:15:162
26186991310,2cyclictest17127-21ls19:55:222
26186991310,2cyclictest13944-21grep21:25:202
2618699130,13cyclictest0-21swapper/200:20:592
2618699130,12cyclictest10778-21grep19:45:002
2618499139,3cyclictest30620-21cat23:30:211
2618499139,3cyclictest27562-21cron20:20:001
2618499131,12cyclictest12522-21kworker/u8:222:03:491
26184991310,2cyclictest9457-21sh22:40:001
26184991310,2cyclictest8487-21uname20:45:181
26184991310,2cyclictest7143-21wc23:35:231
26184991310,2cyclictest26078-21sed00:25:181
26184991310,2cyclictest24470-21grep21:50:261
26184991310,2cyclictest18656-21wc21:05:241
26184991310,2cyclictest12174-21wc23:20:001
2617999139,3cyclictest15488-21tr00:00:160
2617999132,7cyclictest15283-21sendmail_mailqu23:40:230
2617999132,1cyclictest141rcu_preempt21:33:520
26179991310,2cyclictest9041-21grep20:45:250
26179991310,2cyclictest751-21ssh00:10:180
26179991310,2cyclictest32406-21grep23:50:250
26179991310,2cyclictest3155-21grep00:30:260
26179991310,2cyclictest26106-21grep22:30:160
26179991310,2cyclictest22852-21sed23:25:250
26179991310,2cyclictest15660-21cron19:55:000
26179991310,2cyclictest12843-21grep21:25:140
2617999130,13cyclictest0-21swapper/023:21:150
2618999129,2cyclictest882-21perf22:35:003
2618999129,2cyclictest6440-21cat23:55:003
2618999129,2cyclictest410-21cat19:20:233
2618999129,2cyclictest26702-21tr19:10:143
2618999129,2cyclictest24793-21head22:10:173
2618999129,2cyclictest16531-21grep19:55:163
2618999129,2cyclictest11853-21cron20:55:003
2618999128,2cyclictest18216-21ls00:20:233
2618999124,4cyclictest225-21systemd-journal20:20:003
2618999120,3cyclictest0-21swapper/300:14:293
2618699129,2cyclictest7023-21ssh22:19:212
2618699129,2cyclictest6907-21grep19:35:172
2618699129,2cyclictest5790-21grep20:40:152
2618699129,2cyclictest3651-21sh21:20:002
2618699129,2cyclictest28881-21grep22:50:262
2618699129,2cyclictest27980-21cat19:10:262
2618699129,2cyclictest26498-21ssh23:28:112
2618699129,2cyclictest15173-21head23:40:222
2618699129,2cyclictest1339-21grep20:30:192
26186991210,1cyclictest26183-21grep22:30:172
2618699120,1cyclictest0-21swapper/222:39:242
2618699120,12cyclictest0-21swapper/221:33:032
2618699120,11cyclictest13448-21head23:20:162
2618499129,2cyclictest9098-21latency_hist00:35:001
2618499129,2cyclictest7419-21wc19:35:221
2618499129,2cyclictest5719-21grep20:40:151
2618499129,2cyclictest5449-21sh21:20:181
2618499129,2cyclictest29869-21cut21:35:131
2618499129,2cyclictest24334-21wc20:10:211
2618499129,2cyclictest22561-21tr21:30:231
2618499129,2cyclictest13355-21sort23:20:161
2618499129,1cyclictest19233-21diskmemload21:48:531
2618499121,2cyclictest19233-21diskmemload00:17:491
26184991210,1cyclictest32681-21grep19:20:201
26184991210,1cyclictest11220-21cut19:45:141
2618499120,6cyclictest0-21swapper/122:54:101
2618499120,4cyclictest0-21swapper/122:20:131
2617999129,2cyclictest6106-21grep20:40:180
2617999129,2cyclictest4497-21wc22:55:230
2617999129,2cyclictest4161-21ls20:35:270
2617999129,2cyclictest31049-21cut20:25:150
2617999129,2cyclictest30286-21cut23:30:180
2617999129,2cyclictest26635-21grep19:10:140
2617999129,2cyclictest2187-21cut19:25:180
2617999129,2cyclictest20609-21cut23:05:170
2617999129,2cyclictest20609-21cut23:05:170
2617999129,2cyclictest17392-21tr21:05:130
2617999129,2cyclictest17151-21sed19:55:220
2617999129,2cyclictest16047-21sed21:45:270
2617999129,2cyclictest14872-21cat00:00:000
2617999129,2cyclictest12797-21sed19:45:260
2617999129,2cyclictest11952-21wc22:40:260
2617999129,2cyclictest11872-21sleep20:55:000
2617999128,3cyclictest13467-21wc23:40:010
2617999123,8cyclictest24682-21munin-run00:25:000
26179991210,2cyclictest13-21ksoftirqd/022:45:340
2617999120,4cyclictest0-21swapper/023:00:130
2617999120,11cyclictest7780-21sed19:35:260
2617999120,11cyclictest314292sleep021:55:150
2618999118,2cyclictest9893-21ssh22:01:013
2618999118,2cyclictest8884-21ssh23:17:323
2618999118,2cyclictest8621-21tr19:40:123
2618999118,2cyclictest8166-21head20:45:153
2618999118,2cyclictest789-21ssh23:12:473
2618999118,2cyclictest7374-21ls19:35:223
2618999118,2cyclictest5955-21ssh23:35:133
2618999118,2cyclictest5577-21taskset19:31:043
2618999118,2cyclictest5577-21taskset19:31:043
2618999118,2cyclictest3757-21ntpq20:35:213
2618999118,2cyclictest28193-21expr20:20:143
2618999118,2cyclictest27585-21rm00:25:453
2618999118,2cyclictest26158-21taskset00:05:463
2618999118,2cyclictest21827-21grep23:25:163
2618999118,2cyclictest21684-21head21:10:263
2618999118,2cyclictest21439-21tr21:30:163
2618999118,2cyclictest20111-21taskset22:26:283
2618999118,2cyclictest19764-21ssh22:07:103
2618999118,2cyclictest19575-21rm23:05:053
2618999118,2cyclictest19575-21rm23:05:053
2618999118,2cyclictest19232-21turbostat21:10:003
2618999118,2cyclictest18996-21cat22:45:153
2618999118,2cyclictest15974-21grep21:00:233
2618999118,2cyclictest14950-21ssh23:01:393
2618999118,2cyclictest1390-21sh21:37:013
2618999118,2cyclictest13592-21wget19:50:133
2618999118,2cyclictest13115-21tr21:25:153
2618999118,2cyclictest12581-21sh23:20:113
2618999118,2cyclictest12528-21basename19:45:223
2618999115,2cyclictest16027-21/usr/sbin/munin21:45:273
2618999114,4cyclictest225-21systemd-journal20:05:003
2618999112,6cyclictest3642-21cron21:20:003
2618699119,1cyclictest0-21swapper/220:10:132
2618699118,2cyclictest3811-21basename19:30:142
2618699118,2cyclictest3811-21basename19:30:142
2618699118,2cyclictest31512-21head20:25:192
2618699118,2cyclictest29869-21cut19:15:192
2618699118,2cyclictest27610-21cron21:15:002
2618699118,2cyclictest27105-21grep20:15:272
2618699118,2cyclictest25276-21rm00:25:152
2618699118,2cyclictest17714-21grep21:05:152
2618699118,2cyclictest10164-21expr00:35:172
2618699110,7cyclictest0-21swapper/220:55:142
2618499118,2cyclictest8133-21cut23:55:231
2618499118,2cyclictest25788-21sh21:13:361
2618499118,2cyclictest23738-21ssh23:06:581
2618499118,2cyclictest23738-21ssh23:06:581
2618499118,2cyclictest2185-21taskset20:33:001
2618499118,2cyclictest19803-21ssh21:29:221
2618499118,2cyclictest19002-21cut20:00:161
2618499116,5cyclictest38150irq/131-snd_hda22:15:131
2618499110,10cyclictest32541-21wc00:10:151
2617999118,2cyclictest30553-21ssh23:10:410
2617999118,2cyclictest28258-21awk21:15:130
2617999118,2cyclictest25274-21ssh23:46:190
2617999118,2cyclictest2090-21ssh22:35:130
2617999118,2cyclictest1960-21head20:30:260
2617999118,2cyclictest19041-21head20:00:170
2617999118,2cyclictest16516-21chrt21:00:530
2617999118,2cyclictest11936-21rm21:43:400
2617999115,5cyclictest29792-21apache221:38:220
2617999114,2cyclictest6238-21ssh21:20:250
2617999113,3cyclictest225-21systemd-journal22:27:160
26179991110,1cyclictest0-21swapper/000:35:100
2617999110,0cyclictest0-21swapper/000:27:210
2618999108,1cyclictest28999-21cat19:15:143
2618999108,1cyclictest26781-21ssh22:50:043
2618999108,1cyclictest1260-21cut20:30:193
2618999108,1cyclictest12356-21expr20:55:153
2618999107,2cyclictest6012-21tr21:40:133
2618999107,2cyclictest22831-21sleep20:10:003
2618999107,2cyclictest22354-21ssh21:50:053
2618999100,9cyclictest5022-21cut20:40:123
2618699108,1cyclictest1431-21hostname19:25:142
2618699107,2cyclictest3075-21expr20:35:152
2618699107,2cyclictest30328-21ssh23:50:022
2618699107,2cyclictest27856-21apache_volume20:20:122
2618699107,2cyclictest20024-21grep23:05:152
2618699107,2cyclictest20024-21grep23:05:152
2618699107,1cyclictest7482-21wget20:45:132
2618699106,2cyclictest31463-21apt-get19:20:112
2618699100,6cyclictest13004-21sshd19:46:282
2618499108,1cyclictest2968-21expr20:35:151
2618499108,1cyclictest225192sleep120:06:251
2618499108,1cyclictest12189-21cut20:55:151
2618499107,2cyclictest6002-21cat19:35:001
2618499107,2cyclictest6002-21cat19:35:001
2618499107,2cyclictest31919-21chrt20:25:241
2618499107,2cyclictest30893-21chrt19:19:171
2618499107,2cyclictest1376-21chrt19:25:061
2618499104,4cyclictest28147-21cpuspeed_turbos20:20:131
2617999108,1cyclictest8687-21wget19:40:120
2617999108,1cyclictest27924-21wget20:20:130
2617999108,1cyclictest25438-21bc20:15:120
2617999108,1cyclictest230302sleep020:10:000
2617999107,2cyclictest5639-21chrt19:31:500
2617999107,2cyclictest5639-21chrt19:31:500
2617999107,2cyclictest30817-21taskset19:18:180
2617999107,2cyclictest20703-21chrt20:05:110
2617999106,3cyclictest32207-21cut19:20:150
261869998,1cyclictest21038-21expr20:05:132
261869998,1cyclictest20258-21taskset20:02:502
261869998,1cyclictest14747-21dpkg21:00:152
261869997,1cyclictest11573-21taskset20:51:242
261849997,1cyclictest17764-21chrt19:56:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional