You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 02:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Apr 12, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3058829177,10sleep20-21swapper/219:05:462
3083329077,9sleep00-21swapper/019:08:540
3074128161,16sleep30-21swapper/319:07:443
2878327958,6sleep10-21swapper/119:05:051
211192620,0sleep30-21swapper/323:42:083
244442520,0sleep10-21swapper/121:29:461
3100299270,5cyclictest23524-21diskmemload22:15:200
3100299250,25cyclictest0-21swapper/023:02:480
3100299244,4cyclictest23524-21diskmemload23:59:250
3100499230,23cyclictest0-21swapper/121:58:351
3100299232,9cyclictest11007-21ssh23:16:530
31008992221,1cyclictest0-21swapper/200:11:122
3100899220,19cyclictest0-21swapper/223:37:022
3100499220,0cyclictest0-21swapper/121:14:211
3100299224,2cyclictest0-21swapper/022:09:270
3100299224,17cyclictest13048-21/usr/sbin/munin22:40:110
3100299223,4cyclictest23524-21diskmemload00:00:270
31002992222,0cyclictest0-21swapper/000:08:220
3100899210,16cyclictest0-21swapper/221:15:152
31008992014,6cyclictest32-21ksoftirqd/222:58:412
3100899200,0cyclictest0-21swapper/200:19:032
3100499200,20cyclictest0-21swapper/100:19:371
3100499200,18cyclictest0-21swapper/100:22:261
3100299201,7cyclictest13-21ksoftirqd/000:22:160
31002992014,6cyclictest0-21swapper/000:37:530
3100299200,2cyclictest23524-21diskmemload23:09:100
31008991917,2cyclictest0-21swapper/221:52:272
3100899190,18cyclictest0-21swapper/222:10:122
3100499193,1cyclictest23524-21diskmemload23:14:121
31010991817,1cyclictest0-21swapper/322:16:053
3101099181,1cyclictest0-21swapper/323:54:113
3100899182,14cyclictest0-21swapper/221:45:142
31008991814,3cyclictest268872sleep200:24:202
3100899180,1cyclictest0-21swapper/223:50:332
3100899180,18cyclictest0-21swapper/223:21:532
3100899180,18cyclictest0-21swapper/221:41:032
3100899180,17cyclictest0-21swapper/222:50:132
3100899180,0cyclictest0-21swapper/221:36:332
3100499182,16cyclictest38150irq/131-snd_hda23:08:051
31004991817,1cyclictest0-21swapper/100:14:551
3100299188,8cyclictest24740-21grep21:10:170
3100299186,11cyclictest16380-21munin-run20:55:010
3100299181,17cyclictest0-21swapper/022:46:280
3101099170,17cyclictest0-21swapper/300:12:313
3100899170,17cyclictest0-21swapper/223:19:132
3100899170,17cyclictest0-21swapper/200:37:132
3100899170,17cyclictest0-21swapper/200:34:132
3100899170,0cyclictest0-21swapper/223:09:442
3100899170,0cyclictest0-21swapper/222:06:132
3100899170,0cyclictest0-21swapper/221:29:032
3100899170,0cyclictest0-21swapper/221:14:242
3100499177,9cyclictest27802-21apache222:52:451
31004991715,1cyclictest12130-21sort19:35:241
31004991715,1cyclictest0-21swapper/122:26:401
3100299173,3cyclictest4360-21sh21:36:170
3100299173,2cyclictest11978-21apache_accesses22:20:110
3101099160,1cyclictest0-21swapper/321:33:493
3101099160,16cyclictest0-21swapper/300:00:463
31008991615,1cyclictest0-21swapper/223:32:262
3100499162,13cyclictest23524-21diskmemload23:32:311
31004991612,3cyclictest11487-21cut22:00:151
3100499160,16cyclictest0-21swapper/123:04:421
3100499160,15cyclictest23524-21diskmemload22:57:271
3100299169,6cyclictest30737-21chrt23:48:030
3100299169,6cyclictest29685-21wc20:15:000
3100299168,7cyclictest10716-21ssh22:38:340
3100299163,4cyclictest675-21sed21:15:160
3100299162,1cyclictest0-21swapper/020:05:130
3100299161,4cyclictest10662-21sh21:40:190
31002991612,2cyclictest13-21ksoftirqd/019:35:220
31010991513,1cyclictest39-21ksoftirqd/322:20:133
3101099150,15cyclictest0-21swapper/321:43:083
3100899153,10cyclictest0-21swapper/220:40:132
31008991510,3cyclictest0-21swapper/223:28:022
3100499155,1cyclictest18142-21kworker/1:022:05:441
3100499152,12cyclictest25-21ksoftirqd/121:50:161
31004991515,0cyclictest0-21swapper/121:32:241
31004991514,1cyclictest0-21swapper/100:00:271
3100299158,6cyclictest7978-21sed22:55:260
3100299158,6cyclictest5435-21taskset19:21:440
3100299158,6cyclictest31014-21ssh21:52:260
3100299158,6cyclictest30372-21grep22:50:150
3100299158,6cyclictest23109-21sh23:43:410
3100299158,6cyclictest22185-21ssh21:28:000
3100299158,6cyclictest1887-21seq23:11:200
3100299158,6cyclictest11099-21cat22:00:130
3100299158,6cyclictest10282-21tr22:00:000
3100299157,7cyclictest19425-21tr21:45:230
3100299157,2cyclictest31470-21tr19:10:130
3100299157,2cyclictest23430-21idleruntime20:00:170
3100299157,2cyclictest12716-21ssh00:15:200
3100299156,3cyclictest29007-21ls22:10:230
3100299155,5cyclictest23125-1kworker/u9:021:24:590
3100299151,3cyclictest4584-21/usr/sbin/munin20:30:150
3100299151,13cyclictest28811-21/usr/sbin/munin22:30:290
31002991510,4cyclictest31936-21sleep20:20:000
3100299150,3cyclictest0-21swapper/023:28:000
3100299150,2cyclictest0-21swapper/023:22:420
3100299150,1cyclictest0-21swapper/023:50:090
3100299150,12cyclictest28321-21ssh00:25:140
3100299150,0cyclictest0-21swapper/023:38:200
3101099142,1cyclictest141rcu_preempt22:52:263
31010991411,2cyclictest15596-21ls23:00:203
31010991410,3cyclictest31919-21sh20:20:003
31008991413,1cyclictest0-21swapper/222:29:392
31008991412,1cyclictest0-21swapper/221:57:302
31008991410,3cyclictest3315-21latency_hist19:19:592
31008991410,3cyclictest13310-21wc20:45:272
3100899140,1cyclictest0-21swapper/200:28:412
3100899140,14cyclictest0-21swapper/222:46:322
31004991414,0cyclictest0-21swapper/122:37:311
31004991410,3cyclictest14001-21dump-pmu-power20:50:001
3100499140,13cyclictest0-21swapper/123:20:181
3100499140,0cyclictest0-21swapper/123:26:041
3100299148,5cyclictest21401-21wc22:25:170
3100299147,6cyclictest5008-21sort00:30:190
3100299147,6cyclictest21205-21date21:05:010
3100299144,2cyclictest225-21systemd-journal19:49:550
3100299141,2cyclictest225-21systemd-journal19:30:000
3100299140,13cyclictest7018-21sh23:34:250
3100299140,13cyclictest3422-21expr00:10:130
3100299140,13cyclictest32454-21tr20:20:120
3100299140,13cyclictest1785-21forks19:15:150
287842140,0sleep20-21swapper/221:31:372
3101099139,3cyclictest8852-21cat23:15:253
3101099139,3cyclictest5264-21grep20:30:193
3101099139,3cyclictest32469-21grep19:10:223
3101099137,3cyclictest13729-21/usr/sbin/munin22:40:163
31010991311,2cyclictest39-21ksoftirqd/322:06:403
31010991310,2cyclictest7997-21grep23:35:013
31010991310,2cyclictest3072-21grep21:55:173
31010991310,2cyclictest29687-21head22:30:193
31010991310,2cyclictest2905-21sed19:15:243
31010991310,2cyclictest25858-21apt21:50:003
31010991310,2cyclictest25643-21grep23:45:143
31010991310,2cyclictest22334-21dump-pmu-power20:00:003
3101099130,0cyclictest0-21swapper/323:23:393
3100899130,12cyclictest9274-21head21:20:182
3100899130,12cyclictest0-21swapper/200:09:322
3100499139,2cyclictest27953-21tr22:10:161
3100499139,2cyclictest26548-21ssh23:45:191
3100499137,2cyclictest225-21systemd-journal23:51:421
3100499132,7cyclictest0-21swapper/100:35:141
31004991312,1cyclictest0-21swapper/123:59:531
31004991310,2cyclictest4387-21grep00:30:171
31004991310,2cyclictest22192-21cut21:05:171
31004991310,2cyclictest14190-21grep19:40:181
3100499130,13cyclictest0-21swapper/122:40:511
3100499130,12cyclictest31914-21cron20:20:001
3100299138,4cyclictest23424-21taskset21:09:000
3100299138,4cyclictest21930-21awk19:55:270
3100299138,4cyclictest14677-21cat19:40:240
3100299136,1cyclictest8536-21sort19:30:130
3100299132,3cyclictest225-21systemd-journal20:40:530
3100299131,6cyclictest18782-21basename21:00:000
3101099129,2cyclictest9111-21grep21:40:003
3101099129,2cyclictest8404-21wget19:30:123
3101099129,2cyclictest7924-21sed22:55:253
3101099129,2cyclictest6655-21head22:35:273
3101099129,2cyclictest5107-21cut00:30:193
3101099129,2cyclictest4297-21grep19:20:163
3101099129,2cyclictest3785-21grep20:25:263
3101099129,2cyclictest29646-21wc20:15:013
3101099129,2cyclictest27105-21dump-pmu-power20:10:003
3101099129,2cyclictest24400-21cut23:05:243
3101099129,2cyclictest22510-21grep22:45:173
3101099129,2cyclictest18670-21cron22:05:003
3101099129,2cyclictest15226-21sleep19:45:003
3101099128,3cyclictest9753-21ssh23:55:133
3101099128,3cyclictest20603-21run-parts00:40:003
3101099128,3cyclictest17602-21basename19:50:013
3101099121,6cyclictest0-21swapper/300:20:203
3100899129,2cyclictest9056-21awk20:40:012
3100899129,2cyclictest7627-21grep19:25:262
3100899129,2cyclictest5531-21head22:35:182
3100899129,2cyclictest29947-21wc22:30:222
3100899129,2cyclictest27598-21tr20:10:152
3100899129,2cyclictest20291-21wc21:00:232
3100899129,2cyclictest19377-21ssh00:00:192
3100899129,2cyclictest10047-21tr23:55:162
3100899128,3cyclictest9559-21grep19:30:202
3100899128,3cyclictest4460-21sed22:15:172
3100899125,7cyclictest27816-21apache223:02:582
3100899120,4cyclictest0-21swapper/223:41:122
3100899120,11cyclictest4344-21head19:20:152
3100499129,2cyclictest9709-21grep21:20:201
3100499129,2cyclictest9349-21grep23:35:161
3100499129,2cyclictest9152-21sort19:30:161
3100499129,2cyclictest29000-21cut00:25:171
3100499129,2cyclictest22533-21expr22:45:171
3100499129,2cyclictest19738-21grep21:00:181
3100499129,2cyclictest1578-21grep21:35:141
3100499129,2cyclictest13884-21ssh22:20:261
3100499129,2cyclictest1340-21tr21:15:211
3100499128,3cyclictest25661-21grep20:05:161
3100499124,7cyclictest7124-21/usr/sbin/munin19:25:221
3100499120,11cyclictest0-21swapper/121:41:271
3100299129,2cyclictest18634-21grep19:50:180
3100299129,2cyclictest12474-21chrt20:45:180
3100299127,1cyclictest3920-21taskset20:25:380
3100299126,2cyclictest29897-21ssh21:32:280
3100299122,2cyclictest225-21systemd-journal20:39:550
3101099118,2cyclictest331-21tr21:15:153
3101099118,2cyclictest31862-21grep23:10:163
3101099118,2cyclictest31603-21taskset21:52:533
3101099118,2cyclictest29367-21ssh00:25:193
3101099118,2cyclictest28348-21tune2fs22:10:173
3101099118,2cyclictest27353-21ssh22:29:183
3101099118,2cyclictest23364-21grep20:00:163
3101099118,2cyclictest13993-21ssh23:38:083
3101099118,2cyclictest11867-21ssh21:21:323
3101099118,2cyclictest11512-21wget00:15:143
3101099116,3cyclictest0-21swapper/321:12:353
3101099110,10cyclictest31547-21ssh00:07:333
3101099110,10cyclictest24764-21expr23:25:173
3100899118,2cyclictest31967-21head19:10:172
3100899118,2cyclictest27640-21ssh23:45:402
3100899118,2cyclictest23507-21cron21:10:002
3100899118,2cyclictest22327-21sh20:00:002
3100899118,2cyclictest19076-21grep19:50:222
3100899118,2cyclictest1842-21ssh23:11:172
3100899118,2cyclictest17048-21ssh22:22:502
3100899118,2cyclictest13410-21cat19:40:132
3100899118,2cyclictest11546-21ssh22:00:162
3100899116,3cyclictest22719-21grep20:00:142
3100499119,1cyclictest80632sleep120:35:211
3100499118,2cyclictest29538-21ssh00:06:001
3100499118,2cyclictest18352-21cut21:45:161
3100499118,2cyclictest18324-21cut23:40:221
3100499118,2cyclictest1743-21grep19:15:141
3100499118,2cyclictest17353-21head20:55:171
3100499118,2cyclictest1507-21sed20:20:231
3100499117,4cyclictest2048-21ssh22:33:171
3100499117,3cyclictest29021-21cat20:10:261
3100499117,2cyclictest4422-21wc20:30:011
3100499117,2cyclictest0-21swapper/119:20:141
3100499116,4cyclictest15170-21wc23:20:001
3100499116,1cyclictest4757-21wc20:30:161
3100499114,6cyclictest9978-21munin-node22:19:001
3100499113,4cyclictest225-21systemd-journal20:45:001
3101099108,1cyclictest7694-21who19:25:273
3101099108,1cyclictest7167-21expr20:35:163
3101099108,1cyclictest21751-21cat21:05:153
3101099108,1cyclictest199042sleep319:54:283
3101099108,1cyclictest184502sleep320:56:223
3101099108,1cyclictest17713-21gltestperf21:25:173
3101099108,1cyclictest13611-21sed20:45:293
3101099108,1cyclictest11069-21expr19:35:133
3101099108,1cyclictest10082-21head20:40:163
3101099107,2cyclictest882-21cut20:20:173
3101099105,4cyclictest0-21swapper/320:50:153
3100899108,1cyclictest4923-21expr20:30:162
3100899108,1cyclictest29831-21cut20:15:142
3100899108,1cyclictest26917-21chrt20:07:452
3100899108,1cyclictest184292sleep220:56:052
3100899108,1cyclictest128042sleep219:39:572
3100899107,2cyclictest3951-21taskset20:26:002
3100899107,2cyclictest17301-21taskset19:46:312
3100899107,2cyclictest14740-21expr20:50:152
3100899100,9cyclictest165982sleep222:41:362
3100499108,1cyclictest244792sleep120:01:561
3100499108,1cyclictest16104-21chrt20:51:441
3100499107,2cyclictest20294-21wget19:55:141
3100499107,2cyclictest18801-21cut19:50:191
3100499107,2cyclictest16402-21grep19:45:191
310109997,1cyclictest210722sleep321:04:163
310089992,3cyclictest225-21systemd-journal20:23:532
310049996,2cyclictest32289-21wc19:10:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional