You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 13:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Mon Aug 10, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1489299260,2cyclictest23877-21sh00:15:563
1488299263,3cyclictest29320-21cut20:49:211
14879992624,1cyclictest13-21ksoftirqd/000:14:200
1489299250,24cyclictest2416-21ssh00:25:323
14887992522,2cyclictest28653-21latency_hist00:20:002
1488799250,24cyclictest0-21swapper/221:22:522
1488799250,1cyclictest0-21swapper/223:33:132
1488799250,1cyclictest0-21swapper/222:01:392
14882992523,1cyclictest0-21swapper/121:40:551
14879992523,1cyclictest0-21swapper/022:36:270
1487999250,24cyclictest62550irq/127-eno100:09:330
1487999250,24cyclictest27176-21sh23:21:350
1487999250,24cyclictest0-21swapper/022:54:200
1487999250,24cyclictest0-21swapper/020:20:010
14892992423,1cyclictest27183-21sh22:53:443
1489299240,23cyclictest7580-21ssh21:38:513
1489299240,23cyclictest26337-21ssh23:20:243
1489299240,23cyclictest18322-21ssh21:47:443
1489299240,23cyclictest0-21swapper/321:56:443
1489299240,1cyclictest8565-21diskmemload00:01:263
1489299240,1cyclictest0-21swapper/322:39:273
14887992422,1cyclictest3935-21apt-get21:04:212
14887992420,4cyclictest0-21swapper/222:06:342
1488799240,23cyclictest0-21swapper/223:43:172
1488799240,23cyclictest0-21swapper/222:12:452
1488799240,23cyclictest0-21swapper/200:30:062
1488799240,1cyclictest0-21swapper/222:53:442
14882992423,1cyclictest8565-21diskmemload23:09:181
14882992423,0cyclictest8565-21diskmemload21:38:081
14882992422,2cyclictest28650-21apt00:20:001
14882992422,1cyclictest20741-21apt-get00:14:201
14882992422,1cyclictest0-21swapper/123:56:121
1488299240,24cyclictest8565-21diskmemload23:39:401
1488299240,23cyclictest8565-21diskmemload21:49:041
1488299240,23cyclictest0-21swapper/123:42:561
1488299240,23cyclictest0-21swapper/121:27:001
1488299240,0cyclictest8565-21diskmemload22:36:421
1488299240,0cyclictest0-21swapper/121:22:521
14879992424,0cyclictest0-21swapper/023:37:440
14879992422,1cyclictest22065-21cron19:25:010
1487999240,24cyclictest0-21swapper/022:24:530
1487999240,23cyclictest0-21swapper/023:40:540
1487999240,1cyclictest141rcuc/000:04:320
1487999240,0cyclictest0-21swapper/023:46:360
14892992323,0cyclictest8565-21diskmemload23:30:553
14892992323,0cyclictest0-21swapper/322:19:553
14892992323,0cyclictest0-21swapper/321:43:153
14892992323,0cyclictest0-21swapper/300:20:533
14892992322,1cyclictest0-21swapper/322:12:453
14892992320,2cyclictest25746-21cron23:20:013
1489299230,2cyclictest12138-21ssh00:05:053
1489299230,23cyclictest15243-21kworker/u8:021:22:163
1489299230,23cyclictest0-21swapper/323:41:593
1489299230,23cyclictest0-21swapper/323:37:443
1489299230,23cyclictest0-21swapper/323:27:083
1489299230,23cyclictest0-21swapper/322:31:233
1489299230,23cyclictest0-21swapper/322:01:393
1489299230,22cyclictest9956-21ssh21:11:243
1489299230,22cyclictest8565-21diskmemload22:24:453
1489299230,22cyclictest32302-21ssh21:30:533
1489299230,22cyclictest22698-21ssh21:51:053
1489299230,22cyclictest19065-21ssh22:45:073
1489299230,22cyclictest17815-21sh00:10:083
1489299230,22cyclictest14927-21sh21:15:013
1489299230,22cyclictest13999-21rm00:35:563
1489299230,0cyclictest31966-21ssh22:57:343
1489299230,0cyclictest0-21swapper/300:34:263
14887992323,0cyclictest0-21swapper/222:42:442
14887992323,0cyclictest0-21swapper/221:40:552
14887992323,0cyclictest0-21swapper/221:38:082
14887992322,1cyclictest0-21swapper/223:46:042
14887992322,0cyclictest0-21swapper/221:31:422
14887992321,1cyclictest0-21swapper/223:05:542
14887992321,1cyclictest0-21swapper/222:18:112
14887992321,1cyclictest0-21swapper/221:56:312
14887992320,2cyclictest21029-21ssh22:47:512
1488799230,2cyclictest6154-40mandb00:00:012
1488799230,2cyclictest5038-21ssh00:29:092
1488799230,23cyclictest0-21swapper/223:28:102
1488799230,23cyclictest0-21swapper/222:55:262
1488799230,23cyclictest0-21swapper/221:27:002
1488799230,22cyclictest28140-21sh23:50:452
1488799230,22cyclictest14138-21munin-run20:15:012
1488799230,22cyclictest0-21swapper/223:11:072
1488799230,0cyclictest8565-21diskmemload23:18:302
1488799230,0cyclictest0-21swapper/223:02:162
1488799230,0cyclictest0-21swapper/200:20:532
14882992323,0cyclictest0-21swapper/123:52:441
14882992323,0cyclictest0-21swapper/123:21:261
14882992323,0cyclictest0-21swapper/122:50:371
14882992323,0cyclictest0-21swapper/122:42:441
14882992323,0cyclictest0-21swapper/121:11:451
14882992322,1cyclictest0-21swapper/122:27:331
14882992321,1cyclictest0-21swapper/123:46:591
14882992321,1cyclictest0-21swapper/122:57:381
14882992321,1cyclictest0-21swapper/122:05:471
14882992321,1cyclictest0-21swapper/100:05:581
14882992321,1cyclictest0-21swapper/100:03:381
1488299230,23cyclictest0-21swapper/123:27:341
1488299230,22cyclictest4939-21sh00:29:011
1488299230,22cyclictest25773-21dump-pmu-power23:20:011
1488299230,22cyclictest14137-21perf20:15:011
1488299230,22cyclictest13388-21sh00:35:101
1488299230,22cyclictest0-21swapper/123:10:551
1488299230,22cyclictest0-21swapper/122:19:191
1488299230,22cyclictest0-21swapper/121:51:071
1488299230,21cyclictest27277-21awk19:35:011
1488299230,0cyclictest8565-21diskmemload23:32:471
1488299230,0cyclictest8565-21diskmemload21:58:131
14879992323,0cyclictest8565-21diskmemload22:02:080
14879992323,0cyclictest0-21swapper/023:55:100
14879992323,0cyclictest0-21swapper/023:17:440
14879992323,0cyclictest0-21swapper/021:30:260
14879992323,0cyclictest0-21swapper/020:14:250
14879992323,0cyclictest0-21swapper/000:33:400
14879992322,1cyclictest5196-21ssh22:34:160
14879992321,1cyclictest15405-21diskmemload21:15:450
14879992321,1cyclictest14067-21sh00:36:010
14879992321,1cyclictest0-21swapper/019:15:000
14879992320,2cyclictest10365-21ssh21:11:540
14879992319,4cyclictest27246-21kworker/0:223:12:380
1487999230,23cyclictest0-21swapper/021:22:090
1487999230,23cyclictest0-21swapper/000:25:380
1487999230,22cyclictest9336-21dump-pmu-power20:05:010
1487999230,22cyclictest9336-21dump-pmu-power20:05:010
1487999230,22cyclictest522-21sh22:59:000
1487999230,22cyclictest14331-21sh22:41:160
1487999230,22cyclictest10591-21ssh23:07:270
1487999230,22cyclictest0-21swapper/021:51:120
1487999230,22cyclictest0-21swapper/020:39:210
1487999230,21cyclictest1026-21ssh23:27:270
1487999230,21cyclictest0-21swapper/023:33:570
1487999230,21cyclictest0-21swapper/021:45:210
1487999230,0cyclictest8565-21diskmemload22:13:130
1487999230,0cyclictest8565-21diskmemload00:15:440
1487999230,0cyclictest0-21swapper/022:28:580
1487999230,0cyclictest0-21swapper/022:08:080
14892992222,0cyclictest0-21swapper/323:47:003
14892992220,2cyclictest15087-21ssh23:10:553
14892992220,1cyclictest27290-21grep19:35:013
14892992220,1cyclictest13753-21sh22:40:333
1489299220,22cyclictest32709-21kworker/u8:222:08:003
1489299220,21cyclictest921-21wc20:55:013
1489299220,21cyclictest3712-21ssh23:00:353
1489299220,1cyclictest6928-21dump-pmu-power20:00:003
1489299220,0cyclictest0-21swapper/323:06:413
1489299220,0cyclictest0-21swapper/322:25:473
14887992222,0cyclictest8565-21diskmemload21:45:322
14887992222,0cyclictest0-21swapper/223:21:262
14887992222,0cyclictest0-21swapper/222:38:132
14887992220,1cyclictest22688-21ssh21:51:042
14887992220,1cyclictest10850-21ssh23:35:432
1488799220,21cyclictest29142-21apt-get20:49:202
1488799220,21cyclictest27286-21ls19:35:012
1488799220,21cyclictest17979-21ssh00:10:332
1488799220,21cyclictest14087-21ssh00:36:032
1488799220,21cyclictest0-21swapper/222:25:482
1488799220,21cyclictest0-21swapper/221:15:082
1488799220,21cyclictest0-21swapper/200:06:112
1488799220,1cyclictest313-21systemd-journal20:55:012
1488799220,1cyclictest0-21swapper/222:21:392
1488799220,1cyclictest0-21swapper/200:02:392
14882992222,0cyclictest0-21swapper/122:01:321
14882992222,0cyclictest0-21swapper/121:31:421
14882992222,0cyclictest0-21swapper/121:04:211
14882992222,0cyclictest0-21swapper/100:34:591
14882992220,1cyclictest0-21swapper/120:09:211
1488299220,21cyclictest0-21swapper/121:16:041
1488299220,21cyclictest0-21swapper/119:49:201
1488299220,1cyclictest919-21latency_hist20:55:011
1488299220,1cyclictest0-21swapper/122:45:561
1488299220,1cyclictest0-21swapper/122:31:501
1488299220,1cyclictest0-21swapper/122:21:391
1488299220,1cyclictest0-21swapper/122:12:371
14879992222,0cyclictest0-21swapper/021:55:050
14879992221,1cyclictest21455-21ssh22:48:330
14879992221,1cyclictest0-21swapper/023:50:540
14879992220,1cyclictest0-21swapper/021:35:060
1487999220,21cyclictest928-21turbostat.cron20:55:010
1487999220,21cyclictest62550irq/127-eno122:15:400
1487999220,21cyclictest5791-21awk21:05:010
1487999220,21cyclictest3474-21ssh23:00:170
1487999220,21cyclictest29881-21/usr/sbin/munin21:29:220
1487999220,21cyclictest11093-21sh21:40:230
1487999220,21cyclictest0-21swapper/021:09:190
14892992121,0cyclictest26747-21apt-get20:44:213
14892992121,0cyclictest0-21swapper/319:25:023
14892992120,1cyclictest32683-21apt-get19:49:203
1489299210,20cyclictest2668-21apt-get19:54:213
1489299210,20cyclictest17141-21apt-get20:24:203
1489299210,0cyclictest26421-21ssh21:25:333
1489299210,0cyclictest0-21swapper/323:50:423
14887992120,1cyclictest7542-21apt-get20:04:292
14887992120,1cyclictest7542-21apt-get20:04:292
14887992120,1cyclictest14714-21apt-get20:19:202
1488799210,20cyclictest0-21swapper/220:24:202
1488799210,1cyclictest75-21kswapd022:30:022
1488799210,0cyclictest0-21swapper/219:49:202
14882992120,1cyclictest5063-21apt-get19:59:221
14882992120,0cyclictest0-21swapper/120:04:291
14882992120,0cyclictest0-21swapper/120:04:291
14882992120,0cyclictest0-21swapper/119:14:191
1488299210,20cyclictest0-21swapper/120:24:201
1488299210,19cyclictest313-21systemd-journal20:20:011
1488299210,0cyclictest1512-21apt-get20:59:231
1488299210,0cyclictest0-21swapper/120:44:211
1488299210,0cyclictest0-21swapper/120:29:211
1487999210,0cyclictest21933-21apt-get20:34:200
1487999210,0cyclictest0-21swapper/019:49:200
14892992020,0cyclictest0-21swapper/320:59:233
14892992020,0cyclictest0-21swapper/320:29:213
14892992019,1cyclictest5785-21apt-get21:05:013
14892992019,1cyclictest0-21swapper/320:15:013
14892992019,0cyclictest0-21swapper/320:49:203
1489299200,20cyclictest0-21swapper/319:20:013
1489299200,20cyclictest0-21swapper/319:20:013
1489299200,0cyclictest0-21swapper/319:39:233
14887992020,0cyclictest0-21swapper/220:09:212
14887992020,0cyclictest0-21swapper/219:59:222
14887992020,0cyclictest0-21swapper/219:14:192
1488799200,0cyclictest0-21swapper/220:29:212
14879992019,1cyclictest30290-21apt-get19:44:210
14879992018,1cyclictest30956-21apt20:50:000
14879992018,1cyclictest29689-21perf19:40:000
14879992018,1cyclictest0-21swapper/019:19:220
14879992018,1cyclictest0-21swapper/019:19:220
1487999200,0cyclictest0-21swapper/020:24:200
14892991918,1cyclictest15474-21apt-get19:14:193
14887991919,0cyclictest0-21swapper/221:09:192
14887991919,0cyclictest0-21swapper/220:39:212
14887991919,0cyclictest0-21swapper/219:44:212
14882991919,0cyclictest0-21swapper/121:09:191
14882991919,0cyclictest0-21swapper/120:39:211
14882991919,0cyclictest0-21swapper/119:44:211
14882991918,1cyclictest27876-21apt-get19:39:231
14879991917,1cyclictest19538-21apt-get20:29:210
14892991818,0cyclictest9926-21apt-get20:09:213
14892991817,1cyclictest24330-21apt-get20:39:213
1489299180,17cyclictest20286-21apt-get19:24:203
14887991818,0cyclictest0-21swapper/219:39:232
14879991818,0cyclictest0-21swapper/019:59:220
14879991816,1cyclictest13-21ksoftirqd/019:30:010
14892991717,0cyclictest0-21swapper/320:19:203
14892991717,0cyclictest0-21swapper/320:04:293
14892991717,0cyclictest0-21swapper/320:04:293
14892991716,1cyclictest6779-21apt-get21:09:193
14892991716,1cyclictest0-21swapper/319:44:213
14887991716,0cyclictest0-21swapper/219:24:202
14882991716,0cyclictest0-21swapper/119:24:201
14879991715,1cyclictest0-21swapper/020:10:010
1487999170,17cyclictest0-21swapper/020:59:240
14887991616,0cyclictest0-21swapper/220:59:232
14887991616,0cyclictest0-21swapper/219:19:212
14887991616,0cyclictest0-21swapper/219:19:212
14887991614,1cyclictest28551-21perf20:45:002
14882991616,0cyclictest0-21swapper/119:19:211
14882991616,0cyclictest0-21swapper/119:19:211
14879991616,0cyclictest0-21swapper/019:34:200
1488299150,1cyclictest0-21swapper/119:54:301
14879991513,1cyclictest26716-21acpi20:44:210
1487999150,1cyclictest4488-21munin-run19:55:000
14887991414,0cyclictest0-21swapper/219:26:352
1488799140,14cyclictest0-21swapper/219:54:382
14882991414,0cyclictest0-21swapper/120:30:381
14892991313,0cyclictest0-21swapper/320:33:413
14887991313,0cyclictest0-21swapper/220:30:382
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional