You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 17:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 30, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
360929855,39sleep00-21swapper/007:08:340
368929467,9sleep30-21swapper/307:09:383
340729277,10sleep20-21swapper/207:05:552
351328868,6sleep10-21swapper/107:07:181
380899260,26cyclictest0-21swapper/312:17:393
380099260,25cyclictest0-21swapper/212:31:572
379999256,4cyclictest141rcu_preempt11:54:371
380099230,23cyclictest0-21swapper/210:18:432
379999230,22cyclictest0-21swapper/110:43:011
380099220,9cyclictest0-21swapper/209:17:152
3798992222,0cyclictest0-21swapper/010:42:510
379899220,22cyclictest0-21swapper/011:55:130
3800992118,2cyclictest32-21ksoftirqd/210:10:412
380099210,21cyclictest0-21swapper/210:47:052
379999210,3cyclictest0-21swapper/112:35:311
3798992121,0cyclictest0-21swapper/010:30:400
171632210,0sleep00-21swapper/010:58:420
380899200,2cyclictest0-21swapper/310:54:123
3799992018,1cyclictest0-21swapper/111:05:001
3798992014,5cyclictest27246-21apache209:55:030
3798992014,5cyclictest27246-21apache209:55:030
380099190,19cyclictest0-21swapper/211:12:182
379999193,1cyclictest29619-21expr09:10:161
379999191,1cyclictest4024-21kworker/1:010:15:101
379899190,1cyclictest0-21swapper/012:27:460
380899184,2cyclictest141rcu_preempt11:16:123
3800991811,5cyclictest17629-21run-parts10:39:592
380099180,16cyclictest0-21swapper/211:42:272
379999180,16cyclictest0-21swapper/110:25:511
379899182,7cyclictest0-21swapper/008:35:140
379899180,17cyclictest0-21swapper/009:26:450
380899174,3cyclictest141rcu_preempt11:00:043
3808991713,1cyclictest39-21ksoftirqd/312:31:053
380899171,1cyclictest141rcu_preempt10:34:273
380099170,17cyclictest0-21swapper/212:09:572
379999170,1cyclictest0-21swapper/110:32:091
379999170,15cyclictest0-21swapper/109:45:221
379899178,8cyclictest15264-21/usr/sbin/munin09:40:190
379899174,7cyclictest16688-21munin-run10:20:010
3798991716,1cyclictest0-21swapper/011:32:240
3808991612,1cyclictest39-21ksoftirqd/312:09:213
3808991612,1cyclictest39-21ksoftirqd/312:03:053
3808991612,1cyclictest39-21ksoftirqd/311:52:313
3800991615,1cyclictest0-21swapper/210:24:452
380099160,1cyclictest0-21swapper/211:57:512
380099160,1cyclictest0-21swapper/210:28:152
380099160,1cyclictest0-21swapper/207:25:142
380099160,0cyclictest0-21swapper/212:03:212
379999162,13cyclictest0-21swapper/107:45:131
3799991615,1cyclictest0-21swapper/112:03:541
3799991615,1cyclictest0-21swapper/109:34:581
3799991614,2cyclictest0-21swapper/112:29:321
379999161,15cyclictest28260-21diskmemload11:45:441
379999160,15cyclictest0-21swapper/109:40:021
379999160,13cyclictest0-21swapper/111:21:251
3808991511,4cyclictest0-21swapper/309:58:073
3808991511,4cyclictest0-21swapper/309:58:063
3808991510,1cyclictest39-21ksoftirqd/310:15:083
380099151,13cyclictest1948-21ssh09:31:232
380099150,1cyclictest0-21swapper/211:52:402
379999155,6cyclictest225-21systemd-journal08:33:141
379999151,14cyclictest0-21swapper/110:05:031
379999150,5cyclictest0-21swapper/111:41:441
379899156,6cyclictest537-21snmpd11:29:370
379899156,6cyclictest537-21snmpd11:29:370
379899151,9cyclictest20461-21perf08:50:000
3798991514,1cyclictest0-21swapper/009:31:450
3798991513,1cyclictest0-21swapper/010:45:110
379899151,2cyclictest868-21munin-run08:10:000
379899151,10cyclictest0-21swapper/010:53:340
3798991510,3cyclictest3796-21cyclictest11:41:360
379899150,9cyclictest14813-21tr09:40:000
379899150,7cyclictest225-21systemd-journal09:17:250
380899141,7cyclictest0-21swapper/310:23:093
380899141,7cyclictest0-21swapper/308:55:113
3808991413,1cyclictest39-21ksoftirqd/309:30:483
380899141,10cyclictest0-21swapper/310:57:313
3808991410,2cyclictest39-21ksoftirqd/311:08:373
380899140,1cyclictest0-21swapper/310:36:323
3800991411,2cyclictest9831-21df07:20:152
380099140,14cyclictest0-21swapper/209:38:192
379999146,4cyclictest225-21systemd-journal07:50:021
379999142,2cyclictest225-21systemd-journal12:10:001
3799991414,0cyclictest0-21swapper/110:45:441
3799991413,1cyclictest0-21swapper/110:23:411
379999140,12cyclictest0-21swapper/109:25:391
3798991413,1cyclictest0-21swapper/012:38:200
3798991412,2cyclictest13-21ksoftirqd/010:39:390
3798991411,2cyclictest9700-21grep12:30:250
379899140,1cyclictest0-21swapper/011:23:340
379899140,12cyclictest6197-21grep08:20:140
379899140,11cyclictest23978-21apt-get07:50:130
380899139,3cyclictest39-21ksoftirqd/309:15:193
380899138,4cyclictest27264-21apache212:10:213
380899138,2cyclictest22280-21cron09:25:003
380899135,2cyclictest225-21systemd-journal11:25:013
380899134,3cyclictest225-21systemd-journal08:33:143
3808991310,2cyclictest27534-21grep10:25:263
3808991310,2cyclictest1486-21cut08:10:133
380899130,1cyclictest0-21swapper/311:40:123
380899130,12cyclictest8536-21grep07:15:253
380899130,12cyclictest8343-21apt10:15:013
380099139,3cyclictest4330-21ssh10:31:072
380099139,3cyclictest20991-21hdparm10:40:532
3800991310,2cyclictest8335-21fgrep12:10:222
3800991310,2cyclictest2798-21grep10:50:122
380099130,13cyclictest0-21swapper/212:29:222
380099130,11cyclictest0-21swapper/211:29:222
380099130,11cyclictest0-21swapper/211:29:212
379999138,4cyclictest4424-21ssh09:15:021
379999136,5cyclictest5887-21irqstats11:30:181
379999134,4cyclictest9218-21/usr/sbin/munin12:30:211
379999132,10cyclictest26352-21apt-get07:55:111
379999131,1cyclictest6462-21ssh10:13:311
3799991310,2cyclictest6390-21basename07:15:001
3799991310,2cyclictest24494-21head09:00:161
3799991310,2cyclictest2259-21ssh11:28:231
3799991310,2cyclictest2259-21ssh11:28:221
379899139,3cyclictest455-21dbus-daemon09:50:220
379899139,3cyclictest22141-21cut07:45:150
379899139,3cyclictest13530-21grep11:35:160
379899136,6cyclictest13-21ksoftirqd/010:22:000
379899135,7cyclictest20049-21munin-run11:20:010
379899131,6cyclictest0-21swapper/007:35:000
379899131,2cyclictest0-21swapper/007:55:110
3798991310,2cyclictest5838-21grep11:10:260
3798991310,2cyclictest31828-21grep11:45:260
3798991310,2cyclictest27430-21sed10:25:250
3798991310,2cyclictest20650-21grep07:40:270
379899130,10cyclictest0-21swapper/009:20:250
380899129,2cyclictest32408-21grep08:05:223
380899129,2cyclictest31942-21cut12:25:143
380899129,2cyclictest17918-21wc12:35:253
380899128,3cyclictest1946-21sh10:49:483
380899125,6cyclictest27264-21apache209:27:383
380899122,6cyclictest28260-21diskmemload10:03:493
380899122,3cyclictest141rcu_preempt08:20:213
380899121,4cyclictest0-21swapper/309:45:123
3808991210,1cyclictest4985-21wc08:15:213
380899120,7cyclictest0-21swapper/311:25:443
380899120,7cyclictest0-21swapper/311:25:433
380899120,2cyclictest0-21swapper/311:11:543
380099129,2cyclictest4955-21grep08:15:212
380099129,2cyclictest4145-21cron09:15:002
380099129,2cyclictest30421-21grep08:00:272
380099129,2cyclictest25474-21grep09:45:262
380099129,2cyclictest2094-21grep08:10:162
380099129,2cyclictest18577-21grep08:45:162
380099129,2cyclictest16281-21tr10:00:142
380099129,2cyclictest16060-21uniq09:40:172
380099129,2cyclictest11262-21grep10:55:142
380099126,4cyclictest20523-21apt-get11:20:102
380099124,4cyclictest225-21systemd-journal12:39:592
380099120,1cyclictest0-21swapper/209:21:522
380099120,12cyclictest0-21swapper/211:04:482
380099120,11cyclictest32547-21ssh09:50:142
379999129,2cyclictest9759-21grep09:55:271
379999129,2cyclictest9759-21grep09:55:261
379999129,2cyclictest7483-21wc09:55:001
379999129,2cyclictest23756-21grep12:20:131
379999129,2cyclictest19106-21head08:45:181
379999129,2cyclictest17560-21grep08:40:241
379999129,2cyclictest10601-21cron08:30:011
379999129,2cyclictest10053-21grep10:35:121
3799991211,1cyclictest19382-21ssh12:17:121
379999120,11cyclictest14447-21grep11:55:151
379899129,2cyclictest6563-21grep11:50:160
379899129,2cyclictest26828-21grep09:05:170
379899129,2cyclictest25821-21cron09:05:000
379899129,2cyclictest23715-21tr12:20:130
379899129,2cyclictest15847-21sort12:15:150
379899129,2cyclictest12583-21head08:30:260
379899128,3cyclictest30344-21ssh09:10:230
379899128,3cyclictest29051-21tr08:00:150
379899128,3cyclictest22763-21grep12:00:140
379899128,2cyclictest7510-21grep12:10:160
3798991212,0cyclictest0-21swapper/012:06:540
379899120,10cyclictest0-21swapper/011:07:540
380899119,1cyclictest7771-21tail09:35:153
380899119,1cyclictest23103-21grep07:45:253
380899119,1cyclictest14529-21grep07:30:153
380899118,2cyclictest30178-21wc11:45:143
380899118,2cyclictest20962-21chrt07:43:063
380899118,2cyclictest18959-21ssh10:40:153
380899118,2cyclictest16014-21sed11:55:253
380899118,2cyclictest1004-21ssh09:12:313
380899117,1cyclictest23495-21apt-get07:50:023
380899113,4cyclictest0-21swapper/308:35:023
380899111,8cyclictest28260-21diskmemload11:35:163
380899110,9cyclictest0-21swapper/309:44:413
380899110,1cyclictest0-21swapper/310:08:423
380899110,10cyclictest27819-21sh12:22:133
380099119,1cyclictest6949-21grep07:15:122
380099118,2cyclictest896-21ssh11:08:052
380099118,2cyclictest8430-21cat08:25:012
380099118,2cyclictest32415-21taskset10:10:012
380099118,2cyclictest30424-21expr11:45:142
380099118,2cyclictest23999-21tr09:00:142
380099118,2cyclictest22139-21grep07:45:162
380099118,2cyclictest21054-21sleep11:40:002
380099118,2cyclictest15010-21rm12:15:062
380099118,2cyclictest14044-21taskset09:58:432
380099118,2cyclictest14044-21taskset09:58:422
380099118,2cyclictest13692-21basename08:35:152
380099118,2cyclictest12876-21systemd08:33:142
379999119,1cyclictest7759-21grep09:35:161
379999119,1cyclictest28277-21grep11:05:151
379999119,1cyclictest22212-21taskset10:03:351
379999119,1cyclictest13157-21basename11:35:131
379999118,3cyclictest10906-21rm12:12:031
379999118,2cyclictest29588-21head08:00:181
379999118,2cyclictest2622-21/usr/sbin/munin10:50:141
379999118,2cyclictest23558-21cat09:00:001
379999118,2cyclictest16039-21ssh11:16:521
379999112,4cyclictest20954-21sshd08:53:491
379899118,2cyclictest5750-21taskset08:19:010
379899118,2cyclictest5506-21wc07:10:180
379899118,2cyclictest21358-21grep11:00:270
379899118,2cyclictest19033-21sh10:01:040
379899118,2cyclictest166152sleep007:35:110
379899118,2cyclictest13341-21head07:25:260
379899118,2cyclictest1203-21fschecks_count10:10:170
379899113,5cyclictest26851-21users10:05:420
3798991111,0cyclictest0-21swapper/009:45:490
380899109,1cyclictest10622-21basename08:30:003
380899108,1cyclictest5353-21grep07:10:163
380899108,1cyclictest32077-21wget09:50:143
380899108,1cyclictest30620-21chrt08:01:253
380899108,1cyclictest185862sleep307:38:223
380899108,1cyclictest10940-21tail07:20:273
380899107,2cyclictest4595-21ssh11:30:013
380899107,2cyclictest27545-21chrt09:05:253
380899107,2cyclictest26670-21expr07:55:133
380899107,2cyclictest18331-21apache_volume08:45:143
380899107,2cyclictest17211-21wc08:40:213
380899100,6cyclictest13581-21sshd07:27:483
380099108,1cyclictest4864-21rm11:30:132
380099108,1cyclictest26321-21basename07:55:102
380099108,1cyclictest25300-21ssh09:25:492
380099108,1cyclictest23960-21basename07:50:112
380099108,1cyclictest15039-21hddtemp_smartct07:30:162
380099107,2cyclictest26717-21sort09:05:172
380099107,2cyclictest12068-21ssh11:15:082
380099100,9cyclictest23267-21ssh12:20:032
380099100,9cyclictest20045-21expr07:40:192
379999108,2cyclictest5136-21ssh11:10:191
379999108,1cyclictest6732sleep108:07:481
379999108,1cyclictest26082-21cat09:05:121
379999108,1cyclictest16947-21expr07:35:141
379999108,1cyclictest16562-21grep09:20:301
379999106,3cyclictest11082-21ssh10:55:151
379999104,3cyclictest225-21systemd-journal07:27:491
379999102,3cyclictest15668-21sh08:40:011
379999101,7cyclictest19019-21apt-get07:40:121
379899108,1cyclictest177622sleep008:41:130
379899108,1cyclictest11576-21wc07:24:590
379899107,2cyclictest8858-21taskset07:18:230
379899107,2cyclictest22535-21sed08:55:180
38089997,1cyclictest206592sleep308:50:013
38089993,2cyclictest225-21systemd-journal09:04:593
38009998,1cyclictest8895-21cat08:25:132
38009998,1cyclictest21352-21wget08:55:112
38009997,1cyclictest206682sleep208:50:082
38009997,1cyclictest16888-21grep07:35:142
38009997,1cyclictest16294-21cat08:40:142
38009996,2cyclictest4515-21tr07:10:112
38009990,9cyclictest31538-21cat08:05:142
37999998,1cyclictest1226-21dpkg08:10:011
37999998,1cyclictest11046-21taskset07:20:421
37999997,1cyclictest4180-21expr08:15:141
37999997,1cyclictest14521-21expr07:30:141
37999992,4cyclictest6094-21sshd08:20:111
37999992,3cyclictest0-21swapper/107:15:121
37989997,1cyclictest1924-21diskstats08:10:140
37989995,3cyclictest9560-21wc08:25:190
37989991,4cyclictest225-21systemd-journal08:49:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional