You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-03 - 13:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun May 03, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1448829876,16sleep00-21swapper/019:05:250
1494929280,8sleep20-21swapper/219:06:262
1492128963,15sleep10-21swapper/119:06:051
1519928863,9sleep30-21swapper/319:09:413
1531899280,3cyclictest0-21swapper/322:01:493
1531899260,25cyclictest0-21swapper/300:15:173
1531899240,5cyclictest0-21swapper/322:40:123
1531399240,23cyclictest0-21swapper/122:50:471
15318992321,1cyclictest477-21rs:main3
1531899230,21cyclictest0-21swapper/322:58:523
1531799230,1cyclictest0-21swapper/222:46:042
15313992321,1cyclictest477-21rs:main1
1531899222,15cyclictest0-21swapper/322:45:463
15318992221,1cyclictest0-21swapper/323:47:183
15317992119,1cyclictest0-21swapper/221:36:102
15313992119,1cyclictest477-21rs:main1
15310992120,1cyclictest0-21swapper/020:30:130
1531899200,19cyclictest0-21swapper/323:51:313
15313992018,1cyclictest477-21rs:main1
1531899190,4cyclictest0-21swapper/321:14:163
1531899190,17cyclictest0-21swapper/300:04:003
15317991917,1cyclictest0-21swapper/219:55:132
1531799190,1cyclictest141rcu_preempt00:25:412
1531399190,1cyclictest0-21swapper/123:17:481
15310991918,1cyclictest0-21swapper/021:21:340
1531099190,19cyclictest0-21swapper/023:09:480
1531899180,1cyclictest0-21swapper/323:14:353
15317991816,1cyclictest8329-21diskmemload00:35:062
1531799180,1cyclictest0-21swapper/221:10:222
1531799180,18cyclictest0-21swapper/223:48:262
15313991817,1cyclictest8329-21diskmemload23:59:141
15318991715,2cyclictest0-21swapper/321:26:443
1531799179,5cyclictest475-21in:imuxsock22:30:002
15317991716,1cyclictest0-21swapper/200:06:122
1531799170,10cyclictest0-21swapper/222:00:012
15313991717,0cyclictest0-21swapper/123:34:581
15313991712,4cyclictest38150irq/131-snd_hda00:40:001
1531399171,1cyclictest141rcu_preempt22:35:051
1531399171,15cyclictest8329-21diskmemload21:43:131
1531899164,7cyclictest12107-21ssh22:10:133
15318991611,4cyclictest39-21ksoftirqd/320:05:223
1531899161,13cyclictest0-21swapper/321:20:123
15317991615,1cyclictest0-21swapper/223:25:122
1531399160,15cyclictest0-21swapper/123:23:591
1531399160,0cyclictest0-21swapper/122:14:171
1531099162,4cyclictest5406-21df_abs22:25:140
1531899154,6cyclictest11575-21munin-run22:10:003
15318991514,1cyclictest0-21swapper/321:35:183
15317991511,4cyclictest5047-21apache223:16:552
15317991511,1cyclictest0-21swapper/200:22:292
1531799150,15cyclictest0-21swapper/223:02:542
1531799150,15cyclictest0-21swapper/222:22:382
1531799150,14cyclictest0-21swapper/200:30:252
1531399152,13cyclictest8329-21diskmemload22:33:241
15313991513,1cyclictest0-21swapper/121:46:221
15313991510,2cyclictest0-21swapper/123:02:391
1531099157,7cyclictest29555-21munin-run19:40:000
1531099152,7cyclictest9124-21perf20:05:000
1531099150,8cyclictest16271-21perf20:20:000
1531099150,2cyclictest0-21swapper/023:34:320
1531099150,12cyclictest6205-21apache_accesses21:05:120
15318991412,2cyclictest39-21ksoftirqd/300:11:113
15318991410,3cyclictest8858-21uniq23:25:153
15318991410,3cyclictest2997-21grep20:55:253
15318991410,3cyclictest22286-21sort22:55:003
1531899140,2cyclictest0-21swapper/300:05:523
1531899140,1cyclictest0-21swapper/321:44:273
1531899140,1cyclictest0-21swapper/300:34:353
1531799141,9cyclictest937-21cron21:25:002
15317991414,0cyclictest0-21swapper/223:34:102
15317991412,2cyclictest0-21swapper/219:10:142
15317991412,1cyclictest32-21ksoftirqd/222:41:292
15317991410,3cyclictest12831-21wc21:50:272
1531799140,14cyclictest0-21swapper/223:59:082
1531399143,9cyclictest26739-21kworker/1:222:20:131
15313991410,3cyclictest6623-21wc22:25:251
15313991410,3cyclictest14974-21cron23:10:001
1531399140,3cyclictest0-21swapper/123:50:121
1531399140,1cyclictest0-21swapper/121:38:281
15310991412,2cyclictest17490-21ssh00:09:260
15310991412,2cyclictest0-21swapper/023:23:020
1531099140,14cyclictest0-21swapper/022:45:080
1531899139,3cyclictest6713-21awk20:00:003
1531899135,4cyclictest225-21systemd-journal00:23:053
1531899134,3cyclictest21487-21apt-get20:30:143
15318991312,1cyclictest0-21swapper/323:22:333
15318991310,2cyclictest9922-1kworker/u9:223:30:193
15318991310,2cyclictest32691-21perf23:40:003
15318991310,2cyclictest32215-21wc23:00:173
15318991310,2cyclictest13793-21grep00:25:263
1531899130,13cyclictest0-21swapper/323:17:303
1531799139,3cyclictest23892-21grep20:35:142
1531799139,3cyclictest1702-21grep23:20:262
1531799139,2cyclictest23072-21tr22:55:152
1531799137,5cyclictest1035-21packagekitd21:15:432
1531799134,4cyclictest32-21ksoftirqd/222:10:132
1531799131,0cyclictest141rcu_preempt00:18:032
15317991310,2cyclictest7722-21df23:05:152
15317991310,2cyclictest6549-21rm22:06:042
15317991310,2cyclictest32-21ksoftirqd/222:55:002
15317991310,2cyclictest27865-21sed20:40:232
15317991310,2cyclictest21748-21tr20:30:162
15317991310,2cyclictest18983-21wc19:15:202
15317991310,2cyclictest18368-21grep20:20:242
1531399139,3cyclictest6795-21wc20:00:011
1531399139,3cyclictest15601-21grep23:10:141
1531399138,4cyclictest19392-21sed00:10:151
1531399136,5cyclictest25-21ksoftirqd/121:00:141
15313991310,2cyclictest871-21cut19:45:211
15313991310,2cyclictest29507-21sed00:15:271
15313991310,2cyclictest24657-21grep22:55:261
15313991310,2cyclictest20215-21head21:55:161
15313991310,2cyclictest19745-21sh00:30:001
15313991310,2cyclictest18576-21sed21:15:221
15313991310,2cyclictest17713-21grep21:35:011
15313991310,2cyclictest11380-21rm00:25:001
1531399130,13cyclictest0-21swapper/123:26:281
1531399130,13cyclictest0-21swapper/122:48:311
1531399130,12cyclictest2707-21sort00:00:161
1531099139,2cyclictest3323-21cat22:05:000
1531099135,7cyclictest4059-21munin-run00:40:000
1531099134,8cyclictest0-21swapper/020:25:000
1531099131,6cyclictest0-21swapper/022:00:010
15310991310,2cyclictest5428-21cat22:05:250
15310991310,2cyclictest31604-21cat22:40:240
15310991310,2cyclictest27472-21head20:40:190
15310991310,2cyclictest27083-21cron00:15:000
15310991310,2cyclictest24810-21sleep19:30:000
15310991310,2cyclictest24810-21sleep19:30:000
15310991310,2cyclictest22920-21grep22:35:210
15310991310,2cyclictest20063-21cat20:25:160
15310991310,2cyclictest10681-21grep21:10:250
1531099130,1cyclictest8329-21diskmemload23:59:590
1531899129,2cyclictest31452-21grep19:40:263
1531899129,2cyclictest29085-21grep20:45:153
1531899129,2cyclictest20690-21grep21:55:193
1531899129,2cyclictest17891-21wc21:35:013
1531899129,2cyclictest17642-21cron19:15:003
1531899128,3cyclictest29131-21tr22:20:153
1531899123,4cyclictest28276-21idleruntime-cro22:20:003
1531799129,2cyclictest560-21wc20:50:252
1531799129,2cyclictest21798-21ssh23:52:062
1531799129,2cyclictest2046-21cut20:55:152
1531799129,2cyclictest19798-21grep20:25:132
1531799129,2cyclictest10576-21munin-run21:50:002
1531799129,2cyclictest10123-21sed20:05:172
1531799124,5cyclictest28933-21wc22:00:192
15317991210,2cyclictest32-21ksoftirqd/222:33:392
15317991210,1cyclictest32-21ksoftirqd/221:44:502
1531799120,8cyclictest0-21swapper/221:30:332
1531399129,2cyclictest32148-21grep20:50:171
1531399129,2cyclictest27910-21tail20:40:241
1531399129,2cyclictest2233-21grep19:50:131
1531399129,2cyclictest19446-21sed19:15:271
1531399129,2cyclictest12509-21grep21:50:231
1531399129,2cyclictest11809-21cut20:10:131
1531399123,6cyclictest30665-21gltestperf22:40:151
1531399122,9cyclictest32537-21sh22:02:351
1531399120,2cyclictest0-21swapper/123:35:101
1531399120,12cyclictest0-21swapper/123:43:371
1531399120,12cyclictest0-21swapper/100:30:011
1531399120,11cyclictest29257-21rm21:21:481
1531399120,11cyclictest19596-21grep20:25:121
1531099129,2cyclictest9431-21tr23:45:140
1531099129,2cyclictest5400-21cat21:00:260
1531099129,2cyclictest4304-21wc19:55:000
1531099129,2cyclictest30609-21head19:40:170
1531099129,2cyclictest28898-21ntpq00:15:210
1531099129,2cyclictest26500-21cat23:35:250
1531099129,2cyclictest24481-21grep20:35:170
1531099129,2cyclictest13472-21grep22:10:210
1531099129,2cyclictest12937-21cut20:10:200
1531099129,2cyclictest11892-21head21:50:180
1531099129,2cyclictest10563-21sh21:50:000
1531099126,6cyclictest17707-21munin-run21:35:000
1531099122,9cyclictest28841-21apt-get22:20:150
1531099121,7cyclictest0-21swapper/020:05:130
1531899118,2cyclictest7927-21uname20:00:193
1531899118,2cyclictest5940-21apt21:05:003
1531899118,2cyclictest3389-21ssh23:40:563
1531899118,2cyclictest32383-21rm00:36:463
1531899118,2cyclictest303-21grep20:50:223
1531899118,2cyclictest2979-21grep21:45:143
1531899118,2cyclictest27171-21cron19:35:003
1531899118,2cyclictest25767-21rm23:55:023
1531899118,2cyclictest23962-21tr20:35:143
1531899118,2cyclictest18372-21cat19:15:153
1531899118,2cyclictest17149-21cat20:20:133
1531899118,2cyclictest12595-21cat20:10:173
1531899118,2cyclictest11062-21rm22:28:463
1531899118,2cyclictest10992-21ssh21:50:103
1531899117,3cyclictest15733-21grep20:15:253
1531899113,7cyclictest5047-21apache219:25:203
1531899113,7cyclictest5047-21apache219:25:203
1531899110,10cyclictest23713-21ssh22:35:353
1531799119,1cyclictest20564-21wc00:10:262
1531799118,2cyclictest8047-21sh00:03:342
1531799118,2cyclictest4442-21ssh21:26:122
1531799118,2cyclictest30895-21wc19:40:192
1531799118,2cyclictest28188-21grep19:35:172
1531799118,2cyclictest27490-21sh23:35:572
1531799118,2cyclictest26367-21ssh22:37:392
1531799118,2cyclictest25522-21ssh22:17:502
1531799118,2cyclictest23738-21cat19:25:202
1531799118,2cyclictest23738-21cat19:25:192
1531799118,2cyclictest15358-21tr20:15:202
1531799118,2cyclictest1284-21grep23:40:142
1531799116,3cyclictest4812-21missed_timers21:00:202
1531399118,2cyclictest9639-21grep20:05:151
1531399118,2cyclictest8465-21wc21:10:001
1531399118,2cyclictest30066-21grep19:40:141
1531399118,2cyclictest25559-21df19:30:151
1531399118,2cyclictest24586-21taskset19:27:201
1531399118,2cyclictest24586-21taskset19:27:201
1531399118,2cyclictest22154-21taskset19:21:371
1531399118,2cyclictest14463-21ssh23:48:021
1531399117,3cyclictest24948-21ssh22:17:231
1531399117,3cyclictest24532-21grep20:35:171
1531399112,3cyclictest0-21swapper/120:15:101
1531399110,10cyclictest0-21swapper/121:10:371
1531099119,1cyclictest3803-21grep00:20:150
1531099119,1cyclictest21701-21ssh23:52:020
1531099118,2cyclictest8936-21grep23:25:150
1531099118,2cyclictest5276-21uniq19:55:170
1531099118,2cyclictest32501-21tr19:45:160
1531099118,2cyclictest31239-21hostname23:00:120
1531099118,2cyclictest30599-21sh23:00:010
1531099118,2cyclictest26430-21ssh00:33:380
1531099118,2cyclictest21009-21taskset22:15:160
1531099118,2cyclictest1009-21tr23:40:130
15310991110,1cyclictest0-21swapper/021:42:490
15310991110,1cyclictest0-21swapper/000:02:570
1531899108,1cyclictest9631-21ssh23:05:483
1531899108,1cyclictest6548-21expr21:05:153
1531899108,1cyclictest21202sleep319:50:033
1531899108,1cyclictest190622sleep320:25:013
1531899107,2cyclictest20723-21expr19:20:143
1531899107,2cyclictest1889-21cron19:50:003
1531899107,2cyclictest12970-21ssh22:30:053
1531899100,9cyclictest27603-21sendmail_mailtr20:40:203
1531799108,1cyclictest8246-21grep20:00:222
1531799108,1cyclictest82142sleep221:08:512
1531799108,1cyclictest15416-21ssh23:10:102
1531799107,2cyclictest12372-21cut20:10:152
1531799106,3cyclictest26612-21wc19:30:272
1531799101,2cyclictest225-21systemd-journal20:46:282
1531799100,9cyclictest20448-21cut19:20:142
1531399108,1cyclictest6996-21dpkg20:00:121
1531399108,1cyclictest22269-21expr20:30:171
1531399108,1cyclictest17654-21uname20:20:181
1531399107,3cyclictest5047-21apache219:35:211
1531399107,2cyclictest2726-21cut20:55:221
1531399107,2cyclictest15825-21grep19:10:141
1531399100,9cyclictest29993-21chrt20:45:221
1531099108,2cyclictest13101-21ssh22:30:110
1531099108,1cyclictest31426-21apt20:50:120
1531099108,1cyclictest25981-21ssh23:15:310
1531099108,1cyclictest18865-21cut19:15:190
1531099108,1cyclictest17034-21taskset21:15:070
1531099108,1cyclictest15542-21apache_processe19:10:110
1531099107,2cyclictest2932-21rt-features21:25:210
1531099107,2cyclictest29106-21cut20:45:150
1531099107,2cyclictest21932-21grep19:20:250
1531099107,2cyclictest19365-21rm23:11:540
1531099107,2cyclictest15208-21ssh00:26:270
1531099106,3cyclictest26768-21sed19:30:260
1531099102,4cyclictest225-21systemd-journal21:00:000
1531099100,9cyclictest20488-21ssh21:35:390
1531099100,9cyclictest18853-21chrt22:52:200
153189996,2cyclictest27636-21ls19:35:143
153179997,1cyclictest32293-21cut19:45:152
153179997,1cyclictest2983-21basename19:50:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional