You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-30 - 08:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat May 30, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1122629178,9sleep20-21swapper/219:08:032
1124628976,9sleep00-21swapper/019:08:170
1116028667,15sleep30-21swapper/319:07:133
924228060,16sleep10-21swapper/119:05:051
1146299270,24cyclictest0-21swapper/120:10:151
1146299264,19cyclictest10417-21sensors00:25:261
1145899240,23cyclictest0-21swapper/000:33:550
1146699230,21cyclictest0-21swapper/222:48:372
1146699230,1cyclictest0-21swapper/200:15:242
1145899230,21cyclictest0-21swapper/023:15:210
1146699220,0cyclictest0-21swapper/200:12:362
1145899220,3cyclictest0-21swapper/022:34:370
11468992119,2cyclictest0-21swapper/322:01:473
1146699210,0cyclictest0-21swapper/221:30:122
1146299210,20cyclictest7722-21kworker/1:120:45:141
11468992018,2cyclictest0-21swapper/322:12:453
1146899201,18cyclictest0-21swapper/323:20:123
1146299204,15cyclictest4047-21diskmemload22:24:301
11468991918,1cyclictest0-21swapper/322:07:083
1146699195,13cyclictest10668-21apt-get22:50:132
1146699190,18cyclictest0-21swapper/200:37:402
1146299191,2cyclictest19435-21/usr/sbin/munin19:25:151
11458991918,1cyclictest13-21ksoftirqd/000:09:590
1146899184,6cyclictest0-21swapper/322:55:023
11468991815,2cyclictest0-21swapper/321:45:563
11462991817,1cyclictest0-21swapper/122:30:241
11462991817,1cyclictest0-21swapper/122:04:421
1146299181,2cyclictest0-21swapper/121:55:251
1146299180,0cyclictest0-21swapper/123:28:111
1145899186,6cyclictest25861-21munin-run22:39:590
1146899170,16cyclictest0-21swapper/300:08:333
11466991716,1cyclictest0-21swapper/220:58:402
11462991713,3cyclictest4047-21diskmemload23:51:301
1146299170,16cyclictest2894-21cat22:25:251
1145899174,7cyclictest3116-21sendmail20:00:010
1146899160,11cyclictest0-21swapper/300:15:153
1146699165,8cyclictest0-21swapper/222:00:132
1146699163,7cyclictest0-21swapper/220:10:142
1146699160,16cyclictest0-21swapper/222:25:552
1146299162,10cyclictest0-21swapper/121:30:111
11462991616,0cyclictest0-21swapper/122:52:031
1146299161,2cyclictest0-21swapper/121:05:131
11462991612,4cyclictest0-21swapper/100:15:401
11462991610,2cyclictest23975-21sh00:15:001
1145899160,1cyclictest0-21swapper/000:38:530
1146899155,5cyclictest0-21swapper/300:35:133
1146899151,14cyclictest0-21swapper/321:21:003
1146899150,1cyclictest0-21swapper/323:29:583
1146699154,1cyclictest16750-21apt-get22:15:132
1146699152,1cyclictest0-21swapper/219:20:122
11462991514,1cyclictest0-21swapper/100:35:341
11462991511,3cyclictest0-21swapper/121:11:121
11462991510,3cyclictest11449-21cyclictest23:59:531
1146299150,1cyclictest0-21swapper/100:32:351
1146299150,13cyclictest0-21swapper/100:05:121
1145899154,6cyclictest23018-21apache222:28:320
11458991511,2cyclictest969-21ls21:00:250
1146899145,6cyclictest16477-21apt-get19:20:123
11468991411,2cyclictest11432-21grep22:30:243
11468991410,3cyclictest19656-21wc22:35:273
11468991410,3cyclictest1713-21grep19:55:173
11468991410,3cyclictest15026-21cat20:25:013
1146899140,9cyclictest5638-21users23:05:293
1146899140,9cyclictest5638-21users23:05:283
1146699142,11cyclictest55550irq/126-enp2s0-22:30:112
11466991411,2cyclictest6141-21ssh23:45:132
11466991410,3cyclictest17763-21basename20:30:142
11466991410,3cyclictest1200-21taskset00:00:432
1146299143,10cyclictest22366-21ssh23:15:271
11462991413,1cyclictest0-21swapper/123:42:421
11462991413,1cyclictest0-21swapper/121:41:001
1146299141,1cyclictest141rcu_preempt23:31:461
11462991410,3cyclictest3914-21grep23:05:171
11462991410,3cyclictest3914-21grep23:05:161
1146299140,1cyclictest0-21swapper/122:43:111
1146299140,13cyclictest14559-21grep20:20:261
1145899146,6cyclictest3153-21latency_hist23:05:010
1145899144,4cyclictest0-21swapper/022:55:010
11458991411,2cyclictest30363-21basename21:44:590
11458991411,2cyclictest20961-21turbostat19:30:000
11458991410,3cyclictest30546-21wc20:55:180
11458991410,3cyclictest23966-21cron00:15:000
11458991410,3cyclictest18258-21cut20:30:150
1146899139,3cyclictest8457-21wc21:50:213
1146899139,3cyclictest8457-21wc21:50:213
1146899139,3cyclictest20495-21df20:35:153
1146899139,3cyclictest16322-21tr00:10:173
1146899139,3cyclictest16049-21grep21:55:143
1146899139,3cyclictest15197-21tr19:15:183
1146899139,2cyclictest13436-21tr23:30:133
1146899135,4cyclictest0-21swapper/323:05:013
11468991310,2cyclictest8815-21tail20:10:173
11468991310,2cyclictest30090-21ssh23:40:113
11468991310,2cyclictest2364-21grep00:20:263
11468991310,2cyclictest18724-21grep20:30:203
1146899130,2cyclictest0-21swapper/322:53:393
1146699139,3cyclictest25073-21tr20:45:152
1146699139,3cyclictest19324-21ls22:35:232
1146699138,2cyclictest32-21ksoftirqd/223:30:012
11466991310,2cyclictest6410-21grep20:05:182
11466991310,2cyclictest31261-21grep19:50:132
11466991310,2cyclictest23627-21sh23:55:162
11466991310,2cyclictest17346-21grep21:55:242
1146699130,13cyclictest0-21swapper/222:05:112
1146699130,12cyclictest28150-21cron19:45:002
1146699130,12cyclictest13849-21sed23:10:252
1146699130,12cyclictest0-21swapper/219:25:122
1146299139,3cyclictest3465-21ssh22:45:181
1146299139,3cyclictest19362-21tr22:35:221
11462991310,2cyclictest30344-21grep19:45:261
11462991310,2cyclictest21442-21tr19:30:141
11462991310,2cyclictest19081-21wc20:30:251
1146299130,12cyclictest32712-21head19:50:251
1145899139,3cyclictest23756-21sed23:55:160
1145899139,3cyclictest10177-21wc22:10:240
1145899139,2cyclictest6248-21tr23:45:130
1145899139,2cyclictest31970-21wc19:50:190
1145899139,2cyclictest10745-21tr20:15:150
1145899134,3cyclictest225-21systemd-journal20:10:000
1145899131,6cyclictest21159-21munin-run23:35:000
1145899131,6cyclictest19837-21basename23:15:010
11458991313,0cyclictest0-21swapper/021:55:490
11458991310,2cyclictest29298-21wc21:25:000
11458991310,2cyclictest2298-21grep00:20:260
11458991310,2cyclictest13405-21grep19:10:250
1145899130,9cyclictest0-21swapper/023:35:140
1145899130,2cyclictest0-21swapper/022:21:120
1146899129,2cyclictest4799-21ssh21:10:143
1146899129,2cyclictest27753-21sed19:40:253
1146899129,2cyclictest25972-21wc19:40:003
1146899129,2cyclictest21905-21cron21:39:593
1146899129,2cyclictest20588-21tr23:15:113
1146899129,2cyclictest13365-21cat21:15:143
1146899128,3cyclictest8897-21ssh22:29:563
1146899128,3cyclictest4022-21sort22:45:253
1146899128,3cyclictest28855-21grep20:50:253
1146899120,12cyclictest0-21swapper/323:57:193
1146899120,11cyclictest19817-21basename23:15:013
1146699129,2cyclictest5593-21cat23:44:592
1146699129,2cyclictest27128-21sed22:20:252
1146699129,2cyclictest14180-21tr19:15:122
1146699128,2cyclictest29578-21grep23:20:162
1146699125,3cyclictest8640-21sensors_temp21:50:242
1146699125,3cyclictest8640-21sensors_temp21:50:232
1146699123,7cyclictest13274-21apt-get23:30:112
1146699120,1cyclictest0-21swapper/221:19:552
1146699120,11cyclictest15181-21cut23:50:162
1146699120,0cyclictest0-21swapper/200:05:522
1146299129,2cyclictest9210-21chrt23:46:191
1146299129,2cyclictest5906-21sh00:23:081
1146299129,2cyclictest5461-21cron20:05:011
1146299129,2cyclictest32581-21sort21:00:171
1146299128,3cyclictest31525-21cut21:45:161
1146299128,1cyclictest16745-21ssh23:12:371
1146299127,4cyclictest26422-21ssh23:37:341
1146299124,4cyclictest225-21systemd-journal19:44:591
1146299120,12cyclictest0-21swapper/122:06:341
1145899129,2cyclictest9014-21grep00:25:170
1145899129,2cyclictest8423-21grep21:50:210
1145899129,2cyclictest8423-21grep21:50:200
1145899129,2cyclictest6613-21ssh21:10:280
1145899129,2cyclictest3613-21grep20:00:130
1145899129,2cyclictest32355-21wc00:00:200
1145899129,2cyclictest27592-21wc19:40:240
1145899129,2cyclictest25328-21grep20:45:160
1145899129,2cyclictest25291-21tail00:15:150
1145899129,2cyclictest21981-21grep19:30:150
1145899129,2cyclictest15318-21head23:50:170
1145899122,7cyclictest22119-21run-parts20:40:010
1145899121,3cyclictest225-21systemd-journal20:30:010
1145899120,1cyclictest0-21swapper/022:03:070
1146899118,2cyclictest9129-21sh23:46:153
1146899118,2cyclictest8916-21cut00:25:153
1146899118,2cyclictest6867-21grep21:30:183
1146899118,2cyclictest32762-21ssh21:26:133
1146899118,2cyclictest27891-21ssh22:20:563
1146899118,2cyclictest2725-21head21:05:173
1146899118,2cyclictest27172-21expr22:40:173
1146899118,2cyclictest22435-21wget20:40:133
1146899118,2cyclictest22276-21sh21:40:073
1146899118,2cyclictest20940-21sh22:17:093
1146899118,2cyclictest17407-21ssh23:51:103
1146899118,2cyclictest1303-21taskset21:00:293
1146899117,3cyclictest31551-21cpuspeed_turbos00:00:143
1146899117,3cyclictest21447-21grep19:30:143
1146899116,3cyclictest12590-21cron20:20:003
1146899116,1cyclictest17802-21df_inode00:30:153
1146899115,2cyclictest225-21systemd-journal19:54:543
1146899111,7cyclictest0-21swapper/323:35:133
1146899111,2cyclictest225-21systemd-journal20:30:003
1146899110,10cyclictest5491-21basename20:05:003
1146699118,2cyclictest5767-21ssh23:05:342
1146699118,2cyclictest5767-21ssh23:05:332
1146699118,2cyclictest4606-21grep21:10:132
1146699118,2cyclictest30650-21ssh22:42:132
1146699118,2cyclictest30178-21expr21:25:142
1146699118,2cyclictest2586-21cut21:05:172
1146699118,2cyclictest23429-21ssh21:20:282
1146699118,2cyclictest23132-21ssh23:16:002
1146699118,2cyclictest22471-21cut21:40:142
1146699118,2cyclictest21595-21expr19:30:142
1146699118,2cyclictest19845-21ssh00:30:562
1146699118,2cyclictest1752-21head19:55:172
1146699118,2cyclictest16806-21expr20:25:252
1146699118,2cyclictest14544-21tail20:20:262
1146699118,2cyclictest1280-21head00:20:172
1146699118,2cyclictest114042sleep220:15:182
1146699118,2cyclictest10464-21sh00:25:262
11466991111,0cyclictest0-21swapper/223:37:012
1146699110,1cyclictest0-21swapper/223:04:062
1146699110,1cyclictest0-21swapper/222:12:062
1146699110,10cyclictest32290-21rm21:45:222
1146699110,10cyclictest23509-21sh22:57:152
1146699110,10cyclictest20791-21cut20:35:172
1146299119,1cyclictest12600-21idleruntime-cro20:20:001
1146299118,2cyclictest28044-21ssh21:24:051
1146299118,2cyclictest25654-21chrt19:38:441
1146299118,2cyclictest20067-21mii-tool22:55:161
1146299118,2cyclictest1640-21grep19:55:161
1146299118,2cyclictest16092-21ssh21:35:291
1146299118,2cyclictest1275-21ssh23:03:401
1146299118,1cyclictest10403-21ssh22:10:271
1146299115,5cyclictest13638-21uniq21:15:151
1146299113,6cyclictest4047-21diskmemload22:15:151
1146299112,7cyclictest24313-21kworker/1:220:55:131
1146299111,1cyclictest141rcu_preempt23:20:541
11462991110,1cyclictest0-21swapper/121:27:011
1146299110,10cyclictest0-21swapper/121:50:171
1146299110,10cyclictest0-21swapper/121:50:161
1145899118,2cyclictest6474-21grep21:30:150
1145899118,2cyclictest5986-21ssh22:46:430
1145899118,2cyclictest31985-21ssh21:25:380
1145899118,2cyclictest31632-21grep21:45:170
1145899118,2cyclictest30546-21timerwakeupswit23:20:230
1145899118,2cyclictest27266-21rm22:40:170
1145899118,2cyclictest24523-21taskset20:44:320
1145899118,2cyclictest1912-21cat21:05:110
1145899118,2cyclictest18520-21chrt22:15:260
1145899118,2cyclictest18436-21chrt19:23:160
1145899118,2cyclictest15006-21apt20:25:010
1145899118,2cyclictest13807-21ssh21:35:030
1145899118,2cyclictest13038-21ssh21:15:140
1145899118,2cyclictest11536-21ssh23:29:020
1145899118,2cyclictest10988-21tr22:50:140
1145899118,2cyclictest10197-21cat20:14:560
1145899117,3cyclictest28824-21grep20:50:250
1145899117,2cyclictest30657-21ssh23:40:150
1145899114,5cyclictest4788-21/usr/sbin/munin23:05:220
1145899114,5cyclictest4788-21/usr/sbin/munin23:05:210
1146899108,1cyclictest31448-21taskset20:57:003
1146899108,1cyclictest304472sleep319:46:193
1146899108,1cyclictest20209-21run-parts19:25:203
1146899108,1cyclictest12190-21sed19:10:153
1146899107,2cyclictest26712-21taskset20:47:083
1146899104,2cyclictest5739-21apt-get20:05:153
1146699108,2cyclictest29554-21awk19:45:182
1146699108,1cyclictest4810-21cat20:00:242
1146699108,1cyclictest32276-21expr21:00:152
1146699108,1cyclictest22780-21cat20:40:142
1146699108,1cyclictest14191-21aten2_r7power_p21:35:142
1146699107,2cyclictest24805-21ntp_offset19:35:202
1146699106,2cyclictest11929-21grep19:10:142
1146299108,1cyclictest6857-21grep20:05:211
1146299108,1cyclictest27422-21ls20:50:141
1146299108,1cyclictest16450-21basename19:20:101
1146299108,1cyclictest11796-21wget19:10:131
1146299107,2cyclictest3593-21chrt00:02:341
1146299107,2cyclictest24314-21chrt20:41:491
1146299107,2cyclictest16057-21chrt19:18:271
1146299102,6cyclictest22124-21latency_hist20:40:011
1145899108,1cyclictest3272-21sh22:06:250
1145899108,1cyclictest15231-21sh19:15:190
1145899106,2cyclictest25756-21cron19:40:000
1145899103,3cyclictest0-21swapper/019:45:130
114669997,1cyclictest28050-21cat20:50:172
114629997,1cyclictest15674-21expr20:25:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional