You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 07:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Dec 04, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
847210476,9sleep10-21swapper/119:05:391
965210375,9sleep30-21swapper/319:07:103
110928464,16sleep20-21swapper/219:09:012
85128269,9sleep00-21swapper/019:05:420
160832660,0sleep00-21swapper/022:30:290
136822620,0sleep20-21swapper/222:51:182
301232600,1sleep030121-21ssh23:03:070
1266992525,0cyclictest0-21swapper/122:23:301
151632230,0sleep00-21swapper/022:52:490
127399231,0cyclictest0-21swapper/322:42:253
1271992322,1cyclictest0-21swapper/223:30:282
127199231,2cyclictest9039-21ssh21:20:152
1266992220,1cyclictest32613-21python21:35:261
127199211,19cyclictest0-21swapper/221:57:032
127199210,20cyclictest0-21swapper/200:00:182
126699210,20cyclictest0-21swapper/100:25:021
1273992020,0cyclictest0-21swapper/320:05:163
127199200,1cyclictest0-21swapper/221:50:142
125999204,14cyclictest0-21swapper/019:50:120
64162190,0sleep30-21swapper/321:40:153
127199191,1cyclictest0-21swapper/200:23:542
126699190,15cyclictest0-21swapper/122:29:451
1273991817,1cyclictest0-21swapper/321:27:313
127199180,18cyclictest0-21swapper/222:49:152
127199180,17cyclictest0-21swapper/200:31:522
125999179,4cyclictest14568-21apache_volume00:00:140
125999172,8cyclictest224-21systemd-journal23:25:000
287392160,0chrt0-21swapper/021:33:320
127399169,5cyclictest9243-21chrt21:20:153
1273991613,2cyclictest0-21swapper/322:33:343
127199160,16cyclictest0-21swapper/223:28:432
126699166,2cyclictest141rcu_preempt21:59:431
125999161,2cyclictest0-21swapper/022:25:570
125999160,12cyclictest1029-21/usr/sbin/munin23:50:160
127399159,2cyclictest493-21systemd-logind19:28:493
127399152,12cyclictest0-21swapper/323:20:253
127199150,15cyclictest0-21swapper/223:51:232
1266991513,2cyclictest0-21swapper/122:09:241
126699150,0cyclictest0-21swapper/100:10:281
125999156,3cyclictest224-21systemd-journal20:40:000
125999151,11cyclictest25759-21basename20:00:000
311172140,2sleep10-21swapper/100:35:531
127399142,4cyclictest0-21swapper/323:25:133
1273991411,2cyclictest18382-21grep22:55:163
127399140,1cyclictest0-21swapper/321:39:523
127399140,14cyclictest0-21swapper/323:51:283
127199145,3cyclictest224-21systemd-journal20:46:232
127199144,9cyclictest15272-21ssh22:08:452
127199143,1cyclictest0-21swapper/221:33:592
1271991411,2cyclictest7613-21cron20:30:002
127199140,2cyclictest0-21swapper/221:41:492
127199140,1cyclictest0-21swapper/223:41:422
1266991410,3cyclictest8148-21tr20:30:141
125999147,6cyclictest23085-21munin-run21:30:000
125999140,2cyclictest0-21swapper/023:56:390
125999140,13cyclictest25268-21sh00:07:040
127399139,3cyclictest21517-21sed20:55:263
127399139,3cyclictest16079-21fgrep00:00:253
127399139,3cyclictest1000-21tr20:15:143
1273991312,1cyclictest0-21swapper/323:34:493
1273991312,1cyclictest0-21swapper/322:49:243
1273991310,2cyclictest9425-21grep23:55:273
1273991310,2cyclictest26802-21grep23:45:203
1273991310,2cyclictest26309-21tr20:00:163
1273991310,2cyclictest24250-21cut19:55:183
1273991310,2cyclictest23537-21sed00:05:293
1273991310,2cyclictest22332-21grep21:50:283
127399130,2cyclictest0-21swapper/300:15:223
127399130,1cyclictest141rcu_preempt21:47:373
127199139,3cyclictest30517-21apt20:10:002
127199139,3cyclictest20950-21cut22:35:002
127199139,3cyclictest12758-21wc23:35:282
127199138,4cyclictest22587-21expr23:20:182
127199131,1cyclictest2024-21ssh00:14:342
127199131,11cyclictest0-21swapper/223:47:102
1271991310,2cyclictest3848-21grep20:20:192
1271991310,2cyclictest32701-21wc22:20:192
1271991310,2cyclictest24065-21cat22:14:592
1271991310,2cyclictest1779-21tr19:10:132
1271991310,2cyclictest11005-21wc19:30:012
126699139,3cyclictest7616-21basename23:55:141
126699139,3cyclictest6171-21tr21:40:151
126699139,3cyclictest3968-21grep19:15:151
126699139,3cyclictest3087-21tr00:15:171
126699139,2cyclictest27921-21sort21:10:181
126699139,2cyclictest15266-21tr20:45:141
126699138,3cyclictest21509-21tr19:50:141
126699137,3cyclictest0-21swapper/122:10:191
126699135,4cyclictest25738-21munin-run20:00:001
126699132,2cyclictest141rcu_preempt22:15:291
1266991310,2cyclictest4346-21grep22:00:251
1266991310,2cyclictest29970-21grep22:40:201
1266991310,2cyclictest27314-21grep23:45:261
1266991310,2cyclictest25-21ksoftirqd/123:06:411
1266991310,2cyclictest23042-21wc22:35:271
1266991310,2cyclictest22590-21sed23:20:181
1266991310,2cyclictest12086-21ntpq22:50:201
125999139,3cyclictest21235-21grep21:50:190
125999139,3cyclictest16344-21sed23:15:270
125999138,3cyclictest26747-21diskmemload22:46:300
1259991310,2cyclictest5341-21cat19:15:270
1259991310,2cyclictest25275-21grep21:05:170
1259991310,2cyclictest23236-21grep00:30:170
127399129,2cyclictest7251-21taskset19:20:193
127399129,2cyclictest29550-21tail00:10:203
127399129,2cyclictest2786-21tr22:00:143
127399129,2cyclictest21279-21ssh22:35:143
127399129,2cyclictest2026-21sh22:00:003
127399129,2cyclictest19803-21ssh22:11:073
127399129,2cyclictest19600-21sleep20:55:003
127399129,2cyclictest1902-21grep23:05:273
127399129,2cyclictest17076-21tr00:25:223
127399129,2cyclictest1460-21cron00:40:013
127399129,2cyclictest11871-21tr23:35:193
127399129,2cyclictest10332-21cat00:20:223
127399129,2cyclictest10297-21rm23:11:363
127399126,2cyclictest224-21systemd-journal19:16:153
127399120,11cyclictest27200-21tail23:00:283
127399120,11cyclictest17793-21cat19:40:253
127399120,11cyclictest14116-21tr19:35:163
127199129,2cyclictest8329-21tr23:55:172
127199129,2cyclictest31317-21grep21:35:152
127199129,2cyclictest30183-21strings00:35:222
127199129,2cyclictest25727-21grep21:05:202
127199129,2cyclictest22025-21cut19:50:172
127199129,2cyclictest18897-21head22:55:182
127199129,2cyclictest1854-21ls23:05:272
127199129,2cyclictest17586-21cut19:40:222
127199129,2cyclictest119192sleep223:13:062
127199128,3cyclictest8171-21grep20:30:142
127199128,3cyclictest6643-21tr19:20:142
127199128,2cyclictest28400-21sh21:10:212
127199128,2cyclictest24516-21grep22:15:182
127199127,4cyclictest15371-21cat19:35:272
127199120,2cyclictest0-21swapper/223:16:272
127199120,12cyclictest0-21swapper/222:44:542
126699129,2cyclictest2853-21ls20:20:001
126699129,2cyclictest27731-21grep20:00:261
126699129,2cyclictest24325-21tail21:30:181
126699129,2cyclictest22811-21tail00:05:221
126699129,2cyclictest22715-21ssh23:42:491
126699129,2cyclictest2230-21df21:15:171
126699129,2cyclictest16567-21tr21:25:151
126699129,2cyclictest16321-21grep19:40:131
126699129,2cyclictest14356-21ssh22:30:151
126699129,2cyclictest13360-21turbostat.cron19:35:001
126699129,2cyclictest12916-21grep20:40:151
126699128,2cyclictest22300-21tail21:50:291
126699127,4cyclictest20119-21sh23:18:571
126699125,5cyclictest10348-21idleruntime-cro22:50:011
126699122,5cyclictest224-21systemd-journal19:28:491
125999129,2cyclictest8154-21tr20:30:140
125999129,2cyclictest32141-21grep23:50:000
125999129,2cyclictest31841-21grep21:35:170
125999129,2cyclictest29247-21tr23:25:140
125999129,2cyclictest29054-21grep21:10:260
125999129,2cyclictest255432sleep022:15:230
125999129,2cyclictest18013-21grep22:10:190
125999129,2cyclictest16551-21tr19:40:130
125999129,2cyclictest14140-21ssh22:07:400
125999129,2cyclictest13427-21wc19:35:010
125999128,3cyclictest18693-21ssh22:55:170
125999128,3cyclictest10427-21grep19:25:240
125999124,4cyclictest224-21systemd-journal20:24:590
125999120,3cyclictest0-21swapper/021:47:190
125999120,11cyclictest30502-21cron20:10:000
127399118,2cyclictest8406-21sed22:25:273
127399118,2cyclictest3941-21ssh22:23:013
127399118,2cyclictest31114-21expr20:10:153
127399118,2cyclictest28884-21cat21:10:273
127399118,2cyclictest28878-21ssh22:17:523
127399118,2cyclictest26648-21chrt21:08:573
127399118,2cyclictest24350-21grep21:30:173
127399118,2cyclictest2306-21sed21:15:183
127399118,2cyclictest22445-21ssh00:30:053
127399118,2cyclictest18672-21grep23:40:183
127399118,2cyclictest14324-21grep20:40:283
127399118,2cyclictest10896-21wget22:50:143
127399118,2cyclictest10229-21expr22:05:173
127399118,1cyclictest14421-21apache_accesses23:15:123
127199118,2cyclictest7799-21ssh21:19:312
127199118,2cyclictest5970-21ssh22:01:502
127199118,2cyclictest3688-21grep00:15:202
127199118,2cyclictest22443-21cut00:05:192
127199118,2cyclictest21698-21expr22:35:162
127199118,2cyclictest13927-21wc20:40:232
127199118,2cyclictest10705-21sh22:27:322
127199117,3cyclictest16240-21rm21:25:122
127199115,2cyclictest224-21systemd-journal21:02:382
127199113,7cyclictest15957-21aten2_r7power_p00:25:152
127199113,2cyclictest224-21systemd-journal20:01:362
127199111,5cyclictest11244-21apt-get19:30:132
127199110,11cyclictest0-21swapper/223:00:462
126699118,2cyclictest9940-21cut00:20:191
126699118,2cyclictest9021-21ssh21:20:141
126699118,2cyclictest7204-21ssh23:31:501
126699118,2cyclictest6530-21ssh23:54:301
126699118,2cyclictest31322-21grep20:10:151
126699118,2cyclictest30283-21chrt20:07:141
126699118,2cyclictest28863-21ssh23:25:071
126699118,2cyclictest27753-21ssh23:00:541
126699118,2cyclictest22482-21ssh22:57:541
126699118,2cyclictest21955-21cron21:00:001
126699118,2cyclictest20304-21grep19:45:261
126699118,2cyclictest17448-21ssh00:01:211
126699118,2cyclictest16138-21seq21:46:261
126699118,2cyclictest14927-21cat19:35:211
126699118,2cyclictest12202-21taskset20:37:501
126699117,3cyclictest19331-21chrt20:51:491
126699114,3cyclictest224-21systemd-journal00:30:161
126699110,3cyclictest224-21systemd-journal19:23:411
126699110,10cyclictest7418-21seq20:27:321
126699110,10cyclictest13296-21ssh23:35:431
126699110,10cyclictest10278-21ssh23:11:351
125999119,1cyclictest6638-21tr19:20:150
125999119,1cyclictest288882sleep021:55:190
125999118,2cyclictest9836-21chrt21:42:170
125999118,2cyclictest753-21grep20:15:130
125999118,2cyclictest4702-21seq21:16:320
125999118,2cyclictest3998-21ssh23:07:260
125999118,2cyclictest3847-21ssh22:00:200
125999118,2cyclictest3593-21ssh23:30:020
125999118,2cyclictest29022-21sh00:35:160
125999118,2cyclictest26467-21ls20:00:150
125999118,2cyclictest25842-21ssh22:37:540
125999118,2cyclictest21420-21ssh00:29:260
125999118,2cyclictest21116-21ssh23:41:140
125999118,2cyclictest19655-21cat20:55:000
125999118,2cyclictest12954-21tr20:40:150
125999118,2cyclictest12730-21expr23:35:280
125999118,2cyclictest10323-21ssh00:20:210
125999117,3cyclictest29931-21sed22:40:200
125999117,3cyclictest13224-21ssh21:22:350
125999110,10cyclictest2563-21ssh22:21:430
127399108,1cyclictest8493-21sort20:30:173
127399108,1cyclictest50212sleep320:22:143
127399108,1cyclictest242492sleep321:03:413
127399108,1cyclictest21838-21forks19:50:173
127399108,1cyclictest19002-21expr19:45:153
127399108,1cyclictest17039-21seq20:47:553
127399108,1cyclictest1478-21taskset19:10:093
127399108,1cyclictest11415-21ntpq20:35:223
127399107,2cyclictest13208-21taskset19:33:193
127199108,1cyclictest4265-21cat19:15:162
127199108,1cyclictest19820-21chrt20:55:102
127199108,1cyclictest19102-21expr19:45:152
127199108,1cyclictest12452-21date20:40:002
127199107,2cyclictest31258-21df20:10:142
127199107,2cyclictest23604-21acpi19:55:132
127199107,2cyclictest17829-21expr20:50:162
127199107,2cyclictest15501-21ssh21:45:532
127199105,1cyclictest743-21cut20:15:132
126699108,1cyclictest49832sleep120:21:431
126699107,2cyclictest3055-21wc19:10:251
126699107,2cyclictest26328-21head21:05:281
126699107,2cyclictest23863-21sed21:00:271
125999108,1cyclictest32585-21chrt20:11:160
125999108,1cyclictest22210-21chrt21:00:050
125999108,1cyclictest20576-21expr20:55:180
125999108,1cyclictest1602-21grep19:10:140
125999107,2cyclictest5589-21aten2_r7power_p20:25:130
125999107,2cyclictest4712-21sed00:15:300
125999107,2cyclictest28510-21ssh00:10:010
125999107,2cyclictest18910-21cut19:45:140
125999107,2cyclictest15668-21grep20:45:160
125999107,2cyclictest11125-21ssh23:12:240
12739996,2cyclictest5762-21expr20:25:133
12599997,1cyclictest13888-21aten2_r7power_e19:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional