You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 14:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Nov 28, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11703210664,38sleep30-21swapper/319:09:143
11732210187,9sleep10-21swapper/119:09:311
1108529573,17sleep00-21swapper/019:05:280
1174328764,18sleep20-21swapper/219:09:412
1185899320,32cyclictest0-21swapper/300:03:083
1185899310,0cyclictest0-21swapper/323:01:073
11850992726,1cyclictest0-21swapper/121:52:561
1185899240,1cyclictest0-21swapper/322:31:163
1185499220,18cyclictest0-21swapper/223:38:052
1185499210,1cyclictest0-21swapper/223:12:502
1185499210,0cyclictest0-21swapper/221:44:232
11850992121,0cyclictest0-21swapper/100:24:411
1185099210,21cyclictest0-21swapper/121:59:371
1184499218,4cyclictest0-21swapper/020:35:140
1184499210,14cyclictest0-21swapper/023:11:170
11854992018,2cyclictest32-21ksoftirqd/219:55:142
1185499201,14cyclictest0-21swapper/219:20:172
11850992016,3cyclictest25-21ksoftirqd/121:45:201
1185899190,0cyclictest0-21swapper/300:05:133
1185499190,18cyclictest0-21swapper/222:10:142
11844991917,2cyclictest13-21ksoftirqd/023:20:480
115722190,0sleep00-21swapper/021:36:190
1185899180,0cyclictest0-21swapper/322:09:593
1185499180,1cyclictest0-21swapper/200:33:032
1185499180,0cyclictest0-21swapper/223:30:042
11844991813,4cyclictest17380-21apache222:14:240
11854991712,4cyclictest28671-21tr23:40:152
1185499170,17cyclictest0-21swapper/222:41:522
1185499170,16cyclictest0-21swapper/223:05:112
11850991717,0cyclictest0-21swapper/121:40:201
1185099170,0cyclictest0-21swapper/122:35:511
1185499166,4cyclictest224-21systemd-journal20:10:152
1185499160,1cyclictest0-21swapper/200:35:202
1185499160,16cyclictest0-21swapper/222:18:102
1185499160,13cyclictest0-21swapper/221:50:232
1185499160,0cyclictest0-21swapper/200:14:442
1185099162,11cyclictest4908-21diskmemload00:00:051
1184499167,4cyclictest224-21systemd-journal23:55:000
11844991615,1cyclictest0-21swapper/023:33:400
1185899153,10cyclictest0-21swapper/300:38:363
11858991513,1cyclictest4908-21diskmemload22:02:183
1185899151,2cyclictest0-21swapper/323:38:473
1185499151,14cyclictest0-21swapper/222:35:142
1185499150,14cyclictest4908-21diskmemload22:49:182
1185499150,14cyclictest0-21swapper/222:27:362
1185899142,11cyclictest0-21swapper/322:50:133
11858991413,1cyclictest0-21swapper/322:10:323
11858991411,2cyclictest1122-21head21:00:223
11858991410,3cyclictest9496-21grep23:49:593
1185899140,1cyclictest0-21swapper/321:13:033
1185899140,0cyclictest0-21swapper/321:36:303
1185499148,5cyclictest20755-21grep20:35:002
1185499144,6cyclictest30022-21awk21:50:002
1185499140,4cyclictest0-21swapper/221:27:222
1185499140,4cyclictest0-21swapper/221:27:222
1185499140,2cyclictest0-21swapper/200:16:122
1185499140,14cyclictest0-21swapper/222:32:072
1185499140,14cyclictest0-21swapper/200:05:582
1185499140,11cyclictest0-21swapper/200:21:322
1185099149,4cyclictest20681-21wc22:50:001
11850991414,0cyclictest0-21swapper/123:46:011
11850991410,2cyclictest10769-21grep22:20:171
1184499140,4cyclictest0-21swapper/021:41:110
1184499140,1cyclictest0-21swapper/000:29:160
1185899139,3cyclictest2368-21grep23:00:003
11858991311,1cyclictest0-21swapper/321:20:113
11858991310,2cyclictest478-21cut19:50:223
11858991310,2cyclictest433-21sed23:20:183
11858991310,2cyclictest24534-21tr19:35:163
11858991310,2cyclictest22821-21basename21:45:013
11858991310,2cyclictest17032-21cat23:10:013
11858991310,2cyclictest16726-21cut23:55:143
11858991310,2cyclictest12765-21sort19:10:183
11858991310,2cyclictest12298-21sort20:15:203
1185899130,1cyclictest0-21swapper/323:50:373
1185899130,1cyclictest0-21swapper/323:44:083
1185499138,4cyclictest25415-21sh20:45:002
11854991311,1cyclictest0-21swapper/220:55:142
1185499130,1cyclictest0-21swapper/219:16:212
1185499130,13cyclictest0-21swapper/223:53:022
1185499130,12cyclictest21534-21cut20:35:172
1185099139,3cyclictest21707-21chrt20:35:181
1185099139,3cyclictest10713-21grep23:50:201
1185099132,6cyclictest1291-21runrttasks23:05:001
11850991313,0cyclictest0-21swapper/123:21:051
11850991310,2cyclictest3580-21grep00:35:161
11850991310,2cyclictest3452-21grep19:55:261
11850991310,2cyclictest25631-21sed23:15:191
11850991310,2cyclictest2208-21grep21:30:181
11850991310,2cyclictest2072-21grep21:00:321
11850991310,2cyclictest16343-21grep20:25:161
11850991310,2cyclictest12857-21tail00:15:281
1185099130,1cyclictest0-21swapper/123:34:451
1185099130,0cyclictest0-21swapper/121:14:581
1184499138,4cyclictest28668-21wget23:40:140
1184499135,2cyclictest23090-21perf00:00:000
1184499132,8cyclictest11692-21timerandwakeup22:20:260
11844991312,1cyclictest13-21ksoftirqd/023:35:200
11844991310,2cyclictest6415-21head21:55:180
11844991310,2cyclictest29525-21apt00:05:000
11844991310,2cyclictest19025-21chrt20:30:160
11844991310,2cyclictest147612sleep023:07:560
1184499130,12cyclictest0-21swapper/022:18:090
1185899129,2cyclictest9710-21sed20:10:183
1185899129,2cyclictest7311-21head20:05:183
1185899129,2cyclictest6048-21ps00:10:233
1185899129,2cyclictest29600-21grep19:45:133
1185899129,2cyclictest28988-21apt19:45:003
1185899129,2cyclictest2886-21grep19:55:193
1185899129,2cyclictest16923-21wc22:25:003
1185899129,2cyclictest16084-21grep19:15:273
1185899129,2cyclictest12136-21cat00:15:223
1185899128,3cyclictest9302-21cron22:20:003
1185899128,3cyclictest18119-21grep22:25:183
1185899121,11cyclictest0-21swapper/321:50:163
1185499129,2cyclictest7118-21tr23:25:162
1185499129,2cyclictest5734-21wc20:00:262
1185499129,2cyclictest5255-21sh21:10:072
1185499129,2cyclictest2522-21sh21:05:002
1185499129,2cyclictest20323-21sed21:20:182
1185499128,4cyclictest24307-21expr00:25:182
1185499128,3cyclictest12120-21ssh21:15:072
1185499128,3cyclictest0-21swapper/223:16:312
1185499121,8cyclictest15833-21cron20:24:592
1185499120,11cyclictest3133-21cat21:30:282
1185499120,11cyclictest1916-21ssh23:20:482
1185099129,2cyclictest7477-21head22:40:181
1185099129,2cyclictest32159-21grep20:55:271
1185099129,2cyclictest31146-21grep19:45:251
1185099129,2cyclictest30765-21head00:05:161
1185099129,2cyclictest29677-21grep20:50:291
1185099129,2cyclictest28871-21grep22:55:181
1185099129,2cyclictest2595-21basename23:44:591
1185099129,2cyclictest24309-21cron23:15:001
1185099129,2cyclictest24031-21grep20:40:171
1185099129,2cyclictest20154-21sed20:30:261
1185099129,2cyclictest14720-21cat20:20:201
1185099129,2cyclictest10638-21head20:10:271
1185099129,2cyclictest0-21swapper/122:10:131
1185099127,3cyclictest12595-21wc20:15:231
1185099120,10cyclictest17637-21gltestperf19:20:201
1184499129,2cyclictest8581-21cut00:40:000
1184499129,2cyclictest7727-21cut22:40:200
1184499129,2cyclictest31481-21grep00:30:310
1184499129,2cyclictest28847-21tail20:50:190
1184499129,2cyclictest24651-21rm22:30:070
1184499129,2cyclictest24270-21cat20:40:190
1184499129,2cyclictest22461-21head19:30:170
1184499129,2cyclictest22461-21head19:30:170
1184499129,2cyclictest20007-21grep19:25:180
1184499129,2cyclictest16111-21rm22:01:270
1184499129,2cyclictest13040-21gltestperf21:15:170
1184499129,2cyclictest12781-21tune2fs19:10:180
1184499121,4cyclictest0-21swapper/022:35:140
1184499121,2cyclictest0-21swapper/019:20:170
11844991210,1cyclictest9215-21tr20:10:150
11844991210,1cyclictest6442-21seq00:10:270
11844991210,1cyclictest21245-21fw_packets22:05:190
1184499120,11cyclictest6854-21tr20:05:160
1184499120,11cyclictest6583-21ssh23:02:070
1184499120,11cyclictest28250-21ssh21:25:290
1184499120,11cyclictest28250-21ssh21:25:290
1184499120,11cyclictest0-21swapper/022:26:280
185272111,1sleep118587-21ntp_states22:25:211
1185899118,2cyclictest9686-21ssh21:57:293
1185899118,2cyclictest8058-21grep22:40:223
1185899118,2cyclictest6014-21taskset20:01:293
1185899118,2cyclictest4616-21taskset21:06:293
1185899118,2cyclictest3935-21ssh21:31:003
1185899118,2cyclictest370-21ssh00:32:073
1185899118,2cyclictest30736-21ssh21:27:323
1185899118,2cyclictest30736-21ssh21:27:323
1185899118,2cyclictest30203-21sh23:18:443
1185899118,2cyclictest29997-21taskset20:52:523
1185899118,2cyclictest27914-21ssh00:28:213
1185899118,2cyclictest26062-21ssh21:46:153
1185899118,2cyclictest24434-21tr20:40:213
1185899118,2cyclictest21225-21ssh23:12:083
1185899118,2cyclictest19990-21ssh23:33:543
1185899118,2cyclictest19515-21ssh22:48:563
1185899118,2cyclictest19312-21head20:30:183
1185899118,2cyclictest17099-21grep19:20:183
1185899118,2cyclictest14590-21cat20:20:203
1185899118,2cyclictest13269-21sed21:15:193
1185899118,2cyclictest11304-21ssh23:27:443
1185899115,4cyclictest32221-21seq20:55:553
1185899114,3cyclictest0-21swapper/319:30:223
1185899114,3cyclictest0-21swapper/319:30:223
1185899110,10cyclictest18713-21sh00:20:233
1185899110,10cyclictest1786-21ssh22:35:453
1185499119,1cyclictest8581-21taskset20:08:532
1185499118,3cyclictest3796-21grep23:45:192
1185499118,2cyclictest3128-21sh23:00:142
1185499118,2cyclictest28773-21grep20:50:202
1185499118,2cyclictest27978-21grep19:40:192
1185499118,2cyclictest25924-21tr20:45:162
1185499118,2cyclictest25293-21grep19:35:212
1185499118,2cyclictest24537-21if_err_enp0s31f00:00:222
1185499118,2cyclictest23671-21ssh22:51:072
1185499118,2cyclictest22388-21head22:05:302
1185499118,2cyclictest17988-21tr23:55:202
1185499118,2cyclictest11556-21tr20:15:152
1185499118,2cyclictest10264-21chrt21:35:252
1185499117,3cyclictest16306-21sh22:01:382
1185499116,4cyclictest29550-21sendmail_mailst22:55:232
1185499113,4cyclictest224-21systemd-journal22:20:212
1185499110,10cyclictest8623-21sh21:56:282
1185499110,10cyclictest29550-21cut19:45:132
1185099119,1cyclictest5466-21cut00:10:181
1185099118,2cyclictest8901-21egrep21:35:171
1185099118,2cyclictest6628-21ssh23:25:021
1185099118,2cyclictest31386-21chrt21:28:111
1185099118,2cyclictest31386-21chrt21:28:101
1185099118,2cyclictest30493-21head00:30:221
1185099118,2cyclictest27579-21ssh22:55:001
1185099118,2cyclictest25906-21ssh22:30:211
1185099118,2cyclictest24030-21grep00:25:171
1185099118,2cyclictest23939-21ssh23:35:501
1185099118,2cyclictest21070-21grep22:05:181
1185099118,2cyclictest19281-21ssh21:20:051
1185099118,2cyclictest17686-21ssh22:02:521
1185099118,2cyclictest17537-21expr23:55:161
1185099118,2cyclictest1484-21taskset19:54:351
1185099118,2cyclictest13239-21interrupts21:15:181
1185099118,2cyclictest10326-21grep23:05:151
11850991111,0cyclictest0-21swapper/122:16:151
1184499119,2cyclictest7890-21ssh23:48:220
1184499119,2cyclictest30798-21readlink19:45:210
1184499119,2cyclictest14168-21expr20:20:170
1184499119,1cyclictest3245-21ls19:55:250
1184499119,1cyclictest27987-21ssh23:16:380
1184499119,1cyclictest27950-21ssh22:55:080
1184499119,1cyclictest27301-21expr19:40:140
1184499119,1cyclictest24677-21expr19:35:160
1184499119,1cyclictest1756-21expr21:30:150
1184499119,1cyclictest11483-21expr00:15:170
1184499118,2cyclictest32055-21cat19:50:160
1184499118,2cyclictest30764-21tail00:05:170
1184499118,2cyclictest30697-21cat21:50:140
1184499118,2cyclictest24054-21grep21:45:180
1184499118,2cyclictest20985-21ssh22:50:110
1184499118,2cyclictest17371-21ssh00:20:090
1184499114,6cyclictest1316-21/usr/sbin/munin21:00:240
1184499111,9cyclictest0-21swapper/023:25:040
1184499111,10cyclictest14990-21/usr/sbin/munin22:45:210
1184499110,4cyclictest224-21systemd-journal19:19:590
1184499110,10cyclictest19313-21ssh21:20:070
1185899108,1cyclictest21548-21cut20:35:173
1185899108,1cyclictest172332sleep320:25:223
1185899107,2cyclictest26623-21uname20:45:203
1185899107,2cyclictest19618-21expr19:25:163
1185499108,1cyclictest16184-21aten2_r7power_p20:25:162
1185499107,2cyclictest3547-21cut21:05:192
1185499107,2cyclictest22512-21cat19:30:182
1185499107,2cyclictest22512-21cat19:30:182
1185499107,2cyclictest12497-21expr19:10:162
1185499102,4cyclictest224-21systemd-journal19:26:232
1185099108,1cyclictest287422sleep119:42:041
1185099108,1cyclictest2823-21hostname21:05:161
1185099108,1cyclictest26185-21taskset19:37:421
1185099108,1cyclictest236582sleep119:33:311
1185099108,1cyclictest236582sleep119:33:301
1185099107,2cyclictest4911-21grep20:00:171
1185099107,2cyclictest14444-21cat19:15:111
1185099107,2cyclictest13011-21uname19:10:201
1184499109,1cyclictest4301-21aten2_r7power_v20:00:140
1184499108,1cyclictest5252-21ssh21:10:070
1184499108,1cyclictest30498-21basename20:55:140
1184499108,1cyclictest26167-21expr20:45:160
1184499108,1cyclictest16199-21basename20:25:160
1184499107,2cyclictest3093-21seq21:05:170
1184499107,2cyclictest13415-21taskset20:19:470
118549997,1cyclictest11872sleep219:50:552
118509997,1cyclictest6538-21taskset20:05:111
118509997,1cyclictest19304-21apt-get19:25:131
118509985,2cyclictest26926-21grep20:45:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional