You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-11 - 02:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Nov 10, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6101210087,9sleep30-21swapper/307:08:543
404829986,9sleep10-21swapper/107:05:031
613328068,8sleep00-21swapper/007:09:200
601227868,6sleep20-21swapper/207:07:442
627899270,26cyclictest0-21swapper/211:06:472
6278992312,8cyclictest37850irq/131-snd_hda12:39:592
627399210,1cyclictest0-21swapper/109:25:201
627399200,1cyclictest0-21swapper/109:20:591
628099190,18cyclictest0-21swapper/309:55:283
626999195,8cyclictest23638-21/usr/sbin/munin09:35:130
6280991818,0cyclictest0-21swapper/311:45:113
627399181,16cyclictest1291-21runrttasks10:05:301
627899174,12cyclictest19065-21munin-node10:28:342
6278991717,0cyclictest0-21swapper/211:41:262
6278991716,1cyclictest0-21swapper/210:30:142
627399179,6cyclictest26301-21head07:50:201
628099167,8cyclictest7032-21apt-get12:30:173
628099164,10cyclictest0-21swapper/309:05:133
6280991615,1cyclictest0-21swapper/309:18:383
628099161,14cyclictest898-21apt-get10:10:153
6278991616,0cyclictest0-21swapper/212:29:582
6278991616,0cyclictest0-21swapper/210:03:402
6278991614,1cyclictest0-21swapper/209:50:322
6273991616,0cyclictest0-21swapper/110:17:311
627399160,10cyclictest0-21swapper/108:00:151
6280991514,1cyclictest0-21swapper/310:24:423
6280991510,4cyclictest12781-21expr12:10:183
6278991515,0cyclictest0-21swapper/211:03:012
627899150,4cyclictest0-21swapper/210:06:502
627399154,6cyclictest29254-21ls09:40:231
627399150,1cyclictest0-21swapper/110:20:211
627399150,15cyclictest0-21swapper/111:39:001
626999151,4cyclictest224-21systemd-journal12:00:000
626999150,1cyclictest0-21swapper/010:55:440
628099141,12cyclictest4796-21sensors11:35:233
6280991410,3cyclictest12879-21grep09:25:203
627899143,7cyclictest224-21systemd-journal11:38:502
6278991410,3cyclictest6994-21grep10:15:152
627899140,12cyclictest0-21swapper/212:21:372
627399148,4cyclictest0-21swapper/110:55:131
627399148,3cyclictest493-21systemd-logind09:54:221
6273991410,3cyclictest24831-21grep07:45:271
627399140,13cyclictest0-21swapper/110:29:401
627399140,13cyclictest0-21swapper/110:00:401
626999144,3cyclictest14995-21perf10:50:000
6269991410,3cyclictest3895-21cat09:45:210
626999140,1cyclictest0-21swapper/010:12:340
626999140,13cyclictest0-21swapper/012:30:160
628099139,3cyclictest9047-21wc10:45:003
628099139,3cyclictest19513-21cat09:30:253
628099133,6cyclictest26886-21cron11:00:013
628099132,10cyclictest28828-21ntp_kernel_pll_10:05:243
6280991310,2cyclictest31464-21tr10:35:253
6280991310,2cyclictest29801-21grep10:35:003
6280991310,2cyclictest25131-21cron12:20:003
6280991310,2cyclictest20057-21ntpq11:20:213
6280991310,2cyclictest19925-21grep10:25:273
6280991310,2cyclictest18913-21wc08:45:173
6280991310,2cyclictest16799-21tail07:30:173
6280991310,2cyclictest14870-21tr08:35:153
627899139,3cyclictest26708-21grep09:10:172
627899136,6cyclictest7744-21snmpd09:46:402
627899131,1cyclictest28933-21apache212:00:152
6278991310,2cyclictest8137-21ssh12:05:252
6278991310,2cyclictest7236-21grep11:10:152
6278991310,2cyclictest24406-21grep09:35:192
6278991310,2cyclictest21421-21grep07:40:212
627399139,3cyclictest22482-21grep08:55:271
627399135,5cyclictest20827-21apt-get07:40:161
627399135,4cyclictest224-21systemd-journal10:00:001
6273991310,2cyclictest7745-21sed11:10:181
6273991310,2cyclictest28350-21cat09:10:291
6273991310,2cyclictest2067-21grep12:25:281
6273991310,2cyclictest19921-21grep12:15:211
6273991310,2cyclictest17884-21tr09:30:131
6273991310,2cyclictest16297-21tr07:30:141
6273991310,2cyclictest15085-21wc12:35:211
6273991310,2cyclictest10750-21head11:40:281
626999139,3cyclictest12512-21cut08:30:160
626999136,3cyclictest224-21systemd-journal12:05:000
6269991310,2cyclictest6439-21wc08:15:230
6269991310,2cyclictest25120-21head10:30:190
6269991310,2cyclictest23976-21basename09:05:140
6269991310,2cyclictest21692-21grep11:50:190
6269991310,2cyclictest17120-21ssh12:13:570
6269991310,2cyclictest10664-21grep11:40:280
626999130,8cyclictest15799-21basename07:30:010
626999130,13cyclictest0-21swapper/011:35:530
628099129,3cyclictest0-21swapper/308:00:143
628099129,2cyclictest9925-21grep08:25:143
628099129,2cyclictest8924-21tr07:15:133
628099129,2cyclictest3428-21cut08:10:193
628099129,2cyclictest27647-21ssh09:10:213
628099129,2cyclictest24878-21head07:45:273
628099129,2cyclictest24505-21tr11:25:133
628099129,2cyclictest1716-21cat12:00:223
628099129,2cyclictest10685-21tail11:40:293
628099128,3cyclictest28705-21tail07:55:193
628099128,3cyclictest18487-21wc11:20:003
628099128,2cyclictest22417-21tr10:00:213
6280991210,2cyclictest0-21swapper/309:41:383
627899129,2cyclictest7194-21cut07:10:172
627899129,2cyclictest4883-21ps10:40:282
627899129,2cyclictest30926-21tail11:30:202
627899129,2cyclictest3047-21cron10:40:002
627899129,2cyclictest28675-21head07:55:192
627899129,2cyclictest27293-21grep11:55:172
627899129,2cyclictest18394-21head08:45:122
627899129,2cyclictest17710-21cron10:25:002
627899129,2cyclictest14413-21grep07:25:192
627899129,2cyclictest13488-21grep08:30:232
627899129,2cyclictest12068-21cut07:20:172
627899129,2cyclictest11280-21cron09:25:002
627899127,4cyclictest22824-21ssh10:55:262
627899127,4cyclictest11890-21chrt12:10:022
627899123,7cyclictest7781-21tr08:20:152
627899120,2cyclictest0-21swapper/209:26:382
627399129,2cyclictest9989-21sed07:15:231
627399129,2cyclictest5343-21run-parts09:20:001
627399129,2cyclictest31628-21ssh10:35:251
627399129,2cyclictest31428-21cut11:30:241
627399129,2cyclictest2810-21tail11:05:291
627399129,2cyclictest20661-21cat07:40:001
627399129,2cyclictest16766-21munin-run08:40:011
627399129,2cyclictest15240-21wc07:25:281
627399127,2cyclictest24088-21expr09:35:171
626999129,2cyclictest5357-21perf09:20:000
626999129,2cyclictest3948-21ls08:10:240
626999129,2cyclictest3594-21tr10:40:170
626999129,2cyclictest26996-21ssh12:20:240
626999129,2cyclictest243582sleep010:01:530
626999129,2cyclictest21974-21grep08:55:210
626999129,2cyclictest21588-21head07:40:200
626999129,2cyclictest18998-21grep10:25:170
626999129,2cyclictest18978-21grep12:15:160
626999129,2cyclictest11679-21sleep12:10:000
626999127,4cyclictest16591-21needreboot10:50:250
626999127,3cyclictest7256-21head07:10:170
626999122,4cyclictest16753-21idleruntime-cro08:40:010
6269991210,1cyclictest30558-21rm11:30:170
626999120,10cyclictest0-21swapper/011:17:310
628099118,2cyclictest9194-21ssh12:06:183
628099118,2cyclictest8973-21ssh10:15:383
628099118,2cyclictest6858-21ssh09:47:593
628099118,2cyclictest404-21cut12:25:163
628099118,2cyclictest3651-21rm11:06:213
628099118,2cyclictest29661-21ssh12:22:363
628099118,2cyclictest27490-21egrep11:00:143
628099118,2cyclictest26277-21tr07:50:213
628099118,2cyclictest24436-21sed09:35:183
628099118,2cyclictest23690-21ssh11:51:293
628099118,2cyclictest22666-21taskset08:56:033
628099118,2cyclictest22589-21chrt07:40:473
628099118,2cyclictest19177-21sed07:35:183
628099118,2cyclictest16900-21ssh12:36:193
628099118,2cyclictest1619-21ssh11:33:193
628099118,2cyclictest15642-21ssh10:50:203
628099118,2cyclictest12983-21head07:20:273
628099118,2cyclictest12594-21ssh09:52:163
628099118,2cyclictest11693-21sh11:13:593
628099117,3cyclictest7267-21sed07:10:173
628099110,10cyclictest8647-21ssh09:21:503
627899118,2cyclictest346-21grep08:05:142
627899118,2cyclictest29014-21grep09:40:212
627899118,2cyclictest25702-21cut07:50:172
627899118,2cyclictest24601-21grep09:05:172
627899118,2cyclictest23393-21taskset09:02:072
627899118,2cyclictest21807-21sort11:50:202
627899118,2cyclictest21580-21grep08:55:182
627899118,2cyclictest20423-21ssh12:15:272
627899118,2cyclictest20396-21taskset09:59:342
627899118,2cyclictest19432-21ssh11:20:162
627899118,2cyclictest16817-21grep11:45:292
627899118,2cyclictest10746-21cat08:25:192
627899118,1cyclictest20363-21sh09:31:072
627899117,3cyclictest26902-21chrt11:26:252
627899112,2cyclictest224-21systemd-journal09:20:002
627899111,2cyclictest224-21systemd-journal08:44:552
627899110,1cyclictest0-21swapper/208:35:192
627899110,11cyclictest0-21swapper/211:15:482
627899110,10cyclictest16858-21expr10:50:272
627399118,2cyclictest8213-21rm12:30:221
627399118,2cyclictest7925-21expr08:20:161
627399118,2cyclictest7830-21ssh12:05:211
627399118,2cyclictest754-21expr08:05:151
627399118,2cyclictest7180-21grep07:10:171
627399118,2cyclictest356-21sh12:00:041
627399118,2cyclictest3210-21expr09:45:161
627399118,2cyclictest29397-21ssh11:56:011
627399118,2cyclictest28851-21ssh11:29:001
627399118,2cyclictest27218-21ssh11:00:071
627399118,2cyclictest25558-21ntpq10:30:221
627399118,2cyclictest24637-21sshd11:52:391
627399118,2cyclictest19012-21ssh11:48:101
627399118,2cyclictest17420-21sleep08:45:001
627399118,2cyclictest16272-21sh11:17:121
627399118,2cyclictest11965-21/usr/sbin/munin12:10:271
627399118,2cyclictest1148-21expr10:10:171
627399118,2cyclictest10869-21ssh10:45:261
627399115,3cyclictest23899-21apt-get09:05:131
627399113,4cyclictest224-21systemd-journal08:30:001
627399112,6cyclictest18887-21tr08:45:171
627399110,10cyclictest29220-21pmu-power07:55:251
627399110,10cyclictest17982-21ssh10:51:191
626999119,2cyclictest13-21ksoftirqd/009:27:230
626999118,2cyclictest847-21cut12:25:180
626999118,2cyclictest7902-21cat10:15:200
626999118,2cyclictest32712-21ssh10:36:270
626999118,2cyclictest3159-21ssh11:05:450
626999118,2cyclictest31233-21rm09:41:050
626999118,2cyclictest29525-21cat07:55:290
626999118,2cyclictest27735-21cat07:55:000
626999118,2cyclictest26710-21tr09:10:160
626999118,2cyclictest26335-21ssh11:25:410
626999118,2cyclictest18431-21sed08:45:130
626999118,2cyclictest16451-21expr07:30:140
626999118,2cyclictest15780-21grep11:45:210
626999118,2cyclictest15721-21timerwakeupswit12:35:280
626999118,2cyclictest12984-21grep10:20:150
626999118,2cyclictest12290-21ssh09:51:560
626999118,2cyclictest11866-21taskset08:28:250
626999118,2cyclictest1054-21tail08:05:170
626999117,3cyclictest31175-21grep08:00:170
626999117,2cyclictest7021-21aten2_r7power_p11:10:150
626999115,5cyclictest29312-21ssh11:00:300
626999110,10cyclictest6034-21ls09:20:130
628099108,1cyclictest395-21basename08:05:143
628099108,1cyclictest26844-21ssh11:55:023
628099108,1cyclictest20186-21sed08:50:013
628099108,1cyclictest171942sleep308:42:343
628099108,1cyclictest154182sleep307:25:323
628099108,1cyclictest12472-21grep08:30:163
628099108,1cyclictest10123-21expr10:45:203
627899108,1cyclictest7050-21chrt08:17:522
627899108,1cyclictest4501-21taskset08:10:482
627899108,1cyclictest3027-21sh10:10:482
627899108,1cyclictest203052sleep208:51:142
627899108,1cyclictest181212sleep207:34:442
627899108,1cyclictest12701-21ssh10:47:152
627899108,1cyclictest10818-21taskset07:17:482
627899107,2cyclictest7108-21grep12:30:172
627899107,2cyclictest23461-21expr07:45:152
627899107,2cyclictest18911-21df07:35:172
627399108,1cyclictest6705-21cat08:15:261
627399108,1cyclictest3884-21cat08:10:241
627399108,1cyclictest3724-21egrep10:40:201
627399108,1cyclictest12570-21cat08:30:161
627399107,2cyclictest25586-21ssh12:20:121
627399107,2cyclictest23606-21chrt09:04:521
627399107,2cyclictest18691-21ssh11:20:041
627399107,2cyclictest13035-21taskset07:20:281
626999108,2cyclictest12643-21grep07:20:220
626999108,1cyclictest88462sleep007:15:040
626999108,1cyclictest8562-21cat08:20:210
626999108,1cyclictest24311-21ntp_kernel_err07:45:210
626999108,1cyclictest20352-21taskset08:51:490
626999108,1cyclictest19844-21rm09:30:330
626999108,1cyclictest19037-21grep11:20:140
626999107,2cyclictest27477-21ssh10:05:040
626999107,2cyclictest16816-21ssh09:55:260
62809997,1cyclictest94222sleep308:22:343
62809997,1cyclictest5772-21tune2fs08:15:173
62809993,2cyclictest493-21systemd-logind09:01:213
62789997,1cyclictest31964-21timerwakeupswit08:00:282
62739997,1cyclictest20319-21taskset08:51:241
62699997,1cyclictest23428-21chrt09:02:330
62699996,2cyclictest18666-21expr07:35:160
62699986,1cyclictest170322sleep008:40:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional