You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 14:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Nov 20, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8589210086,9sleep10-21swapper/119:08:081
872129985,9sleep30-21swapper/319:09:553
852429383,6sleep20-21swapper/219:07:172
744728066,9sleep00-21swapper/019:05:210
8805992813,3cyclictest13-21ksoftirqd/023:56:420
8805992622,1cyclictest13-21ksoftirqd/023:20:500
8805992621,1cyclictest13-21ksoftirqd/000:29:030
8805992621,1cyclictest13-21ksoftirqd/000:29:020
8818992322,1cyclictest0-21swapper/321:26:563
881899223,17cyclictest19092-21diskmemload00:08:253
881899222,19cyclictest17868-21latency_hist19:59:593
881899221,15cyclictest0-21swapper/322:20:193
8818992120,1cyclictest0-21swapper/322:10:203
880599212,14cyclictest13-21ksoftirqd/022:25:240
880599210,5cyclictest161rcuc/022:35:140
880599207,5cyclictest13-21ksoftirqd/021:00:170
8805992017,2cyclictest13-21ksoftirqd/023:50:160
880599200,19cyclictest0-21swapper/000:32:530
208992200,0sleep30-21swapper/321:35:033
881899190,18cyclictest0-21swapper/323:52:593
8814991917,1cyclictest8489-21fschecks_count21:25:202
8814991915,2cyclictest141rcu_preempt00:18:342
880599197,8cyclictest13-21ksoftirqd/023:07:060
8805991914,2cyclictest13-21ksoftirqd/023:45:240
880599190,1cyclictest0-21swapper/021:15:120
880599190,16cyclictest13-21ksoftirqd/023:40:150
881899181,2cyclictest0-21swapper/320:30:183
881899181,14cyclictest0-21swapper/320:25:143
881499187,10cyclictest32-21ksoftirqd/221:42:202
880999188,9cyclictest24987-21apache222:56:491
880599189,1cyclictest141rcu_preempt21:23:130
880599181,5cyclictest13-21ksoftirqd/022:15:170
880599180,1cyclictest0-21swapper/022:33:510
8818991715,1cyclictest0-21swapper/321:30:123
881499172,2cyclictest0-21swapper/222:19:302
8809991716,1cyclictest0-21swapper/100:27:191
8809991716,1cyclictest0-21swapper/100:27:181
880599176,7cyclictest141rcu_preempt00:20:010
880599171,6cyclictest13-21ksoftirqd/021:28:540
8805991713,2cyclictest1937-21ssh00:02:360
880599171,2cyclictest141rcu_preempt23:39:200
8814991614,1cyclictest0-21swapper/223:22:082
881499160,15cyclictest0-21swapper/223:01:082
8809991615,1cyclictest0-21swapper/100:15:091
880599161,1cyclictest141rcu_preempt22:20:250
8818991513,1cyclictest16536-21sh00:12:313
881499153,4cyclictest8803-21cyclictest22:59:022
881499151,2cyclictest141rcu_preempt21:10:222
881499151,2cyclictest141rcu_preempt21:10:212
880599159,5cyclictest13-21ksoftirqd/000:10:170
880599154,9cyclictest14360-21apt-get21:30:120
880599153,3cyclictest141rcu_preempt21:35:140
880599152,6cyclictest11268-21munin-run19:30:010
880599151,7cyclictest13-21ksoftirqd/021:10:230
880599151,7cyclictest13-21ksoftirqd/021:10:230
880599150,0cyclictest0-21swapper/022:02:340
881899145,3cyclictest224-21systemd-journal22:07:203
881899143,10cyclictest0-21swapper/322:29:203
8818991411,2cyclictest987-21cat21:20:093
8818991410,3cyclictest13213-21grep00:35:243
881899140,14cyclictest0-21swapper/321:56:163
8814991411,2cyclictest19091-21turbostat21:10:002
881499140,6cyclictest0-21swapper/223:35:112
881499140,12cyclictest0-21swapper/222:06:232
881499140,10cyclictest0-21swapper/222:38:002
880999141,12cyclictest0-21swapper/122:05:131
880599145,4cyclictest141rcu_preempt23:16:400
880599143,1cyclictest141rcu_preempt23:26:220
8805991411,2cyclictest13-21ksoftirqd/022:48:520
8805991411,2cyclictest13-21ksoftirqd/020:40:010
8805991411,2cyclictest13-21ksoftirqd/000:20:230
880599140,2cyclictest0-21swapper/020:05:110
881899139,3cyclictest8699-21wc20:45:213
881899139,3cyclictest23241-21grep23:55:143
881899139,3cyclictest22871-21tr22:00:173
881899139,3cyclictest13622-21tr23:25:243
8818991311,1cyclictest0-21swapper/323:49:393
8818991310,2cyclictest29067-21head20:20:223
8818991310,2cyclictest22387-21grep22:45:273
8818991310,2cyclictest22049-21wc20:05:253
8818991310,2cyclictest19910-21tr21:10:173
8818991310,2cyclictest19910-21tr21:10:163
8818991310,2cyclictest15158-21turbostat.cron21:55:003
8818991310,2cyclictest1431-21grep23:40:113
8818991310,2cyclictest12564-21grep22:15:273
881899130,13cyclictest0-21swapper/322:52:173
881499136,3cyclictest32-21ksoftirqd/221:48:582
881499133,1cyclictest32-21ksoftirqd/222:25:052
8814991312,1cyclictest17693-21kworker/2:120:25:212
8814991312,1cyclictest0-21swapper/223:15:142
8814991311,1cyclictest37850irq/131-snd_hda23:56:132
881499131,10cyclictest1158-21apt-get23:40:092
8814991310,2cyclictest31918-21grep00:00:262
881499130,1cyclictest0-21swapper/221:50:162
881499130,13cyclictest0-21swapper/222:03:092
881499130,0cyclictest141rcu_preempt20:40:212
880999139,2cyclictest0-21swapper/122:45:141
880999132,5cyclictest224-21systemd-journal23:34:591
8809991310,2cyclictest515-21cat22:30:211
8809991310,2cyclictest32435-21cron20:29:591
8809991310,2cyclictest28794-21ssh21:40:161
8809991310,2cyclictest21646-21uname20:05:201
880999130,13cyclictest0-21swapper/122:17:161
880999130,12cyclictest18967-21tr22:20:201
880599138,3cyclictest13-21ksoftirqd/022:44:320
880599138,3cyclictest13-21ksoftirqd/022:44:320
880599136,2cyclictest13-21ksoftirqd/022:12:000
880599133,8cyclictest0-21swapper/022:50:120
880599133,2cyclictest224-21systemd-journal20:42:190
8805991310,2cyclictest30255-21cron23:15:000
8805991310,2cyclictest25485-21cut20:15:170
8805991310,2cyclictest2393-21cron20:34:590
8805991310,2cyclictest17294-21wc21:55:260
880599130,12cyclictest10861-21wc20:50:210
881899129,2cyclictest800-21grep00:25:263
881899129,2cyclictest800-21grep00:25:263
881899129,2cyclictest30250-21ssh23:14:593
881899129,2cyclictest28675-21grep21:40:153
881899129,2cyclictest18190-21ls23:05:223
881899129,2cyclictest15312-21grep21:00:183
881899129,2cyclictest10327-21ssh23:00:163
881899128,3cyclictest6212-21apt-config22:35:003
881899127,3cyclictest929-21wc23:40:013
881899122,6cyclictest224-21systemd-journal21:17:003
881899120,11cyclictest19394-21head20:00:333
881899120,11cyclictest17219-21tr21:05:153
881499129,2cyclictest9628-31su00:07:582
881499129,2cyclictest6644-21tr00:30:242
881499129,2cyclictest29329-21cat22:50:242
881499129,2cyclictest177552sleep221:31:572
881499129,2cyclictest17048-21wc19:55:392
881499128,4cyclictest24987-21apache221:36:582
881499128,3cyclictest3895-21grep20:35:212
881499128,3cyclictest11890-21sleep19:35:002
881499128,1cyclictest22182-21ssh23:08:532
881499122,8cyclictest37850irq/131-snd_hda21:20:012
8814991211,1cyclictest25666-21awk23:10:252
8814991211,1cyclictest19092-21diskmemload00:36:012
881499120,12cyclictest0-21swapper/222:32:132
880999129,2cyclictest9995-21grep23:00:151
880999129,2cyclictest6595-21tr00:30:251
880999129,2cyclictest5848-21head20:40:181
880999129,2cyclictest21344-21grep21:35:151
880999129,2cyclictest16827-21tr21:55:201
880999129,2cyclictest1517-21sed20:30:251
880999129,2cyclictest13020-21df00:10:161
880999129,2cyclictest11281-21basename19:30:011
880999129,2cyclictest10763-21chrt22:37:431
880999128,2cyclictest16265-21expr23:50:181
8809991210,1cyclictest8569-21grep21:25:211
8809991210,1cyclictest15320-21cat23:50:011
880999120,1cyclictest0-21swapper/122:10:511
880999120,11cyclictest0-21swapper/122:42:511
880999120,11cyclictest0-21swapper/122:42:501
880599129,2cyclictest9737-21sed21:50:180
880599129,2cyclictest9736-21ssh23:00:140
880599129,2cyclictest30616-21tr20:25:160
880599129,2cyclictest13500-21uname20:55:200
880599129,2cyclictest10644-21cron19:25:000
880599126,4cyclictest13-21ksoftirqd/000:05:320
880599125,2cyclictest13-21ksoftirqd/021:45:510
880599122,1cyclictest141rcu_preempt22:58:480
880599121,2cyclictest141rcu_preempt23:32:500
8805991211,1cyclictest18040-21ls21:05:210
138242120,1sleep10-21swapper/121:53:361
881899118,2cyclictest8171-21ssh23:21:243
881899118,2cyclictest5725-21tr00:30:183
881899118,2cyclictest5344-21ssh21:46:423
881899118,2cyclictest25960-21grep00:20:163
881899118,2cyclictest2580-21ssh22:55:133
881899118,2cyclictest2457-21sh23:17:413
881899118,2cyclictest23063-21chrt23:32:193
881899118,2cyclictest2010-21ssh22:31:023
881899118,2cyclictest14088-21ls22:40:193
881899118,2cyclictest14088-21ls22:40:183
881899118,2cyclictest13682-21chrt19:49:283
881899118,2cyclictest12956-21cut20:55:143
881899118,2cyclictest11546-21chrt19:30:493
881899118,2cyclictest10096-21cat19:20:003
881899117,3cyclictest301132sleep300:00:133
881899115,3cyclictest224-21systemd-journal19:53:143
881899110,10cyclictest5382-21wc20:40:173
881499119,1cyclictest9404-21cut23:45:232
881499119,1cyclictest20548-21grep23:30:222
881499119,1cyclictest12237-21ssh00:10:032
881499118,2cyclictest4644-21head22:10:152
881499118,2cyclictest3516-21sh21:21:002
881499118,2cyclictest32119-21ssh00:25:182
881499118,2cyclictest32119-21ssh00:25:172
881499118,2cyclictest24568-21sed20:10:302
881499118,2cyclictest21838-21head22:45:222
881499118,2cyclictest18902-21head20:00:192
881499118,2cyclictest18892-21cat22:20:202
881499118,2cyclictest17316-21sendmail_mailtr23:50:252
881499118,2cyclictest13081-21grep23:25:202
881499116,1cyclictest141rcu_preempt22:40:212
881499116,1cyclictest141rcu_preempt22:40:202
881499115,1cyclictest13750-21munin-run19:50:002
881499110,1cyclictest0-21swapper/200:20:262
880999119,1cyclictest26700-21grep22:25:251
880999119,1cyclictest17890-21sh20:00:001
880999118,2cyclictest9434-21wc19:15:001
880999118,2cyclictest6983-21ssh21:48:261
880999118,2cyclictest31683-21sh23:38:371
880999118,2cyclictest31551-21grep23:15:181
880999118,2cyclictest31064-21ssh00:00:181
880999118,2cyclictest30068-21latency_hist20:24:591
880999118,2cyclictest29499-21ssh21:16:261
880999118,2cyclictest25165-21ssh00:20:101
880999118,2cyclictest2404-21wc21:20:201
880999118,2cyclictest22679-21rm23:55:011
880999118,2cyclictest21557-21ssh21:10:291
880999118,2cyclictest21557-21ssh21:10:291
880999118,2cyclictest21368-21ssh23:08:041
880999118,2cyclictest18317-21chrt21:32:321
880999118,2cyclictest17390-21expr21:05:161
880999118,2cyclictest12269-21grep20:55:121
880999118,2cyclictest12053-21ssh23:25:031
880999118,2cyclictest11837-21ssh00:35:121
880999118,2cyclictest10297-21expr20:50:171
880999115,2cyclictest224-21systemd-journal20:02:171
880999114,3cyclictest30266-21awk23:14:591
880999113,4cyclictest224-21systemd-journal00:10:011
880999110,10cyclictest31251-21chrt22:51:541
880599119,1cyclictest16382-21sort19:55:130
880599118,2cyclictest30335-21ssh22:05:170
880599118,2cyclictest28607-21cut21:40:150
880599118,2cyclictest23639-21grep20:10:190
880599118,2cyclictest18894-21grep20:00:190
880599118,2cyclictest13429-21taskset19:46:080
880599118,2cyclictest12954-21chrt19:42:560
880599117,3cyclictest30082-21basename20:24:590
880599115,1cyclictest141rcu_preempt19:51:480
880599111,4cyclictest141rcu_preempt20:49:210
880599111,1cyclictest141rcu_preempt19:14:580
880599110,10cyclictest13125-21ntp_states00:35:220
195692110,7sleep20-21swapper/221:57:302
881899108,1cyclictest4504-21chrt20:36:363
881899108,1cyclictest19767-21taskset00:15:163
881899108,1cyclictest121362sleep319:35:293
881899107,2cyclictest9515-21sort19:15:003
881899107,2cyclictest26128-21sed20:15:183
881899107,2cyclictest11415-21sed20:50:293
881899106,3cyclictest13110-21cron19:45:013
881499108,2cyclictest25774-21ls20:15:162
881499108,2cyclictest1420-21grep20:30:242
881499108,2cyclictest10613-21taskset19:24:422
881499108,1cyclictest91432sleep219:11:482
881499108,1cyclictest30139-21wc20:25:012
881499108,1cyclictest11970-21ls20:55:002
881499107,2cyclictest12481-21grep20:55:122
881499102,4cyclictest6999-21kworker/2:020:45:122
880999108,1cyclictest9771-21cat20:50:011
880999108,1cyclictest6466-21ntpq23:20:191
880999108,1cyclictest4020-21chrt23:41:151
880999108,1cyclictest26712-21ssh22:02:391
880999108,1cyclictest10103-21date19:20:011
880999107,2cyclictest23626-21grep20:10:191
880999107,2cyclictest15732-21tr21:00:201
880999107,2cyclictest14134-21hostname19:50:381
880999107,2cyclictest12324-21taskset19:38:001
880999100,9cyclictest11826-21seq19:34:301
880599108,1cyclictest96692sleep019:15:360
880599108,1cyclictest123042sleep019:37:440
880599108,1cyclictest115882sleep019:31:220
88189997,1cyclictest110482sleep319:27:203
88189991,4cyclictest23210-21cstates20:10:183
88149998,1cyclictest124012sleep219:39:022
88149997,1cyclictest150992sleep219:52:312
88149996,2cyclictest15379-21tail21:00:172
88149991,2cyclictest224-21systemd-journal19:16:192
88149991,1cyclictest224-21systemd-journal20:09:542
88099998,1cyclictest27152-21taskset20:16:131
88099998,1cyclictest10517-21taskset19:23:241
88099998,1cyclictest0-21swapper/119:45:171
88099997,1cyclictest2746-21awk20:35:151
88149987,1cyclictest13137-21perf19:45:002
88149981,4cyclictest0-21swapper/219:26:202
88099987,1cyclictest13001-21taskset19:43:341
88189972,2cyclictest224-21systemd-journal19:21:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional