You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 20:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Nov 15, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
670210779,9sleep10-21swapper/107:07:561
760210679,9sleep30-21swapper/307:09:063
59228164,13sleep20-21swapper/207:07:002
55228068,8sleep00-21swapper/007:06:310
172332580,0sleep00-21swapper/009:57:310
91399321,30cyclictest22872-21if_err_enp2s010:55:200
91399312,24cyclictest25217-21/usr/sbin/munin12:15:130
91399310,30cyclictest0-21swapper/007:40:150
92199300,1cyclictest0-21swapper/307:40:163
92199290,2cyclictest0-21swapper/312:05:143
91899291,25cyclictest0-21swapper/209:27:442
91899290,26cyclictest32-21ksoftirqd/212:07:272
91899290,25cyclictest32-21ksoftirqd/210:50:172
91399291,27cyclictest25990-21diskmemload11:30:190
92199280,3cyclictest0-21swapper/311:15:263
92199280,1cyclictest0-21swapper/309:47:543
91899280,25cyclictest32-21ksoftirqd/212:27:032
91899280,25cyclictest32-21ksoftirqd/212:12:162
91899280,25cyclictest32-21ksoftirqd/211:10:132
91899280,25cyclictest32-21ksoftirqd/211:01:562
91899280,25cyclictest32-21ksoftirqd/209:43:332
1200922825,1sleep339-21ksoftirqd/309:53:433
92199272,24cyclictest25815-21sed12:15:173
92199272,24cyclictest0-21swapper/311:00:163
91899271,24cyclictest32-21ksoftirqd/210:39:222
91899270,24cyclictest32-21ksoftirqd/212:36:272
91899270,24cyclictest0-21swapper/211:20:092
91399271,22cyclictest25990-21diskmemload11:23:100
91399270,1cyclictest0-21swapper/012:20:140
92199263,22cyclictest0-21swapper/308:15:143
92199260,2cyclictest0-21swapper/310:19:333
91899262,1cyclictest37850irq/131-snd_hda10:07:042
91899260,24cyclictest0-21swapper/209:16:302
92199253,21cyclictest25990-21diskmemload09:21:443
92199253,20cyclictest5589-21apt11:35:003
92199252,2cyclictest23511-21cut07:55:193
92199250,2cyclictest0-21swapper/312:13:453
918992521,4cyclictest32-21ksoftirqd/212:15:152
91899251,23cyclictest0-21swapper/212:23:442
91399250,24cyclictest0-21swapper/011:15:270
91399250,24cyclictest0-21swapper/009:25:160
92199243,18cyclictest11604-21interrupts07:30:203
92199241,4cyclictest25990-21diskmemload11:25:233
92199240,2cyclictest0-21swapper/307:10:123
92199240,1cyclictest0-21swapper/309:10:143
918992423,1cyclictest904-21cyclictest11:32:382
918992423,1cyclictest904-21cyclictest11:17:582
918992423,1cyclictest904-21cyclictest10:24:332
918992423,1cyclictest904-21cyclictest09:37:052
918992423,1cyclictest904-21cyclictest09:32:202
918992422,1cyclictest14263-21apache212:34:372
91499240,23cyclictest0-21swapper/111:26:041
91399243,19cyclictest25990-21diskmemload11:12:270
91399242,21cyclictest26188-21/usr/sbin/munin10:05:240
91399241,21cyclictest23510-21grep07:55:180
91399240,23cyclictest0-21swapper/011:00:170
91399240,22cyclictest0-21swapper/011:40:180
91399240,1cyclictest0-21swapper/011:05:140
92199232,3cyclictest25990-21diskmemload10:46:393
92199232,19cyclictest5620-21gpgv10:15:003
92199231,2cyclictest0-21swapper/312:30:203
92199231,21cyclictest0-21swapper/312:37:593
92199231,1cyclictest25990-21diskmemload11:51:253
92199230,22cyclictest0-21swapper/310:55:163
92199230,22cyclictest0-21swapper/310:20:013
92199230,19cyclictest0-21swapper/312:02:143
918992322,1cyclictest904-21cyclictest11:37:052
918992322,1cyclictest904-21cyclictest11:10:002
918992322,1cyclictest904-21cyclictest09:24:452
918992322,1cyclictest0-21swapper/207:44:042
91399230,2cyclictest0-21swapper/012:04:550
91399230,22cyclictest0-21swapper/011:25:220
91399230,1cyclictest0-21swapper/009:53:430
92199223,18cyclictest25990-21diskmemload10:03:443
92199221,5cyclictest3005-21chrt12:20:413
92199221,20cyclictest25990-21diskmemload10:25:043
92199221,1cyclictest25990-21diskmemload10:32:183
92199221,1cyclictest0-21swapper/310:41:273
92199221,1cyclictest0-21swapper/310:41:273
92199220,2cyclictest0-21swapper/311:37:423
92199220,1cyclictest0-21swapper/311:20:033
92199220,1cyclictest0-21swapper/309:58:093
92199220,1cyclictest0-21swapper/309:05:153
918992221,1cyclictest904-21cyclictest10:33:342
91399221,14cyclictest1291-21runrttasks11:55:130
91399220,21cyclictest0-21swapper/009:10:140
91399220,19cyclictest0-21swapper/008:15:140
92199213,18cyclictest0-21swapper/311:58:013
92199212,19cyclictest0-21swapper/310:05:243
92199212,18cyclictest25990-21diskmemload11:05:363
92199212,17cyclictest11329-21aten2_r7power_e09:25:173
92199211,1cyclictest25990-21diskmemload09:34:483
918992120,1cyclictest904-21cyclictest10:56:262
918992120,1cyclictest0-21swapper/211:28:432
91499210,21cyclictest0-21swapper/112:04:561
91399210,2cyclictest0-21swapper/010:35:470
91399210,20cyclictest25990-21diskmemload10:12:150
91399210,1cyclictest0-21swapper/012:12:220
91399210,1cyclictest0-21swapper/010:32:180
91399210,17cyclictest0-21swapper/009:47:550
92199201,18cyclictest8843-21df_abs07:25:173
92199200,2cyclictest3406-21sort08:20:463
92199200,2cyclictest0-21swapper/310:50:273
92199200,1cyclictest0-21swapper/311:43:043
92199200,1cyclictest0-21swapper/309:41:343
92199200,1cyclictest0-21swapper/309:16:363
92199200,1cyclictest0-21swapper/307:45:153
918992018,1cyclictest904-21cyclictest09:46:542
91399208,8cyclictest1944-21sh09:16:240
91399201,14cyclictest25990-21diskmemload09:34:000
91399200,3cyclictest0-21swapper/012:28:460
91399200,1cyclictest0-21swapper/011:51:250
91399200,1cyclictest0-21swapper/009:21:440
91399200,19cyclictest0-21swapper/009:05:150
92199193,15cyclictest20795-21awk08:55:273
92199192,16cyclictest0-21swapper/311:46:343
92199191,1cyclictest25990-21diskmemload09:36:353
92199190,2cyclictest7416-21cpuspeed_turbos08:30:163
92199190,18cyclictest0-21swapper/307:50:153
91899191,15cyclictest32512-21kworker/2:112:03:132
91399198,8cyclictest16420-21chrt08:49:470
91399194,13cyclictest0-21swapper/011:35:240
91399192,15cyclictest17396-21fschecks_count12:30:190
91399191,16cyclictest0-21swapper/007:45:160
91399190,18cyclictest0-21swapper/007:30:150
91399190,16cyclictest0-21swapper/010:43:310
91399190,16cyclictest0-21swapper/010:43:310
92199180,2cyclictest0-21swapper/308:50:143
92199180,2cyclictest0-21swapper/308:50:133
92199180,17cyclictest0-21swapper/312:25:313
91899182,1cyclictest0-21swapper/210:40:552
91899182,1cyclictest0-21swapper/210:40:552
918991817,1cyclictest0-21swapper/211:55:142
918991814,4cyclictest311rcuc/209:58:272
918991812,5cyclictest311rcuc/209:10:002
91499186,3cyclictest141rcu_preempt11:12:271
91399188,5cyclictest27646-21sh12:36:190
91399182,13cyclictest25990-21diskmemload10:20:000
91399182,13cyclictest0-21swapper/007:10:120
91399181,16cyclictest0-21swapper/010:50:270
91399181,14cyclictest25990-21diskmemload11:49:190
91399180,3cyclictest6775-21tr10:15:180
91399180,2cyclictest0-21swapper/009:41:330
91399180,1cyclictest0-21swapper/010:00:420
91399180,17cyclictest0-21swapper/007:50:150
92199177,8cyclictest7036-21tail07:20:253
92199171,3cyclictest4290-21gltestperf07:15:173
92199171,1cyclictest31170-21fschecks_time10:35:193
91899179,3cyclictest0-21swapper/210:26:072
91899170,1cyclictest0-21swapper/208:31:182
914991716,1cyclictest0-21swapper/112:22:561
91499170,16cyclictest0-21swapper/110:53:221
91499170,14cyclictest0-21swapper/112:13:461
91399178,6cyclictest22785-21ssh10:29:050
91399170,1cyclictest0-21swapper/012:06:180
91399170,1cyclictest0-21swapper/010:45:080
92199168,7cyclictest32276-21wc08:15:013
92199168,7cyclictest12131-21tr08:40:163
92199162,2cyclictest0-21swapper/311:10:163
92199161,2cyclictest10629-21tr08:35:223
92199160,2cyclictest0-21swapper/307:37:573
91899166,8cyclictest25990-21diskmemload10:18:202
91899160,13cyclictest32-21ksoftirqd/210:00:002
914991612,3cyclictest25-21ksoftirqd/109:59:041
914991612,3cyclictest24854-21ls09:05:201
91499160,2cyclictest141rcu_preempt08:45:191
91499160,1cyclictest0-21swapper/112:34:091
91499160,16cyclictest0-21swapper/112:18:491
91399160,15cyclictest0-21swapper/008:20:460
91399160,14cyclictest0-21swapper/007:25:160
918991510,4cyclictest0-21swapper/209:10:142
91499150,1cyclictest0-21swapper/109:45:221
91399157,5cyclictest27257-21chrt08:03:400
91399151,13cyclictest0-21swapper/008:25:180
91399150,1cyclictest0-21swapper/007:20:230
91399150,14cyclictest0-21swapper/009:36:350
91399150,14cyclictest0-21swapper/008:50:130
91399150,14cyclictest0-21swapper/008:50:130
921991410,3cyclictest28599-21grep08:05:203
92199140,1cyclictest0-21swapper/308:00:183
91899141,9cyclictest32-21ksoftirqd/208:23:072
918991413,1cyclictest1169-21kworker/2:109:53:442
914991411,2cyclictest241rcuc/110:55:221
914991410,3cyclictest24779-21tr11:50:181
91399148,5cyclictest22595-21cut09:00:210
91399148,5cyclictest11351-21taskset08:36:460
91399144,9cyclictest28429-21hddtemp_smartct08:05:180
91399140,1cyclictest0-21swapper/008:30:160
92199135,7cyclictest23529-21chrt09:04:183
92199131,2cyclictest224-21systemd-journal08:50:003
91899139,3cyclictest311rcuc/211:45:112
91899139,3cyclictest11875-21/usr/sbin/munin08:40:212
91899137,4cyclictest311rcuc/207:50:152
918991310,2cyclictest2092-21cut07:10:192
918991310,2cyclictest20043-21ls07:49:592
91499139,1cyclictest141rcu_preempt12:05:441
91499138,3cyclictest26753-21df10:05:171
91499134,4cyclictest224-21systemd-journal09:25:001
914991310,2cyclictest740-21sed08:15:181
914991310,2cyclictest6739-21grep11:35:241
914991310,2cyclictest26008-21grep08:00:181
914991310,2cyclictest21405-21cat09:00:001
91499130,8cyclictest3527-21cron10:40:001
91499130,12cyclictest26045-21ls12:35:271
91499130,12cyclictest21813-21expr07:50:231
91399131,6cyclictest19095-21apt-get08:55:150
92199121,4cyclictest5501-21sed08:25:173
91899129,2cyclictest27375-21ls08:05:002
91899129,2cyclictest14742-21grep07:35:272
91899122,4cyclictest0-21swapper/208:48:112
91899121,8cyclictest25990-21diskmemload11:43:482
91899121,4cyclictest32-21ksoftirqd/207:22:532
918991210,2cyclictest32-21ksoftirqd/208:05:172
91899120,11cyclictest0-21swapper/210:45:302
91499129,2cyclictest587-21tail11:30:231
91499129,2cyclictest31463-21head11:55:221
91499129,2cyclictest28441-21basename09:40:171
91499129,2cyclictest25827-21cat10:05:001
91499129,2cyclictest17429-21head08:50:181
91499129,2cyclictest17429-21head08:50:171
91499129,2cyclictest14213-21tr07:35:221
91499129,2cyclictest10629-21grep10:45:171
91499129,2cyclictest1045-21uname10:10:221
91499121,9cyclictest8451-21wget07:25:151
91499120,5cyclictest12166-21kworker/u8:111:22:581
91499120,12cyclictest0-21swapper/111:02:371
91499120,12cyclictest0-21swapper/110:32:191
91499120,12cyclictest0-21swapper/110:18:211
91499120,11cyclictest0-21swapper/109:35:011
91399129,2cyclictest13132-21head08:40:240
91399126,5cyclictest31620-21grep08:10:270
91399122,9cyclictest4291-21gltestperf07:15:180
91399121,2cyclictest224-21systemd-journal07:39:470
91899119,1cyclictest9444-21ntpq07:25:212
91899118,3cyclictest5323-21chrt07:15:582
91899118,2cyclictest25341-21grep11:50:212
91899117,3cyclictest23513-21fschecks_time07:55:192
91899116,2cyclictest32-21ksoftirqd/209:00:222
91899115,3cyclictest32-21ksoftirqd/208:38:142
91899114,2cyclictest224-21systemd-journal08:58:132
91899111,6cyclictest32-21ksoftirqd/210:10:192
91899110,8cyclictest0-21swapper/208:53:132
91899110,8cyclictest0-21swapper/208:53:122
91899110,10cyclictest11356-21fschecks_time07:30:182
91499118,2cyclictest7805-21grep08:30:181
91499118,2cyclictest6590-21ssh10:41:141
91499118,2cyclictest6590-21ssh10:41:141
91499118,2cyclictest32613-21cut09:15:361
91499118,2cyclictest3185-21sed08:20:441
91499118,2cyclictest24344-21cat07:55:271
91499118,2cyclictest21696-21cat09:00:171
91499118,2cyclictest21571-21ssh10:27:251
91499118,2cyclictest20491-21grep11:45:281
91499118,2cyclictest19037-21ssh11:19:481
91499118,2cyclictest18866-21cat09:30:281
91499118,2cyclictest1684-21taskset11:05:141
91499118,2cyclictest1448-21grep07:10:141
91499118,2cyclictest14439-21ssh10:20:361
91499118,2cyclictest12839-21forks11:40:181
91499118,2cyclictest12490-21ssh12:26:581
91499118,2cyclictest11682-21expr09:25:181
91499118,2cyclictest11216-21ssh09:52:421
91499118,2cyclictest10697-21pmu-power08:35:221
91499117,3cyclictest8053-21dump-pmu-power07:25:001
91499111,3cyclictest224-21systemd-journal07:40:171
91499110,10cyclictest5496-21taskset07:18:111
91499110,10cyclictest19293-21sensors_temp07:45:251
91899108,2cyclictest6479-21uptime08:25:282
91899108,1cyclictest1109-21tr08:15:222
91899102,1cyclictest141rcu_preempt08:12:492
91499108,1cyclictest5639-21cat08:25:191
91499108,1cyclictest30053-21tr08:10:151
91499108,1cyclictest11272-21df07:30:181
91499107,2cyclictest28736-21ssh09:10:551
91499107,2cyclictest12637-21grep08:40:191
9149996,2cyclictest27827-21egrep08:05:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional