You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 03:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Nov 28, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
269082238163,24sleep20-21swapper/219:06:482
268862238164,24sleep30-21swapper/319:06:333
268182224193,20sleep00-21swapper/019:05:370
269012203165,25sleep10-21swapper/119:06:431
2439621700,4sleep22725999cyclictest20:15:132
2826021530,3sleep32726399cyclictest19:10:183
84042580,3sleep30-21swapper/300:30:283
308802550,5sleep330881-21fschecks_time19:15:163
149552500,1sleep314954-21users19:50:283
27259994537,3cyclictest0-21swapper/221:05:122
2726399381,11cyclictest3897-21expr21:55:133
2724799389,13cyclictest24145-21munin-run20:15:010
2726399374,8cyclictest28334-40dnf20:22:333
2726399361,9cyclictest28912-21ntp_states22:50:223
27263993618,15cyclictest9512-21grep22:05:283
2726399352,5cyclictest8745-21munin-run19:40:013
2726399351,13cyclictest0-21swapper/319:40:173
2726399341,9cyclictest0-21swapper/322:32:183
2726399341,12cyclictest11163-21missed_timers22:10:193
2726399331,5cyclictest30490-21munin-run00:10:003
2726399331,10cyclictest2087-21/usr/sbin/munin23:05:263
2724799335,13cyclictest14429-21basename22:20:010
2726399322,9cyclictest15975-21diskmemload23:25:563
2726399322,8cyclictest0-21swapper/300:20:233
2726399322,3cyclictest86550irq/126-enp1s0-20:25:173
2726399321,9cyclictest0-21swapper/321:51:303
2726399321,5cyclictest86550irq/126-enp1s0-21:49:093
2726399321,4cyclictest0-21swapper/300:12:363
27259993227,3cyclictest25024-21egrep21:30:142
2726399317,9cyclictest13211-21cat22:15:173
2726399314,3cyclictest0-21swapper/322:48:273
2726399311,19cyclictest0-21swapper/322:20:213
2726399311,11cyclictest0-21swapper/323:14:553
2725299313,15cyclictest0-21swapper/120:38:311
2724799311,14cyclictest17529-21munin-run20:00:000
2726399301,5cyclictest0-21swapper/322:26:033
2726399301,3cyclictest0-21swapper/323:46:153
2726399301,11cyclictest0-21swapper/321:29:323
2725299301,12cyclictest0-21swapper/120:10:251
2725299301,10cyclictest0-21swapper/121:11:131
2725299301,10cyclictest0-21swapper/121:11:131
2726399299,3cyclictest15953-21hddtemp_smartct23:35:153
2726399291,12cyclictest0-21swapper/300:15:243
2725299291,9cyclictest0-21swapper/123:53:441
260402290,9sleep121-21rcuc/121:30:221
2726399281,9cyclictest8842-21timerwakeupswit20:50:273
2726399281,8cyclictest15975-21diskmemload23:18:313
2726399281,4cyclictest21242-21/usr/sbin/munin22:35:253
2726399281,11cyclictest0-21swapper/323:50:293
2725999286,3cyclictest15975-21diskmemload21:39:222
2725299286,7cyclictest780-21rngd19:55:291
2724799281,14cyclictest2786-21munin-run20:40:010
102542281,2sleep010363-21/usr/sbin/munin00:35:200
2726399271,3cyclictest0-21swapper/323:55:333
2726399270,13cyclictest0-21swapper/321:35:253
2726399270,13cyclictest0-21swapper/300:25:253
2726399270,13cyclictest0-21swapper/300:25:253
2725299278,10cyclictest0-21swapper/119:50:111
2725299271,20cyclictest0-21swapper/123:34:441
2725299271,12cyclictest0-21swapper/100:35:141
2725299271,10cyclictest4396-21latency_hist19:30:011
2724799274,2cyclictest26793-21kworker/0:121:34:190
40212260,22sleep04206-21/usr/sbin/munin00:20:270
295922261,3sleep129715-21rt-features00:05:231
2726399261,12cyclictest0-21swapper/323:30:453
2726399261,12cyclictest0-21swapper/321:21:013
2726399261,10cyclictest0-21swapper/320:14:293
2726399260,12cyclictest0-21swapper/321:07:253
2725299264,12cyclictest780-21rngd21:15:291
2725299263,15cyclictest0-21swapper/122:29:281
2725299261,6cyclictest0-21swapper/100:15:491
27252992612,4cyclictest16317-21ntp_kernel_pll_23:35:201
2725299261,18cyclictest0-21swapper/122:06:391
2725299260,5cyclictest0-21swapper/122:54:461
2724799261,15cyclictest16321-21kworker/0:022:42:350
2726399251,9cyclictest908-21ntpq23:00:213
2726399251,12cyclictest0-21swapper/320:05:173
2726399251,12cyclictest0-21swapper/320:05:173
2726399251,12cyclictest0-21swapper/319:55:493
2726399250,11cyclictest0-21swapper/320:56:373
2725999251,4cyclictest0-21swapper/221:27:512
2725299252,3cyclictest23-21ksoftirqd/122:10:251
2725299251,5cyclictest3729-21kworker/u8:223:24:441
2725299251,5cyclictest3729-21kworker/u8:223:24:441
2725299251,17cyclictest0-21swapper/121:20:501
2725299251,12cyclictest780-21rngd20:05:231
2725299251,12cyclictest780-21rngd20:05:231
2724799251,12cyclictest18183-21munin-run21:15:010
2724799251,12cyclictest18183-21munin-run21:15:010
2724799250,12cyclictest7835-21idleruntime-cro22:05:000
2726399245,3cyclictest0-21swapper/300:01:433
2726399241,11cyclictest0-21swapper/321:42:533
2726399240,11cyclictest0-21swapper/321:03:413
2725999241,6cyclictest0-21swapper/221:48:332
2725299248,3cyclictest27244-21hddtemp_smartct20:20:171
2725299241,6cyclictest0-21swapper/123:01:271
2725299241,6cyclictest0-21swapper/122:22:351
2725299241,5cyclictest3424-21perf21:55:011
27252992412,10cyclictest1848-21kworker/u8:219:30:051
2725299241,20cyclictest28801-21users21:35:291
2725299241,20cyclictest0-21swapper/123:40:221
2725299241,15cyclictest0-21swapper/120:49:111
2725299241,12cyclictest3899-21proc_pri00:20:251
2725299241,12cyclictest13518-21/usr/sbin/munin22:15:231
2725299241,10cyclictest0-21swapper/120:17:231
2725299240,7cyclictest0-21swapper/100:25:281
2725299240,7cyclictest0-21swapper/100:25:281
2724799248,14cyclictest0-21swapper/023:20:010
2724799241,2cyclictest88250irq/135-enp2s0-20:40:430
2724799241,10cyclictest15595-21users21:05:290
2726399233,2cyclictest86750irq/128-enp1s0-00:38:153
2726399231,9cyclictest0-21swapper/323:44:493
2726399231,8cyclictest0-21swapper/320:40:573
2726399231,12cyclictest0-21swapper/319:21:573
2726399231,11cyclictest26079-21ntp_states21:30:233
2726399230,6cyclictest0-21swapper/321:12:293
2726399230,6cyclictest0-21swapper/321:12:293
2726399230,12cyclictest0-21swapper/322:02:373
2725999231,6cyclictest11558-21switchtime22:10:252
2725999231,6cyclictest0-21swapper/221:51:112
2725999231,5cyclictest32-21ksoftirqd/200:10:202
2725299239,3cyclictest11230-21df_inode23:25:161
27252992320,2cyclictest0-21swapper/122:40:231
2725299231,7cyclictest0-21swapper/122:03:491
2725299231,3cyclictest19634-21grep22:30:171
2725299231,14cyclictest0-21swapper/100:34:011
2725299231,13cyclictest0-21swapper/121:25:381
2725299231,10cyclictest0-21swapper/121:00:031
2724799233,2cyclictest88050irq/133-enp2s0-21:22:310
2724799232,11cyclictest21394-21kworker/0:100:00:570
2724799231,11cyclictest27254-21kworker/0:222:50:010
2724799231,10cyclictest0-21swapper/022:35:130
2726399227,3cyclictest0-21swapper/320:16:053
2726399223,2cyclictest86850irq/129-enp1s0-19:28:273
27263992210,3cyclictest0-21swapper/322:44:573
2725999221,6cyclictest0-21swapper/223:12:192
2725999221,5cyclictest0-21swapper/223:22:372
2725999221,5cyclictest0-21swapper/223:22:372
2725999221,4cyclictest0-21swapper/223:03:152
2725999221,18cyclictest2785-21perf20:40:022
2725299227,8cyclictest0-21swapper/100:01:261
2725299225,8cyclictest88050irq/133-enp2s0-19:35:271
2725299225,3cyclictest0-21swapper/100:13:461
2725299223,2cyclictest86750irq/128-enp1s0-23:58:211
2725299222,6cyclictest2427-21df_inode23:05:141
2725299222,2cyclictest0-21swapper/119:24:151
27252992216,3cyclictest5352-21ntp_states23:10:231
2725299221,16cyclictest0-21swapper/122:59:001
27252992211,3cyclictest0-21swapper/123:15:261
2724799227,5cyclictest11-21rcuc/000:14:500
2724799226,3cyclictest0-21swapper/023:35:360
2724799223,4cyclictest7910-21kworker/0:119:21:590
2724799221,6cyclictest0-21swapper/021:44:420
2724799221,5cyclictest0-21swapper/000:19:350
2724799221,12cyclictest8513-21kworker/0:120:06:350
2724799221,12cyclictest8513-21kworker/0:120:06:350
2726399215,6cyclictest0-21swapper/323:20:313
2726399215,6cyclictest0-21swapper/323:20:313
2726399211,5cyclictest0-21swapper/322:56:213
2725999215,6cyclictest768-21in:imjournal00:25:012
2725999215,6cyclictest30-21rcuc/221:15:272
2725999211,8cyclictest0-21swapper/219:22:352
2725999211,7cyclictest0-21swapper/222:44:542
2725999211,5cyclictest0-21swapper/223:41:492
2725999211,5cyclictest0-21swapper/200:07:212
2725999211,3cyclictest0-21swapper/222:58:532
2725999210,8cyclictest14591-21grep23:30:252
2725299218,3cyclictest0-21swapper/121:40:181
2725299213,3cyclictest86750irq/128-enp1s0-22:35:211
2725299212,3cyclictest0-21swapper/120:33:401
2725299212,10cyclictest0-21swapper/121:47:481
2725299211,18cyclictest21-21rcuc/119:45:161
2725299211,15cyclictest86750irq/128-enp1s0-23:46:371
2724799219,9cyclictest1980-21fschecks_time21:50:170
2724799216,2cyclictest16912-21kworker/0:023:43:510
2724799215,2cyclictest10554-21kworker/0:023:24:310
2724799215,2cyclictest10554-21kworker/0:023:24:310
2724799214,3cyclictest88050irq/133-enp2s0-21:36:030
2724799213,3cyclictest88050irq/133-enp2s0-19:10:180
2724799213,3cyclictest86750irq/128-enp1s0-00:05:190
2724799211,9cyclictest21576-21kworker/0:323:54:410
2724799211,7cyclictest0-21swapper/023:46:130
2724799211,6cyclictest0-21swapper/023:30:150
2724799211,6cyclictest0-21swapper/021:45:290
2724799211,6cyclictest0-21swapper/000:30:310
2724799211,13cyclictest0-21swapper/021:59:410
2724799210,9cyclictest0-21swapper/020:00:110
2726399202,10cyclictest11164-21/usr/sbin/munin19:45:183
2726399200,6cyclictest0-21swapper/320:32:193
2725999205,5cyclictest0-21swapper/200:03:362
2725999202,6cyclictest780-21rngd23:08:592
2725999201,7cyclictest0-21swapper/223:49:032
2725999201,5cyclictest0-21swapper/200:39:352
2725999201,4cyclictest0-21swapper/222:00:372
2725999201,2cyclictest0-21swapper/200:30:132
2725299205,13cyclictest21-21rcuc/119:15:011
2725299203,3cyclictest88050irq/133-enp2s0-21:05:151
2725299203,3cyclictest21-21rcuc/121:55:161
2725299201,7cyclictest3010-21/usr/sbin/munin20:40:141
27252992016,3cyclictest0-21swapper/120:58:531
2725299201,18cyclictest0-21swapper/119:40:261
2725299201,18cyclictest0-21swapper/119:19:571
2725299201,17cyclictest0-21swapper/120:25:051
2725299201,10cyclictest0-21swapper/122:46:011
2725299200,18cyclictest0-21swapper/120:50:281
2724799204,8cyclictest23947-21kworker/0:220:20:190
2724799202,4cyclictest3152-21ntp_states23:05:210
2724799201,9cyclictest8311-21kworker/0:323:29:110
2724799201,7cyclictest10249-21/usr/sbin/munin22:10:160
2724799201,5cyclictest0-21swapper/023:58:210
2724799201,10cyclictest5206-21/usr/sbin/munin20:45:220
2726399191,9cyclictest5194-21gltestperf19:30:153
2726399191,10cyclictest724-21/usr/sbin/munin20:35:273
2726399191,10cyclictest0-21swapper/321:19:093
2725999195,5cyclictest9810-21expr20:55:142
2725999194,5cyclictest30-21rcuc/219:50:232
2725999193,6cyclictest32-21ksoftirqd/219:10:162
2725999192,5cyclictest0-21swapper/221:55:152
2725999191,7cyclictest31894-21unixbench_singl19:15:262
2725999191,7cyclictest0-21swapper/222:36:242
2725999191,6cyclictest0-21swapper/223:37:192
2725999191,6cyclictest0-21swapper/221:42:322
2725999191,6cyclictest0-21swapper/200:25:242
2725999191,6cyclictest0-21swapper/200:25:242
2725999191,5cyclictest0-21swapper/223:51:512
2725999191,4cyclictest0-21swapper/222:19:062
27259991914,3cyclictest18901-21pmu-power20:00:232
2725999191,15cyclictest3484-21ls19:25:232
27259991910,2cyclictest86750irq/128-enp1s0-22:49:212
2725299191,12cyclictest0-21swapper/120:01:131
2724799195,3cyclictest29512-21kworker/0:022:52:150
2724799194,5cyclictest88050irq/133-enp2s0-19:38:270
2724799193,2cyclictest88250irq/135-enp2s0-22:21:270
2724799191,7cyclictest0-21swapper/022:30:070
2724799191,7cyclictest0-21swapper/021:15:260
2724799191,6cyclictest8970-21/usr/sbin/munin19:40:230
2724799191,6cyclictest30381-21df_inode22:55:140
27247991913,3cyclictest2378-21/usr/sbin/munin19:25:250
2724799191,10cyclictest0-21swapper/020:50:290
2724799191,10cyclictest0-21swapper/019:15:200
63712181,2sleep00-21swapper/019:32:560
2726399185,2cyclictest86650irq/127-enp1s0-20:45:153
27263991811,4cyclictest18252-21tune2fs20:00:143
2725999189,3cyclictest86750irq/128-enp1s0-22:52:072
2725999188,3cyclictest86850irq/129-enp1s0-00:15:282
2725999187,3cyclictest30-21rcuc/219:40:172
2725999184,5cyclictest30-21rcuc/221:20:272
2725999181,7cyclictest17114-21grep19:55:262
2725999181,7cyclictest0-21swapper/222:30:052
2725999181,7cyclictest0-21swapper/219:35:272
2725999181,6cyclictest0-21swapper/223:15:122
2725999181,6cyclictest0-21swapper/222:28:162
2725999181,5cyclictest0-21swapper/222:21:152
27259991812,3cyclictest784-21NetworkManager21:11:472
27259991812,3cyclictest784-21NetworkManager21:11:472
2725999180,8cyclictest9219-21proc_pri22:05:252
2724799184,2cyclictest1571-21kworker/0:123:10:190
2724799184,2cyclictest12967-21kworker/0:219:49:190
2724799181,6cyclictest0-21swapper/022:06:060
2724799181,6cyclictest0-21swapper/021:25:030
2724799181,5cyclictest0-21swapper/023:03:060
27247991814,3cyclictest7-21ksoftirqd/021:00:220
2724799181,3cyclictest5268-21cstates00:25:150
2724799181,3cyclictest5268-21cstates00:25:150
2724799181,15cyclictest0-21swapper/020:26:420
2724799180,16cyclictest0-21swapper/020:58:410
166082181,2sleep00-21swapper/022:25:010
133672180,14sleep00-21swapper/019:50:140
2725999175,3cyclictest12224-21timerandwakeup23:25:262
2725999172,7cyclictest11804-21/usr/sbin/munin21:00:172
2725999171,6cyclictest0-21swapper/223:55:082
2725999171,5cyclictest0-21swapper/220:50:152
27259991712,3cyclictest6122-21missed_timers20:45:212
27259991712,3cyclictest29027-21expr20:25:152
2725999171,15cyclictest0-21swapper/220:44:572
27259991710,4cyclictest27038-21fschecks_time20:20:172
2724799172,7cyclictest0-21swapper/020:31:290
2724799171,9cyclictest0-21swapper/020:17:210
2725999162,5cyclictest30-21rcuc/219:45:152
2725999161,6cyclictest30-21rcuc/220:30:252
2725999161,5cyclictest23648-21unixbench-2d20:10:272
2725999161,14cyclictest0-21swapper/220:05:102
2725999161,14cyclictest0-21swapper/220:05:102
27259991611,3cyclictest4838-21cat19:30:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional