You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-26 - 02:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri Dec 26, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
263442236164,23sleep30-21swapper/319:06:473
263442236164,23sleep30-21swapper/319:06:463
265542203166,24sleep10-21swapper/119:09:301
265542203166,24sleep10-21swapper/119:09:301
263512203166,25sleep00-21swapper/019:06:520
263512203166,25sleep00-21swapper/019:06:520
246362201165,24sleep20-21swapper/219:05:092
246362201165,24sleep20-21swapper/219:05:092
1086221690,4sleep02667299cyclictest19:45:120
26686991170,111cyclictest0-21swapper/221:26:402
26686991170,111cyclictest0-21swapper/221:26:402
26686999895,2cyclictest0-21swapper/222:11:192
26686996259,2cyclictest0-21swapper/200:12:402
101672560,5sleep230-21rcuc/220:55:242
76452550,2sleep20-21swapper/200:30:232
284372510,1sleep30-21swapper/300:05:203
257392470,4sleep025742-21idleruntime22:45:190
234312470,1sleep00-21swapper/023:55:020
2667299451,26cyclictest10217-21grep22:10:160
2667299421,27cyclictest28851-21latency_hist19:15:010
2667299400,29cyclictest2317-21grep21:50:260
2667299391,24cyclictest13423-21/usr/sbin/munin21:05:260
2667299391,23cyclictest2428-21/usr/sbin/munin20:40:210
2667299381,23cyclictest26922-21expr21:35:130
2667299381,22cyclictest24193-21munin-run21:30:010
2667299381,22cyclictest24193-21munin-run21:30:010
2667299371,26cyclictest11910-21egrep22:15:120
2667299371,24cyclictest27235-21/usr/sbin/munin22:50:260
2667299361,26cyclictest31633-21acpi23:00:140
2667299361,26cyclictest12794-21/usr/sbin/munin19:50:200
2667299361,22cyclictest31056-21cpu21:45:130
224702360,4sleep30-21swapper/321:25:153
224702360,4sleep30-21swapper/321:25:153
2667299356,14cyclictest31748-21munin-run19:20:010
2667299351,31cyclictest7455-21/usr/sbin/munin22:05:170
2667299351,31cyclictest10270-21/usr/sbin/munin23:25:170
2667299351,26cyclictest29434-21/usr/sbin/munin22:55:190
2667299351,24cyclictest17057-21expr23:40:130
2667299351,21cyclictest19340-21cstates23:45:140
2667299341,29cyclictest23796-21/usr/sbin/munin20:15:240
2667299341,24cyclictest22831-21/usr/sbin/munin22:40:170
2667299341,24cyclictest22831-21/usr/sbin/munin22:40:160
2667299340,26cyclictest0-21swapper/020:55:170
2667299335,17cyclictest6528-21switchtime22:00:260
2667299332,19cyclictest26644-21grep00:00:190
2667299332,18cyclictest907-21systemd-logind00:35:390
2667299331,29cyclictest27883-21/usr/sbin/munin00:05:230
2667299331,25cyclictest16608-21chrt23:39:490
2667299331,24cyclictest14407-21fschecks_count22:20:150
2667299331,18cyclictest0-21swapper/000:32:330
2667299331,13cyclictest0-21swapper/023:05:510
2667299321,28cyclictest6590-21idleruntime-cro20:50:020
2667299321,28cyclictest28814-21/usr/sbin/munin21:40:130
2667299321,19cyclictest32592-21/usr/sbin/munin20:35:150
2667299321,19cyclictest21214-21processes21:20:240
2667299321,18cyclictest13728-21smart_sda23:30:270
2667299314,13cyclictest6303-21awk00:30:010
2667299312,25cyclictest4086-21/usr/sbin/munin21:55:210
2667299312,25cyclictest32631-21diskstats00:15:150
2667299311,27cyclictest14989-21/usr/sbin/munin19:55:210
2667299311,26cyclictest21100-21tune2fs22:35:150
2667299311,17cyclictest28193-21/usr/sbin/munin20:25:240
2667299310,23cyclictest0-21swapper/019:40:160
2668199301,2cyclictest0-21swapper/123:15:321
2667299307,20cyclictest18751-21head20:00:250
2667299304,19cyclictest1812-21/usr/sbin/munin19:25:240
2667299302,25cyclictest18439-21/usr/sbin/munin22:30:210
2667299301,16cyclictest31161-21ntpq00:10:230
2667299300,3cyclictest0-21swapper/023:20:120
2668199291,6cyclictest0-21swapper/121:23:421
2667299295,21cyclictest31027-21head20:30:180
2667299291,25cyclictest21596-21/usr/sbin/munin20:10:280
26672992911,15cyclictest7191-21sed19:35:200
2667299290,26cyclictest19399-21/usr/sbin/munin20:05:120
2667299290,17cyclictest3403-21switchtime00:20:250
2669399281,13cyclictest0-21swapper/300:28:583
2668199281,25cyclictest0-21swapper/120:01:451
2667299282,23cyclictest6223-21diskstats23:15:140
2667299281,5cyclictest0-21swapper/022:25:120
2667299281,2cyclictest8079-21/usr/sbin/munin20:50:250
2667299281,24cyclictest4012-21/usr/sbin/munin19:30:270
2667299281,24cyclictest3773-21cpuspeed_turbos23:10:150
2667299281,23cyclictest23846-21tune2fs23:55:150
2667299281,18cyclictest25083-21gltestperf21:30:170
2667299281,16cyclictest0-21swapper/021:10:320
2667299281,14cyclictest19044-21rt-features21:15:270
2667299281,13cyclictest86950irq/126-enp1s0-20:20:170
2667299281,13cyclictest86950irq/126-enp1s0-20:20:170
2668199273,17cyclictest0-21swapper/100:20:291
2668199271,24cyclictest0-21swapper/123:09:441
2668199271,24cyclictest0-21swapper/123:01:101
2667299271,23cyclictest904-21sssd_nss19:20:120
2667299271,23cyclictest12033-21grep21:00:200
2668199261,6cyclictest0-21swapper/123:58:151
2668199261,6cyclictest0-21swapper/122:17:171
2668199261,3cyclictest32730-21cpuspeed_turbos20:35:141
2668199261,23cyclictest0-21swapper/122:56:441
2668199261,16cyclictest0-21swapper/121:11:221
2668199260,24cyclictest0-21swapper/121:56:111
2669399256,3cyclictest86950irq/126-enp1s0-22:00:233
2669399251,6cyclictest31011-21/usr/sbin/munin21:45:193
26686992514,3cyclictest0-21swapper/223:47:182
26681992519,3cyclictest18837-21munin-run23:45:011
2668199251,6cyclictest0-21swapper/123:53:111
2668199251,4cyclictest24698-21kworker/u8:122:48:301
2668199251,21cyclictest6702-21expr00:30:121
2668199251,21cyclictest4662-21df_inode00:25:151
2668199251,21cyclictest24012-21cat20:15:161
2668199251,21cyclictest13770-21diskstats21:05:141
2668199251,21cyclictest11184-21memory23:25:201
2668199251,14cyclictest0-21swapper/119:48:551
2668199251,12cyclictest5542-21memory20:45:221
2668199251,11cyclictest30456-21smartctl19:15:171
2668199251,11cyclictest16446-21kworker/u8:223:39:071
2668199250,16cyclictest0-21swapper/119:10:011
2669399243,18cyclictest0-21swapper/323:30:153
2669399241,4cyclictest0-21swapper/322:51:413
2669399241,10cyclictest29066-21grep19:15:023
2668199242,10cyclictest6313-21cpuspeed_turbos19:35:131
26681992414,2cyclictest0-21swapper/120:52:181
26681992413,3cyclictest0-21swapper/119:20:371
2668199241,15cyclictest0-21swapper/121:16:291
2668199241,15cyclictest0-21swapper/119:27:131
2668199241,12cyclictest6373-21pmu-power22:00:241
2668199241,11cyclictest2316-21grep21:50:261
2668199240,16cyclictest0-21swapper/120:59:051
2669399231,11cyclictest0-21swapper/322:20:163
2668199238,4cyclictest0-21swapper/122:10:211
2668199231,20cyclictest0-21swapper/122:43:101
2668199231,20cyclictest0-21swapper/122:43:091
2668199231,15cyclictest0-21swapper/122:53:521
2668199231,14cyclictest0-21swapper/121:31:081
2668199231,11cyclictest20635-21/usr/sbin/munin22:35:251
2668199231,11cyclictest19668-21gltestperf23:45:171
2668199231,11cyclictest0-21swapper/123:22:121
2668199231,10cyclictest4194-21cstates19:30:121
2668199231,10cyclictest0-21swapper/122:07:111
2668199230,9cyclictest0-21swapper/100:15:141
2668199230,14cyclictest0-21swapper/121:40:181
2669399226,6cyclictest0-21swapper/322:39:513
2669399226,3cyclictest0-21swapper/300:04:143
2669399223,10cyclictest22831-21/usr/sbin/munin22:40:223
2669399223,10cyclictest22831-21/usr/sbin/munin22:40:223
2668699229,2cyclictest25975-21kworker/2:020:35:562
2668699226,2cyclictest0-21swapper/222:25:032
2668699223,8cyclictest0-21swapper/222:57:032
2668699221,7cyclictest9736-21netstat00:35:232
2668699221,4cyclictest0-21swapper/223:56:422
2668199227,3cyclictest0-21swapper/121:04:381
2668199221,19cyclictest0-21swapper/120:28:151
2668199221,11cyclictest15392-21diskmemload00:11:251
2668199220,21cyclictest0-21swapper/123:11:571
2668199220,21cyclictest0-21swapper/100:09:391
2668199220,20cyclictest0-21swapper/121:49:011
37932211,2sleep33991-21unixbench_singl20:40:263
2669399216,5cyclictest0-21swapper/300:32:213
2669399215,4cyclictest0-21swapper/323:05:183
2669399211,11cyclictest801-21snmpd20:25:093
2669399211,11cyclictest4629-21/usr/sbin/munin20:45:303
2669399211,11cyclictest26612-21/usr/sbin/munin21:35:193
2669399211,11cyclictest23454-21/usr/sbin/munin23:55:253
2668699214,7cyclictest6756-21unixbench_multi22:00:272
2668699214,4cyclictest17538-21diskstats20:00:152
2668699212,7cyclictest26000-21users21:30:272
2668699211,8cyclictest0-21swapper/223:23:122
2668699211,7cyclictest14485-21/usr/sbin/munin21:05:202
2668699211,11cyclictest0-21swapper/223:40:072
2668199211,4cyclictest0-21swapper/122:25:031
2668199211,2cyclictest9769-21ntpq00:35:231
2668199210,14cyclictest0-21swapper/119:43:241
246012211,17sleep324755-21df21:30:133
158042211,4sleep115921-21cat19:55:171
2669399209,3cyclictest7732-21kworker/u8:023:25:153
2669399209,3cyclictest30088-21/usr/sbin/munin00:10:273
2669399207,3cyclictest0-21swapper/322:25:253
2669399206,6cyclictest0-21swapper/300:35:413
2669399205,7cyclictest18682-21kworker/u8:023:45:293
2669399201,9cyclictest0-21swapper/323:13:423
2669399201,7cyclictest0-21swapper/320:54:233
2669399201,11cyclictest29777-21/usr/sbin/munin19:15:203
2669399201,10cyclictest0-21swapper/322:18:333
2669399200,9cyclictest0-21swapper/322:58:463
2669399200,4cyclictest0-21swapper/323:22:243
2668699208,3cyclictest86950irq/126-enp1s0-22:15:152
2668699205,8cyclictest12050irq/123-ahci[0000:05:012
2668699205,7cyclictest86950irq/126-enp1s0-23:05:132
2668699205,4cyclictest0-21swapper/200:15:152
2668699204,7cyclictest6941-21kworker/2:120:10:162
2668699203,6cyclictest6190-40dnf00:28:582
2668699201,9cyclictest15987-21forks21:10:162
2668699201,7cyclictest7546-21grep19:35:262
2668699201,5cyclictest3878-21uname21:55:192
2668699201,5cyclictest3681-21/usr/sbin/munin23:10:152
26686992012,5cyclictest32271-21smart_sda21:45:252
2668699201,11cyclictest1639-21irqstats21:50:182
2668699201,10cyclictest7535-21cpuspeed_turbos22:05:142
2668199203,4cyclictest21596-21/usr/sbin/munin20:10:181
2668199201,4cyclictest25794-21tr00:00:121
2668199201,18cyclictest0-21swapper/122:32:081
2668199201,17cyclictest0-21swapper/120:05:141
2668199201,16cyclictest13857-21ntp_states19:50:201
2668199201,11cyclictest0-21swapper/121:27:091
2668199201,11cyclictest0-21swapper/121:27:091
2668199201,10cyclictest25989-21/usr/sbin/munin20:20:211
2668199201,10cyclictest25989-21/usr/sbin/munin20:20:211
2668199201,10cyclictest0-21swapper/122:21:241
2668199200,18cyclictest0-21swapper/120:32:331
2668199200,14cyclictest0-21swapper/123:33:441
2669399196,6cyclictest0-21swapper/321:56:403
2669399196,3cyclictest0-21swapper/322:05:223
2669399195,6cyclictest0-21swapper/321:53:153
2669399192,7cyclictest21933-21hddtemp_smartct23:50:183
26693991913,3cyclictest15146-21cpuspeed_turbos19:55:113
2669399191,13cyclictest0-21swapper/319:35:343
2669399191,11cyclictest0-21swapper/323:04:173
2669399191,11cyclictest0-21swapper/321:20:513
2669399191,11cyclictest0-21swapper/321:15:013
2669399191,10cyclictest2122-21/usr/sbin/munin00:20:283
2669399191,10cyclictest15392-21diskmemload22:45:153
2669399191,10cyclictest0-21swapper/321:41:493
2669399191,10cyclictest0-21swapper/320:20:273
2669399191,10cyclictest0-21swapper/320:20:273
2669399190,8cyclictest0-21swapper/323:35:053
2669399190,8cyclictest0-21swapper/323:17:443
2668699195,3cyclictest6086-21expr23:15:132
2668699194,3cyclictest12050irq/123-ahci[0020:35:012
2668699193,4cyclictest3074-21smart_sda19:25:242
2668699192,6cyclictest751-21rngd23:25:472
2668699192,4cyclictest12130-21memory21:00:192
2668699191,9cyclictest22181-21latency_hist21:25:022
2668699191,7cyclictest86950irq/126-enp1s0-19:24:572
2668699191,7cyclictest0-21swapper/222:22:332
2668699191,6cyclictest27246-21/usr/sbin/munin19:10:142
2668699191,6cyclictest1273-21cat23:05:012
2668699191,6cyclictest10843-21cat19:45:122
2668699191,5cyclictest904-21sssd_nss21:20:002
2668699191,5cyclictest6590-21crond20:50:012
2668699191,5cyclictest0-21swapper/223:35:252
2668699191,5cyclictest0-21swapper/219:15:462
2668699191,11cyclictest0-21swapper/200:05:532
2668199191,2cyclictest0-21swapper/120:42:351
84662181,2sleep20-21swapper/220:51:012
290672181,2sleep20-21swapper/222:53:252
2669399184,6cyclictest41-21ksoftirqd/321:05:163
2669399182,13cyclictest10716-21ntp_states22:10:223
2669399181,9cyclictest0-21swapper/320:36:403
2669399181,7cyclictest19078-21cut22:30:163
2669399181,6cyclictest3384-21kworker/3:220:58:163
2669399181,6cyclictest0-21swapper/321:15:033
2669399181,5cyclictest32440-21tr00:15:143
26693991812,3cyclictest582-21ntpq19:20:203
2668699184,8cyclictest0-21swapper/223:32:412
2668699183,8cyclictest86950irq/126-enp1s0-22:35:242
2668699183,5cyclictest0-21swapper/221:38:522
2668699183,4cyclictest24060-21cstates20:15:162
2668699183,4cyclictest19077-21sed22:30:162
2668699182,8cyclictest30-21rcuc/223:50:272
2668699182,3cyclictest86950irq/126-enp1s0-21:40:132
2668699181,9cyclictest801-21snmpd20:20:082
2668699181,9cyclictest801-21snmpd20:20:082
2668699181,7cyclictest19738-21df_inode20:05:142
2668699181,5cyclictest25843-21irqstats22:45:202
2668699181,5cyclictest13606-21irqstats19:50:172
2668699181,5cyclictest0-21swapper/220:25:232
2668699181,4cyclictest0-21swapper/222:44:052
2668699181,4cyclictest0-21swapper/222:44:052
2668199180,14cyclictest0-21swapper/121:38:461
2669399179,5cyclictest0-21swapper/321:00:203
2669399178,3cyclictest0-21swapper/320:00:183
2669399176,3cyclictest86950irq/126-enp1s0-19:50:123
2669399175,3cyclictest15392-21diskmemload23:40:143
2669399171,3cyclictest0-21swapper/320:34:163
26693991713,3cyclictest0-21swapper/319:25:383
2669399171,13cyclictest24865-21ntp_states20:15:253
2669399171,10cyclictest0-21swapper/319:45:223
2668699173,3cyclictest86950irq/126-enp1s0-19:30:122
2668699171,5cyclictest0-21swapper/219:44:232
2668699171,3cyclictest0-21swapper/219:55:132
2668699171,10cyclictest0-21swapper/200:21:352
2669399161,9cyclictest0-21swapper/319:30:233
2669399161,3cyclictest0-21swapper/320:07:453
26693991610,3cyclictest9987-21uptime19:40:263
26693991610,3cyclictest21869-21expr20:10:143
26686991611,3cyclictest2883-21cut20:40:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional