You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-24 - 05:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Sep 24, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
292192238162,24sleep10-21swapper/119:07:241
293072236163,24sleep20-21swapper/219:08:342
291112227150,24sleep00-21swapper/019:05:580
291762204168,24sleep30-21swapper/319:06:493
584421800,5sleep12951399cyclictest23:05:271
254321590,4sleep22951999cyclictest21:45:212
2951999105102,2cyclictest0-21swapper/219:54:072
29519997775,1cyclictest0-21swapper/221:11:302
29519996657,8cyclictest0-21swapper/220:48:002
24212510,3sleep230-21rcuc/223:00:132
106212380,1sleep30-21swapper/300:30:253
29509993611,11cyclictest1247-21idleruntime-cro21:45:010
2951399322,16cyclictest87550irq/128-enp1s0-22:05:481
2951399301,2cyclictest26054-21perf23:55:011
233552300,3sleep023359-21pmu-power20:05:250
121102300,5sleep230-21rcuc/200:35:172
2951399291,11cyclictest10062-21awk22:05:021
2951399288,9cyclictest0-21swapper/100:04:301
2951399282,3cyclictest7844-21munin-run22:00:011
2951399281,17cyclictest0-21swapper/122:56:271
2951399281,17cyclictest0-21swapper/122:56:271
2951399281,17cyclictest0-21swapper/122:39:541
2951399274,6cyclictest31151-40dnf21:36:531
2951399273,11cyclictest0-21swapper/123:57:331
2951399271,3cyclictest0-21swapper/122:21:031
2951399271,16cyclictest0-21swapper/121:25:461
163532270,3sleep32952599cyclictest23:30:203
319432260,1sleep20-21swapper/220:25:192
319432260,1sleep20-21swapper/220:25:192
2951399261,16cyclictest0-21swapper/100:33:091
29513992611,3cyclictest87550irq/128-enp1s0-22:25:131
2952599258,3cyclictest24789-21fschecks_count20:10:143
2951999255,12cyclictest24139-21hddtemp_smartct22:35:162
2951399251,6cyclictest0-21swapper/121:14:521
2951399251,17cyclictest0-21swapper/121:30:191
2951399250,17cyclictest0-21swapper/121:40:051
2950999259,9cyclictest0-21swapper/021:35:200
162462250,3sleep316250-21gltestperf19:50:153
2952599249,3cyclictest26615-21/usr/sbin/munin20:15:183
2952599241,7cyclictest8564-1kworker/u9:122:23:103
2951399248,3cyclictest15209-21fschecks_time22:15:171
2951399241,10cyclictest0-21swapper/121:48:411
2951399240,21cyclictest0-21swapper/123:43:071
2951399240,13cyclictest0-21swapper/123:25:131
2952599233,6cyclictest0-21swapper/300:28:533
2952599232,10cyclictest2438-21/usr/sbin/munin19:20:243
2952599230,4cyclictest7514-21kworker/u8:022:00:183
2951999231,4cyclictest0-21swapper/223:43:072
2951999231,16cyclictest0-21swapper/222:10:032
2951999231,12cyclictest25010-21cat21:25:022
2951399239,2cyclictest13673-21date00:40:021
2951399231,2cyclictest0-21swapper/123:15:151
2951399231,19cyclictest25889-21expr22:40:121
2951399231,19cyclictest2091-21awk00:10:231
2951399231,13cyclictest0-21swapper/121:52:361
2951399231,11cyclictest0-21swapper/122:14:181
2951399231,11cyclictest0-21swapper/100:15:511
2951399231,10cyclictest0-21swapper/100:09:241
2951399231,10cyclictest0-21swapper/100:09:241
29513992310,3cyclictest16198-21memory23:30:181
2951399230,6cyclictest2324-21kworker/u8:100:24:171
2950999236,6cyclictest0-21swapper/000:12:100
2952599225,4cyclictest0-21swapper/322:25:243
2952599222,3cyclictest10955-21hddtemp_smartct22:05:163
2951999222,3cyclictest12050irq/123-ahci[0019:35:012
2951999221,19cyclictest0-21swapper/219:46:212
2951999221,10cyclictest26589-21missed_timers22:40:192
2951999221,10cyclictest16760-21cat22:20:022
2951399227,4cyclictest12871-21munin-run23:25:021
29513992216,3cyclictest32125-21ntp_states19:15:031
2951399221,19cyclictest0-21swapper/122:32:311
2951399221,12cyclictest0-21swapper/123:39:041
2951399221,10cyclictest24424-21acpi20:10:101
2951399221,10cyclictest20828-21cstates21:15:131
2950999229,8cyclictest0-21swapper/020:02:100
2950999226,9cyclictest12241-21crond22:10:010
2950999226,9cyclictest0-21swapper/021:35:010
2950999225,9cyclictest4678-21fschecks_count23:05:160
2950999224,10cyclictest7-21ksoftirqd/023:30:250
2950999221,14cyclictest0-21swapper/021:50:170
2952599215,6cyclictest0-21swapper/323:19:463
2952599215,5cyclictest0-21swapper/321:59:043
2952599212,11cyclictest17483-21/usr/sbin/munin23:35:283
2952599211,5cyclictest27633-21munin-run22:45:013
2952599211,5cyclictest0-21swapper/323:02:153
2952599211,15cyclictest0-21swapper/321:04:133
2951999217,3cyclictest12050irq/123-ahci[0023:55:012
2951999216,3cyclictest12050irq/123-ahci[0020:15:152
2951999215,7cyclictest26548-21cstates23:55:132
2951999215,4cyclictest32-21ksoftirqd/222:00:142
2951999215,3cyclictest10325-21strings00:30:202
2951999214,7cyclictest87350irq/126-enp1s0-22:55:142
2951999214,7cyclictest87350irq/126-enp1s0-22:55:142
2951999214,6cyclictest0-21swapper/200:29:212
2951999213,6cyclictest27240-21/usr/sbin/munin21:30:272
2951999211,8cyclictest0-21swapper/221:40:032
2951999211,7cyclictest25710-21hddtemp_smartct21:25:162
2951999211,7cyclictest12613-21cat20:55:172
2951999211,7cyclictest11193-21memory22:05:192
2951999211,6cyclictest0-21swapper/200:11:052
29519992115,3cyclictest309862chrt21:35:262
2951999211,4cyclictest0-21swapper/223:15:102
2951399219,3cyclictest22000-21awk20:05:011
2951399216,3cyclictest0-21swapper/123:04:511
2951399211,5cyclictest22250-21cpu20:05:161
2951399211,18cyclictest0-21swapper/122:45:221
2951399211,18cyclictest0-21swapper/120:38:151
2951399211,18cyclictest0-21swapper/119:38:311
2951399211,14cyclictest0-21swapper/120:54:201
2951399211,11cyclictest27171-21perl20:15:151
2951399211,11cyclictest1-21systemd21:07:481
2951399210,18cyclictest28800-21taskset20:20:031
2951399210,14cyclictest22560-21kworker/u8:021:21:531
2950999217,4cyclictest3268-21sshd21:47:510
2950999217,2cyclictest27268-21strings23:55:190
2950999216,3cyclictest150552dump-pmu-power23:30:020
2950999215,8cyclictest22163-21sed22:30:180
2950999215,7cyclictest28411-21fschecks_time22:45:140
2950999214,10cyclictest24417-21/usr/sbin/munin20:10:120
29509992119,1cyclictest0-21swapper/000:15:220
2950999211,18cyclictest0-21swapper/022:00:130
2950999211,11cyclictest19405-21df22:25:120
2952599209,2cyclictest12525-21sshd23:20:493
2952599206,4cyclictest0-21swapper/322:10:163
2952599205,5cyclictest0-21swapper/300:36:173
2952599204,7cyclictest13161-21sort23:25:103
2952599203,11cyclictest16035-21kworker/u8:221:15:003
2952599201,9cyclictest0-21swapper/323:10:323
2952599201,6cyclictest0-21swapper/323:50:243
2952599201,17cyclictest0-21swapper/320:55:283
2952599201,13cyclictest0-21swapper/322:55:293
2952599201,13cyclictest0-21swapper/322:55:293
2952599201,11cyclictest0-21swapper/319:30:253
2952599201,10cyclictest21277-21users23:40:283
2952599201,10cyclictest0-21swapper/321:45:523
2952599200,12cyclictest0-21swapper/321:31:323
2952599200,10cyclictest2324-21kworker/u8:100:20:313
2951999205,3cyclictest87350irq/126-enp1s0-23:37:512
2951999205,3cyclictest2841-21awk20:35:012
2951999204,3cyclictest5237-21grep21:50:292
2951999203,3cyclictest0-21swapper/223:09:342
2951999202,6cyclictest22528-21smart_sda22:30:242
2951999202,4cyclictest12050irq/123-ahci[0020:45:002
2951999202,15cyclictest31933-21find00:05:242
2951999202,15cyclictest31933-21find00:05:242
2951999201,9cyclictest30805-21sadc22:50:212
2951999201,9cyclictest22156-21grep21:15:272
2951999201,9cyclictest16852-21grep21:05:162
2951999201,9cyclictest124772sleep223:20:242
2951999201,6cyclictest5550-21memory19:25:172
2951999201,6cyclictest23846-21latency_hist23:50:012
2951999201,5cyclictest0-21swapper/220:23:542
2951999201,10cyclictest0-21swapper/222:48:402
2951999201,10cyclictest0-21swapper/219:22:202
2951399206,4cyclictest0-21swapper/123:14:221
2951399201,6cyclictest8632-21kworker/1:219:20:261
29513992016,3cyclictest0-21swapper/100:26:351
2951399201,17cyclictest0-21swapper/120:47:421
2951399201,17cyclictest0-21swapper/119:51:031
2951399201,11cyclictest0-21swapper/123:47:321
2951399200,11cyclictest18457-21gltestperf19:55:161
2950999205,7cyclictest538-21fschecks_count19:15:160
2950999205,3cyclictest87550irq/128-enp1s0-19:45:170
2950999201,7cyclictest0-21swapper/000:20:160
2950999201,3cyclictest0-21swapper/021:25:270
29509992012,5cyclictest32275-21smart_sda20:25:240
29509992012,5cyclictest32275-21smart_sda20:25:240
2950999201,16cyclictest10589-21head19:35:260
2950999201,11cyclictest6504-21/usr/sbin/munin23:10:290
2950999201,11cyclictest31952-21NetworkManager22:54:070
2950999201,10cyclictest30315-21sshd00:03:120
2950999201,10cyclictest22344-21tune2fs23:45:140
2950999201,10cyclictest11530-21/usr/sbin/munin00:35:210
2950999201,10cyclictest0-21swapper/022:59:410
2950999201,10cyclictest0-21swapper/022:59:410
2950999201,10cyclictest0-21swapper/020:48:160
2952599198,3cyclictest29435-21/usr/sbin/munin21:35:163
2952599192,14cyclictest14674-21/usr/sbin/munin22:15:183
2952599192,13cyclictest30066-21/usr/sbin/munin22:50:163
2952599192,13cyclictest28501-21grep22:45:163
2952599191,9cyclictest20093-21cat20:00:133
2952599191,9cyclictest0-21swapper/300:01:103
2952599191,8cyclictest27000-21latency_hist21:30:003
29525991914,3cyclictest2093-21find20:30:263
2952599191,3cyclictest0-21swapper/300:07:203
2952599191,3cyclictest0-21swapper/300:07:203
29525991913,3cyclictest25214-40dnf22:37:233
2952599191,14cyclictest754-21unixbench_singl21:40:243
2952599191,11cyclictest0-21swapper/320:05:283
2952599191,10cyclictest808-21snmpd20:39:213
2952599191,10cyclictest21881-21/usr/sbin/munin23:45:183
2952599191,10cyclictest14481-21ntpq19:45:223
2952599191,10cyclictest1067-21grep19:15:193
2952599191,10cyclictest0-21swapper/320:25:153
2952599191,10cyclictest0-21swapper/320:25:153
2952599190,8cyclictest0-21swapper/323:58:413
2952599190,8cyclictest0-21swapper/321:18:223
2951999195,3cyclictest87350irq/126-enp1s0-23:15:012
2951999195,3cyclictest30-21rcuc/220:10:142
2951999191,8cyclictest0-21swapper/219:37:102
2951999191,8cyclictest0-21swapper/200:00:212
2951999191,6cyclictest15937-21runrttasks21:04:052
2951999191,5cyclictest31281-21grep19:10:272
2951999191,5cyclictest31281-21grep19:10:272
2951999191,10cyclictest0-21swapper/222:25:162
2951999191,10cyclictest0-21swapper/200:18:022
2951399198,3cyclictest7153-21cstates19:30:151
2951399191,6cyclictest0-21swapper/120:25:171
2951399191,6cyclictest0-21swapper/120:25:171
2951399191,5cyclictest30537-21kernelversion19:10:181
2951399191,5cyclictest30537-21kernelversion19:10:171
2951399191,3cyclictest0-21swapper/119:26:041
2951399191,2cyclictest0-21swapper/120:34:111
2951399191,17cyclictest0-21swapper/122:51:001
2951399191,16cyclictest0-21swapper/119:41:201
2951399190,17cyclictest0-21swapper/120:57:461
2950999197,2cyclictest0-21swapper/020:51:480
2950999195,3cyclictest87550irq/128-enp1s0-23:00:170
2950999195,3cyclictest87550irq/128-enp1s0-22:40:180
2950999194,5cyclictest14884-21kworker/u8:123:35:210
2950999194,4cyclictest8868-21expr23:15:120
2950999194,2cyclictest32423-21kworker/0:200:28:070
2950999193,8cyclictest87550irq/128-enp1s0-21:55:120
2950999191,9cyclictest17856-21strings22:20:180
2950999191,8cyclictest839-21lldpd21:06:500
2950999191,6cyclictest0-21swapper/022:35:260
29509991916,2cyclictest0-21swapper/022:10:220
2950999191,5cyclictest0-21swapper/000:30:250
29509991914,3cyclictest3130-21cpuspeed_turbos20:35:100
29509991914,3cyclictest21344-21sshd23:41:090
2950999191,14cyclictest87350irq/126-enp1s0-23:20:260
2950999191,12cyclictest0-21swapper/021:15:180
2950999191,11cyclictest18473-21cpu21:10:150
2952599188,3cyclictest0-21swapper/320:45:253
2952599185,4cyclictest5028-21perf20:44:563
2952599185,3cyclictest12050irq/123-ahci[0021:10:013
2952599183,9cyclictest22643-21kworker/3:021:21:483
2952599181,15cyclictest4331-21users00:15:263
2952599181,15cyclictest401ktimersoftd/323:05:173
2952599181,15cyclictest0-21swapper/319:41:483
2952599181,13cyclictest10296-21gltestperf20:50:173
2952599181,10cyclictest0-21swapper/322:31:053
2952599181,10cyclictest0-21swapper/320:20:193
2952599181,10cyclictest0-21swapper/300:11:403
2951999184,3cyclictest12050irq/123-ahci[0022:25:022
2951999181,8cyclictest87350irq/126-enp1s0-19:40:232
2951999181,5cyclictest15285-21/usr/sbin/munin23:30:152
2951999181,4cyclictest18496-21hddtemp_smartct19:55:172
29519991812,3cyclictest23004-21if_err_enp1s020:05:212
2951999181,10cyclictest0-21swapper/219:18:302
2951999181,10cyclictest0-21swapper/200:22:132
2951399181,2cyclictest0-21swapper/120:40:451
2951399181,16cyclictest0-21swapper/119:47:141
2951399181,15cyclictest0-21swapper/121:04:571
2950999188,1cyclictest0-21swapper/020:40:170
2950999187,2cyclictest0-21swapper/021:00:140
2950999185,3cyclictest31105-21fschecks_count00:05:150
2950999185,3cyclictest31105-21fschecks_count00:05:150
2950999184,3cyclictest808-21snmpd22:15:110
2950999183,9cyclictest0-21swapper/021:20:440
2950999181,6cyclictest18218-21diskmemload23:50:070
2950999181,5cyclictest11229-21/usr/sbin/munin19:40:240
2950999181,3cyclictest1853-21cut20:30:220
29509991812,3cyclictest30052-21df_inode19:10:120
29509991812,3cyclictest30052-21df_inode19:10:120
29509991812,3cyclictest28817-21acpi20:20:120
2950999181,14cyclictest0-21swapper/019:55:160
193072181,2sleep30-21swapper/319:55:263
2952599174,4cyclictest5510-21latency19:25:173
2952599171,14cyclictest401ktimersoftd/319:10:183
2952599171,14cyclictest401ktimersoftd/319:10:183
2952599171,13cyclictest3970-21cstates21:50:163
2951999172,3cyclictest13194-21expr23:25:112
2951999171,7cyclictest0-21swapper/220:36:482
2951999171,4cyclictest32-21ksoftirqd/221:55:112
2951999171,2cyclictest87550irq/128-enp1s0-20:00:162
2950999172,3cyclictest808-21snmpd20:59:250
2950999171,5cyclictest3325-21latency19:20:190
2950999171,4cyclictest4636-21/usr/sbin/munin19:25:140
2950999171,2cyclictest0-21swapper/019:30:150
2952599165,3cyclictest87650irq/129-enp1s0-19:35:223
2951999161,8cyclictest0-21swapper/220:51:022
2950999161,5cyclictest0-21swapper/020:15:130
2950999161,3cyclictest0-21swapper/019:51:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional