You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 18:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 24, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
167642240163,25sleep20-21swapper/207:07:272
167642240163,25sleep20-21swapper/207:07:272
169392232163,56sleep30-21swapper/307:09:413
169392232163,56sleep30-21swapper/307:09:413
167432224154,57sleep10-21swapper/107:07:101
167432224154,57sleep10-21swapper/107:07:101
169562218147,59sleep00-21swapper/007:09:560
169562218147,59sleep00-21swapper/007:09:550
1308821810,3sleep21706099cyclictest11:50:232
1308821810,3sleep21706099cyclictest11:50:232
2143321670,3sleep01704899cyclictest12:10:200
17065998678,2cyclictest0-21swapper/309:09:213
17065997470,3cyclictest0-21swapper/308:48:033
17065997267,2cyclictest28257-21sendmail_mailqu12:25:253
17065996562,2cyclictest0-21swapper/312:01:203
17065996359,3cyclictest0-21swapper/311:17:233
271032520,1sleep20-21swapper/207:30:142
184022520,1sleep30-21swapper/308:25:013
113382510,1sleep30-21swapper/309:20:203
17065994946,2cyclictest0-21swapper/308:09:283
1705699478,27cyclictest0-21swapper/111:40:011
17065994642,3cyclictest0-21swapper/309:32:313
1705699424,6cyclictest22522-40dnf08:32:291
17065994036,3cyclictest0-21swapper/307:31:453
1705699401,27cyclictest30196-21/usr/sbin/munin12:30:191
1705699401,26cyclictest0-21swapper/107:54:181
17065993936,2cyclictest0-21swapper/308:37:043
1705699391,25cyclictest13372-21chrt11:52:191
1705699391,25cyclictest13372-21chrt11:52:191
250332380,4sleep11705699cyclictest11:05:201
1705699386,29cyclictest7901-21sshd09:14:261
1705699383,3cyclictest27715-21switchtime11:10:251
1705699381,7cyclictest5768-21diskmemload09:16:391
1705699378,18cyclictest9247-21fschecks_count10:30:151
1705699371,6cyclictest537-21/usr/sbin/munin11:25:191
17065993632,3cyclictest0-21swapper/311:53:573
17065993632,3cyclictest0-21swapper/311:53:573
1705699366,24cyclictest0-21swapper/107:14:071
1705699361,4cyclictest2558-21diskstats10:15:141
1705699361,26cyclictest18235-21/usr/sbin/munin12:05:251
17056993610,23cyclictest5179-21/usr/sbin/munin10:20:191
1705699360,34cyclictest0-21swapper/109:25:161
1705699360,34cyclictest0-21swapper/109:25:161
1705699354,4cyclictest32722-21/usr/sbin/munin10:10:141
17056993511,21cyclictest2833-21/usr/sbin/munin11:30:241
1705699341,3cyclictest29458-21cat08:50:021
1705699341,10cyclictest0-21swapper/107:16:141
1704899341,25cyclictest0-21swapper/008:25:470
1705699335,26cyclictest23-21ksoftirqd/109:42:571
1705699333,2cyclictest86550irq/126-enp1s0-09:20:201
1705699332,4cyclictest21437-21cpu09:45:121
1705699331,5cyclictest27024-21/usr/sbin/munin12:25:271
1705699331,5cyclictest17835-21latency09:35:181
1705699331,4cyclictest17890-21expr10:50:141
17056993311,19cyclictest22963-21/usr/sbin/munin08:35:241
1704899331,24cyclictest0-21swapper/007:48:210
1704899331,24cyclictest0-21swapper/007:43:450
1704899331,23cyclictest0-21swapper/007:29:240
1705699328,16cyclictest0-21swapper/110:40:201
1705699328,16cyclictest0-21swapper/110:40:201
1705699325,3cyclictest86850irq/129-enp1s0-09:05:161
1705699321,3cyclictest21016-21processes10:55:241
1705699321,12cyclictest0-21swapper/108:26:071
17056993211,18cyclictest5520-21expr07:55:131
17056993210,19cyclictest22343-21/usr/sbin/munin07:20:221
17056993210,18cyclictest25775-21/usr/sbin/munin09:55:121
1704899320,25cyclictest0-21swapper/011:17:290
1705699318,20cyclictest25118-21/usr/sbin/munin07:25:151
1705699318,20cyclictest16708-21hwlatdetect12:00:171
1705699316,2cyclictest32096-21switchtime11:20:241
1705699313,25cyclictest9817-21grep11:45:131
1705699312,4cyclictest32093-21diskstats08:55:141
17056993115,13cyclictest24092-21cat11:05:011
1705699311,27cyclictest25409-21expr08:40:131
1705699311,12cyclictest0-21swapper/109:53:311
1704899319,11cyclictest0-21swapper/008:00:000
1704899311,14cyclictest0-21swapper/008:10:080
1704899311,12cyclictest0-21swapper/010:52:250
1704899311,12cyclictest0-21swapper/008:18:480
1704899310,24cyclictest0-21swapper/012:05:530
1704899310,24cyclictest0-21swapper/010:27:150
1705699309,18cyclictest8490-21smartctl11:40:211
1705699308,19cyclictest15439-21/usr/sbin/munin08:15:271
1705699308,19cyclictest10227-21tune2fs08:05:151
1705699306,21cyclictest31742-21grep10:05:261
1705699304,3cyclictest23673-21/usr/sbin/munin12:15:201
1705699301,4cyclictest30107-21fschecks_time08:50:161
1705699301,11cyclictest0-21swapper/108:21:511
17056993011,16cyclictest14974-21cat09:30:121
17056993010,17cyclictest29804-21processes11:15:231
17056993010,17cyclictest21780-21wc12:10:261
17056993010,17cyclictest11195-21ls10:35:131
1705699300,11cyclictest0-21swapper/112:21:231
1704899301,6cyclictest0-21swapper/009:43:260
1704899301,23cyclictest0-21swapper/007:37:280
1704899301,13cyclictest1330-21idleruntime-cro09:00:010
1704899301,13cyclictest11543-21chrt08:07:440
290652290,4sleep011-21rcuc/010:00:210
1705699298,18cyclictest833-21runrttasks12:37:061
1705699291,4cyclictest3165-21/usr/sbin/munin09:00:271
1705699291,4cyclictest31130-21/usr/sbin/munin07:40:221
1705699291,4cyclictest1986-21netstat07:45:191
1705699291,4cyclictest16433-21/usr/sbin/munin10:45:231
1705699291,2cyclictest0-21swapper/107:35:121
17056992910,16cyclictest8623-21ntp_states08:00:231
17056992910,16cyclictest833-21runrttasks07:34:101
1705699290,11cyclictest0-21swapper/110:27:041
1704899291,14cyclictest0-21swapper/009:59:120
1704899291,13cyclictest0-21swapper/011:44:300
1704899291,12cyclictest0-21swapper/009:06:160
1704899291,11cyclictest0-21swapper/007:32:560
1704899290,12cyclictest0-21swapper/012:29:510
17065992814,3cyclictest18370-21kworker/3:007:51:043
1705699281,4cyclictest28320-21/usr/sbin/munin10:00:141
1705699281,3cyclictest11983-21egrep08:10:141
1705699281,24cyclictest14028-21expr11:55:131
1704899281,18cyclictest0-21swapper/008:50:400
1704899281,13cyclictest0-21swapper/011:57:410
1704899281,13cyclictest0-21swapper/011:37:450
1704899281,12cyclictest0-21swapper/009:12:450
1704899281,11cyclictest0-21swapper/008:35:160
1704899281,11cyclictest0-21swapper/007:18:520
1704899281,10cyclictest0-21swapper/010:14:370
1706599277,3cyclictest18370-21kworker/3:007:10:163
1704899271,9cyclictest0-21swapper/008:47:120
1704899271,6cyclictest0-21swapper/009:29:250
1704899271,6cyclictest0-21swapper/009:29:250
1704899271,6cyclictest0-21swapper/009:15:440
1704899271,13cyclictest23366-21idleruntime-cro09:50:020
1704899271,12cyclictest0-21swapper/011:05:050
1704899271,12cyclictest0-21swapper/010:55:330
1704899271,12cyclictest0-21swapper/008:20:480
1704899271,11cyclictest0-21swapper/012:22:450
1704899271,11cyclictest0-21swapper/010:33:170
1704899271,11cyclictest0-21swapper/008:44:000
1704899271,10cyclictest0-21swapper/008:31:180
1704899270,11cyclictest0-21swapper/010:18:590
1706099260,11cyclictest0-21swapper/209:35:172
1704899261,9cyclictest0-21swapper/009:31:170
1704899261,9cyclictest0-21swapper/009:01:470
1704899261,12cyclictest0-21swapper/010:36:250
1704899261,11cyclictest0-21swapper/011:21:330
1704899261,11cyclictest0-21swapper/010:45:450
1704899261,10cyclictest0-21swapper/012:33:410
1704899261,10cyclictest0-21swapper/011:48:580
1704899261,10cyclictest0-21swapper/010:07:170
1704899261,10cyclictest0-21swapper/007:24:340
1704899258,9cyclictest0-21swapper/009:51:530
1704899253,15cyclictest3476-21expr07:50:130
1704899251,9cyclictest0-21swapper/011:51:450
1704899251,9cyclictest0-21swapper/011:51:450
1704899251,9cyclictest0-21swapper/009:22:080
1704899251,15cyclictest0-21swapper/011:10:290
1704899251,12cyclictest0-21swapper/012:36:170
1704899251,11cyclictest0-21swapper/011:27:410
1704899250,17cyclictest0-21swapper/007:12:460
1706599241,11cyclictest18370-21kworker/3:009:26:003
1706599241,11cyclictest18370-21kworker/3:009:26:003
17065992410,3cyclictest18370-21kworker/3:010:24:413
17065992410,2cyclictest18370-21kworker/3:010:05:253
1704899246,3cyclictest0-21swapper/012:15:570
1704899241,14cyclictest0-21swapper/008:02:580
1704899241,11cyclictest0-21swapper/012:03:410
1704899240,12cyclictest0-21swapper/010:22:130
1706599232,12cyclictest18370-21kworker/3:009:18:143
1706599232,12cyclictest18370-21kworker/3:007:41:283
1706599232,10cyclictest18370-21kworker/3:012:20:133
17065992318,3cyclictest4692-40dnf11:33:503
17065992312,2cyclictest18370-21kworker/3:008:29:283
1704899231,9cyclictest0-21swapper/010:41:130
1704899231,9cyclictest0-21swapper/010:41:130
1704899230,9cyclictest0-21swapper/011:01:430
301492220,4sleep339-21rcuc/310:05:023
1706599224,3cyclictest86750irq/128-enp1s0-11:25:183
1706599221,5cyclictest0-21swapper/309:53:093
1706599221,10cyclictest0-21swapper/309:35:163
1704899228,3cyclictest3509-21hddtemp_smartct11:30:170
1704899221,9cyclictest0-21swapper/009:38:400
1706599215,7cyclictest18370-21kworker/3:010:56:193
1706599213,10cyclictest26408-21/usr/sbin/munin11:10:243
1706599212,2cyclictest18370-21kworker/3:012:10:133
17065992113,5cyclictest836-21lldpd09:45:263
1706599210,11cyclictest18370-21kworker/3:007:29:043
1706099216,4cyclictest0-21swapper/209:42:032
1706099215,3cyclictest86850irq/129-enp1s0-10:05:142
1706099214,4cyclictest0-21swapper/210:15:152
1706599209,2cyclictest18370-21kworker/3:008:40:163
1706599207,2cyclictest18370-21kworker/3:007:35:043
1706599203,10cyclictest18370-21kworker/3:012:35:213
1706599202,10cyclictest7559-21/usr/sbin/munin08:00:273
1706599201,8cyclictest18370-21kworker/3:008:59:443
1706599201,7cyclictest18370-21kworker/3:010:32:253
1706599201,6cyclictest26741-21sed09:55:193
1706599201,11cyclictest31460-21sshd08:54:253
1706599200,8cyclictest18370-21kworker/3:008:12:243
1706099207,3cyclictest510-21unixbench_singl12:35:252
1706099205,7cyclictest0-21swapper/209:17:012
1706099205,3cyclictest0-21swapper/208:24:302
1706099204,3cyclictest86850irq/129-enp1s0-12:13:422
1706099202,7cyclictest11402-21munin-run11:50:012
1706099202,15cyclictest25164-21tr07:25:152
17060992015,3cyclictest10273-21grep10:30:252
322952190,4sleep230-21rcuc/207:40:222
1706599198,2cyclictest18370-21kworker/3:009:43:043
1706599197,4cyclictest851-21sssd_nss11:41:263
1706599194,6cyclictest10556-21sed11:45:213
1706599192,4cyclictest18370-21kworker/3:011:35:093
1706599191,10cyclictest6607-21/usr/sbin/munin10:25:133
1706599191,10cyclictest15403-21/usr/sbin/munin10:45:203
1706599191,10cyclictest14163-21/usr/sbin/munin08:15:173
1706599190,8cyclictest18370-21kworker/3:011:02:493
1706099199,3cyclictest86850irq/129-enp1s0-07:35:232
1706099197,3cyclictest25771-1kworker/u9:010:14:162
1706099194,12cyclictest5330-21grep09:05:252
1706099193,10cyclictest812-21snmpd11:33:502
1706099191,9cyclictest31351-21sed12:35:032
1706099191,6cyclictest23543-21memory12:15:202
17060991914,3cyclictest11733-21turbostat.cron08:10:012
17060991913,3cyclictest27698-21df_inode08:45:142
1706099191,2cyclictest0-21swapper/211:16:462
17060991912,4cyclictest30024-21fschecks_time08:50:162
1706099191,15cyclictest25547-21munin-run09:55:012
1706099191,10cyclictest22692-21diskstats07:20:132
1706099191,10cyclictest15843-21ntpq09:30:202
1706099190,4cyclictest0-21swapper/210:21:272
1706099190,11cyclictest0-21swapper/208:55:072
1706599187,3cyclictest812-21snmpd12:19:063
1706599185,4cyclictest18370-21kworker/3:010:11:443
1706599185,2cyclictest39-21rcuc/310:40:173
1706599185,2cyclictest39-21rcuc/310:40:173
1706599184,7cyclictest86750irq/128-enp1s0-07:57:323
1706599184,3cyclictest18370-21kworker/3:009:14:563
1706599183,3cyclictest86850irq/129-enp1s0-12:32:533
1706599181,7cyclictest0-21swapper/310:53:123
17065991813,3cyclictest2552-21users07:45:263
17065991812,4cyclictest3733-21cut09:05:013
17065991812,3cyclictest3679-21sed10:15:253
17065991812,3cyclictest19613-21ntpq07:15:033
1706099189,2cyclictest21066-21smartctl10:55:252
1706099188,2cyclictest17599-21diskstats07:10:132
1706099185,4cyclictest6846-21sshd07:55:262
1706099184,4cyclictest16223-21latency_hist08:20:012
1706099183,5cyclictest0-21swapper/209:20:132
1706099182,5cyclictest25771-1kworker/u9:011:22:172
1706099181,8cyclictest28986-21netstat10:00:192
1706099181,6cyclictest26794-21idleruntime-cro12:25:022
1706099181,6cyclictest0-21swapper/211:05:302
1706099181,5cyclictest851-21sssd_nss10:00:012
1706099181,4cyclictest0-21swapper/211:28:532
17060991811,4cyclictest0-21swapper/208:10:262
200372171,2sleep20-21swapper/212:08:022
1706599172,5cyclictest31608-21irqstats11:20:183
1706599171,6cyclictest0-21swapper/312:06:293
1706599171,4cyclictest0-21swapper/310:35:193
17065991712,3cyclictest836-21lldpd11:05:273
1706599170,3cyclictest86850irq/129-enp1s0-08:33:363
1706099172,11cyclictest2160-21tune2fs09:00:162
1706099171,7cyclictest6905-21memory09:10:202
1706099171,7cyclictest12180-21processes10:35:232
1706099171,6cyclictest19865-21sed08:25:262
1706099171,6cyclictest16472-21ntp_states10:45:252
1706099171,3cyclictest86550irq/126-enp1s0-11:13:162
1706099171,3cyclictest0-21swapper/211:00:232
17060991712,3cyclictest24930-21latency_hist08:40:022
1706099171,15cyclictest0-21swapper/208:04:152
1706099171,15cyclictest0-21swapper/207:46:212
17060991711,3cyclictest7837-21gltestperf11:40:132
17060991711,3cyclictest6607-21/usr/sbin/munin10:25:122
17060991711,3cyclictest6127-21open_inodes11:35:242
17060991711,3cyclictest27237-21expr12:25:152
17060991711,3cyclictest21655-21unixbench_multi07:15:252
17060991711,3cyclictest13543-21df_inode10:40:132
17060991711,3cyclictest13543-21df_inode10:40:132
1706099171,10cyclictest0-21swapper/209:26:392
1706099171,10cyclictest0-21swapper/209:26:392
1706599163,4cyclictest22365-21grep07:20:123
1706599163,2cyclictest86650irq/127-enp1s0-11:58:053
1706099161,6cyclictest0-21swapper/210:50:192
1706099161,6cyclictest0-21swapper/209:45:052
1706099161,2cyclictest0-21swapper/212:00:202
17060991611,3cyclictest21577-21kernelversion08:30:172
17060991611,3cyclictest15406-21uptime11:55:252
1706099161,12cyclictest3577-21fschecks_count07:50:142
17060991610,3cyclictest25362-21cat08:40:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional