You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-03 - 12:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Aug 03, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
169622233162,23sleep20-21swapper/219:06:012
1540622300,12sleep10-21swapper/119:05:151
171582229162,54sleep30-21swapper/319:08:353
171142203165,25sleep00-21swapper/019:08:000
1053321780,5sleep31736899cyclictest21:20:023
1667821740,3sleep31736899cyclictest00:00:143
2233321300,4sleep31736899cyclictest20:30:263
17362998682,3cyclictest0-21swapper/223:17:212
319492740,1sleep30-21swapper/320:55:023
1735299486,11cyclictest23671-21basename21:50:000
302922470,2sleep1221ktimersoftd/123:15:261
44532450,1sleep20-21swapper/219:50:202
188292400,2sleep30-21swapper/300:05:173
17352993910,13cyclictest17682-21munin-run22:50:010
17362993836,1cyclictest0-21swapper/221:46:042
1735899371,9cyclictest0-21swapper/121:50:561
1735299356,14cyclictest18315-21munin-run00:05:000
1735899341,15cyclictest0-21swapper/123:35:161
1735899331,14cyclictest0-21swapper/100:20:311
1735899321,25cyclictest0-21swapper/100:02:121
1735899301,8cyclictest0-21swapper/122:27:111
1735899301,6cyclictest0-21swapper/122:19:231
1735899301,12cyclictest0-21swapper/119:37:181
1735899301,11cyclictest0-21swapper/119:50:221
1735299302,17cyclictest27748-21kworker/0:121:59:330
1735899291,12cyclictest0-21swapper/122:24:511
1735899291,12cyclictest0-21swapper/121:29:201
1735899291,10cyclictest30799-21smart_sda00:30:251
1735899290,12cyclictest0-21swapper/123:43:341
1735299292,11cyclictest11513-21kworker/0:223:53:260
1735299291,9cyclictest24695-21kworker/0:000:21:130
1735899281,9cyclictest0-21swapper/120:46:031
1735899281,9cyclictest0-21swapper/100:09:121
1735899281,8cyclictest0-21swapper/100:19:551
1735899281,6cyclictest0-21swapper/121:32:041
17358992812,3cyclictest0-21swapper/123:53:121
1735899281,13cyclictest572-21needreboot20:55:211
1735899281,12cyclictest0-21swapper/119:42:561
1735899281,12cyclictest0-21swapper/119:27:561
1735899281,10cyclictest0-21swapper/123:45:281
1735899281,10cyclictest0-21swapper/121:49:161
1735899281,10cyclictest0-21swapper/120:10:581
1735899281,10cyclictest0-21swapper/120:06:161
1735899280,13cyclictest0-21swapper/119:20:541
17352992812,2cyclictest31528-21kworker/0:321:05:180
1735299281,12cyclictest26828-21awk19:30:010
1735299281,12cyclictest22089-21basename23:00:010
1735299281,10cyclictest32213-21kworker/0:022:13:570
1735299281,10cyclictest13733-21kworker/0:023:56:370
1735299281,10cyclictest13489-21kworker/0:022:40:340
1735299281,10cyclictest10106-21kworker/0:121:18:380
1735899279,3cyclictest0-21swapper/121:35:041
1735899271,9cyclictest0-21swapper/119:32:021
1735899271,12cyclictest0-21swapper/120:25:031
1735899271,12cyclictest0-21swapper/119:55:401
1735899271,11cyclictest0-21swapper/123:57:001
1735899271,11cyclictest0-21swapper/120:03:121
1735899271,10cyclictest0-21swapper/123:09:231
1735899271,10cyclictest0-21swapper/100:29:151
1735899270,12cyclictest0-21swapper/123:10:141
1735299278,2cyclictest2179-21kworker/0:222:22:090
1735299272,10cyclictest31115-21kworker/0:019:46:370
1735299271,10cyclictest29150-21kworker/0:000:29:530
1736299261,12cyclictest0-21swapper/223:43:592
1736299260,11cyclictest0-21swapper/200:36:352
1735899268,2cyclictest0-21swapper/123:26:121
1735899261,9cyclictest0-21swapper/121:57:361
1735899261,11cyclictest0-21swapper/121:17:201
1735899261,11cyclictest0-21swapper/120:44:311
1735899261,10cyclictest0-21swapper/122:32:231
1735899261,10cyclictest0-21swapper/119:11:171
1735899261,10cyclictest0-21swapper/100:35:311
1735899260,12cyclictest0-21swapper/121:06:031
1735299262,9cyclictest14636-21kworker/0:321:31:180
1735299262,11cyclictest31212-21munin-run19:40:020
1735299261,10cyclictest24284-21kworker/0:219:32:140
1736299251,9cyclictest0-21swapper/222:54:032
1736299251,8cyclictest0-21swapper/200:13:522
1736299251,8cyclictest0-21swapper/200:13:522
1736299250,12cyclictest0-21swapper/223:36:352
1735899251,9cyclictest0-21swapper/122:36:031
1735899251,9cyclictest0-21swapper/121:41:121
1735899251,14cyclictest0-21swapper/122:10:291
1735899251,11cyclictest0-21swapper/123:21:041
1735899250,11cyclictest0-21swapper/120:54:291
1735899250,10cyclictest0-21swapper/122:00:141
1735299259,14cyclictest4502-21perf22:20:010
1735299252,9cyclictest22673-21kworker/0:120:33:210
1735299251,10cyclictest7273-21kworker/0:020:00:370
1735299251,10cyclictest24148-21kworker/0:223:38:380
17352992510,2cyclictest17554-21kworker/0:319:12:370
1736899241,5cyclictest0-21swapper/321:52:563
1736299241,7cyclictest0-21swapper/219:40:032
1735899241,8cyclictest0-21swapper/122:43:331
1735899241,15cyclictest0-21swapper/121:21:241
1735899241,15cyclictest0-21swapper/121:10:391
1735899241,13cyclictest0-21swapper/120:35:151
1735899241,13cyclictest0-21swapper/120:35:151
1735899241,11cyclictest0-21swapper/122:59:511
1735899241,11cyclictest0-21swapper/120:30:591
1735899241,10cyclictest0-21swapper/119:15:271
1735899240,9cyclictest0-21swapper/120:15:531
1735299241,9cyclictest27070-21kworker/0:120:51:460
1735299241,9cyclictest24148-21kworker/0:223:09:220
1735299241,9cyclictest18575-21kworker/0:020:35:450
1735299241,9cyclictest18575-21kworker/0:020:35:450
1735299241,9cyclictest17310-21kworker/0:123:02:380
1735299241,9cyclictest17310-21kworker/0:123:02:380
1735299241,8cyclictest18533-21kworker/0:000:09:090
17352992411,2cyclictest16954-21kworker/0:019:15:140
1735299241,11cyclictest2980-21kworker/0:220:09:050
1735299241,10cyclictest14636-21kworker/0:321:42:020
17352992410,3cyclictest14636-21kworker/0:321:39:380
17352992410,2cyclictest2980-21kworker/0:220:15:370
1735299240,10cyclictest6050-21kworker/0:021:10:420
1736299231,9cyclictest0-21swapper/223:56:402
1736299230,9cyclictest0-21swapper/200:05:442
1735899239,9cyclictest0-21swapper/120:20:191
1735899235,3cyclictest0-21swapper/121:00:111
1735899231,9cyclictest0-21swapper/123:30:561
1735899231,9cyclictest0-21swapper/122:07:311
17358992314,2cyclictest16687-21strings22:45:181
1735899231,10cyclictest0-21swapper/122:52:031
1735299238,3cyclictest28328-21kworker/0:023:13:540
1735299236,5cyclictest2179-21kworker/0:222:34:460
1735299235,3cyclictest2179-21kworker/0:222:29:500
1735299232,12cyclictest12666-21munin-run21:25:010
1735299231,9cyclictest31407-21kworker/0:100:34:220
1735299231,8cyclictest14636-21kworker/0:321:27:540
1735299231,10cyclictest11513-21kworker/0:200:17:250
17352992310,3cyclictest31459-21kworker/0:300:35:300
1735299230,9cyclictest29964-21kworker/0:322:04:450
1736299221,6cyclictest0-21swapper/223:10:552
1736299221,5cyclictest0-21swapper/221:40:442
1736299220,8cyclictest0-21swapper/220:10:162
1735899228,2cyclictest0-21swapper/123:00:171
1735899228,2cyclictest0-21swapper/123:00:171
1735899221,6cyclictest0-21swapper/119:46:201
1735299226,5cyclictest24148-21kworker/0:223:17:020
1735299226,4cyclictest9288-21kworker/0:023:44:300
1735299226,4cyclictest18575-21kworker/0:020:25:130
1735299224,3cyclictest2179-21kworker/0:222:37:380
1735299222,10cyclictest2542-21kworker/0:323:29:260
1735299222,10cyclictest1550-21kworker/0:021:03:140
1735299221,9cyclictest11513-21kworker/0:200:12:250
1735299221,9cyclictest11513-21kworker/0:200:12:250
1735299221,10cyclictest28793-21kworker/0:319:51:090
17352992210,3cyclictest16359-21kworker/0:320:20:130
1736899214,3cyclictest41-21ksoftirqd/319:40:013
1736899211,13cyclictest0-21swapper/323:45:123
1736899211,13cyclictest0-21swapper/300:15:183
1736899211,11cyclictest0-21swapper/323:50:273
1736299212,16cyclictest17944-21cut21:35:172
1736299211,6cyclictest0-21swapper/222:11:392
1736299211,5cyclictest0-21swapper/222:05:262
1735899214,3cyclictest0-21swapper/100:14:291
1735899214,3cyclictest0-21swapper/100:14:291
1735299218,2cyclictest30538-21kworker/0:123:20:200
17352992116,2cyclictest27070-21kworker/0:120:45:020
1735299211,10cyclictest24284-21kworker/0:219:24:140
1735299210,13cyclictest27070-21kworker/0:120:55:180
1736899209,3cyclictest87650irq/129-enp1s0-21:10:163
1736899207,3cyclictest21588-21latency_hist21:45:013
1736899205,6cyclictest0-21swapper/321:30:263
1736899204,9cyclictest41-21ksoftirqd/322:20:233
1736899203,14cyclictest28827-21cut22:00:163
1736899202,15cyclictest16087-21grep00:00:003
1736899201,6cyclictest11616-21expr22:35:133
1736899201,4cyclictest0-21swapper/322:18:013
17368992013,4cyclictest12051-21sleep20:10:023
1736899201,11cyclictest4098-21/usr/sbin/munin21:05:203
1736899201,11cyclictest27341-21/usr/sbin/munin00:25:133
1736899201,11cyclictest1633-21sshd20:59:553
1736899201,10cyclictest0-21swapper/320:25:273
1736299204,7cyclictest0-21swapper/221:16:022
1736299201,7cyclictest25900-21ntp_offset19:25:212
1736299201,6cyclictest0-21swapper/223:08:122
1736299201,4cyclictest0-21swapper/221:30:502
1736299201,4cyclictest0-21swapper/200:27:582
17362992014,4cyclictest24335-21ntp_states20:35:242
17362992014,4cyclictest24335-21ntp_states20:35:242
1736299201,10cyclictest10401-21hddtemp_smartct23:45:172
1735299206,4cyclictest7273-21kworker/0:019:58:290
1735299203,4cyclictest17310-21kworker/0:122:54:220
1735299201,7cyclictest10759-21ntpq23:45:210
1735299201,10cyclictest27070-21kworker/0:120:45:420
177642190,4sleep230-21rcuc/200:00:262
1736899198,3cyclictest27658-21/usr/sbin/munin20:45:273
1736899198,3cyclictest0-21swapper/320:15:223
1736899197,11cyclictest87350irq/126-enp1s0-21:20:263
1736899195,7cyclictest41-21ksoftirqd/300:40:023
1736899195,6cyclictest755-21rngd23:04:293
1736899195,6cyclictest755-21rngd23:04:293
1736899193,7cyclictest401ktimersoftd/320:25:013
1736899192,4cyclictest3204-21ls21:00:253
1736899191,9cyclictest0-21swapper/300:13:173
1736899191,9cyclictest0-21swapper/300:13:173
1736899191,8cyclictest0-21swapper/321:45:133
1736899191,7cyclictest7868-21acpi20:00:133
1736899191,7cyclictest0-21swapper/321:56:063
1736899191,6cyclictest0-21swapper/323:42:173
17368991912,4cyclictest31741-21grep23:20:153
1736899191,11cyclictest839-21lldpd21:38:163
1736899191,10cyclictest6918-21/usr/sbin/munin22:25:243
1736899191,10cyclictest29534-21/usr/sbin/munin00:30:173
1736899191,10cyclictest24507-21/usr/sbin/munin23:05:153
1736299199,3cyclictest87650irq/129-enp1s0-22:15:342
1736299195,3cyclictest87650irq/129-enp1s0-21:25:252
1736299194,3cyclictest23376-21open_files23:00:222
1736299194,3cyclictest23376-21open_files23:00:222
1736299191,9cyclictest0-21swapper/200:33:352
1736299191,3cyclictest0-21swapper/222:56:062
17362991912,4cyclictest20798-21diskstats19:15:142
1736299191,10cyclictest0-21swapper/222:40:072
1735299192,10cyclictest19196-21kworker/0:021:53:090
1735299191,13cyclictest2980-21kworker/0:220:13:330
1735299191,10cyclictest30483-21/usr/sbin/munin22:05:180
1736899188,3cyclictest22649-21/usr/sbin/munin19:20:193
1736899185,3cyclictest18245-1kworker/u9:222:08:383
1736899182,14cyclictest401ktimersoftd/319:16:233
1736899181,9cyclictest32676-21/usr/sbin/munin22:10:243
1736899181,9cyclictest0-21swapper/322:48:303
1736899181,5cyclictest0-21swapper/322:54:213
1736899181,3cyclictest0-21swapper/323:33:393
17368991812,4cyclictest26489-21grep00:20:243
1736899181,10cyclictest808-21snmpd19:46:553
1736899180,5cyclictest0-21swapper/322:42:133
1736299188,2cyclictest87550irq/128-enp1s0-20:45:212
1736299187,9cyclictest7646-21awk20:00:022
1736299186,3cyclictest4640-21fschecks_time21:05:152
1736299183,6cyclictest0-21swapper/200:15:072
1736299183,5cyclictest0-21swapper/222:21:122
1736299181,7cyclictest12535-21gltestperf23:50:172
1736299181,7cyclictest0-21swapper/219:45:572
1736299181,6cyclictest2882-21needreboot21:00:192
1736299181,6cyclictest0-21swapper/222:00:452
1736299181,4cyclictest0-21swapper/222:37:022
17362991812,3cyclictest25602-21sshd21:51:552
1736299180,10cyclictest0-21swapper/223:23:152
1735299186,3cyclictest0-21swapper/023:31:080
1735299182,10cyclictest28793-21kworker/0:319:41:380
270802171,2sleep20-21swapper/200:24:472
1736899174,3cyclictest0-21swapper/323:10:233
1736899173,4cyclictest10037-21ls22:30:203
1736899171,9cyclictest0-21swapper/322:58:453
1736899171,6cyclictest0-21swapper/323:17:403
1736899171,6cyclictest0-21swapper/319:50:173
1736899171,14cyclictest0-21swapper/320:40:203
1736899171,10cyclictest0-21swapper/320:35:133
1736899171,10cyclictest0-21swapper/320:35:133
1736299176,3cyclictest87350irq/126-enp1s0-20:40:142
1736299171,7cyclictest31827-21munin-run20:55:012
1736299171,6cyclictest19090-21grep19:10:272
1736299171,6cyclictest0-21swapper/222:25:482
1736299171,6cyclictest0-21swapper/220:30:182
1736299171,4cyclictest23719-21ntp_states19:20:202
1736299171,3cyclictest0-21swapper/223:25:252
1736299171,2cyclictest87650irq/129-enp1s0-21:22:142
17362991712,3cyclictest28632-21users19:30:272
17362991712,3cyclictest18659-21basename20:25:012
17362991712,3cyclictest17015-21sed22:45:232
1736299171,15cyclictest0-21swapper/222:32:222
1736299170,14cyclictest8506-21gltestperf20:00:182
1736899162,11cyclictest12892-21/usr/sbin/munin21:25:173
1736899161,6cyclictest32109-21hddtemp_smartct19:40:143
1736899161,6cyclictest12268-21/usr/sbin/munin20:10:143
1736899161,5cyclictest0-21swapper/323:35:463
1736899161,4cyclictest0-21swapper/323:25:033
17368991612,3cyclictest0-21swapper/319:55:083
1736899161,14cyclictest0-21swapper/319:12:243
1736899160,10cyclictest0-21swapper/319:30:183
1736299169,4cyclictest1-21systemd20:16:542
1736299161,9cyclictest0-21swapper/221:55:242
1736299161,7cyclictest32308-21cstates20:55:142
17362991612,3cyclictest0-21swapper/221:12:402
17362991612,3cyclictest0-21swapper/220:07:452
17362991611,3cyclictest29238-21/usr/sbin/munin19:35:222
17362991610,3cyclictest3131-21/usr/sbin/munin23:30:162
1736899151,6cyclictest0-21swapper/319:25:223
17362991510,3cyclictest19639-21if_err_enp1s020:25:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional