You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 17:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Mar 26, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28182241163,25sleep20-21swapper/207:06:042
28172241164,24sleep10-21swapper/107:06:041
29082238164,24sleep30-21swapper/307:07:133
175522180,13sleep011-21rcuc/007:05:160
770721790,4sleep1321499cyclictest12:10:231
477921310,3sleep0320999cyclictest07:10:230
3226998063,15cyclictest0-21swapper/308:20:193
209742660,1sleep10-21swapper/110:15:131
166512640,2sleep216650-21grep07:35:252
3226996357,5cyclictest0-21swapper/310:08:123
3226995956,2cyclictest0-21swapper/308:32:043
120682590,2sleep30-21swapper/308:40:163
3226994946,2cyclictest0-21swapper/308:19:133
249122480,4sleep230-21rcuc/211:35:242
54382380,3sleep15441-21if_enp1s008:25:171
3226993625,2cyclictest25495-21kworker/3:210:25:223
320999347,13cyclictest19863-21awk09:00:010
3226993228,3cyclictest0-21swapper/311:07:303
3226993228,3cyclictest0-21swapper/311:07:293
3226993228,3cyclictest0-21swapper/310:47:573
320999321,15cyclictest24262-21munin-run09:10:010
322699312,18cyclictest25495-21kworker/3:208:02:253
3226993117,4cyclictest25495-21kworker/3:208:07:493
322662310,3sleep3322699cyclictest10:40:153
320999313,13cyclictest21445-21munin-run07:50:010
322699302,11cyclictest25495-21kworker/3:208:52:223
322699301,19cyclictest25495-21kworker/3:207:59:373
3226992925,3cyclictest0-21swapper/312:35:383
322699291,17cyclictest25495-21kworker/3:208:36:183
322699291,10cyclictest25495-21kworker/3:210:30:183
321499296,16cyclictest5176-21sed09:40:011
322699280,10cyclictest25495-21kworker/3:211:14:593
3226992712,3cyclictest25495-21kworker/3:209:50:223
321499274,7cyclictest13408-21expr12:25:121
3214992714,4cyclictest17468-40dnf08:52:491
320999272,12cyclictest6339-21awk12:10:010
322699267,3cyclictest25495-21kworker/3:211:02:193
321999261,11cyclictest0-21swapper/210:20:022
321499265,6cyclictest0-21swapper/111:53:511
3214992610,14cyclictest7883-21kworker/u8:011:25:211
322699259,3cyclictest25495-21kworker/3:208:49:273
322699259,3cyclictest25495-21kworker/3:208:49:263
322699252,11cyclictest25495-21kworker/3:211:47:423
322699251,10cyclictest25495-21kworker/3:208:12:173
3226992510,3cyclictest25495-21kworker/3:212:25:303
3226992510,2cyclictest29994-21kworker/3:007:17:353
3219992512,5cyclictest311ktimersoftd/209:45:142
321999251,10cyclictest0-21swapper/211:50:202
321499257,7cyclictest25771-1kworker/u9:010:36:071
320999252,4cyclictest31100-21kworker/0:110:36:070
320999251,5cyclictest0-21swapper/010:43:290
320999250,12cyclictest28664-21awk09:20:010
265672250,2sleep10-21swapper/110:25:241
322699242,8cyclictest25495-21kworker/3:209:19:023
3226992420,3cyclictest0-21swapper/307:46:323
3226992414,2cyclictest29994-21kworker/3:007:33:253
3226992412,3cyclictest29994-21kworker/3:007:55:013
3226992410,3cyclictest29994-21kworker/3:007:20:053
3226992410,3cyclictest25495-21kworker/3:211:18:343
321999243,4cyclictest32-21ksoftirqd/211:31:052
321499244,8cyclictest15060-21df11:15:121
321499244,2cyclictest11982-21kworker/u8:112:00:021
3214992416,5cyclictest11890-21grep11:05:311
3214992416,5cyclictest11890-21grep11:05:311
321499241,16cyclictest0-21swapper/111:47:501
321499240,3cyclictest0-21swapper/111:11:081
321499240,18cyclictest0-21swapper/109:15:161
178982240,2sleep20-21swapper/208:55:112
322699232,8cyclictest25495-21kworker/3:210:18:583
322699232,7cyclictest25495-21kworker/3:212:03:543
322699232,10cyclictest29994-21kworker/3:007:43:093
322699231,8cyclictest25495-21kworker/3:212:06:383
322699231,10cyclictest25495-21kworker/3:209:03:293
3226992310,3cyclictest25495-21kworker/3:212:12:103
3226992310,2cyclictest29994-21kworker/3:007:25:453
321999236,8cyclictest7027-21sort12:10:152
321999234,9cyclictest20482-21perf10:15:012
321499238,2cyclictest86750irq/128-enp1s0-10:30:181
321499231,9cyclictest2191-21tail10:45:171
321499231,4cyclictest7579-21hddtemp_smartct08:30:161
320999235,11cyclictest17474-21chrt08:52:490
320999231,8cyclictest10356-21grep09:50:160
3209992316,4cyclictest30103-21/usr/sbin/munin11:50:110
320999231,10cyclictest0-21swapper/010:14:430
322699229,4cyclictest0-21swapper/311:57:273
322699229,3cyclictest25495-21kworker/3:212:34:303
322699229,3cyclictest25495-21kworker/3:208:26:103
322699227,3cyclictest25495-21kworker/3:209:10:413
322699225,7cyclictest0-21swapper/312:21:563
322699225,7cyclictest0-21swapper/312:21:563
322699222,12cyclictest29994-21kworker/3:007:14:013
3226992219,2cyclictest0-21swapper/311:20:333
322699221,16cyclictest25495-21kworker/3:211:44:063
322699221,12cyclictest888-21/usr/sbin/munin09:30:133
3219992211,4cyclictest86750irq/128-enp1s0-10:40:032
321499228,12cyclictest5416-21kworker/u8:108:45:041
321499228,12cyclictest5416-21kworker/u8:108:45:041
321499228,12cyclictest31447-21kworker/u8:110:54:411
321499227,7cyclictest0-21swapper/109:10:101
321499225,7cyclictest0-21swapper/109:50:201
321499222,14cyclictest86750irq/128-enp1s0-12:09:231
321499221,5cyclictest17452-21grep10:05:221
321499221,3cyclictest21138-21ntp_states09:00:201
321499221,3cyclictest17455-21fschecks_time11:20:141
321499221,2cyclictest86750irq/128-enp1s0-11:41:091
321499221,18cyclictest22739-21ntp_states07:50:211
321499221,18cyclictest1094-21irqstats08:15:181
321499221,15cyclictest0-21swapper/110:02:571
3214992210,3cyclictest8276-21munin-run07:20:001
320999228,3cyclictest0-21swapper/007:20:180
320999220,4cyclictest3326-21kworker/u8:210:51:310
322699219,2cyclictest25495-21kworker/3:211:33:223
322699218,2cyclictest25495-21kworker/3:209:06:013
322699217,3cyclictest25495-21kworker/3:210:53:143
322699217,3cyclictest25495-21kworker/3:210:35:343
322699217,3cyclictest25495-21kworker/3:209:26:303
322699213,3cyclictest0-21swapper/311:38:193
322699211,4cyclictest9495-21perf09:50:013
322699210,9cyclictest0-21swapper/310:58:483
321999215,8cyclictest2845-21/usr/sbin/munin08:20:132
321999214,7cyclictest25833-21latency_hist08:00:032
321999213,7cyclictest311ktimersoftd/207:50:022
321999211,9cyclictest250232cut09:10:142
321999210,8cyclictest11982-21kworker/u8:111:30:022
321499217,12cyclictest6190-21kworker/u8:012:18:401
321499213,13cyclictest86750irq/128-enp1s0-11:58:001
321499211,2cyclictest6255-21cat09:40:201
321499211,2cyclictest241062sleep109:08:191
321499211,18cyclictest0-21swapper/109:46:301
321499211,12cyclictest0-21swapper/110:25:011
3214992110,3cyclictest9042-21memory11:00:181
3214992110,3cyclictest86750irq/128-enp1s0-11:32:341
321499210,6cyclictest0-21swapper/109:28:181
320999218,3cyclictest6845-21memory10:55:170
320999218,11cyclictest22686-21perf10:20:010
320999216,12cyclictest8524-21munin-run12:15:010
320999212,5cyclictest0-21swapper/009:20:250
320999211,9cyclictest0-21swapper/010:07:390
320999211,13cyclictest0-21swapper/011:00:170
320999211,11cyclictest28268-21/usr/sbin/munin08:05:130
320999211,11cyclictest0-21swapper/007:30:170
320999211,10cyclictest24250-21fschecks_count07:55:130
320999211,10cyclictest0-21swapper/008:40:160
322699208,3cyclictest25495-21kworker/3:209:21:263
322699201,6cyclictest0-21swapper/311:29:533
322699201,11cyclictest25495-21kworker/3:210:20:063
322699200,8cyclictest25495-21kworker/3:209:41:293
321999205,7cyclictest24265-21awk09:10:012
321999204,7cyclictest28268-21/usr/sbin/munin08:05:242
321999204,3cyclictest0-21swapper/212:27:442
321999202,8cyclictest86850irq/129-enp1s0-07:30:202
321999202,7cyclictest27910-21expr10:30:172
321999202,6cyclictest0-21swapper/212:21:162
321999202,6cyclictest0-21swapper/212:21:162
321999201,7cyclictest0-21swapper/209:52:592
321999201,16cyclictest8796-21sed11:00:152
321999201,16cyclictest31156-21cat09:25:102
321999201,12cyclictest0-21swapper/209:22:552
321999201,11cyclictest0-21swapper/210:40:272
321499207,11cyclictest5416-21kworker/u8:108:21:161
321499206,7cyclictest0-21swapper/111:35:201
3214992017,2cyclictest0-21swapper/109:20:521
321499201,18cyclictest0-21swapper/109:32:591
321499201,17cyclictest0-21swapper/110:42:041
321499201,17cyclictest0-21swapper/107:55:041
321499201,17cyclictest0-21swapper/107:40:161
321499201,10cyclictest0-21swapper/109:56:421
321499201,10cyclictest0-21swapper/109:56:421
321499200,9cyclictest0-21swapper/112:39:301
321499200,7cyclictest0-21swapper/112:33:581
321499200,5cyclictest0-21swapper/112:20:441
321499200,5cyclictest0-21swapper/112:20:431
320999209,3cyclictest15085-21/usr/sbin/munin07:35:140
320999205,4cyclictest0-21swapper/009:46:090
320999202,7cyclictest10379-21cpu11:05:100
320999202,7cyclictest10379-21cpu11:05:100
320999201,9cyclictest13698-21ps11:10:220
320999201,7cyclictest0-21swapper/011:40:370
320999201,6cyclictest0-21swapper/012:01:030
320999201,12cyclictest0-21swapper/009:39:130
320999201,12cyclictest0-21swapper/008:15:200
322699199,3cyclictest24253-21diskmemload09:35:163
322699195,3cyclictest29994-21kworker/3:007:38:573
322699191,12cyclictest0-21swapper/309:56:253
322699191,12cyclictest0-21swapper/309:56:253
322699191,11cyclictest0-21swapper/312:19:253
322699191,10cyclictest20475-21munin-run10:15:013
322699191,10cyclictest0-21swapper/310:00:143
321999199,3cyclictest86550irq/126-enp1s0-11:40:122
321999197,4cyclictest86550irq/126-enp1s0-12:03:202
321999195,8cyclictest30-21rcuc/207:25:122
321999192,7cyclictest15795-21ntp_states11:15:222
321999192,6cyclictest0-21swapper/209:39:442
321999192,2cyclictest86750irq/128-enp1s0-10:50:012
321999191,15cyclictest15992-21grep12:30:162
321999191,12cyclictest0-21swapper/212:06:332
321999191,10cyclictest32-21ksoftirqd/210:05:142
321999191,10cyclictest0-21swapper/212:38:112
321499192,3cyclictest86750irq/128-enp1s0-10:59:441
3214991915,3cyclictest0-21swapper/108:06:451
3214991912,2cyclictest0-21swapper/107:10:171
321499191,17cyclictest0-21swapper/107:36:031
321499191,16cyclictest0-21swapper/107:21:421
321499191,15cyclictest18639-21tr10:10:121
321499191,10cyclictest0-21swapper/108:55:201
321499190,15cyclictest0-21swapper/108:01:041
320999196,11cyclictest86550irq/126-enp1s0-11:35:120
320999191,9cyclictest0-21swapper/010:24:580
320999191,8cyclictest0-21swapper/011:25:400
320999191,8cyclictest0-21swapper/009:00:250
320999191,7cyclictest27299-21/usr/sbin/munin10:30:230
320999191,6cyclictest0-21swapper/012:15:180
320999191,6cyclictest0-21swapper/011:20:400
320999191,5cyclictest0-21swapper/011:31:060
320999191,5cyclictest0-21swapper/009:58:170
320999191,5cyclictest0-21swapper/009:58:170
320999191,4cyclictest14880-21/usr/sbin/munin10:00:160
320999191,10cyclictest786-21NetworkManager12:29:140
320999191,10cyclictest0-21swapper/010:25:170
320999190,9cyclictest22058-21fschecks_count07:50:140
135162191,2sleep20-21swapper/209:55:322
135162191,2sleep20-21swapper/209:55:322
42432180,3sleep00-21swapper/008:22:080
32562180,2sleep00-21swapper/010:46:590
3226991812,3cyclictest25495-21kworker/3:208:55:293
322699181,10cyclictest0-21swapper/311:50:593
321999187,3cyclictest118542fschecks_time08:40:132
321999185,4cyclictest6408-21sshd08:26:442
321999184,8cyclictest32-21ksoftirqd/209:31:402
321999184,6cyclictest0-21swapper/210:55:162
321999182,15cyclictest0-21swapper/211:05:262
321999182,15cyclictest0-21swapper/211:05:262
321999181,7cyclictest27235-21perl09:15:162
321999181,7cyclictest0-21swapper/211:12:392
321999181,7cyclictest0-21swapper/210:23:492
321999181,6cyclictest0-21swapper/208:00:182
3219991813,3cyclictest18886-21sh11:25:012
3219991813,3cyclictest17618-21df07:40:122
3219991811,4cyclictest1358-21sh08:15:222
321999181,11cyclictest0-21swapper/210:04:252
321499182,13cyclictest10668-21head08:35:251
321499181,16cyclictest0-21swapper/108:44:191
321499181,16cyclictest0-21swapper/108:11:281
3214991810,3cyclictest0-21swapper/107:46:471
321499180,16cyclictest0-21swapper/107:31:141
320999182,7cyclictest10691-21/usr/sbin/munin07:25:130
320999181,9cyclictest26769-21idleruntime08:00:160
320999181,9cyclictest0-21swapper/011:58:360
320999181,5cyclictest0-21swapper/009:10:240
3209991813,3cyclictest32638-21grep09:25:240
320999181,2cyclictest19084-21grep12:35:250
3209991812,3cyclictest18713-21grep07:40:230
3209991812,3cyclictest14153-21sed08:45:150
3209991812,3cyclictest14153-21sed08:45:150
320999181,10cyclictest0-21swapper/009:40:090
320999181,10cyclictest0-21swapper/008:25:180
171752181,2sleep00-21swapper/012:33:300
14802181,2sleep00-21swapper/009:30:190
50542170,2sleep20-21swapper/207:10:582
321999179,3cyclictest0-21swapper/209:40:052
321999179,3cyclictest0-21swapper/209:01:452
321999178,2cyclictest30-21rcuc/208:30:122
321999176,2cyclictest86750irq/128-enp1s0-11:45:182
321999174,2cyclictest86850irq/129-enp1s0-11:55:202
321999173,7cyclictest86550irq/126-enp1s0-07:20:172
321999171,8cyclictest6369-21cpuspeed_turbos07:15:132
321999171,7cyclictest5700-21latency_hist10:55:012
321999171,6cyclictest0-21swapper/212:16:282
321999171,3cyclictest25095-21/usr/sbin/munin10:25:142
321999171,13cyclictest22316-21sed07:50:162
321499171,2cyclictest0-21swapper/107:25:031
320999178,3cyclictest86750irq/128-enp1s0-08:10:180
320999171,9cyclictest0-21swapper/008:32:430
320999171,5cyclictest7-21ksoftirqd/012:20:230
320999171,5cyclictest7-21ksoftirqd/012:20:230
320999171,4cyclictest14725-21/usr/sbin/munin11:15:140
3209991712,3cyclictest28166-21expr11:45:120
3209991711,3cyclictest7006-21basename07:15:170
107612171,2sleep20-21swapper/208:36:142
321999163,2cyclictest13487-21/usr/sbin/munin08:45:182
321999163,2cyclictest13487-21/usr/sbin/munin08:45:182
321999161,14cyclictest0-21swapper/208:50:212
3209991611,3cyclictest10061-21cat08:35:190
321999154,3cyclictest12050irq/123-ahci[0008:10:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional