You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-30 - 16:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Thu Jul 30, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
87832238165,24sleep20-21swapper/207:07:252
87252234163,58sleep10-21swapper/107:06:401
87492204169,23sleep00-21swapper/007:07:000
86922204167,25sleep30-21swapper/307:06:153
1745921560,5sleep3908699cyclictest12:20:143
9081998683,2cyclictest0-21swapper/211:22:002
9081997668,7cyclictest0-21swapper/209:21:542
67332750,2sleep10-21swapper/108:15:151
156802500,4sleep121-21rcuc/109:50:021
907199478,11cyclictest0-21swapper/009:55:010
907199461,13cyclictest30994-21munin-run10:25:000
907199461,13cyclictest30994-21munin-run10:25:000
312202460,2sleep30-21swapper/309:10:193
312202460,2sleep30-21swapper/309:10:193
907199436,12cyclictest20638-21perf11:15:010
908699411,29cyclictest9622-21perf10:50:023
9081994138,2cyclictest0-21swapper/208:11:422
908699403,24cyclictest31417-21cstates10:25:103
908699398,22cyclictest18646-21/usr/sbin/munin11:10:163
908699391,30cyclictest19303-21/usr/sbin/munin12:25:153
908699391,24cyclictest27534-21cpuspeed_turbos11:30:123
908699391,24cyclictest27534-21cpuspeed_turbos11:30:123
908699384,17cyclictest12050irq/123-ahci[0009:45:013
908699381,24cyclictest3918-21/usr/sbin/munin11:50:183
908699379,20cyclictest9267-21/usr/sbin/munin07:10:103
908699371,26cyclictest14406-21irqstats09:45:203
908699371,25cyclictest6700-21latency_hist09:30:013
908699370,29cyclictest0-21swapper/310:55:133
907199379,14cyclictest15563-21munin-run09:50:020
908699361,26cyclictest14971-21/usr/sbin/munin08:35:143
908699361,24cyclictest30131-21diskmemload11:41:513
908699361,24cyclictest0-21swapper/310:05:183
908699361,21cyclictest1533-21latency_hist08:05:013
908699361,21cyclictest1533-21latency_hist08:05:013
908699355,23cyclictest0-21swapper/311:05:243
908699351,31cyclictest29737-21/usr/sbin/munin07:55:183
908699351,21cyclictest25342-21/usr/sbin/munin07:45:243
908699345,21cyclictest7110-21missed_timers08:15:183
908699345,20cyclictest87350irq/126-enp1s0-11:25:233
908699341,23cyclictest7578-21unixbench-2d11:55:243
908699341,21cyclictest29355-21df10:20:123
908699341,21cyclictest29355-21df10:20:123
908699341,21cyclictest22256-21expr11:15:243
908699341,20cyclictest4383-21kworker/u8:009:30:213
908699341,20cyclictest1781-21/usr/sbin/munin08:05:233
908699331,18cyclictest11120-21seq09:39:413
908699330,21cyclictest0-21swapper/308:30:253
907199334,13cyclictest23526-21idleruntime-cro08:55:000
908699325,16cyclictest87350irq/126-enp1s0-12:35:123
908699322,26cyclictest27167-21diskstats10:15:123
908699321,24cyclictest16413-21gltestperf09:50:153
908699321,24cyclictest14981-21cat11:00:163
908699321,22cyclictest21567-21sshd08:50:123
908699321,18cyclictest87650irq/129-enp1s0-10:30:263
908699321,18cyclictest4636-21latency_hist09:25:023
908699321,18cyclictest0-21swapper/310:35:233
908699321,15cyclictest0-21swapper/310:10:123
908699320,23cyclictest0-21swapper/310:50:133
9071993219,5cyclictest14677-21kworker/0:112:17:090
908699313,17cyclictest774-21avahi-daemon10:01:063
908699312,25cyclictest15620-21smart_sda07:20:243
908699311,27cyclictest29631-21/usr/sbin/munin11:35:183
908699311,26cyclictest20632-21smart_sda08:45:263
908699311,17cyclictest19992-21latency_hist10:00:003
908699311,17cyclictest17394-21expr08:40:143
9086993110,18cyclictest12673-21latency_hist12:10:013
908699310,20cyclictest0-21swapper/312:15:013
908699310,20cyclictest0-21swapper/309:15:173
908699310,17cyclictest25654-21sshd08:59:133
907199312,11cyclictest13799-21kworker/0:311:05:140
9071993112,7cyclictest16847-21kworker/0:008:42:570
9071993112,3cyclictest3567-21kworker/0:108:12:250
908699302,25cyclictest13424-21smart_sda07:15:233
908699302,18cyclictest87650irq/129-enp1s0-07:30:123
908699301,26cyclictest6545-21/usr/sbin/munin10:40:203
908699301,26cyclictest20946-21/usr/sbin/munin07:35:143
908699301,17cyclictest22755-21/usr/sbin/munin12:30:243
9086993010,17cyclictest10915-21df_inode08:25:123
9081993015,3cyclictest0-21swapper/212:05:002
9071993018,4cyclictest21011-21kworker/0:109:01:140
9071993017,3cyclictest18976-21kworker/0:207:35:220
219152300,5sleep0907199cyclictest11:15:210
908699298,18cyclictest27540-21acpi07:50:093
908699291,25cyclictest9128-21irqstats12:00:183
908699291,25cyclictest8372-21/usr/sbin/munin08:20:133
908699291,25cyclictest5242-21smart_sda08:10:233
908699291,25cyclictest28164-21/usr/sbin/munin09:05:243
908699291,25cyclictest25953-21/usr/sbin/munin09:00:173
908699291,25cyclictest2338-21/usr/sbin/munin11:45:133
908699291,25cyclictest23140-21/usr/sbin/munin07:40:243
908699291,25cyclictest23039-21/usr/sbin/munin11:20:103
908699291,25cyclictest17624-21/usr/sbin/munin07:25:213
908699291,25cyclictest15095-21cstates12:15:123
9071992912,9cyclictest14677-21kworker/0:112:29:050
907199286,9cyclictest6640-21kworker/0:208:29:250
907199285,2cyclictest14072-21kworker/0:107:27:410
9071992812,8cyclictest29208-21kworker/0:007:54:220
907199273,5cyclictest29208-21kworker/0:007:59:540
9071992712,8cyclictest29208-21kworker/0:008:02:100
9071992712,8cyclictest29208-21kworker/0:008:02:100
9071992712,3cyclictest21011-21kworker/0:108:55:050
9071992712,3cyclictest14072-21kworker/0:107:33:580
908199268,3cyclictest0-21swapper/209:40:412
908199261,14cyclictest0-21swapper/207:40:252
907199268,8cyclictest10966-21kworker/0:007:11:580
907199266,9cyclictest4871-21kworker/0:010:49:490
907199266,10cyclictest30948-21kworker/0:009:10:580
907199266,10cyclictest30948-21kworker/0:009:10:580
907199265,4cyclictest3598-21kworker/0:112:00:090
9071992614,3cyclictest14072-21kworker/0:107:24:210
907199260,9cyclictest4871-21kworker/0:010:37:370
908199251,14cyclictest0-21swapper/211:49:542
908199251,13cyclictest0-21swapper/211:54:222
908199251,13cyclictest0-21swapper/210:42:522
9081992510,3cyclictest0-21swapper/210:07:402
907599251,20cyclictest13367-21sed12:10:161
907199258,2cyclictest22483-21kworker/0:211:25:180
907199257,8cyclictest31405-21kworker/0:211:47:100
907199257,2cyclictest30941-21kworker/0:110:31:570
907199256,9cyclictest4871-21kworker/0:010:56:340
907199256,2cyclictest31405-21kworker/0:211:39:020
907199255,10cyclictest1993-21kworker/0:209:19:100
9071992513,3cyclictest8935-21kworker/0:010:00:020
9071992511,3cyclictest29172-21kworker/0:111:40:180
9071992511,3cyclictest21030-21kworker/0:212:33:030
907199251,12cyclictest14748-21munin-run08:35:010
9071992510,3cyclictest24768-21kworker/0:307:48:420
908199246,9cyclictest0-21swapper/210:39:122
908199245,2cyclictest0-21swapper/207:58:112
908199241,9cyclictest0-21swapper/209:12:452
908199241,9cyclictest0-21swapper/209:12:452
9081992416,3cyclictest0-21swapper/211:37:112
908199241,13cyclictest0-21swapper/209:59:302
908199241,13cyclictest0-21swapper/207:24:342
908199241,12cyclictest0-21swapper/209:30:332
907199249,7cyclictest6640-21kworker/0:208:22:370
907199249,2cyclictest6640-21kworker/0:208:37:010
907199248,9cyclictest13799-21kworker/0:311:00:140
907199244,3cyclictest24047-21kworker/0:210:18:450
907199242,10cyclictest8935-21kworker/0:009:55:300
907199242,10cyclictest21011-21kworker/0:109:07:370
9071992412,3cyclictest24768-21kworker/0:307:43:500
9071992412,2cyclictest4331-21kworker/0:109:27:460
9071992411,7cyclictest11628-21kworker/0:110:54:220
907199241,13cyclictest10966-21kworker/0:007:15:220
9071992410,2cyclictest16847-21kworker/0:008:45:050
908199234,3cyclictest0-21swapper/211:28:192
9081992316,4cyclictest20348-21tr10:00:102
908199230,13cyclictest0-21swapper/207:36:212
907599235,4cyclictest27319-21turbostat07:50:011
907599231,9cyclictest0-21swapper/112:20:011
9075992317,3cyclictest808-21snmpd09:38:001
9075992317,3cyclictest27941-21munin-run09:05:001
9075992311,6cyclictest23-21ksoftirqd/108:23:171
907599231,16cyclictest0-21swapper/108:41:071
907599231,13cyclictest0-21swapper/111:42:271
907199237,2cyclictest12211-21kworker/0:212:11:090
907199235,4cyclictest8935-21kworker/0:009:37:500
907199232,9cyclictest24047-21kworker/0:210:11:170
9071992312,2cyclictest30948-21kworker/0:009:23:340
9071992311,2cyclictest8935-21kworker/0:010:05:170
9071992311,2cyclictest22483-21kworker/0:211:20:300
9071992311,2cyclictest1366-21kworker/0:011:52:340
9071992310,2cyclictest6640-21kworker/0:208:17:370
9071992310,2cyclictest13166-21kworker/0:109:43:180
908199229,3cyclictest0-21swapper/208:00:252
908199229,3cyclictest0-21swapper/208:00:252
908199228,5cyclictest827-21runrttasks10:23:482
908199228,5cyclictest827-21runrttasks10:23:482
9081992218,3cyclictest0-21swapper/211:07:542
9081992211,3cyclictest0-21swapper/210:54:452
908199221,13cyclictest0-21swapper/209:45:162
908199221,12cyclictest0-21swapper/212:25:122
908199221,12cyclictest0-21swapper/211:34:292
908199221,12cyclictest0-21swapper/211:34:292
908199221,12cyclictest0-21swapper/207:53:452
9081992210,3cyclictest0-21swapper/210:30:182
907599229,4cyclictest5016-21df09:25:131
907599221,19cyclictest0-21swapper/110:59:121
907599221,18cyclictest4064-21sed09:20:271
907599221,18cyclictest21845-21df12:30:131
907599221,18cyclictest16326-21timerandwakeup08:35:231
907599221,16cyclictest0-21swapper/111:27:251
907599221,16cyclictest0-21swapper/107:34:421
907599221,15cyclictest0-21swapper/108:31:281
907599220,16cyclictest0-21swapper/110:38:411
907199227,6cyclictest26958-21kworker/0:011:30:500
907199227,6cyclictest26958-21kworker/0:011:30:500
907199226,4cyclictest14677-21kworker/0:112:20:170
907199225,8cyclictest6677-21kworker/0:309:34:140
907199221,15cyclictest87350irq/126-enp1s0-10:25:090
9071992211,2cyclictest29208-21kworker/0:008:05:260
162682220,3sleep29-21rcu_preempt09:50:152
908199219,3cyclictest0-21swapper/212:31:102
908199218,3cyclictest0-21swapper/211:01:332
908199211,9cyclictest6175-21/usr/sbin/munin08:15:262
9081992114,4cyclictest9345-21tr09:35:122
9081992110,3cyclictest0-21swapper/210:18:342
908199210,13cyclictest0-21swapper/208:36:022
907599219,3cyclictest8077-21turbostat12:00:021
907599219,3cyclictest11812-21latency_hist10:55:011
907599217,12cyclictest19952-21kworker/u8:010:25:181
907599217,11cyclictest24751-21needreboot08:55:191
907599214,3cyclictest0-21swapper/111:22:181
907599211,8cyclictest0-21swapper/111:36:311
9075992117,3cyclictest0-21swapper/111:03:391
907599211,18cyclictest0-21swapper/109:33:131
907599211,18cyclictest0-21swapper/109:15:291
907599211,17cyclictest32648-21basename08:00:171
907599211,17cyclictest32648-21basename08:00:171
907599211,17cyclictest19821-21fschecks_count08:45:161
907599211,11cyclictest9389-21ntp_states12:00:231
907599210,19cyclictest0-21swapper/111:13:321
907199218,3cyclictest4871-21kworker/0:010:40:090
908199202,15cyclictest27512-21grep09:00:242
908199201,15cyclictest0-21swapper/208:40:222
908199201,11cyclictest0-21swapper/208:05:202
9081992010,3cyclictest0-21swapper/210:56:572
907599208,3cyclictest21912-21ntp_states11:15:211
907599201,9cyclictest0-21swapper/109:55:161
907599201,6cyclictest0-21swapper/110:33:031
9075992016,3cyclictest0-21swapper/110:18:321
9075992016,3cyclictest0-21swapper/107:42:321
907599201,5cyclictest0-21swapper/111:30:231
907599201,5cyclictest0-21swapper/111:30:231
9075992013,3cyclictest0-21swapper/107:21:391
9075992013,2cyclictest0-21swapper/107:35:061
907599201,14cyclictest0-21swapper/108:06:181
907599201,14cyclictest0-21swapper/107:17:501
907599201,11cyclictest16447-21/usr/sbin/munin11:05:241
9075992010,3cyclictest24038-21df12:35:111
9075992010,2cyclictest11691-21proc_pri12:05:231
907599200,15cyclictest0-21swapper/107:54:371
907199207,4cyclictest0-21swapper/012:40:000
907199206,2cyclictest3598-21kworker/0:111:55:290
907199205,2cyclictest9977-21kworker/0:012:07:130
92652191,2sleep10-21swapper/110:45:361
908199199,3cyclictest0-21swapper/207:27:402
908199198,3cyclictest0-21swapper/210:28:262
908199196,3cyclictest0-21swapper/210:12:022
908199191,15cyclictest0-21swapper/210:45:582
908199191,14cyclictest0-21swapper/211:58:542
908199191,13cyclictest6710-21munin-run09:30:012
908199191,13cyclictest442-21df_inode09:15:132
908199191,13cyclictest20351-21needreboot08:45:222
908199191,13cyclictest0-21swapper/211:40:572
908199191,10cyclictest20447-21runrttasks07:31:382
907599198,9cyclictest8011-21kworker/u8:112:25:031
907599197,11cyclictest23-21ksoftirqd/109:41:441
907599195,8cyclictest808-21snmpd11:48:281
9075991916,2cyclictest0-21swapper/107:13:221
907599191,17cyclictest0-21swapper/112:23:191
907599191,16cyclictest0-21swapper/110:09:451
907599191,12cyclictest0-21swapper/110:02:231
907599191,10cyclictest6617-21ps10:40:201
907599191,10cyclictest5257-21grep11:50:261
907599191,10cyclictest30086-21ntpq10:20:221
907599191,10cyclictest30086-21ntpq10:20:221
907599191,10cyclictest22549-21needreboot08:50:201
907599191,10cyclictest16761-21cstates07:25:111
907599190,14cyclictest0-21swapper/108:10:101
907599190,13cyclictest0-21swapper/107:56:211
116342190,2sleep10-21swapper/108:25:201
908199185,3cyclictest0-21swapper/212:15:162
908199181,6cyclictest11586-21ntp_states12:05:222
908199181,5cyclictest25786-21tune2fs07:45:142
9081991812,3cyclictest10325-21ntp_offset07:10:192
9081991811,3cyclictest18243-21grep12:20:222
908199181,13cyclictest0-21swapper/211:15:172
908199181,13cyclictest0-21swapper/208:51:342
907599182,13cyclictest28975-21irqstats09:05:161
907599181,8cyclictest0-21swapper/109:11:071
907599181,8cyclictest0-21swapper/109:11:071
907599181,14cyclictest260262sleep110:10:261
185852181,2sleep20-21swapper/211:10:012
908199171,8cyclictest25237-21/usr/sbin/munin12:35:252
908199171,3cyclictest14722-21taskset08:34:542
908199171,2cyclictest0-21swapper/209:05:282
908199171,13cyclictest0-21swapper/207:15:342
908199170,13cyclictest0-21swapper/208:55:022
907599175,9cyclictest16015-21sadc09:50:131
908199165,3cyclictest87350irq/126-enp1s0-12:10:232
908199161,3cyclictest0-21swapper/208:25:142
9081991612,2cyclictest8372-21/usr/sbin/munin08:20:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional