You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 14:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Jan 01, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
324452230163,55sleep10-21swapper/107:06:111
324452230163,55sleep10-21swapper/107:06:111
325712202164,25sleep20-21swapper/207:07:492
325712202164,25sleep20-21swapper/207:07:482
326332201165,24sleep30-21swapper/307:08:383
326332201165,24sleep30-21swapper/307:08:383
327072199163,24sleep00-21swapper/007:09:360
327072199163,24sleep00-21swapper/007:09:360
1451521730,4sleep035499cyclictest11:20:130
448821460,4sleep035499cyclictest10:55:240
366991061,2cyclictest0-21swapper/208:33:002
36699105101,3cyclictest0-21swapper/210:11:292
3669910299,2cyclictest0-21swapper/208:17:572
3669910299,2cyclictest0-21swapper/208:17:572
366998985,3cyclictest0-21swapper/207:57:412
57292720,3sleep09-21rcu_preempt11:00:140
366997067,2cyclictest0-21swapper/207:20:412
320072550,4sleep232009-21kernelversion10:45:192
163772550,4sleep236699cyclictest08:55:242
109352550,7sleep1221ktimersoftd/111:10:221
12992510,1sleep10-21swapper/109:35:191
37299364,23cyclictest2269-21kworker/u8:109:52:143
37299311,3cyclictest0-21swapper/311:56:073
37299302,7cyclictest16803-21munin-run12:40:013
36299290,20cyclictest0-21swapper/108:40:121
372992812,9cyclictest21536-21diskmemload10:35:433
36299271,19cyclictest0-21swapper/111:40:011
36299271,17cyclictest20532-21users07:50:261
35499279,8cyclictest1615-21kworker/0:012:13:320
362992616,6cyclictest18046-21systemd-journal11:25:541
36299261,17cyclictest0-21swapper/110:30:161
36299261,16cyclictest0-21swapper/110:35:171
16702260,3sleep236699cyclictest10:50:182
36699259,4cyclictest0-21swapper/210:15:222
366992513,7cyclictest1075-21systemd-journal12:01:022
36299253,7cyclictest0-21swapper/112:24:411
36299251,7cyclictest1349-21users08:20:261
36299251,17cyclictest0-21swapper/111:50:251
36299251,17cyclictest0-21swapper/111:21:051
36299251,16cyclictest5238-21ntpq08:30:241
36299251,12cyclictest0-21swapper/111:05:091
36299250,20cyclictest0-21swapper/110:09:471
37299248,6cyclictest39-21rcuc/309:30:413
37299247,4cyclictest86950irq/126-enp1s0-07:34:563
37299241,21cyclictest0-21swapper/310:50:133
37299241,11cyclictest5133-21df_inode09:45:133
37299241,11cyclictest4321-21if_enp2s007:15:173
37299241,11cyclictest25549-21users09:15:263
37299241,11cyclictest18543-21NetworkManager07:47:563
37299241,11cyclictest17323-21memory11:25:203
37299241,11cyclictest0-21swapper/308:15:263
37299241,11cyclictest0-21swapper/308:15:263
36299248,10cyclictest0-21swapper/110:53:191
36299247,10cyclictest0-21swapper/112:07:481
36299241,16cyclictest0-21swapper/110:12:111
37299239,3cyclictest31174-21/usr/sbin/munin10:45:183
37299233,18cyclictest39-21rcuc/311:15:153
37299233,10cyclictest0-21swapper/312:15:233
37299232,19cyclictest39-21rcuc/310:05:253
37299231,9cyclictest35702awk08:25:273
37299231,20cyclictest401ktimersoftd/307:35:173
37299231,19cyclictest801-21snmpd12:08:483
37299231,19cyclictest801-21snmpd10:18:193
37299231,19cyclictest401ktimersoftd/310:15:023
37299231,19cyclictest21766-21/usr/sbin/munin09:10:243
37299231,19cyclictest0-21swapper/311:03:213
37299231,14cyclictest0-21swapper/308:58:393
37299231,14cyclictest0-21swapper/308:08:593
37299231,11cyclictest0-21swapper/311:10:373
37299231,10cyclictest27730-21df11:50:163
37299230,11cyclictest826-21lldpd10:40:273
37299230,11cyclictest826-21lldpd10:40:273
36299231,19cyclictest15220-21hddtemp_smartct07:40:151
36299231,16cyclictest10235-21ntp_kernel_pll_09:55:201
362992311,6cyclictest0-21swapper/112:26:391
36299231,16cyclictest0-21swapper/111:03:021
36299231,16cyclictest0-21swapper/110:00:381
36299231,15cyclictest0-21swapper/109:21:131
362992311,4cyclictest22962-21perl10:25:161
36299231,13cyclictest786-21sssd_be09:25:511
36299230,16cyclictest0-21swapper/108:09:461
35499235,15cyclictest23731-21idleruntime-cro09:15:000
35499230,7cyclictest11033-21kworker/0:010:08:180
37299221,9cyclictest0-21swapper/308:02:473
37299221,19cyclictest86950irq/126-enp1s0-09:02:423
37299221,19cyclictest0-21swapper/311:46:083
37299221,19cyclictest0-21swapper/311:05:183
37299221,11cyclictest0-21swapper/310:00:473
37299221,10cyclictest801-21snmpd07:12:433
36699222,5cyclictest25630-21ntpq10:30:222
36299228,6cyclictest563-21systemd-journal07:10:051
36299223,14cyclictest751-21rngd11:55:571
362992214,3cyclictest0-21swapper/111:41:361
36299221,19cyclictest0-21swapper/108:00:191
362992211,6cyclictest0-21swapper/112:14:381
36299221,16cyclictest0-21swapper/109:12:121
36299221,15cyclictest0-21swapper/107:31:531
36299221,15cyclictest0-21swapper/107:20:261
362992211,3cyclictest0-21swapper/109:54:501
36299221,13cyclictest0-21swapper/109:47:211
36299220,5cyclictest0-21swapper/111:46:221
36299220,5cyclictest0-21swapper/110:41:321
36299220,5cyclictest0-21swapper/110:41:321
36299220,16cyclictest22410-21kworker/u8:012:03:161
37299219,3cyclictest0-21swapper/307:43:243
37299218,2cyclictest1639-21smart_sda09:35:253
37299217,2cyclictest0-21swapper/309:25:183
37299211,7cyclictest29002-21smart_sda08:10:243
37299211,6cyclictest0-21swapper/310:32:263
37299211,3cyclictest3939-21timerandwakeup09:40:263
37299211,11cyclictest0-21swapper/311:44:013
37299211,11cyclictest0-21swapper/311:38:563
36699215,4cyclictest21536-21diskmemload09:25:182
366992117,3cyclictest0-21swapper/209:59:422
36299217,3cyclictest87150irq/128-enp1s0-12:35:251
36299217,3cyclictest3717-21fschecks_time10:55:151
36299211,7cyclictest15188-21cpu08:55:111
36299211,5cyclictest0-21swapper/110:15:361
362992113,3cyclictest0-21swapper/112:32:161
36299211,17cyclictest712-21perf10:50:021
36299211,15cyclictest0-21swapper/108:28:521
36299211,15cyclictest0-21swapper/107:57:011
36299211,14cyclictest0-21swapper/107:39:561
362992111,3cyclictest9544-21users07:25:271
36299211,11cyclictest21758-21kworker/u8:209:30:241
36299211,11cyclictest20189-21users11:30:261
36299211,11cyclictest0-21swapper/109:42:191
36299211,10cyclictest0-21swapper/112:18:221
36299211,10cyclictest0-21swapper/109:15:481
36299210,16cyclictest0-21swapper/108:46:411
36299210,15cyclictest0-21swapper/110:24:361
35499215,8cyclictest176950irq/12-ttyS009:25:160
123172210,5sleep339-21rcuc/308:45:253
90342200,3sleep230-21rcuc/208:40:152
37299208,3cyclictest0-21swapper/312:03:453
37299204,4cyclictest32164-21cpu08:20:103
37299201,9cyclictest0-21swapper/311:20:203
37299201,8cyclictest0-21swapper/312:28:123
37299201,2cyclictest401ktimersoftd/310:28:353
37299201,2cyclictest401ktimersoftd/308:35:163
37299201,2cyclictest0-21swapper/307:53:213
37299201,1cyclictest0-21swapper/308:40:173
37299201,10cyclictest0-21swapper/311:30:283
37299200,2cyclictest0-21swapper/312:20:163
36699204,4cyclictest30896-21switchtime11:55:262
36699204,4cyclictest23158-21sed09:10:252
36699204,4cyclictest20525-21sh10:20:162
36299201,14cyclictest0-21swapper/107:49:161
36299200,14cyclictest0-21swapper/109:09:181
35499206,8cyclictest0-21swapper/010:53:140
35499205,9cyclictest0-21swapper/008:20:080
35499202,10cyclictest1982-21/usr/sbin/munin08:25:180
35499201,6cyclictest10259-21ntp_states09:55:220
35499201,13cyclictest86950irq/126-enp1s0-08:50:270
35499201,12cyclictest0-21swapper/011:46:550
35499201,12cyclictest0-21swapper/010:17:250
35499201,10cyclictest648-21df09:35:130
37299195,3cyclictest86950irq/126-enp1s0-08:30:133
37299193,4cyclictest835-21runrttasks10:58:453
37299192,5cyclictest7022-21smart_sda07:20:253
37299191,6cyclictest0-21swapper/312:30:183
37299191,11cyclictest0-21swapper/310:20:053
37299190,1cyclictest0-21swapper/309:55:243
36699191,8cyclictest782-21in:imjournal07:49:412
36699191,5cyclictest27115-21df_inode10:35:122
36699191,10cyclictest0-21swapper/209:38:052
362991916,2cyclictest0-21swapper/108:50:471
362991912,2cyclictest0-21swapper/109:00:111
36299191,16cyclictest0-21swapper/108:14:261
36299191,11cyclictest0-21swapper/108:35:191
36299191,10cyclictest3715-21expr07:15:111
36299190,14cyclictest0-21swapper/111:16:481
36299190,14cyclictest0-21swapper/108:16:111
36299190,14cyclictest0-21swapper/108:16:111
35499199,3cyclictest14209-21sshd07:38:410
35499198,3cyclictest22999-21/usr/sbin/munin11:40:160
35499198,3cyclictest21136-21/usr/sbin/munin07:55:220
35499198,3cyclictest17362-21/usr/sbin/munin09:00:140
35499194,9cyclictest0-21swapper/011:35:150
35499194,8cyclictest87150irq/128-enp1s0-10:30:010
35499192,9cyclictest2597-21/usr/sbin/munin09:40:150
35499192,7cyclictest0-21swapper/009:20:130
35499192,4cyclictest12745-21awk08:50:010
35499191,8cyclictest0-21swapper/009:30:130
35499191,12cyclictest0-21swapper/012:29:320
35499191,12cyclictest0-21swapper/012:06:520
35499191,12cyclictest0-21swapper/011:18:290
35499191,12cyclictest0-21swapper/009:17:580
35499191,10cyclictest31788-21/usr/sbin/munin12:00:280
77342181,2sleep00-21swapper/012:16:410
37299181,5cyclictest12966-21/usr/sbin/munin08:50:173
37299181,3cyclictest3876-21cpu12:10:143
37299181,15cyclictest0-21swapper/309:05:093
37299181,11cyclictest86950irq/126-enp1s0-07:26:253
37299181,10cyclictest0-21swapper/309:21:263
36699184,4cyclictest24423-21latency_hist10:30:002
36699183,5cyclictest0-21swapper/210:58:022
36699182,7cyclictest18939-21/usr/sbin/munin07:50:132
36699182,5cyclictest0-21swapper/212:30:072
36699181,7cyclictest26762-21grep09:20:162
36699181,7cyclictest10151-21df11:10:132
36699181,6cyclictest0-21swapper/212:26:392
36699181,5cyclictest7388-21latency_hist11:05:022
36699181,5cyclictest6039-21/usr/sbin/munin12:15:252
36699181,5cyclictest2614-21cpu09:40:122
36699181,15cyclictest0-21swapper/211:50:282
366991811,4cyclictest17642-21fgrep11:25:252
35499187,3cyclictest23336-21/usr/sbin/munin08:00:250
35499187,3cyclictest14549-21/usr/sbin/munin07:40:110
35499184,3cyclictest176950irq/12-ttyS008:55:210
35499182,7cyclictest4177-21/usr/sbin/munin08:30:150
35499181,9cyclictest573-21/usr/sbin/munin07:10:270
35499181,9cyclictest26775-21/usr/sbin/munin10:35:270
35499181,7cyclictest0-21swapper/012:21:410
35499181,6cyclictest21544-21perf09:10:000
35499181,6cyclictest0-21swapper/009:50:170
35499181,5cyclictest9562-21grep08:40:190
35499181,3cyclictest86950irq/126-enp1s0-10:46:370
354991811,4cyclictest8353-21perf08:40:010
35499181,11cyclictest0-21swapper/012:37:270
35499181,11cyclictest0-21swapper/011:53:370
35499181,10cyclictest0-21swapper/010:12:450
35499180,11cyclictest0-21swapper/010:21:310
37299171,7cyclictest21385-21cat07:55:093
36699179,5cyclictest0-21swapper/210:05:212
36699177,3cyclictest0-21swapper/211:41:422
36699172,12cyclictest13511-21wc10:05:012
36699171,7cyclictest32131-21/usr/sbin/munin08:20:232
36699171,5cyclictest7957-21/usr/sbin/munin07:25:162
36699171,5cyclictest2583-21kworker/2:211:45:172
36699171,4cyclictest86950irq/126-enp1s0-12:35:282
36699171,4cyclictest30239-21smart_sda10:40:262
36699171,4cyclictest30239-21smart_sda10:40:262
36699171,4cyclictest16180-21latency_hist11:25:022
36699171,3cyclictest0-21swapper/209:05:162
36699171,2cyclictest0-21swapper/209:47:252
366991712,3cyclictest7353-21needreboot08:35:212
366991711,3cyclictest3052-21cat07:15:022
366991710,4cyclictest18681-21cat11:30:122
35499178,5cyclictest0-21swapper/007:29:570
35499177,3cyclictest5761-21/usr/sbin/munin07:20:270
35499173,2cyclictest87250irq/129-enp1s0-10:40:220
35499173,2cyclictest87250irq/129-enp1s0-10:40:220
35499172,12cyclictest7619-21/usr/sbin/munin11:05:180
35499172,12cyclictest24579-21/usr/sbin/munin10:30:170
35499171,7cyclictest18603-21/usr/sbin/munin11:30:200
35499171,6cyclictest0-21swapper/007:15:130
35499171,5cyclictest30612-21netstat11:55:200
35499171,4cyclictest801-21snmpd08:07:540
35499171,4cyclictest13376-21munin-run10:05:020
35499171,4cyclictest13139-21fschecks_time12:30:160
354991711,3cyclictest16746-21/usr/sbin/munin07:45:150
35499171,10cyclictest0-21swapper/011:13:450
354991710,4cyclictest17638-21/usr/sbin/munin11:25:240
35499170,6cyclictest0-21swapper/009:46:410
257732171,2sleep20-21swapper/209:18:082
36699166,3cyclictest0-21swapper/211:05:332
36699165,3cyclictest86950irq/126-enp1s0-09:50:202
36699164,3cyclictest11136-21uname07:30:222
36699161,7cyclictest0-21swapper/207:35:242
36699161,5cyclictest27757-21/usr/sbin/munin08:10:112
36699161,5cyclictest12021-21acpi11:15:122
36699161,5cyclictest0-21swapper/212:23:292
366991612,3cyclictest0-21swapper/212:10:522
36699161,14cyclictest0-21swapper/208:08:462
366991611,3cyclictest32331-21sshd09:32:412
366991611,3cyclictest23336-21/usr/sbin/munin08:00:202
366991611,3cyclictest21050-21expr11:35:142
366991611,3cyclictest18247-21cat09:00:202
36699161,13cyclictest15840-21switchtime07:40:232
366991611,3cyclictest1361-21basename07:10:172
366991611,3cyclictest13233-21expr08:50:142
366991611,3cyclictest10769-21/usr/sbin/munin08:45:162
366991610,3cyclictest1731-21cut12:05:122
35499161,6cyclictest27905-21cpuspeed_turbos08:10:120
354991611,3cyclictest30903-21cat08:15:200
354991611,3cyclictest30903-21cat08:15:190
354991611,3cyclictest10382-21expr07:30:160
35499161,12cyclictest18939-21/usr/sbin/munin07:50:150
366991510,3cyclictest2589-21grep08:25:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional