You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-15 - 23:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 15, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
226402238163,24sleep20-21swapper/207:07:002
228012232163,21sleep10-21swapper/107:09:041
225792204168,24sleep30-21swapper/307:06:143
227262202164,25sleep00-21swapper/007:08:070
401621700,3sleep22296799cyclictest08:50:222
3087221510,4sleep12296099cyclictest08:40:011
1018721440,2sleep10-21swapper/109:05:191
632521400,3sleep22296799cyclictest11:25:012
135621330,3sleep12296099cyclictest07:30:231
22967991110,109cyclictest0-21swapper/208:15:242
22967998581,3cyclictest0-21swapper/208:23:182
22967997774,2cyclictest0-21swapper/212:24:272
221172580,4sleep011-21rcuc/010:45:190
266652520,1sleep00-21swapper/008:30:090
123102490,3sleep20-21swapper/210:25:012
123102490,3sleep20-21swapper/210:25:012
42702480,1sleep00-21swapper/011:20:010
116952450,2sleep30-21swapper/311:35:173
116952450,2sleep30-21swapper/311:35:173
22967994126,3cyclictest0-21swapper/207:29:482
66542400,3sleep26650-21munin-run07:45:012
2295399407,10cyclictest909-21sssd_nss09:00:240
22967993929,7cyclictest0-21swapper/211:41:322
2296799391,23cyclictest0-21swapper/210:50:352
22953993912,3cyclictest5309-21df_inode12:35:150
2297299381,27cyclictest22073-21if_enp2s010:45:173
2296799381,22cyclictest0-21swapper/208:28:232
2295399388,14cyclictest20469-21munin-run09:30:010
2296799361,27cyclictest0-21swapper/209:55:552
2296799361,20cyclictest17983-21ntp_states10:35:232
2295399361,15cyclictest31710-21cut08:40:170
22967993524,3cyclictest0-21swapper/212:29:032
2296799351,23cyclictest0-21swapper/207:24:072
2296799351,13cyclictest11668-21diskmemload09:13:192
2296799351,13cyclictest11668-21diskmemload09:13:192
2296799350,24cyclictest0-21swapper/208:55:432
22953993512,3cyclictest8122-21/usr/sbin/munin10:15:170
22967993422,6cyclictest0-21swapper/211:32:432
22967993418,8cyclictest0-21swapper/210:06:112
2296799341,22cyclictest0-21swapper/208:34:472
2296799341,21cyclictest0-21swapper/211:14:232
2296799341,20cyclictest0-21swapper/211:15:592
2296799341,20cyclictest0-21swapper/207:54:152
2296799341,18cyclictest0-21swapper/209:16:562
2295399348,3cyclictest30118-21/usr/sbin/munin11:05:270
2295399348,2cyclictest20582-21latency11:55:190
22967993319,4cyclictest0-21swapper/210:25:232
22967993318,3cyclictest0-21swapper/212:08:192
2296799331,21cyclictest0-21swapper/210:17:392
2296799331,20cyclictest0-21swapper/210:31:112
2296799331,19cyclictest0-21swapper/210:49:592
2296799331,18cyclictest11668-21diskmemload11:45:562
2296799331,14cyclictest0-21swapper/209:07:512
2295399332,11cyclictest20071-21/usr/sbin/munin08:15:240
22953993312,3cyclictest32657-21diskstats11:10:150
2295399331,10cyclictest26053-21/usr/sbin/munin07:15:270
2297299327,12cyclictest29092-21expr08:35:133
22967993216,7cyclictest0-21swapper/212:03:472
2296799321,22cyclictest0-21swapper/209:32:362
2296799321,21cyclictest11668-21diskmemload11:58:362
2296799321,21cyclictest0-21swapper/211:35:132
2296799321,21cyclictest0-21swapper/211:35:132
2296799321,19cyclictest0-21swapper/212:31:472
2296799321,13cyclictest0-21swapper/210:12:032
2295399321,6cyclictest17301-21kworker/u8:208:55:220
22953993212,3cyclictest25407-21sort07:15:020
2295399320,16cyclictest25498-21awk10:55:020
2296799311,20cyclictest0-21swapper/211:26:392
2296799311,20cyclictest0-21swapper/208:39:112
2296799311,19cyclictest0-21swapper/211:04:272
2296799311,14cyclictest0-21swapper/210:56:352
2296799311,13cyclictest0-21swapper/207:37:202
2296799311,11cyclictest0-21swapper/207:49:162
2296799311,10cyclictest0-21swapper/207:17:312
2295399315,10cyclictest295232chrt11:00:390
22953993112,3cyclictest5091-21cat10:05:260
2295399310,10cyclictest0-21swapper/010:56:160
2297299301,15cyclictest0-21swapper/307:45:173
2297299301,12cyclictest0-21swapper/311:30:193
2297299301,12cyclictest0-21swapper/311:15:553
2296799301,26cyclictest0-21swapper/209:45:172
2296799301,21cyclictest0-21swapper/207:33:242
2296799301,20cyclictest0-21swapper/208:05:312
2296799301,19cyclictest0-21swapper/210:41:432
2296799301,15cyclictest0-21swapper/212:35:432
2296799301,14cyclictest0-21swapper/211:50:082
2295399305,3cyclictest0-21swapper/009:12:530
2295399305,3cyclictest0-21swapper/009:12:530
2295399301,3cyclictest20952-21expr09:30:110
22953993012,3cyclictest11273-21/usr/sbin/munin07:55:240
22953993012,3cyclictest11273-21/usr/sbin/munin07:55:240
2295399301,10cyclictest26072-21sshd12:09:410
110982300,4sleep02295399cyclictest10:20:170
110982300,4sleep02295399cyclictest10:20:170
288152290,4sleep22296799cyclictest12:15:132
2297299297,11cyclictest0-21swapper/312:13:323
22967992915,4cyclictest0-21swapper/209:41:072
22967992914,8cyclictest0-21swapper/210:02:552
22967992913,7cyclictest0-21swapper/211:08:312
2295399291,10cyclictest0-21swapper/011:33:510
2297299288,3cyclictest0-21swapper/309:35:233
22972992813,10cyclictest0-21swapper/312:25:163
2296799281,17cyclictest0-21swapper/207:12:432
2296799281,16cyclictest0-21swapper/208:10:512
2296799281,13cyclictest0-21swapper/209:38:282
2295399283,3cyclictest0-21swapper/012:31:360
2297299271,13cyclictest0-21swapper/310:27:563
22967992718,3cyclictest0-21swapper/209:52:192
22967992717,3cyclictest0-21swapper/209:00:312
2296799271,19cyclictest0-21swapper/208:47:032
2296799271,19cyclictest0-21swapper/208:41:112
2296799271,13cyclictest0-21swapper/212:11:352
2296799271,12cyclictest0-21swapper/209:23:162
2296099272,14cyclictest8708-40dnf09:00:251
2296099271,14cyclictest0-21swapper/107:35:131
2295399274,11cyclictest6665-21awk07:45:000
2295399271,6cyclictest0-21swapper/012:29:200
2295399271,11cyclictest0-21swapper/009:22:110
2295399271,10cyclictest6413-21kworker/0:107:45:210
2297299261,15cyclictest0-21swapper/309:53:243
2296799261,9cyclictest0-21swapper/208:00:132
22967992617,3cyclictest0-21swapper/207:57:472
22967992617,3cyclictest0-21swapper/207:57:472
2296099263,6cyclictest808-21snmpd08:01:371
2295399261,4cyclictest21074-21latency_hist10:45:020
2295399260,5cyclictest0-21swapper/012:18:010
2297299252,8cyclictest0-21swapper/309:14:153
2297299252,8cyclictest0-21swapper/309:14:153
2297299251,22cyclictest0-21swapper/308:25:353
22972992510,3cyclictest15480-21if_err_enp1s010:30:163
2295399252,5cyclictest15677-21/usr/sbin/munin08:05:240
2295399251,9cyclictest0-21swapper/010:12:350
2295399251,5cyclictest24860-21munin-run09:40:010
2295399251,12cyclictest17317-21awk11:50:010
304302240,4sleep121-21rcuc/109:50:201
252782240,5sleep325283-21pmu-power12:05:233
2297299248,3cyclictest0-21swapper/309:28:113
22972992411,5cyclictest0-21swapper/309:58:393
22972992410,3cyclictest0-21swapper/307:26:353
22967992415,3cyclictest0-21swapper/209:26:402
2295399246,15cyclictest32430-21awk07:30:010
2295399241,4cyclictest8596-21cat11:30:010
2295399241,3cyclictest27394-21cut09:45:130
2295399241,15cyclictest0-21swapper/009:19:060
2297299235,3cyclictest0-21swapper/310:00:153
2297299234,11cyclictest87350irq/126-enp1s0-10:55:313
2297299231,8cyclictest0-21swapper/312:00:273
2297299231,5cyclictest22045-21perf08:20:023
22972992312,3cyclictest0-21swapper/312:22:283
22972992311,3cyclictest18562unixbench_singl07:30:273
2296099239,4cyclictest15110-21munin-run11:45:011
2296099232,7cyclictest0-21swapper/110:30:251
2296099232,18cyclictest4686-21df_inode11:20:151
2296099231,19cyclictest29513-21needreboot12:15:201
2296099231,16cyclictest0-21swapper/108:25:591
2295399233,11cyclictest88950irq/132-enp2s0-10:33:370
2295399232,7cyclictest0-21swapper/010:29:280
2295399231,11cyclictest5081-21basename08:55:010
2295399230,8cyclictest30334-21kworker/u8:012:20:270
2295399230,10cyclictest18616-21kworker/0:110:38:210
2297299228,3cyclictest18825-21ls08:10:203
2297299224,3cyclictest0-21swapper/307:21:153
22972992214,3cyclictest0-21swapper/308:55:153
2297299221,2cyclictest0-21swapper/307:10:233
22972992212,5cyclictest9025-21unixbench_singl09:00:253
2297299221,10cyclictest0-21swapper/308:30:353
22972992210,3cyclictest0-21swapper/311:41:313
22972992210,3cyclictest0-21swapper/309:32:553
2297299220,5cyclictest15799-21kworker/u8:009:24:313
2296099222,7cyclictest0-21swapper/109:16:571
2296099222,18cyclictest0-21swapper/107:19:301
2296099221,3cyclictest5069-21munin-run08:55:001
2296099221,3cyclictest4307-21fschecks_time10:05:171
2296099221,3cyclictest24540-21timerandwakeup07:10:251
2296099221,2cyclictest578-21perf08:45:001
2296099221,16cyclictest0-21swapper/110:03:581
2296099221,16cyclictest0-21swapper/108:30:511
2296099221,11cyclictest909-21sssd_nss12:10:221
2296099221,10cyclictest0-21swapper/112:06:251
2295399226,10cyclictest0-21swapper/011:39:480
2295399226,10cyclictest0-21swapper/011:39:480
2297299217,3cyclictest87350irq/126-enp1s0-12:15:273
2297299216,4cyclictest909-21sssd_nss10:45:013
2297299214,11cyclictest87350irq/126-enp1s0-10:16:203
2297299213,9cyclictest87350irq/126-enp1s0-09:45:033
2297299212,3cyclictest0-21swapper/307:18:233
2297299212,11cyclictest13467-21/usr/sbin/munin08:00:263
2297299211,6cyclictest0-21swapper/309:19:053
2296099216,12cyclictest182472chrt09:24:561
2296099213,2cyclictest87650irq/129-enp1s0-11:11:191
2296099212,3cyclictest0-21swapper/107:24:151
2296099212,18cyclictest0-21swapper/108:22:411
2296099212,11cyclictest21077-21dump-pmu-power10:49:561
22960992116,3cyclictest26384-21smartctl10:55:181
2296099211,18cyclictest0-21swapper/110:26:551
2296099211,18cyclictest0-21swapper/109:10:231
2296099211,18cyclictest0-21swapper/109:10:231
2296099211,15cyclictest0-21swapper/110:21:271
2296099211,15cyclictest0-21swapper/110:21:271
2296099211,11cyclictest2729-21fw_packets11:15:161
22960992110,3cyclictest6269-21diskstats10:10:141
22960992110,3cyclictest26350-21smart_sda09:40:251
2295399217,3cyclictest32347-21kworker/0:007:31:170
2295399216,3cyclictest12642-21kworker/0:311:43:170
2295399216,2cyclictest1147-21kworker/0:110:02:090
2295399214,2cyclictest2208-21kworker/0:207:37:170
2295399211,5cyclictest0-21swapper/011:23:140
240842201,3sleep30-21swapper/308:23:123
2297299206,3cyclictest0-21swapper/308:44:353
2297299204,10cyclictest0-21swapper/310:38:353
2297299203,9cyclictest87350irq/126-enp1s0-11:12:513
2297299203,11cyclictest87350irq/126-enp1s0-11:27:033
2297299202,10cyclictest3729-21/usr/sbin/munin10:05:173
2297299201,6cyclictest0-21swapper/307:50:233
2297299201,5cyclictest0-21swapper/310:22:383
2297299201,5cyclictest0-21swapper/310:22:383
2297299201,3cyclictest17243-21grep08:05:283
2297299201,15cyclictest10385-21hwlatdetect09:05:193
2297299201,12cyclictest0-21swapper/310:10:203
2297299201,12cyclictest0-21swapper/307:35:273
2297299200,10cyclictest0-21swapper/312:33:483
2296099209,3cyclictest326-21cat12:25:011
2296099202,3cyclictest0-21swapper/107:57:531
2296099202,3cyclictest0-21swapper/107:57:531
2296099202,16cyclictest0-21swapper/111:45:361
22960992016,3cyclictest0-21swapper/111:58:501
22960992016,3cyclictest0-21swapper/109:57:031
2296099201,5cyclictest0-21swapper/112:29:241
2296099201,17cyclictest3250-21cut12:30:141
2296099201,17cyclictest0-21swapper/111:50:201
2296099201,17cyclictest0-21swapper/111:35:161
2296099201,17cyclictest0-21swapper/111:35:161
22960992011,5cyclictest23641-40dnf12:01:431
2296099201,12cyclictest0-21swapper/107:40:121
2296099201,10cyclictest18837-21diskstats09:25:141
2295399208,3cyclictest19725-21kworker/0:308:23:530
2295399205,4cyclictest24680-21kworker/0:209:50:450
2295399201,9cyclictest24680-21kworker/0:209:56:370
2295399201,10cyclictest0-21swapper/012:00:450
2295399200,3cyclictest28853-21kworker/0:208:35:140
2297299196,4cyclictest11668-21diskmemload11:03:023
2297299195,3cyclictest87650irq/129-enp1s0-12:37:033
2297299192,11cyclictest87650irq/129-enp1s0-09:42:453
22972991912,4cyclictest1217-21df_inode08:45:143
2297299190,7cyclictest0-21swapper/311:49:033
2296099198,3cyclictest23858-21diskstats10:50:151
2296099192,16cyclictest0-21swapper/110:39:491
2296099191,7cyclictest0-21swapper/110:15:271
22960991916,2cyclictest0-21swapper/107:49:101
22960991915,3cyclictest0-21swapper/111:08:211
22960991915,3cyclictest0-21swapper/107:52:041
2296099191,16cyclictest28838-21unixbench_singl09:45:271
2296099191,16cyclictest11668-21diskmemload11:29:201
2296099191,16cyclictest0-21swapper/111:30:061
2296099191,10cyclictest925-21cpu08:45:131
2296099191,10cyclictest21074-21latency_hist10:45:021
2296099191,10cyclictest20144-21tr08:15:131
2296099191,10cyclictest17888-21cpu08:10:121
2296099191,10cyclictest0-21swapper/111:00:171
2296099190,17cyclictest0-21swapper/108:05:391
2295399196,4cyclictest22183-21kworker/0:207:23:370
2295399195,2cyclictest28417-21kworker/0:008:49:140
2295399193,4cyclictest9296-21kworker/0:309:07:370
2295399193,3cyclictest6410-21kworker/0:008:01:490
2295399193,2cyclictest89150irq/134-enp2s0-08:13:010
2295399192,10cyclictest24680-21kworker/0:209:43:010
2297299186,3cyclictest19165-21taskset11:50:493
2297299183,2cyclictest87350irq/126-enp1s0-11:20:073
2297299181,11cyclictest0-21swapper/308:51:553
2296099181,9cyclictest7257-21latency_hist09:00:011
2296099181,3cyclictest0-21swapper/109:31:051
22960991812,4cyclictest30917-21grep07:25:141
2296099181,10cyclictest6225-21smart_sda12:35:261
2296099180,2cyclictest0-21swapper/109:35:211
2295399185,7cyclictest26036-21kworker/0:112:14:470
2295399182,2cyclictest89150irq/134-enp2s0-08:28:170
2295399181,11cyclictest0-21swapper/011:53:170
214402181,3sleep30-21swapper/311:56:533
2297299174,2cyclictest11668-21diskmemload10:51:193
2297299173,2cyclictest87350irq/126-enp1s0-07:43:253
22972991711,3cyclictest30885-21if_err_enp1s011:05:203
2297299171,13cyclictest11273-21/usr/sbin/munin07:55:223
2297299171,13cyclictest11273-21/usr/sbin/munin07:55:223
2295399173,2cyclictest87350irq/126-enp1s0-07:51:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional