You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-02 - 01:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 01, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
274562236160,25sleep00-21swapper/007:09:180
274052236163,22sleep10-21swapper/107:08:391
273242203167,24sleep20-21swapper/207:07:352
272892201163,25sleep30-21swapper/307:07:083
1178621800,5sleep32760799cyclictest11:25:163
2614321760,4sleep32760799cyclictest08:15:263
2906021630,4sleep22760299cyclictest07:10:242
22182610,4sleep32760799cyclictest12:15:253
223502580,6sleep230-21rcuc/210:35:192
27602995248,3cyclictest0-21swapper/209:58:252
27602995248,3cyclictest0-21swapper/209:58:252
153402500,3sleep115338-21turbostat11:35:011
90432490,2sleep10-21swapper/110:05:161
313462390,1sleep20-21swapper/212:10:142
2758999368,13cyclictest31687-21munin-run09:45:010
2759399346,20cyclictest17147-21users11:35:231
18672340,3sleep32760799cyclictest07:20:243
120532330,5sleep230-21rcuc/210:10:232
27602993219,8cyclictest16035-40dnf10:20:242
27602993211,14cyclictest28145-21users12:00:262
2758999327,12cyclictest3731-21awk09:55:010
2759399305,13cyclictest16295-21diskmemload11:55:151
2759399301,14cyclictest7118-21/usr/sbin/munin07:35:201
2759399301,14cyclictest7118-21/usr/sbin/munin07:35:201
2759399301,12cyclictest4312-21users07:25:261
2760299297,14cyclictest11516-21diskstats11:25:142
27602992910,6cyclictest31559-21sendmail_mailqu10:55:242
2759399297,13cyclictest2037-21expr09:50:141
2759399296,8cyclictest0-21swapper/112:00:021
2759399295,15cyclictest16295-21diskmemload11:45:261
2759399294,15cyclictest0-21swapper/111:15:151
2759399292,14cyclictest5614-21fschecks_count12:25:161
277342280,4sleep227738-21kernelversion08:20:182
2760299283,12cyclictest29852-21cpuspeed_turbos09:40:122
27602992815,7cyclictest32555-40dnf07:18:552
2759399285,16cyclictest23808-21kworker/u8:211:54:351
2759399284,13cyclictest22081-21chrt08:07:371
2759399281,15cyclictest0-21swapper/110:55:271
2758999285,10cyclictest11568-21idleruntime-cro12:40:010
2760299277,10cyclictest16295-21diskmemload12:15:142
2760299275,9cyclictest25713-21awk10:45:012
2760299274,5cyclictest17856-21dump-pmu-power07:59:562
2759399275,8cyclictest0-21swapper/112:24:081
2759399275,15cyclictest7856-21kworker/u8:010:30:511
2759399273,18cyclictest87550irq/128-enp1s0-11:01:311
2759399273,12cyclictest22170-21sed10:35:171
2759399272,22cyclictest0-21swapper/109:25:221
27593992712,7cyclictest28789-21strings07:10:191
2759399271,13cyclictest21733-21kernelversion09:20:171
2758999279,15cyclictest2770-21munin-run12:20:010
2758999279,15cyclictest15680-21munin-run07:55:010
2758999271,15cyclictest18332-21kworker/0:009:18:420
2758999271,10cyclictest12875-21kworker/0:211:28:400
2760299264,7cyclictest30109-21perf10:55:012
2760299264,10cyclictest7971-21/usr/sbin/munin12:30:152
2760299262,14cyclictest1750-21/usr/sbin/munin09:50:212
2760299262,10cyclictest6564-21fschecks_count10:00:142
2759399268,15cyclictest600-21date12:15:011
2759399266,3cyclictest31261-21sed09:40:251
2759399265,15cyclictest29396-21kworker/u8:109:45:021
2759399264,12cyclictest17685-21proc_pri09:10:221
27593992614,8cyclictest25965-21smart_sda08:15:251
2759399261,23cyclictest0-21swapper/109:05:211
2759399261,18cyclictest0-21swapper/111:10:211
2759399261,16cyclictest0-21swapper/110:44:381
27593992610,13cyclictest28661-21cut10:50:151
2758999262,9cyclictest10583-21kworker/0:111:31:440
2758999261,19cyclictest25614-21kworker/0:111:01:160
2760299259,4cyclictest87550irq/128-enp1s0-07:35:112
2760299259,4cyclictest87550irq/128-enp1s0-07:35:112
2760299257,5cyclictest16295-21diskmemload11:05:172
2760299257,5cyclictest16295-21diskmemload11:05:172
2760299257,11cyclictest18686-21cat09:15:022
2760299257,11cyclictest17495-21/usr/sbin/munin10:25:152
2760299253,9cyclictest20736-21if_err_enp1s011:45:162
2760299251,13cyclictest5391-21cstates12:25:132
2759399254,13cyclictest16295-21diskmemload11:29:481
2759399252,2cyclictest87550irq/128-enp1s0-10:19:121
27593992513,6cyclictest13476-21latency_hist07:50:011
2759399251,17cyclictest0-21swapper/109:15:451
27593992510,3cyclictest11541-21cut10:10:171
2759399250,19cyclictest11638-21kworker/u8:210:23:131
2759399250,17cyclictest31486-21if_err_enp2s007:15:181
2758999251,8cyclictest27179-21kworker/0:207:23:020
2758999251,16cyclictest11586-21kworker/0:109:33:300
2760299246,6cyclictest5644-21ntp_states11:10:222
2760299246,5cyclictest29462-21sort12:05:172
2760299243,5cyclictest87550irq/128-enp1s0-08:25:242
2760299242,13cyclictest22154-21/usr/sbin/munin11:50:162
2760299242,13cyclictest22051-21pmu-power09:20:212
2760299241,8cyclictest0-21swapper/209:25:182
2759399248,2cyclictest0-21swapper/108:31:361
2759399247,14cyclictest26227-21/usr/sbin/munin09:30:181
2759399246,4cyclictest29831-21ntp_states12:05:221
2759399245,16cyclictest0-21swapper/110:25:231
2759399243,10cyclictest12459-21df_inode09:00:141
2759399241,20cyclictest6896-21munin-run07:35:011
2759399240,16cyclictest87550irq/128-enp1s0-09:55:141
2759399240,16cyclictest87550irq/128-enp1s0-09:55:141
2758999242,9cyclictest2731-21kworker/0:209:56:560
2758999242,9cyclictest2731-21kworker/0:209:56:560
2758999242,8cyclictest28555-21kworker/0:308:24:180
2758999241,5cyclictest0-21swapper/011:05:170
2758999241,5cyclictest0-21swapper/011:05:170
2760799239,4cyclictest20508-21perl11:45:143
2760799233,16cyclictest16035-40dnf10:20:263
27607992316,4cyclictest7416-21gltestperf11:15:153
2760299237,8cyclictest426-21/usr/sbin/munin07:20:192
2760299237,5cyclictest15302-21sshd09:05:192
2760299236,3cyclictest12050irq/123-ahci[0010:35:012
2760299235,14cyclictest10459-21pmu-power07:40:222
2760299232,12cyclictest87350irq/126-enp1s0-09:30:122
2760299231,9cyclictest10368-21sed11:20:262
2760299231,10cyclictest19277-21perl09:15:152
2759399237,4cyclictest28558-21ntpq09:35:201
2759399237,13cyclictest16721-21/usr/sbin/munin07:55:171
2759399233,6cyclictest27149-21/usr/sbin/munin10:45:231
2759399233,5cyclictest9775-21irqstats11:20:191
27593992319,3cyclictest0-21swapper/108:38:111
27593992319,2cyclictest0-21swapper/112:35:251
2759399231,16cyclictest0-21swapper/108:25:231
2759399231,15cyclictest87550irq/128-enp1s0-12:32:201
27593992311,4cyclictest11447-21sed07:45:011
2759399231,10cyclictest27320-21fschecks_count08:20:151
2759399231,10cyclictest15687-21perf07:55:011
2759399231,10cyclictest0-21swapper/112:15:341
2758999237,2cyclictest15354-21kworker/0:108:15:020
2758999231,8cyclictest16080-21kworker/0:209:10:300
2758999231,5cyclictest0-21swapper/011:18:390
2758999231,13cyclictest8017-21kworker/0:110:04:180
2758999230,8cyclictest7370-21kworker/0:008:51:260
2760799228,6cyclictest0-21swapper/309:26:443
2760799221,16cyclictest0-21swapper/310:41:293
2760799221,14cyclictest0-21swapper/311:12:043
2760799221,10cyclictest0-21swapper/310:37:193
2760299226,4cyclictest17610-21cat11:40:022
2760299223,6cyclictest87350irq/126-enp1s0-12:35:162
2760299222,8cyclictest27512-21/usr/sbin/munin09:35:262
2760299222,5cyclictest27913-21munin-run10:50:012
2760299221,9cyclictest14689-21taskset10:19:512
27602992214,5cyclictest14435-21ntp_states11:30:222
2759399226,8cyclictest7191-21sshd08:46:151
2759399226,8cyclictest7191-21sshd08:46:151
2759399224,15cyclictest7898-21taskset10:02:161
2759399224,10cyclictest18314-21fschecks_time11:40:141
2759399222,17cyclictest24661-21sort08:15:021
2759399221,15cyclictest0-21swapper/108:50:071
2759399221,15cyclictest0-21swapper/108:00:171
2759399221,15cyclictest0-21swapper/107:25:011
2759399220,15cyclictest32183-21kworker/u8:111:07:481
2759399220,15cyclictest32183-21kworker/u8:111:07:481
2758999225,14cyclictest21943-21awk11:50:020
2758999222,9cyclictest30429-21kworker/0:112:10:520
2758999222,9cyclictest15165-21kworker/0:011:57:160
2758999222,8cyclictest7386-21kworker/0:308:56:360
2758999222,8cyclictest5716-21kworker/0:010:08:200
2758999222,8cyclictest32380-21kworker/0:107:19:120
2758999222,8cyclictest18808-21kworker/0:110:34:260
2758999221,8cyclictest28220-21kworker/0:212:05:560
2758999221,8cyclictest25614-21kworker/0:110:57:060
2758999221,5cyclictest0-21swapper/009:49:110
2758999221,10cyclictest8017-21kworker/0:110:14:400
2758999220,8cyclictest30429-21kworker/0:112:20:200
155692221,17sleep015601-21switchtime09:05:240
2760799216,4cyclictest24349-21chrt11:55:103
2760799211,12cyclictest4920-21/usr/sbin/munin07:30:163
2760799211,12cyclictest0-21swapper/312:35:073
2760799211,10cyclictest27280-21latency_hist09:35:013
27607992110,3cyclictest8636-21memory08:50:173
27607992110,3cyclictest808-21snmpd08:28:443
2760299216,12cyclictest26077-21cat08:15:262
27602992113,5cyclictest6183-21smart_sda07:30:242
27602992113,5cyclictest3340-21diskstats12:20:142
27602992112,6cyclictest8070-21diskstats08:50:132
27602992111,7cyclictest9921-21/usr/sbin/munin08:55:212
2760299211,13cyclictest0-21swapper/207:25:262
2760299210,8cyclictest12120-21/usr/sbin/munin09:00:152
2759399210,15cyclictest0-21swapper/108:40:501
2758999212,9cyclictest6508-21kworker/0:011:22:280
2758999212,9cyclictest11586-21kworker/0:109:37:320
2758999212,8cyclictest27179-21kworker/0:207:30:420
2758999212,14cyclictest15165-21kworker/0:011:44:580
2758999212,11cyclictest23118-21/usr/sbin/munin09:25:270
2758999211,9cyclictest25614-21kworker/0:111:11:120
2758999211,9cyclictest15971-21kworker/0:010:23:080
2758999211,8cyclictest30429-21kworker/0:112:31:020
2758999211,6cyclictest2914-21kworker/0:108:45:460
2758999211,6cyclictest2914-21kworker/0:108:45:460
2758999211,2cyclictest12185-21kworker/0:208:00:180
239722211,3sleep00-21swapper/011:53:110
208032211,3sleep320929-21users10:30:273
2760799205,6cyclictest0-21swapper/312:29:463
2760799204,9cyclictest39-21rcuc/307:40:183
2760799204,8cyclictest0-21swapper/309:45:193
2760799204,6cyclictest0-21swapper/309:50:483
2760799202,10cyclictest12120-21/usr/sbin/munin09:00:253
2760799201,6cyclictest0-21swapper/311:38:223
2760799201,5cyclictest0-21swapper/310:45:413
2760799201,11cyclictest0-21swapper/310:19:003
2760799201,10cyclictest19878-21latency_hist11:45:023
2760799201,10cyclictest0-21swapper/309:40:183
2760799201,10cyclictest0-21swapper/309:23:393
2760799200,5cyclictest0-21swapper/310:59:003
27602992012,5cyclictest9026-21hddtemp_smartct10:05:162
27602992012,5cyclictest25417-21ntp_states11:55:222
27602992012,5cyclictest18104-21diskstats11:40:132
27602992011,6cyclictest3748-21/usr/sbin/munin08:40:142
27602992010,7cyclictest6769-21/usr/sbin/munin11:15:122
2760299200,9cyclictest5863-21taskset08:45:152
2760299200,9cyclictest5863-21taskset08:45:152
2758999206,2cyclictest25614-21kworker/0:110:52:340
2758999202,9cyclictest23353-21kworker/0:210:42:420
2758999202,9cyclictest10583-21kworker/0:111:35:360
2758999202,8cyclictest8017-21kworker/0:110:16:120
2758999202,7cyclictest6840-21kworker/0:007:39:240
2758999202,7cyclictest6840-21kworker/0:007:39:240
2758999201,9cyclictest12185-21kworker/0:208:05:040
2758999201,6cyclictest15165-21kworker/0:012:04:360
2758999201,11cyclictest25928-21/usr/sbin/munin10:45:220
2758999201,10cyclictest18297-21sshd10:25:230
2758999200,9cyclictest30429-21kworker/0:112:27:160
2758999200,10cyclictest26627-21kworker/0:008:39:260
2760799198,4cyclictest0-21swapper/312:30:563
2760799198,3cyclictest4055-21sshd11:06:243
2760799198,3cyclictest4055-21sshd11:06:243
2760799196,3cyclictest28913-21grep12:05:133
2760799195,12cyclictest401ktimersoftd/312:00:113
2760799194,5cyclictest0-21swapper/312:20:393
2760799193,3cyclictest0-21swapper/309:16:403
2760799192,2cyclictest0-21swapper/309:35:143
2760799191,8cyclictest8122-21ls10:05:013
2760799191,8cyclictest0-21swapper/312:10:553
2760799191,5cyclictest0-21swapper/311:02:033
2760799191,3cyclictest29710-21grep10:50:253
2760799191,10cyclictest9809-21grep11:20:193
2760799191,10cyclictest4387-21ntpq08:40:203
2760799191,10cyclictest24093-21users08:10:263
2760799191,10cyclictest1118-21/usr/sbin/munin08:35:133
2760799190,11cyclictest0-21swapper/310:09:193
2760299196,10cyclictest15379-21runrttasks07:51:182
2760299192,9cyclictest1109-21sed11:00:202
2760299191,6cyclictest22721-21cat08:10:132
27602991911,5cyclictest31655-21/usr/sbin/munin08:30:142
27602991911,5cyclictest11511-21/usr/sbin/munin07:45:222
27602991910,6cyclictest31910-21/usr/sbin/munin09:45:132
27602991910,6cyclictest20312-21/usr/sbin/munin08:05:182
27602991910,6cyclictest18100-21/usr/sbin/munin08:00:122
27602991910,6cyclictest1465-21/usr/sbin/munin08:35:142
2759399194,5cyclictest10875-21wc08:55:201
2758999196,2cyclictest12185-21kworker/0:207:47:400
2758999191,9cyclictest27179-21kworker/0:207:13:580
2758999191,9cyclictest18808-21kworker/0:110:37:360
2758999191,8cyclictest6840-21kworker/0:007:42:160
2758999191,12cyclictest0-21swapper/009:20:460
2758999191,11cyclictest0-21swapper/008:25:220
2758999191,10cyclictest15905-21/usr/sbin/munin07:55:140
2760799183,9cyclictest39-21rcuc/308:32:193
2760799183,2cyclictest0-21swapper/307:10:243
2760799183,13cyclictest41-21ksoftirqd/307:35:183
2760799183,13cyclictest41-21ksoftirqd/307:35:183
2760799181,7cyclictest0-21swapper/310:25:413
2760799181,6cyclictest16295-21diskmemload09:55:593
2760799181,6cyclictest16295-21diskmemload09:55:593
2760799181,6cyclictest0-21swapper/309:14:453
2760799181,11cyclictest0-21swapper/307:45:133
2760799181,10cyclictest16295-21diskmemload11:31:373
2760799180,10cyclictest10622-21cpuspeed_turbos10:10:093
2758999185,2cyclictest15354-21kworker/0:108:12:000
27589991813,3cyclictest13255-21head09:00:220
2760799177,7cyclictest11466-21unixbench_singl08:55:273
2760799173,9cyclictest41-21ksoftirqd/307:28:323
2760799173,11cyclictest18100-21/usr/sbin/munin08:00:163
2760799171,6cyclictest0-21swapper/307:55:233
2760799171,2cyclictest23495-21sshd11:50:243
27607991710,4cyclictest0-21swapper/307:15:163
2758999172,2cyclictest26627-21kworker/0:008:31:520
2758999171,6cyclictest0-21swapper/007:25:120
220622171,3sleep30-21swapper/308:07:243
2760799161,2cyclictest401ktimersoftd/308:45:233
2760799161,2cyclictest401ktimersoftd/308:45:233
2760799161,14cyclictest0-21swapper/308:23:253
27607991611,3cyclictest14783-21ntpq07:50:223
2760799161,13cyclictest0-21swapper/309:05:213
27589991612,3cyclictest0-21swapper/008:41:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional