You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-08 - 22:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jul 08, 2025 12:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
109212241164,26sleep20-21swapper/207:07:132
111402239164,23sleep30-21swapper/307:10:003
109342221191,20sleep00-21swapper/007:07:230
110192205168,24sleep10-21swapper/107:08:301
628221400,3sleep01123199cyclictest10:35:250
40962730,2sleep10-21swapper/108:05:171
109332630,4sleep110930-21awk09:35:001
20382550,4sleep21124299cyclictest09:13:582
126602550,1sleep10-21swapper/112:05:141
100832480,1sleep00-21swapper/010:45:180
100832480,1sleep00-21swapper/010:45:180
3262420,4sleep011-21rcuc/007:55:170
11231993611,12cyclictest9383-21munin-run10:45:020
11231993521,3cyclictest4333-21munin-run09:20:020
1123199351,13cyclictest19117-21kworker/0:009:07:300
1123199341,17cyclictest14601-21kworker/0:208:29:460
1123199340,11cyclictest21797-21kworker/0:010:00:240
1123199327,12cyclictest18487-21munin-run07:25:010
1123199311,13cyclictest17403-21kworker/0:110:05:390
1123199311,13cyclictest10124-21kworker/0:308:20:280
1123199310,13cyclictest20908-21kworker/0:107:39:040
1123199303,13cyclictest8739-21basename09:30:010
1123199301,7cyclictest7732-21kworker/0:212:08:360
1123199301,6cyclictest1109-21kworker/0:211:39:300
1123199301,5cyclictest7732-21kworker/0:211:54:260
1123199301,14cyclictest15043-21kworker/0:009:42:280
1123199301,14cyclictest15043-21kworker/0:009:42:280
1123199300,9cyclictest9952-21kworker/0:008:16:180
1123199300,5cyclictest31169-21kworker/0:111:41:080
1123199300,5cyclictest27279-21kworker/0:212:37:310
1123199300,14cyclictest11372-21kworker/0:110:52:440
1123199300,12cyclictest17403-21kworker/0:109:53:080
11242992923,3cyclictest22177-21sa110:00:092
1123799295,15cyclictest0-21swapper/111:51:511
1123799291,24cyclictest0-21swapper/108:03:331
1123199291,6cyclictest436-21kworker/0:210:24:390
1123199291,12cyclictest20238-21kworker/0:311:31:480
1123199291,11cyclictest12804-21kworker/0:209:45:220
1123199290,7cyclictest17403-21kworker/0:110:16:000
1123199290,13cyclictest3388-21kworker/0:011:46:020
1123199290,13cyclictest3388-21kworker/0:011:46:020
1123199290,13cyclictest13764-21kworker/0:011:03:380
1123199290,11cyclictest27072-21kworker/0:007:44:280
1124799281,12cyclictest9774-21cpuspeed_turbos10:45:143
1124799281,12cyclictest9774-21cpuspeed_turbos10:45:143
11237992825,2cyclictest0-21swapper/107:53:001
1123799281,24cyclictest12561-21ls07:10:221
1123199289,3cyclictest9482-21kworker/0:109:31:360
1123199281,10cyclictest7732-21kworker/0:212:24:280
1123199281,10cyclictest11372-21kworker/0:110:55:320
1123199280,8cyclictest20238-21kworker/0:311:27:470
1123199280,6cyclictest27589-21kworker/0:209:12:100
1123199280,12cyclictest7732-21kworker/0:212:02:180
1123199280,10cyclictest3388-21kworker/0:011:55:040
1123199280,10cyclictest20238-21kworker/0:311:14:040
1123199280,10cyclictest18075-21kworker/0:111:08:100
11237992724,2cyclictest0-21swapper/107:31:061
11237992724,2cyclictest0-21swapper/107:31:061
11237992723,3cyclictest0-21swapper/111:34:451
1123799271,20cyclictest0-21swapper/110:03:381
1123199271,10cyclictest17403-21kworker/0:110:10:510
1123199270,6cyclictest2761-21kworker/0:010:33:530
1123199270,5cyclictest16464-21kworker/0:012:14:480
1123199270,13cyclictest28483-21kworker/0:310:27:050
1123199270,13cyclictest0-21swapper/012:30:210
1124299265,8cyclictest18135-40dnf11:04:342
11237992623,2cyclictest0-21swapper/108:13:151
11237992623,2cyclictest0-21swapper/107:22:591
1123799261,19cyclictest0-21swapper/110:07:141
1123799261,17cyclictest14634-21snmpd09:52:521
1123799261,16cyclictest0-21swapper/112:10:471
11237992611,3cyclictest28473-21smart_sda11:25:281
1123199264,3cyclictest0-21swapper/011:22:290
1123199261,17cyclictest0-21swapper/009:35:180
1123199261,11cyclictest7732-21kworker/0:212:19:000
1123799254,10cyclictest32736-21latency07:55:161
1123799253,17cyclictest18135-40dnf11:04:341
11237992515,3cyclictest1664-21sed09:10:261
1123799251,22cyclictest0-21swapper/112:00:141
1123799251,21cyclictest29463-21sh07:50:011
1123799251,16cyclictest0-21swapper/109:05:151
1123799250,16cyclictest0-21swapper/109:58:591
11231992514,2cyclictest15967-21kworker/0:307:27:320
1123199251,10cyclictest19576-21kworker/0:309:57:080
1123199250,13cyclictest0-21swapper/007:15:170
1123199250,12cyclictest27589-21kworker/0:209:00:180
1123199250,11cyclictest14601-21kworker/0:208:41:160
1124299246,5cyclictest29505-21basename10:15:172
1123799244,8cyclictest0-21swapper/112:30:191
1123799242,11cyclictest15940-21diskstats09:45:141
1123799241,21cyclictest0-21swapper/108:15:241
1123799241,21cyclictest0-21swapper/107:37:111
1123799241,15cyclictest0-21swapper/112:24:181
1123799241,15cyclictest0-21swapper/110:13:081
1123799241,11cyclictest12749-21users09:35:291
1123799241,11cyclictest0-21swapper/111:16:311
1123199241,14cyclictest3661-21kworker/0:208:07:220
1123199241,11cyclictest24798-21perf11:20:020
1123199240,13cyclictest31538-21kworker/0:207:54:180
11247992320,2cyclictest0-21swapper/311:25:233
11247992319,2cyclictest21527-21/usr/sbin/munin08:45:213
1124799231,4cyclictest0-21swapper/312:15:523
1124799231,13cyclictest0-21swapper/311:38:363
1124299235,6cyclictest30185-21cat11:30:182
1123799237,3cyclictest2535-21seq09:15:141
1123799232,9cyclictest0-21swapper/109:20:261
1123799231,3cyclictest14685-21taskset08:29:531
11237992313,2cyclictest0-21swapper/108:59:071
1123799231,20cyclictest0-21swapper/110:38:511
1123799231,19cyclictest29814-21chrt09:01:331
1123799231,14cyclictest0-21swapper/112:29:421
11237992311,3cyclictest0-21swapper/109:26:341
1123799231,11cyclictest19461-21grep11:05:231
1123799231,10cyclictest30882-21cat10:20:021
1123799231,10cyclictest19511-21cat07:25:201
1123199231,13cyclictest4237-21kworker/0:109:21:160
1123199231,11cyclictest0-21swapper/008:35:020
1123199230,9cyclictest20983-21kworker/0:012:28:520
1123199230,15cyclictest20908-21kworker/0:107:48:400
154142221,17sleep315598-21/usr/sbin/munin10:55:283
1124799229,3cyclictest16305-21awk07:20:013
1124799226,4cyclictest0-21swapper/311:56:273
1124799222,7cyclictest0-21swapper/309:40:173
1124799222,7cyclictest0-21swapper/309:40:173
1124799222,10cyclictest0-21swapper/311:04:343
11247992219,2cyclictest0-21swapper/310:20:273
11247992212,3cyclictest0-21swapper/310:34:143
1124799221,14cyclictest0-21swapper/312:25:183
1124799221,11cyclictest0-21swapper/308:15:153
1124299228,4cyclictest0-21swapper/212:25:192
1124299223,5cyclictest5230-1kworker/u9:112:31:102
1123799227,7cyclictest31600-21/usr/sbin/munin11:35:261
1123799222,17cyclictest12163-21diskstats10:50:151
1123799221,9cyclictest32296-21diskmemload12:15:271
1123799221,9cyclictest0-21swapper/110:57:161
11237992214,4cyclictest26984-21munin-run11:25:011
1123799221,3cyclictest8451-21tune2fs11:55:201
1123799221,19cyclictest0-21swapper/110:25:161
1123799221,19cyclictest0-21swapper/108:50:181
1123799221,19cyclictest0-21swapper/108:43:101
1123799221,19cyclictest0-21swapper/108:36:411
1123799221,19cyclictest0-21swapper/108:36:411
1123799221,19cyclictest0-21swapper/107:16:041
1123799221,11cyclictest0-21swapper/111:45:151
1123799221,11cyclictest0-21swapper/111:45:151
1123799221,10cyclictest32309-21sed10:20:251
1123799221,10cyclictest21871-21sshd11:10:261
1123199221,18cyclictest3716-21munin-run08:05:020
1123199221,13cyclictest6764-21kworker/0:108:11:220
1123199221,11cyclictest19117-21kworker/0:008:52:360
1124799215,8cyclictest86850irq/129-enp1s0-09:15:193
1124799215,8cyclictest86550irq/126-enp1s0-07:35:233
1124799215,3cyclictest30913-21grep11:30:273
1124799214,9cyclictest32296-21diskmemload10:16:333
1124799212,10cyclictest14647-21/usr/sbin/munin12:10:163
1124799211,6cyclictest3322-21users08:00:303
1124799211,15cyclictest0-21swapper/309:21:143
1124799211,11cyclictest6695-21grep10:35:293
1124299212,15cyclictest5772-21perl10:35:192
1124299211,14cyclictest0-21swapper/210:58:332
1124299211,13cyclictest0-21swapper/210:25:242
1123799217,3cyclictest26890-21smart_sda12:35:261
1123799211,2cyclictest32296-21diskmemload10:41:181
11237992112,2cyclictest0-21swapper/109:41:331
11237992112,2cyclictest0-21swapper/109:41:331
1123799211,19cyclictest0-21swapper/108:45:581
1123799211,18cyclictest6019-21kworker/1:210:45:141
1123799211,18cyclictest6019-21kworker/1:210:45:141
1123199216,3cyclictest27589-21kworker/0:208:59:060
1123199211,12cyclictest26703-21kworker/0:207:10:030
1123199211,12cyclictest15967-21kworker/0:307:30:080
1123199211,12cyclictest15967-21kworker/0:307:30:080
1123199210,13cyclictest16886-21kworker/0:108:35:160
1123199210,13cyclictest16886-21kworker/0:108:35:160
1123199210,12cyclictest19117-21kworker/0:008:45:360
1124799207,8cyclictest0-21swapper/309:10:133
1124799206,6cyclictest0-21swapper/311:09:223
1124799204,8cyclictest86550irq/126-enp1s0-10:40:193
1124799204,3cyclictest31215-21sed07:50:323
1124799202,6cyclictest14412-21latency_hist12:10:023
1124799202,10cyclictest8955-21/usr/sbin/munin09:30:243
1124799202,10cyclictest32296-21diskmemload11:52:483
1124799201,8cyclictest25007-21/usr/sbin/munin11:20:213
1124799201,7cyclictest8703-21taskset09:29:543
1124799201,7cyclictest21289-21latency_hist08:45:013
1124799201,7cyclictest0-21swapper/310:25:133
1124799201,6cyclictest13923-21ps08:25:353
11247992014,3cyclictest19037-21/usr/sbin/munin12:20:163
11247992013,4cyclictest23871-21grep08:50:123
1124799201,16cyclictest19347-21grep09:50:273
1124799201,12cyclictest4199-21kworker/u8:209:38:163
1124799201,10cyclictest3942-21/usr/sbin/munin08:05:203
1124799201,10cyclictest21322-21sshd09:55:263
1124799201,10cyclictest0-21swapper/310:00:253
11247992010,3cyclictest86850irq/129-enp1s0-10:10:063
1124299201,7cyclictest848-21sssd_nss09:45:202
1124299201,6cyclictest8481-21ntpq10:40:232
1123799201,2cyclictest0-21swapper/111:43:041
1123799201,18cyclictest0-21swapper/108:31:411
1123799201,18cyclictest0-21swapper/108:24:491
1123799201,10cyclictest0-21swapper/110:30:521
1123799200,19cyclictest0-21swapper/107:42:531
260332191,2sleep20-21swapper/210:05:582
1124799195,9cyclictest0-21swapper/310:05:153
1124799195,3cyclictest86850irq/129-enp1s0-07:55:203
1124799195,3cyclictest86550irq/126-enp1s0-11:40:143
1124799194,3cyclictest27545-21cpu07:45:143
1124799191,8cyclictest26678-21ntp_kernel_pll_12:35:233
1124799191,6cyclictest0-21swapper/310:50:253
1124799191,5cyclictest27270-21latency_hist07:45:013
11247991915,2cyclictest0-21swapper/311:45:233
11247991915,2cyclictest0-21swapper/311:45:233
11247991914,2cyclictest31079-21if_enp2s009:05:163
11247991913,4cyclictest842-21lldpd07:22:283
11247991913,3cyclictest17068-21unixbench-2d09:45:253
1124799191,12cyclictest0-21swapper/308:35:253
1124799191,12cyclictest0-21swapper/308:35:253
1124799191,11cyclictest842-21lldpd07:13:283
1124799191,10cyclictest6156-21cpu08:10:143
1124799191,10cyclictest25920-21/usr/sbin/munin08:55:153
1124799191,10cyclictest0-21swapper/311:15:103
1124299198,3cyclictest0-21swapper/212:10:312
1124299196,6cyclictest0-21swapper/210:20:132
1124299193,7cyclictest3942-21/usr/sbin/munin08:05:222
1124299192,5cyclictest5230-1kworker/u9:110:46:012
1124299192,5cyclictest5230-1kworker/u9:110:46:012
1124299192,11cyclictest0-21swapper/211:38:012
1124299191,8cyclictest24777-21sh11:20:012
1124299191,7cyclictest0-21swapper/211:47:012
1124299191,7cyclictest0-21swapper/211:47:012
1124299191,5cyclictest24604-21kworker/u8:011:42:212
11242991914,3cyclictest29698-21grep09:00:302
195522181,2sleep20-21swapper/209:52:332
1124799187,4cyclictest842-21lldpd08:23:593
1124799181,7cyclictest0-21swapper/312:31:013
1124799181,6cyclictest21610-21strings11:10:223
1124799181,6cyclictest0-21swapper/312:01:263
1124799181,6cyclictest0-21swapper/308:30:293
1124799181,6cyclictest0-21swapper/307:25:223
11247991812,3cyclictest28128-21acpi09:00:163
1124299183,6cyclictest86850irq/129-enp1s0-11:10:212
1124299182,15cyclictest0-21swapper/210:34:532
1124299182,10cyclictest0-21swapper/207:54:272
1124299181,7cyclictest23513-21latency_hist08:50:012
1124299181,7cyclictest0-21swapper/211:06:392
1124299181,6cyclictest14412-21latency_hist12:10:012
1124299181,6cyclictest10927-21munin-run09:35:012
1124299181,5cyclictest0-21swapper/210:51:072
1124299181,4cyclictest20099-21ntp_states12:20:232
11242991814,3cyclictest0-21swapper/209:43:512
11242991814,3cyclictest0-21swapper/209:43:512
1124299181,11cyclictest0-21swapper/209:28:332
1124299181,10cyclictest0-21swapper/208:23:062
11242991810,3cyclictest0-21swapper/210:11:162
256022171,2sleep20-21swapper/208:54:042
1124799171,3cyclictest86550irq/126-enp1s0-07:35:013
1124799171,3cyclictest86550irq/126-enp1s0-07:35:013
1124299176,3cyclictest24089-21needreboot07:35:202
1124299176,3cyclictest18023-21cat08:35:212
1124299176,3cyclictest18023-21cat08:35:212
1124299175,6cyclictest496-21jbd2/dm-0-811:56:312
1124299175,3cyclictest16831-21sshd08:34:182
1124299174,8cyclictest0-21swapper/209:16:312
1124299174,4cyclictest18100-21sensors12:15:252
1124299171,6cyclictest19952-21gltestperf08:40:162
1124299171,6cyclictest0-21swapper/211:20:092
1124299171,6cyclictest0-21swapper/208:00:452
1124299171,6cyclictest0-21swapper/207:15:432
1124299171,4cyclictest26984-21ntp_states08:55:232
1124299171,3cyclictest0-21swapper/209:24:202
11242991712,3cyclictest25940-21tail07:40:162
11242991712,3cyclictest11175-21sed12:00:202
1124299171,11cyclictest0-21swapper/211:50:182
1124299171,10cyclictest0-21swapper/208:15:482
11242991710,4cyclictest27869-21gltestperf11:25:202
225222160,2sleep20-21swapper/207:30:342
225222160,2sleep20-21swapper/207:30:342
1124299169,4cyclictest28491-21needreboot07:45:222
1124299163,6cyclictest0-21swapper/207:26:142
1124299162,5cyclictest31130-21irqstats09:05:172
1124299162,3cyclictest26523-21latency12:35:202
1124299161,9cyclictest0-21swapper/208:13:312
1124299161,8cyclictest0-21swapper/208:25:132
11242991612,3cyclictest0-21swapper/209:55:142
1124299161,14cyclictest0-21swapper/207:58:152
11242991611,3cyclictest12186-21nfs4_client09:35:222
11242991610,4cyclictest17871-21grep07:20:282
11242991510,3cyclictest11517-21cpuspeed_turbos07:10:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional