You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-02 - 22:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot6s.osadl.org (updated Sun Nov 02, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16221211693,18sleep00-21swapper/007:08:040
16244211496,13sleep10-21swapper/107:08:211
16198211494,15sleep20-21swapper/207:07:462
16194211390,18sleep30-21swapper/307:07:423
289822990,0sleep30-21swapper/311:18:073
133052790,1sleep2301ktimersoftd/208:20:162
8382750,0sleep30-21swapper/310:49:583
28972730,0sleep20-21swapper/211:25:092
249022700,7sleep1332250irq/24-eth111:46:401
249022700,7sleep1332250irq/24-eth111:46:391
213562550,1sleep10-21swapper/108:40:151
16406993914,18cyclictest0-21swapper/307:30:123
16406993717,19cyclictest0-21swapper/311:50:133
1640699311,3cyclictest17287-21ssh10:00:123
52922290,2sleep05289-21ssh12:00:130
106012280,1sleep00-21swapper/012:39:260
16405992219,2cyclictest8792-21kworker/u8:208:13:042
16405992219,2cyclictest28462-21kworker/u8:109:21:052
16405992219,2cyclictest16809-21kworker/u8:210:38:062
1640499223,2cyclictest333-21diskmemload09:45:051
176062210,1sleep20-21swapper/208:30:182
16405992118,2cyclictest6313-21kworker/u8:008:28:022
16405992118,2cyclictest6313-21kworker/u8:008:18:022
16405992118,2cyclictest6313-21kworker/u8:008:02:052
16405992118,2cyclictest4791-21kworker/u8:012:15:052
16405992118,2cyclictest4791-21kworker/u8:012:04:052
16405992118,2cyclictest4070-21kworker/u8:107:57:052
16405992118,2cyclictest31893-21kworker/u8:209:46:052
16405992118,2cyclictest31496-21kworker/u8:007:28:052
16405992118,2cyclictest31496-21kworker/u8:007:10:052
16405992118,2cyclictest28932-21kworker/u8:210:48:052
16405992118,2cyclictest27643-21kworker/u8:111:18:042
16405992118,2cyclictest26188-21kworker/u8:107:33:052
16405992118,2cyclictest25647-21kworker/u8:110:20:052
16405992118,2cyclictest20822-21kworker/u8:207:23:042
16405992118,2cyclictest20425-21kworker/u8:211:12:062
16405992118,2cyclictest20046-21kworker/u8:010:05:052
16405992118,2cyclictest12423-21kworker/u8:112:20:042
16404992118,2cyclictest9832-21kworker/u8:110:59:041
16404992118,2cyclictest25647-21kworker/u8:110:31:051
1640399211,18cyclictest0-21swapper/007:15:150
16405992017,2cyclictest6313-21kworker/u8:008:55:052
16405992017,2cyclictest28462-21kworker/u8:109:41:052
16405992017,2cyclictest28462-21kworker/u8:109:37:042
16405992017,2cyclictest28462-21kworker/u8:109:25:042
16405992017,2cyclictest26188-21kworker/u8:107:45:042
16405992017,2cyclictest22875-21kworker/u8:110:50:042
16405992017,2cyclictest20822-21kworker/u8:207:40:052
16405992017,2cyclictest20425-21kworker/u8:211:38:052
16405992017,2cyclictest1820-21kworker/u8:207:54:022
1640599201,17cyclictest0-21swapper/211:53:052
1640399200,18cyclictest0-21swapper/009:45:150
16405991916,2cyclictest6313-21kworker/u8:008:45:042
16405991916,2cyclictest6313-21kworker/u8:008:38:042
16405991916,2cyclictest6313-21kworker/u8:008:05:052
16405991916,2cyclictest27643-21kworker/u8:111:32:042
16405991916,2cyclictest27643-21kworker/u8:111:22:062
16405991916,2cyclictest14802-21kworker/u8:011:07:042
16405991916,2cyclictest14453-21kworker/u8:210:02:042
1640499197,7cyclictest332250irq/24-eth109:20:051
16404991916,2cyclictest30596-21kworker/u8:010:18:041
16404991916,2cyclictest25647-21kworker/u8:110:23:051
243062180,1sleep20-21swapper/210:40:142
16405991815,2cyclictest28932-21kworker/u8:211:00:052
16405991815,2cyclictest28462-21kworker/u8:109:31:052
16405991815,2cyclictest16325-21kworker/u8:111:41:052
16405991815,2cyclictest16325-21kworker/u8:111:41:042
16405991815,2cyclictest11178-21kworker/u8:011:49:052
16405991815,2cyclictest11178-21kworker/u8:011:49:042
16404991815,2cyclictest28932-21kworker/u8:210:51:051
16404991815,2cyclictest14453-21kworker/u8:210:13:051
16404991714,2cyclictest649-21kworker/u8:209:13:051
16404991714,2cyclictest4791-21kworker/u8:012:21:041
16404991714,2cyclictest4070-21kworker/u8:108:08:051
16404991714,2cyclictest30804-21kworker/u8:312:28:051
16404991714,2cyclictest11172-21kworker/u8:310:27:051
1640499170,2cyclictest0-21swapper/112:16:061
1640499170,2cyclictest0-21swapper/112:11:041
16403991714,2cyclictest0-21swapper/012:08:270
16403991714,2cyclictest0-21swapper/009:28:060
1640399170,15cyclictest0-21swapper/007:40:280
16406991612,2cyclictest0-21swapper/312:30:003
16405991613,2cyclictest6313-21kworker/u8:008:51:052
16405991613,2cyclictest28932-21kworker/u8:210:57:052
16404991613,2cyclictest6313-21kworker/u8:008:37:041
16404991613,2cyclictest16648-21kworker/u8:010:41:041
16404991613,2cyclictest14453-21kworker/u8:210:04:051
16404991613,2cyclictest12423-21kworker/u8:112:08:051
16404991613,2cyclictest11178-21kworker/u8:011:39:051
16404991613,2cyclictest11178-21kworker/u8:011:33:041
1640499160,11cyclictest0-21swapper/111:15:051
1640399162,2cyclictest0-21swapper/009:56:070
1640399162,12cyclictest1219-21ssh10:50:130
1640399161,3cyclictest11915-21sensors08:15:260
16403991613,2cyclictest0-21swapper/011:57:370
16403991613,2cyclictest0-21swapper/011:26:170
16403991613,2cyclictest0-21swapper/009:40:160
16403991612,2cyclictest0-21swapper/007:10:130
16403991611,3cyclictest10818-21ssh09:53:040
1640399160,14cyclictest0-21swapper/009:10:160
1640699151,2cyclictest3479-21cut11:25:183
1640699151,12cyclictest21594-21ssh10:36:533
16406991511,2cyclictest16155-21iostat_ios09:25:183
16406991510,3cyclictest5330-21ntp_states10:20:203
1640699150,13cyclictest0-21swapper/312:18:363
16405991512,2cyclictest16750-21kworker/u8:107:18:062
1640499152,11cyclictest332250irq/24-eth111:53:241
16404991512,2cyclictest8792-21kworker/u8:208:11:041
16404991512,2cyclictest6313-21kworker/u8:008:53:051
16404991512,2cyclictest4070-21kworker/u8:107:58:051
16404991512,2cyclictest28462-21kworker/u8:109:52:041
16404991512,2cyclictest18347-21kworker/u8:108:31:051
16404991512,2cyclictest17850-21kworker/u8:209:33:051
16404991512,2cyclictest16750-21kworker/u8:107:16:051
16404991512,2cyclictest16750-21kworker/u8:107:12:041
16404991512,2cyclictest16325-21kworker/u8:111:43:051
16404991512,2cyclictest16325-21kworker/u8:111:43:051
1640499151,13cyclictest23-21ksoftirqd/109:40:051
16404991510,3cyclictest0-21swapper/108:25:231
1640499150,8cyclictest333-21diskmemload10:08:021
1640499150,2cyclictest0-21swapper/112:34:061
1640499150,2cyclictest0-21swapper/112:03:051
1640399153,2cyclictest0-21swapper/010:02:360
1640399152,7cyclictest0-21swapper/011:15:010
1640399152,7cyclictest0-21swapper/008:50:010
1640399152,11cyclictest0-21swapper/009:00:150
1640399151,2cyclictest0-21swapper/011:21:260
16403991512,2cyclictest101ktimersoftd/010:40:560
16403991512,2cyclictest0-21swapper/010:05:140
16403991512,2cyclictest0-21swapper/009:31:100
1640399151,12cyclictest5129-21runrttasks09:22:430
1640399151,12cyclictest28312-21ntp_kernel_err11:50:190
1640399151,12cyclictest20259-21ssh11:09:380
1640399151,12cyclictest10568-21sh11:32:050
1640699149,3cyclictest15939-21ssh09:58:313
1640699141,3cyclictest18020-21ssh11:06:133
1640699141,2cyclictest23991-21ssh11:45:293
1640699141,2cyclictest23991-21ssh11:45:283
1640699140,13cyclictest8528-21ssh09:50:183
1640699140,13cyclictest0-21swapper/311:39:383
1640699140,13cyclictest0-21swapper/308:45:273
1640699140,12cyclictest4538-21ssh09:46:043
1640699140,12cyclictest0-21swapper/311:55:123
1640599141,2cyclictest6476-21apt-get12:35:122
1640499149,3cyclictest8163-21sh09:16:541
1640499149,3cyclictest0-21swapper/109:36:341
1640499142,7cyclictest0-21swapper/108:00:001
1640499142,6cyclictest0-21swapper/108:50:011
1640499142,11cyclictest221ktimersoftd/111:23:531
1640499141,2cyclictest15945-21apt-get11:05:111
16404991411,2cyclictest26188-21kworker/u8:107:41:051
16404991411,2cyclictest26188-21kworker/u8:107:35:041
16404991411,2cyclictest20822-21kworker/u8:207:25:051
1640499141,11cyclictest30957-21munin-node1
1640499140,2cyclictest0-21swapper/108:23:051
1640499140,12cyclictest0-21swapper/111:03:371
1640399149,3cyclictest26827-21ssh11:15:240
1640399142,6cyclictest0-21swapper/012:10:000
1640399142,10cyclictest0-21swapper/008:34:410
16403991411,2cyclictest0-21swapper/012:28:290
16403991411,2cyclictest0-21swapper/010:47:480
16403991411,2cyclictest0-21swapper/010:30:040
16403991411,2cyclictest0-21swapper/009:18:340
1640399141,10cyclictest0-21swapper/008:39:590
16403991410,2cyclictest0-21swapper/010:21:130
1640399140,2cyclictest333-21diskmemload10:37:270
1640399140,12cyclictest333-21diskmemload11:45:100
1640399140,12cyclictest333-21diskmemload11:45:090
1640699139,2cyclictest22112-21ssh09:30:473
1640699138,3cyclictest0-21swapper/312:10:273
16406991310,2cyclictest4656-21ssh12:00:093
16406991310,2cyclictest0-21swapper/312:37:163
16406991310,2cyclictest0-21swapper/311:34:273
16406991310,2cyclictest0-21swapper/311:03:173
16406991310,2cyclictest0-21swapper/310:53:303
16406991310,2cyclictest0-21swapper/310:40:263
16406991310,2cyclictest0-21swapper/310:12:483
1640699130,3cyclictest16543-21ssh10:31:363
1640699130,2cyclictest0-21swapper/310:15:193
1640699130,2cyclictest0-21swapper/309:39:493
1640699130,12cyclictest0-21swapper/310:26:173
1640699130,12cyclictest0-21swapper/309:44:133
1640699130,12cyclictest0-21swapper/309:15:383
1640699130,12cyclictest0-21swapper/309:13:273
1640699130,11cyclictest2625-21ssh12:30:243
1640599139,2cyclictest13122-21ssh12:08:212
16405991310,2cyclictest0-21swapper/210:26:372
16404991310,2cyclictest30610-21kworker/u8:009:05:051
16404991310,2cyclictest21366-21kworker/u8:211:59:041
16404991310,2cyclictest14802-21kworker/u8:011:10:051
1640499130,2cyclictest333-21diskmemload10:45:161
1640499130,2cyclictest333-21diskmemload10:38:151
1640499130,2cyclictest21606-21ps07:20:181
1640499130,2cyclictest0-21swapper/111:25:441
1640499130,11cyclictest0-21swapper/112:37:261
1640499130,11cyclictest0-21swapper/107:30:201
1640399139,2cyclictest333-21diskmemload11:40:020
1640399139,2cyclictest333-21diskmemload11:40:010
1640399131,11cyclictest101ktimersoftd/009:05:160
16403991311,1cyclictest0-21swapper/010:57:160
1640399131,10cyclictest14542-21chrt08:23:590
16403991310,2cyclictest0-21swapper/012:34:540
16403991310,2cyclictest0-21swapper/011:38:330
16403991310,2cyclictest0-21swapper/010:10:160
16403991310,2cyclictest0-21swapper/008:05:230
1640399130,12cyclictest0-21swapper/012:20:070
1640399130,12cyclictest0-21swapper/009:35:470
1640399130,11cyclictest333-21diskmemload12:16:190
1640699129,2cyclictest11163-21ssh09:20:163
1640699129,2cyclictest0-21swapper/312:20:183
1640699122,6cyclictest0-21swapper/307:55:003
1640699120,2cyclictest0-21swapper/311:22:153
1640699120,2cyclictest0-21swapper/310:55:063
1640699120,2cyclictest0-21swapper/310:06:043
1640699120,11cyclictest18873-21ssh11:40:263
1640699120,11cyclictest18873-21ssh11:40:253
1640699120,11cyclictest0-21swapper/312:05:453
1640699120,11cyclictest0-21swapper/311:10:353
1640599129,2cyclictest0-21swapper/212:31:082
1640599129,2cyclictest0-21swapper/210:31:022
1640599122,7cyclictest0-21swapper/209:09:022
1640599120,2cyclictest0-21swapper/211:56:342
1640599120,2cyclictest0-21swapper/209:11:372
1640599120,11cyclictest0-21swapper/212:25:412
1640599120,11cyclictest0-21swapper/212:12:582
1640599120,11cyclictest0-21swapper/210:13:372
1640599120,10cyclictest4324-21nscd10:16:402
1640499129,2cyclictest22438-21kworker/u8:209:03:041
1640499129,2cyclictest0-21swapper/108:55:221
1640499120,2cyclictest0-21swapper/109:26:481
1640499120,2cyclictest0-21swapper/107:50:141
1640499120,11cyclictest0-21swapper/109:55:051
1640399129,2cyclictest333-21diskmemload11:01:520
1640399129,2cyclictest101ktimersoftd/010:25:150
1640399129,2cyclictest0-21swapper/008:10:190
1640399129,2cyclictest0-21swapper/007:25:250
1640399121,2cyclictest0-21swapper/010:15:580
1640399120,11cyclictest0-21swapper/007:30:120
1640699116,3cyclictest4906-21cut08:00:133
1640699116,3cyclictest16227-21head08:25:263
1640599118,2cyclictest0-21swapper/209:50:072
1640499118,2cyclictest0-21swapper/108:17:401
1640399116,3cyclictest0-21swapper/007:20:190
1640399112,7cyclictest0-21swapper/008:25:230
1640399111,3cyclictest0-21swapper/008:55:140
1640699106,2cyclictest30859-21telnet07:45:103
1640699105,3cyclictest0-21swapper/308:30:153
1640699102,6cyclictest0-21swapper/308:40:243
1640699102,6cyclictest0-21swapper/307:10:243
1640699100,2cyclictest0-21swapper/307:35:013
1640599107,2cyclictest0-21swapper/209:55:192
1640599101,6cyclictest0-21swapper/209:16:592
1640599100,7cyclictest0-21swapper/207:35:592
1640399107,2cyclictest0-21swapper/008:00:140
1640399104,4cyclictest930-21grep07:50:140
1640399104,4cyclictest32207-21unixbench_singl07:45:250
1640399104,4cyclictest2939-21df_abs07:55:150
1640399100,6cyclictest0-21swapper/008:53:440
164069994,3cyclictest8921-21aten2.4-expect08:10:133
164069994,3cyclictest4654-21avahi-daemon07:15:293
164069994,3cyclictest31251-21grep09:05:133
164069994,3cyclictest29476-21grep09:00:143
164069994,3cyclictest25503-21hddtemp_smartct08:50:143
164069993,4cyclictest3277-21cut07:55:173
164069992,6cyclictest0-21swapper/307:40:273
164069991,3cyclictest13234-21df08:20:153
164059995,2cyclictest21748-21grep08:40:182
164059990,7cyclictest0-21swapper/209:00:102
164039993,4cyclictest27034-21grep07:35:130
164039993,4cyclictest20907-21apt-get08:40:120
164069983,3cyclictest4367-21nscd08:05:133
164069983,3cyclictest27176-21grep08:55:143
164069983,3cyclictest23984-21unixbench_multi07:25:263
164069983,3cyclictest19460-21chrt08:35:163
164069982,4cyclictest31496-21kworker/u8:007:20:133
164069980,2cyclictest0-21swapper/308:15:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional