You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-02 - 13:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 02, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22402251165,37sleep20-21swapper/219:05:512
25402238163,24sleep30-21swapper/319:09:433
25022236163,25sleep10-21swapper/119:09:121
24222203166,24sleep00-21swapper/019:08:100
3253021840,4sleep1265699cyclictest20:15:151
2784921160,5sleep3266799cyclictest22:30:223
28802970,2sleep2266199cyclictest00:00:212
2661999693,2cyclictest0-21swapper/221:23:152
2661998985,3cyclictest0-21swapper/219:59:092
235752780,2sleep0264999cyclictest23:35:180
326212710,4sleep011-21rcuc/022:40:290
2661996561,3cyclictest0-21swapper/220:46:002
2661996561,3cyclictest0-21swapper/220:46:002
2661993818,2cyclictest2615-21kworker/2:122:26:262
2661993734,2cyclictest0-21swapper/219:16:162
2661993716,2cyclictest2615-21kworker/2:120:38:022
264999368,13cyclictest17905-21perf22:10:010
2661993524,5cyclictest2615-21kworker/2:100:14:022
85652340,4sleep230-21rcuc/219:20:172
2661993425,2cyclictest2615-21kworker/2:120:58:242
2661993329,2cyclictest2615-21kworker/2:120:06:482
266199332,11cyclictest2615-21kworker/2:123:24:102
2661993316,6cyclictest2615-21kworker/2:119:29:102
266199331,24cyclictest2615-21kworker/2:122:49:122
264999329,11cyclictest11257-21perf21:55:000
264999329,11cyclictest11257-21perf21:55:000
264999321,5cyclictest30188-21kworker/0:121:34:550
2661993124,2cyclictest2615-21kworker/2:100:07:162
2661993124,2cyclictest2615-21kworker/2:100:07:162
266199302,22cyclictest2615-21kworker/2:121:44:162
2661993019,3cyclictest2615-21kworker/2:119:31:062
2661993016,3cyclictest2615-21kworker/2:120:29:142
265699301,22cyclictest0-21swapper/120:09:561
266799291,10cyclictest0-21swapper/321:33:013
2661992924,3cyclictest2615-21kworker/2:120:04:262
2661992921,2cyclictest2615-21kworker/2:122:39:442
266199291,10cyclictest2615-21kworker/2:121:53:082
266199291,10cyclictest2615-21kworker/2:121:53:082
264999292,12cyclictest4003-21awk00:05:010
106452290,2sleep011-21rcuc/000:20:010
2667992813,9cyclictest23778-21diskmemload21:57:143
266199289,3cyclictest2615-21kworker/2:121:48:582
266199282,20cyclictest2615-21kworker/2:100:32:402
2661992819,2cyclictest2615-21kworker/2:100:35:222
2661992816,3cyclictest2615-21kworker/2:120:52:422
2661992815,7cyclictest2615-21kworker/2:100:18:362
2661992814,7cyclictest2615-21kworker/2:121:11:002
266199281,25cyclictest2615-21kworker/2:122:02:442
2661992812,3cyclictest2615-21kworker/2:120:10:562
2661992811,3cyclictest2615-21kworker/2:123:01:362
266199280,26cyclictest2615-21kworker/2:121:55:542
266199280,13cyclictest2615-21kworker/2:123:30:462
264999281,12cyclictest18668-21awk23:25:010
94222270,2sleep00-21swapper/019:20:270
266799271,10cyclictest0-21swapper/321:39:493
266199272,19cyclictest2615-21kworker/2:122:54:122
266199271,19cyclictest2615-21kworker/2:123:49:542
2661992711,3cyclictest2615-21kworker/2:100:24:562
266199271,12cyclictest2615-21kworker/2:120:44:522
264999271,12cyclictest18376-21kworker/0:123:29:270
264999270,11cyclictest29613-21kworker/0:023:54:260
266799261,17cyclictest0-21swapper/320:15:013
266799261,16cyclictest0-21swapper/300:05:023
266799261,16cyclictest0-21swapper/300:05:023
266799261,14cyclictest0-21swapper/321:54:133
266799261,14cyclictest0-21swapper/321:54:133
266199265,2cyclictest2615-21kworker/2:120:20:162
266199262,14cyclictest2615-21kworker/2:123:37:242
266199262,10cyclictest2615-21kworker/2:119:50:142
2661992614,3cyclictest2615-21kworker/2:123:15:262
266199261,19cyclictest2615-21kworker/2:121:39:162
266199261,16cyclictest2615-21kworker/2:119:44:022
2661992611,3cyclictest2615-21kworker/2:119:49:082
266199261,11cyclictest2615-21kworker/2:121:32:582
266199260,11cyclictest2615-21kworker/2:100:27:342
265699261,17cyclictest0-21swapper/122:39:071
264999260,13cyclictest11899-21kworker/0:323:09:050
264999260,11cyclictest4546-21kworker/0:222:15:550
264999260,11cyclictest4546-21kworker/0:221:41:230
264999260,11cyclictest11781-21kworker/0:023:14:330
266799251,7cyclictest11741-21memory00:20:203
266199258,2cyclictest2615-21kworker/2:122:57:322
266199251,14cyclictest2615-21kworker/2:120:31:482
2661992510,3cyclictest2615-21kworker/2:121:03:142
264999252,14cyclictest4546-21kworker/0:222:22:150
264999251,7cyclictest10207-21kworker/0:220:52:420
264999251,11cyclictest30391-21perf21:25:010
309722241,2sleep231222-21head21:25:182
266799241,9cyclictest0-21swapper/322:01:093
266799240,8cyclictest0-21swapper/300:26:463
266199248,3cyclictest2615-21kworker/2:121:05:222
266199246,3cyclictest2615-21kworker/2:123:44:182
266199241,12cyclictest2615-21kworker/2:122:14:162
265699241,20cyclictest24509-21munin-run22:25:021
265699241,17cyclictest0-21swapper/122:50:101
265699241,17cyclictest0-21swapper/121:41:121
265699241,11cyclictest0-21swapper/100:11:061
264999244,2cyclictest86750irq/128-enp1s0-20:00:260
264999241,7cyclictest31819-21kworker/0:223:57:320
264999241,11cyclictest7629-21awk23:00:000
264999241,11cyclictest20951-21perf19:50:000
264999240,10cyclictest20658-21kworker/0:023:32:420
142652240,3sleep30-21swapper/323:15:023
308252230,2sleep00-21swapper/021:25:140
266799232,8cyclictest20791-1kworker/u9:222:50:173
266199239,3cyclictest2615-21kworker/2:123:07:362
266199237,3cyclictest2615-21kworker/2:123:27:062
266199235,3cyclictest2615-21kworker/2:119:37:322
266199232,14cyclictest2615-21kworker/2:122:40:342
2661992312,2cyclictest2615-21kworker/2:121:17:482
266199231,16cyclictest2615-21kworker/2:122:07:122
265699232,15cyclictest0-21swapper/120:27:151
265699232,11cyclictest4955-21munin-node20:10:301
265699231,7cyclictest0-21swapper/122:25:171
265699231,5cyclictest0-21swapper/100:38:411
265699231,20cyclictest0-21swapper/121:57:141
2656992311,7cyclictest0-21swapper/121:20:591
265699231,16cyclictest0-21swapper/121:30:051
265699231,15cyclictest23778-21diskmemload21:52:401
265699231,15cyclictest23778-21diskmemload21:52:401
265699231,15cyclictest0-21swapper/122:02:321
265699231,13cyclictest7412-21taskset21:45:131
264999234,2cyclictest86750irq/128-enp1s0-20:13:280
264999231,11cyclictest30450-21kworker/0:019:10:040
266799224,9cyclictest0-21swapper/321:16:073
266799222,11cyclictest19959-21ntp_states23:25:273
266199228,3cyclictest2615-21kworker/2:123:11:002
266199227,3cyclictest2615-21kworker/2:123:56:282
266199225,3cyclictest2615-21kworker/2:120:16:202
2661992214,3cyclictest2615-21kworker/2:119:11:212
266199221,3cyclictest0-21swapper/222:23:252
265699228,2cyclictest0-21swapper/122:44:271
265699223,6cyclictest20791-1kworker/u9:200:26:591
265699221,6cyclictest0-21swapper/123:51:371
2656992214,3cyclictest0-21swapper/123:28:441
2656992214,3cyclictest0-21swapper/121:11:571
265699221,3cyclictest21759-21hddtemp_smartct23:30:201
265699221,3cyclictest15994-21if_enp1s000:30:171
265699221,18cyclictest28158-21gunzip22:30:261
265699221,18cyclictest23778-21diskmemload23:45:021
265699221,18cyclictest16040-21awk23:15:261
265699221,15cyclictest0-21swapper/122:47:541
2656992211,3cyclictest0-21swapper/122:10:201
265699221,13cyclictest0-21swapper/100:07:221
265699221,13cyclictest0-21swapper/100:07:221
265699221,10cyclictest0-21swapper/121:19:221
264999225,6cyclictest2130-21kworker/0:021:39:270
264999224,6cyclictest31770-21kworker/0:220:16:200
264999224,11cyclictest780-21rngd20:06:460
264999222,2cyclictest86750irq/128-enp1s0-21:19:270
264999221,5cyclictest0-21swapper/022:30:170
264999220,7cyclictest6393-21kworker/0:123:04:150
264999220,12cyclictest12658-21kworker/0:200:28:510
264999220,10cyclictest9862-21kworker/0:019:30:070
266799215,6cyclictest0-21swapper/323:16:383
266799212,8cyclictest23135-21munin-run19:55:003
266799211,13cyclictest0-21swapper/321:05:333
266799211,13cyclictest0-21swapper/319:48:293
266799211,13cyclictest0-21swapper/319:22:133
266799211,12cyclictest0-21swapper/320:53:093
266799211,12cyclictest0-21swapper/319:12:463
266799211,11cyclictest1984-21/usr/sbin/munin00:00:173
266799211,11cyclictest0-21swapper/320:25:093
266199217,2cyclictest2615-21kworker/2:122:30:562
266199212,9cyclictest2615-21kworker/2:122:18:082
2661992115,3cyclictest2615-21kworker/2:123:50:262
265699216,6cyclictest3060-21ntp_states00:00:231
2656992117,3cyclictest0-21swapper/119:54:501
265699211,4cyclictest8184-21df23:00:161
2656992113,3cyclictest0-21swapper/123:11:081
265699211,18cyclictest0-21swapper/120:00:201
265699211,18cyclictest0-21swapper/119:11:251
265699211,15cyclictest0-21swapper/122:58:051
265699211,15cyclictest0-21swapper/121:07:301
265699211,15cyclictest0-21swapper/120:55:121
265699211,15cyclictest0-21swapper/120:52:261
265699211,15cyclictest0-21swapper/120:46:251
265699211,15cyclictest0-21swapper/120:46:251
265699211,14cyclictest0-21swapper/120:33:131
265699211,13cyclictest0-21swapper/100:16:001
265699211,11cyclictest19321-21diskstats19:45:171
265699210,15cyclictest0-21swapper/123:05:281
265699210,15cyclictest0-21swapper/121:00:211
264999216,8cyclictest773-21kworker/0:322:51:430
264999212,7cyclictest22940-21kworker/0:319:59:080
264999211,4cyclictest0-21swapper/021:14:540
264999210,3cyclictest28819-21kworker/0:122:39:150
266799205,5cyclictest0-21swapper/323:24:303
266799205,5cyclictest0-21swapper/322:23:033
266799202,10cyclictest25568-21/usr/sbin/munin20:00:283
266799201,8cyclictest0-21swapper/300:37:323
266799201,11cyclictest27772-21/usr/sbin/munin20:05:183
266799200,10cyclictest0-21swapper/321:46:273
265699202,16cyclictest0-21swapper/120:40:031
2656992017,2cyclictest0-21swapper/121:38:561
2656992016,3cyclictest0-21swapper/120:37:171
2656992013,2cyclictest0-21swapper/122:08:561
2656992013,2cyclictest0-21swapper/120:20:251
2656992012,3cyclictest0-21swapper/123:24:581
265699201,17cyclictest0-21swapper/119:59:301
2656992011,5cyclictest784-21NetworkManager19:39:241
265699201,12cyclictest21-21rcuc/122:15:191
265699201,11cyclictest649-21needreboot23:55:231
265699201,10cyclictest9909-21sh19:25:011
265699201,10cyclictest23618-21df23:35:171
264999206,2cyclictest86650irq/127-enp1s0-20:30:030
264999205,8cyclictest3897-21kworker/0:020:29:040
264999205,8cyclictest12536-21kworker/0:120:43:550
264999205,5cyclictest0-21swapper/019:17:260
264999204,7cyclictest31770-21kworker/0:220:22:160
264999202,10cyclictest7078-21/usr/sbin/munin21:45:160
2649992015,2cyclictest16542-21dump-pmu-power19:40:000
2649992013,4cyclictest21325-21tr19:50:130
264999201,12cyclictest0-21swapper/021:00:220
264999201,11cyclictest27726-21/usr/sbin/munin23:45:180
264999201,11cyclictest15261-21/usr/sbin/munin00:30:190
264999200,9cyclictest10207-21kworker/0:220:47:380
264999200,9cyclictest10207-21kworker/0:220:47:380
264999200,3cyclictest86650irq/127-enp1s0-20:35:390
264999200,10cyclictest4546-21kworker/0:222:12:550
266799198,3cyclictest5745-21/usr/sbin/munin19:15:223
266799198,3cyclictest22156-21ntp_states23:30:263
266799198,3cyclictest1184-21/usr/sbin/munin22:45:203
266799196,6cyclictest0-21swapper/322:25:463
266799195,3cyclictest0-21swapper/323:08:523
266799191,3cyclictest20558-21kworker/u8:123:47:073
2667991913,3cyclictest7727-21cut23:00:023
2667991913,3cyclictest16961-21ntpq22:05:203
2667991913,3cyclictest16333-21ntpq00:30:233
2667991913,3cyclictest14568-21grep20:45:273
2667991913,3cyclictest14568-21grep20:45:273
266799191,2cyclictest0-21swapper/321:04:293
266799191,2cyclictest0-21swapper/319:45:013
266799191,12cyclictest0-21swapper/323:35:503
266799191,11cyclictest0-21swapper/323:42:463
266799191,11cyclictest0-21swapper/323:04:143
266799191,10cyclictest6437-21/usr/sbin/munin00:10:243
266799191,10cyclictest4862-21/usr/sbin/munin21:40:173
266799191,10cyclictest31571-21sshd23:51:083
265699192,3cyclictest0-21swapper/100:23:401
2656991912,4cyclictest17375-21grep19:40:191
265699191,17cyclictest0-21swapper/119:33:451
265699191,14cyclictest0-21swapper/121:25:311
265699190,18cyclictest0-21swapper/123:43:561
265699190,18cyclictest0-21swapper/119:26:071
264999193,2cyclictest86850irq/129-enp1s0-19:42:150
264999191,8cyclictest86750irq/128-enp1s0-23:40:170
264999191,6cyclictest0-21swapper/022:47:100
264999191,10cyclictest16147-21sshd23:16:090
264999190,10cyclictest0-21swapper/000:23:490
155952191,2sleep00-21swapper/022:04:090
266799187,3cyclictest0-21swapper/300:15:323
266799183,12cyclictest32163-21/usr/sbin/munin20:15:173
266799181,9cyclictest30607-21/usr/sbin/munin21:25:193
266799181,7cyclictest29173-21egrep22:35:153
2667991813,3cyclictest18061-21hddtemp_smartct20:55:163
2667991812,3cyclictest14480-21cat19:35:013
266799181,11cyclictest0-21swapper/319:38:213
266799181,10cyclictest0-21swapper/322:17:443
266799181,10cyclictest0-21swapper/321:10:253
266799180,7cyclictest0-21swapper/321:20:273
265699181,16cyclictest0-21swapper/119:18:251
264999184,6cyclictest21368-21kworker/0:021:07:030
264999183,8cyclictest0-21swapper/022:26:210
264999183,8cyclictest0-21swapper/000:36:450
264999182,10cyclictest0-21swapper/000:05:500
264999182,10cyclictest0-21swapper/000:05:500
264999181,9cyclictest16847-21kworker/0:020:55:050
264999181,9cyclictest0-21swapper/000:15:010
15962181,3sleep30-21swapper/323:58:343
114612180,2sleep00-21swapper/021:55:060
105022181,2sleep30-21swapper/320:39:223
2667991712,3cyclictest31353-21date22:40:023
2667991711,3cyclictest19257-21grep22:10:263
266799171,13cyclictest14634-21snmpd19:25:503
266799171,10cyclictest0-21swapper/320:30:173
264999179,5cyclictest0-21swapper/019:25:160
124012171,2sleep30-21swapper/320:40:303
266799162,11cyclictest23364-21/usr/sbin/munin19:55:163
2667991612,3cyclictest0-21swapper/320:20:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional