You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 22:16
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot2.osadl.org (updated Fri Nov 28, 2025 12:44:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
61792199183,11sleep10-21swapper/107:09:361
59612194178,11sleep30-21swapper/307:06:493
59582188172,11sleep00-21swapper/007:06:470
60672116100,12sleep20-21swapper/207:08:122
13042550,0sleep20-21swapper/209:57:142
108112460,0sleep10-21swapper/109:14:221
168292440,0sleep316826-21aten2_r5power_v12:15:123
63949991,7cyclictest0-21swapper/311:20:113
63899960,6cyclictest0-21swapper/212:10:102
63899960,6cyclictest0-21swapper/211:33:572
63899960,6cyclictest0-21swapper/209:30:132
63949955,0cyclictest8958-21kworker/3:109:47:243
63949955,0cyclictest8958-21kworker/3:108:40:163
63949955,0cyclictest18444-21kworker/3:010:43:223
63949951,2cyclictest0-21swapper/308:20:123
63949951,2cyclictest0-21swapper/307:45:123
63949950,5cyclictest17604-21ssh11:42:413
63949950,4cyclictest0-21swapper/312:35:143
63899950,5cyclictest0-21swapper/212:17:102
63899950,5cyclictest0-21swapper/211:55:552
63899950,5cyclictest0-21swapper/211:19:322
63899950,5cyclictest0-21swapper/207:30:312
63899950,4cyclictest12434-21ssh09:15:142
63899950,0cyclictest0-21swapper/212:30:492
63859950,5cyclictest0-21swapper/112:03:091
63859950,5cyclictest0-21swapper/111:30:251
63859950,5cyclictest0-21swapper/111:20:301
63859950,5cyclictest0-21swapper/109:59:271
63799950,5cyclictest0-21swapper/010:19:230
29239250,0sleep30-21swapper/311:00:123
1008250,0sleep20-21swapper/212:06:462
63949944,0cyclictest18444-21kworker/3:011:17:003
63949944,0cyclictest0-21swapper/309:40:013
63949943,1cyclictest31852-21grep07:59:593
63949943,1cyclictest15622-21grep11:25:173
63949943,1cyclictest0-21swapper/310:18:443
63949941,3cyclictest8358-21grep12:10:183
63949941,2cyclictest0-21swapper/308:05:123
63949941,2cyclictest0-21swapper/307:25:113
63949940,4cyclictest61-21kswapd011:45:143
63949940,4cyclictest29701-21ssh10:11:113
63949940,4cyclictest14614-21ssh10:35:593
63949940,4cyclictest0-21swapper/312:32:523
63949940,4cyclictest0-21swapper/312:25:093
63949940,4cyclictest0-21swapper/312:20:053
63949940,4cyclictest0-21swapper/312:05:133
63949940,4cyclictest0-21swapper/311:35:123
63949940,4cyclictest0-21swapper/311:05:113
63949940,4cyclictest0-21swapper/310:55:123
63949940,4cyclictest0-21swapper/310:30:133
63949940,4cyclictest0-21swapper/310:29:223
63949940,4cyclictest0-21swapper/310:00:133
63949940,4cyclictest0-21swapper/309:35:143
63949940,4cyclictest0-21swapper/309:05:143
63949940,3cyclictest9118-21diskstats07:15:023
63949940,3cyclictest9118-21diskstats07:15:013
63949940,3cyclictest714-21snmpd10:20:103
63949940,3cyclictest5265-21rm09:10:543
63949940,3cyclictest4442-21sh09:26:383
63949940,3cyclictest31523-21ssh10:45:123
63949940,3cyclictest26497-21sh11:30:233
63949940,3cyclictest22692-21ssh09:20:113
63949940,3cyclictest19100-21grep09:50:153
63949940,3cyclictest16864-21df_inode10:05:133
63949940,3cyclictest14418-21ssh09:31:343
63949940,3cyclictest12968-21irqstats08:25:143
63949940,2cyclictest0-21swapper/309:15:133
63949940,1cyclictest9317-21/usr/sbin/munin10:50:203
63949940,1cyclictest32742-21which11:50:153
63949940,1cyclictest32017-21/usr/sbin/munin08:00:193
63949940,1cyclictest19244-21/usr/sbin/munin12:00:123
63949940,1cyclictest17157-21/usr/sbin/munin07:30:153
63949940,1cyclictest12225-21/usr/sbin/munin07:20:183
63899944,0cyclictest0-21swapper/210:59:102
63899944,0cyclictest0-21swapper/210:46:212
63899944,0cyclictest0-21swapper/207:39:572
63899942,1cyclictest714-21snmpd12:02:562
63899942,1cyclictest714-21snmpd10:54:132
63899941,3cyclictest18925-21ssh11:43:282
63899940,4cyclictest10784-21ssh10:18:212
63899940,4cyclictest10441-21ssh10:01:402
63899940,4cyclictest0-21swapper/211:45:582
63899940,4cyclictest0-21swapper/211:27:582
63899940,4cyclictest0-21swapper/211:05:112
63899940,4cyclictest0-21swapper/211:04:172
63899940,4cyclictest0-21swapper/210:36:272
63899940,4cyclictest0-21swapper/210:34:322
63899940,4cyclictest0-21swapper/210:25:042
63899940,4cyclictest0-21swapper/210:10:102
63899940,4cyclictest0-21swapper/210:05:122
63899940,4cyclictest0-21swapper/209:45:232
63899940,4cyclictest0-21swapper/209:35:142
63899940,4cyclictest0-21swapper/209:25:102
63899940,4cyclictest0-21swapper/209:20:232
63899940,4cyclictest0-21swapper/208:20:132
63899940,4cyclictest0-21swapper/207:20:202
63899940,3cyclictest720-21ssh12:23:112
63899940,3cyclictest5821-21ssh11:36:302
63899940,3cyclictest3686-21ntp_states09:10:162
63899940,3cyclictest31702-21ssh11:50:072
63899940,3cyclictest26773-21cat08:55:002
63899940,3cyclictest26391-21ls11:15:002
63899940,3cyclictest25942-21ssh09:53:462
63899940,3cyclictest25331-21/usr/sbin/munin12:35:182
63899940,3cyclictest19177-21ssh10:22:202
63899940,3cyclictest14427-21sed08:30:002
63899940,3cyclictest13404-21ssh11:24:442
63899940,3cyclictest0-21swapper/209:00:112
63899940,0cyclictest0-21swapper/210:41:512
63899940,0cyclictest0-21swapper/208:49:572
63859944,0cyclictest0-21swapper/112:35:091
63859944,0cyclictest0-21swapper/111:50:111
63859944,0cyclictest0-21swapper/109:42:441
63859944,0cyclictest0-21swapper/109:39:401
63859941,2cyclictest0-21swapper/112:10:021
63859941,2cyclictest0-21swapper/109:15:141
63859941,2cyclictest0-21swapper/107:20:121
63859941,1cyclictest0-21swapper/112:25:101
63859940,4cyclictest0-21swapper/110:39:381
63859940,0cyclictest0-21swapper/112:19:081
63859940,0cyclictest0-21swapper/110:19:591
63859940,0cyclictest0-21swapper/110:13:091
63859940,0cyclictest0-21swapper/110:06:191
63799944,0cyclictest0-21swapper/011:52:210
63799944,0cyclictest0-21swapper/011:41:440
63799944,0cyclictest0-21swapper/010:48:460
63799944,0cyclictest0-21swapper/010:35:570
63799944,0cyclictest0-21swapper/009:17:490
63799943,1cyclictest27527-21sed12:20:140
63799943,1cyclictest10203-21grep08:20:140
63799942,2cyclictest714-21snmpd09:30:010
63799942,2cyclictest24370-21kworker/0:007:40:130
63799942,1cyclictest714-21snmpd08:04:540
63799942,1cyclictest28418-21ssh11:15:190
63799941,3cyclictest0-21swapper/009:45:210
63799941,2cyclictest0-21swapper/010:20:100
63799941,2cyclictest0-21swapper/009:30:130
63799940,4cyclictest30125-21ssh09:23:430
63799940,4cyclictest1802-21diskmemload11:27:150
63799940,4cyclictest0-21swapper/012:25:090
63799940,4cyclictest0-21swapper/012:00:110
63799940,4cyclictest0-21swapper/011:37:100
63799940,4cyclictest0-21swapper/011:30:110
63799940,4cyclictest0-21swapper/011:10:190
63799940,4cyclictest0-21swapper/011:05:200
63799940,4cyclictest0-21swapper/011:00:130
63799940,4cyclictest0-21swapper/010:50:130
63799940,4cyclictest0-21swapper/010:10:130
63799940,4cyclictest0-21swapper/010:05:320
63799940,4cyclictest0-21swapper/010:04:570
63799940,4cyclictest0-21swapper/009:51:270
63799940,4cyclictest0-21swapper/009:00:120
63799940,4cyclictest0-21swapper/008:45:100
63799940,4cyclictest0-21swapper/007:35:230
63799940,3cyclictest5433-21ssh10:31:300
63799940,3cyclictest31914-21memory09:40:160
63799940,3cyclictest27341-21df_inode08:55:130
63799940,3cyclictest26982-21latency_hist07:50:000
63799940,3cyclictest25434-21memory08:50:150
63799940,3cyclictest2129-21ssh09:10:040
63799940,3cyclictest19935-21diskstats08:40:130
63799940,3cyclictest15713-21ntp_states08:30:160
63799940,3cyclictest14655-21df_inode12:30:160
63799940,3cyclictest12850irq/132-ahci[0012:00:000
63799940,3cyclictest0-21swapper/010:55:100
63799940,1cyclictest8839-21munin-node07:15:010
63799940,1cyclictest19163-21latency_hist08:40:000
63799940,0cyclictest0-21swapper/011:20:290
63799940,0cyclictest0-21swapper/009:36:520
25658240,0sleep1201rcuc/109:20:551
63949931,2cyclictest0-21swapper/308:10:103
63949931,1cyclictest0-21swapper/309:55:053
63949930,3cyclictest7136-21wc08:15:123
63949930,3cyclictest270612sleep307:50:133
63949930,3cyclictest23413-21/usr/sbin/munin07:40:203
63949930,3cyclictest0-21swapper/311:55:023
63949930,3cyclictest0-21swapper/309:00:133
63949930,3cyclictest0-21swapper/308:55:103
63949930,3cyclictest0-21swapper/308:50:113
63949930,3cyclictest0-21swapper/308:45:103
63949930,3cyclictest0-21swapper/308:35:093
63949930,3cyclictest0-21swapper/308:30:023
63949930,3cyclictest0-21swapper/307:35:113
63949930,3cyclictest0-21swapper/307:10:133
63949930,2cyclictest17303-21diskstats11:10:143
63899932,1cyclictest714-21snmpd12:25:142
63899930,3cyclictest9387-21tune2fs07:15:012
63899930,3cyclictest9387-21tune2fs07:15:012
63899930,3cyclictest17654-21perl08:35:112
63899930,3cyclictest0-21swapper/209:05:142
63899930,3cyclictest0-21swapper/208:55:102
63899930,3cyclictest0-21swapper/208:40:112
63899930,3cyclictest0-21swapper/208:30:102
63899930,3cyclictest0-21swapper/208:15:112
63899930,3cyclictest0-21swapper/208:05:112
63899930,3cyclictest0-21swapper/208:00:112
63899930,3cyclictest0-21swapper/207:55:102
63899930,3cyclictest0-21swapper/207:50:132
63899930,3cyclictest0-21swapper/207:45:112
63899930,3cyclictest0-21swapper/207:40:132
63899930,3cyclictest0-21swapper/207:25:112
63899930,3cyclictest0-21swapper/207:10:112
63899930,2cyclictest4606-21aten2_r5power_p08:10:092
63899930,2cyclictest30555-21ssh09:40:062
63859933,0cyclictest0-21swapper/112:32:061
63859933,0cyclictest0-21swapper/111:13:411
63859931,2cyclictest0-21swapper/111:25:131
63859931,1cyclictest0-21swapper/112:20:111
63859931,1cyclictest0-21swapper/112:05:141
63859931,1cyclictest0-21swapper/111:55:101
63859931,1cyclictest0-21swapper/111:43:281
63859931,1cyclictest0-21swapper/111:35:131
63859931,1cyclictest0-21swapper/111:15:101
63859931,1cyclictest0-21swapper/110:55:101
63859931,1cyclictest0-21swapper/110:30:141
63859931,1cyclictest0-21swapper/110:25:131
63859931,1cyclictest0-21swapper/110:20:091
63859931,1cyclictest0-21swapper/109:50:131
63859931,1cyclictest0-21swapper/109:25:111
63859931,1cyclictest0-21swapper/109:05:131
63859931,1cyclictest0-21swapper/108:55:111
63859931,1cyclictest0-21swapper/108:45:111
63859931,1cyclictest0-21swapper/108:44:591
63859931,1cyclictest0-21swapper/108:35:081
63859931,1cyclictest0-21swapper/108:30:111
63859931,1cyclictest0-21swapper/108:20:121
63859931,1cyclictest0-21swapper/108:15:111
63859931,1cyclictest0-21swapper/107:50:121
63859931,1cyclictest0-21swapper/107:45:111
63859931,1cyclictest0-21swapper/107:10:121
63859930,3cyclictest0-21swapper/109:01:141
63859930,3cyclictest0-21swapper/107:35:071
63859930,2cyclictest32495-21chrt11:01:071
63859930,2cyclictest15770-21chrt11:09:441
63859930,2cyclictest13557-21chrt10:03:381
63859930,2cyclictest0-21swapper/111:45:131
63859930,2cyclictest0-21swapper/108:50:121
63859930,2cyclictest0-21swapper/108:25:111
63859930,2cyclictest0-21swapper/108:00:121
63859930,2cyclictest0-21swapper/107:15:111
63859930,2cyclictest0-21swapper/107:15:101
63859930,1cyclictest0-21swapper/109:45:131
63799933,0cyclictest24370-21kworker/0:010:25:130
63799933,0cyclictest24370-21kworker/0:007:30:000
63799932,1cyclictest714-21snmpd09:05:040
63799932,1cyclictest714-21snmpd08:10:060
63799932,1cyclictest6532-21gpg-connect-age12:10:020
63799932,1cyclictest28150-21ssh09:55:020
63799931,2cyclictest0-21swapper/012:15:060
63799931,1cyclictest0-21swapper/008:05:120
63799930,3cyclictest9402-21perl07:15:020
63799930,3cyclictest9402-21perl07:15:020
63799930,3cyclictest29455-21cut12:05:140
63799930,3cyclictest24212-21grep12:35:150
63799930,3cyclictest21702-21rm11:45:060
63799930,3cyclictest0-21swapper/010:40:020
63799930,3cyclictest0-21swapper/008:25:000
63799930,3cyclictest0-21swapper/008:15:100
63799930,3cyclictest0-21swapper/007:55:010
63799930,3cyclictest0-21swapper/007:50:110
63799930,3cyclictest0-21swapper/007:25:100
63799930,3cyclictest0-21swapper/007:20:130
63859921,1cyclictest0-21swapper/110:50:011
63859921,0cyclictest0-21swapper/107:40:141
63859920,1cyclictest0-21swapper/110:45:141
63859920,1cyclictest0-21swapper/110:40:111
63859920,1cyclictest0-21swapper/109:30:121
63859920,1cyclictest0-21swapper/108:10:091
63859920,1cyclictest0-21swapper/107:55:101
63859920,1cyclictest0-21swapper/107:30:111
63859920,1cyclictest0-21swapper/107:25:101
63859910,1cyclictest0-21swapper/108:05:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional