You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 16:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Thu Nov 27, 2025 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4581997132,22cyclictest29566irq/58-eth010:30:123
4577996414,13cyclictest131rcu_preempt10:20:122
455999622,44cyclictest29673-21ssh09:35:100
4581996135,22cyclictest29566irq/58-eth010:20:133
456999573,41cyclictest14173-21ssh10:50:131
4559995746,4cyclictest23983-21diskmemload10:05:150
455999573,40cyclictest25990-21ssh12:20:130
455999552,40cyclictest17157-21ssh12:15:130
4559995424,21cyclictest12-21ksoftirqd/011:10:130
457799536,7cyclictest131rcu_preempt08:35:122
455999492,37cyclictest10556-21ssh11:30:130
458199473,31cyclictest29566irq/58-eth011:20:163
456999471,37cyclictest399-21ssh12:05:131
455999473,35cyclictest18550-21rm10:10:150
456999461,35cyclictest2823-21ssh11:25:151
455999453,34cyclictest18336-21ssh11:35:120
458199442,27cyclictest29566irq/58-eth008:45:163
457799434,29cyclictest261rcuc/212:35:112
458199428,24cyclictest29566irq/58-eth011:30:133
4559994217,18cyclictest20989-21ssh09:28:340
458199412,23cyclictest29566irq/58-eth010:50:133
458199412,23cyclictest29566irq/58-eth010:25:103
457799419,19cyclictest27-21ksoftirqd/210:30:122
4559994119,17cyclictest27345-1kworker/u9:210:30:120
458199402,25cyclictest29566irq/58-eth011:45:143
458199402,23cyclictest29566irq/58-eth010:10:143
458199402,21cyclictest29566irq/58-eth010:55:133
457799401,32cyclictest27613-21ssh11:20:142
457799401,25cyclictest13090-21ssh09:45:132
458199384,21cyclictest29566irq/58-eth009:20:133
4577993819,6cyclictest27-21ksoftirqd/211:30:142
456999382,21cyclictest13091-21ssh09:45:131
455999381,34cyclictest26238-21ssh10:15:140
457799372,18cyclictest6313-21ssh09:19:342
457799364,6cyclictest131rcu_preempt09:25:522
4559993610,20cyclictest12-21ksoftirqd/010:00:130
457799352,27cyclictest4922-21/usr/sbin/munin07:10:172
458199343,18cyclictest29566irq/58-eth011:55:133
457799349,4cyclictest131rcu_preempt11:05:112
457799333,14cyclictest26346irq/40-dwc2_hso08:25:132
4569993310,16cyclictest21-21ksoftirqd/108:10:121
4559993316,12cyclictest0-21swapper/009:45:130
458199322,19cyclictest29566irq/58-eth008:40:113
458199322,18cyclictest29566irq/58-eth011:05:123
456999321,23cyclictest9069-21ssh11:50:131
457799313,20cyclictest27-21ksoftirqd/210:35:132
458199303,16cyclictest29566irq/58-eth008:55:113
458199303,16cyclictest0-21swapper/309:35:103
458199301,17cyclictest29566irq/58-eth011:25:143
457799302,23cyclictest11290-21ssh12:30:132
457799302,17cyclictest26346irq/40-dwc2_hso10:05:142
457799301,19cyclictest26346irq/40-dwc2_hso11:10:122
456999301,21cyclictest2378-21sh12:25:111
456999301,16cyclictest26231-21ssh10:15:141
4559993018,8cyclictest0-21swapper/010:55:120
458199295,11cyclictest29566irq/58-eth009:45:133
458199293,15cyclictest29566irq/58-eth012:35:123
457799292,20cyclictest0-21swapper/208:45:172
4577992913,7cyclictest27-21ksoftirqd/209:50:132
458199282,11cyclictest29566irq/58-eth007:30:123
457799285,18cyclictest131rcu_preempt10:25:592
457799285,17cyclictest0-21swapper/209:35:342
457799283,11cyclictest0-21swapper/209:22:342
455999283,17cyclictest12-21ksoftirqd/009:55:120
455999283,17cyclictest12-21ksoftirqd/009:55:120
458199273,14cyclictest29666irq/59-eth010:35:123
457799272,9cyclictest0-21swapper/209:05:102
457799272,20cyclictest0-21swapper/211:44:342
457799272,20cyclictest0-21swapper/211:44:342
457799272,20cyclictest0-21swapper/209:34:342
457799272,15cyclictest26346irq/40-dwc2_hso11:45:142
457799272,10cyclictest0-21swapper/210:59:352
457799272,10cyclictest0-21swapper/207:20:142
457799271,19cyclictest26346irq/40-dwc2_hso07:40:132
457799271,17cyclictest26346irq/40-dwc2_hso11:55:142
456999273,21cyclictest8715-21ssh12:10:141
4569992710,11cyclictest201rcuc/107:15:131
458199269,14cyclictest0-21swapper/310:40:133
458199262,13cyclictest29566irq/58-eth009:40:143
457799262,17cyclictest26346irq/40-dwc2_hso11:35:132
457799262,17cyclictest26346irq/40-dwc2_hso11:00:142
457799262,16cyclictest26346irq/40-dwc2_hso08:20:112
457799261,17cyclictest26346irq/40-dwc2_hso07:35:122
456999268,11cyclictest0-21swapper/108:35:111
456999266,9cyclictest21-21ksoftirqd/112:39:591
456999265,5cyclictest21-21ksoftirqd/107:10:011
456999263,18cyclictest4065-21ssh11:05:131
4569992610,5cyclictest131rcu_preempt11:37:401
458199251,17cyclictest29566irq/58-eth011:35:123
457799251,21cyclictest27659-21ls08:40:162
457799251,19cyclictest22836-21ssh12:16:342
456999258,11cyclictest21-21ksoftirqd/107:35:101
456999254,8cyclictest21-21ksoftirqd/109:33:321
455999255,12cyclictest12-21ksoftirqd/008:15:130
455999252,18cyclictest26175-21rm11:40:130
455999252,18cyclictest26175-21rm11:40:120
4559992512,8cyclictest629-21Async14:00:480
458199243,14cyclictest0-21swapper/309:19:343
458199243,13cyclictest0-21swapper/307:10:163
458199242,3cyclictest0-21swapper/312:20:133
457799241,8cyclictest23983-21diskmemload10:45:162
457799241,17cyclictest26346irq/40-dwc2_hso07:25:112
457799241,10cyclictest93266snmpd12:13:392
456999248,9cyclictest21-21ksoftirqd/109:27:421
456999247,7cyclictest21-21ksoftirqd/111:33:541
456999246,9cyclictest21-21ksoftirqd/107:40:191
456999246,7cyclictest21-21ksoftirqd/112:23:261
456999245,9cyclictest21-21ksoftirqd/110:57:581
456999245,3cyclictest131rcu_preempt12:33:251
456999245,3cyclictest131rcu_preempt10:35:291
456999243,6cyclictest21-21ksoftirqd/110:25:171
456999243,5cyclictest21-21ksoftirqd/107:10:161
456999243,17cyclictest1-21systemd11:40:131
456999243,17cyclictest1-21systemd11:40:131
456999241,5cyclictest21-21ksoftirqd/110:35:021
4569992413,9cyclictest0-21swapper/111:13:161
455999246,7cyclictest12-21ksoftirqd/012:30:130
455999241,4cyclictest131rcu_preempt09:40:160
455999241,18cyclictest16377-21cut07:55:150
457799232,6cyclictest0-21swapper/209:56:342
457799232,6cyclictest0-21swapper/209:56:332
457799231,16cyclictest26346irq/40-dwc2_hso09:10:122
457799231,15cyclictest0-21swapper/208:55:132
457799231,14cyclictest0-21swapper/207:30:132
456999239,8cyclictest21-21ksoftirqd/109:44:021
456999238,4cyclictest21-21ksoftirqd/109:13:141
456999237,8cyclictest21-21ksoftirqd/110:00:321
456999236,9cyclictest21-21ksoftirqd/111:15:241
456999236,9cyclictest21-21ksoftirqd/111:15:231
456999235,6cyclictest21-21ksoftirqd/109:37:501
456999234,9cyclictest21-21ksoftirqd/109:51:501
456999233,8cyclictest21-21ksoftirqd/110:42:561
456999231,18cyclictest19504-21cut09:05:141
4581992214,5cyclictest29566irq/58-eth011:50:103
457799228,11cyclictest0-21swapper/211:50:212
457799222,6cyclictest0-21swapper/212:22:342
457799222,15cyclictest0-21swapper/212:09:342
457799221,5cyclictest0-21swapper/212:00:102
4577992211,8cyclictest12763-21cron09:45:022
4577992211,8cyclictest0-21swapper/211:25:162
4577992211,8cyclictest0-21swapper/207:55:142
457799221,11cyclictest7166-21ps07:45:192
456999225,9cyclictest21-21ksoftirqd/110:12:401
456999225,8cyclictest21-21ksoftirqd/109:24:321
456999225,12cyclictest21-21ksoftirqd/112:18:351
456999224,8cyclictest21-21ksoftirqd/108:15:121
456999222,2cyclictest131rcu_preempt11:45:271
4569992215,3cyclictest21-21ksoftirqd/107:45:131
4569992212,7cyclictest13824-21cron10:50:011
456999221,17cyclictest31298-21ssh11:20:281
456999221,16cyclictest29159-21find11:00:121
455999222,14cyclictest2397-21ssh12:25:130
455999222,13cyclictest842-21munin-node11:20:160
458199218,6cyclictest0-21swapper/311:05:023
458199211,11cyclictest29566irq/58-eth010:05:143
457799218,10cyclictest1214-21kthreadcore08:10:212
457799214,9cyclictest16940-21kthreadcore09:00:202
457799214,9cyclictest16940-21kthreadcore09:00:202
457799214,6cyclictest131rcu_preempt10:15:242
457799214,14cyclictest29255-21missed_timers08:05:212
457799213,10cyclictest9584-21/usr/sbin/munin07:15:152
457799211,17cyclictest2358-21kthreadcore10:40:212
457799211,16cyclictest21174-21fschecks_count08:00:142
457799211,16cyclictest17944-21grep08:30:172
457799211,11cyclictest20186-21kthreadcore10:10:212
457799211,10cyclictest7019-21tail08:15:282
457799211,10cyclictest18336-21grep10:50:292
457799211,10cyclictest10921-21latency_hist12:30:012
456999216,8cyclictest21-21ksoftirqd/108:30:251
456999214,8cyclictest21-21ksoftirqd/112:00:391
456999213,4cyclictest21-21ksoftirqd/109:00:141
456999213,4cyclictest21-21ksoftirqd/109:00:141
456999212,4cyclictest21-21ksoftirqd/108:40:241
456999211,4cyclictest21-21ksoftirqd/109:19:051
4569992113,5cyclictest201rcuc/110:20:191
456999211,17cyclictest9546-21find08:55:121
456999211,16cyclictest16723-21ssh11:55:131
456999211,13cyclictest19421-21find07:25:101
455999219,9cyclictest9366-21cron07:15:010
455999219,9cyclictest7555-21memory11:05:230
455999218,8cyclictest12-21ksoftirqd/008:25:120
455999217,3cyclictest131rcu_preempt11:45:250
455999215,7cyclictest4545-21cyclictest09:30:140
458199203,12cyclictest0-21swapper/309:59:343
458199203,12cyclictest0-21swapper/309:59:343
4581992012,5cyclictest29566irq/58-eth010:00:133
4581992011,6cyclictest0-21swapper/311:40:203
4581992011,6cyclictest0-21swapper/311:40:193
457799208,9cyclictest13759-21kthreadcore07:50:242
457799208,9cyclictest13759-21kthreadcore07:50:242
457799202,10cyclictest23152-21latency11:15:242
457799202,10cyclictest23152-21latency11:15:232
457799201,9cyclictest6951-21kthreadcore08:50:222
457799201,10cyclictest4647-21latency_hist07:10:012
456999207,6cyclictest32294-21apt-get08:45:171
456999207,4cyclictest21-21ksoftirqd/107:55:161
456999205,3cyclictest131rcu_preempt09:55:331
456999205,3cyclictest131rcu_preempt09:55:331
456999203,8cyclictest21-21ksoftirqd/110:07:321
4569992011,3cyclictest21-21ksoftirqd/107:20:141
4569992010,4cyclictest21-21ksoftirqd/107:50:181
4569992010,4cyclictest21-21ksoftirqd/107:50:171
455999206,7cyclictest131rcu_preempt07:15:130
455999204,10cyclictest31742-21ssh10:40:120
455999203,13cyclictest8871-21ssh10:25:110
455999201,16cyclictest14426-21apt-get09:00:110
455999201,16cyclictest14426-21apt-get09:00:110
455999201,10cyclictest19728-21cut11:55:220
4559992010,7cyclictest27345-1kworker/u9:210:35:190
455999200,15cyclictest10536-21ls10:45:290
458199199,6cyclictest0-21swapper/310:49:213
458199193,11cyclictest0-21swapper/312:16:263
458199193,11cyclictest0-21swapper/309:30:343
458199193,11cyclictest0-21swapper/309:25:493
4581991911,5cyclictest29566irq/58-eth012:10:273
4581991911,5cyclictest29566irq/58-eth010:15:143
458199191,13cyclictest0-21swapper/307:50:203
458199191,13cyclictest0-21swapper/307:50:203
4581991910,4cyclictest0-21swapper/312:05:463
457799192,9cyclictest4412-21kthreadcore10:00:182
456999191,11cyclictest21-21ksoftirqd/108:25:141
456999190,16cyclictest25889-21find08:05:111
455999199,7cyclictest2393-21cut12:05:190
455999193,11cyclictest19643-21acpi11:15:130
455999193,11cyclictest19643-21acpi11:15:130
455999192,11cyclictest0-21swapper/009:05:050
455999191,9cyclictest18078-21switchtime10:50:280
455999191,14cyclictest13860-21users11:50:300
455999191,13cyclictest6818-21tr07:45:180
455999190,12cyclictest32449-21cut08:45:190
458199189,6cyclictest0-21swapper/309:10:173
458199187,6cyclictest0-21swapper/312:26:533
458199186,8cyclictest29566irq/58-eth012:30:253
458199182,8cyclictest0-21swapper/309:00:113
458199182,8cyclictest0-21swapper/309:00:103
458199182,10cyclictest29566irq/58-eth007:40:163
458199182,10cyclictest0-21swapper/311:15:053
458199182,10cyclictest0-21swapper/311:15:053
458199182,10cyclictest0-21swapper/309:50:143
458199182,10cyclictest0-21swapper/308:25:133
458199182,10cyclictest0-21swapper/308:20:113
458199182,10cyclictest0-21swapper/308:15:143
4581991812,4cyclictest0-21swapper/312:04:113
458199181,13cyclictest0-21swapper/307:25:153
458199181,12cyclictest0-21swapper/308:30:213
458199181,11cyclictest0-21swapper/309:05:123
456999189,3cyclictest0-21swapper/108:20:261
456999187,9cyclictest28112-21grep07:30:281
456999187,3cyclictest131rcu_preempt08:50:131
455999189,7cyclictest7636-21sshd11:25:340
455999189,6cyclictest3385-21kthreadcore10:20:210
455999189,6cyclictest23983-21diskmemload09:20:230
455999185,3cyclictest131rcu_preempt08:55:010
4559991810,6cyclictest25668-21cron08:05:020
458199178,4cyclictest0-21swapper/311:10:323
458199172,10cyclictest0-21swapper/308:35:123
458199172,10cyclictest0-21swapper/308:05:133
458199172,10cyclictest0-21swapper/307:35:133
458199172,10cyclictest0-21swapper/307:20:143
458199171,12cyclictest0-21swapper/307:15:123
455999179,6cyclictest32623-21kthreadcore09:15:190
455999177,7cyclictest24269-21apt-get07:30:130
455999175,8cyclictest0-21swapper/012:35:120
455999173,11cyclictest24139-21kthreadcore09:50:220
455999172,12cyclictest30943-21ntp_states08:40:240
455999171,13cyclictest12560-21kthreadcore08:55:220
455999171,12cyclictest32596-21ssh12:05:010
455999171,12cyclictest24457-21diskstats09:10:140
455999171,12cyclictest17943-21wc08:30:160
455999171,11cyclictest0-21swapper/007:50:170
455999171,11cyclictest0-21swapper/007:50:160
4559991710,5cyclictest8924-21fschecks_time12:10:170
455999170,8cyclictest9985-21tr08:20:180
456999165,8cyclictest21-21ksoftirqd/108:00:111
455999165,8cyclictest29109-21apt-get07:35:120
455999163,10cyclictest23986-21kthreadcore08:35:180
455999162,9cyclictest0-21swapper/007:20:130
455999162,11cyclictest27097-21kthreadcore08:05:190
4559991611,3cyclictest3440-21kthreadcore07:40:220
455999161,11cyclictest31267-21irqrtprio08:10:170
458199151,12cyclictest0-21swapper/308:55:023
455999152,9cyclictest0-21swapper/007:25:120
458199139,2cyclictest33-21ksoftirqd/308:10:213
458199133,8cyclictest0-21swapper/308:00:143
458199132,8cyclictest0-21swapper/307:45:123
455999131,9cyclictest4687-21latency_hist07:10:020
458199121,8cyclictest0-21swapper/308:00:013
458199101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional