You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-11 - 17:55
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7.osadl.org (updated Wed Sep 11, 2024 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1236521040,6chrt12348-21gpgv08:33:240
1448499972,62cyclictest20249-21taskset07:18:491
14505999546,12cyclictest141rcu_preempt10:33:243
14505999546,12cyclictest141rcu_preempt10:33:243
289072930,5sleep10-21swapper/110:13:531
1449799923,22cyclictest31827-21apt-get07:53:362
1449799923,22cyclictest31827-21apt-get07:53:362
1433529111,44sleep214370-21munin-run07:03:222
14497998540,23cyclictest33-21ksoftirqd/208:13:472
14497998442,20cyclictest33-21ksoftirqd/207:13:382
14484998446,31cyclictest355-21systemd-logind07:43:231
14484998443,33cyclictest0-21swapper/108:13:221
14497998259,17cyclictest26629-21unixbench_singl10:08:572
14497998259,17cyclictest26629-21unixbench_singl10:08:572
14497998250,13cyclictest33-21ksoftirqd/209:35:032
14497998241,19cyclictest33-21ksoftirqd/208:08:502
14480998234,20cyclictest141rcu_preempt11:39:180
14497998052,9cyclictest33-21ksoftirqd/211:42:352
14484998062,10cyclictest2366-21munin-plugin-st08:03:211
14484998032,36cyclictest0-21swapper/109:08:441
14480998051,19cyclictest22942-21idleruntime-cro11:03:210
134622809,49sleep013499-21grep06:58:570
1316927911,36sleep313218-21fschecks_count06:58:483
14484997865,7cyclictest0-21swapper/107:53:361
14484997865,7cyclictest0-21swapper/107:53:361
14480997861,10cyclictest20316-21latency_hist10:58:230
14480997846,9cyclictest141rcu_preempt10:03:560
14480997846,9cyclictest141rcu_preempt10:03:560
14497997749,7cyclictest33-21ksoftirqd/211:56:112
14497997736,26cyclictest33-21ksoftirqd/208:53:552
1448499774,42cyclictest23471-21apt-get11:03:361
1448499771,47cyclictest10920-21ssh11:38:381
14480997754,13cyclictest10570-21latency_hist11:38:210
14480997742,8cyclictest141rcu_preempt09:53:150
14480997739,9cyclictest141rcu_preempt07:03:560
14480997739,9cyclictest141rcu_preempt07:03:560
1448499762,48cyclictest4022-21ssh12:21:071
14480997651,16cyclictest31564-21latency_hist11:18:230
14497997544,12cyclictest141rcu_preempt10:59:492
14497997532,23cyclictest33-21ksoftirqd/209:23:232
1448499753,41cyclictest2781-21diskstats08:03:401
1448499752,45cyclictest15245-21sshd11:44:581
1448499752,44cyclictest22345-21diskmemload10:56:451
14480997557,10cyclictest31730-21unixbench_multi09:18:590
14480997531,19cyclictest141rcu_preempt08:39:000
141912757,46sleep10-21swapper/107:01:301
14505997446,20cyclictest18082-21grep10:54:013
14497997442,24cyclictest33-21ksoftirqd/208:58:442
14497997420,23cyclictest0-21swapper/210:55:412
14484997451,16cyclictest8860-21munin-plugin-st08:23:221
14484997451,16cyclictest8860-21munin-plugin-st08:23:221
1448499744,59cyclictest14347-21apt-get09:48:341
1448499744,53cyclictest0-21swapper/110:43:531
1448499743,57cyclictest10808-21latency_hist12:33:241
14484997425,40cyclictest0-21swapper/111:23:451
1448499742,44cyclictest22345-21diskmemload12:12:551
1448499742,44cyclictest22345-21diskmemload12:12:551
1448499742,43cyclictest22345-21diskmemload11:20:381
1450599732,46cyclictest14900-21apt-get10:48:523
14497997352,8cyclictest33-21ksoftirqd/211:08:402
14497997352,8cyclictest33-21ksoftirqd/211:08:402
1448499732,41cyclictest29307-21users07:43:571
14480997327,25cyclictest141rcu_preempt11:30:290
14497997257,10cyclictest0-21swapper/209:38:432
14497997257,10cyclictest0-21swapper/209:38:432
14497997253,9cyclictest0-21swapper/211:13:442
14497997227,22cyclictest33-21ksoftirqd/210:14:022
14484997236,30cyclictest0-21swapper/111:58:241
14484997233,32cyclictest22972-21apt-config11:03:221
14480997243,16cyclictest141rcu_preempt09:28:450
14480997243,16cyclictest141rcu_preempt09:28:450
14480997234,15cyclictest141rcu_preempt12:08:440
14480997234,15cyclictest141rcu_preempt12:08:440
14497997155,10cyclictest17455-21munin-plugin-st08:48:222
14497997155,10cyclictest17455-21munin-plugin-st08:48:222
14497997139,17cyclictest33-21ksoftirqd/208:29:302
14497997130,19cyclictest33-21ksoftirqd/210:50:512
14497997129,23cyclictest141rcu_preempt10:26:022
14484997128,35cyclictest0-21swapper/111:28:411
1448499712,39cyclictest22800-21apt-get10:03:331
1448499712,39cyclictest22800-21apt-get10:03:331
14484997117,47cyclictest22573-21ssh11:58:361
14480997126,18cyclictest141rcu_preempt08:13:410
14505997038,15cyclictest141rcu_preempt12:13:423
14505997029,18cyclictest141rcu_preempt07:43:453
14497997045,16cyclictest31443-21apt-key07:53:242
14497997024,24cyclictest33-21ksoftirqd/209:56:532
14497997024,24cyclictest33-21ksoftirqd/209:56:532
14497997024,19cyclictest141rcu_preempt09:16:512
14497997010,28cyclictest141rcu_preempt12:13:242
14497997010,28cyclictest141rcu_preempt12:13:242
14484997048,12cyclictest12090-21sendmail08:33:221
14484997043,18cyclictest3544-21latency_hist10:28:231
14480997035,9cyclictest22345-21diskmemload10:23:460
14497996928,14cyclictest183-21pr/ttyS012:25:132
14497996926,19cyclictest363-21in:imklog10:44:132
14497996925,21cyclictest141rcu_preempt07:43:492
14484996940,20cyclictest22388-21latency_hist09:03:231
1448499692,41cyclictest24962-21sh09:08:011
14480996940,19cyclictest19789-21cut11:53:430
14480996935,8cyclictest141rcu_preempt07:13:240
1448099692,34cyclictest11966-21apt-get10:43:510
14480996916,19cyclictest141rcu_preempt08:53:540
287692680,4sleep00-21swapper/010:13:470
14505996846,13cyclictest16801-21kworker/u8:210:38:423
14505996812,34cyclictest41-21ksoftirqd/307:03:433
14505996812,34cyclictest41-21ksoftirqd/307:03:433
14497996840,7cyclictest141rcu_preempt12:29:322
14497996836,19cyclictest33-21ksoftirqd/211:38:012
14497996828,9cyclictest33-21ksoftirqd/210:03:412
14497996828,9cyclictest33-21ksoftirqd/210:03:412
14497996825,24cyclictest0-21swapper/211:06:552
14497996816,25cyclictest33-21ksoftirqd/211:25:502
1448499683,25cyclictest12611-21apt-get08:33:351
1448499682,54cyclictest22218-21/usr/sbin/munin07:23:471
14480996844,14cyclictest14341-21latency_hist08:38:250
14480996820,23cyclictest141rcu_preempt09:14:170
14497996733,19cyclictest141rcu_preempt10:20:362
14497996733,19cyclictest141rcu_preempt10:20:362
14497996731,16cyclictest141rcu_preempt07:43:242
1449799672,23cyclictest141rcu_preempt10:28:422
1449799672,23cyclictest141rcu_preempt10:28:422
1448499674,53cyclictest10525-21latency_hist08:28:231
1448499673,53cyclictest31993-21/usr/sbin/munin12:13:421
14480996733,15cyclictest363-21in:imklog10:33:370
14480996730,22cyclictest13830-21sshd09:47:450
14505996647,12cyclictest26290-21sort11:08:393
14505996647,12cyclictest26290-21sort11:08:393
14505996643,14cyclictest23261-21latency_hist07:28:243
14505996640,9cyclictest141rcu_preempt09:38:243
14505996638,8cyclictest141rcu_preempt10:45:053
14505996635,9cyclictest141rcu_preempt11:38:573
14505996630,8cyclictest141rcu_preempt09:43:483
14497996638,8cyclictest141rcu_preempt12:23:212
14497996629,16cyclictest183-21pr/ttyS011:58:522
14497996620,19cyclictest141rcu_preempt11:18:392
14497996618,24cyclictest0-21swapper/210:38:552
1448499664,51cyclictest8549-21modprobe09:38:211
14484996642,12cyclictest0-21swapper/108:48:431
14484996627,30cyclictest0-21swapper/108:39:111
1448499662,55cyclictest24215-21/usr/sbin/munin07:28:541
14484996624,16cyclictest0-21swapper/107:13:551
1448099667,46cyclictest0-21swapper/010:48:490
14480996621,23cyclictest141rcu_preempt11:25:160
14480996618,25cyclictest141rcu_preempt10:02:550
14505996548,11cyclictest12100-21idleruntime-cro08:33:223
14505996545,7cyclictest141rcu_preempt11:01:523
14505996540,7cyclictest141rcu_preempt12:03:453
14505996531,7cyclictest141rcu_preempt12:02:503
14505996531,17cyclictest141rcu_preempt09:08:523
14497996542,10cyclictest141rcu_preempt08:18:482
14497996542,10cyclictest141rcu_preempt08:18:482
14497996533,15cyclictest141rcu_preempt11:46:332
14497996526,17cyclictest0-21swapper/212:16:322
14484996549,10cyclictest14471-21cyclictest09:23:381
14484996511,47cyclictest22470-21cron10:03:211
14480996533,18cyclictest141rcu_preempt07:33:500
14480996525,17cyclictest141rcu_preempt12:15:570
14505996450,9cyclictest29265-21unixbench_singl10:13:563
14505996449,10cyclictest14471-21cyclictest09:18:553
14505996448,8cyclictest0-21swapper/311:53:213
14505996446,9cyclictest4834-21idleruntime-cro12:23:213
14505996437,13cyclictest0-21swapper/308:03:423
1450599642,17cyclictest41-21ksoftirqd/307:13:213
14497996430,9cyclictest0-21swapper/211:51:502
14497996425,7cyclictest141rcu_preempt08:06:342
14484996448,10cyclictest31311-21idleruntime-cro07:53:221
1448499644,49cyclictest918-21latency_hist10:23:241
1448499644,49cyclictest918-21latency_hist10:23:241
1448499643,52cyclictest16890-21grep11:48:411
1448499643,51cyclictest19801-21/usr/sbin/munin08:53:461
14484996427,28cyclictest0-21swapper/109:43:511
1448499642,55cyclictest18082-21sed09:53:551
1448499642,55cyclictest18082-21sed09:53:551
1448499642,53cyclictest11272-21latency_hist09:43:241
1448499642,53cyclictest11272-21latency_hist09:43:241
1448499642,50cyclictest3354-21ssh09:28:321
1448499642,50cyclictest3354-21ssh09:28:321
14480996444,7cyclictest141rcu_preempt09:23:510
14480996442,13cyclictest17022-21unixbench_multi08:43:550
14480996442,13cyclictest17022-21unixbench_multi08:43:550
14480996429,12cyclictest141rcu_preempt09:08:570
1448099642,46cyclictest23528-21ssh11:03:410
14480996424,12cyclictest141rcu_preempt07:48:410
14505996339,8cyclictest13774-21/usr/sbin/munin11:43:553
14505996327,8cyclictest141rcu_preempt07:18:253
1449799638,26cyclictest141rcu_preempt07:58:552
1448499638,34cyclictest14900-21apt-get10:48:361
1448499634,49cyclictest0-21swapper/111:08:331
1448499634,49cyclictest0-21swapper/111:08:331
1448499633,50cyclictest11815-21cat10:43:251
1448499633,47cyclictest25592-21apt-get10:08:371
1448499633,47cyclictest25592-21apt-get10:08:371
14484996327,26cyclictest0-21swapper/107:03:431
14484996327,26cyclictest0-21swapper/107:03:431
1448499632,54cyclictest25691-21ssh12:03:351
1448499632,53cyclictest337-21dbus-daemon12:28:211
14484996319,35cyclictest0-21swapper/109:21:511
14480996337,8cyclictest141rcu_preempt10:18:440
14480996337,8cyclictest141rcu_preempt10:18:440
14480996333,8cyclictest141rcu_preempt12:23:540
14480996329,15cyclictest141rcu_preempt12:32:300
14505996245,10cyclictest31302-21latency_hist07:53:253
14505996239,8cyclictest141rcu_preempt07:28:543
14505996228,7cyclictest141rcu_preempt07:58:243
14505996228,7cyclictest141rcu_preempt07:58:243
14505996228,19cyclictest141rcu_preempt09:48:393
14505996226,14cyclictest141rcu_preempt10:08:183
14505996226,14cyclictest141rcu_preempt10:08:183
14497996229,21cyclictest0-21swapper/208:34:012
14497996227,21cyclictest141rcu_preempt12:06:212
14497996220,22cyclictest5422-21fschecks_time11:28:422
1448499623,48cyclictest28886-21/usr/sbin/munin11:13:481
1448499623,46cyclictest3708-21apt-get10:28:331
1448499623,46cyclictest3708-21apt-get10:28:331
1448499622,51cyclictest25110-21cpu07:33:371
1448499622,49cyclictest6182-21/usr/sbin/munin08:13:421
14484996223,28cyclictest0-21swapper/111:33:511
14480996238,8cyclictest141rcu_preempt09:53:520
14480996238,8cyclictest141rcu_preempt09:53:520
14480996237,8cyclictest141rcu_preempt11:48:230
14480996237,18cyclictest5612-21idleruntime-cro08:13:220
14480996231,8cyclictest141rcu_preempt07:23:580
14480996218,23cyclictest141rcu_preempt07:19:000
14505996143,9cyclictest0-21swapper/308:13:223
1450599614,20cyclictest3075-21find11:23:563
14505996139,8cyclictest141rcu_preempt12:28:533
14505996139,7cyclictest141rcu_preempt10:11:033
14505996139,7cyclictest141rcu_preempt10:11:033
14505996133,8cyclictest141rcu_preempt12:23:413
14505996130,9cyclictest141rcu_preempt08:23:243
14505996130,9cyclictest141rcu_preempt08:23:243
14497996138,7cyclictest141rcu_preempt08:23:542
14497996126,9cyclictest141rcu_preempt09:53:062
14497996112,17cyclictest141rcu_preempt09:08:572
1448499613,49cyclictest17479-21/usr/sbin/munin07:08:561
14480996140,13cyclictest26920-21ssh10:09:470
14480996140,13cyclictest26920-21ssh10:09:470
14480996138,11cyclictest0-21swapper/011:58:570
14480996131,13cyclictest141rcu_preempt10:30:530
14480996131,13cyclictest141rcu_preempt10:30:530
14480996129,9cyclictest141rcu_preempt09:03:330
1450599604,28cyclictest12611-21apt-get08:33:513
14505996036,9cyclictest141rcu_preempt08:02:103
14505996031,13cyclictest141rcu_preempt09:31:143
14505996031,13cyclictest141rcu_preempt09:31:143
14505996030,10cyclictest141rcu_preempt09:03:243
14505996022,8cyclictest141rcu_preempt09:38:433
14505996022,8cyclictest141rcu_preempt09:38:433
14497996031,9cyclictest33-21ksoftirqd/209:03:282
14497996026,7cyclictest141rcu_preempt10:01:242
14497996024,15cyclictest141rcu_preempt07:08:562
1448499604,48cyclictest16426-21if_eth008:43:431
1448499604,48cyclictest16426-21if_eth008:43:431
1448499603,48cyclictest28698-21/usr/sbin/munin09:13:481
14484996022,31cyclictest400-21runrttasks10:35:391
14480996041,12cyclictest2039-21journal-offline07:59:270
14480996036,13cyclictest0-21swapper/007:33:240
14480996031,13cyclictest141rcu_preempt11:08:310
14480996031,13cyclictest141rcu_preempt11:08:310
1450599596,32cyclictest0-21swapper/311:03:483
14505995939,7cyclictest141rcu_preempt09:53:523
14505995939,7cyclictest141rcu_preempt09:53:523
14505995938,12cyclictest26442-21modprobe07:38:213
14505995935,12cyclictest0-21swapper/308:53:423
14505995929,7cyclictest141rcu_preempt09:03:383
14497995939,8cyclictest33-21ksoftirqd/208:39:102
14497995925,13cyclictest33-21ksoftirqd/210:38:062
14497995923,13cyclictest33-21ksoftirqd/209:31:232
14497995923,13cyclictest33-21ksoftirqd/209:31:232
14497995917,9cyclictest141rcu_preempt07:28:422
14497995916,9cyclictest141rcu_preempt08:48:422
14480995934,7cyclictest141rcu_preempt11:19:400
14480995933,7cyclictest141rcu_preempt09:33:520
14480995932,8cyclictest141rcu_preempt07:17:470
14480995930,7cyclictest141rcu_preempt12:20:150
14480995927,14cyclictest120-21pr/ttymxc012:08:160
14480995927,11cyclictest0-21swapper/009:38:540
14480995927,11cyclictest0-21swapper/009:38:540
14480995920,8cyclictest141rcu_preempt08:07:220
1450599588,11cyclictest41-21ksoftirqd/311:14:513
14505995837,8cyclictest141rcu_preempt09:13:463
14505995836,8cyclictest141rcu_preempt11:38:023
14505995824,10cyclictest141rcu_preempt08:23:403
14505995817,19cyclictest41-21ksoftirqd/308:13:403
1448099589,12cyclictest141rcu_preempt10:43:250
1450599578,17cyclictest41-21ksoftirqd/310:33:563
14505995742,8cyclictest18065-21kworker/u8:308:53:213
14505995733,7cyclictest141rcu_preempt11:53:553
14505995732,8cyclictest141rcu_preempt11:28:313
1450599573,21cyclictest2633-21ssh09:27:073
14497995733,7cyclictest33-21ksoftirqd/209:47:372
14497995732,17cyclictest20505-21fschecks_time07:19:002
14497995726,14cyclictest141rcu_preempt07:33:522
14497995722,10cyclictest141rcu_preempt09:23:382
14480995724,17cyclictest141rcu_preempt08:23:570
1450599565,20cyclictest41-21ksoftirqd/308:43:243
14505995634,7cyclictest141rcu_preempt07:23:243
14505995632,8cyclictest141rcu_preempt11:23:213
14480995631,16cyclictest17726-21unixbench_singl11:48:560
14480995629,8cyclictest141rcu_preempt08:18:540
14480995629,8cyclictest141rcu_preempt08:18:540
1448099562,15cyclictest400-21runrttasks07:55:360
1448099562,15cyclictest400-21runrttasks07:55:360
1450599558,26cyclictest41-21ksoftirqd/310:18:383
1450599558,26cyclictest41-21ksoftirqd/310:18:383
14505995532,9cyclictest141rcu_preempt09:58:253
14505995526,9cyclictest141rcu_preempt10:23:373
14480995511,23cyclictest141rcu_preempt07:43:430
1450599534,10cyclictest41-21ksoftirqd/312:08:443
1450599534,10cyclictest41-21ksoftirqd/312:08:443
14505995315,12cyclictest41-21ksoftirqd/307:38:353
14497995328,13cyclictest33-21ksoftirqd/207:03:382
14497995328,13cyclictest33-21ksoftirqd/207:03:382
14497995323,13cyclictest141rcu_preempt07:23:472
14480995320,17cyclictest141rcu_preempt08:58:490
14480995219,8cyclictest141rcu_preempt07:43:240
1450599512,21cyclictest17453-21munin-run08:48:223
1450599512,21cyclictest17453-21munin-run08:48:223
14480995010,16cyclictest141rcu_preempt08:48:390
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional