You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 03:46
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7.osadl.org (updated Fri Jul 26, 2024 12:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1766212918,78sleep11782-21modprobe07:06:321
172421006,14sleep00-21swapper/007:05:580
192999943,62cyclictest400-21runrttasks09:41:593
1912998759,18cyclictest24954-21sendmail08:16:321
1912998759,18cyclictest24954-21sendmail08:16:321
110028722,30sleep30-21swapper/307:02:403
1929998333,20cyclictest363-21in:imklog09:22:513
191999834,55cyclictest31467-21apt-get11:36:462
1912998351,24cyclictest25-21ksoftirqd/108:31:351
1912998351,24cyclictest25-21ksoftirqd/108:31:351
1903998237,8cyclictest141rcu_preempt11:43:190
1903998237,8cyclictest141rcu_preempt11:43:190
1929998135,13cyclictest141rcu_preempt10:34:423
191299817,26cyclictest29858-21ssh09:41:511
1903998147,8cyclictest141rcu_preempt10:45:550
1903998033,27cyclictest141rcu_preempt07:54:250
1903998033,26cyclictest141rcu_preempt10:21:510
1929997861,8cyclictest0-21swapper/311:06:313
1919997826,21cyclictest21955-21sh11:21:262
1912997834,23cyclictest141rcu_preempt10:08:351
1929997740,17cyclictest141rcu_preempt08:52:113
1929997734,8cyclictest141rcu_preempt09:34:393
1929997734,8cyclictest141rcu_preempt09:34:393
1919997717,28cyclictest8141-21/usr/sbin/munin11:51:512
1912997734,22cyclictest25-21ksoftirqd/109:27:421
1929997644,10cyclictest141rcu_preempt08:01:463
1929997644,10cyclictest141rcu_preempt08:01:463
1929997642,7cyclictest141rcu_preempt09:41:033
1929997642,7cyclictest141rcu_preempt09:41:033
1929997641,12cyclictest141rcu_preempt12:35:173
1929997641,12cyclictest141rcu_preempt12:35:173
1929997628,10cyclictest141rcu_preempt10:52:063
1919997629,17cyclictest4267-21/usr/sbin/munin10:51:342
1912997642,11cyclictest25-21ksoftirqd/110:11:541
1919997559,8cyclictest0-21swapper/208:37:052
1919997550,16cyclictest18937-21unixbench_multi09:22:082
1919997530,17cyclictest24276-21sh12:18:052
1919997521,25cyclictest1352-21latency_hist10:46:352
1912997556,11cyclictest31512-21fschecks_time10:41:531
1912997545,23cyclictest25-21ksoftirqd/111:23:551
1912997541,8cyclictest25-21ksoftirqd/108:46:521
1903997534,12cyclictest141rcu_preempt09:42:220
1903997529,12cyclictest141rcu_preempt12:15:190
1929997457,10cyclictest31418-21sendmail08:36:323
1929997436,8cyclictest141rcu_preempt12:23:073
1929997432,14cyclictest141rcu_preempt09:12:063
1919997419,24cyclictest9992-21awk09:07:052
1912997447,21cyclictest25-21ksoftirqd/112:21:321
1903997459,9cyclictest5703-21munin-run08:56:320
1903997436,11cyclictest141rcu_preempt10:35:300
1903997435,22cyclictest141rcu_preempt07:24:100
1903997424,27cyclictest363-21in:imklog09:06:350
1929997343,9cyclictest141rcu_preempt10:40:113
1929997343,9cyclictest141rcu_preempt10:40:113
1929997336,12cyclictest141rcu_preempt10:11:043
1929997335,13cyclictest141rcu_preempt12:04:523
1919997320,20cyclictest8950-21diskmemload12:11:502
1919997319,23cyclictest26766-21ssh10:32:212
1903997334,13cyclictest141rcu_preempt12:10:170
1912997238,24cyclictest25-21ksoftirqd/110:31:351
191299723,39cyclictest31719-21/usr/sbin/munin12:32:101
191299723,39cyclictest31719-21/usr/sbin/munin12:32:101
1912997210,27cyclictest0-21swapper/111:37:001
1903997234,17cyclictest141rcu_preempt11:00:140
1903997234,17cyclictest141rcu_preempt11:00:140
1929997148,15cyclictest3720-21ssh10:49:513
1929997146,15cyclictest14396-21grep11:07:073
1929997146,15cyclictest14396-21grep11:07:073
1929997145,7cyclictest141rcu_preempt11:39:493
1929997142,7cyclictest141rcu_preempt10:26:083
1929997141,14cyclictest141rcu_preempt08:16:543
1929997137,8cyclictest141rcu_preempt11:13:033
1929997133,19cyclictest363-21in:imklog10:42:273
1929997123,14cyclictest141rcu_preempt08:23:193
1919997153,11cyclictest1315-21unixbench_singl09:47:092
1919997147,16cyclictest16843-21kworker/u8:210:16:332
1919997124,17cyclictest8638-21sh10:57:132
1919997124,17cyclictest8638-21sh10:57:132
1919997118,24cyclictest25802-21proc_pri08:17:042
1903997139,12cyclictest141rcu_preempt11:20:180
1903997136,16cyclictest141rcu_preempt07:12:100
1903997124,12cyclictest0-21swapper/009:56:350
1929997044,6cyclictest141rcu_preempt10:11:533
1929997033,14cyclictest141rcu_preempt08:49:523
1919997052,11cyclictest29805-21kworker/u8:108:31:332
1919997052,11cyclictest29805-21kworker/u8:108:31:332
1919997049,14cyclictest10369-21munin-plugin-st07:31:322
1912997031,13cyclictest25-21ksoftirqd/108:21:561
1903997042,8cyclictest141rcu_preempt11:56:320
1903997042,10cyclictest141rcu_preempt12:27:520
1903997032,10cyclictest141rcu_preempt11:26:590
1929996941,8cyclictest141rcu_preempt11:23:143
1929996939,10cyclictest183-21pr/ttyS011:33:053
1929996936,7cyclictest141rcu_preempt07:53:273
1929996934,11cyclictest141rcu_preempt07:31:583
1929996927,16cyclictest141rcu_preempt09:27:203
1929996915,18cyclictest41-21ksoftirqd/311:46:533
1919996923,16cyclictest24499-21diskmemload10:29:092
1919996922,19cyclictest20005-21taskset08:00:352
1919996919,21cyclictest8950-21diskmemload11:43:492
1919996919,21cyclictest8950-21diskmemload11:43:492
1912996938,8cyclictest25-21ksoftirqd/108:36:511
1912996934,9cyclictest25-21ksoftirqd/107:22:061
1903996944,8cyclictest141rcu_preempt10:52:040
1903996940,14cyclictest141rcu_preempt08:06:390
1903996939,15cyclictest141rcu_preempt11:47:030
1903996934,17cyclictest141rcu_preempt09:27:400
1903996934,11cyclictest141rcu_preempt10:11:330
1903996927,19cyclictest120-21pr/ttymxc007:46:560
1929996842,7cyclictest141rcu_preempt09:08:133
1929996835,8cyclictest141rcu_preempt12:31:183
1929996831,7cyclictest141rcu_preempt12:00:353
1929996831,11cyclictest141rcu_preempt11:55:143
1929996829,20cyclictest52-21kcompactd011:42:303
1929996829,20cyclictest52-21kcompactd011:42:303
1919996835,23cyclictest4335-21apt-get08:52:032
1919996833,26cyclictest17102-21apt-get07:51:432
1919996828,11cyclictest12808-21ntp_states07:37:052
1919996828,11cyclictest12808-21ntp_states07:37:052
1919996825,18cyclictest0-21swapper/207:32:112
1919996823,18cyclictest0-21swapper/211:15:052
1919996822,17cyclictest25351-21chrt09:33:162
1919996822,17cyclictest25351-21chrt09:33:162
1919996818,20cyclictest30296-21chrt11:34:512
1919996817,19cyclictest22518-21diskstats11:21:522
1912996846,8cyclictest25-21ksoftirqd/109:48:231
1912996845,12cyclictest0-21swapper/109:21:321
1912996836,9cyclictest25-21ksoftirqd/111:41:531
1912996836,9cyclictest25-21ksoftirqd/111:41:531
1912996836,23cyclictest25-21ksoftirqd/107:46:351
1903996838,12cyclictest141rcu_preempt08:15:490
1903996838,12cyclictest141rcu_preempt08:15:490
1903996835,7cyclictest141rcu_preempt09:49:270
1903996833,12cyclictest141rcu_preempt10:31:340
1903996832,22cyclictest141rcu_preempt07:26:540
1903996828,11cyclictest141rcu_preempt10:41:240
1903996828,11cyclictest141rcu_preempt10:41:240
1929996741,7cyclictest141rcu_preempt09:20:383
1929996741,13cyclictest141rcu_preempt08:08:533
1929996740,7cyclictest141rcu_preempt10:19:003
1929996733,7cyclictest141rcu_preempt09:56:093
1929996727,10cyclictest141rcu_preempt07:43:583
1919996717,19cyclictest6126-21df08:56:522
1919996717,19cyclictest6126-21df08:56:522
1919996717,17cyclictest14660-21sendmail-mta10:22:562
1912996744,15cyclictest18473-21sendmail07:56:331
1912996730,12cyclictest25-21ksoftirqd/108:11:341
1903996736,10cyclictest141rcu_preempt07:40:030
1903996736,10cyclictest141rcu_preempt07:40:030
1903996733,17cyclictest141rcu_preempt08:36:500
1903996731,12cyclictest141rcu_preempt12:04:300
1929996639,8cyclictest141rcu_preempt12:11:343
1929996636,7cyclictest141rcu_preempt11:19:043
1929996635,8cyclictest141rcu_preempt12:11:563
1929996628,22cyclictest363-21in:imklog09:06:093
1929996627,12cyclictest141rcu_preempt08:56:523
1929996627,12cyclictest141rcu_preempt08:56:523
1919996649,10cyclictest29524-21unixbench_multi10:37:062
1919996649,10cyclictest29524-21unixbench_multi10:37:062
1919996634,25cyclictest16749-21ssh09:18:462
1919996622,17cyclictest8950-21diskmemload09:40:562
1919996622,17cyclictest8950-21diskmemload09:40:562
1919996622,16cyclictest400-21runrttasks11:09:082
1919996622,16cyclictest400-21runrttasks11:09:082
1919996619,20cyclictest4459-21sh09:53:022
1919996619,18cyclictest31189-21chrt09:42:592
1912996643,15cyclictest12538-21systemd-journal07:40:151
1912996643,15cyclictest12538-21systemd-journal07:40:151
1912996633,12cyclictest25-21ksoftirqd/110:58:081
1912996633,12cyclictest25-21ksoftirqd/110:58:081
1912996632,9cyclictest25-21ksoftirqd/109:21:561
1912996632,12cyclictest25-21ksoftirqd/109:11:521
1912996627,12cyclictest25-21ksoftirqd/111:07:081
1912996627,12cyclictest25-21ksoftirqd/111:07:081
1903996643,8cyclictest141rcu_preempt11:37:230
1903996634,9cyclictest0-21swapper/008:06:350
1903996634,9cyclictest0-21swapper/008:06:350
1903996633,12cyclictest141rcu_preempt08:41:540
1903996625,7cyclictest141rcu_preempt09:22:490
4462659,30sleep21969-21kworker/u8:307:01:542
1929996537,8cyclictest141rcu_preempt08:11:573
1929996537,8cyclictest141rcu_preempt08:11:573
1929996531,8cyclictest141rcu_preempt11:01:203
1929996531,8cyclictest141rcu_preempt11:01:203
1929996531,13cyclictest141rcu_preempt07:12:053
1929996523,13cyclictest363-21in:imklog09:57:063
1919996545,13cyclictest362-21in:imuxsock10:11:322
1919996518,19cyclictest8950-21diskmemload12:21:472
1919996517,19cyclictest9421-21ssh10:02:032
1919996517,19cyclictest9421-21ssh10:02:032
1912996538,9cyclictest5491-21kworker/u8:107:21:341
1912996533,21cyclictest25-21ksoftirqd/107:27:041
1912996531,13cyclictest25-21ksoftirqd/111:11:531
1912996531,10cyclictest25-21ksoftirqd/109:56:571
1912996526,13cyclictest25-21ksoftirqd/110:36:561
1912996526,13cyclictest25-21ksoftirqd/110:36:561
1912996511,36cyclictest4335-21apt-get08:51:511
1903996539,9cyclictest141rcu_preempt11:06:350
1903996537,15cyclictest141rcu_preempt10:50:480
1903996535,25cyclictest183-21pr/ttyS012:18:260
1903996532,13cyclictest52-21kcompactd009:20:030
1903996531,7cyclictest141rcu_preempt10:06:300
1903996531,7cyclictest141rcu_preempt10:06:300
1903996531,21cyclictest141rcu_preempt11:15:070
1903996523,7cyclictest141rcu_preempt11:25:110
1903996523,19cyclictest183-21pr/ttyS010:12:070
1929996447,12cyclictest52-21kcompactd007:46:563
1929996432,7cyclictest141rcu_preempt11:31:103
1929996432,7cyclictest141rcu_preempt08:26:413
1929996432,7cyclictest141rcu_preempt08:26:413
1929996422,22cyclictest363-21in:imklog12:17:293
1919996442,10cyclictest10437-21apt-get11:01:452
1919996441,15cyclictest0-21swapper/207:11:322
1919996428,26cyclictest28320-21latency_hist08:26:352
1919996424,10cyclictest17579-21ps12:07:032
1919996420,16cyclictest5373-21sh11:46:562
1919996417,17cyclictest5751-21ssh10:52:222
1919996416,21cyclictest8950-21diskmemload11:28:392
1919996416,19cyclictest5943-21apt-get09:56:422
1919996415,19cyclictest24032-21memory08:11:592
1919996415,19cyclictest24032-21memory08:11:592
1912996445,11cyclictest28348-21cpuspeed11:31:471
1912996438,15cyclictest0-21swapper/112:07:061
1912996432,10cyclictest25-21ksoftirqd/110:51:521
190399646,22cyclictest363-21in:imklog09:07:020
1903996432,13cyclictest141rcu_preempt09:57:470
1903996428,10cyclictest141rcu_preempt08:18:250
1929996336,7cyclictest141rcu_preempt07:06:423
1929996330,8cyclictest141rcu_preempt07:57:033
1929996316,21cyclictest41-21ksoftirqd/307:37:073
1929996316,21cyclictest41-21ksoftirqd/307:37:073
1919996342,10cyclictest4148-21apt-get07:11:462
1919996317,19cyclictest1308-21ssh12:34:432
1919996317,19cyclictest1308-21ssh12:34:432
1912996343,10cyclictest0-21swapper/110:33:431
1912996342,16cyclictest25-21ksoftirqd/108:57:111
1912996342,16cyclictest25-21ksoftirqd/108:57:111
1912996342,11cyclictest0-21swapper/110:17:071
1912996337,9cyclictest25-21ksoftirqd/109:36:321
1912996337,9cyclictest25-21ksoftirqd/109:36:321
191299633,31cyclictest0-21swapper/110:21:561
1903996341,8cyclictest141rcu_preempt08:46:480
1903996339,7cyclictest141rcu_preempt11:35:210
1903996337,8cyclictest141rcu_preempt12:21:590
1903996333,16cyclictest141rcu_preempt07:42:050
1903996330,6cyclictest141rcu_preempt11:07:300
1903996330,6cyclictest141rcu_preempt11:07:300
1903996326,11cyclictest141rcu_preempt12:34:160
1903996326,11cyclictest141rcu_preempt12:34:160
1903996322,10cyclictest0-21swapper/009:41:160
1903996322,10cyclictest0-21swapper/009:41:160
1929996235,7cyclictest141rcu_preempt07:21:353
1929996232,7cyclictest141rcu_preempt07:28:453
1929996229,7cyclictest141rcu_preempt07:21:533
1929996224,7cyclictest141rcu_preempt08:44:323
1919996240,16cyclictest14830-21unixbench_singl07:42:092
1919996229,23cyclictest6766-21grep07:17:102
1912996242,7cyclictest25-21ksoftirqd/111:17:011
1912996240,12cyclictest0-21swapper/107:47:101
1903996238,8cyclictest141rcu_preempt10:16:480
1903996237,11cyclictest141rcu_preempt07:10:220
1903996236,8cyclictest141rcu_preempt09:33:380
1903996236,8cyclictest141rcu_preempt09:33:380
1903996234,10cyclictest141rcu_preempt07:57:070
1903996233,16cyclictest141rcu_preempt09:11:570
1903996229,15cyclictest141rcu_preempt12:00:420
1903996229,12cyclictest52-21kcompactd008:31:350
1903996229,12cyclictest52-21kcompactd008:31:350
1929996136,7cyclictest141rcu_preempt10:26:363
1929996135,8cyclictest141rcu_preempt08:36:403
1929996130,11cyclictest141rcu_preempt09:47:213
1919996139,13cyclictest12702-21ssh09:11:572
1919996131,23cyclictest11939-21grep11:57:082
1919996129,23cyclictest1811-21unixbench_singl08:42:082
1919996126,12cyclictest0-21swapper/212:26:562
1919996123,33cyclictest21963-21ssh09:27:232
1919996114,17cyclictest18828-21ssh10:18:512
191299614,17cyclictest8950-21diskmemload09:06:321
1912996137,16cyclictest6043-21unixbench_singl11:47:091
1912996137,13cyclictest0-21swapper/110:06:321
1912996137,13cyclictest0-21swapper/110:06:321
1912996127,12cyclictest25-21ksoftirqd/111:02:041
1912996116,10cyclictest0-21swapper/111:27:101
1903996134,7cyclictest141rcu_preempt07:17:020
1919996042,11cyclictest16846-21idleruntime-cro07:51:322
1919996031,19cyclictest7879-21seq07:21:582
1919996028,20cyclictest3474-21/usr/sbin/munin08:47:092
1919996015,18cyclictest21740-21munin-run08:06:342
1919996015,18cyclictest21740-21munin-run08:06:342
1912996042,12cyclictest1997-21uniq10:46:561
1912996035,8cyclictest25-21ksoftirqd/109:37:001
1912996035,8cyclictest25-21ksoftirqd/109:37:001
1912996035,16cyclictest0-21swapper/108:41:541
1912996033,18cyclictest25-21ksoftirqd/107:07:081
1903996019,19cyclictest12-21ksoftirqd/008:32:070
1912995934,13cyclictest0-21swapper/108:01:321
1903995930,9cyclictest141rcu_preempt08:57:030
1903995930,9cyclictest141rcu_preempt08:57:030
1929995821,17cyclictest141rcu_preempt10:04:463
1929995821,17cyclictest141rcu_preempt10:04:463
1919995836,15cyclictest14111-21ssh12:01:512
1912995837,8cyclictest141rcu_preempt07:36:351
1912995821,13cyclictest25-21ksoftirqd/111:52:001
1903995833,13cyclictest141rcu_preempt07:32:090
1903995825,10cyclictest183-21pr/ttyS008:21:510
1919995729,20cyclictest22422-21/usr/sbin/munin08:07:002
1919995729,19cyclictest31414-21apt08:36:322
1912995723,12cyclictest0-21swapper/112:21:471
1912995635,15cyclictest1-21systemd12:01:321
1919995529,18cyclictest7821-21fschecks_time09:01:532
1912995533,12cyclictest0-21swapper/108:16:551
1912995531,7cyclictest25-21ksoftirqd/112:12:071
1912995437,11cyclictest0-21swapper/107:16:331
191299543,31cyclictest52-21kcompactd008:01:571
191299543,31cyclictest52-21kcompactd008:01:571
1912995329,14cyclictest25-21ksoftirqd/112:06:331
1912995231,14cyclictest25-21ksoftirqd/109:06:561
1912995230,13cyclictest0-21swapper/109:51:581
1912995226,10cyclictest0-21swapper/108:31:591
1912995132,12cyclictest355-21systemd-logind12:31:331
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional