You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-20 - 14:50
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7.osadl.org (updated Sat Apr 20, 2024 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2936621750,9sleep1241ktimers/119:01:111
282952960,5chrt28294-21diskstats20:35:511
30260999242,43cyclictest2267-21latency_hist21:55:323
30241999022,52cyclictest4763-21apt-get22:55:480
30260998815,19cyclictest22352-21cron22:30:323
30254998813,52cyclictest4193-21apt-get22:55:482
30249998725,32cyclictest5146-21diskmemload21:32:291
3024199845,52cyclictest18347-21apt-get20:06:130
30260998315,14cyclictest141rcu_preempt22:30:533
3026099825,22cyclictest5190-21latency_hist22:00:353
3026099824,21cyclictest5693-21fschecks_time22:00:523
3026099824,21cyclictest5693-21fschecks_time22:00:523
3026099824,19cyclictest13473-21/usr/sbin/munin23:10:543
3026099824,17cyclictest17304-21grep00:11:123
3026099824,17cyclictest17304-21grep00:11:123
3026099823,32cyclictest19173-21cstates00:15:483
30241998241,26cyclictest12-21ksoftirqd/021:11:070
30260998131,42cyclictest0-21swapper/320:56:083
30260998126,46cyclictest0-21swapper/320:50:353
3026099804,20cyclictest32694-21cut19:10:553
3026099804,18cyclictest31516-21fschecks_count22:45:533
3026099803,23cyclictest27137-21modprobe00:30:313
30260998032,39cyclictest0-21swapper/321:05:363
30260998015,12cyclictest0-21swapper/319:36:093
2978028014,30sleep00-21swapper/019:01:190
30260997962,10cyclictest22968-21munin-plugin-st20:20:323
30260997961,11cyclictest17505-21unixbench_multi23:16:083
3026099795,18cyclictest13645-21cut22:15:323
3026099793,21cyclictest13162-21cron00:05:313
3026099793,18cyclictest8196-21grep23:55:543
3026099793,18cyclictest8196-21grep23:55:543
30260997925,12cyclictest0-21swapper/319:21:033
30260997925,12cyclictest0-21swapper/319:21:033
30260997918,11cyclictest0-21swapper/323:39:453
3026099785,17cyclictest31144-21latency_hist20:45:343
3026099784,22cyclictest8554-21fschecks_time22:05:533
3026099784,21cyclictest21017-21sed23:22:013
3026099784,19cyclictest16321-21latency_hist20:00:363
3026099783,22cyclictest7860-21latency_hist23:55:373
3026099783,22cyclictest27448-21grep20:31:073
3026099783,19cyclictest14972-21sed22:16:073
3026099783,18cyclictest29424-21cstates21:45:473
3026099783,18cyclictest29424-21cstates21:45:473
3026099782,22cyclictest22885-21sed00:21:063
3026099775,17cyclictest24918-21cpuspeed20:25:483
3026099775,17cyclictest24918-21cpuspeed20:25:483
3026099774,20cyclictest28369-21ssh23:35:343
3026099774,19cyclictest20888-21sed20:11:063
3026099774,19cyclictest20888-21sed20:11:063
3026099774,19cyclictest17967-21apt-key20:05:353
3026099774,19cyclictest17967-21apt-key20:05:353
3026099774,18cyclictest29484-21latency_hist20:40:333
3026099774,18cyclictest10030-21grep19:40:553
3026099774,17cyclictest24365-21switchtime21:36:083
3026099774,17cyclictest24365-21switchtime21:36:083
3026099773,22cyclictest29074-21wc22:41:073
3026099773,20cyclictest13232-21grep19:51:043
3026099773,19cyclictest2585-21ntp_states23:46:013
30260997719,17cyclictest276-21systemd-journal23:05:323
3026099764,20cyclictest5237-21tune2fs22:56:403
3026099763,22cyclictest30405-21latency_hist19:05:373
3026099763,20cyclictest18627-21fschecks_count20:06:583
30260997621,12cyclictest14865-21fschecks_count21:20:533
30260997611,12cyclictest0-21swapper/322:20:413
30260997610,13cyclictest0-21swapper/322:50:533
30260997610,13cyclictest0-21swapper/322:50:533
252792760,14sleep1276-21systemd-journal23:30:321
252792760,14sleep1276-21systemd-journal23:30:321
3026099754,17cyclictest12315-21sed19:46:123
3026099753,26cyclictest1102-21ps20:51:023
3026099753,26cyclictest1102-21ps20:51:023
3026099753,21cyclictest23058-21kernelversion23:25:583
3026099753,21cyclictest23058-21kernelversion23:25:583
30260997522,8cyclictest0-21swapper/321:15:473
3025499756,50cyclictest16507-21kworker/u8:320:06:132
3026099748,16cyclictest0-21swapper/321:06:473
3026099744,17cyclictest2594-21grep19:16:053
3026099743,17cyclictest9689-21grep21:11:073
30260997423,8cyclictest0-21swapper/321:25:593
30254997455,13cyclictest14503-21unixbench_multi00:06:062
30241997437,30cyclictest24333-21munin-plugin-st00:25:320
3026099735,17cyclictest21539-21ssh21:31:063
3026099734,20cyclictest5092-21fschecks_count19:25:553
3026099733,19cyclictest7497-21sed19:31:103
30241997319,17cyclictest20382-21ls21:30:350
3026099724,21cyclictest4997-21kworker/u8:021:40:573
3026099724,16cyclictest24126-21unixbench_multi20:21:093
3025499724,35cyclictest0-21swapper/221:06:372
30249997257,9cyclictest17858-21fschecks_time21:25:521
30249997234,28cyclictest0-21swapper/121:51:091
3024199729,38cyclictest19700-21kworker/u8:323:20:590
3024199724,50cyclictest0-21swapper/021:05:540
3026099713,17cyclictest14474-21unixbench_multi00:06:063
30241997123,39cyclictest32047-21/usr/sbin/munin23:41:000
30241997123,39cyclictest32047-21/usr/sbin/munin23:41:000
3026099703,20cyclictest26450-21ssh22:36:093
3026099703,19cyclictest13167-21latency_hist23:10:343
3026099703,19cyclictest13167-21latency_hist23:10:343
30254997041,16cyclictest0-21swapper/220:15:332
30254997041,16cyclictest0-21swapper/220:15:332
3024999703,58cyclictest0-21swapper/122:05:441
3024999703,51cyclictest5141-21apt-get21:05:581
30249997026,37cyclictest15569-21unixbench_singl19:56:071
3024999702,60cyclictest16279-21ssh00:10:461
3024999702,60cyclictest16279-21ssh00:10:461
3024999702,59cyclictest0-21swapper/100:25:401
30241997018,21cyclictest21726-21sort00:20:340
3026099694,23cyclictest29778-21munin-plugin-st00:35:323
3026099693,22cyclictest399-21runrttasks23:51:513
30254996949,13cyclictest14329-21systemd-run21:20:332
30249996949,14cyclictest19136-21basename23:20:321
30249996854,9cyclictest306-21runrttasks19:10:561
30249996851,9cyclictest25848-21fschecks_count23:30:551
30249996833,26cyclictest0-21swapper/121:20:531
30249996750,11cyclictest9516-21munin-plugin-st19:40:331
3024999674,55cyclictest0-21swapper/122:50:431
3024999674,55cyclictest0-21swapper/122:50:431
30241996721,29cyclictest28868-21chrt21:45:110
30260996623,8cyclictest0-21swapper/323:41:263
30260996623,8cyclictest0-21swapper/323:41:263
30254996645,14cyclictest2712-21unixbench_singl22:51:082
30254996645,14cyclictest2712-21unixbench_singl22:51:082
3024999663,57cyclictest0-21swapper/100:30:461
3024999663,55cyclictest0-21swapper/121:35:451
3024999663,55cyclictest0-21swapper/121:35:451
30249996632,27cyclictest0-21swapper/123:46:071
3024999662,56cyclictest0-21swapper/121:00:471
30249996611,47cyclictest16764-21sort20:01:411
30249996611,47cyclictest16764-21sort20:01:411
30241996647,13cyclictest14981-21unixbench_multi22:16:070
30241996641,17cyclictest10876-21latency_hist22:10:340
30241996623,34cyclictest9422-21runrttasks23:57:210
30241996623,34cyclictest9422-21runrttasks23:57:210
30254996550,10cyclictest16000-21munin-plugin-st20:00:332
30254996543,12cyclictest21548-21runrttasks00:20:312
30254996542,13cyclictest12675-21kworker/u8:000:15:352
30254996542,13cyclictest12675-21kworker/u8:000:15:352
3024999654,55cyclictest0-21swapper/100:20:451
3024999653,55cyclictest31282-21ssh22:45:471
3024999653,54cyclictest0-21swapper/122:00:461
3024999653,54cyclictest0-21swapper/122:00:461
30249996531,28cyclictest0-21swapper/100:16:071
30249996531,26cyclictest0-21swapper/120:11:051
30249996531,26cyclictest0-21swapper/120:11:051
30249996530,27cyclictest0-21swapper/123:35:561
30241996515,16cyclictest399-21runrttasks00:05:480
30241996512,17cyclictest13006-21rm22:13:400
3024999643,54cyclictest0-21swapper/119:15:441
3024999642,56cyclictest0-21swapper/123:21:351
3024999642,56cyclictest0-21swapper/121:45:431
3024999642,56cyclictest0-21swapper/121:45:431
3024999642,55cyclictest29300-21cat22:41:101
3024999642,51cyclictest6526-21apt-get19:30:451
3024999642,50cyclictest22851-21ssh22:30:511
30249996412,44cyclictest324-21dbus-daemon20:30:341
30249996412,44cyclictest324-21dbus-daemon20:30:341
30254996344,13cyclictest12299-21runrttasks19:46:122
30254996341,12cyclictest0-21swapper/223:35:322
3024999636,49cyclictest11680-21ssh21:15:501
3024999633,53cyclictest0-21swapper/119:45:501
3024999633,52cyclictest11028-21fschecks_time00:00:541
3024999633,51cyclictest25501-21/usr/sbin/munin22:35:481
3024999633,50cyclictest7924-21apt-get23:55:441
3024999633,50cyclictest7924-21apt-get23:55:441
3024999633,50cyclictest4763-21apt-get22:56:181
30249996328,14cyclictest26563-21ssh21:40:521
3024999632,55cyclictest0-21swapper/120:30:461
3024999632,54cyclictest0-21swapper/119:25:461
3024999632,53cyclictest19754-21apt-get22:25:421
3024999632,53cyclictest13501-21ssh23:10:461
3024999632,52cyclictest2892-21/usr/sbin/munin21:55:541
30241996347,9cyclictest25521-21apt-get22:35:430
30241996343,14cyclictest7624-21install23:55:320
30241996343,14cyclictest7624-21install23:55:320
30241996315,16cyclictest24321-21sh23:27:500
30241996315,16cyclictest24321-21sh23:27:500
30254996240,12cyclictest0-21swapper/221:50:362
30254996235,16cyclictest0-21swapper/219:05:552
3024999623,51cyclictest0-21swapper/120:50:431
3024999623,51cyclictest0-21swapper/120:50:431
3024999622,53cyclictest0-21swapper/123:00:421
3024999622,52cyclictest13272-21sort19:51:041
3024199627,15cyclictest0-21swapper/021:57:380
30241996238,15cyclictest0-21swapper/022:25:530
30241996216,16cyclictest3757-21modprobe21:00:350
3025499615,29cyclictest19669-21apt-get23:20:592
30254996144,8cyclictest0-21swapper/222:40:322
30254996140,14cyclictest0-21swapper/219:30:552
3024999618,46cyclictest24160-21grep20:21:101
3024999614,49cyclictest31479-21apt-get23:40:431
3024999614,49cyclictest31479-21apt-get23:40:431
3024999613,51cyclictest30654-21fschecks_count19:05:551
3024999613,51cyclictest19363-21tail20:07:141
3024999613,49cyclictest11065-21/usr/sbin/munin22:11:071
3024999612,52cyclictest0-21swapper/120:40:481
3024999612,49cyclictest3072-21tail20:56:071
3024999612,49cyclictest16751-21latency_hist22:20:351
3024999612,49cyclictest13553-21/usr/sbin/munin00:05:461
3024199617,26cyclictest141rcu_preempt22:21:050
30241996143,11cyclictest19189-21rm23:20:330
30241996139,14cyclictest29402-21sh21:45:470
30241996139,14cyclictest29402-21sh21:45:470
30241996120,33cyclictest18168-21latency_hist20:05:350
30241996120,33cyclictest18168-21latency_hist20:05:350
30241996115,30cyclictest22932-21runrttasks20:20:220
30241996113,15cyclictest2097-21wc22:50:560
30241996113,15cyclictest2097-21wc22:50:560
30241996112,19cyclictest17000-21tr00:11:080
30241996112,19cyclictest17000-21tr00:11:080
2895626111,25sleep20-21swapper/219:00:572
30254996038,13cyclictest0-21swapper/223:06:062
30254996038,13cyclictest0-21swapper/223:06:062
3025499602,26cyclictest141rcu_preempt19:55:322
3024999603,50cyclictest5146-21diskmemload23:51:011
3024999603,50cyclictest5146-21diskmemload23:51:011
3024999603,48cyclictest3477-21sort19:20:531
3024999603,48cyclictest3477-21sort19:20:531
3024999602,52cyclictest0-21swapper/122:20:421
3024999602,49cyclictest10492-21acpi23:05:391
3024999602,49cyclictest10492-21acpi23:05:391
3024999602,48cyclictest8800-21/usr/sbin/munin21:10:501
30241996045,8cyclictest26169-21latency_hist21:40:340
30241996045,8cyclictest26169-21latency_hist21:40:340
30241996037,12cyclictest26498-21apt-get20:30:430
3024199603,16cyclictest141rcu_preempt19:16:050
30241996015,17cyclictest23444-21grep20:20:560
30241996015,17cyclictest2082-21ssh23:45:470
30254995940,10cyclictest0-21swapper/221:40:322
30254995940,10cyclictest0-21swapper/221:40:322
30254995910,42cyclictest7596-21munin-plugin-st23:55:322
30254995910,42cyclictest7596-21munin-plugin-st23:55:322
3024999593,47cyclictest32343-21/usr/sbin/munin20:46:081
3024999592,52cyclictest0-21swapper/119:40:491
3024999592,51cyclictest0-21swapper/120:15:421
30241995941,12cyclictest9516-21munin-plugin-st19:40:330
30241995936,15cyclictest4897-21/usr/sbin/munin19:26:060
30241995932,19cyclictest13010-21apt-get19:50:580
30241995917,13cyclictest0-21swapper/021:51:090
30241995916,14cyclictest24606-21/usr/sbin/munin00:25:460
3025499589,19cyclictest141rcu_preempt22:10:352
30254995843,9cyclictest31863-21modprobe21:50:322
30254995836,12cyclictest0-21swapper/223:30:322
30254995836,12cyclictest0-21swapper/223:30:322
30241995839,8cyclictest28562-21ssh23:35:410
30241995825,21cyclictest29753-21apt-get20:40:450
30241995812,35cyclictest13404-21latency_hist00:05:350
30241995810,17cyclictest0-21swapper/023:00:430
3025499578,42cyclictest0-21swapper/220:01:072
3025499578,42cyclictest0-21swapper/220:01:072
30254995738,9cyclictest0-21swapper/223:20:322
30254995738,13cyclictest2550-21ntp_kernel_pll_23:45:592
3024199579,28cyclictest20116-21df_inode20:10:500
3024199579,28cyclictest20116-21df_inode20:10:500
30241995745,7cyclictest12-21ksoftirqd/023:31:270
30241995732,14cyclictest26244-21idleruntime-cro20:30:330
30241995732,14cyclictest26244-21idleruntime-cro20:30:330
30241995726,19cyclictest28970-21/usr/sbin/munin22:41:020
3024199572,18cyclictest0-21swapper/022:01:250
3024199572,18cyclictest0-21swapper/022:01:250
30254995639,11cyclictest27832-21cron20:35:322
30254995639,11cyclictest0-21swapper/220:55:332
30254995639,11cyclictest0-21swapper/220:55:332
30254995639,10cyclictest25314-21processes00:26:022
30254995633,12cyclictest0-21swapper/221:25:332
3024199567,27cyclictest0-21swapper/000:30:560
3024199565,17cyclictest0-21swapper/022:30:530
30241995635,10cyclictest11651-21apt-get21:15:470
30241995614,32cyclictest13210-21wc23:10:340
30241995614,32cyclictest13210-21wc23:10:340
30241995611,17cyclictest16114-21rm23:15:310
30254995532,11cyclictest0-21swapper/221:55:342
30254995531,14cyclictest0-21swapper/223:56:082
30254995531,14cyclictest0-21swapper/223:56:082
30241995539,11cyclictest0-21swapper/021:35:350
30241995534,13cyclictest0-21swapper/020:55:550
30241995521,26cyclictest31468-21cut22:45:520
30241995518,29cyclictest414-21latency_hist20:50:350
30241995517,8cyclictest15219-21ntp_states21:21:010
30260995435,11cyclictest347-21in:imklog19:10:333
3025499544,10cyclictest0-21swapper/221:41:062
3025499543,13cyclictest141rcu_preempt22:50:322
30254995431,16cyclictest0-21swapper/220:36:082
30254995429,14cyclictest0-21swapper/219:25:352
30254995429,14cyclictest0-21swapper/219:25:352
30254995416,9cyclictest5146-21diskmemload23:45:322
30254995416,9cyclictest5146-21diskmemload23:45:322
30254995336,10cyclictest16213-21systemd23:15:342
3025499533,23cyclictest21790-21sort20:15:522
30254995328,15cyclictest0-21swapper/219:26:102
30241995332,13cyclictest11596-21grep19:45:580
30241995330,13cyclictest0-21swapper/019:56:060
30241995327,19cyclictest30459-21chrt19:05:460
30241995312,16cyclictest0-21swapper/019:41:050
30254995235,11cyclictest5261-21latency_hist22:00:352
30254995232,13cyclictest0-21swapper/221:35:332
30254995229,13cyclictest0-21swapper/221:05:342
30254995228,14cyclictest0-21swapper/221:25:532
30241995236,10cyclictest276-21systemd-journal20:55:350
30241995236,10cyclictest276-21systemd-journal20:55:350
30241995213,31cyclictest32477-21/usr/sbin/munin19:11:090
3025499516,19cyclictest141rcu_preempt22:27:082
30254995136,9cyclictest16554-21systemd22:20:342
3025499513,19cyclictest11528-21cat22:10:572
30254995129,12cyclictest0-21swapper/219:40:542
30254995128,12cyclictest0-21swapper/220:40:542
30241995135,10cyclictest0-21swapper/019:35:330
3025499504,12cyclictest3193-21latency_hist19:20:352
30254995033,10cyclictest5433-21cpu22:00:462
30254995033,10cyclictest5433-21cpu22:00:462
3025499502,19cyclictest17053-21fschecks_count22:20:522
30241995018,7cyclictest141rcu_preempt20:35:530
3025499498,9cyclictest33-21ksoftirqd/220:55:502
30254994930,12cyclictest0-21swapper/220:50:322
3024199494,18cyclictest0-21swapper/019:21:070
3024199494,18cyclictest0-21swapper/019:21:070
30254994831,9cyclictest0-21swapper/200:00:522
30254994831,11cyclictest0-21swapper/222:35:322
30254994830,8cyclictest0-21swapper/219:14:592
30254994822,18cyclictest28816-21fschecks_count23:35:502
30254994810,9cyclictest5146-21diskmemload23:03:532
3025499473,16cyclictest5146-21diskmemload22:42:432
30254994730,9cyclictest179-21pr/ttyS019:35:362
30254994729,9cyclictest0-21swapper/221:10:532
30254994727,12cyclictest399-21runrttasks00:31:032
3025499453,23cyclictest0-21swapper/220:25:352
30254994531,7cyclictest0-21swapper/200:21:062
30254994529,10cyclictest276-21systemd-journal20:30:352
30254994529,10cyclictest276-21systemd-journal20:30:352
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional