You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-11-29 - 14:29

x86 Intel Atom E3845 @1910 MHz, Linux 4.19.37-rt19 (Profile)

Latency plot of system in rack #d, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Up99 -i200 -h400 --smi -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackdslot3.osadl.org (updated Tue Nov 29, 2022 00:46:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22462161139,10sleep30-21swapper/319:09:073
22462161139,10sleep30-21swapper/319:09:073
576981300,13rtkit-daemon0-21swapper/019:05:460
576981300,13rtkit-daemon0-21swapper/019:05:460
266199815,14cyclictest28003-21date00:40:002
266199709,13cyclictest25222-21needreboot21:05:262
266199707,13cyclictest16149-21tune2fs23:05:242
21052704,53sleep10-21swapper/119:07:151
21052704,53sleep10-21swapper/119:07:151
30212670,2sleep30-21swapper/322:35:363
266199659,11cyclictest1414-21switchtime23:40:282
266199656,18cyclictest3872-21if_enp1s020:20:232
266199643,14cyclictest20778-21irqrtprio22:05:292
825591630,1ptp4l401ktimersoftd/323:00:223
825591630,1ptp4l401ktimersoftd/321:15:293
266199634,14cyclictest20259-21mailstats00:20:302
825591610,1ptp4l401ktimersoftd/321:00:253
266199616,9cyclictest20907-21/usr/sbin/munin19:45:322
266199615,10cyclictest8633-21ls19:20:252
266199605,10cyclictest21100-21hddtemp_smartct23:15:262
266199596,10cyclictest1417-21sshd20:15:202
266199585,12cyclictest19472-21munin-node20:55:142
266199584,10cyclictest4191-21/usr/sbin/munin19:10:302
266199584,10cyclictest4191-21/usr/sbin/munin19:10:292
266199575,19cyclictest2355-21gnome-shell20:30:582
266199575,16cyclictest28628-21apt-get20:05:162
266199574,9cyclictest9982-21ls21:40:312
266199574,9cyclictest27112-21apt-get22:20:142
266199574,10cyclictest11426-21processes20:35:272
825591550,1ptp4l401ktimersoftd/322:29:533
266199556,15cyclictest6808-21ptp4l-jitter20:25:252
266199555,17cyclictest1598-21/usr/sbin/munin22:35:282
266199535,10cyclictest15597-21ls19:35:272
266199534,9cyclictest22914-21hddtemp_smartct22:10:202
266199534,9cyclictest22914-21hddtemp_smartct22:10:192
266199534,9cyclictest11053-21apt-get22:55:192
233892530,2sleep30-21swapper/300:30:003
266199526,11cyclictest12735-21fschecks_count19:30:162
266199524,10cyclictest8596-21ls23:55:362
266199524,10cyclictest4751-21iostat_ios22:40:202
2661995210,20cyclictest27765-21tr23:30:152
57698510,2rtkit-daemon0-21swapper/022:41:330
2662995145,3cyclictest41-21ksoftirqd/320:20:003
2662995043,4cyclictest41-21ksoftirqd/321:39:593
266199483,18cyclictest0-21swapper/219:55:152
266199475,11cyclictest19074-21kernelversion23:10:232
266199475,10cyclictest19046-21mailstats22:00:272
202124715,20sleep20-21swapper/219:06:152
202124715,20sleep20-21swapper/219:06:152
825591460,1ptp4l401ktimersoftd/321:24:073
266199461,15cyclictest5215-21/usr/sbin/munin19:15:282
266099461,4cyclictest121rcu_preempt00:15:011
2659994628,15cyclictest9-21ksoftirqd/023:11:260
825591450,1ptp4l401ktimersoftd/320:25:263
2662994538,4cyclictest41-21ksoftirqd/322:20:013
2662994538,4cyclictest41-21ksoftirqd/320:09:593
266199455,10cyclictest30736-21mailstats22:25:272
266199454,9cyclictest10462-21ls00:00:342
2660994538,4cyclictest9650-21cron19:25:001
825591440,1ptp4l401ktimersoftd/323:55:323
825591440,1ptp4l401ktimersoftd/321:50:293
825591440,16ptp4l3137-21iostat23:45:253
825591440,16ptp4l3137-21iostat23:45:243
825591440,11ptp4l401ktimersoftd/319:10:013
825591440,11ptp4l401ktimersoftd/319:10:013
2662994437,4cyclictest41-21ksoftirqd/300:25:013
2662994437,4cyclictest41-21ksoftirqd/300:20:013
266199443,15cyclictest6101-21apt-get21:35:182
266199431,39cyclictest18957-21cron19:45:002
2659994324,4cyclictest9-21ksoftirqd/021:26:080
2659994320,5cyclictest15417-21cat23:04:590
2662994235,4cyclictest41-21ksoftirqd/323:35:163
2662994235,4cyclictest41-21ksoftirqd/319:40:003
2662994235,4cyclictest41-21ksoftirqd/319:15:053
2662994136,3cyclictest20256-21expr19:45:243
2662994135,3cyclictest41-21ksoftirqd/320:15:003
2662994134,4cyclictest41-21ksoftirqd/320:00:013
266199414,11cyclictest23362-21ls21:00:322
2660994134,4cyclictest25-21ksoftirqd/123:50:001
2660994134,4cyclictest25-21ksoftirqd/123:50:001
825591400,1ptp4l401ktimersoftd/320:04:593
2662994034,3cyclictest41-21ksoftirqd/321:30:013
2662994033,4cyclictest41-21ksoftirqd/321:50:003
2660994035,3cyclictest25-21ksoftirqd/119:15:301
2660994033,4cyclictest25-21ksoftirqd/100:03:261
2660994032,5cyclictest25-21ksoftirqd/122:25:001
2659994020,4cyclictest9-21ksoftirqd/019:56:360
2662993933,4cyclictest41-21ksoftirqd/322:44:093
2662993933,3cyclictest41-21ksoftirqd/323:40:223
2662993933,3cyclictest41-21ksoftirqd/321:40:153
2662993932,4cyclictest41-21ksoftirqd/323:19:593
2660993931,5cyclictest25-21ksoftirqd/120:45:501
2660993931,5cyclictest25-21ksoftirqd/120:45:501
2662993832,4cyclictest41-21ksoftirqd/319:27:123
2662993832,4cyclictest41-21ksoftirqd/319:22:083
2662993831,4cyclictest41-21ksoftirqd/300:08:453
2661993833,3cyclictest33-21ksoftirqd/219:52:192
2661993831,4cyclictest33-21ksoftirqd/223:48:332
2661993831,4cyclictest33-21ksoftirqd/223:48:332
2661993831,4cyclictest33-21ksoftirqd/223:35:152
266199380,6cyclictest24112-21grep23:20:272
2660993832,4cyclictest15387-21kworker/1:023:28:581
2660993832,3cyclictest25-21ksoftirqd/123:44:531
2660993831,4cyclictest25-21ksoftirqd/123:40:011
2660993831,4cyclictest25-21ksoftirqd/122:54:251
2660993831,4cyclictest25-21ksoftirqd/119:32:031
2660993831,4cyclictest25-21ksoftirqd/100:18:151
2660993830,5cyclictest25-21ksoftirqd/123:56:511
2660993830,5cyclictest25-21ksoftirqd/120:27:361
2659993832,3cyclictest9-21ksoftirqd/019:25:550
2659993831,4cyclictest9-21ksoftirqd/000:02:120
2662993732,3cyclictest41-21ksoftirqd/322:35:003
2662993732,3cyclictest41-21ksoftirqd/320:40:003
2662993731,4cyclictest41-21ksoftirqd/323:10:003
2662993731,4cyclictest41-21ksoftirqd/322:25:003
2662993731,4cyclictest41-21ksoftirqd/321:10:543
2662993731,3cyclictest41-21ksoftirqd/321:08:193
2662993730,4cyclictest41-21ksoftirqd/320:50:143
2662993729,5cyclictest41-21ksoftirqd/321:30:483
2661993732,3cyclictest33-21ksoftirqd/200:09:192
2661993730,4cyclictest33-21ksoftirqd/220:48:452
2661993730,4cyclictest33-21ksoftirqd/220:48:442
2660993731,4cyclictest25-21ksoftirqd/123:22:351
2660993731,3cyclictest25-21ksoftirqd/121:07:381
2660993730,4cyclictest25-21ksoftirqd/122:43:331
2660993730,4cyclictest25-21ksoftirqd/122:17:191
2660993730,4cyclictest25-21ksoftirqd/121:42:311
2660993730,4cyclictest25-21ksoftirqd/119:42:401
2660993729,5cyclictest25-21ksoftirqd/122:26:551
2660993729,5cyclictest25-21ksoftirqd/122:00:461
2659993731,4cyclictest9-21ksoftirqd/020:16:470
2659993731,3cyclictest9-21ksoftirqd/021:52:030
2659993731,3cyclictest9-21ksoftirqd/021:47:590
2659993731,3cyclictest9-21ksoftirqd/000:19:280
2659993730,4cyclictest9-21ksoftirqd/022:34:320
2662993631,3cyclictest41-21ksoftirqd/300:40:003
2662993630,4cyclictest41-21ksoftirqd/322:50:303
2662993630,3cyclictest41-21ksoftirqd/323:20:253
2662993629,4cyclictest41-21ksoftirqd/323:50:223
2662993629,4cyclictest41-21ksoftirqd/322:06:023
2662993629,4cyclictest41-21ksoftirqd/320:43:193
2662993629,4cyclictest41-21ksoftirqd/320:43:193
2662993628,5cyclictest41-21ksoftirqd/320:33:283
2661993629,4cyclictest33-21ksoftirqd/223:52:342
2661993629,4cyclictest33-21ksoftirqd/222:33:262
2661993629,4cyclictest33-21ksoftirqd/221:53:202
2660993630,3cyclictest25-21ksoftirqd/122:30:581
2660993629,4cyclictest25-21ksoftirqd/123:11:111
2660993629,4cyclictest25-21ksoftirqd/122:05:141
2660993629,4cyclictest25-21ksoftirqd/121:13:411
2660993629,4cyclictest25-21ksoftirqd/120:58:091
2660993629,4cyclictest25-21ksoftirqd/120:34:141
2660993629,4cyclictest25-21ksoftirqd/100:07:561
2660993628,5cyclictest25-21ksoftirqd/123:52:371
2660993628,5cyclictest25-21ksoftirqd/123:02:591
2659993630,4cyclictest9-21ksoftirqd/019:46:170
2659993630,4cyclictest9-21ksoftirqd/019:16:460
2659993630,3cyclictest9-21ksoftirqd/023:38:260
2659993630,3cyclictest9-21ksoftirqd/020:42:550
2659993630,3cyclictest9-21ksoftirqd/020:42:540
2659993630,3cyclictest9-21ksoftirqd/000:07:370
2659993629,4cyclictest9-21ksoftirqd/021:23:030
2659993629,4cyclictest9-21ksoftirqd/019:43:550
2659993629,4cyclictest9-21ksoftirqd/019:23:360
2662993530,3cyclictest41-21ksoftirqd/323:30:233
2662993530,3cyclictest41-21ksoftirqd/321:00:003
2662993529,4cyclictest41-21ksoftirqd/321:55:143
2662993529,3cyclictest41-21ksoftirqd/300:32:183
2661993529,3cyclictest33-21ksoftirqd/221:55:592
2661993528,4cyclictest33-21ksoftirqd/222:46:272
2661993528,4cyclictest33-21ksoftirqd/200:30:112
2660993529,3cyclictest25-21ksoftirqd/121:17:511
2660993529,3cyclictest25-21ksoftirqd/120:05:521
2660993528,4cyclictest25-21ksoftirqd/123:17:451
2660993528,4cyclictest25-21ksoftirqd/121:55:481
2660993528,4cyclictest25-21ksoftirqd/119:53:321
2659993529,4cyclictest9-21ksoftirqd/019:14:090
2659993529,4cyclictest9-21ksoftirqd/019:14:080
2659993529,4cyclictest206-21systemd-journal19:54:550
2659993529,3cyclictest9-21ksoftirqd/000:33:250
2659993528,5cyclictest9-21ksoftirqd/022:47:410
2659993528,4cyclictest9-21ksoftirqd/023:40:130
2659993528,4cyclictest9-21ksoftirqd/022:51:380
2659993528,4cyclictest9-21ksoftirqd/021:09:260
2659993527,5cyclictest9-21ksoftirqd/022:00:370
825591340,1ptp4l401ktimersoftd/319:40:333
825591340,12ptp4l401ktimersoftd/320:45:323
825591340,12ptp4l401ktimersoftd/320:45:313
2662993428,3cyclictest41-21ksoftirqd/319:51:433
2662993427,4cyclictest41-21ksoftirqd/323:25:343
2662993427,4cyclictest41-21ksoftirqd/322:55:023
2662993427,4cyclictest41-21ksoftirqd/320:23:303
2661993428,4cyclictest33-21ksoftirqd/223:00:152
2661993427,4cyclictest33-21ksoftirqd/222:51:242
2661993427,4cyclictest33-21ksoftirqd/219:28:562
2660993428,3cyclictest25-21ksoftirqd/121:52:441
2660993428,3cyclictest25-21ksoftirqd/120:15:451
2660993427,4cyclictest25-21ksoftirqd/123:32:421
2660993427,4cyclictest25-21ksoftirqd/122:57:301
2660993427,4cyclictest25-21ksoftirqd/122:11:141
2660993427,4cyclictest25-21ksoftirqd/122:11:131
2660993427,4cyclictest25-21ksoftirqd/121:20:411
2660993427,4cyclictest25-21ksoftirqd/119:59:291
2660993427,4cyclictest25-21ksoftirqd/119:35:331
2660993426,5cyclictest25-21ksoftirqd/123:09:301
2659993428,4cyclictest9-21ksoftirqd/023:16:030
2659993428,3cyclictest9-21ksoftirqd/021:37:590
2659993428,3cyclictest9-21ksoftirqd/021:33:370
2659993428,3cyclictest9-21ksoftirqd/020:51:090
2659993427,4cyclictest9-21ksoftirqd/023:58:280
2659993427,4cyclictest9-21ksoftirqd/022:28:380
2659993427,4cyclictest9-21ksoftirqd/022:20:390
2659993427,4cyclictest9-21ksoftirqd/021:16:570
2659993427,4cyclictest9-21ksoftirqd/020:57:050
2659993427,4cyclictest9-21ksoftirqd/020:47:550
2659993427,4cyclictest9-21ksoftirqd/020:47:540
2659993425,6cyclictest9-21ksoftirqd/020:35:060
2662993327,4cyclictest41-21ksoftirqd/319:30:053
2662993327,4cyclictest41-21ksoftirqd/300:10:253
2661993326,4cyclictest33-21ksoftirqd/221:20:442
2660993326,4cyclictest25-21ksoftirqd/121:36:501
2660993326,4cyclictest25-21ksoftirqd/121:32:001
2660993326,4cyclictest25-21ksoftirqd/120:35:101
2660993326,4cyclictest25-21ksoftirqd/120:01:451
2659993327,4cyclictest9-21ksoftirqd/021:04:360
2659993327,4cyclictest9-21ksoftirqd/019:36:130
2659993327,4cyclictest9-21ksoftirqd/000:26:400
2659993327,3cyclictest9-21ksoftirqd/023:49:220
2659993327,3cyclictest9-21ksoftirqd/023:49:210
2659993327,3cyclictest9-21ksoftirqd/023:31:360
2659993327,3cyclictest9-21ksoftirqd/021:55:450
2659993326,4cyclictest9-21ksoftirqd/022:58:330
2659993326,4cyclictest9-21ksoftirqd/022:11:380
2659993326,4cyclictest9-21ksoftirqd/022:11:380
2659993326,4cyclictest9-21ksoftirqd/021:44:110
825591320,1ptp4l401ktimersoftd/323:11:073
2662993226,4cyclictest41-21ksoftirqd/300:00:413
2662993226,3cyclictest41-21ksoftirqd/322:49:013
2662993225,4cyclictest41-21ksoftirqd/322:11:033
2662993225,4cyclictest41-21ksoftirqd/322:11:033
2662993225,4cyclictest41-21ksoftirqd/322:01:083
2661993226,4cyclictest33-21ksoftirqd/223:25:352
2661993225,4cyclictest33-21ksoftirqd/221:34:462
2661993220,6cyclictest17223-21apt-get20:50:112
2660993226,4cyclictest25-21ksoftirqd/122:35:461
2660993225,4cyclictest25-21ksoftirqd/119:11:371
2660993225,4cyclictest25-21ksoftirqd/119:11:361
2660993225,4cyclictest25-21ksoftirqd/100:36:131
2660993225,4cyclictest25-21ksoftirqd/100:25:451
2659993226,4cyclictest9-21ksoftirqd/020:29:040
2659993225,4cyclictest9-21ksoftirqd/023:51:560
2659993225,4cyclictest9-21ksoftirqd/023:22:180
2659993225,4cyclictest9-21ksoftirqd/022:37:420
2659993225,4cyclictest9-21ksoftirqd/019:34:290
2661993125,3cyclictest33-21ksoftirqd/200:28:002
2661993124,4cyclictest33-21ksoftirqd/221:12:372
2660993126,3cyclictest25-21ksoftirqd/122:45:371
2660993126,3cyclictest25-21ksoftirqd/120:13:101
2660993125,4cyclictest25-21ksoftirqd/100:32:041
2660993124,4cyclictest25-21ksoftirqd/121:29:301
2660993124,4cyclictest25-21ksoftirqd/120:20:451
2660993124,4cyclictest25-21ksoftirqd/119:29:191
2659993125,3cyclictest9-21ksoftirqd/023:09:070
2659993124,4cyclictest9-21ksoftirqd/020:11:260
2659993124,4cyclictest9-21ksoftirqd/020:05:390
2661993017,7cyclictest30947-21apt-get20:10:132
2660993024,4cyclictest25-21ksoftirqd/120:43:191
2660993024,4cyclictest25-21ksoftirqd/120:43:181
2660993024,3cyclictest25-21ksoftirqd/119:47:411
2660993023,4cyclictest25-21ksoftirqd/100:21:031
2659993024,4cyclictest9-21ksoftirqd/022:16:270
2659993024,3cyclictest9-21ksoftirqd/000:24:420
2659993023,4cyclictest9-21ksoftirqd/000:39:200
257072300,2sleep20-21swapper/222:15:312
2661992924,3cyclictest33-21ksoftirqd/221:15:492
2661992923,4cyclictest33-21ksoftirqd/221:45:082
2660992924,3cyclictest25-21ksoftirqd/121:01:121
2659992923,4cyclictest9-21ksoftirqd/020:33:250
2659992923,3cyclictest9-21ksoftirqd/022:05:200
2659992922,4cyclictest9-21ksoftirqd/020:21:290
266199289,16cyclictest14602-21latency_hist20:45:012
266199289,16cyclictest14602-21latency_hist20:45:002
266199289,15cyclictest14432-21tr00:10:172
2659992822,4cyclictest9-21ksoftirqd/000:11:220
2659992822,3cyclictest9-21ksoftirqd/021:12:020
2661992720,4cyclictest33-21ksoftirqd/200:19:022
2659992722,3cyclictest9-21ksoftirqd/020:02:560
2659992720,4cyclictest9-21ksoftirqd/023:28:040
2661992615,8cyclictest27233-21missed_timers20:00:252
2661992610,11cyclictest0-21swapper/221:25:162
2660992419,3cyclictest25-21ksoftirqd/121:45:251
2660992418,4cyclictest25-21ksoftirqd/120:53:421
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional