You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-31 - 05:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  

x86 Intel Core i3-8145UE @2200 MHz, Linux 5.10.27-rt36 (Profile)

Latency plot of shadow in rack #e, slot #5
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Jan 31, 2023 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1729099260,2cyclictest19554-21ssh22:53:162
1728399261,24cyclictest0-21swapper/023:02:280
1728399260,2cyclictest8143-21cron21:05:010
1729599250,0cyclictest10547-21diskmemload00:35:223
1729599250,0cyclictest0-21swapper/323:36:263
17290992522,2cyclictest11712-21fschecks_time00:20:392
1729099250,24cyclictest0-21swapper/200:00:122
1728699250,24cyclictest0-21swapper/121:12:201
1728699250,24cyclictest0-21swapper/121:12:201
1728399253,21cyclictest62150irq/126-eno123:25:490
17283992523,1cyclictest10834-21ssh22:45:140
1728399250,1cyclictest13701-21ssh00:21:530
17295992423,1cyclictest12607-21ssh23:50:133
17295992421,2cyclictest324-21rm00:10:073
1729599240,2cyclictest3527-21latency_hist23:10:003
1729599240,23cyclictest5821-21ssh23:43:183
1729599240,23cyclictest1068-21sh21:30:523
17290992424,0cyclictest0-21swapper/221:11:262
17290992424,0cyclictest0-21swapper/221:11:262
1729099240,23cyclictest30989-21awk21:30:012
1729099240,23cyclictest10547-21diskmemload23:54:012
1729099240,23cyclictest10547-21diskmemload22:45:052
1729099240,1cyclictest23339-21ssh21:53:532
1729099240,1cyclictest20830-21turbostat21:20:012
1729099240,1cyclictest16738-21ssh21:46:282
1729099240,1cyclictest10547-21diskmemload21:21:162
17286992422,1cyclictest19295-21sh21:50:121
1728699242,21cyclictest11371-21wc20:05:001
17286992422,1cyclictest0-21swapper/123:27:351
1728699240,24cyclictest0-21swapper/122:32:041
1728699240,23cyclictest5783-21munin-plugin-st21:00:001
1728699240,23cyclictest22855-21ntp_states00:30:441
1728699240,23cyclictest0-21swapper/122:52:061
1728699240,23cyclictest0-21swapper/100:20:491
1728699240,0cyclictest0-21swapper/123:00:161
1728699240,0cyclictest0-21swapper/122:46:151
17283992424,0cyclictest62150irq/126-eno122:15:070
17283992422,1cyclictest0-21swapper/000:03:170
17283992420,3cyclictest5384-21ntp_states21:35:390
1728399241,22cyclictest29415-21rm21:27:310
1728399240,24cyclictest0-21swapper/023:49:560
1728399240,24cyclictest0-21swapper/000:16:590
1728399240,23cyclictest14598-21ssh23:20:330
1728399240,23cyclictest10547-21diskmemload23:42:170
1728399240,23cyclictest10547-21diskmemload23:14:270
1728399240,23cyclictest0-21swapper/021:12:500
1728399240,23cyclictest0-21swapper/021:12:500
1728399240,1cyclictest600-21systemd-logind00:00:000
1728399240,1cyclictest0-21swapper/021:48:430
1728399240,0cyclictest0-21swapper/022:26:380
1728399240,0cyclictest0-21swapper/000:26:510
17295992323,0cyclictest10547-21diskmemload23:10:403
17295992323,0cyclictest10547-21diskmemload22:25:583
17295992323,0cyclictest0-21swapper/323:47:553
17295992323,0cyclictest0-21swapper/323:20:123
17295992323,0cyclictest0-21swapper/319:20:313
17295992323,0cyclictest0-21swapper/300:27:093
17295992322,1cyclictest10394-21ssh21:40:463
17295992321,1cyclictest20602-21memory23:25:373
17295992321,1cyclictest17524-21sh23:55:043
17295992321,1cyclictest0-21swapper/323:00:353
17295992321,1cyclictest0-21swapper/322:35:343
17295992320,2cyclictest14460-21apt-get21:45:333
1729599230,2cyclictest30204-21sh22:31:093
1729599230,23cyclictest0-21swapper/323:30:433
1729599230,23cyclictest0-21swapper/323:30:433
1729599230,23cyclictest0-21swapper/322:00:333
1729599230,22cyclictest26571-21ssh21:55:523
1729599230,22cyclictest22081-21sendmail-msp19:20:003
1729599230,22cyclictest14646-21sh00:23:253
1729599230,22cyclictest10547-21diskmemload21:27:053
1729599230,0cyclictest10547-21diskmemload22:21:463
1729599230,0cyclictest10547-21diskmemload22:11:483
1729599230,0cyclictest0-21swapper/322:42:033
1729599230,0cyclictest0-21swapper/322:18:483
17290992323,0cyclictest10547-21diskmemload21:59:272
17290992323,0cyclictest10547-21diskmemload00:35:472
17290992323,0cyclictest0-21swapper/223:42:262
17290992323,0cyclictest0-21swapper/223:36:082
17290992323,0cyclictest0-21swapper/223:24:452
17290992323,0cyclictest0-21swapper/223:10:302
17290992323,0cyclictest0-21swapper/222:20:562
17290992322,0cyclictest0-21swapper/222:11:482
17290992321,1cyclictest11145-21ssh23:15:552
17290992321,1cyclictest0-21swapper/200:27:202
17290992320,2cyclictest29189-21munin-plugin-st23:35:012
17290992320,2cyclictest29189-21munin-plugin-st23:35:012
1729099230,2cyclictest7012-21ssh21:37:122
1729099230,23cyclictest0-21swapper/223:06:252
1729099230,23cyclictest0-21swapper/222:35:232
1729099230,23cyclictest0-21swapper/222:30:472
1729099230,23cyclictest0-21swapper/221:30:032
1729099230,22cyclictest8799-21ssh22:42:212
1729099230,22cyclictest13122-21ssh21:43:512
1729099230,22cyclictest11917-21wc23:50:012
1729099230,22cyclictest10547-21diskmemload22:56:282
1729099230,22cyclictest10547-21diskmemload00:05:512
1729099230,22cyclictest0-21swapper/200:30:382
17286992323,0cyclictest0-21swapper/123:41:561
17286992323,0cyclictest0-21swapper/122:01:431
17286992323,0cyclictest0-21swapper/121:35:111
17286992322,1cyclictest5163-21ssh22:07:091
17286992321,1cyclictest0-21swapper/123:53:041
17286992321,1cyclictest0-21swapper/122:14:171
17286992321,1cyclictest0-21swapper/121:43:421
17286992321,1cyclictest0-21swapper/100:02:281
1728699230,23cyclictest0-21swapper/123:18:131
1728699230,23cyclictest0-21swapper/123:06:451
1728699230,23cyclictest0-21swapper/100:16:001
1728699230,22cyclictest0-21swapper/123:38:071
1728699230,22cyclictest0-21swapper/123:10:581
1728699230,1cyclictest0-21swapper/122:15:111
1728699230,0cyclictest10547-21diskmemload21:47:061
1728699230,0cyclictest0-21swapper/123:57:281
1728699230,0cyclictest0-21swapper/122:43:571
1728699230,0cyclictest0-21swapper/100:39:151
1728699230,0cyclictest0-21swapper/100:11:141
1728699230,0cyclictest0-21swapper/100:05:221
17283992323,0cyclictest10547-21diskmemload23:06:430
17283992323,0cyclictest0-21swapper/023:50:270
17283992323,0cyclictest0-21swapper/022:53:220
17283992323,0cyclictest0-21swapper/021:42:440
17283992323,0cyclictest0-21swapper/021:16:030
17283992321,1cyclictest11430-21sh23:16:250
17283992320,2cyclictest31323-21sh21:30:120
1728399230,23cyclictest0-21swapper/022:24:530
1728399230,23cyclictest0-21swapper/000:35:200
1728399230,23cyclictest0-21swapper/000:05:010
1728399230,22cyclictest24739-21ssh21:23:090
1728399230,22cyclictest24476-21ssh23:30:130
1728399230,22cyclictest24476-21ssh23:30:130
1728399230,22cyclictest20730-21apt-get00:30:020
1728399230,22cyclictest10547-21diskmemload23:37:180
1728399230,22cyclictest10547-21diskmemload21:55:080
1728399230,1cyclictest3025-21fw_packets22:05:340
1728399230,0cyclictest11056-21ssh22:13:010
1728399230,0cyclictest0-21swapper/022:56:450
1728399230,0cyclictest0-21swapper/022:30:030
1728399230,0cyclictest0-21swapper/022:01:390
1728399230,0cyclictest0-21swapper/000:11:320
17295992222,0cyclictest0-21swapper/322:55:353
17295992222,0cyclictest0-21swapper/321:50:153
17295992222,0cyclictest0-21swapper/321:20:213
17295992222,0cyclictest0-21swapper/321:15:093
17295992221,1cyclictest20139-21ssh22:54:133
17295992219,2cyclictest8888-21munin-plugin-st21:40:003
1729599220,22cyclictest0-21swapper/322:45:463
1729599220,22cyclictest0-21swapper/300:16:323
1729599220,22cyclictest0-21swapper/300:01:003
1729599220,21cyclictest6071-21apt-get21:00:313
1729599220,21cyclictest2756-21expr22:05:333
1729599220,21cyclictest23659-21apt-get20:30:323
1729599220,21cyclictest21452-21ssh00:30:163
1729599220,21cyclictest21263-21apt-get20:25:313
1729599220,21cyclictest16444-21apt-get20:15:323
1729599220,1cyclictest9178-21apt-get23:15:323
1729599220,0cyclictest10547-21diskmemload00:09:453
17290992222,0cyclictest0-21swapper/222:25:332
17290992222,0cyclictest0-21swapper/200:16:362
17290992220,1cyclictest31684-21cron19:40:012
17290992220,1cyclictest30587-21awk20:45:002
1729099220,22cyclictest0-21swapper/223:01:582
1729099220,22cyclictest0-21swapper/222:00:322
1729099220,21cyclictest8466-21apt-get21:05:322
1729099220,21cyclictest2781-21ssh00:11:112
1729099220,21cyclictest0-21swapper/223:25:082
1729099220,21cyclictest0-21swapper/222:05:592
1729099220,1cyclictest16964-21ssh22:19:262
1729099220,0cyclictest0-21swapper/223:55:102
17286992222,0cyclictest0-21swapper/122:27:411
17286992221,1cyclictest16444-21apt-get20:15:331
17286992221,1cyclictest10547-21diskmemload21:15:221
17286992221,1cyclictest0-21swapper/121:30:361
17286992220,1cyclictest31722-21sendmail-msp19:40:011
1728699220,22cyclictest19443-21sh00:27:571
1728699220,21cyclictest503-21apt-get20:50:021
1728699220,21cyclictest10547-21diskmemload23:45:151
1728699220,21cyclictest10547-21diskmemload22:35:031
1728699220,21cyclictest10547-21diskmemload22:20:371
1728699220,21cyclictest0-21swapper/121:20:141
1728699220,21cyclictest0-21swapper/120:25:311
1728699220,21cyclictest0-21swapper/119:30:321
1728699220,1cyclictest24427-21ssh21:55:141
1728699220,0cyclictest0-21swapper/123:20:191
1728399220,21cyclictest6845-21apt-get19:55:310
1728399220,21cyclictest28479-21apt-get20:40:360
1728399220,21cyclictest0-21swapper/022:42:330
1728399220,21cyclictest0-21swapper/019:50:310
17295992119,1cyclictest6848-21seq19:55:303
1729599210,1cyclictest5786-21munin-run21:00:003
17290992121,0cyclictest0-21swapper/219:30:322
17290992120,1cyclictest4451-21apt-get19:50:312
17290992120,0cyclictest1114-21apt-get20:50:022
17290992119,1cyclictest1672-21cron19:45:012
17290992118,2cyclictest18561-21turbostat20:20:012
1729099210,20cyclictest0-21swapper/219:55:312
1728699210,2cyclictest1706-21latency_hist19:45:011
1728699210,20cyclictest0-21swapper/119:50:311
1728699210,0cyclictest0-21swapper/123:30:201
1728699210,0cyclictest0-21swapper/123:30:201
17283992121,0cyclictest10547-21diskmemload22:35:160
1728399210,21cyclictest0-21swapper/019:40:340
1728399210,20cyclictest0-21swapper/020:10:350
1728399210,20cyclictest0-21swapper/019:45:320
1728399210,0cyclictest17544-21apt-get19:10:280
1728399210,0cyclictest0-21swapper/021:05:320
17295992020,0cyclictest2001-21apt-get19:45:323
17295992020,0cyclictest0-21swapper/319:29:033
1729599200,0cyclictest30873-21apt-get20:45:333
1729599200,0cyclictest0-21swapper/320:20:323
17290992020,0cyclictest0-21swapper/220:45:332
17290992020,0cyclictest0-21swapper/220:25:312
17290992018,1cyclictest22063-21perf19:20:002
1729099201,4cyclictest23295-21/usr/sbin/munin19:20:362
1729099200,0cyclictest14054-21apt-get20:10:352
17286992020,0cyclictest0-21swapper/120:10:351
1728699200,0cyclictest0-21swapper/121:00:311
17283992020,0cyclictest0-21swapper/020:55:320
17283992020,0cyclictest0-21swapper/020:15:330
17283992020,0cyclictest0-21swapper/019:35:310
17283992018,1cyclictest0-21swapper/020:30:010
1728399200,0cyclictest24752-21apt-get19:25:310
1728399200,0cyclictest0-21swapper/020:50:330
17295991919,0cyclictest9264-21apt-get20:00:323
17295991919,0cyclictest0-21swapper/320:50:023
17295991919,0cyclictest0-21swapper/320:05:313
17290991919,0cyclictest0-21swapper/221:00:322
17290991919,0cyclictest0-21swapper/219:25:312
17286991919,0cyclictest0-21swapper/120:45:331
17286991919,0cyclictest0-21swapper/119:10:281
17283991919,0cyclictest0-21swapper/020:35:320
17295991818,0cyclictest0-21swapper/321:05:323
17295991818,0cyclictest0-21swapper/320:40:373
17295991818,0cyclictest0-21swapper/320:10:353
17290991817,0cyclictest0-21swapper/219:45:322
17286991818,0cyclictest0-21swapper/120:05:311
17286991818,0cyclictest0-21swapper/119:20:321
17283991818,0cyclictest0-21swapper/019:15:310
17283991816,1cyclictest512-21latency_hist20:50:000
17290991717,0cyclictest0-21swapper/220:05:312
17290991717,0cyclictest0-21swapper/219:15:002
17290991716,0cyclictest0-21swapper/220:30:312
17286991717,0cyclictest0-21swapper/119:55:311
17295991616,0cyclictest0-21swapper/319:30:323
17295991613,2cyclictest309-21systemd-journal19:45:013
17290991616,0cyclictest3651-21apt-get20:55:322
17290991616,0cyclictest26056-21apt-get20:35:322
17286991616,0cyclictest0-21swapper/119:25:311
1728699160,0cyclictest0-21swapper/120:35:331
17283991616,0cyclictest27197-21apt-get19:30:320
17283991614,1cyclictest0-21swapper/020:35:010
17295991515,0cyclictest0-21swapper/320:35:323
17295991515,0cyclictest0-21swapper/319:35:313
17295991515,0cyclictest0-21swapper/319:10:283
17295991513,1cyclictest774-21runrttasks19:50:573
1729099150,14cyclictest0-21swapper/220:20:332
17286991515,0cyclictest18870-21apt-get20:20:321
17286991515,0cyclictest0-21swapper/121:05:321
17286991515,0cyclictest0-21swapper/119:45:331
17283991513,1cyclictest22359-21apt-get19:20:310
17286991414,0cyclictest0-21swapper/120:30:311
1728399140,14cyclictest0-21swapper/020:20:560
1728399140,14cyclictest0-21swapper/020:01:360
1728399140,0cyclictest0-21swapper/020:05:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional