You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-29 - 21:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  

x86 Intel Core i3-8145UE @2200 MHz, Linux 5.10.27-rt36 (Profile)

Latency plot of shadow in rack #e, slot #5
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Wed Mar 29, 2023 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27411992623,2cyclictest597-21dbus-daemon07:30:000
27420992525,0cyclictest0-21swapper/212:01:392
27420992522,0cyclictest0-21swapper/209:11:162
27420992522,0cyclictest0-21swapper/209:11:162
2742099250,24cyclictest0-21swapper/210:04:242
2741199250,24cyclictest24249-21sh12:21:520
2741199250,24cyclictest21426-21diskmemload12:15:410
2741199250,24cyclictest21426-21diskmemload11:46:240
2741199250,24cyclictest0-21swapper/011:31:370
2741199250,24cyclictest0-21swapper/009:22:580
2741199250,1cyclictest6237-21if_enp1s011:01:360
2741199250,1cyclictest21426-21diskmemload11:21:560
2742299241,23cyclictest0-21swapper/311:48:423
2742299240,23cyclictest21426-21diskmemload09:23:093
2742299240,23cyclictest11036-21tune2fs11:06:323
2742299240,23cyclictest0-21swapper/312:25:143
2742299240,1cyclictest9978-21ssh12:08:233
2742299240,1cyclictest0-21swapper/309:16:383
27420992422,1cyclictest0-21swapper/209:15:182
2742099240,24cyclictest0-21swapper/211:39:282
2742099240,23cyclictest32481-21ssh11:58:492
2742099240,23cyclictest21426-21diskmemload10:43:392
2742099240,23cyclictest21426-21diskmemload10:43:392
2742099240,23cyclictest0-21swapper/210:21:242
2742099240,1cyclictest21426-21diskmemload10:06:072
27415992421,2cyclictest24236-21ptp4l-jitter09:11:381
27415992421,2cyclictest24236-21ptp4l-jitter09:11:381
27415992421,2cyclictest14071-21sh10:05:561
2741599241,2cyclictest12363-21/usr/sbin/munin10:36:411
2741599241,22cyclictest345-21chrt09:21:271
2741599240,23cyclictest0-21swapper/112:16:131
2741599240,23cyclictest0-21swapper/111:36:401
2741599240,23cyclictest0-21swapper/110:28:141
2741599240,0cyclictest21426-21diskmemload12:23:351
2741599240,0cyclictest21426-21diskmemload11:10:491
2741199243,21cyclictest62150irq/126-eno109:53:500
27411992423,1cyclictest21426-21diskmemload11:06:460
27411992422,1cyclictest62150irq/126-eno109:44:020
27411992421,2cyclictest23174-21cat09:11:310
27411992421,2cyclictest23174-21cat09:11:310
2741199240,24cyclictest0-21swapper/011:56:180
2741199240,24cyclictest0-21swapper/010:38:320
2741199240,23cyclictest17517-21ssh11:44:470
2741199240,1cyclictest0-21swapper/010:17:130
2741199240,0cyclictest0-21swapper/009:38:200
27422992323,0cyclictest21426-21diskmemload10:36:423
27422992323,0cyclictest21426-21diskmemload10:15:363
27422992323,0cyclictest0-21swapper/311:44:143
27422992323,0cyclictest0-21swapper/309:35:453
27422992323,0cyclictest0-21swapper/309:13:243
27422992323,0cyclictest0-21swapper/309:13:243
27422992322,1cyclictest8483-21ssh11:36:133
27422992322,1cyclictest21426-21diskmemload09:50:333
27422992321,1cyclictest309-21systemd-journal09:45:023
27422992321,1cyclictest29777-21perf07:15:003
27422992321,1cyclictest21426-21diskmemload10:51:303
27422992321,1cyclictest19064-21munin-run09:05:003
27422992321,1cyclictest11469-21cron08:50:003
2742299230,2cyclictest28916-21sh11:55:503
2742299230,23cyclictest0-21swapper/311:33:173
2742299230,23cyclictest0-21swapper/310:43:403
2742299230,23cyclictest0-21swapper/310:43:403
2742299230,23cyclictest0-21swapper/310:30:403
2742299230,22cyclictest7396-21ssh09:58:443
2742299230,22cyclictest32021-21rm12:30:203
2742299230,22cyclictest27851-21sh11:54:303
2742299230,22cyclictest2581-21ssh10:57:183
2742299230,22cyclictest25043-21ssh11:20:183
2742299230,22cyclictest16333-21ssh12:15:073
2742299230,22cyclictest13949-21sh09:33:313
2742299230,22cyclictest0-21swapper/312:36:313
2742299230,1cyclictest30740-21open_files10:21:383
2742299230,0cyclictest21426-21diskmemload12:23:413
2742299230,0cyclictest21426-21diskmemload09:26:553
2742299230,0cyclictest0-21swapper/312:03:013
2742299230,0cyclictest0-21swapper/310:45:343
27420992323,0cyclictest0-21swapper/211:42:252
27420992322,1cyclictest32280-21sh09:20:372
27420992320,2cyclictest4830-21ssh11:00:312
27420992320,2cyclictest18554-21ssh11:46:032
2742099230,23cyclictest0-21swapper/211:21:582
2742099230,23cyclictest0-21swapper/210:18:372
2742099230,22cyclictest7136-21sh09:26:392
2742099230,22cyclictest5436-21ssh11:31:422
2742099230,22cyclictest30505-21rm11:25:552
2742099230,22cyclictest2432-21ssh10:57:032
2742099230,22cyclictest22031-21ssh10:45:382
2742099230,22cyclictest21426-21diskmemload09:37:092
2742099230,22cyclictest13142-21ssh09:32:122
2742099230,22cyclictest11994-21ssh10:35:532
2742099230,0cyclictest21426-21diskmemload12:38:422
2742099230,0cyclictest21426-21diskmemload11:05:142
2742099230,0cyclictest0-21swapper/212:33:512
2742099230,0cyclictest0-21swapper/212:26:572
2742099230,0cyclictest0-21swapper/212:22:322
2742099230,0cyclictest0-21swapper/212:12:512
2742099230,0cyclictest0-21swapper/211:50:562
2742099230,0cyclictest0-21swapper/211:16:342
2742099230,0cyclictest0-21swapper/210:28:172
2742099230,0cyclictest0-21swapper/209:41:242
27415992323,0cyclictest21426-21diskmemload11:41:021
27415992323,0cyclictest0-21swapper/112:01:551
27415992323,0cyclictest0-21swapper/111:46:211
27415992323,0cyclictest0-21swapper/111:21:561
27415992323,0cyclictest0-21swapper/111:05:111
27415992323,0cyclictest0-21swapper/110:45:551
27415992323,0cyclictest0-21swapper/110:42:491
27415992323,0cyclictest0-21swapper/110:42:491
27415992323,0cyclictest0-21swapper/109:58:471
27415992323,0cyclictest0-21swapper/109:31:471
27415992321,1cyclictest0-21swapper/108:30:011
2741599230,23cyclictest21426-21diskmemload09:45:101
2741599230,23cyclictest0-21swapper/110:33:361
2741599230,22cyclictest28770-21ntp_states10:51:371
2741599230,22cyclictest20525-21sh11:16:141
2741599230,22cyclictest0-21swapper/112:35:271
2741599230,22cyclictest0-21swapper/112:27:361
2741599230,22cyclictest0-21swapper/110:11:301
2741599230,22cyclictest0-21swapper/109:25:221
2741599230,1cyclictest0-21swapper/112:10:481
2741599230,0cyclictest0-21swapper/111:00:491
2741599230,0cyclictest0-21swapper/110:15:421
2741599230,0cyclictest0-21swapper/110:00:071
2741599230,0cyclictest0-21swapper/109:35:191
2741599230,0cyclictest0-21swapper/109:17:571
27411992323,0cyclictest21426-21diskmemload11:35:450
27411992323,0cyclictest0-21swapper/011:28:510
27411992323,0cyclictest0-21swapper/010:49:010
27411992323,0cyclictest0-21swapper/010:12:440
27411992323,0cyclictest0-21swapper/007:31:330
27411992322,1cyclictest6959-21ssh12:06:170
27411992321,1cyclictest17352-21sed11:11:410
27411992321,1cyclictest16302-21perf09:00:000
27411992321,1cyclictest0-21swapper/010:56:560
27411992321,1cyclictest0-21swapper/009:30:370
27411992321,1cyclictest0-21swapper/009:15:220
27411992320,2cyclictest75-21kswapd011:50:010
2741199230,2cyclictest15033-21sh12:13:240
2741199230,22cyclictest5573-21ssh09:25:570
2741199230,22cyclictest27000-21sh12:25:240
2741199230,22cyclictest21426-21diskmemload12:31:040
2741199230,22cyclictest21426-21diskmemload12:00:360
2741199230,22cyclictest21426-21diskmemload11:18:300
2741199230,22cyclictest19260-21ssh10:41:400
2741199230,22cyclictest19260-21ssh10:41:400
2741199230,22cyclictest0-21swapper/009:00:020
2741199230,0cyclictest0-21swapper/010:00:280
27422992222,0cyclictest21426-21diskmemload11:12:553
27422992221,1cyclictest7736-21apt-get07:36:293
27422992221,1cyclictest7736-21apt-get07:36:293
27422992220,2cyclictest9085-21cron08:45:003
2742299220,22cyclictest21426-21diskmemload11:02:213
2742299220,21cyclictest21426-21diskmemload11:25:153
2742299220,21cyclictest21426-21diskmemload11:15:103
2742299220,21cyclictest21426-21diskmemload10:25:273
2742299220,21cyclictest21426-21diskmemload10:00:073
2742299220,21cyclictest18718-21sh10:10:183
2742299220,21cyclictest12300-21apt-get07:45:023
2742299220,1cyclictest20219-21sh09:40:053
2742299220,0cyclictest0-21swapper/312:11:403
27420992222,0cyclictest0-21swapper/210:30:012
27420992222,0cyclictest0-21swapper/209:55:552
27420992222,0cyclictest0-21swapper/209:47:092
27420992220,1cyclictest19060-21awk09:05:002
27420992220,1cyclictest12300-21apt-get07:45:012
2742099220,22cyclictest0-21swapper/210:51:052
2742099220,21cyclictest21426-21diskmemload12:05:422
2742099220,21cyclictest21426-21diskmemload09:50:142
2742099220,21cyclictest21305-21sh10:11:412
2742099220,21cyclictest0-21swapper/211:10:122
27415992222,0cyclictest0-21swapper/111:51:171
27415992222,0cyclictest0-21swapper/111:31:091
27415992222,0cyclictest0-21swapper/110:21:311
27415992221,1cyclictest0-21swapper/109:40:151
27415992220,1cyclictest7362-21turbostat07:35:011
27415992220,1cyclictest17431-21wc07:55:011
27415992219,2cyclictest16320-21munin-run09:00:001
2741599220,21cyclictest2587-21munin-plugin-st07:25:001
2741599220,21cyclictest0-21swapper/111:25:281
2741599220,21cyclictest0-21swapper/109:50:291
2741599220,21cyclictest0-21swapper/108:01:281
2741599220,1cyclictest0-21swapper/111:56:411
2741599220,1cyclictest0-21swapper/110:55:211
27411992222,0cyclictest0-21swapper/012:36:370
27411992222,0cyclictest0-21swapper/010:25:540
27411992222,0cyclictest0-21swapper/010:05:130
27411992220,1cyclictest28980-21sh10:20:320
27411992220,1cyclictest12262-21cron07:45:010
27411992220,1cyclictest11866-21apt-get08:51:300
2741199220,22cyclictest0-21swapper/009:57:530
2741199220,21cyclictest30480-21apt-get07:16:280
2741199220,21cyclictest0-21swapper/010:50:220
2741199220,21cyclictest0-21swapper/009:45:480
2741199220,1cyclictest9901-21sendmail-msp07:40:010
2741199220,1cyclictest9901-21sendmail-msp07:40:010
2741199220,0cyclictest0-21swapper/010:30:050
27422992121,0cyclictest0-21swapper/309:06:293
27422992120,1cyclictest5346-21apt-get07:31:333
2742299210,21cyclictest0-21swapper/307:15:023
2742099210,21cyclictest0-21swapper/212:15:462
27415992121,0cyclictest4659-21apt-get08:36:291
27415992120,1cyclictest24643-21perf08:10:001
27415992119,1cyclictest9884-21munin-run07:40:011
27415992119,1cyclictest9884-21munin-run07:40:011
2741599210,0cyclictest0-21swapper/108:51:301
27422992020,0cyclictest0-21swapper/308:56:293
27422992020,0cyclictest0-21swapper/308:31:303
27422992019,1cyclictest22610-21apt-get08:06:303
27422992018,1cyclictest0-21swapper/308:40:013
2742299200,0cyclictest441-21apt-get07:21:293
27420992019,0cyclictest0-21swapper/207:31:332
27420992018,1cyclictest27018-21latency_hist08:15:002
27420992017,2cyclictest597-21dbus-daemon08:45:002
27415992020,0cyclictest0-21swapper/109:01:291
27415992020,0cyclictest0-21swapper/108:46:291
27415992019,0cyclictest29830-21apt-get08:21:321
27415992019,0cyclictest27746-21apt-get07:11:281
2741599200,0cyclictest0-21swapper/109:06:291
2741599200,0cyclictest0-21swapper/107:41:331
27411992020,0cyclictest0-21swapper/008:01:290
27411992020,0cyclictest0-21swapper/007:21:290
27420991919,0cyclictest0-21swapper/208:56:292
27420991919,0cyclictest0-21swapper/207:16:282
27415991919,0cyclictest7086-21apt-get08:41:371
27415991919,0cyclictest0-21swapper/108:16:301
27411991919,0cyclictest0-21swapper/008:41:370
27411991919,0cyclictest0-21swapper/007:46:330
27422991818,0cyclictest0-21swapper/307:56:283
27420991818,0cyclictest0-21swapper/209:06:282
27420991818,0cyclictest0-21swapper/208:31:302
27420991818,0cyclictest0-21swapper/208:06:302
27420991818,0cyclictest0-21swapper/207:21:292
27415991818,0cyclictest0-21swapper/107:26:321
27411991818,0cyclictest2264-21apt-get08:31:300
27411991818,0cyclictest0-21swapper/008:26:370
27411991818,0cyclictest0-21swapper/007:51:300
27411991817,0cyclictest0-21swapper/008:11:290
2741199180,17cyclictest9472-21apt-get08:46:290
27422991717,0cyclictest2945-21apt-get07:26:323
27422991717,0cyclictest0-21swapper/307:51:293
27422991710,6cyclictest1718-21sendmail-mta07:42:033
27420991717,0cyclictest0-21swapper/207:56:282
27420991717,0cyclictest0-21swapper/207:46:332
27415991716,1cyclictest0-21swapper/108:31:301
27415991715,1cyclictest19286-21timerandwakeup07:56:371
27411991717,0cyclictest0-21swapper/008:36:290
27411991716,0cyclictest19421-21apt-get09:06:290
27422991616,0cyclictest0-21swapper/308:11:293
2742299160,15cyclictest32230-21apt-get08:26:373
27420991616,0cyclictest0-21swapper/208:01:292
27420991616,0cyclictest0-21swapper/207:11:292
27420991615,1cyclictest28-21ksoftirqd/208:36:292
27420991615,1cyclictest0-21swapper/208:50:002
2742099160,15cyclictest0-21swapper/208:55:002
27411991616,0cyclictest0-21swapper/008:21:320
27422991515,0cyclictest0-21swapper/308:16:303
27422991513,1cyclictest297312sleep308:20:403
27420991515,0cyclictest27403-21apt-get08:16:302
27420991515,0cyclictest0-21swapper/208:26:362
27420991515,0cyclictest0-21swapper/207:26:322
2741599150,1cyclictest0-21swapper/108:11:331
27411991515,0cyclictest0-21swapper/007:56:280
2742299140,14cyclictest0-21swapper/308:51:353
2742299140,14cyclictest0-21swapper/308:00:223
2742099144,9cyclictest28-21ksoftirqd/208:21:322
2742099140,14cyclictest0-21swapper/207:36:102
2742099140,14cyclictest0-21swapper/207:36:102
27415991414,0cyclictest0-21swapper/107:46:331
27415991414,0cyclictest0-21swapper/107:15:481
27411991414,0cyclictest0-21swapper/008:15:310
27411991414,0cyclictest0-21swapper/008:05:020
2741199140,0cyclictest0-21swapper/007:11:510
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional