You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-27 - 05:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sat Apr 27, 2024 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17799992525,0cyclictest0-21swapper/323:04:453
1779999250,0cyclictest0-21swapper/321:39:113
1779399251,1cyclictest173272sleep221:15:282
1779399250,24cyclictest31543-21sh22:35:002
17790992523,2cyclictest7239-21latency23:17:561
1779099250,1cyclictest0-21swapper/119:20:011
17785992523,1cyclictest30171-21perf19:35:000
1779999240,2cyclictest13731-21ssh21:42:543
1779999240,23cyclictest11478-21diskmemload21:32:323
1779999240,23cyclictest11478-21diskmemload21:21:003
1779999240,23cyclictest0-21swapper/322:45:333
1779999240,0cyclictest0-21swapper/300:24:193
1779399240,23cyclictest7422-21ssh00:27:492
1779399240,23cyclictest17558-21ssh21:47:042
1779399240,23cyclictest0-21swapper/222:21:002
1779399240,23cyclictest0-21swapper/222:10:192
1779399240,1cyclictest11478-21diskmemload22:50:122
1779399240,0cyclictest0-21swapper/200:15:122
1779099240,24cyclictest0-21swapper/122:18:491
1779099240,23cyclictest5297-21apt-get22:07:491
1779099240,23cyclictest27132-21sh22:30:031
1779099240,23cyclictest11600-21ssh21:40:461
17785992424,0cyclictest0-21swapper/023:35:290
17785992423,1cyclictest0-21swapper/021:50:120
17785992422,1cyclictest603-21in:imuxsock21:55:540
1778599240,23cyclictest27588-21rm23:04:570
1778599240,0cyclictest11478-21diskmemload00:12:450
17799992323,0cyclictest0-21swapper/322:52:423
17799992323,0cyclictest0-21swapper/322:01:403
17799992323,0cyclictest0-21swapper/321:56:573
17799992323,0cyclictest0-21swapper/321:10:383
17799992323,0cyclictest0-21swapper/300:38:013
17799992321,1cyclictest23617-21cron23:35:003
17799992321,1cyclictest0-21swapper/319:50:003
1779999231,22cyclictest34-21ksoftirqd/323:27:183
1779999230,22cyclictest14613-21turbostat20:09:593
1779999230,22cyclictest12580-21sh22:14:303
1779999230,22cyclictest11478-21diskmemload22:23:463
1779999230,22cyclictest0-21swapper/322:40:003
1779999230,0cyclictest0-21swapper/321:50:013
1779999230,0cyclictest0-21swapper/321:25:313
17793992323,0cyclictest0-21swapper/221:40:222
17793992322,1cyclictest11478-21diskmemload22:35:152
17793992321,1cyclictest597-21dbus-daemon21:10:002
17793992321,1cyclictest11628-21apt-get23:57:502
17793992321,1cyclictest0-21swapper/200:00:152
17793992321,0cyclictest6965-21apt-get23:52:482
17793992321,0cyclictest6965-21apt-get23:52:482
17793992320,2cyclictest27907-21ssh21:25:332
1779399230,23cyclictest0-21swapper/223:13:472
1779399230,23cyclictest0-21swapper/223:03:232
1779399230,23cyclictest0-21swapper/221:50:162
1779399230,23cyclictest0-21swapper/221:20:122
1779399230,23cyclictest0-21swapper/200:36:242
1779399230,22cyclictest8222-21latency_hist22:10:002
1779399230,22cyclictest5839-21rm23:16:582
1779399230,22cyclictest20152-21perf19:15:002
1779399230,22cyclictest11478-21diskmemload22:26:332
1779399230,0cyclictest0-21swapper/221:33:312
1779399230,0cyclictest0-21swapper/219:42:482
17790992323,0cyclictest0-21swapper/123:44:361
17790992323,0cyclictest0-21swapper/122:36:161
17790992323,0cyclictest0-21swapper/122:25:461
17790992323,0cyclictest0-21swapper/121:36:431
17790992323,0cyclictest0-21swapper/121:18:261
17790992323,0cyclictest0-21swapper/100:32:161
17790992322,1cyclictest11478-21diskmemload23:31:501
17790992322,0cyclictest31709-21ssh22:00:241
17790992321,1cyclictest32459-21rm21:29:501
17790992321,1cyclictest21832-21apt-get20:25:011
17790992321,1cyclictest0-21swapper/121:20:331
17790992321,1cyclictest0-21swapper/121:12:501
1779099230,2cyclictest23063-21rm21:52:361
1779099230,2cyclictest1853-21rm21:31:381
1779099230,22cyclictest31737-21ssh23:09:021
1779099230,22cyclictest30632-21apt-get19:37:471
1779099230,22cyclictest29935-21apt-get20:42:501
1779099230,22cyclictest21305-21rm22:23:221
1779099230,22cyclictest11478-21diskmemload00:17:591
1779099230,1cyclictest13244-21apt-get22:50:011
1779099230,0cyclictest0-21swapper/123:27:241
1779099230,0cyclictest0-21swapper/122:14:321
17785992323,0cyclictest11478-21diskmemload22:44:130
17785992323,0cyclictest0-21swapper/022:07:030
17785992323,0cyclictest0-21swapper/021:40:460
17785992320,2cyclictest4411-21sh23:14:460
17785992320,2cyclictest32142-21ssh23:09:420
1778599231,22cyclictest62150irq/126-eno121:28:080
1778599230,23cyclictest11478-21diskmemload23:22:110
1778599230,23cyclictest0-21swapper/019:12:490
1778599230,23cyclictest0-21swapper/000:00:060
1778599230,22cyclictest6462-21rm21:35:480
1778599230,22cyclictest3555-21latency_hist22:05:000
1778599230,22cyclictest32586-21cron19:40:000
1778599230,22cyclictest29569-21ssh00:15:540
1778599230,22cyclictest2408-21diskmemload00:22:070
1778599230,22cyclictest19035-21rm22:55:570
1778599230,22cyclictest18169-21ssh22:54:480
1778599230,22cyclictest11996-21ssh21:10:240
1778599230,22cyclictest1022-21ssh23:45:270
1778599230,22cyclictest0-21swapper/021:31:290
1778599230,1cyclictest20615-21apt-get19:17:460
1778599230,1cyclictest11478-21diskmemload00:09:090
1778599230,0cyclictest23675-21kworker/u8:023:19:040
1778599230,0cyclictest11478-21diskmemload21:49:390
1778599230,0cyclictest0-21swapper/000:36:400
17799992222,0cyclictest11478-21diskmemload22:33:093
17799992222,0cyclictest0-21swapper/323:55:393
17799992222,0cyclictest0-21swapper/323:50:263
17799992222,0cyclictest0-21swapper/323:50:263
17799992222,0cyclictest0-21swapper/323:37:193
17799992222,0cyclictest0-21swapper/323:19:043
17799992222,0cyclictest0-21swapper/323:10:423
17799992222,0cyclictest0-21swapper/322:57:063
17799992222,0cyclictest0-21swapper/322:15:433
17799992221,1cyclictest16309-21ssh21:45:053
17799992220,1cyclictest19282-21munin-run00:05:013
17799992220,1cyclictest0-21swapper/323:45:513
17799992220,1cyclictest0-21swapper/300:32:483
1779999221,0cyclictest34-21ksoftirqd/300:19:593
1779999220,2cyclictest10314-21awk00:30:003
1779999220,22cyclictest11478-21diskmemload22:41:163
1779999220,22cyclictest0-21swapper/322:27:093
1779999220,21cyclictest30179-21basename19:35:003
1779999220,21cyclictest0-21swapper/323:07:033
1779999220,21cyclictest0-21swapper/320:42:503
1779999220,21cyclictest0-21swapper/319:12:493
1779999220,21cyclictest0-21swapper/300:12:503
1779999220,1cyclictest0-21swapper/321:17:473
1779999220,0cyclictest0-21swapper/320:32:453
1779999220,0cyclictest0-21swapper/300:05:223
17793992222,0cyclictest11478-21diskmemload23:06:252
17793992222,0cyclictest0-21swapper/223:49:502
17793992222,0cyclictest0-21swapper/223:42:332
17793992222,0cyclictest0-21swapper/221:36:392
17793992220,1cyclictest3539-21cron22:05:002
17793992220,1cyclictest27501-21apt-get20:37:452
17793992220,1cyclictest24622-21cron20:30:002
17793992220,1cyclictest24607-21taskset23:36:282
17793992220,1cyclictest19545-21ssh23:30:402
17793992220,1cyclictest13661-21ssh22:16:022
17793992220,1cyclictest0-21swapper/219:30:012
1779399221,1cyclictest22543-21idleruntime-cro19:20:002
1779399220,22cyclictest0-21swapper/200:06:312
1779399220,21cyclictest0-21swapper/223:21:372
1779399220,21cyclictest0-21swapper/222:41:552
1779399220,21cyclictest0-21swapper/221:55:112
1779399220,21cyclictest0-21swapper/221:10:222
1779399220,21cyclictest0-21swapper/219:50:002
1779399220,1cyclictest0-21swapper/223:26:002
1779399220,1cyclictest0-21swapper/222:45:472
1779399220,1cyclictest0-21swapper/200:10:412
17790992222,0cyclictest11478-21diskmemload00:35:381
17790992222,0cyclictest0-21swapper/123:37:491
17790992222,0cyclictest0-21swapper/123:21:151
17790992222,0cyclictest0-21swapper/122:58:331
17790992222,0cyclictest0-21swapper/122:46:181
17790992222,0cyclictest0-21swapper/100:12:551
17790992221,1cyclictest1339-21ssh00:20:061
17790992221,0cyclictest15089-21apt-get20:12:471
17790992220,1cyclictest1898-21perf20:50:001
17790992218,3cyclictest1-21systemd00:00:001
1779099220,21cyclictest4134-21ssh22:40:021
1779099220,21cyclictest11478-21diskmemload00:05:261
1779099220,21cyclictest1091-21sh23:45:351
1779099220,21cyclictest0-21swapper/119:32:461
1779099220,1cyclictest6797-21ssh00:26:411
1779099220,1cyclictest5836-21diskmemload23:50:431
1779099220,1cyclictest5836-21diskmemload23:50:431
1779099220,1cyclictest19914-21apt-get20:22:461
1779099220,1cyclictest15905-21sh00:02:021
1779099220,1cyclictest12693-21apt-get20:07:461
1779099220,0cyclictest7883-21apt-get19:57:481
17785992222,0cyclictest0-21swapper/021:21:570
17785992221,1cyclictest0-21swapper/023:42:060
17785992220,2cyclictest0-21swapper/021:19:150
17785992220,1cyclictest228332sleep022:25:440
1778599220,22cyclictest0-21swapper/023:59:040
1778599220,22cyclictest0-21swapper/022:15:110
1778599220,22cyclictest0-21swapper/022:12:340
1778599220,22cyclictest0-21swapper/000:33:190
1778599220,21cyclictest22459-21apt-get20:25:010
1778599220,21cyclictest19235-21rm23:30:070
1778599220,1cyclictest0-21swapper/023:52:400
1778599220,1cyclictest0-21swapper/023:52:400
1778599220,0cyclictest0-21swapper/022:35:560
1778599220,0cyclictest0-21swapper/022:20:470
17799992121,0cyclictest11478-21diskmemload22:05:023
17799992121,0cyclictest0-21swapper/323:22:013
1779999210,21cyclictest331rcuc/323:40:083
1779999210,21cyclictest0-21swapper/320:22:463
1779999210,20cyclictest0-21swapper/321:02:463
1779999210,0cyclictest0-21swapper/320:27:493
17793992121,0cyclictest0-21swapper/222:56:252
1779399210,20cyclictest0-21swapper/221:05:002
1779399210,20cyclictest0-21swapper/219:22:482
1779399210,1cyclictest17008-21dump-pmu-power20:15:002
1779399210,0cyclictest0-21swapper/220:47:462
1779399210,0cyclictest0-21swapper/200:20:212
17790992118,2cyclictest6694-21apt20:59:591
1779099210,21cyclictest0-21swapper/121:55:041
1779099210,20cyclictest3087-21apt-get19:47:461
1779099210,20cyclictest18220-21apt-get19:12:491
1779099210,20cyclictest0-21swapper/121:07:471
1779099210,1cyclictest2362-21apt-get20:52:461
1779099210,1cyclictest23300-21ssh23:00:111
1779099210,1cyclictest12235-21perf20:05:011
17785992121,0cyclictest0-21swapper/023:25:430
17785992121,0cyclictest0-21swapper/020:57:460
17785992121,0cyclictest0-21swapper/019:45:000
17785992119,1cyclictest19451-21sendmail-msp20:20:000
17785992119,1cyclictest14619-21perf20:09:590
1778599210,21cyclictest0-21swapper/022:48:430
1778599210,21cyclictest0-21swapper/000:26:040
1778599210,20cyclictest0-21swapper/020:37:450
1778599210,1cyclictest0-21swapper/022:32:510
17799992020,0cyclictest0-21swapper/320:37:473
17799992019,1cyclictest0-21swapper/320:12:473
17799992018,1cyclictest22560-21perf19:20:003
1779999200,19cyclictest0-21swapper/320:47:463
1779999200,0cyclictest0-21swapper/319:52:473
17793992019,1cyclictest25103-21apt-get20:32:452
17793992019,0cyclictest0-21swapper/220:07:462
17793992018,1cyclictest27346-21turbostat19:30:002
17785992020,0cyclictest0-21swapper/021:02:460
17799991919,0cyclictest0-21swapper/319:37:483
17799991919,0cyclictest0-21swapper/319:22:493
17799991917,1cyclictest6663-21cron20:59:593
17790991919,0cyclictest0-21swapper/120:37:451
17785991919,0cyclictest0-21swapper/020:12:470
17785991919,0cyclictest0-21swapper/019:55:010
17785991916,2cyclictest21853-21perf20:25:000
17799991818,0cyclictest0-21swapper/320:16:243
1779999180,17cyclictest0-21swapper/319:57:493
17793991818,0cyclictest4760-21apt-get20:57:452
17793991818,0cyclictest0-21swapper/220:42:502
17790991817,1cyclictest25425-21apt-get19:27:461
17790991816,1cyclictest9102-21turbostat.cron21:05:011
17799991717,0cyclictest0-21swapper/321:07:473
17799991716,1cyclictest660-21apt-get19:42:483
17793991717,0cyclictest0-21swapper/220:22:462
17790991717,0cyclictest0-21swapper/119:42:491
17785991717,0cyclictest0-21swapper/020:32:450
17799991614,1cyclictest4353-21munin-run20:55:003
17799991613,2cyclictest0-21swapper/320:02:473
17799991613,2cyclictest0-21swapper/319:28:033
17793991616,0cyclictest0-21swapper/219:50:432
17790991616,0cyclictest0-21swapper/119:52:471
17785991616,0cyclictest0-21swapper/020:47:460
1778599160,15cyclictest0-21swapper/020:05:000
17793991515,0cyclictest0-21swapper/219:57:492
17793991513,1cyclictest0-21swapper/220:17:562
1779399150,1cyclictest10483-21expr20:02:472
17785991515,0cyclictest0-21swapper/019:57:480
17785991514,0cyclictest0-21swapper/020:52:470
1778599150,1cyclictest0-21swapper/019:47:530
1779399140,14cyclictest0-21swapper/220:50:322
1779399140,14cyclictest0-21swapper/219:35:292
17790991414,0cyclictest0-21swapper/120:30:181
17790991414,0cyclictest0-21swapper/120:15:181
1779099140,14cyclictest0-21swapper/119:20:441
17785991414,0cyclictest0-21swapper/019:20:160
1778599140,14cyclictest0-21swapper/021:05:040
1778599140,14cyclictest0-21swapper/020:41:110
1778599140,0cyclictest0-21swapper/019:26:030
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional