You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-16 - 14:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 16, 2024 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25231210288,9sleep10-21swapper/119:06:391
2538629986,9sleep30-21swapper/319:08:383
2313028066,9sleep00-21swapper/019:05:010
2548327857,6sleep20-21swapper/219:09:542
2558699310,30cyclictest0-21swapper/122:31:101
2558399290,25cyclictest0-21swapper/023:34:090
2558699280,27cyclictest0-21swapper/100:23:521
2558899270,27cyclictest0-21swapper/221:58:132
2558399270,26cyclictest0-21swapper/022:25:100
25588992424,0cyclictest0-21swapper/221:28:132
2558899240,18cyclictest15809-21kworker/2:121:05:132
2558699231,2cyclictest0-21swapper/121:35:131
21852231,1sleep30-21swapper/319:25:373
2558899221,20cyclictest0-21swapper/221:34:232
2558899210,20cyclictest0-21swapper/222:09:502
2558699210,20cyclictest0-21swapper/123:47:351
2558899200,20cyclictest31366-21kworker/2:123:37:002
2558699200,19cyclictest0-21swapper/122:41:531
2558899192,17cyclictest0-21swapper/223:19:122
2558899192,16cyclictest0-21swapper/223:00:472
25588991918,1cyclictest0-21swapper/200:10:392
2558899191,18cyclictest0-21swapper/222:37:342
2558899190,0cyclictest0-21swapper/223:34:092
2558399190,2cyclictest0-21swapper/023:21:090
25592991817,1cyclictest0-21swapper/300:09:493
2558899180,1cyclictest0-21swapper/222:53:032
2558699188,3cyclictest25-21ksoftirqd/100:10:111
2558699180,16cyclictest0-21swapper/100:26:461
25583991812,5cyclictest13-21ksoftirqd/023:29:450
2558399180,17cyclictest0-21swapper/021:21:420
321332170,0sleep00-21swapper/022:02:480
25592991710,3cyclictest224-21systemd-journal20:07:183
2559299170,17cyclictest0-21swapper/322:21:303
2558899171,0cyclictest0-21swapper/222:10:532
2558699179,3cyclictest9432-21tail19:40:371
2558699179,2cyclictest8051-21ssh23:15:181
2558699179,2cyclictest29728-21grep22:00:371
2558699179,2cyclictest14326-21grep19:50:341
25586991710,2cyclictest32448-21sed21:40:331
25586991710,2cyclictest18276-21grep23:00:171
2558699170,3cyclictest0-21swapper/121:56:071
2558399171,15cyclictest0-21swapper/020:45:130
2558399170,2cyclictest0-21swapper/022:19:440
25592991615,1cyclictest0-21swapper/321:57:493
25588991615,1cyclictest24717-21kworker/2:121:43:192
25588991615,1cyclictest11514-21kworker/2:121:54:392
25588991615,1cyclictest0-21swapper/223:54:242
25588991615,1cyclictest0-21swapper/200:25:272
2558899160,16cyclictest32562-21kworker/2:219:54:262
2558899160,14cyclictest0-21swapper/200:21:582
2558699169,2cyclictest27244-21sed23:05:391
2558699169,2cyclictest21075-21cut20:05:331
25586991615,1cyclictest0-21swapper/121:16:361
25586991613,3cyclictest0-21swapper/121:23:341
2558399163,7cyclictest2162-21sendmail-msp00:20:010
2558399163,12cyclictest17482-21apt-get20:00:170
2558399162,12cyclictest1291-21runrttasks23:39:210
2558399162,10cyclictest24223-21ssh21:35:260
215782160,0sleep10-21swapper/123:24:591
25592991510,3cyclictest18561-21grep23:45:183
2558899152,13cyclictest0-21swapper/223:41:012
2558899152,12cyclictest20547-21ssh00:33:332
2558899151,4cyclictest0-21swapper/221:15:062
2558699159,4cyclictest27010-21apt-get20:20:141
2558699159,1cyclictest16987-21head21:05:201
2558699159,1cyclictest15437-21grep00:05:221
2558699158,2cyclictest595-21apache221:27:051
2558699158,2cyclictest2448-21sort20:35:231
2558699157,3cyclictest27817-21grep23:50:371
2558399159,4cyclictest13158-21cron22:35:010
2558399153,9cyclictest4409-21munin-node23:14:060
25583991513,2cyclictest595-21apache221:59:130
25583991513,2cyclictest13-21ksoftirqd/019:25:360
2559299149,3cyclictest23685-21ssh00:35:333
25592991413,1cyclictest0-21swapper/321:52:103
25592991412,1cyclictest0-21swapper/323:10:153
25592991410,3cyclictest5090-21cat21:45:013
25592991410,3cyclictest11421-21tr20:55:163
2559299140,9cyclictest0-21swapper/321:11:533
2558899142,2cyclictest18381-21diskmemload23:46:002
25588991413,1cyclictest32-21ksoftirqd/221:39:482
25588991413,1cyclictest18381-21diskmemload21:21:162
2558899140,4cyclictest0-21swapper/223:10:482
2558899140,2cyclictest0-21swapper/223:57:072
2558899140,2cyclictest0-21swapper/223:21:102
2558899140,1cyclictest0-21swapper/222:19:452
2558699147,1cyclictest14134-21awk00:05:011
25586991410,3cyclictest6255-21tune2fs21:45:191
25586991410,3cyclictest23098-21tail00:35:231
274192130,1sleep00-21swapper/022:21:290
2559299139,3cyclictest17264-21grep23:00:013
25592991310,2cyclictest22249-21head00:10:203
25592991310,2cyclictest16273-21cut21:30:193
25592991310,2cyclictest14823-21grep19:55:003
2558899132,11cyclictest3855-21kworker/2:220:42:222
25588991313,0cyclictest0-21swapper/219:28:222
25588991312,1cyclictest0-21swapper/222:21:582
25588991311,2cyclictest27589-21kworker/2:022:02:192
25588991310,2cyclictest23157-21grep00:35:242
25588991310,2cyclictest19785-31su00:08:552
2558899130,1cyclictest15787-21/usr/sbin/munin19:55:222
2558899130,12cyclictest6508-21cat20:45:102
2558899130,12cyclictest18381-21diskmemload22:57:072
25586991310,2cyclictest640-21grep23:10:191
25586991310,2cyclictest371-21grep22:25:231
25586991310,2cyclictest12037-21sed20:55:231
2558699130,12cyclictest30583-21wc23:30:281
25583991312,1cyclictest0-21swapper/022:36:520
2558399131,11cyclictest18256-21chrt21:08:420
25583991310,2cyclictest18919-21cut21:10:150
2559299129,2cyclictest9699-21head20:50:243
2559299129,2cyclictest8430-21cut19:40:213
2559299129,2cyclictest6596-21tr19:35:283
2559299129,2cyclictest4802-21ssh22:05:483
2559299129,2cyclictest31363-21grep19:20:193
2559299129,2cyclictest28116-21wc19:15:003
2559299129,2cyclictest27392-21ssh23:50:293
2559299129,2cyclictest23258-21expr21:35:143
2559299129,2cyclictest23005-21tail20:10:203
2559299129,2cyclictest18103-21cut20:00:263
2559299129,2cyclictest11564-21ls19:45:343
2559299129,2cyclictest10566-21tr00:25:233
2559299128,3cyclictest7532-21cut23:15:153
2559299120,3cyclictest0-21swapper/321:45:123
2559299120,11cyclictest10326-21sed21:25:403
2558899129,2cyclictest367-21grep22:25:242
2558899129,2cyclictest30203-21cat00:15:442
2558899129,2cyclictest29867-21grep19:15:422
2558899129,2cyclictest26129-21cat20:15:372
2558899129,2cyclictest21330-21grep22:40:182
2558899129,2cyclictest20599-21grep20:05:222
2558899127,4cyclictest32576-21basename20:30:222
2558899122,4cyclictest15762-21sshd21:02:082
25588991210,1cyclictest12028-21head20:55:232
2558899120,11cyclictest31516-21cat20:29:592
2558699129,2cyclictest30096-21grep20:25:201
2558699129,2cyclictest29695-21ssh00:15:351
2558699129,2cyclictest2455-21wc22:50:001
2558699128,3cyclictest5898-21tail23:35:361
2558699128,3cyclictest23602-21ssh23:25:291
2558699128,2cyclictest23915-21grep20:10:341
2558699123,6cyclictest0-21swapper/122:20:001
2558699121,10cyclictest0-21swapper/122:24:321
2558399129,2cyclictest6682-21fgrep19:35:270
2558399129,2cyclictest6439-21ssh21:45:210
2558399129,2cyclictest5366-21grep20:40:320
2558399129,2cyclictest3409-21cut00:20:180
2558399129,2cyclictest3400-21grep20:35:380
2558399129,2cyclictest23177-21cut20:10:220
2558399129,2cyclictest19502-21wc23:00:350
2558399129,2cyclictest14243-21perf23:19:590
2558399128,3cyclictest7196-21ssh22:07:570
2558399128,3cyclictest24699-21rm00:11:510
2558399128,3cyclictest19885-21sh21:32:380
2558399128,3cyclictest18227-21ssh23:45:050
2558399128,3cyclictest17208-21ntpq00:30:250
2558399128,3cyclictest12099-21ssh22:55:320
2558399128,3cyclictest11435-21ssh21:26:380
2558399122,9cyclictest11277-21apache_processe20:55:120
25583991210,2cyclictest13-21ksoftirqd/023:53:330
2558399120,11cyclictest28075-21sed20:20:240
2559299118,2cyclictest8711-21grep00:00:183
2559299118,2cyclictest8167-21grep20:45:363
2559299118,2cyclictest6018-21ssh00:22:023
2559299118,2cyclictest5464-21ssh22:51:093
2559299118,2cyclictest4366-21cat23:35:173
2559299118,2cyclictest4084-21cat20:40:003
2559299118,2cyclictest32118-21ssh22:02:463
2559299118,2cyclictest31639-21cut22:25:123
2559299118,2cyclictest31487-21ssh21:18:513
2559299118,2cyclictest31153-21taskset20:25:533
2559299118,2cyclictest29760-21ssh22:45:323
2559299118,2cyclictest28859-21ssh00:15:193
2559299118,2cyclictest26059-21head23:05:243
2559299118,2cyclictest24170-21ssh23:25:453
2559299118,2cyclictest2403-21ssh23:34:013
2559299118,2cyclictest23287-21ssh22:40:453
2559299118,2cyclictest18950-21rm22:15:393
2559299118,2cyclictest17701-21sh23:00:113
2559299118,2cyclictest17210-21ssh00:30:253
2559299118,2cyclictest1712-21tr21:20:203
2559299118,2cyclictest15824-21grep19:55:223
2559299118,2cyclictest15659-21cat22:35:363
2559299118,2cyclictest13317-21ssh22:11:553
2559299117,3cyclictest1378-21ntpd20:24:583
2559299111,6cyclictest11060-21/usr/sbin/munin23:40:243
2559299110,10cyclictest20458-21ssh23:24:083
2558899119,2cyclictest27965-21rm22:45:062
2558899119,1cyclictest30818-21cut19:20:152
2558899119,1cyclictest27605-21grep20:20:182
2558899118,3cyclictest9216-21taskset22:31:202
2558899118,3cyclictest25774-21taskset23:05:212
2558899118,2cyclictest3444-21cat19:30:152
2558899118,2cyclictest26108-21tr19:10:172
2558899118,2cyclictest24652-21ssh23:26:132
2558899118,2cyclictest20836-21sort21:10:412
2558899118,2cyclictest19244-21taskset20:00:502
2558899118,2cyclictest11082-21ssh21:48:452
2558899118,2cyclictest10228-21mktemp19:45:012
25588991110,1cyclictest18381-21diskmemload00:03:282
2558699119,2cyclictest0-21swapper/123:57:371
2558699118,2cyclictest7046-21sed19:35:331
2558699118,2cyclictest4375-21ssh22:50:331
2558699118,2cyclictest30864-21tr19:20:151
2558699118,2cyclictest2820-21wc19:30:011
2558699118,2cyclictest20772-21ssh00:33:471
2558699118,2cyclictest18986-21cut21:10:161
2558699118,2cyclictest14135-21ssh22:57:111
2558699118,2cyclictest13616-21ssh22:35:021
2558699118,2cyclictest12901-21ssh22:11:311
2558699118,2cyclictest12072-21head19:45:411
2558699115,2cyclictest224-21systemd-journal19:35:001
2558699114,3cyclictest224-21systemd-journal22:05:541
2558699111,9cyclictest0-21swapper/121:34:101
2558699110,10cyclictest17443-21chrt20:00:121
2558699110,10cyclictest14090-21cut21:50:231
2558699110,10cyclictest11955-21tail23:40:191
2558399118,2cyclictest9105-21grep00:00:240
2558399118,2cyclictest9088-21ssh00:25:030
2558399118,2cyclictest6745-21sh22:52:240
2558399118,2cyclictest29431-21cut20:25:140
2558399118,2cyclictest29047-21ssh22:45:210
2558399118,2cyclictest27318-21cat21:15:300
2558399118,2cyclictest26542-21rm23:05:300
2558399118,2cyclictest25267-21grep20:15:220
2558399118,2cyclictest23755-21grep00:35:320
2558399118,2cyclictest21791-21chrt20:08:000
2558399118,2cyclictest18997-21rm21:54:150
2558399118,2cyclictest18935-21sh00:08:080
2558399118,2cyclictest16190-21grep19:55:270
2558399118,2cyclictest12525-21sh23:40:260
2558399117,3cyclictest300-21sh23:55:020
25583991111,0cyclictest0-21swapper/022:10:530
2558399110,4cyclictest0-21swapper/021:40:130
2558399110,10cyclictest32071-21cat20:30:160
2558399110,10cyclictest22284-21cat22:40:280
2559299108,1cyclictest6662-21expr22:30:203
2559299108,1cyclictest4127-21ntpq19:30:273
2559299108,1cyclictest247232sleep320:15:163
2559299108,1cyclictest13897-21aten2_r7power_c21:00:143
2559299107,2cyclictest754-21systemctl20:30:313
2559299107,2cyclictest5821-21tail20:40:393
2559299107,2cyclictest2267-21ssh23:55:313
2559299107,2cyclictest16389-21basename21:05:143
2558899109,1cyclictest8913-21sh19:40:282
2558899109,1cyclictest2645-21if_enp2s020:35:242
2558899109,1cyclictest22898-21fschecks_time20:10:202
2558899108,2cyclictest9310-21expr20:50:192
2558899102,2cyclictest1-21systemd19:37:452
2558699108,1cyclictest15047-21taskset19:55:101
2558699108,1cyclictest14183-21cat21:00:161
2558699108,1cyclictest10234-21sendmail_mailqu20:50:341
2558699107,2cyclictest5371-21cut20:40:321
2558699107,2cyclictest28229-21grep19:15:151
2558699107,2cyclictest27738-21taskset19:12:461
2558699107,2cyclictest26450-21taskset20:16:451
2558699107,2cyclictest1029-21grep20:30:371
2558399108,1cyclictest4297-21grep19:30:310
2558399108,1cyclictest15606-21grep21:00:340
2558399107,2cyclictest9780-21chrt19:44:130
2558399107,2cyclictest9358-21expr20:50:190
2558399107,2cyclictest32723-21sleep19:25:010
2558399107,2cyclictest28189-21apache_accesses19:15:110
2558399107,2cyclictest13427-21expr19:50:200
2558399107,2cyclictest11539-21pluginstate19:45:340
2558399103,3cyclictest224-21systemd-journal19:10:350
255929992,3cyclictest224-21systemd-journal19:15:373
255869996,2cyclictest7080-21df20:45:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional