You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-19 - 22:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Sep 19, 2024 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13095210087,9sleep30-21swapper/307:05:573
1334129985,9sleep10-21swapper/107:09:071
1311729265,8sleep00-21swapper/007:06:140
1324027868,6sleep20-21swapper/207:07:472
13495992119,1cyclictest0-21swapper/210:41:302
1349199210,20cyclictest0-21swapper/111:45:311
13486992113,6cyclictest13227-21ls10:10:220
1349599202,14cyclictest25225-21/usr/sbin/munin12:25:182
1349599200,19cyclictest0-21swapper/211:28:532
166072190,0sleep00-21swapper/011:59:410
1349599190,19cyclictest0-21swapper/210:36:312
13486991916,1cyclictest26313-21/usr/sbin/munin09:55:140
13486991818,0cyclictest0-21swapper/012:33:460
13491991716,1cyclictest0-21swapper/110:51:071
1350099163,8cyclictest6520-21diskmemload11:55:543
13500991615,1cyclictest0-21swapper/310:22:173
1349199167,5cyclictest0-21swapper/112:33:381
13491991616,0cyclictest0-21swapper/109:51:061
13491991616,0cyclictest0-21swapper/109:51:061
13500991514,1cyclictest0-21swapper/312:01:383
13500991511,3cyclictest15405-21grep09:45:263
13500991510,3cyclictest28700-21ssh11:44:553
1349599154,10cyclictest4532-21ssh09:36:252
13495991515,0cyclictest1346-21kworker/2:210:59:512
13495991510,2cyclictest0-21swapper/211:32:472
13491991514,1cyclictest0-21swapper/110:14:481
13500991413,1cyclictest61950irq/127-enp2s0-09:27:073
13500991411,2cyclictest5916-21chrt08:00:143
13500991411,2cyclictest5916-21chrt08:00:143
13500991411,2cyclictest13164-21df08:15:163
1350099141,12cyclictest0-21swapper/309:23:353
1349599149,4cyclictest19541-21hwlatdetect07:20:172
13495991412,2cyclictest37850irq/131-snd_hda09:45:292
13495991412,2cyclictest32-21ksoftirqd/209:42:432
13495991410,3cyclictest22517-21sed11:40:162
1349599140,3cyclictest0-21swapper/211:09:122
1349199144,8cyclictest0-21swapper/107:10:151
13491991413,1cyclictest0-21swapper/110:01:281
13491991411,2cyclictest8011-21grep10:05:251
13491991410,3cyclictest7602-21cron11:30:001
13491991410,3cyclictest10787-21wc10:35:241
1348699149,3cyclictest12859-21tr08:15:150
1348699149,2cyclictest8802-21grep09:10:270
1348699147,6cyclictest31364-21munin-run10:55:000
1348699145,8cyclictest0-21swapper/010:25:130
1348699141,8cyclictest25940-21apache_processe07:35:130
13486991410,2cyclictest26512-21grep08:40:270
1348699140,1cyclictest0-21swapper/010:08:200
1348699140,14cyclictest0-21swapper/011:53:160
74512130,0chrt0-21swapper/309:39:553
1350099139,3cyclictest17430-21wc11:10:203
1350099139,3cyclictest15853-21munin-run07:15:003
1350099139,2cyclictest4426-21grep09:05:153
13500991310,2cyclictest7950-21tr12:15:143
13500991310,2cyclictest3453-21sh11:50:003
13500991310,2cyclictest26993-21cron08:45:003
1350099130,12cyclictest28903-21sed08:45:243
1350099130,12cyclictest11848-21sed08:10:273
1349599139,3cyclictest20144-21head09:20:282
1349599132,9cyclictest37850irq/131-snd_hda08:30:132
13495991312,1cyclictest0-21swapper/207:50:132
13495991311,1cyclictest24095-21apache212:13:262
13495991310,2cyclictest30260-21wc09:30:182
13495991310,2cyclictest14455-21grep09:15:382
1349199139,3cyclictest19993-21wc07:20:211
1349199132,7cyclictest22999-21ssh11:40:181
13491991310,2cyclictest6623-21cat09:10:001
13491991310,2cyclictest28889-21grep08:45:241
13491991310,2cyclictest25-21ksoftirqd/110:40:181
13491991310,2cyclictest22904-21cut11:15:151
13491991310,2cyclictest18880-21grep10:15:141
13491991310,2cyclictest1521-21cron09:35:001
1349199130,12cyclictest2503-21grep09:00:161
1348699139,3cyclictest24134-21grep08:35:330
1348699139,3cyclictest19095-21cat12:00:260
1348699139,3cyclictest16109-21cat11:10:000
1348699135,7cyclictest28201-21munin-run11:20:000
1348699134,9cyclictest11726-21packagekitd11:28:230
13486991310,2cyclictest8322-21sh12:35:140
13486991310,2cyclictest7254-21wc12:15:000
13486991310,2cyclictest5042-21sort09:05:180
13486991310,2cyclictest30881-21wc11:45:260
13486991310,2cyclictest2805-21grep07:54:590
13486991310,2cyclictest17104-21grep11:10:190
1350099129,2cyclictest9699-21wc12:35:223
1350099129,2cyclictest8781-21tail09:40:183
1350099129,2cyclictest675-21grep12:10:203
1350099129,2cyclictest4416-21df10:30:133
1350099129,2cyclictest4247-21tr11:50:163
1350099129,2cyclictest30434-21sed08:50:333
1350099129,2cyclictest30432-21tr09:30:203
1350099129,2cyclictest30368-21grep10:25:133
1350099129,2cyclictest25758-21wc12:25:223
1350099129,2cyclictest24046-21sed08:35:333
1350099129,2cyclictest18427-21grep10:15:133
1350099129,2cyclictest1438-21grep12:30:243
1350099128,3cyclictest31456-21cat12:10:003
1350099128,3cyclictest22144-21cron08:35:013
1350099125,3cyclictest224-21systemd-journal09:04:593
1350099124,4cyclictest224-21systemd-journal08:30:013
13500991211,1cyclictest0-21swapper/311:32:183
1350099120,11cyclictest17897-21ssh12:20:273
1349599129,2cyclictest8473-21tail12:15:172
1349599129,2cyclictest3436-21cron10:30:012
1349599129,2cyclictest32340-21grep07:45:282
1349599129,2cyclictest23510-21cut08:35:202
1349599129,2cyclictest18278-21tail12:00:182
1349599129,2cyclictest16479-21tr11:10:152
1349599128,3cyclictest15908-21cron12:40:002
1349599126,5cyclictest10988-21sed08:10:182
13495991210,2cyclictest61950irq/127-enp2s0-12:23:302
1349199129,2cyclictest25646-21head08:40:181
1349199129,2cyclictest24642-21wc09:25:231
1349199129,2cyclictest17428-21cat11:10:201
1349199128,3cyclictest20758-21head08:30:171
1349199128,3cyclictest1378-21ntpd12:10:441
1349199126,3cyclictest513-21polkitd08:59:591
13491991211,1cyclictest0-21swapper/110:56:081
1349199120,11cyclictest15740-21grep11:35:161
1348699129,2cyclictest9597-21sed12:15:250
1348699129,2cyclictest5347-21cat08:00:010
1348699129,2cyclictest31449-21tail07:45:170
1348699129,2cyclictest30207-21wc11:20:270
1348699129,2cyclictest27510-21grep08:45:140
1348699129,2cyclictest25231-21head10:20:190
1348699129,2cyclictest24137-21cat10:20:000
1348699129,2cyclictest23535-21grep07:30:130
1348699129,2cyclictest13558-21head09:15:230
1348699129,2cyclictest10619-21nfs4_client10:35:210
1348699126,2cyclictest224-21systemd-journal07:25:000
1348699124,4cyclictest224-21systemd-journal07:15:000
1348699124,4cyclictest20616-21ssh12:22:340
1350099118,2cyclictest8462-21sh09:10:233
1350099118,2cyclictest8037-21ssh11:01:533
1350099118,2cyclictest6334-21grep10:05:143
1350099118,2cyclictest5338-21ssh11:27:173
1350099118,2cyclictest4529-21ssh10:03:313
1350099118,2cyclictest32559-21ssh11:23:223
1350099118,2cyclictest27667-21uptime07:35:293
1350099118,2cyclictest25296-21ssh11:16:213
1350099118,2cyclictest2385-21sed07:50:263
1350099118,2cyclictest22849-21vmstat07:25:273
1350099118,2cyclictest22002-21chrt09:50:493
1350099118,2cyclictest22002-21chrt09:50:483
1350099118,2cyclictest17991-21sh10:42:243
1350099118,2cyclictest17164-21chrt10:14:043
1350099118,2cyclictest17139-21ssh11:35:423
1350099118,2cyclictest13251-21ssh11:06:423
1350099117,3cyclictest262782sleep309:55:133
1350099117,2cyclictest3804-21head07:55:183
1349599119,1cyclictest21231-21head10:45:172
1349599118,2cyclictest792-21sed10:00:172
1349599118,2cyclictest7689-21ssh11:01:272
1349599118,2cyclictest5569-21cut08:00:132
1349599118,2cyclictest5569-21cut08:00:122
1349599118,2cyclictest4783-21expr09:05:172
1349599118,2cyclictest4744-21if_enp0s31f610:30:162
1349599118,2cyclictest4731-21grep07:55:282
1349599118,2cyclictest3913-21ssh11:50:142
1349599118,2cyclictest32750-21head08:55:152
1349599118,2cyclictest31772-21grep08:55:002
1349599118,2cyclictest29380-21sh08:49:592
1349599118,2cyclictest29126-21ssh11:20:152
1349599118,2cyclictest27756-21ssh11:19:292
1349599118,2cyclictest27075-21ntpq07:35:212
1349599118,2cyclictest26365-21rm10:20:302
1349599118,2cyclictest26004-21sh09:55:002
1349599118,2cyclictest23849-21ssh09:53:002
1349599118,2cyclictest23849-21ssh09:52:592
1349599118,2cyclictest23691-21cat09:25:142
1349599118,2cyclictest21158-21taskset07:25:142
1349599118,2cyclictest18262-21uniq08:25:162
1349599118,2cyclictest16837-21head07:15:162
1349599118,1cyclictest27002-21sh12:05:552
1349599118,1cyclictest13861-21head08:15:212
1349599117,3cyclictest29831-21ssh10:52:592
1349599117,2cyclictest16988-21munin-run12:00:012
1349599117,1cyclictest9818-21ssh10:07:262
1349599116,4cyclictest687-21fschecks_count12:30:182
1349599111,9cyclictest17172-21cron08:25:012
1349599110,10cyclictest659-21ssh11:47:142
1349199118,2cyclictest9034-21uname11:30:171
1349199118,2cyclictest7820-21sh11:01:361
1349199118,2cyclictest7646-21sort09:10:171
1349199118,2cyclictest7580-21cron08:05:001
1349199118,2cyclictest7580-21cron08:04:591
1349199118,2cyclictest5243-21ntpq11:50:231
1349199118,2cyclictest3689-21grep07:55:181
1349199118,2cyclictest30938-21ssh10:25:151
1349199118,2cyclictest26692-21ssh12:25:481
1349199118,2cyclictest26586-21ssh12:05:321
1349199118,2cyclictest25243-21head10:20:191
1349199118,2cyclictest24554-21chrt08:39:381
1349199118,2cyclictest20334-21ssh10:45:051
1349199118,2cyclictest19089-21ssh12:21:221
1349199118,2cyclictest18551-21tr09:20:151
1349199118,2cyclictest18254-21perf07:19:591
1349199118,2cyclictest1784-21ssh09:35:021
1349199118,2cyclictest17475-21dpkg12:00:141
1349199118,2cyclictest1621-21uname07:50:191
1349199118,2cyclictest16010-21ssh09:46:091
1349199118,2cyclictest15797-21head08:20:211
1349199118,2cyclictest15196-21ssh09:16:251
1349199118,2cyclictest12765-21grep08:15:131
1349199118,2cyclictest12703-21ssh12:37:251
1349199118,2cyclictest10597-21ssh09:41:181
1349199118,2cyclictest10567-21aten2_r7power_p11:05:141
1349199118,2cyclictest10563-21ssh11:55:021
1349199117,2cyclictest4126-21grep10:30:141
1349199115,3cyclictest224-21systemd-journal07:50:001
1349199113,2cyclictest30321-21cut08:50:331
13491991110,1cyclictest0-21swapper/112:18:481
1349199110,10cyclictest388-21ssh11:23:451
1349199110,10cyclictest0-21swapper/109:56:311
1348699118,2cyclictest5567-21wget08:00:130
1348699118,2cyclictest5567-21wget08:00:120
1348699118,2cyclictest4934-21basename11:00:120
1348699118,2cyclictest4670-21which10:30:160
1348699118,2cyclictest32624-21sort08:55:150
1348699118,2cyclictest30190-21ssh09:30:180
1348699118,2cyclictest28536-21egrep07:40:150
1348699118,2cyclictest27493-21ssh12:06:190
1348699118,2cyclictest27301-21sh12:26:170
1348699118,2cyclictest25770-21chrt11:42:090
1348699118,2cyclictest21350-21ssh10:45:180
1348699118,2cyclictest20940-21ssh11:39:240
1348699118,2cyclictest20746-21ssh09:21:190
1348699118,2cyclictest20743-21cat09:50:190
1348699118,2cyclictest20743-21cat09:50:190
1348699118,2cyclictest20686-21sort08:30:170
1348699118,2cyclictest19891-21ssh10:44:500
1348699118,2cyclictest17947-21grep08:25:150
1348699118,2cyclictest13968-21expr09:45:130
1348699118,2cyclictest11165-21chrt09:42:010
1348699118,2cyclictest10208-21acpi08:10:110
1348699117,3cyclictest24032-21sed09:25:180
1348699117,2cyclictest15062-21cut08:20:130
1348699117,2cyclictest1020-21grep10:55:270
1348699116,3cyclictest10030-21ssh11:30:300
1348699113,4cyclictest224-21systemd-journal10:05:010
1348699110,10cyclictest8712-21taskset08:05:190
1350099108,1cyclictest9331-21ssh10:35:093
1350099108,1cyclictest7949-21cut08:05:143
1350099108,1cyclictest30767-21cut07:45:133
1350099108,1cyclictest25838-21seq10:50:043
1350099108,1cyclictest24786-21grep07:30:213
1350099108,1cyclictest22032-21cat10:45:263
1350099108,1cyclictest20717-21chrt07:23:273
1350099108,1cyclictest16160-21ntpq08:20:243
1350099107,2cyclictest903-21grep08:55:203
1350099107,2cyclictest31612-21ssh10:55:043
1350099107,2cyclictest28589-21tr07:40:163
1350099107,2cyclictest12782-21aten2_r7power_e09:15:203
1349599107,2cyclictest9675-21sshd08:06:262
1349599107,2cyclictest9496-21ssh09:11:332
1349599107,2cyclictest18245-21cut10:15:122
1349599107,2cyclictest17294-21ssh11:35:512
1349599107,2cyclictest12230-21cut10:10:162
1349599106,3cyclictest28583-21tr07:40:162
1349599102,6cyclictest25669-21cron07:34:592
1349199108,1cyclictest17661-21apache_volume08:25:121
1349199107,2cyclictest8000-21grep08:05:151
1349199107,2cyclictest30305-21chrt07:43:121
1349199107,2cyclictest23455-21seq07:30:011
1349199107,2cyclictest12089-21taskset08:11:491
1349199103,3cyclictest224-21systemd-journal07:40:001
1348699108,1cyclictest1933-21ssh09:35:110
1348699108,1cyclictest16231-21cut07:15:120
1348699107,2cyclictest31090-21wc08:50:410
1348699107,2cyclictest3042-21expr09:00:210
1348699107,2cyclictest21152-21tr07:25:130
135009997,1cyclictest17873-21chrt07:15:343
134959997,1cyclictest2938-21cut09:00:202
134959997,1cyclictest25122-21tr08:40:162
134959997,1cyclictest14652-21uname07:10:212
134919996,2cyclictest21936-21chrt07:25:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional