You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-23 - 08:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 23, 2024 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2093599495,32cyclictest26866irq/58-eth023:40:113
2093599481,13cyclictest26866irq/58-eth022:30:223
2093599383,25cyclictest26866irq/58-eth000:10:123
2093599383,23cyclictest26866irq/58-eth021:05:143
2093599376,22cyclictest26866irq/58-eth000:15:113
2093599376,22cyclictest26866irq/58-eth000:15:103
2092999376,9cyclictest27-21ksoftirqd/221:10:132
2093599362,24cyclictest26866irq/58-eth000:05:143
2093599343,14cyclictest0-21swapper/321:14:463
20929993312,11cyclictest0-21swapper/221:10:022
2092299338,4cyclictest21-21ksoftirqd/122:30:231
20911993313,4cyclictest12-21ksoftirqd/023:35:130
2093599302,13cyclictest26866irq/58-eth022:50:093
2093599291,14cyclictest26866irq/58-eth021:20:113
2092299292,16cyclictest28799-21ssh22:40:111
2093599286,13cyclictest26866irq/58-eth022:40:103
2093599284,12cyclictest26866irq/58-eth022:45:143
2093599283,18cyclictest0-21swapper/321:29:463
2093599283,13cyclictest26866irq/58-eth022:10:123
2092999281,10cyclictest1656166ssh22:50:202
2093599276,18cyclictest26866irq/58-eth022:15:133
2093599274,11cyclictest26866irq/58-eth000:00:133
2093599273,18cyclictest0-21swapper/321:42:463
2093599271,16cyclictest26866irq/58-eth023:50:173
2093599266,12cyclictest26866irq/58-eth023:25:163
2093599263,12cyclictest0-21swapper/320:55:123
2093599256,9cyclictest26966irq/59-eth022:35:143
2093599255,11cyclictest0-21swapper/322:00:133
2093599253,14cyclictest26866irq/58-eth021:35:133
20935992511,9cyclictest26866irq/58-eth021:15:143
2092299255,5cyclictest21-21ksoftirqd/123:55:121
2093599243,13cyclictest0-21swapper/323:47:453
2092999245,3cyclictest131rcu_preempt23:10:142
2092999241,7cyclictest795966ssh23:43:522
2092999241,7cyclictest776566ssh21:30:022
2092999241,7cyclictest707466ssh21:29:222
2092999241,7cyclictest3237266ssh22:40:242
2092999241,6cyclictest1162766ssh22:48:282
20929992413,9cyclictest17245-21ssh23:49:582
2091199243,18cyclictest7904-21apt-get19:30:150
2093599233,13cyclictest0-21swapper/321:55:143
2093599232,13cyclictest26866irq/58-eth022:25:123
2092999233,17cyclictest18751-21/usr/sbin/munin22:15:142
2092999233,13cyclictest21487-21latency00:10:222
2092999231,7cyclictest1134466ssh23:06:112
2092999231,6cyclictest1463166ssh00:27:532
2092999231,5cyclictest877766kworker/u8:200:15:222
2092999231,5cyclictest877766kworker/u8:200:15:212
2092999231,14cyclictest2711866ssh22:20:062
2091199237,3cyclictest131rcu_preempt22:15:320
2093599223,12cyclictest0-21swapper/321:34:473
2092999223,11cyclictest868-21snmpd19:50:132
2092999222,18cyclictest20803-21kthreadcore20:15:242
2092999222,17cyclictest25787-21kthreadcore20:20:192
2092999222,17cyclictest24986-21kthreadcore19:45:232
2092999222,17cyclictest16065-21latency19:35:212
2092999222,17cyclictest10695-21fschecks_time20:40:182
2092999222,13cyclictest6811-21kthreadcore20:00:222
2092999222,13cyclictest28520-21switchtime20:55:312
2092999222,13cyclictest25918-21kthreadcore23:35:222
2092999221,8cyclictest0-21swapper/223:34:072
2092999221,7cyclictest3062966ssh21:23:332
2092999221,7cyclictest2663966ssh22:56:002
2092999221,4cyclictest0-21swapper/223:00:392
20929992212,8cyclictest10708-21ssh22:28:552
20929992212,8cyclictest0-21swapper/223:25:242
20929992211,9cyclictest5075-21ssh23:21:572
20929992211,9cyclictest31379-21ssh21:43:022
2092999221,13cyclictest1738866ssh21:55:052
2092999221,13cyclictest135366ssh23:59:252
20929992210,5cyclictest28215-21ssh23:15:512
2091199226,4cyclictest1478-21find00:20:130
2091199223,16cyclictest1116-21sendmail00:20:000
2091199223,16cyclictest1116-21sendmail00:20:000
2091199221,4cyclictest31259-21munin-run23:20:010
2093599213,12cyclictest0-21swapper/321:54:463
2092999214,14cyclictest12075-21ssh22:30:222
2092999213,16cyclictest2455-21kthreadcore20:30:202
2092999213,16cyclictest2418-21kthreadcore00:20:212
2092999212,17cyclictest4184-21kthreadcore00:00:222
2092999212,16cyclictest3820-21kthreadcore19:25:212
2092999212,16cyclictest20330-21hwlatdetect20:50:182
2092999212,12cyclictest22753-21kthreadcore22:35:232
2092999211,6cyclictest2874466ssh00:38:052
2092999211,6cyclictest2195566ssh21:18:022
2092999211,6cyclictest1686566ssh00:08:392
2092999211,6cyclictest1568266ssh21:52:582
2092999211,12cyclictest575266ssh21:45:432
2092999211,12cyclictest575266ssh21:45:422
2092299213,5cyclictest21-21ksoftirqd/119:50:011
20922992111,7cyclictest23681-21apt-get21:20:101
2091199214,4cyclictest12-21ksoftirqd/021:45:230
2091199214,4cyclictest12-21ksoftirqd/021:45:230
2091199211,9cyclictest1320066ssh23:45:310
2091199211,11cyclictest1139066ssh23:25:310
2092999208,10cyclictest156-21systemd-udevd22:06:452
2092999207,6cyclictest18042-21cut21:35:222
2092999202,16cyclictest21183-21cpuspeed19:10:132
2092999202,16cyclictest21183-21cpuspeed19:10:122
2092999202,16cyclictest16274-21irqcore00:30:202
2092999202,12cyclictest458-21kthreadcore19:20:222
2092999202,11cyclictest431-21kthreadcore21:00:212
2092999202,11cyclictest17947-21irqcore19:40:222
2092999201,6cyclictest131rcu_preempt22:00:342
2092999201,5cyclictest627366kworker/u8:123:51:252
2092299205,3cyclictest131rcu_preempt23:54:411
2092299204,2cyclictest131rcu_preempt21:17:251
2092299202,4cyclictest21-21ksoftirqd/123:02:201
2091199209,8cyclictest31875-21fw_packets23:20:170
2091199209,3cyclictest12-21ksoftirqd/023:30:250
2091199208,3cyclictest131rcu_preempt23:45:010
2091199204,3cyclictest131rcu_preempt19:30:010
2091199201,5cyclictest0-21swapper/021:43:530
2091199201,16cyclictest20489-21apt-get22:35:130
2091199201,15cyclictest1852-21apt-get22:05:140
2093599199,6cyclictest26866irq/58-eth000:30:143
2093599198,7cyclictest26866irq/58-eth022:55:133
2093599193,11cyclictest0-21swapper/323:04:473
2093599193,11cyclictest0-21swapper/322:20:143
2093599192,11cyclictest0-21swapper/321:45:063
2093599192,11cyclictest0-21swapper/321:45:053
2093599191,13cyclictest0-21swapper/319:45:303
2093599191,12cyclictest0-21swapper/323:20:013
2093599191,12cyclictest0-21swapper/320:35:033
2093599191,12cyclictest0-21swapper/319:35:153
2093599191,12cyclictest0-21swapper/319:25:023
20935991910,6cyclictest0-21swapper/300:27:233
2092999194,7cyclictest5690-21apt-get20:35:112
2092999192,15cyclictest28602-21kthreadcore19:15:232
2092999192,10cyclictest12408-21kthreadcore20:05:222
2092999191,16cyclictest8236-21grep19:30:202
2092999191,10cyclictest868-21snmpd20:10:132
2092999191,10cyclictest28564-21diskstats20:25:152
2092299197,8cyclictest0-21swapper/121:40:151
2092299195,5cyclictest1467-21kworker/1:122:50:271
2092299194,3cyclictest131rcu_preempt23:41:551
2092299192,3cyclictest131rcu_preempt23:09:031
20922991912,5cyclictest21-21ksoftirqd/123:45:261
20922991912,5cyclictest14584-1kworker/u9:023:15:161
2092299191,13cyclictest20516-21ssh22:35:141
2092299191,13cyclictest16212-21ssh23:30:151
2092299191,12cyclictest12933-21ssh21:30:471
20922991910,7cyclictest8716-21apt-get00:25:151
2092299190,3cyclictest0-21swapper/122:04:151
2092299190,16cyclictest12741-21sed19:35:011
2091199199,8cyclictest15354-21apt-get21:15:130
2091199195,2cyclictest131rcu_preempt19:20:020
2091199193,13cyclictest10621-21grep20:40:180
2091199192,4cyclictest12-21ksoftirqd/023:05:180
2091199192,15cyclictest9781-21cron00:05:010
2091199192,13cyclictest3322-21ssh22:25:120
2091199192,13cyclictest26985-21apt-get19:50:040
2091199191,4cyclictest0-21swapper/022:00:250
20911991912,4cyclictest0-21swapper/020:25:240
2091199191,15cyclictest28549-21grep20:55:300
2091199191,14cyclictest32048-21apt-get21:25:140
2091199191,14cyclictest30739-21apt-get19:20:120
20911991910,6cyclictest10669-21cut21:50:210
2093599189,6cyclictest0-21swapper/323:05:443
2093599189,6cyclictest0-21swapper/300:35:043
2093599189,6cyclictest0-21swapper/300:21:063
2093599188,4cyclictest26966irq/59-eth022:05:163
2093599188,3cyclictest0-21swapper/323:57:183
2093599183,11cyclictest0-21swapper/323:20:143
2093599183,10cyclictest0-21swapper/323:30:143
2093599182,10cyclictest0-21swapper/320:45:173
2093599182,10cyclictest0-21swapper/320:10:153
2093599182,10cyclictest0-21swapper/320:05:123
2093599182,10cyclictest0-21swapper/319:50:053
2093599182,10cyclictest0-21swapper/319:10:123
2093599182,10cyclictest0-21swapper/319:10:123
20935991810,6cyclictest0-21swapper/323:11:063
2092999185,6cyclictest11012-21kthreadcore22:10:192
2092999183,9cyclictest18574-21latency20:45:282
2092999183,12cyclictest32665-21irqrtprio19:55:242
2092299188,8cyclictest23787-21df_inode20:20:121
2092299184,3cyclictest131rcu_preempt21:10:271
2092299183,2cyclictest131rcu_preempt23:26:551
2092299183,2cyclictest131rcu_preempt23:11:581
2092299182,2cyclictest92-21pr/ttyS019:40:011
2092299182,2cyclictest131rcu_preempt21:29:171
2092299181,4cyclictest24151-1kworker/u9:200:15:121
2092299181,4cyclictest24151-1kworker/u9:200:15:121
2092299181,14cyclictest24741-21ssh23:35:141
2092299181,14cyclictest17500-21ssh21:55:131
2092299181,14cyclictest15800-21ssh00:30:151
2092299181,13cyclictest2164-21ssh00:00:131
20922991810,6cyclictest1539-21sed21:45:211
20922991810,6cyclictest1539-21sed21:45:201
2091199189,7cyclictest14953-21cat22:10:310
2091199188,7cyclictest14084-21apt-get20:10:140
2091199183,13cyclictest25826-21ssh22:55:340
2091199183,12cyclictest24712-21cron20:55:020
2091199182,14cyclictest8314-21sed21:30:170
2091199182,13cyclictest307-21uname19:55:230
2091199182,13cyclictest15437-21grep20:45:200
2091199181,9cyclictest1875666ssh22:33:110
2091199181,5cyclictest0-21swapper/023:04:090
20911991812,3cyclictest25500-21sh22:00:020
20911991811,4cyclictest7252-21fschecks_count21:10:160
2093599172,10cyclictest0-21swapper/320:20:123
2093599172,10cyclictest0-21swapper/319:40:173
2092299179,5cyclictest6721-21diskmemload21:35:131
2092299178,7cyclictest11223-21cat22:10:201
2092299172,4cyclictest21-21ksoftirqd/120:48:461
2092299172,2cyclictest131rcu_preempt22:45:021
2092299172,2cyclictest131rcu_preempt22:06:161
2092299171,9cyclictest15510-21ssh21:52:431
2092299171,3cyclictest131rcu_preempt22:27:531
2092299171,2cyclictest131rcu_preempt22:16:211
2092299171,2cyclictest0-21swapper/123:23:391
2092299171,2cyclictest0-21swapper/100:14:461
2092299171,14cyclictest21276-21tune2fs19:10:151
2092299171,14cyclictest21276-21tune2fs19:10:141
20922991711,4cyclictest0-21swapper/119:40:351
20922991711,4cyclictest0-21swapper/119:25:261
2092299171,13cyclictest27315-21apt-get19:50:121
2092299171,12cyclictest30889-21ntp_states22:20:301
20922991710,4cyclictest0-21swapper/122:58:061
2091199179,6cyclictest0-21swapper/022:52:150
2091199173,12cyclictest9734-21tr00:25:230
2091199173,12cyclictest30011-21ssh21:22:410
2091199173,12cyclictest22077-21kthreadcore20:15:250
2091199172,13cyclictest6002-21grep21:05:340
2091199172,13cyclictest27621-21sed22:20:190
2091199172,12cyclictest32311-21missed_timers22:40:230
2091199172,12cyclictest27138-21ntp_states20:20:250
2091199172,12cyclictest22304-21grep19:45:020
2091199172,12cyclictest10275-21idleruntime-cro20:40:010
2091199171,14cyclictest14472-21kthreadcore19:35:190
2091199171,13cyclictest13998-21cat00:05:320
2093599167,7cyclictest0-21swapper/323:35:093
2093599162,7cyclictest0-21swapper/321:00:113
2093599162,10cyclictest0-21swapper/320:35:123
2093599162,10cyclictest0-21swapper/320:25:133
2093599162,10cyclictest0-21swapper/319:25:133
2093599161,12cyclictest0-21swapper/320:00:013
2092999161,7cyclictest21076-21grep19:10:022
2092299168,6cyclictest9396-21df_inode20:05:141
2092299168,6cyclictest4526-21apt-get20:00:111
2092299161,2cyclictest131rcu_preempt00:37:111
2092299161,12cyclictest696-21ntp_192.168.11421:00:251
2092299161,12cyclictest14178-21diskstats20:10:151
2092299161,12cyclictest10137-21ssh00:05:141
2092299161,11cyclictest2198-21irqstats00:20:211
2091199168,6cyclictest25701-21cron19:15:000
2091199168,6cyclictest25701-21cron19:15:000
2091199163,11cyclictest4973-21fschecks_time22:45:180
2091199162,12cyclictest5124-21sed20:00:200
2091199162,12cyclictest23141-21/usr/sbin/munin00:35:190
2091199162,12cyclictest1503-21uname20:30:200
2091199162,12cyclictest10991-21kthreadcore20:05:200
2091199161,12cyclictest29725-21apt-get21:00:100
2091199161,12cyclictest26985-21apt-get19:50:010
2091199161,12cyclictest26258-21df23:55:150
2091199161,12cyclictest15792-21apt-get00:30:140
2091199161,12cyclictest14571-21apt-get23:10:110
20911991610,4cyclictest18664-21sh00:10:180
20911991610,3cyclictest16378-21cpuspeed21:35:150
2091199160,10cyclictest18097-21grep23:50:220
20935991510,3cyclictest0-21swapper/320:53:463
2092299159,4cyclictest0-21swapper/120:40:301
2092299158,5cyclictest4619-21timerwakeupswit20:30:291
2092299157,6cyclictest31446-21kthreadcore19:20:191
2092299151,3cyclictest131rcu_preempt19:56:441
2092299151,11cyclictest28470-21apt-get20:25:131
2092299151,11cyclictest28309-21ntp_192.168.11420:55:261
2092299151,11cyclictest23453-21munin-run20:20:011
2092299147,5cyclictest27819-21kthreadcore19:15:221
2092299141,6cyclictest8574-21cut20:35:221
2092299141,10cyclictest23543-21ntp_192.168.11420:50:261
2092299141,10cyclictest2315-21fw_packets21:05:191
2093599133,8cyclictest0-21swapper/320:00:163
2093599132,7cyclictest0-21swapper/319:30:143
2093599131,7cyclictest0-21swapper/320:40:153
2092299137,4cyclictest21035-21latency_hist19:10:011
2091199131,9cyclictest20910-21cyclictest19:10:020
2093599127,3cyclictest0-21swapper/319:19:473
2093599121,8cyclictest0-21swapper/319:10:013
2093599112,5cyclictest0-21swapper/320:15:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional