You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-03 - 09:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Jul 03, 2025 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24168998319,13cyclictest131rcu_preempt00:05:150
24168998319,13cyclictest131rcu_preempt00:05:150
24200996323,27cyclictest32466irq/58-eth023:45:183
2417999634,51cyclictest1758-21ssh22:45:181
2417999634,51cyclictest1758-21ssh22:45:171
2417999594,8cyclictest131rcu_preempt21:55:131
2416899552,39cyclictest18113-21ssh22:55:160
24179995112,7cyclictest131rcu_preempt00:00:151
2416899492,37cyclictest24891-21ssh22:20:160
2417999471,5cyclictest0-21swapper/119:10:131
2417999471,3cyclictest0-21swapper/120:25:131
2416899474,36cyclictest31097-21ssh21:25:140
2420099464,28cyclictest32466irq/58-eth000:05:143
2420099464,28cyclictest32466irq/58-eth000:05:143
2417999462,7cyclictest0-21swapper/120:40:131
24168994631,10cyclictest0-21swapper/000:20:140
2420099456,24cyclictest32466irq/58-eth023:00:163
2419099451,32cyclictest10173-21ssh00:30:142
2417999452,2cyclictest7556-21apt-get21:50:161
2417999452,2cyclictest3825-21apt-get23:45:171
2417999452,2cyclictest27276-21apt-get00:20:131
2416899452,32cyclictest19426-21ssh00:15:130
2419099432,22cyclictest18770-21ssh23:15:142
2419099431,31cyclictest28426-21ssh00:00:162
2417999431,3cyclictest0-21swapper/122:15:141
2417999430,2cyclictest31720-21ssh21:45:131
2420099422,23cyclictest32466irq/58-eth021:30:143
2417999421,4cyclictest0-21swapper/120:30:171
2417999421,37cyclictest0-21swapper/120:00:141
2417999413,5cyclictest0-21swapper/120:10:141
2417999411,5cyclictest0-21swapper/119:15:121
2417999411,4cyclictest31090-21apt-get21:25:141
2417999411,3cyclictest0-21swapper/123:40:181
24190994021,11cyclictest23522-21ssh21:40:142
24190994021,11cyclictest23522-21ssh21:40:132
2417999401,2cyclictest0-21swapper/122:05:131
24168994025,10cyclictest626-21tcf-agent21:15:150
24168994018,12cyclictest151rcuc/021:50:160
2417999392,33cyclictest0-21swapper/123:55:161
2417999391,4cyclictest0-21swapper/120:20:141
2417999391,3cyclictest0-21swapper/120:55:161
24168993929,6cyclictest151rcuc/023:25:140
2416899392,31cyclictest11705-21ssh00:10:150
2417999381,3cyclictest0-21swapper/120:15:131
24168993814,4cyclictest131rcu_preempt22:35:210
2417999371,4cyclictest2240-21apt-get23:05:141
2417999371,3cyclictest3792-21apt-get00:05:151
2417999371,3cyclictest3792-21apt-get00:05:151
2417999371,3cyclictest0-21swapper/121:05:171
2417999370,5cyclictest6425-21apt-get19:25:151
2416899373,27cyclictest27988-21ssh23:40:160
2416899371,31cyclictest3826-21ssh23:45:160
2417999361,4cyclictest11232-21apt-get19:30:131
2417999361,2cyclictest0-21swapper/121:40:131
2417999361,2cyclictest0-21swapper/121:40:131
2417999360,3cyclictest2674-21apt-get00:25:131
2420099353,23cyclictest32466irq/58-eth000:30:153
2419099351,30cyclictest825-21ssh22:25:152
2417999352,4cyclictest0-21swapper/122:30:141
2417999352,24cyclictest8205-21sh22:10:141
2417999351,3cyclictest24883-21apt-get22:20:181
2417999351,3cyclictest0-21swapper/120:35:131
2417999351,2cyclictest16070-21apt-get19:35:141
2417999351,2cyclictest11357-21apt-get23:30:131
2417999351,11cyclictest2970-21apt-get23:25:121
2417999350,4cyclictest20906-21apt-get19:40:151
2420099349,16cyclictest32466irq/58-eth023:50:163
24200993421,8cyclictest32466irq/58-eth023:40:183
2417999341,4cyclictest23046-21diskstats21:20:151
2417999341,12cyclictest7038-21diskstats21:30:161
2419099330,25cyclictest31102-21ssh21:25:152
2417999332,26cyclictest0-21swapper/122:04:521
2417999331,9cyclictest16133-21tail23:50:311
2417999331,9cyclictest1459-21sort22:45:011
2417999331,8cyclictest28959-21tr23:00:241
2417999331,8cyclictest14065-21tr00:10:241
2417999331,6cyclictest19537-21diskstats00:15:151
2417999331,11cyclictest4517-21ntp_states22:25:271
2417999331,11cyclictest15977-21irqstats21:35:201
2417999331,10cyclictest31544-21getconf23:20:311
2417999331,10cyclictest31544-21getconf23:20:311
2417999331,10cyclictest18364-21ssh23:15:001
2419099329,10cyclictest131rcu_preempt19:20:122
2419099325,25cyclictest3848-21ssh23:45:182
24190993216,13cyclictest628-21Async09:12:572
2417999326,24cyclictest18933-21cat21:15:331
2417999321,9cyclictest10072-21/usr/sbin/munin22:50:161
2417999321,8cyclictest22267-21memory23:15:251
2417999321,8cyclictest12587-21ssh21:12:401
2417999321,7cyclictest21219-21kthreadcore22:55:241
2417999321,3cyclictest0-21swapper/119:55:101
2417999321,10cyclictest19299-21df_inode20:50:151
2416899329,3cyclictest131rcu_preempt19:20:120
2416899324,18cyclictest12-21ksoftirqd/023:10:160
24168993218,10cyclictest151rcuc/020:40:120
2419099313,19cyclictest3791-21ssh00:05:132
2419099313,19cyclictest3791-21ssh00:05:132
2419099313,17cyclictest5908-21diskmemload22:50:152
2419099311,15cyclictest16489-21ssh22:15:162
2417999312,5cyclictest31287-21kthreadcore21:00:231
2417999311,9cyclictest14909-21awk00:30:341
2417999311,7cyclictest14827-21cut20:45:201
2417999311,7cyclictest14827-21cut20:45:201
2416899312,7cyclictest12-21ksoftirqd/020:00:140
24168993117,8cyclictest12-21ksoftirqd/023:34:520
2420099303,14cyclictest32466irq/58-eth022:15:143
2420099302,21cyclictest32466irq/58-eth021:00:143
2419099303,21cyclictest23346irq/40-dwc2_hso20:50:132
2419099302,22cyclictest916-21snmpd22:40:182
2419099301,21cyclictest8191-21ssh22:10:132
2417999301,7cyclictest23898-21sed00:40:011
2417999301,7cyclictest23220-21missed_timers23:35:241
2417999301,6cyclictest25734-21/usr/sbin/munin19:45:281
2417999301,6cyclictest17529-21find22:35:181
2417999301,5cyclictest8683-21cut20:05:201
2417999301,5cyclictest1569-21/usr/sbin/munin19:20:261
2420099294,22cyclictest32466irq/58-eth023:20:153
2420099294,22cyclictest32466irq/58-eth023:20:153
2419099293,9cyclictest916-21snmpd20:55:162
2419099293,18cyclictest23346irq/40-dwc2_hso21:20:142
2419099293,17cyclictest23346irq/40-dwc2_hso20:00:142
2419099293,17cyclictest23346irq/40-dwc2_hso19:35:132
2419099292,20cyclictest0-21swapper/219:45:152
2419099292,18cyclictest23346irq/40-dwc2_hso21:35:152
2417999291,6cyclictest30643-21sshd19:50:491
2420099283,18cyclictest32466irq/58-eth023:55:153
2419099283,16cyclictest23346irq/40-dwc2_hso00:25:142
2419099282,21cyclictest25417-21ssh23:36:522
2419099282,20cyclictest28008-21ssh23:40:182
2419099282,20cyclictest24032-21ssh22:00:132
2419099282,11cyclictest0-21swapper/222:22:512
2419099282,11cyclictest0-21swapper/220:10:142
2419099281,17cyclictest23346irq/40-dwc2_hso19:30:142
2419099281,16cyclictest23346irq/40-dwc2_hso21:50:142
2419099281,16cyclictest1763-21ssh22:45:182
2419099281,16cyclictest1763-21ssh22:45:182
2417999280,6cyclictest24204-21latency_hist19:10:021
2416899283,17cyclictest19678-21ssh23:35:150
2420099273,16cyclictest0-21swapper/300:28:433
2420099273,12cyclictest0-21swapper/300:39:523
2419099273,17cyclictest0-21swapper/220:25:132
2419099273,16cyclictest23346irq/40-dwc2_hso20:20:142
2419099272,20cyclictest0-21swapper/200:10:152
2419099272,17cyclictest23346irq/40-dwc2_hso19:55:092
2419099272,11cyclictest0-21swapper/221:15:172
2419099272,10cyclictest0-21swapper/223:51:522
24190992712,12cyclictest0-21swapper/223:30:132
2416899272,19cyclictest9272-21ssh00:28:430
2416899272,17cyclictest26376-21ssh23:00:140
2420099261,13cyclictest32466irq/58-eth000:20:143
2419099262,17cyclictest23346irq/40-dwc2_hso19:25:142
24190992614,9cyclictest0-21swapper/221:05:172
2416899261,4cyclictest131rcu_preempt19:25:140
24168992613,10cyclictest28129-21ssh00:00:020
24168992610,3cyclictest131rcu_preempt22:15:150
2420099253,11cyclictest0-21swapper/300:15:133
2420099252,18cyclictest0-21swapper/321:15:173
2420099252,14cyclictest0-21swapper/322:10:133
2420099251,17cyclictest0-21swapper/322:21:403
2419099253,19cyclictest11045-21kthreadcore22:30:212
2419099253,17cyclictest0-21swapper/200:39:532
2419099252,8cyclictest23128-21ssh22:55:522
2419099251,18cyclictest31842-21ssh21:45:162
2416899259,11cyclictest12-21ksoftirqd/023:23:530
2416899259,11cyclictest12-21ksoftirqd/023:23:530
2420099247,12cyclictest0-21swapper/323:15:153
2420099243,13cyclictest0-21swapper/322:58:523
24200992411,10cyclictest0-21swapper/321:55:133
2419099249,12cyclictest27-21ksoftirqd/221:55:142
2419099243,18cyclictest23346irq/40-dwc2_hso23:55:152
2419099241,6cyclictest4890-21grep20:35:172
2419099241,18cyclictest8074-21ssh23:25:522
2416899244,13cyclictest5908-21diskmemload21:30:130
2416899243,16cyclictest2262-21ssh23:05:140
2416899241,16cyclictest23533-21ssh21:40:140
2416899241,16cyclictest23533-21ssh21:40:130
2420099239,10cyclictest0-21swapper/320:40:133
2419099233,11cyclictest10912-21timerwakeupswit21:30:302
2419099232,6cyclictest0-21swapper/223:00:522
2419099232,6cyclictest0-21swapper/200:15:522
2419099232,16cyclictest0-21swapper/223:22:532
2419099232,16cyclictest0-21swapper/223:22:532
24190992311,9cyclictest8683-21ssh23:07:522
2416899237,12cyclictest12-21ksoftirqd/021:45:130
24168992313,7cyclictest151rcuc/020:45:150
24168992313,7cyclictest151rcuc/020:45:140
24168992310,10cyclictest12-21ksoftirqd/023:50:150
2416899230,19cyclictest824-21sh22:25:150
2420099229,10cyclictest0-21swapper/322:35:253
2420099227,10cyclictest32466irq/58-eth021:50:163
24200992214,3cyclictest0-21swapper/320:10:143
24190992216,3cyclictest27253-21ssh00:20:122
2419099221,12cyclictest0-21swapper/219:10:122
2416899228,9cyclictest12-21ksoftirqd/023:15:150
2420099217,9cyclictest32466irq/58-eth023:05:143
2420099214,9cyclictest0-21swapper/320:05:153
2420099212,13cyclictest32466irq/58-eth023:35:143
24200992113,4cyclictest0-21swapper/320:15:123
2419099217,6cyclictest0-21swapper/220:40:172
2419099212,9cyclictest8982-21cut21:10:242
2419099212,5cyclictest0-21swapper/220:30:172
2419099212,15cyclictest15187-21usage_EthcTask23:10:352
2419099211,8cyclictest18224-21unixbench_multi20:45:312
2419099211,8cyclictest18224-21unixbench_multi20:45:312
2419099211,17cyclictest8256-21fschecks_count20:05:182
2419099211,17cyclictest29224-21ps21:00:192
2419099211,10cyclictest19187-21kthreadcore20:15:202
24168992111,7cyclictest11763-21ssh22:50:210
2416899211,16cyclictest32369-21ssh22:05:160
2420099202,8cyclictest0-21swapper/320:30:183
2420099202,14cyclictest0-21swapper/322:25:153
2420099202,11cyclictest0-21swapper/323:30:143
2420099202,11cyclictest0-21swapper/319:20:143
24200992014,3cyclictest0-21swapper/322:45:173
24200992014,3cyclictest0-21swapper/322:45:173
24200992011,5cyclictest0-21swapper/321:25:143
2419099203,13cyclictest1548-21latency_hist19:20:002
2419099201,10cyclictest2634-21kthreadcore22:05:242
2419099201,10cyclictest18035-21irqcore22:35:252
2416899203,4cyclictest131rcu_preempt20:15:210
2416899202,15cyclictest22944-21ssh21:20:130
2416899201,6cyclictest25533-21kthreadcore20:55:220
24168992015,3cyclictest12-21ksoftirqd/022:10:290
24168992014,3cyclictest12-21ksoftirqd/022:00:020
24168992010,8cyclictest1389-21sed22:45:010
24168992010,2cyclictest12-21ksoftirqd/000:39:150
2416899200,17cyclictest6556-21ls21:10:200
2420099199,8cyclictest0-21swapper/321:45:393
2420099193,8cyclictest0-21swapper/321:40:133
2420099193,8cyclictest0-21swapper/321:40:123
2420099192,11cyclictest0-21swapper/319:10:133
24200991911,3cyclictest0-21swapper/319:45:143
2420099191,13cyclictest0-21swapper/319:15:143
2420099191,12cyclictest0-21swapper/319:55:123
24200991910,6cyclictest32466irq/58-eth023:25:143
2416899196,4cyclictest12-21ksoftirqd/000:00:200
2416899196,10cyclictest16306-21kthreadcore21:35:220
2416899192,2cyclictest131rcu_preempt22:46:550
2416899192,2cyclictest131rcu_preempt22:46:550
2416899192,11cyclictest0-21swapper/019:30:150
24168991914,3cyclictest12-21ksoftirqd/020:30:270
2416899191,12cyclictest12-21ksoftirqd/019:35:130
2420099189,6cyclictest0-21swapper/322:42:233
2420099182,10cyclictest0-21swapper/322:30:053
2420099182,10cyclictest0-21swapper/320:25:143
2420099181,8cyclictest32466irq/58-eth000:00:143
2420099181,11cyclictest0-21swapper/320:50:273
24200991810,6cyclictest0-21swapper/323:10:463
24200991810,6cyclictest0-21swapper/322:04:423
24200991810,6cyclictest0-21swapper/321:22:533
24200991810,6cyclictest0-21swapper/300:10:293
24200991810,5cyclictest32466irq/58-eth022:50:153
2419099181,9cyclictest24307-21latency_hist19:10:012
2419099181,7cyclictest30672-21munin-plugin-st19:55:012
2416899182,11cyclictest0-21swapper/019:40:150
2416899181,3cyclictest131rcu_preempt20:30:010
24168991813,3cyclictest12-21ksoftirqd/019:10:160
2416899181,2cyclictest0-21swapper/020:05:330
2416899181,14cyclictest15832-21cut20:10:220
2416899181,14cyclictest12761-21ssh22:30:250
2416899181,14cyclictest10948-21kthreadcore00:30:220
2420099179,6cyclictest0-21swapper/321:37:233
2420099179,6cyclictest0-21swapper/321:13:273
2420099179,6cyclictest0-21swapper/320:01:163
2420099179,3cyclictest0-21swapper/319:33:483
2420099172,10cyclictest0-21swapper/320:55:173
2420099172,10cyclictest0-21swapper/320:20:143
24200991710,3cyclictest0-21swapper/322:05:143
2416899179,2cyclictest12-21ksoftirqd/020:20:190
2416899173,12cyclictest32561-21needreboot19:15:210
2416899172,7cyclictest27759-21kthreadcore19:45:220
2416899172,13cyclictest2864-21kthreadcore21:05:220
2416899171,2cyclictest131rcu_preempt19:55:160
2416899171,2cyclictest131rcu_preempt19:50:070
24168991712,3cyclictest0-21swapper/022:00:160
24200991612,3cyclictest321rcuc/319:53:113
24200991611,3cyclictest0-21swapper/320:35:173
2416899162,9cyclictest0-21swapper/020:35:150
2416899161,12cyclictest30593-21kthreadcore21:00:220
2416899161,11cyclictest23112-21grep20:50:280
2420099151,8cyclictest0-21swapper/321:05:213
2420099151,8cyclictest0-21swapper/320:45:253
2420099151,8cyclictest0-21swapper/320:45:243
2420099149,3cyclictest0-21swapper/319:35:043
2420099149,3cyclictest0-21swapper/319:26:403
2420099143,8cyclictest0-21swapper/319:40:303
2416899142,9cyclictest24187-21latency_hist19:10:010
2420099111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional