You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 14:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Sat Jul 12, 2025 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17096996549,13cyclictest23346irq/40-dwc2_hso22:35:142
17104996241,14cyclictest32466irq/58-eth023:05:133
1709699573,33cyclictest31584-21ssh22:45:142
17096995719,29cyclictest27-21ksoftirqd/222:50:172
17104995440,7cyclictest32466irq/58-eth023:55:143
17087995336,10cyclictest22016-21kworker/u8:100:10:141
17087995336,10cyclictest22016-21kworker/u8:100:10:141
1708799503,10cyclictest131rcu_preempt20:00:141
17087994724,12cyclictest131rcu_preempt21:20:191
17087994724,12cyclictest131rcu_preempt21:20:181
1707699472,35cyclictest32276-21ssh23:05:120
1707699471,2cyclictest0-21swapper/021:10:130
1707699461,3cyclictest0-21swapper/021:00:150
1709699442,22cyclictest738-21ssh00:25:152
1708799446,32cyclictest9257-21ssh23:30:121
1710499439,21cyclictest32466irq/58-eth023:25:123
1707699432,36cyclictest0-21swapper/021:30:130
1707699431,2cyclictest7922-21apt-get00:30:160
1707699421,3cyclictest0-21swapper/020:05:150
1708799413,26cyclictest22395-21ssh22:20:141
17087994121,11cyclictest131rcu_preempt22:15:171
1707699411,5cyclictest0-21swapper/020:25:130
1707699411,3cyclictest0-21swapper/022:25:130
1707699403,29cyclictest17682-21sh23:35:190
1707699402,2cyclictest14915-21apt-get22:35:140
1710499397,13cyclictest32466irq/58-eth023:20:133
1707699391,3cyclictest0-21swapper/020:35:150
1707699391,35cyclictest0-21swapper/022:50:160
1707699391,2cyclictest22379-21apt-get22:20:150
1707699391,2cyclictest13994-21apt-get22:15:160
1707699390,6cyclictest24703-21ssh23:57:520
1707699390,5cyclictest727-21apt-get00:25:150
17087993817,9cyclictest131rcu_preempt22:10:141
17087993813,20cyclictest21-21ksoftirqd/122:45:131
1707699381,3cyclictest23308-21apt-get22:40:160
1707699381,3cyclictest23308-21apt-get22:40:150
1707699381,33cyclictest28808-21ssh21:45:140
1707699381,2cyclictest0-21swapper/022:00:150
1709699371,29cyclictest28074-21ssh21:25:142
1708799372,28cyclictest14891-21ssh00:35:131
1707699372,7cyclictest0-21swapper/020:55:130
1707699372,31cyclictest0-21swapper/020:50:120
1707699371,3cyclictest0-21swapper/022:55:140
1707699371,3cyclictest0-21swapper/021:40:140
1707699371,2cyclictest0-21swapper/023:20:130
1707699370,3cyclictest11313-21apt-get20:45:140
17104993615,14cyclictest0-21swapper/322:05:133
17087993630,4cyclictest0-21swapper/122:50:151
1707699361,2cyclictest0-21swapper/022:05:130
1707699361,2cyclictest0-21swapper/019:40:160
17096993519,11cyclictest27-21ksoftirqd/222:05:142
1708799354,2cyclictest0-21swapper/120:45:131
1708799352,27cyclictest15697-21ssh22:55:141
1708799352,23cyclictest17819-21ssh00:15:141
1707699356,26cyclictest2677-21diskmemload00:00:310
1707699355,26cyclictest0-21swapper/000:15:130
1707699351,3cyclictest0-21swapper/021:35:150
1707699351,3cyclictest0-21swapper/021:25:150
1707699351,3cyclictest0-21swapper/019:10:120
1707699351,2cyclictest346-21apt-get20:00:130
1707699351,2cyclictest19739-21apt-get21:20:180
1707699351,2cyclictest19739-21apt-get21:20:180
1707699351,2cyclictest0-21swapper/023:15:130
1707699351,10cyclictest32014-21cut19:25:190
1707699341,6cyclictest11395-21cat23:50:230
1707699340,5cyclictest25825-21apt-get23:40:130
1708799335,20cyclictest21-21ksoftirqd/120:50:151
1707699331,9cyclictest30066-21ssh23:02:230
1707699331,8cyclictest6615-21diskstats20:40:170
1707699331,8cyclictest5953-21/usr/sbin/munin22:10:190
1707699331,6cyclictest5109-21cut00:05:210
1707699331,4cyclictest31562-21apt-get22:45:130
1707699331,3cyclictest0-21swapper/019:15:160
1710499326,14cyclictest0-21swapper/300:25:143
17104993224,3cyclictest32466irq/58-eth023:40:123
1709699327,6cyclictest0-21swapper/222:15:172
1709699322,24cyclictest9266-21ssh23:30:132
1707699322,22cyclictest9280-21ssh23:30:140
1707699321,8cyclictest6049-21ssh23:25:590
1707699321,8cyclictest18325-21memory00:35:240
1707699321,7cyclictest21829-21ntp_192.168.11419:45:280
1707699321,7cyclictest16357-21ssh23:15:030
1707699321,6cyclictest29330-21apt20:30:110
1707699321,6cyclictest0-21swapper/020:20:150
1707699321,4cyclictest31460-21ntp_states19:55:290
1707699321,3cyclictest0-21swapper/020:10:140
1707699320,3cyclictest4693-21ssh21:50:150
17096993114,2cyclictest0-21swapper/200:10:142
17096993114,2cyclictest0-21swapper/200:10:132
1708799311,23cyclictest24017-21ssh23:00:131
1707699311,8cyclictest15970-21cut20:15:200
1707699311,8cyclictest15970-21cut20:15:190
1707699311,7cyclictest13169-21uniq21:55:200
1707699311,7cyclictest13169-21uniq21:55:190
1707699311,6cyclictest6672-21entropy22:30:160
1707699311,6cyclictest29409-21kthreadcore19:20:190
1707699311,5cyclictest1716-21/usr/sbin/munin23:45:110
1707699311,3cyclictest25522-21apt-get00:20:120
1707699311,3cyclictest0-21swapper/021:05:140
1710499307,17cyclictest32466irq/58-eth021:10:143
1710499304,16cyclictest32466irq/58-eth021:20:183
1710499304,16cyclictest32466irq/58-eth021:20:173
1709699303,21cyclictest27-21ksoftirqd/221:20:172
1709699303,21cyclictest27-21ksoftirqd/221:20:162
1707699301,4cyclictest10236-21apt-get00:10:140
1707699301,4cyclictest10236-21apt-get00:10:140
1707699301,25cyclictest25106-21kthreadcore19:50:210
1707699300,6cyclictest17699-21ssh21:17:430
1707699300,6cyclictest17699-21ssh21:17:420
1709699296,14cyclictest23346irq/40-dwc2_hso19:10:132
1708799294,22cyclictest4699-21ssh21:50:171
1708799291,22cyclictest16561-21ssh23:15:161
1707699291,6cyclictest8783-21/usr/sbin/munin19:35:130
1709699283,18cyclictest23346irq/40-dwc2_hso19:20:122
1709699283,17cyclictest23346irq/40-dwc2_hso22:25:132
1709699283,17cyclictest23346irq/40-dwc2_hso22:20:122
1709699283,16cyclictest23346irq/40-dwc2_hso21:00:152
1709699282,16cyclictest23346irq/40-dwc2_hso20:30:132
1709699281,20cyclictest10095-21ssh23:50:162
1707699281,6cyclictest4014-21awk19:30:110
1708799273,17cyclictest14460-21apt-get22:35:031
1707699271,5cyclictest17094-21latency_hist19:10:020
1707699271,5cyclictest17094-21latency_hist19:10:020
1710499263,13cyclictest32466irq/58-eth020:55:133
1710499262,14cyclictest32466irq/58-eth000:35:133
17104992611,10cyclictest32466irq/58-eth021:45:153
1709699262,18cyclictest0-21swapper/223:05:132
1709699262,10cyclictest0-21swapper/200:20:132
1708799261,21cyclictest23316-21ssh22:40:161
1708799261,21cyclictest23316-21ssh22:40:151
1710499252,17cyclictest0-21swapper/322:50:033
17104992519,4cyclictest321rcuc/300:12:363
17104992519,4cyclictest321rcuc/300:12:353
1710499251,16cyclictest32466irq/58-eth022:40:163
1710499251,16cyclictest32466irq/58-eth022:40:153
1709699259,13cyclictest23618-21cron23:00:002
17096992515,5cyclictest17064-21cyclictest21:55:172
17096992515,5cyclictest17064-21cyclictest21:55:162
17096992513,6cyclictest20368-21ssh00:37:232
1708799251,18cyclictest24914-21ssh23:20:141
1708799251,16cyclictest6556-21ssh22:30:141
1710499244,11cyclictest32466irq/58-eth022:35:133
17104992418,4cyclictest321rcuc/321:40:353
1709699246,12cyclictest23346irq/40-dwc2_hso20:35:142
1709699242,15cyclictest23346irq/40-dwc2_hso20:25:132
1709699241,9cyclictest11336-21ssh21:15:152
1709699241,9cyclictest11336-21ssh21:15:142
17096992412,5cyclictest0-21swapper/221:40:252
1709699241,21cyclictest0-21swapper/219:40:182
1709699241,18cyclictest5625-21rm22:10:152
1709699241,13cyclictest3891-21ssh21:30:152
17087992413,8cyclictest612-21dbus-daemon21:47:241
1710499233,12cyclictest0-21swapper/323:35:193
1709699235,9cyclictest22358-21tail19:15:192
1709699232,6cyclictest0-21swapper/221:05:132
1709699232,14cyclictest23346irq/40-dwc2_hso00:30:142
17096992311,10cyclictest19458-21cron20:20:012
17096992311,10cyclictest19458-21cron20:20:002
1708799235,12cyclictest877-21ssh23:25:141
1708799232,17cyclictest26209-21latency_hist00:00:021
1708799232,16cyclictest2116-21ssh22:05:341
1708799232,13cyclictest20477-21ssh21:40:121
1708799231,2cyclictest131rcu_preempt00:05:121
17104992217,3cyclictest321rcuc/323:51:533
17104992217,3cyclictest321rcuc/320:35:133
17104992216,4cyclictest321rcuc/321:39:283
1709699229,10cyclictest26167-21if_eth123:40:172
1709699225,14cyclictest11076-21cron20:45:002
1709699223,14cyclictest23346irq/40-dwc2_hso20:00:152
1709699223,10cyclictest25643-21ps23:20:202
1709699222,14cyclictest23346irq/40-dwc2_hso23:25:122
1709699221,9cyclictest1223166ssh21:35:162
1709699221,16cyclictest10026-21cat00:10:012
1708799222,2cyclictest131rcu_preempt20:30:121
1708799222,16cyclictest23226-21apt-get19:50:141
1708799221,16cyclictest4732-21ls00:25:301
17104992115,4cyclictest321rcuc/300:00:123
1709699219,9cyclictest1-21systemd19:37:052
1709699216,6cyclictest26127-21kthreadcore19:50:232
1709699214,14cyclictest21065-21cut20:55:152
1709699212,10cyclictest29343-21kthreadcore19:55:222
1709699211,9cyclictest5785-21timerwakeupswit23:45:282
17096992114,5cyclictest916-21snmpd22:30:142
17096992113,5cyclictest21760-21ssh22:00:202
1709699211,12cyclictest21227-21ssh23:35:292
1709699211,11cyclictest19445-21cut23:15:232
1709699211,10cyclictest21935-21wc19:45:292
1709699211,10cyclictest16199-21latency_hist23:15:022
17096992110,4cyclictest0-21swapper/200:00:142
1708799219,3cyclictest25050-21irqrtprio20:25:171
1708799212,16cyclictest31612-21ls19:55:311
1708799212,13cyclictest7941-21ssh00:30:161
1708799211,9cyclictest18368-21irqstats23:35:241
1708799211,17cyclictest30731-21apt-get22:25:111
1708799211,13cyclictest16173-21latency_hist23:15:011
1708799211,10cyclictest2677-21diskmemload23:42:431
1710499202,14cyclictest0-21swapper/322:10:163
17104992015,3cyclictest321rcuc/323:14:273
17104992013,5cyclictest321rcuc/321:25:163
1709699208,5cyclictest683-21in:imuxsock21:55:012
1709699207,6cyclictest21118-21cut20:20:222
1709699205,6cyclictest5961-21kthreadcore21:10:232
1709699201,9cyclictest8606-21latency_hist19:35:012
1709699201,9cyclictest18158-21kthreadcore20:50:222
1709699201,16cyclictest23872-21ps22:40:212
1709699201,16cyclictest23872-21ps22:40:202
1709699201,16cyclictest0-21swapper/220:05:132
1709699201,10cyclictest25972-21ps23:00:212
1709699201,10cyclictest20296-21ps23:55:222
1709699201,10cyclictest14457-21kthreadcore20:45:242
1708799205,2cyclictest131rcu_preempt21:35:151
1708799204,2cyclictest131rcu_preempt20:20:161
1708799201,3cyclictest131rcu_preempt21:00:151
1710499198,6cyclictest0-21swapper/322:20:073
1710499197,5cyclictest0-21swapper/323:15:163
17104991914,3cyclictest321rcuc/322:46:113
17104991914,3cyclictest321rcuc/320:00:283
17104991913,4cyclictest0-21swapper/323:49:343
1710499191,14cyclictest32466irq/58-eth023:30:133
1710499191,13cyclictest0-21swapper/320:40:173
1710499191,12cyclictest0-21swapper/322:00:023
1710499191,12cyclictest0-21swapper/322:00:013
1710499191,12cyclictest0-21swapper/320:50:163
1710499191,11cyclictest0-21swapper/320:05:303
17104991910,6cyclictest0-21swapper/321:54:333
1709699192,9cyclictest10127-21fschecks_count20:10:162
1709699191,9cyclictest31823-21fschecks_time19:25:172
1709699191,9cyclictest29184-21irqcore21:45:182
1709699191,8cyclictest18000-21fschecks_time00:15:182
1708799199,8cyclictest5203-21apt-get20:05:151
1708799191,3cyclictest131rcu_preempt20:10:121
17087991911,6cyclictest9088-21sshd21:12:191
1708799191,12cyclictest13482-21ls23:50:241
1708799190,2cyclictest131rcu_preempt20:15:131
1708799190,2cyclictest131rcu_preempt20:15:121
1710499182,9cyclictest0-21swapper/323:00:123
1710499182,10cyclictest0-21swapper/321:15:163
1710499182,10cyclictest0-21swapper/321:15:153
1710499182,10cyclictest0-21swapper/320:10:143
17104991812,4cyclictest0-21swapper/322:57:463
17104991812,4cyclictest0-21swapper/322:31:143
17104991812,4cyclictest0-21swapper/322:28:073
17104991812,4cyclictest0-21swapper/321:30:113
17104991812,4cyclictest0-21swapper/320:20:183
17104991812,4cyclictest0-21swapper/300:33:073
17104991812,4cyclictest0-21swapper/300:21:143
17104991812,4cyclictest0-21swapper/300:17:433
1710499181,13cyclictest0-21swapper/322:15:323
1710499181,12cyclictest0-21swapper/322:00:253
1710499181,11cyclictest0-21swapper/300:05:283
1708799188,7cyclictest21-21ksoftirqd/121:00:021
1708799187,8cyclictest2778-21date21:10:021
1708799185,10cyclictest3397-21kthreadcore23:45:201
1708799183,11cyclictest0-21swapper/100:00:141
1708799181,7cyclictest11457-21diskstats21:15:171
1708799181,7cyclictest11457-21diskstats21:15:161
17087991812,3cyclictest21-21ksoftirqd/121:30:131
17087991811,5cyclictest16025-21cut21:55:261
17087991811,5cyclictest16025-21cut21:55:261
1708799181,14cyclictest13630-21apt-get19:40:161
17087991810,6cyclictest1-21systemd23:07:381
17087991810,5cyclictest28328-21idleruntime21:25:191
1710499172,10cyclictest0-21swapper/321:00:153
1710499172,10cyclictest0-21swapper/319:55:183
1710499172,10cyclictest0-21swapper/319:10:133
1710499171,9cyclictest0-21swapper/319:35:013
1710499171,13cyclictest0-21swapper/319:15:193
1710499171,12cyclictest0-21swapper/321:05:133
1709699170,9cyclictest17141-21latency_hist19:10:022
1709699170,9cyclictest17141-21latency_hist19:10:012
1708799179,6cyclictest18914-21irqrtprio19:45:221
1708799178,7cyclictest7403-21ntp_192.168.11419:30:231
1708799177,7cyclictest4733-21kthreadcore20:35:221
1708799177,7cyclictest25464-21ntp_kernel_err19:15:261
1708799176,8cyclictest0-21swapper/100:20:131
1708799171,8cyclictest19820-21cut19:10:211
1708799171,13cyclictest29347-21kthreadcore19:20:201
1708799171,12cyclictest24303-21cut22:00:241
1708799171,12cyclictest13373-21cron19:40:011
1710499162,10cyclictest0-21swapper/319:50:153
1710499162,10cyclictest0-21swapper/319:45:183
17104991611,3cyclictest0-21swapper/320:30:283
17104991611,3cyclictest0-21swapper/319:20:213
1708799168,5cyclictest7009-21irqcore20:40:201
1708799161,11cyclictest32141-21sed19:25:191
1710499152,10cyclictest0-21swapper/320:15:143
1710499152,10cyclictest0-21swapper/320:15:133
1710499151,8cyclictest0-21swapper/319:40:173
1710499151,11cyclictest0-21swapper/320:25:173
17104991510,3cyclictest0-21swapper/319:38:253
1710499149,3cyclictest0-21swapper/320:45:293
1710499149,3cyclictest0-21swapper/319:25:293
1708799142,10cyclictest17064-21cyclictest19:10:021
1708799142,10cyclictest17064-21cyclictest19:10:011
171049991,6cyclictest0-21swapper/319:10:023
171049991,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional