You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-15 - 22:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Mar 15, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2445899625,46cyclictest2309-21ssh11:30:131
2444799613,48cyclictest423-21ssh12:05:120
2445899594,44cyclictest19314-21ssh09:15:131
2444799582,45cyclictest22837-21ssh09:35:120
2446699572,38cyclictest17290-21ssh09:50:112
2446699542,20cyclictest131rcu_preempt08:40:122
2445899545,5cyclictest131rcu_preempt11:10:121
2446699521,31cyclictest26322-21ssh09:55:132
2446699511,28cyclictest20800-21rm10:10:122
2445899513,4cyclictest131rcu_preempt08:50:131
2444799518,5cyclictest131rcu_preempt09:20:110
2445899494,5cyclictest131rcu_preempt07:25:121
24466994717,22cyclictest27-21ksoftirqd/210:15:122
2445899462,38cyclictest5807-21ssh11:50:121
24447994529,10cyclictest0-21swapper/011:55:140
2446699448,19cyclictest27-21ksoftirqd/210:00:132
2447499435,21cyclictest28366irq/58-eth011:55:133
2447499432,24cyclictest28366irq/58-eth011:15:123
2447499422,27cyclictest28366irq/58-eth010:05:123
2445899422,32cyclictest4707-21ssh09:25:121
24447994226,10cyclictest0-21swapper/011:40:130
24447994127,10cyclictest16686-21ssh11:20:120
2447499406,19cyclictest28366irq/58-eth011:35:133
2447499403,23cyclictest28366irq/58-eth011:45:113
2447499403,23cyclictest28366irq/58-eth011:45:103
24466994020,15cyclictest27-21ksoftirqd/209:20:122
2446699392,28cyclictest27478-21ssh12:20:132
2446699392,28cyclictest27478-21ssh12:20:122
24466993918,10cyclictest27-21ksoftirqd/212:30:142
2445899391,32cyclictest22075-21ssh11:05:121
2444799391,31cyclictest11260-21ssh11:35:140
2446699382,19cyclictest20398-21ntp_states11:20:242
2446699372,18cyclictest11304-21ssh12:28:232
24447993720,12cyclictest0-21swapper/011:48:230
24447993720,12cyclictest0-21swapper/011:48:220
2447499363,19cyclictest28366irq/58-eth012:05:103
2446699362,18cyclictest23396-21ssh10:29:242
2445899364,21cyclictest21-21ksoftirqd/110:40:131
2446699353,24cyclictest0-21swapper/209:45:242
2445899341,24cyclictest4238-21ssh10:55:141
24447993417,12cyclictest12-21ksoftirqd/010:04:210
24458993314,14cyclictest21-21ksoftirqd/108:30:101
2445899331,24cyclictest13763-21ssh09:30:131
2444799333,20cyclictest25755-21ssh11:25:140
24447993323,3cyclictest12-21ksoftirqd/011:15:120
24447993210,17cyclictest12-21ksoftirqd/008:00:120
24458993112,10cyclictest633-21Async12:11:531
24458993112,10cyclictest633-21Async12:11:531
2444799312,14cyclictest4087-21ssh12:25:140
2447499301,17cyclictest28366irq/58-eth007:30:143
2446699302,21cyclictest9415-21ssh12:10:142
2446699302,21cyclictest9415-21ssh12:10:132
2446699302,20cyclictest22346irq/40-dwc2_hso07:50:132
2446699302,18cyclictest261rcuc/211:45:122
2446699302,18cyclictest261rcuc/211:45:112
2445899302,22cyclictest21258-21ssh11:04:281
2444799304,18cyclictest7731-21kworker/u8:010:40:140
2444799302,22cyclictest5402-21ssh10:19:230
2444799302,22cyclictest1982-21ssh09:59:220
2444799302,22cyclictest10320-21ssh09:26:240
24447993018,7cyclictest0-21swapper/010:50:140
2447499293,14cyclictest28366irq/58-eth008:35:133
2446699293,19cyclictest0-21swapper/209:26:232
2446699293,11cyclictest0-21swapper/211:09:322
2446699292,18cyclictest22346irq/40-dwc2_hso08:10:142
2446699292,11cyclictest4026-21ntp_192.168.11412:05:232
2445899292,21cyclictest6009-21ssh12:06:231
2445899292,21cyclictest19053-21ssh10:25:261
2445899292,19cyclictest908-21missed_timers10:15:221
2444799292,22cyclictest32444-21ssh10:34:270
2444799291,25cyclictest13177-21ssh11:00:120
2447499284,16cyclictest0-21swapper/310:03:243
2446699283,17cyclictest22346irq/40-dwc2_hso08:55:132
2446699282,21cyclictest32408-21ssh10:34:252
2446699282,21cyclictest16959-21ssh10:43:232
2446699282,20cyclictest22346irq/40-dwc2_hso07:25:132
2446699282,11cyclictest0-21swapper/211:29:332
2445899282,3cyclictest0-21swapper/110:35:121
2444799282,21cyclictest29330-21ssh12:38:230
2444799282,19cyclictest11955-21ssh09:45:240
2447499274,12cyclictest28366irq/58-eth008:25:133
2447499273,18cyclictest0-21swapper/310:34:253
2447499271,13cyclictest28366irq/58-eth011:40:133
24474992710,12cyclictest28366irq/58-eth011:20:123
2446699277,16cyclictest27-21ksoftirqd/209:30:132
2446699275,13cyclictest27-21ksoftirqd/210:20:142
2445899273,2cyclictest131rcu_preempt10:30:131
2445899272,19cyclictest13922-21ssh09:46:231
2445899271,20cyclictest2951-21ssh10:00:141
2444799271,21cyclictest17306-21ssh09:50:120
2447499267,11cyclictest28366irq/58-eth009:20:113
2447499263,17cyclictest0-21swapper/312:38:233
2446699263,19cyclictest22346irq/40-dwc2_hso12:15:122
2446699263,19cyclictest22346irq/40-dwc2_hso12:15:122
2446699262,17cyclictest22346irq/40-dwc2_hso07:35:112
24466992610,10cyclictest22346irq/40-dwc2_hso11:58:242
24458992613,7cyclictest0-21swapper/109:24:171
24474992518,5cyclictest321rcuc/309:27:593
2447499251,12cyclictest28366irq/58-eth009:55:133
2446699252,8cyclictest30364-21ssh12:39:352
24466992512,4cyclictest15925-21cron07:35:012
24466992510,11cyclictest27-21ksoftirqd/212:04:222
2445899253,15cyclictest20783-21ssh10:10:121
2445899252,18cyclictest9986-21diskmemload11:20:231
2445899252,18cyclictest31929-21ssh09:56:221
2444799256,4cyclictest131rcu_preempt07:35:120
24447992511,5cyclictest0-21swapper/007:15:120
2447499243,10cyclictest0-21swapper/310:59:303
24474992418,4cyclictest321rcuc/312:15:133
24474992418,4cyclictest321rcuc/312:15:133
24474992418,4cyclictest321rcuc/309:35:263
2446699243,16cyclictest22346irq/40-dwc2_hso09:00:132
2446699243,14cyclictest22346irq/40-dwc2_hso07:10:132
2446699242,18cyclictest12314-21ssh10:59:282
2446699242,17cyclictest0-21swapper/210:05:242
2446699242,13cyclictest22346irq/40-dwc2_hso08:25:122
2446699242,13cyclictest22346irq/40-dwc2_hso07:40:112
24466992416,5cyclictest20203-21ssh11:40:132
24466992410,9cyclictest22346irq/40-dwc2_hso11:15:042
24458992410,8cyclictest0-21swapper/108:05:121
24447992413,3cyclictest0-21swapper/012:00:150
24447992412,4cyclictest12-21ksoftirqd/009:40:010
2447499238,8cyclictest0-21swapper/309:40:123
2447499234,13cyclictest0-21swapper/309:45:143
24474992317,3cyclictest28366irq/58-eth010:10:113
2446699234,15cyclictest22346irq/40-dwc2_hso09:40:122
2446699233,16cyclictest22346irq/40-dwc2_hso11:35:122
2446699233,13cyclictest22346irq/40-dwc2_hso08:50:132
2446699232,15cyclictest22346irq/40-dwc2_hso08:45:112
2445899231,13cyclictest30654-21ls07:50:131
2444799238,6cyclictest0-21swapper/011:06:130
2444799234,11cyclictest12-21ksoftirqd/012:15:170
2444799234,11cyclictest12-21ksoftirqd/012:15:160
2444799232,17cyclictest18857-21memory10:25:240
24447992312,8cyclictest29392-21cron10:15:000
24447992311,9cyclictest616-21dbus-daemon11:14:300
2444799231,16cyclictest18717-21ssh10:45:130
2447499223,13cyclictest0-21swapper/311:04:293
2447499223,10cyclictest28366irq/58-eth009:50:123
24474992217,3cyclictest321rcuc/311:31:043
2447499221,17cyclictest0-21swapper/312:10:143
2447499221,17cyclictest0-21swapper/312:10:133
24474992210,9cyclictest0-21swapper/309:14:583
2446699222,17cyclictest0-21swapper/211:10:152
2446699221,7cyclictest2798366ssh09:20:012
24466992216,4cyclictest924-21snmpd10:52:452
2446699221,15cyclictest261rcuc/211:30:142
2446699221,13cyclictest15216-21users09:10:322
2446699221,12cyclictest924-21snmpd11:02:452
2446699221,11cyclictest12402-21apt-config08:05:022
2445899222,16cyclictest11438-21ssh11:15:261
2445899222,15cyclictest15877-21ls10:05:271
2445899220,19cyclictest1729-21find07:20:111
2444799228,8cyclictest12-21ksoftirqd/009:31:310
2444799225,6cyclictest12-21ksoftirqd/011:30:230
2444799224,3cyclictest0-21swapper/008:35:130
2444799221,4cyclictest0-21swapper/009:00:130
24447992213,6cyclictest0-21swapper/010:05:110
2447499217,8cyclictest0-21swapper/310:45:133
2447499212,12cyclictest0-21swapper/311:25:133
24474992115,4cyclictest321rcuc/311:54:523
2446699219,8cyclictest0-21swapper/207:55:162
2446699217,12cyclictest13537-21kthreadcore08:05:182
2446699215,10cyclictest22346irq/40-dwc2_hso09:35:132
2446699214,9cyclictest8059-21unixbench_singl08:30:282
2446699212,15cyclictest0-21swapper/208:35:112
2446699212,11cyclictest18687-21apt-get10:45:112
2446699211,16cyclictest5772-21/usr/sbin/munin11:50:142
2446699211,16cyclictest29339-21apt-get07:15:132
2445899219,3cyclictest131rcu_preempt09:35:131
2445899213,10cyclictest0-21swapper/108:00:121
2445899211,18cyclictest25954-21ls09:55:011
2445899211,17cyclictest3123-21cut07:55:171
2445899211,15cyclictest10466-21ssh09:10:131
2445899211,13cyclictest9374-21find12:10:121
2445899211,13cyclictest9374-21find12:10:111
2444799215,10cyclictest0-21swapper/009:17:350
2444799213,14cyclictest9986-21diskmemload09:10:130
2444799212,16cyclictest11574-21hwlatdetect07:30:170
24447992113,6cyclictest9526-21ssh12:10:150
24447992113,6cyclictest9526-21ssh12:10:140
24447992113,4cyclictest0-21swapper/007:45:130
2444799211,15cyclictest887-21ssh10:35:140
24447992110,5cyclictest12-21ksoftirqd/011:53:080
2447499203,12cyclictest0-21swapper/310:25:243
24474992014,4cyclictest0-21swapper/312:22:253
24474992014,4cyclictest0-21swapper/312:22:243
24474992014,4cyclictest0-21swapper/310:39:203
24474992014,4cyclictest0-21swapper/309:19:443
24474992013,4cyclictest0-21swapper/311:09:573
2446699205,7cyclictest27154-21df_inode08:20:142
2446699202,10cyclictest5514-21fschecks_count09:05:152
2446699201,5cyclictest22335-21ls08:15:152
2446699201,16cyclictest29630-21grep07:45:282
2446699201,16cyclictest1631-21kthreadcore10:35:202
2445899202,15cyclictest29364-21kthreadcore08:55:191
2445899202,15cyclictest22171-21latency_hist08:15:021
2445899202,14cyclictest6364-21ssh10:20:141
2445899201,9cyclictest21841-21kthreadcore08:45:211
2445899201,4cyclictest131rcu_preempt11:40:211
2445899201,14cyclictest25525-21latency08:15:231
2444799208,4cyclictest12-21ksoftirqd/007:10:280
2444799207,4cyclictest12-21ksoftirqd/008:15:250
2444799202,6cyclictest53146usb-storage12:30:050
24447992014,4cyclictest0-21swapper/010:55:130
24447992014,4cyclictest0-21swapper/008:55:260
24447992010,8cyclictest17450-21apt-get08:10:130
24447992010,3cyclictest12-21ksoftirqd/010:20:120
2447499197,9cyclictest0-21swapper/312:00:303
2447499196,8cyclictest321rcuc/310:40:143
2447499195,8cyclictest0-21swapper/312:25:153
24474991913,4cyclictest321rcuc/310:15:143
24474991913,4cyclictest0-21swapper/312:31:183
24474991913,4cyclictest0-21swapper/310:51:473
24474991913,4cyclictest0-21swapper/310:22:333
24474991912,4cyclictest0-21swapper/309:31:253
2447499191,14cyclictest28366irq/58-eth008:20:113
2447499191,14cyclictest0-21swapper/309:00:223
2446699191,9cyclictest4009-21kthreadcore07:20:192
2445899199,8cyclictest22648-21threads10:45:291
2445899197,9cyclictest1-21systemd11:59:351
2445899192,15cyclictest5768-21irqcore09:05:181
24458991913,4cyclictest31436-21switchtime12:20:281
24458991913,4cyclictest31436-21switchtime12:20:281
24458991912,4cyclictest21-21ksoftirqd/111:45:201
24458991912,4cyclictest21-21ksoftirqd/111:45:191
2445899191,15cyclictest27255-21fschecks_time08:20:161
24458991911,5cyclictest11210-21/usr/sbin/munin11:35:271
2445899191,14cyclictest23836-21sh12:00:141
2445899191,11cyclictest4066-21find12:25:141
2445899191,10cyclictest18133-21cut07:35:201
2444799197,4cyclictest12-21ksoftirqd/007:25:250
2444799195,3cyclictest131rcu_preempt07:40:120
2444799191,7cyclictest1487-21sshd12:22:230
2444799191,7cyclictest1487-21sshd12:22:220
24447991913,4cyclictest0-21swapper/007:55:300
24447991913,3cyclictest531-21usb-storage08:05:050
24447991913,3cyclictest0-21swapper/007:50:130
2444799191,11cyclictest0-21swapper/008:25:130
2447499182,10cyclictest0-21swapper/307:35:123
24474991812,4cyclictest0-21swapper/311:12:113
2447499181,12cyclictest0-21swapper/309:05:283
2447499181,12cyclictest0-21swapper/308:15:193
2445899184,11cyclictest0-21swapper/107:40:111
2445899181,8cyclictest32531-21/usr/sbin/munin10:50:311
2445899181,13cyclictest0-21swapper/107:15:121
2445899181,11cyclictest29657-21ssh11:25:291
2445899181,11cyclictest14159-21cut08:40:161
2444799188,7cyclictest18627-21apt-get08:45:110
2444799186,4cyclictest12-21ksoftirqd/008:30:210
24447991813,3cyclictest0-21swapper/007:20:140
24447991812,4cyclictest0-21swapper/008:40:260
24447991810,5cyclictest0-21swapper/007:10:020
2447499171,9cyclictest0-21swapper/307:50:113
2447499171,12cyclictest0-21swapper/307:20:153
2447499171,11cyclictest0-21swapper/308:55:013
2446699175,9cyclictest24478-21latency_hist07:10:022
2445899175,10cyclictest15839-21kthreadcore12:30:231
2445899172,12cyclictest557-21apt-get09:00:121
2445899171,12cyclictest31998-21df_inode09:40:151
2445899171,11cyclictest23280-21kthreadcore12:35:201
24447991713,3cyclictest0-21swapper/008:50:240
24447991710,3cyclictest0-21swapper/009:05:130
2447499161,9cyclictest0-21swapper/307:45:163
2447499161,8cyclictest0-21swapper/307:10:013
24474991611,3cyclictest0-21swapper/308:10:123
24474991611,3cyclictest0-21swapper/307:55:143
2445899169,5cyclictest0-21swapper/107:10:161
2445899168,5cyclictest1107-21cut08:25:201
2445899167,7cyclictest13551-21users08:35:301
2445899163,10cyclictest14914-21mailstats07:30:271
2445899161,12cyclictest25976-21fschecks_time07:45:161
24447991611,3cyclictest0-21swapper/008:20:130
2447499157,3cyclictest0-21swapper/308:40:153
2447499152,9cyclictest0-21swapper/307:10:143
2447499151,8cyclictest0-21swapper/308:55:283
2447499151,8cyclictest0-21swapper/307:40:223
24474991510,3cyclictest0-21swapper/308:00:183
24474991510,3cyclictest0-21swapper/307:25:153
24474991510,3cyclictest0-21swapper/307:15:123
2447499149,3cyclictest0-21swapper/308:45:223
2445899141,9cyclictest24523-21latency_hist07:10:011
2447499134,6cyclictest0-21swapper/308:30:123
2447499121,3cyclictest0-21swapper/308:05:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional