You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-23 - 18:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Dec 23, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2275429279,9sleep20-21swapper/207:08:522
2258329178,9sleep00-21swapper/007:06:370
2282728661,9sleep30-21swapper/307:09:503
2276327969,6sleep10-21swapper/107:08:591
2292899216,9cyclictest0-21swapper/211:45:142
2292699210,20cyclictest0-21swapper/110:55:161
22928992020,0cyclictest0-21swapper/212:05:132
22926992018,1cyclictest21922-21grep09:20:161
2291999207,6cyclictest13725-21munin-run11:20:000
2291999200,1cyclictest0-21swapper/008:35:140
22926991917,1cyclictest0-21swapper/110:30:121
2291999197,6cyclictest12124-21awk07:55:000
2291999182,7cyclictest19452-21munin-run12:40:010
2291999174,7cyclictest26943-21munin-run08:25:000
2291999174,7cyclictest26943-21munin-run08:25:000
2291999160,9cyclictest0-21swapper/008:50:150
2293299154,5cyclictest225-21systemd-journal08:13:163
22928991513,1cyclictest15989-21hddtemp_smartct10:15:172
2292699153,6cyclictest0-21swapper/111:55:131
2291999153,9cyclictest1142-21runrttasks09:30:160
2291999151,7cyclictest23343-21sendmail-msp11:39:590
22919991511,2cyclictest13801-21sed12:25:220
2291999150,14cyclictest10181-21munin-plugin-st10:05:000
2293299140,13cyclictest11603-21grep07:50:243
22928991414,0cyclictest0-21swapper/209:30:182
22928991411,2cyclictest23852-21head08:15:262
22928991410,3cyclictest13192-21cat09:00:272
2292699148,5cyclictest38150irq/131-snd_hda08:20:001
2292699142,8cyclictest6170-21unixbench_singl11:00:271
2292699140,12cyclictest0-21swapper/109:48:411
22919991410,2cyclictest807-21cut09:45:120
22919991410,2cyclictest22127-21cron10:30:000
22919991410,2cyclictest17950-21wc09:10:260
2291999140,9cyclictest27652-21idleruntime-cro07:20:000
22932991310,2cyclictest32205-21cat09:40:243
22932991310,2cyclictest28074-21dump-pmu-power11:49:593
2292899139,3cyclictest27003-21cat08:25:002
2292899139,3cyclictest27003-21cat08:25:002
2292899139,2cyclictest19056-21tr09:15:142
22928991310,2cyclictest30048-21grep11:50:272
22928991310,2cyclictest2407-21grep09:45:242
22928991310,2cyclictest2167-21grep10:55:172
22928991310,2cyclictest14001-21cat07:55:252
22928991310,2cyclictest10611-21cat08:55:262
22926991310,2cyclictest538-21cat09:45:001
22926991310,2cyclictest13707-21tr12:25:201
22926991310,2cyclictest10958-21wc07:50:171
2292699130,12cyclictest0-21swapper/111:15:121
2291999139,2cyclictest20924-21sh11:35:000
2291999135,7cyclictest22259-21memory09:20:180
2291999132,9cyclictest448-21apt-get07:30:150
2291999131,6cyclictest25707-21munin-run11:45:000
22919991310,2cyclictest8544-21grep11:05:260
22919991310,2cyclictest6194-21grep11:00:270
22919991310,2cyclictest4509-21wc12:05:250
22919991310,2cyclictest3616-21head07:35:170
22919991310,2cyclictest32199-21tr10:50:180
22919991310,2cyclictest30813-21grep11:55:130
22919991310,2cyclictest27442-21grep10:40:130
22919991310,2cyclictest10714-21tail08:55:270
2293299129,2cyclictest3809-21tr12:05:173
2293299129,2cyclictest29823-21grep10:45:143
2293299129,2cyclictest27621-21grep09:30:283
2293299129,2cyclictest26894-21sendmail10:40:003
2293299129,2cyclictest25247-21grep11:40:273
2293299129,2cyclictest24832-21grep08:20:153
2293299129,2cyclictest24832-21grep08:20:153
2293299129,2cyclictest24533-21sleep10:35:003
2293299129,2cyclictest22818-21grep11:35:303
2293299129,2cyclictest22464-21head09:20:203
2293299129,2cyclictest19646-21cat12:40:013
2293299129,2cyclictest13090-21grep10:10:143
2293299129,2cyclictest10693-21grep10:05:153
2293299126,5cyclictest20624-21apache208:35:133
22932991210,1cyclictest30931-21grep11:55:143
22932991210,1cyclictest13744-21awk11:20:003
22932991210,1cyclictest0-21swapper/312:20:123
2293299120,11cyclictest23935-21grep07:10:193
2292899129,2cyclictest8855-21grep10:00:172
2292899129,2cyclictest7124-21grep11:05:142
2292899129,2cyclictest6665-21cat09:55:202
2292899129,2cyclictest606-21grep08:35:202
2292899129,2cyclictest31549-21grep07:25:202
2292899129,2cyclictest2987-21sort08:40:202
2292899129,2cyclictest27939-21grep08:25:182
2292899129,2cyclictest25340-21sed09:25:282
2292899129,2cyclictest25259-21cron07:15:002
2292899129,2cyclictest22899-21sed11:35:302
2292899129,2cyclictest19522-21head11:30:172
2292899129,2cyclictest15258-21grep09:05:232
2292899129,2cyclictest14658-21cut11:20:172
2292899128,3cyclictest23545-21grep10:30:192
2292899128,3cyclictest21892-21sh08:15:002
2292899128,3cyclictest18876-21wc12:35:262
2292899128,3cyclictest10967-21grep12:20:152
2292899120,11cyclictest8883-21wc07:45:222
2292699129,2cyclictest9145-21chrt12:15:221
2292699129,2cyclictest9079-21wc11:10:001
2292699129,2cyclictest7777-21cron10:00:001
2292699129,2cyclictest31049-21grep07:25:151
2292699129,2cyclictest30254-21grep10:45:161
2292699129,2cyclictest27396-21tr10:40:131
2292699129,2cyclictest21745-21grep10:25:281
2292699129,2cyclictest18983-21grep12:35:271
2292699129,2cyclictest17156-21tail09:10:171
2292699129,2cyclictest16866-21tail10:15:261
2292699129,2cyclictest1631-21basename10:55:001
2292699129,2cyclictest15203-21tr12:30:161
2292699129,2cyclictest15013-21tr08:00:121
22926991210,1cyclictest5024-21grep08:45:171
22926991210,1cyclictest3207-21basename09:50:131
22926991210,1cyclictest27219-21tr09:30:241
22926991210,1cyclictest21896-21turbostat.cron08:15:001
2292699120,9cyclictest0-21swapper/107:55:141
2291999129,2cyclictest7135-21head12:10:260
2291999129,2cyclictest7135-21head12:10:260
2291999129,2cyclictest29331-21wc09:35:200
2291999129,2cyclictest27804-21sort08:25:180
2291999129,2cyclictest2621-21grep08:40:180
2291999129,2cyclictest20949-21ls09:20:000
2291999129,2cyclictest18032-21grep11:25:260
2291999129,2cyclictest14797-21grep09:05:180
2291999129,2cyclictest12813-21tr07:55:150
2291999129,2cyclictest11438-21wc10:05:200
2291999128,2cyclictest24346-21grep09:25:190
2291999128,2cyclictest19455-21cron08:10:000
2291999121,6cyclictest26927-21awk10:40:000
2291999121,6cyclictest225-21systemd-journal09:40:000
2293299119,1cyclictest6151-21tail12:10:173
2293299119,1cyclictest6151-21tail12:10:173
2293299119,1cyclictest3580-21grep10:55:283
2293299119,1cyclictest28425-21grep08:25:233
2293299119,1cyclictest17596-21tr08:05:153
2293299119,1cyclictest16838-21grep10:15:263
2293299119,1cyclictest1176-21sed10:50:303
2293299118,2cyclictest9390-21fgrep10:00:223
2293299118,2cyclictest6917-21grep08:50:163
2293299118,2cyclictest31683-21ls08:35:003
2293299118,2cyclictest29599-21wc11:50:233
2293299118,2cyclictest15737-21grep09:05:283
2293299118,2cyclictest13333-21head12:25:173
2293299117,3cyclictest39-21ksoftirqd/312:20:003
2293299112,4cyclictest14545-21sshd07:58:163
2293299112,3cyclictest19455-21sshd11:30:173
2293299112,2cyclictest141rcu_preempt07:45:003
2293299110,10cyclictest9851-21tune2fs11:10:163
2292899118,2cyclictest869-21expr07:30:162
2292899118,2cyclictest6083-21grep12:10:162
2292899118,2cyclictest6083-21grep12:10:162
2292899118,2cyclictest5274-21head11:00:182
2292899118,2cyclictest4960-21cron07:40:002
2292899118,2cyclictest4292-21chrt08:45:122
2292899118,2cyclictest30222-21grep10:45:162
2292899118,2cyclictest28688-21tr07:20:182
2292899118,2cyclictest25772-21grep07:15:132
2292899118,2cyclictest20039-21tr10:25:152
2292899118,2cyclictest19462-21sh08:10:002
2292899118,2cyclictest17997-21sed11:25:262
2292899118,2cyclictest15202-21grep12:30:162
2292899118,2cyclictest13199-21cat10:10:152
2292899115,2cyclictest225-21systemd-journal08:51:172
2292899113,2cyclictest225-21systemd-journal11:55:002
2292699119,1cyclictest7375-21grep08:50:181
2292699118,2cyclictest9271-21tr08:55:151
2292699118,2cyclictest8768-21awk07:45:211
2292699118,2cyclictest5901-21uniq07:40:151
2292699118,2cyclictest4989-21chrt12:07:491
2292699118,2cyclictest29750-21chrt07:21:171
2292699118,2cyclictest27648-21cron07:20:001
2292699118,2cyclictest2742-21sort07:35:001
2292699118,2cyclictest26677-21tail11:45:171
2292699118,2cyclictest25526-21cut10:35:161
2292699118,2cyclictest18290-21chrt11:27:341
2292699118,2cyclictest15348-21wc09:05:231
2292699118,2cyclictest12311-21cron12:25:001
2292699113,4cyclictest225-21systemd-journal11:53:211
2292699111,6cyclictest24840-21tr08:20:151
2292699111,6cyclictest24840-21tr08:20:151
2292699110,10cyclictest3584-21tail07:35:171
2292699110,10cyclictest10894-21cat10:05:151
2291999118,2cyclictest9462-21grep11:10:140
2291999118,2cyclictest4707-21grep09:50:250
2291999118,2cyclictest4540-21tr08:45:150
2291999118,2cyclictest3487-21grep10:55:270
2291999118,2cyclictest31371-21wc07:25:180
2291999118,2cyclictest30096-21grep10:45:150
2291999118,2cyclictest29853-21cut08:30:140
2291999118,2cyclictest29589-21grep07:20:270
2291999118,2cyclictest23132-21chrt07:10:140
2291999118,2cyclictest22365-21taskset10:30:090
2291999118,2cyclictest20467-21grep08:10:170
2291999118,2cyclictest19715-21sleep10:25:000
2291999118,2cyclictest16627-21tail08:00:240
2291999118,2cyclictest15001-21ls10:15:000
2291999118,2cyclictest1365-21cut12:00:170
2291999117,2cyclictest8844-21sed07:45:220
2291999116,3cyclictest26186-21tr11:45:150
2291999113,4cyclictest225-21systemd-journal09:05:000
2291999110,10cyclictest7380-21idleruntime-cro07:45:000
2293299108,2cyclictest6420-21/usr/sbin/munin08:50:013
2293299108,2cyclictest23564-21python08:15:243
2293299108,1cyclictest9720-21uniq08:55:173
2293299108,1cyclictest3552-21cat09:50:143
2293299108,1cyclictest3120-21ls08:40:223
2293299108,1cyclictest23895-21grep09:25:163
2293299108,1cyclictest15212-21tr08:00:133
2293299108,1cyclictest13321-21grep09:00:273
2293299107,2cyclictest4686-21cat11:00:153
2293299107,2cyclictest29012-21tune2fs09:35:183
2293299107,2cyclictest21539-21wc10:25:273
2293299107,2cyclictest14110-21wget11:20:153
2293299106,3cyclictest7917-21cat11:05:193
2293299105,3cyclictest0-21swapper/307:25:003
2293299101,6cyclictest27178-21apt-get10:40:123
2293299100,8cyclictest0-21swapper/312:30:153
2293299100,10cyclictest20671-21chrt09:16:343
2292899108,1cyclictest8271-21cat12:15:152
2292899108,1cyclictest4013-21kernelversion09:50:182
2292899108,1cyclictest319362sleep210:50:172
2292899108,1cyclictest29678-21grep09:35:232
2292899108,1cyclictest27375-21grep10:40:132
2292899108,1cyclictest25095-21cat10:35:142
2292899108,1cyclictest125472sleep212:25:092
2292899108,1cyclictest12203-21fschecks_count11:15:152
2292899108,1cyclictest10026-21sh11:10:172
2292899108,1cyclictest10005-21aten2_r7power_c07:50:122
2292899107,2cyclictest31693-21munin-plugin-st08:35:002
2292899107,2cyclictest30796-21grep09:40:132
2292899107,2cyclictest2354-21sed12:00:252
2292899107,2cyclictest22028-21sh09:20:162
2292899107,2cyclictest17014-21cut09:10:162
2292899107,2cyclictest14979-21apt08:00:112
2292899107,2cyclictest11099-21sort10:05:172
2292899100,9cyclictest5246-21apache_accesses07:40:112
2292899100,9cyclictest24178-21entropy11:40:172
2292699108,1cyclictest28163-21wc08:25:201
2292699108,1cyclictest23978-21expr09:25:161
2292699108,1cyclictest22838-21grep11:35:301
2292699108,1cyclictest17800-21grep10:20:161
2292699108,1cyclictest15408-21swap11:20:241
2292699108,1cyclictest12052-21cat09:00:161
2292699107,2cyclictest818-21uname08:35:221
2292699107,2cyclictest814-21cut12:00:151
2292699107,2cyclictest235882sleep111:40:131
2292699107,2cyclictest23158-21bc07:10:141
2292699107,2cyclictest2073-21wc08:40:141
2292699107,2cyclictest18915-21sed08:05:261
2292699107,2cyclictest18867-21cut09:15:131
2292699106,3cyclictest6035-21cut12:10:161
2292699106,3cyclictest6035-21cut12:10:161
2292699106,3cyclictest30253-21sort08:30:161
2292699106,2cyclictest12958-21cut10:10:141
2292699105,2cyclictest225-21systemd-journal11:30:171
2291999108,1cyclictest15506-21ls10:15:150
2291999108,1cyclictest10240-21wget12:20:120
2291999107,2cyclictest8251-21expr12:15:150
2291999107,2cyclictest5583-21sort09:55:010
2291999107,2cyclictest24379-21sed08:20:000
2291999107,2cyclictest17031-21taskset12:34:440
2291999100,9cyclictest28413-21grep11:50:150
2291999100,9cyclictest159442sleep011:22:570
229329998,1cyclictest7070-21chrt09:55:253
229329998,1cyclictest2612-21seq09:47:243
229329998,1cyclictest16601-21wc09:10:153
229329998,1cyclictest1526-21basename12:00:183
229329997,1cyclictest7630-21apt-get07:45:143
229329997,1cyclictest30639-21expr07:25:133
229329997,1cyclictest25796-21grep07:15:133
229329997,1cyclictest2142-21grep07:30:273
229329997,1cyclictest16592-21cpuspeed_turbos11:25:143
229329996,2cyclictest19290-21grep10:20:273
229329990,9cyclictest4605-21seq07:35:303
229289996,2cyclictest18386-21smartctl10:20:192
229269997,1cyclictest9209-21cat11:10:111
229269997,1cyclictest28625-21expr09:35:161
229269996,2cyclictest8680-21grep10:00:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional