You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 09:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Aug 31, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3093229143,43sleep10-21swapper/119:05:221
3197529069,16sleep00-21swapper/019:09:550
3172628660,8sleep30-21swapper/319:06:413
3183728371,8sleep20-21swapper/219:08:092
50622570,0sleep30-21swapper/322:34:523
3206899260,26cyclictest0-21swapper/200:38:432
3205899230,2cyclictest0-21swapper/021:59:410
3205899230,1cyclictest0-21swapper/021:37:170
3206899225,14cyclictest2592-31find00:10:202
3206899220,22cyclictest0-21swapper/200:15:382
3206899220,21cyclictest0-21swapper/221:38:512
32063992221,1cyclictest0-21swapper/123:30:141
3206399220,2cyclictest0-21swapper/122:12:431
3205899225,2cyclictest141rcu_preempt21:30:230
3205899220,9cyclictest0-21swapper/021:28:100
32072992120,1cyclictest0-21swapper/320:20:153
3207299210,2cyclictest0-21swapper/300:30:153
3206899210,20cyclictest0-21swapper/222:36:442
3206899210,20cyclictest0-21swapper/200:28:002
3205899218,12cyclictest14102-21/usr/sbin/munin22:40:200
3205899212,2cyclictest19884-21munin-run20:59:590
32058992119,2cyclictest0-21swapper/023:02:460
3205899211,4cyclictest8562-21ssh23:35:000
3206899200,20cyclictest0-21swapper/223:39:502
3206399200,0cyclictest0-21swapper/121:30:461
3206899190,19cyclictest0-21swapper/222:23:302
32063991917,1cyclictest3752-21cstates21:55:141
3205899197,1cyclictest0-21swapper/000:01:100
3205899195,7cyclictest29315-21munin-run22:30:000
3205899195,2cyclictest31837-21apt-get23:10:120
3205899194,3cyclictest4468-21apache_accesses22:15:120
32058991910,8cyclictest24727-21grep23:05:190
3205899190,5cyclictest0-21swapper/023:45:200
3205899190,4cyclictest0-21swapper/022:55:140
3206899180,18cyclictest0-21swapper/222:09:422
32063991813,4cyclictest38150irq/131-snd_hda21:21:201
3206399180,2cyclictest0-21swapper/121:15:431
3206399180,14cyclictest0-21swapper/123:05:551
3206399180,0cyclictest0-21swapper/122:25:001
3205899189,8cyclictest5118-21grep00:30:190
3206899177,0cyclictest0-21swapper/200:04:422
3206899170,17cyclictest0-21swapper/221:25:192
3206399170,4cyclictest0-21swapper/123:59:111
32058991712,4cyclictest32512-21ls20:15:240
32058991710,6cyclictest11274-21ssh21:40:170
3207299160,2cyclictest225-21systemd-journal00:08:133
3206899160,16cyclictest0-21swapper/222:52:122
32063991616,0cyclictest0-21swapper/123:23:591
3206399161,14cyclictest0-21swapper/120:35:131
3206399160,15cyclictest0-21swapper/122:51:341
3205899169,6cyclictest2845-21sort00:10:000
3205899168,2cyclictest25927-21wc20:05:010
3205899168,2cyclictest25927-21wc20:05:000
3205899168,2cyclictest20021-21basename21:45:170
3205899168,1cyclictest12091-21seq22:38:510
3205899164,7cyclictest24244-21awk23:25:010
3205899163,6cyclictest18735-21perf19:50:000
3205899161,14cyclictest0-21swapper/000:25:130
3205899160,1cyclictest0-21swapper/023:28:260
209742160,1sleep20-21swapper/223:41:372
32072991511,3cyclictest3591-21pmu-power21:35:243
3206899150,1cyclictest0-21swapper/223:03:182
3206899150,15cyclictest0-21swapper/222:59:512
3206899150,14cyclictest0-21swapper/222:48:472
3206399152,11cyclictest38150irq/131-snd_hda20:50:131
3206399150,1cyclictest0-21swapper/123:11:121
3205899158,6cyclictest11688-21grep20:40:190
3205899158,2cyclictest959-21tail19:10:190
3205899158,2cyclictest766-21ssh22:31:300
3205899158,2cyclictest4070-21ssh00:10:190
3205899158,2cyclictest32165-21ssh21:52:320
3205899158,2cyclictest29851-21tr22:10:210
3205899158,2cyclictest25701-21ssh00:23:180
3205899158,2cyclictest13951-21ssh00:15:390
3205899158,1cyclictest13462-21ssh22:00:250
3205899157,2cyclictest3293-21expr19:15:200
3205899157,2cyclictest14002-21ntp_kernel_pll_00:35:190
3205899156,2cyclictest225-21systemd-journal19:40:000
3205899153,2cyclictest225-21systemd-journal23:45:000
3205899151,3cyclictest225-21systemd-journal21:20:010
3205899150,1cyclictest0-21swapper/023:51:360
3205899150,0cyclictest0-21swapper/022:09:380
3207299140,13cyclictest0-21swapper/320:45:153
3206899146,8cyclictest12740-21apache221:14:482
32068991410,2cyclictest26043-21cat20:05:002
32068991410,2cyclictest26043-21cat20:05:002
3206899140,14cyclictest0-21swapper/221:21:522
3206899140,13cyclictest0-21swapper/219:15:142
3206399142,11cyclictest1142-21runrttasks23:49:191
32063991412,2cyclictest0-21swapper/122:41:081
32063991411,2cyclictest28576-21wc21:50:201
32063991411,2cyclictest13955-21munin-plugin-st19:40:001
3206399140,13cyclictest24689-21diskmemload00:10:201
3205899148,2cyclictest8971-21tail20:35:170
3205899148,1cyclictest26767-21sh22:47:130
3205899148,1cyclictest16720-21which20:50:220
3205899148,1cyclictest16414-21ssh21:24:150
3205899147,2cyclictest31500-21awk22:50:170
3205899147,2cyclictest20900-21chrt19:52:450
3205899147,2cyclictest15894-21tail19:40:270
3205899146,2cyclictest11499-21ssh23:55:230
3205899143,2cyclictest15145-21awk20:49:590
3205899140,9cyclictest27692-21ssh21:10:490
3205899140,10cyclictest13-21ksoftirqd/019:30:140
3205899140,10cyclictest0-21swapper/023:15:130
3205899140,10cyclictest0-21swapper/023:15:120
3207299139,3cyclictest328-21sed20:15:253
3207299139,3cyclictest14442-21grep22:40:133
3207299137,2cyclictest0-21swapper/323:10:133
32072991310,2cyclictest29417-21grep20:10:163
32072991310,2cyclictest27280-21uname23:45:223
32072991310,2cyclictest19569-21cat00:20:003
32072991310,2cyclictest17124-21cat23:40:003
3207299130,12cyclictest0-21swapper/323:24:533
32068991310,2cyclictest32552-21grep19:10:142
32068991310,2cyclictest30499-21grep22:10:272
32068991310,2cyclictest26848-21grep20:05:172
32068991310,2cyclictest18540-21sed20:55:162
32068991310,2cyclictest16867-21grep19:45:142
32068991310,2cyclictest11988-21grep23:55:272
32068991310,2cyclictest11513-21apt19:35:002
3206899130,12cyclictest31320-21cut20:15:132
3206399139,4cyclictest0-21swapper/123:50:561
3206399139,3cyclictest29739-21wc20:10:201
3206399139,2cyclictest27714-21sed00:05:161
3206399138,3cyclictest38150irq/131-snd_hda20:25:221
32063991312,1cyclictest4266-21ssh22:34:151
32063991312,1cyclictest24689-21diskmemload00:39:121
32063991311,1cyclictest11375-21sh21:40:161
3206399131,11cyclictest0-21swapper/119:10:141
32063991310,2cyclictest6495-21sort20:30:171
32063991310,2cyclictest5423-21sed19:20:191
32063991310,2cyclictest27189-21grep21:10:271
3206399130,8cyclictest0-21swapper/121:45:211
3206399130,1cyclictest0-21swapper/100:01:591
3206399130,10cyclictest38150irq/131-snd_hda19:15:131
3205899139,3cyclictest3730-21grep20:25:150
3205899137,2cyclictest13811-21ssh23:37:320
3205899137,1cyclictest7919-21cat19:25:180
3205899137,1cyclictest23495-21irqstats21:05:190
3205899136,2cyclictest13602-21expr22:20:160
32058991310,2cyclictest29332-21cut20:10:170
3205899130,4cyclictest225-21systemd-journal19:25:010
3207299129,2cyclictest8608-21cat23:35:013
3207299129,2cyclictest4183-21cat22:15:013
3207299129,2cyclictest31589-21uniq22:50:163
3207299129,2cyclictest28447-21wc20:10:003
3207299129,2cyclictest20607-21timerwakeupswit00:00:293
3207299129,2cyclictest19466-21grep19:50:153
3207299129,2cyclictest14907-21grep19:40:163
3207299129,2cyclictest12584-21taskset21:40:283
3207299129,2cyclictest10687-21cut20:40:133
3207299129,2cyclictest10340-21sleep20:39:593
3207299128,3cyclictest7490-21sed20:30:283
3207299121,9cyclictest24689-21diskmemload00:35:123
3207299120,11cyclictest28240-21grep21:50:173
3206899129,2cyclictest27835-21wc21:50:152
3206899129,2cyclictest27069-21tr21:30:192
3206899129,2cyclictest26135-21grep23:25:212
3206899129,2cyclictest20594-21sed19:50:252
3206899129,2cyclictest15713-21basename23:20:002
3206899129,2cyclictest15713-21basename23:20:002
3206899129,2cyclictest14458-21grep19:40:152
3206899129,2cyclictest1267-21grep23:10:262
3206899129,2cyclictest11321-21grep21:40:162
3206899129,2cyclictest11301-21wc22:00:002
3206899128,3cyclictest9641-21cat23:55:002
3206899128,3cyclictest537-21snmpd23:07:092
3206899125,3cyclictest225-21systemd-journal20:30:002
32068991212,0cyclictest0-21swapper/222:41:352
32068991210,1cyclictest0-21swapper/222:04:492
3206899120,5cyclictest0-21swapper/223:20:152
3206899120,11cyclictest12325-21grep19:35:152
3206399129,2cyclictest20371-21grep00:20:161
3206399129,2cyclictest0-21swapper/123:25:131
3206399128,3cyclictest6843-21grep22:55:141
3206399128,3cyclictest28238-21sleep20:10:001
3206399128,3cyclictest1169-21grep20:20:151
3206399127,2cyclictest15999-21taskset19:40:481
32063991210,1cyclictest23290-21grep21:05:171
3206399120,0cyclictest0-21swapper/123:00:321
3205899127,1cyclictest23275-21taskset19:57:470
3205899127,1cyclictest221362sleep021:02:580
3207299119,1cyclictest3016-21grep19:15:183
3207299119,1cyclictest265302sleep323:06:143
3207299118,3cyclictest55750irq/128-enp2s0-23:57:083
3207299118,2cyclictest9477-21ssh22:55:593
3207299118,2cyclictest8776-21ssh22:36:153
3207299118,2cyclictest7562-21ssh21:57:073
3207299118,2cyclictest7489-21chrt23:53:223
3207299118,2cyclictest4677-21grep00:10:233
3207299118,2cyclictest31405-21ssh00:26:133
3207299118,2cyclictest29535-21ssh21:31:263
3207299118,2cyclictest25127-21ssh21:10:113
3207299118,2cyclictest24663-21ssh22:45:353
3207299118,2cyclictest2267-21cat21:15:193
3207299118,2cyclictest21834-21seq21:00:243
3207299118,2cyclictest21564-21ls19:55:153
3207299118,2cyclictest21220-21ssh22:25:013
3207299118,2cyclictest20885-21grep00:20:173
3207299118,2cyclictest20445-21ssh23:41:123
3207299118,2cyclictest19888-21grep21:45:183
3207299118,2cyclictest19475-21basename21:25:243
3207299118,2cyclictest17176-21ssh22:22:033
3207299118,2cyclictest16248-21ssh22:02:203
3207299118,2cyclictest15635-21expr23:00:143
3207299118,2cyclictest15367-21basename20:50:113
3207299118,2cyclictest10631-21head19:30:213
3207299116,4cyclictest24676-21/usr/sbin/munin23:25:113
3207299112,5cyclictest12723-21apache222:05:123
3207299110,10cyclictest9344-21rm21:20:083
3207299110,10cyclictest8616-21grep23:15:183
3207299110,10cyclictest8616-21grep23:15:173
3207299110,10cyclictest18377-21chrt19:45:513
3206899118,2cyclictest9808-21sed20:35:262
3206899118,2cyclictest9294-21date19:30:012
3206899118,2cyclictest6040-21taskset22:15:222
3206899118,2cyclictest4230-21ssh00:30:102
3206899118,2cyclictest29665-21ssh22:30:062
3206899118,2cyclictest28735-21cut20:10:142
3206899118,2cyclictest27242-21cut00:05:162
3206899118,2cyclictest23846-21ssh22:25:422
3206899118,2cyclictest23355-21sed21:05:172
3206899118,2cyclictest20903-21cut21:00:152
3206899118,2cyclictest1979-21head21:15:172
3206899118,2cyclictest19778-21ssh00:20:022
3206899118,2cyclictest14116-21grep20:45:222
3206899118,2cyclictest1223-21sh23:30:162
3206899117,3cyclictest26695-21sed23:45:172
3206899110,2cyclictest0-21swapper/221:45:172
3206399118,2cyclictest8782-21taskset19:25:341
3206399118,2cyclictest5673-21ssh22:35:111
3206399118,2cyclictest38150irq/131-snd_hda22:05:211
3206399118,2cyclictest3047-21sed21:35:191
3206399118,2cyclictest25167-21rm22:45:581
3206399118,2cyclictest19635-21taskset20:57:071
3206399118,2cyclictest19427-21head23:40:261
3206399118,2cyclictest19348-21ssh22:04:461
3206399118,2cyclictest17727-21ssh21:25:061
3206399118,2cyclictest13736-21tail20:45:191
3206399118,2cyclictest13144-21wc00:15:231
3206399118,2cyclictest10989-21tail23:35:271
3206399117,3cyclictest5571-21ssh00:30:221
3206399117,3cyclictest4510-21sh22:15:121
3206399111,6cyclictest225-21systemd-journal20:00:001
3206399110,10cyclictest23148-21taskset22:25:271
3205899118,2cyclictest8076-21cat20:35:000
3205899118,2cyclictest26193-21tr20:05:130
237142110,0sleep10-21swapper/120:00:131
237142110,0sleep10-21swapper/120:00:121
3207299108,1cyclictest25479-21who20:00:283
3207299108,1cyclictest25479-21who20:00:273
3207299108,1cyclictest12237-21expr19:35:153
3207299107,2cyclictest4938-21tr19:20:173
3207299107,2cyclictest4598-21ntpq20:25:203
3207299107,2cyclictest4349-21ssh22:15:073
3207299107,2cyclictest18214-21expr20:55:153
3206899108,1cyclictest6279-21cat20:30:152
3206899108,1cyclictest4657-21apache_processe19:20:142
3206899108,1cyclictest232822sleep219:57:522
3206899108,1cyclictest12342-21chrt20:40:392
3206899107,2cyclictest2172-21grep20:20:212
3206899107,2cyclictest15698-21cstates20:50:142
3206399108,1cyclictest686-21taskset20:19:031
3206399108,1cyclictest11315-21taskset19:32:361
3206399108,1cyclictest10898-21ls20:40:141
3206399107,2cyclictest7497-21ssh23:15:011
3206399107,2cyclictest7497-21ssh23:15:011
3206399107,2cyclictest29215-21grep00:25:161
3206399107,2cyclictest20696-21df_abs21:00:141
3206399107,2cyclictest19681-21grep19:50:151
3206399100,9cyclictest16819-21wc19:45:141
3205899107,2cyclictest1328-21tr20:20:150
320729997,1cyclictest7029-21apache_volume19:25:133
320729997,1cyclictest32710-21expr19:10:143
320729996,2cyclictest22613-21tr21:05:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional