You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-03 - 15:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun May 03, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2570529884,9sleep20-21swapper/207:08:232
2430729573,17sleep00-21swapper/007:05:170
2558428564,6sleep10-21swapper/107:06:511
2560628468,12sleep30-21swapper/307:07:093
190072480,0sleep319006-21ssh12:04:373
25918992726,1cyclictest0-21swapper/211:44:052
2591899270,26cyclictest0-21swapper/210:27:452
2591399270,1cyclictest0-21swapper/109:00:131
309612260,1chrt241rcuc/110:34:551
25920992624,1cyclictest39-21ksoftirqd/307:15:003
2592099262,23cyclictest29570-21ssh09:55:073
2591899260,26cyclictest0-21swapper/212:23:272
2591399260,2cyclictest0-21swapper/112:32:121
2591399260,25cyclictest0-21swapper/112:00:261
2591899250,25cyclictest0-21swapper/209:47:282
2590999250,21cyclictest0-21swapper/010:09:120
2591399240,24cyclictest0-21swapper/110:12:491
2591899228,2cyclictest141rcu_preempt11:11:362
2591899220,22cyclictest0-21swapper/210:13:372
2590999220,0cyclictest0-21swapper/010:38:060
25913992119,1cyclictest0-21swapper/107:15:001
25909992111,7cyclictest21287-21apache207:40:110
2590999210,1cyclictest0-21swapper/009:43:320
25920992017,1cyclictest39-21ksoftirqd/309:39:453
2591899203,17cyclictest0-21swapper/209:29:242
2591899201,3cyclictest0-21swapper/211:36:272
25913992018,1cyclictest0-21swapper/111:40:151
2591399200,4cyclictest0-21swapper/111:00:121
2590999200,1cyclictest0-21swapper/012:36:510
2590999200,1cyclictest0-21swapper/010:47:440
2590999200,1cyclictest0-21swapper/010:30:450
25918991917,1cyclictest12423-21fschecks_time12:00:172
2591899190,18cyclictest0-21swapper/210:41:212
2590999190,1cyclictest0-21swapper/012:31:280
25920991817,1cyclictest0-21swapper/312:24:093
2591899182,15cyclictest0-21swapper/211:22:062
2591899180,17cyclictest0-21swapper/212:19:502
2591899180,17cyclictest0-21swapper/209:40:042
2591399181,16cyclictest9701-21aten2_r7power_e08:50:141
2590999182,2cyclictest141rcu_preempt11:43:580
2590999180,4cyclictest0-21swapper/009:10:130
2590999180,1cyclictest0-21swapper/012:12:510
2590999180,1cyclictest0-21swapper/011:20:290
2590999180,1cyclictest0-21swapper/011:15:590
25920991715,2cyclictest39-21ksoftirqd/309:32:003
25920991712,3cyclictest39-21ksoftirqd/307:45:133
2592099170,2cyclictest0-21swapper/311:09:023
25918991716,1cyclictest0-21swapper/212:12:512
2591899170,1cyclictest0-21swapper/212:38:192
2591899170,17cyclictest0-21swapper/211:57:332
2591899170,15cyclictest0-21swapper/212:30:522
2591399175,7cyclictest13240-21df_abs12:20:131
2591399173,9cyclictest18992-21ssh11:45:121
2591399170,16cyclictest0-21swapper/112:25:141
25909991710,5cyclictest0-21swapper/009:51:580
25920991613,1cyclictest0-21swapper/309:52:493
25918991615,1cyclictest0-21swapper/210:09:122
2591899160,15cyclictest0-21swapper/210:30:452
25913991616,0cyclictest0-21swapper/109:54:271
2590999169,4cyclictest12125-21munin-run12:20:000
25909991614,1cyclictest0-21swapper/010:29:090
2590999160,0cyclictest0-21swapper/011:26:500
25920991513,2cyclictest39-21ksoftirqd/309:40:293
2591899151,14cyclictest0-21swapper/210:55:142
2591899150,14cyclictest0-21swapper/211:05:572
25913991514,1cyclictest0-21swapper/110:04:531
2591399151,14cyclictest0-21swapper/110:23:121
2590999150,15cyclictest0-21swapper/010:01:470
2592099149,4cyclictest39-21ksoftirqd/311:10:173
2592099149,3cyclictest9182-21sleep08:50:003
2592099149,3cyclictest5863-21tr10:00:153
2592099149,1cyclictest0-21swapper/310:51:103
25920991411,2cyclictest21048-21cat09:50:003
25920991410,3cyclictest968-21grep10:55:173
25920991410,3cyclictest13738-21grep07:50:173
2592099140,0cyclictest141rcu_preempt10:39:323
2591899146,8cyclictest24204-21/usr/sbin/munin10:50:192
2591899141,13cyclictest0-21swapper/211:28:442
2591899140,1cyclictest0-21swapper/209:15:472
2591899140,1cyclictest0-21swapper/209:10:132
2591899140,14cyclictest0-21swapper/210:37:072
25913991413,1cyclictest0-21swapper/107:30:241
25913991411,2cyclictest0-21swapper/110:27:151
25913991410,3cyclictest3392-21grep08:35:201
25913991410,3cyclictest21095-21ls09:49:591
25913991410,2cyclictest24510-21grep08:10:251
25913991410,2cyclictest24510-21grep08:10:251
2590999142,11cyclictest13-21ksoftirqd/009:30:140
25909991414,0cyclictest0-21swapper/009:15:070
25909991411,2cyclictest0-21swapper/010:56:310
116752140,0sleep10-21swapper/109:44:181
2592099139,3cyclictest9924-21sed07:40:253
2592099139,3cyclictest32372-21tr08:30:133
2592099139,3cyclictest24466-21sed08:10:253
2592099139,3cyclictest24466-21sed08:10:253
2592099139,1cyclictest141rcu_preempt11:35:123
2592099138,3cyclictest39-21ksoftirqd/311:51:113
2592099132,10cyclictest39-21ksoftirqd/309:20:133
25920991310,2cyclictest4073-21head07:30:183
25920991310,2cyclictest29716-21basename08:25:003
25918991311,2cyclictest0-21swapper/212:09:122
25918991310,2cyclictest17173-21open_inodes10:45:212
2591899130,3cyclictest0-21swapper/209:38:392
2591899130,12cyclictest7782-21grep08:45:172
2591399139,3cyclictest38150irq/131-snd_hda10:53:311
2591399139,3cyclictest31748-21cut10:35:151
2591399139,3cyclictest29690-21cron08:25:001
2591399139,3cyclictest1802-21sh11:35:001
2591399139,3cyclictest14056-21grep09:25:281
2591399138,4cyclictest27892-21chrt09:34:401
2591399138,3cyclictest0-21swapper/111:19:241
2591399135,4cyclictest225-21systemd-journal07:55:001
25913991310,2cyclictest28345-21ssh12:10:101
25913991310,2cyclictest28252-21grep09:15:151
25913991310,2cyclictest20765-21ls09:10:201
25913991310,2cyclictest18886-21cron09:10:001
25913991310,2cyclictest1043-21grep08:30:201
2591399130,12cyclictest4210-21idleruntime-cro09:40:001
2590999137,6cyclictest487-21avahi-daemon11:52:250
2590999130,8cyclictest0-21swapper/010:44:590
2590999130,5cyclictest0-21swapper/012:25:000
2590999130,12cyclictest2043-21basename08:35:000
2592099129,2cyclictest4513-21grep12:15:143
2592099129,2cyclictest27456-21cut11:30:183
2592099129,2cyclictest21036-21tr12:25:143
2592099129,2cyclictest18626-21dnf11:25:153
2592099129,2cyclictest1596-21cut11:15:153
2592099129,2cyclictest11740-21cron08:55:003
25920991211,1cyclictest16287-21ssh10:25:233
2592099121,10cyclictest24181-21ssh11:47:543
25920991210,1cyclictest0-21swapper/308:40:133
2592099120,11cyclictest7797-21uname10:20:193
2591899129,2cyclictest9584-21ssh09:23:352
2591899129,2cyclictest7374-21tail10:20:172
2591899129,2cyclictest32601-21sed07:20:252
2591899129,2cyclictest30993-21cat08:25:222
2591899129,2cyclictest30012-21cat07:15:262
2591899129,2cyclictest27755-21grep07:10:262
2591899129,2cyclictest1631-21grep08:30:252
2591899129,2cyclictest11902-21wc07:45:222
2591899127,1cyclictest18907-21diskmemload09:58:412
25918991210,1cyclictest11594-21ssh10:03:362
2591899120,1cyclictest0-21swapper/211:15:022
2591399129,2cyclictest5905-21grep07:35:131
2591399129,2cyclictest401-21ssh10:55:151
2591399129,2cyclictest31242-21ssh10:15:161
2591399129,2cyclictest30698-21perf07:20:001
2591399129,2cyclictest18985-21sed08:00:211
2591399129,2cyclictest18741-21grep11:25:161
2591399128,3cyclictest31065-21sh09:55:211
2591399128,3cyclictest2851-21taskset07:27:571
2591399128,2cyclictest5451-21grep08:40:171
2591399127,4cyclictest9471-21ssh09:23:291
2591399121,8cyclictest9585-21/usr/sbin/munin11:20:181
2591399120,7cyclictest0-21swapper/112:15:151
2591399120,2cyclictest0-21swapper/111:35:341
2590999129,2cyclictest7332-21grep10:20:160
2590999129,2cyclictest30261-21wc09:35:220
2590999129,2cyclictest23640-21tr10:10:210
2590999129,2cyclictest18580-21wc11:05:260
2590999129,2cyclictest15635-21tr09:00:210
2590999125,6cyclictest11139-21munin-run12:00:000
2590999125,5cyclictest10175-21sendmail-msp11:40:010
2590999124,5cyclictest18907-21diskmemload11:35:000
2590999120,11cyclictest0-21swapper/010:19:500
2592099119,1cyclictest39-21ksoftirqd/308:35:163
2592099118,2cyclictest988-21cut07:25:153
2592099118,2cyclictest964-21ssh12:12:483
2592099118,2cyclictest6519-21ssh11:56:223
2592099118,2cyclictest6423-21head07:35:163
2592099118,2cyclictest32506-21grep10:15:263
2592099118,2cyclictest29047-21expr07:15:153
2592099118,2cyclictest26342-21ssh10:12:043
2592099118,2cyclictest24860-21ssh10:30:233
2592099118,2cyclictest23622-21ssh12:06:393
2592099118,2cyclictest22416-21rm09:11:113
2592099118,2cyclictest20314-21ssh10:47:223
2592099118,2cyclictest18929-21tr08:00:203
2592099118,2cyclictest18457-21cron11:45:013
2592099118,2cyclictest15123-21head09:00:173
2592099118,2cyclictest13871-21taskset08:56:563
2592099118,2cyclictest13768-21ssh10:05:073
2592099118,2cyclictest13009-21ssh11:02:253
2592099118,2cyclictest12548-21ssh11:21:083
2592099117,3cyclictest7764-21tr10:40:153
2592099116,4cyclictest6069-21ssh12:35:183
2592099116,4cyclictest39-21ksoftirqd/309:15:183
2592099115,2cyclictest225-21systemd-journal09:10:003
2592099110,10cyclictest29970-21sed12:30:153
2591899119,2cyclictest30900-21sh11:32:242
2591899119,1cyclictest4913-21tr08:40:152
2591899119,1cyclictest30639-21tr10:15:142
2591899119,1cyclictest22414-21grep09:30:262
2591899119,1cyclictest18392-21grep09:05:272
2591899119,1cyclictest16177-21grep07:55:172
2591899119,1cyclictest14514-21tr09:00:142
2591899118,2cyclictest8358-21ssh11:00:072
2591899118,2cyclictest4710-21cat07:30:272
2591899118,2cyclictest3019-21sh07:30:002
2591899118,2cyclictest24378-21cat08:10:252
2591899118,2cyclictest24378-21cat08:10:252
2591899118,2cyclictest14116-21sh09:00:002
2591899118,1cyclictest23270-21sed09:50:252
2591899113,3cyclictest0-21swapper/208:50:162
2591899112,4cyclictest29542-21sshd08:23:102
25918991110,1cyclictest0-21swapper/212:28:292
2591399118,2cyclictest9737-21ssh10:40:391
2591399118,2cyclictest7340-21tr08:45:151
2591399118,2cyclictest631-21cron07:25:001
2591399118,2cyclictest5213-21ssh12:35:121
2591399118,2cyclictest5058-21expr11:55:271
2591399118,2cyclictest28898-21sendmail_mailst11:50:221
2591399118,2cyclictest27925-21ssh11:10:491
2591399118,2cyclictest22212-21chrt08:05:531
2591399118,2cyclictest19876-21ssh10:47:021
2591399118,2cyclictest18951-21ssh11:05:321
2591399118,2cyclictest12504-21df08:55:151
2591399110,10cyclictest22886-21ssh12:06:051
2591399110,0cyclictest0-21swapper/110:09:261
2590999119,1cyclictest31953-21ls07:20:180
2590999118,2cyclictest7581-21sed08:45:160
2590999118,2cyclictest4034-21ls09:20:140
2590999118,2cyclictest31459-21ssh09:55:260
2590999118,2cyclictest30716-21head08:25:180
2590999118,2cyclictest29694-21awk07:15:210
2590999118,2cyclictest29336-21chrt08:20:300
2590999118,2cyclictest28065-21rm11:10:550
2590999118,2cyclictest26524-21awk08:15:210
2590999118,2cyclictest24726-21chrt08:12:410
2590999118,2cyclictest24726-21chrt08:12:410
2590999118,2cyclictest24045-21ssh10:50:040
2590999118,2cyclictest23103-21ssh12:25:300
2590999118,2cyclictest22209-21ssh11:46:220
2590999118,2cyclictest21790-21cut08:05:220
2590999118,2cyclictest20254-21sh12:05:140
2590999118,2cyclictest17538-21cut09:05:180
2590999118,2cyclictest17435-21chrt07:59:180
2590999118,2cyclictest15727-21taskset09:26:420
2590999118,2cyclictest14015-21basename09:45:170
2590999118,2cyclictest11686-21cut12:00:140
2590999118,2cyclictest10689-21rm11:00:360
2590999118,2cyclictest1052-21grep07:25:150
2590999111,3cyclictest225-21systemd-journal08:45:010
2590999110,5cyclictest225-21systemd-journal07:55:000
2592099108,1cyclictest299652sleep308:25:143
2592099108,1cyclictest26409-21ls08:15:223
2592099108,1cyclictest15799-21expr07:55:153
2592099107,2cyclictest20946-21taskset08:05:153
2592099104,2cyclictest225-21systemd-journal07:20:103
2592099104,2cyclictest11874-21apt-get09:25:023
2591899108,2cyclictest8210-21gpgconf07:40:012
2591899108,2cyclictest19077-21grep11:45:152
2591899108,1cyclictest2582-21cstates08:35:132
2591899108,1cyclictest18047-21grep08:00:142
2591899107,2cyclictest27045-21seq08:16:532
2591899107,2cyclictest13661-21grep07:50:172
2591399108,1cyclictest30401-21expr08:25:161
2591399108,1cyclictest11002-21expr07:45:151
2591399107,2cyclictest17003-21sed07:55:251
2591399101,7cyclictest1037-21gdbus07:40:011
2590999108,1cyclictest40532sleep008:35:450
2590999108,1cyclictest18438-21cut08:00:160
2590999107,2cyclictest4997-21sed07:30:280
2590999107,2cyclictest28425-21sed07:15:000
2590999107,2cyclictest10653-21apt-get07:45:140
2590999101,6cyclictest14115-21cron09:00:000
2590999101,5cyclictest0-21swapper/008:50:160
259189998,1cyclictest6902-21awk07:35:212
259189998,1cyclictest27919-21ssh11:50:152
259189998,1cyclictest20708-21expr08:05:142
259139996,2cyclictest26531-21tr08:15:221
259099996,2cyclictest7669-21chrt07:38:510
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional