You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-06 - 18:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Apr 06, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
692028975,9sleep00-21swapper/007:07:110
702428572,9sleep20-21swapper/207:08:342
686428467,13sleep10-21swapper/107:06:271
684128067,9sleep30-21swapper/307:06:103
723999250,4cyclictest0-21swapper/110:12:161
723699250,1cyclictest0-21swapper/011:57:370
7236992423,1cyclictest0-21swapper/010:28:020
7239992322,1cyclictest25-21ksoftirqd/110:34:471
7239992219,2cyclictest29082-21grep10:05:161
7239992214,4cyclictest0-21swapper/111:45:221
7236992222,0cyclictest0-21swapper/010:44:290
723699220,0cyclictest0-21swapper/012:23:440
193712220,0sleep30-21swapper/312:34:523
723999212,2cyclictest25-21ksoftirqd/111:31:121
7239992118,1cyclictest0-21swapper/111:17:521
723999210,3cyclictest0-21swapper/109:31:231
723999210,2cyclictest0-21swapper/110:01:231
723699210,18cyclictest0-21swapper/009:17:490
7246992020,0cyclictest0-21swapper/311:20:383
723999205,15cyclictest25-21ksoftirqd/110:26:111
723999200,2cyclictest0-21swapper/112:27:361
723699202,1cyclictest141rcu_preempt09:47:160
723999198,5cyclictest25-21ksoftirqd/109:45:441
723699190,1cyclictest0-21swapper/009:34:210
7246991812,4cyclictest0-21swapper/309:10:173
724699180,1cyclictest0-21swapper/309:52:403
724699180,17cyclictest0-21swapper/312:07:053
724299180,1cyclictest0-21swapper/209:55:042
724299180,17cyclictest0-21swapper/209:35:132
723999188,9cyclictest0-21swapper/109:50:171
723999183,13cyclictest241rcuc/109:24:151
7239991818,0cyclictest25-21ksoftirqd/112:33:431
7239991816,2cyclictest0-21swapper/112:05:141
723999180,1cyclictest0-21swapper/110:23:561
723999180,17cyclictest0-21swapper/109:12:051
723699187,6cyclictest1852-21perf12:05:000
7236991817,1cyclictest0-21swapper/009:11:080
723699180,14cyclictest0-21swapper/011:11:300
291712180,1sleep00-21swapper/010:05:180
7246991717,0cyclictest0-21swapper/311:40:073
724699170,1cyclictest0-21swapper/310:21:423
724699170,17cyclictest0-21swapper/312:18:073
724299170,15cyclictest0-21swapper/209:23:372
723999173,3cyclictest141rcu_preempt10:15:041
7239991713,4cyclictest25-21ksoftirqd/111:43:101
723999170,1cyclictest0-21swapper/111:35:211
7236991715,1cyclictest411-21diskmemload12:08:020
723699170,3cyclictest0-21swapper/012:13:200
723699170,1cyclictest0-21swapper/010:54:110
724699164,4cyclictest225-21systemd-journal11:55:003
7246991616,0cyclictest0-21swapper/311:34:343
724699161,2cyclictest141rcu_preempt12:20:363
724699160,5cyclictest0-21swapper/310:17:533
723999167,7cyclictest25-21ksoftirqd/112:15:181
723999166,9cyclictest31873-21df_abs11:05:141
7239991615,0cyclictest0-21swapper/109:27:241
7239991613,2cyclictest241rcuc/109:44:371
7239991612,4cyclictest0-21swapper/111:50:371
723699164,6cyclictest20431-21perf10:20:000
723699163,6cyclictest11059-21latency_hist09:55:010
723699160,15cyclictest0-21swapper/008:30:130
724699151,6cyclictest14340-21fschecks_count10:35:173
7242991514,1cyclictest0-21swapper/209:15:482
724299150,7cyclictest225-21systemd-journal11:13:402
724299150,15cyclictest0-21swapper/210:47:182
724299150,15cyclictest0-21swapper/209:25:512
723999152,12cyclictest13662-21ssh10:35:141
7239991512,2cyclictest16092-21runrttasks09:57:261
7239991511,4cyclictest411-21diskmemload10:58:251
7239991510,2cyclictest225-21systemd-journal07:22:391
723999150,0cyclictest0-21swapper/111:28:491
723699158,4cyclictest13-21ksoftirqd/008:25:210
723699156,7cyclictest411-21diskmemload11:40:040
723699150,1cyclictest0-21swapper/012:30:480
724699149,5cyclictest31270-21apache209:41:163
724699149,3cyclictest29637-21wc09:00:213
724699142,11cyclictest12065-21ssh12:10:203
724699141,0cyclictest0-21swapper/311:07:223
724699140,13cyclictest0-21swapper/310:56:273
724299143,5cyclictest225-21systemd-journal10:59:552
724299143,10cyclictest14946-21ntp_states10:35:212
724299141,8cyclictest26652-21cpuspeed_turbos12:00:142
7242991413,1cyclictest55650irq/127-enp2s0-10:50:342
7242991413,1cyclictest0-21swapper/210:28:272
724299140,5cyclictest0-21swapper/211:53:392
724299140,14cyclictest0-21swapper/209:44:032
724299140,13cyclictest0-21swapper/211:31:492
7239991412,2cyclictest0-21swapper/112:35:141
723999141,12cyclictest25-21ksoftirqd/110:40:131
7239991410,3cyclictest18504-21cron08:39:591
723999140,3cyclictest0-21swapper/111:10:161
723999140,14cyclictest0-21swapper/111:57:201
723699149,3cyclictest19054-21tr08:40:140
723699142,8cyclictest31433-21if_err_enp0s31f10:45:210
723699142,11cyclictest18450-21awk09:40:000
723699141,7cyclictest9149-21munin-run11:50:000
7236991411,2cyclictest27541-21grep08:55:270
723699140,13cyclictest0-21swapper/009:23:360
724699139,2cyclictest16330-21grep11:15:153
724699138,1cyclictest2297-21apt-get09:30:133
724699133,9cyclictest14032-21apache_processe08:30:113
724699133,2cyclictest6566-21taskset10:50:143
7246991312,1cyclictest0-21swapper/312:01:013
7246991312,1cyclictest0-21swapper/310:43:263
7246991310,2cyclictest5820-21grep10:30:153
7246991310,2cyclictest19994-21ls08:40:213
7246991310,2cyclictest10139-21cat08:20:183
724699130,9cyclictest0-21swapper/310:09:383
724699130,13cyclictest0-21swapper/311:47:533
724699130,12cyclictest17993-21ssh11:35:203
724299137,6cyclictest7662sleep209:48:302
724299131,8cyclictest0-21swapper/212:20:142
7242991310,2cyclictest5200-21cat12:25:272
7242991310,2cyclictest26894-21apt12:20:002
7242991310,2cyclictest22461-21grep07:40:142
7242991310,2cyclictest22371-21sed08:45:212
7242991310,2cyclictest18642-21grep07:30:212
7242991310,2cyclictest17318-21cat11:15:242
7242991310,2cyclictest17131-21cron07:29:592
7242991310,2cyclictest10220-21munin-plugin-st12:10:002
7242991310,2cyclictest0-21swapper/210:06:132
723999139,4cyclictest25-21ksoftirqd/112:12:361
723999139,3cyclictest22105-21munin-plugin-st07:40:001
723999138,3cyclictest25-21ksoftirqd/109:35:131
723999136,5cyclictest25-21ksoftirqd/111:01:271
7239991310,2cyclictest8802-21ls08:20:011
7239991310,2cyclictest32088-21latency_hist11:25:001
7239991310,2cyclictest25-21ksoftirqd/110:50:491
7239991310,2cyclictest241rcuc/107:30:121
723699139,2cyclictest31979-21grep09:05:230
723699139,2cyclictest21128-21tr07:35:200
7236991313,0cyclictest0-21swapper/010:00:200
7236991310,2cyclictest26824-21cat09:45:010
7236991310,2cyclictest11200-21perf08:25:010
723699130,6cyclictest0-21swapper/007:50:130
724699129,2cyclictest2175-21tr08:05:143
724699129,2cyclictest18603-21sed09:20:173
724699129,2cyclictest18547-21tr07:30:203
724699129,2cyclictest13300-21sed08:25:263
724699128,3cyclictest4966-21ssh10:10:173
724699128,3cyclictest133142sleep311:13:293
724699128,2cyclictest25665-21dump-pmu-power08:55:003
724699123,5cyclictest27602-21grep08:55:263
7246991212,0cyclictest0-21swapper/309:57:063
7246991211,1cyclictest0-21swapper/309:25:143
724699120,12cyclictest0-21swapper/311:56:163
724699120,10cyclictest0-21swapper/312:27:073
724299129,2cyclictest9691-21grep08:20:152
724299129,2cyclictest32194-21head11:05:172
724299129,2cyclictest30956-21cat07:55:242
724299129,2cyclictest28154-21basename12:40:002
724299129,2cyclictest2574-21sort09:10:242
724299129,2cyclictest24276-21grep11:20:142
724299129,2cyclictest18075-21grep08:35:252
724299129,2cyclictest17005-21tr11:35:142
724299129,2cyclictest16132-21cron08:35:012
724299129,2cyclictest13773-21apt08:30:012
724299129,2cyclictest12029-21open_files12:10:212
724299123,8cyclictest3326-21/usr/sbin/munin09:30:172
724299123,3cyclictest22933-21/usr/sbin/munin11:00:112
7242991210,1cyclictest8873-21cut07:10:212
724299120,11cyclictest20202-21tr10:00:162
723999129,2cyclictest31820-21tr10:45:251
723999129,2cyclictest31690-21uname09:05:201
723999127,4cyclictest15891-21taskset08:31:541
723999121,5cyclictest225-21systemd-journal08:00:091
723999121,5cyclictest225-21systemd-journal08:00:091
723999121,11cyclictest25-21ksoftirqd/108:50:121
723999120,4cyclictest225-21systemd-journal07:49:551
723999120,3cyclictest0-21swapper/112:20:511
723699129,2cyclictest4907-21wc12:25:220
723699129,2cyclictest24992-21grep07:45:140
723699129,2cyclictest23809-21wc07:40:250
723699129,2cyclictest22402-21tr08:45:220
723699129,2cyclictest2019-21head11:25:270
723699129,2cyclictest14095-21tail07:20:250
723699129,2cyclictest13453-21grep09:55:250
723699128,3cyclictest3585-21grep08:05:270
723699122,8cyclictest13644-21apt-get10:35:140
7236991210,2cyclictest0-21swapper/011:03:350
723699120,7cyclictest0-21swapper/012:15:370
724699118,2cyclictest7927-21chrt08:15:223
724699118,2cyclictest5618-21wc08:10:213
724699118,2cyclictest30644-21ssh10:25:203
724699118,2cyclictest30266-21grep07:55:163
724699118,2cyclictest30159-21ssh10:45:073
724699118,2cyclictest29619-21ssh11:04:053
724699118,2cyclictest27386-21cut07:50:143
724699118,2cyclictest21545-21ssh12:35:203
724699118,2cyclictest21163-21grep10:00:213
724699118,2cyclictest1685-21ssh09:49:123
724699118,2cyclictest1642-21taskset08:04:573
724699118,2cyclictest1642-21taskset08:04:563
724699118,2cyclictest16146-21cut07:25:193
724699118,2cyclictest12505-21cut07:20:133
724699118,2cyclictest11365-21ssh09:15:263
724699117,1cyclictest0-21swapper/309:35:443
724699111,4cyclictest0-21swapper/311:25:213
724299119,1cyclictest22669-21grep10:40:162
724299119,1cyclictest1294-21grep11:25:202
724299118,2cyclictest8765-21ssh10:12:242
724299118,2cyclictest8314-21ssh11:49:222
724299118,2cyclictest6929-21expr08:15:142
724299118,2cyclictest6396-21cron08:15:012
724299118,2cyclictest6018-21ssh09:51:042
724299118,2cyclictest5373-21grep10:30:142
724299118,2cyclictest30598-21sh11:43:102
724299118,2cyclictest27095-21ssh10:23:412
724299118,2cyclictest25791-21cat07:45:202
724299118,2cyclictest2358-21expr08:05:162
724299118,2cyclictest14480-21ssh12:31:022
724299118,2cyclictest12385-21ssh10:15:032
724299111,3cyclictest225-21systemd-journal08:02:452
724299111,3cyclictest225-21systemd-journal08:02:442
723999118,2cyclictest7740-21cut07:10:131
723999118,2cyclictest3024-21grep08:05:201
723999118,2cyclictest27180-21ssh12:00:171
723999118,2cyclictest26659-21grep08:55:181
723999118,2cyclictest24504-21apt07:44:591
723999118,2cyclictest13681-21taskset08:29:011
723999118,2cyclictest12009-21taskset09:15:571
723999117,3cyclictest17250-21cat07:30:001
723999113,7cyclictest6394-21cron08:15:001
723999111,9cyclictest25-21ksoftirqd/107:15:121
723699119,1cyclictest16647-21tr08:35:120
723699118,2cyclictest4528-21tr10:10:150
723699118,2cyclictest32521-21sed08:00:150
723699118,2cyclictest32521-21sed08:00:140
723699118,2cyclictest29493-21cat07:55:110
723699118,2cyclictest29098-21cut09:00:160
723699118,2cyclictest28280-21ssh10:24:360
723699118,2cyclictest26854-21ssh09:25:160
723699118,2cyclictest23925-21expr08:50:140
723699118,2cyclictest21914-21python12:35:260
723699118,2cyclictest21343-21ssh11:18:190
723699118,2cyclictest17760-21sh11:35:180
723699118,2cyclictest16682-21tail07:25:260
723699118,2cyclictest16389-21wc10:55:260
723699118,2cyclictest14309-21ssh11:33:300
723699118,2cyclictest11688-21grep07:15:260
723699118,2cyclictest10926-21ntp_offset11:50:200
723699118,2cyclictest1018-21ssh11:05:310
723699110,2cyclictest0-21swapper/010:34:470
724699108,1cyclictest8831-21ntpq07:10:203
724699108,1cyclictest307032sleep309:05:133
724699108,1cyclictest22392-21seq07:40:133
724699108,1cyclictest21549-21expr08:45:153
724699108,1cyclictest20854-21cat07:35:183
724699107,2cyclictest17310-21cat08:35:173
724699107,2cyclictest10313-21expr07:15:143
724699101,8cyclictest0-21swapper/307:45:133
724299108,1cyclictest388-21seq09:09:592
724299108,1cyclictest28724-21expr09:00:142
724299108,1cyclictest20324-21expr07:35:142
724299108,1cyclictest19220-21expr08:40:152
724299108,1cyclictest12446-21apache_processe07:20:112
724299107,2cyclictest26330-21expr08:55:152
724299107,2cyclictest24372-21basename08:50:172
724299107,2cyclictest11770-21head07:15:262
724299100,9cyclictest17863-21sh11:55:082
723999108,1cyclictest90392sleep108:20:081
723999108,1cyclictest28653-21sed07:50:251
723999107,2cyclictest29547-21tr07:55:131
723999107,2cyclictest21256-21cut08:45:141
723999106,3cyclictest28350-21cut09:00:121
723999104,2cyclictest25-21ksoftirqd/108:40:381
723699108,1cyclictest5027-21expr08:10:150
723699107,2cyclictest9757-21chrt07:14:520
723699107,2cyclictest24772-21cut11:20:160
72429997,1cyclictest27118-21basename07:50:112
72369997,1cyclictest6701-21wget08:15:130
72369997,1cyclictest194572sleep007:34:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional