You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-16 - 05:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Sep 16, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2604829079,6sleep00-21swapper/019:08:390
2580929069,16sleep20-21swapper/219:05:352
2590528467,13sleep10-21swapper/119:06:471
2582728169,8sleep30-21swapper/319:05:483
224182310,0sleep30-21swapper/322:29:003
26246992723,1cyclictest13-21ksoftirqd/023:29:560
2624799260,26cyclictest0-21swapper/100:27:051
26253992422,1cyclictest0-21swapper/323:54:233
2625399241,20cyclictest20709-21ssh23:25:263
2624999242,1cyclictest19467-21diskmemload21:51:182
2624799241,23cyclictest0-21swapper/122:46:151
2625399234,19cyclictest16808-21kworker/3:023:37:263
26249992322,1cyclictest0-21swapper/222:26:032
2624799230,23cyclictest0-21swapper/122:38:401
2624799230,22cyclictest0-21swapper/121:55:121
26246992218,3cyclictest29770-21grep23:50:210
9162210,2sleep20-21swapper/223:52:472
2625399210,19cyclictest16808-21kworker/3:023:34:143
2624999210,21cyclictest0-21swapper/223:04:142
26246992118,2cyclictest13-21ksoftirqd/023:20:100
2625399201,19cyclictest0-21swapper/300:17:223
2624799201,18cyclictest19467-21diskmemload00:13:531
2624799200,20cyclictest0-21swapper/122:17:591
26246992010,3cyclictest13-21ksoftirqd/022:42:380
26253991918,1cyclictest0-21swapper/323:23:583
2625399190,2cyclictest0-21swapper/322:48:563
26249991918,1cyclictest0-21swapper/222:46:242
26249991916,2cyclictest32-21ksoftirqd/219:50:142
2624799190,18cyclictest0-21swapper/122:25:121
2624699192,2cyclictest0-21swapper/019:15:140
26246991917,1cyclictest13-21ksoftirqd/022:50:440
26246991917,1cyclictest13-21ksoftirqd/021:51:250
26246991914,1cyclictest13-21ksoftirqd/023:03:350
2624699190,1cyclictest0-21swapper/022:15:130
26247991817,1cyclictest7335-21ssh21:21:171
2624799180,18cyclictest0-21swapper/121:54:371
2624699182,4cyclictest3347-21ssh23:35:140
26246991815,2cyclictest13-21ksoftirqd/000:25:170
26246991811,4cyclictest13-21ksoftirqd/021:44:450
2624799172,15cyclictest9857-21ssh21:42:281
2624699178,5cyclictest13-21ksoftirqd/022:56:050
2624699175,6cyclictest5982-21perf00:35:000
26246991717,0cyclictest0-21swapper/000:37:190
26246991716,1cyclictest13-21ksoftirqd/021:40:000
26246991714,2cyclictest13-21ksoftirqd/023:05:070
26246991713,3cyclictest13-21ksoftirqd/022:01:390
2624999167,4cyclictest141rcu_preempt22:50:152
26249991614,1cyclictest7491-21ssh22:20:122
26249991613,2cyclictest21672-21munin-run00:25:002
2624799160,1cyclictest0-21swapper/122:14:061
2624699167,8cyclictest27864-21munin-run23:50:000
2624699166,5cyclictest7524-21perf20:45:000
2624699166,5cyclictest7524-21perf20:45:000
26246991613,1cyclictest13-21ksoftirqd/021:10:370
26246991612,1cyclictest13-21ksoftirqd/022:11:110
2625399158,4cyclictest0-21swapper/320:45:143
2624999150,14cyclictest0-21swapper/200:07:342
2624799158,2cyclictest29817-21ssh23:30:471
2624799150,1cyclictest0-21swapper/123:15:251
2624699153,6cyclictest14692-21munin-run21:00:000
2624699153,3cyclictest13-21ksoftirqd/022:07:380
2624699152,2cyclictest141rcu_preempt21:58:520
2624699152,2cyclictest141rcu_preempt00:03:130
2624699151,1cyclictest141rcu_preempt21:24:340
2625399143,10cyclictest0-21swapper/300:07:333
26253991411,2cyclictest17381-21grep21:05:143
2625399140,13cyclictest0-21swapper/321:45:273
2625399140,12cyclictest11650-21wc20:50:253
2624999147,6cyclictest28590-21cpuspeed_turbos21:15:152
2624999141,1cyclictest0-21swapper/221:25:082
26249991410,3cyclictest15242-21ssh21:45:232
2624799147,3cyclictest14294-21latency_hist22:05:001
2624799141,1cyclictest13750-21ssh23:40:291
26247991410,3cyclictest26350-21seq21:13:511
26246991411,2cyclictest7218-21wc19:35:180
2625399139,3cyclictest27078-21sed20:15:253
2625399139,2cyclictest25912-21uniq22:50:163
2625399139,2cyclictest0-21swapper/323:16:333
26253991310,2cyclictest6986-21ls22:20:013
26253991310,2cyclictest6527-21tr20:40:213
26253991310,2cyclictest6527-21tr20:40:213
26253991310,2cyclictest32145-21sed21:55:243
26253991310,2cyclictest30760-21grep21:35:243
26253991310,2cyclictest15781-21grep00:20:263
26253991310,2cyclictest13465-21grep19:50:143
2625399130,6cyclictest9265-21/usr/sbin/munin22:40:163
2625399130,11cyclictest0-21swapper/321:40:153
2624999139,3cyclictest5654-21cut21:20:222
2624999133,2cyclictest32-21ksoftirqd/221:01:062
26249991312,1cyclictest55750irq/128-enp2s0-23:56:082
26249991310,2cyclictest4229-21sed23:15:272
26249991310,2cyclictest32220-21sed19:20:172
26249991310,2cyclictest32-21ksoftirqd/222:40:002
26249991310,2cyclictest30260-21ls19:15:222
26249991310,2cyclictest2773-21cron23:35:002
26249991310,2cyclictest23847-21grep20:10:192
26249991310,2cyclictest22881-21grep00:25:162
2624999130,12cyclictest5479-21grep19:30:262
2624799139,3cyclictest25310-21wc20:15:001
2624799138,3cyclictest29453-21ssh23:50:191
2624799134,8cyclictest19467-21diskmemload00:01:101
26247991310,2cyclictest7416-21ps19:35:201
26247991310,2cyclictest29516-21grep21:35:171
26247991310,2cyclictest27961-21sh23:50:001
26247991310,2cyclictest16645-21ssh21:46:141
26247991310,2cyclictest15700-21tail21:00:211
2624799130,13cyclictest0-21swapper/121:32:191
2624799130,12cyclictest32561-21cron20:30:001
2624699139,3cyclictest8799-21wc20:45:210
2624699139,2cyclictest19121-21sort20:00:210
2624699137,3cyclictest0-21swapper/021:31:240
2624699132,1cyclictest141rcu_preempt00:15:210
2624699132,1cyclictest0-21swapper/019:10:140
2624699131,3cyclictest141rcu_preempt22:47:010
26246991313,0cyclictest0-21swapper/022:30:140
26246991313,0cyclictest0-21swapper/000:10:120
26246991311,1cyclictest13-21ksoftirqd/023:14:060
26246991310,2cyclictest15026-21grep19:50:270
26246991310,2cyclictest11841-21head20:50:250
2625399129,2cyclictest7136-21grep22:00:193
2625399129,2cyclictest5416-21sed19:30:263
2625399129,2cyclictest5115-21tail23:55:183
2625399129,2cyclictest4444-21grep21:20:153
2625399129,2cyclictest381rcuc/300:00:173
2625399129,2cyclictest30220-21sed21:15:263
2625399129,2cyclictest29690-21grep00:10:133
2625399129,2cyclictest28155-21grep20:20:163
2625399129,2cyclictest23857-21tail20:10:203
2625399129,2cyclictest2069-21sort19:25:173
2625399128,3cyclictest27419-21grep19:10:173
2625399125,3cyclictest225-21systemd-journal20:00:003
2625399122,7cyclictest22797-21ssh00:25:163
26253991211,1cyclictest0-21swapper/322:05:503
2625399120,4cyclictest0-21swapper/321:25:563
2625399120,1cyclictest0-21swapper/322:58:553
2625399120,11cyclictest0-21swapper/323:12:453
2624999129,2cyclictest9515-21cat19:40:202
2624999129,2cyclictest7694-21grep00:35:202
2624999129,2cyclictest28792-21cron21:35:002
2624999129,2cyclictest20299-21tr21:10:142
2624999129,2cyclictest19992-21grep23:05:272
2624999129,2cyclictest16497-21head19:55:192
2624999129,2cyclictest11810-21grep20:50:252
2624999128,3cyclictest3979-21grep23:35:172
26249991210,2cyclictest311rcuc/200:15:222
2624799129,2cyclictest3737-21grep20:35:161
2624799129,2cyclictest27223-21grep23:10:161
2624799129,2cyclictest21816-21grep20:05:201
2624799129,2cyclictest19638-21cat20:00:271
2624799129,2cyclictest14084-21head19:50:171
2624799128,3cyclictest5396-21sh00:15:111
2624799128,3cyclictest5048-21wc19:30:221
2624799128,3cyclictest358-21ssh00:30:381
2624799128,3cyclictest12250-21cat00:00:001
26247991211,1cyclictest0-21swapper/120:45:201
26247991210,2cyclictest19467-21diskmemload00:37:311
26247991210,1cyclictest13924-21grep21:25:211
2624799120,2cyclictest225-21systemd-journal00:05:571
2624799120,12cyclictest0-21swapper/122:59:421
2624799120,12cyclictest0-21swapper/122:32:541
2624799120,11cyclictest13033-21ssh22:23:101
2624699129,3cyclictest13-21ksoftirqd/022:20:120
2624699129,2cyclictest6268-21head23:55:270
2624699129,2cyclictest29518-21sed20:20:280
2624699129,2cyclictest29198-21grep21:15:170
2624699129,2cyclictest23374-21grep20:10:160
2624699129,2cyclictest22205-21head00:05:190
2624699129,2cyclictest162712sleep021:26:420
2624699129,2cyclictest161rcuc/019:40:230
2624699129,2cyclictest15113-21cut21:00:180
2624699129,1cyclictest13-21ksoftirqd/019:55:230
2624699128,3cyclictest18576-21tr21:05:210
2624699128,3cyclictest13-21ksoftirqd/020:05:220
2624699124,7cyclictest2537-21munin-run20:35:000
2624699123,8cyclictest3075-21fschecks_time23:15:170
2624699122,3cyclictest141rcu_preempt21:46:350
2624699121,3cyclictest225-21systemd-journal19:34:540
26246991210,2cyclictest13-21ksoftirqd/023:30:310
26246991210,1cyclictest19467-21diskmemload22:35:130
2624699120,0cyclictest0-21swapper/000:20:300
2625399119,1cyclictest682-21tr20:30:153
2625399119,1cyclictest18756-21sort20:00:183
2625399118,2cyclictest965-21taskset19:22:523
2625399118,2cyclictest7449-21ssh22:20:103
2625399118,2cyclictest6343-21sh00:35:083
2625399118,2cyclictest5351-21ssh00:34:323
2625399118,2cyclictest4420-21ssh22:37:143
2625399118,2cyclictest26865-21ssh22:11:443
2625399118,2cyclictest25484-21sh21:13:113
2625399118,2cyclictest25164-21cat22:30:183
2625399118,2cyclictest23295-21sh21:50:213
2625399118,2cyclictest21099-21kernelversion23:45:193
2625399118,2cyclictest20943-21ssh21:30:103
2625399118,2cyclictest18726-21ssh23:05:163
2625399118,2cyclictest15043-21wget21:00:173
2625399118,2cyclictest12799-21taskset19:46:193
2625399118,2cyclictest11781-21sh23:40:133
2625399117,3cyclictest14678-21cron21:00:003
2624999119,1cyclictest3667-21tune2fs20:35:172
2624999119,1cyclictest23506-21tr22:10:152
2624999119,1cyclictest11182-21taskset22:02:452
2624999118,2cyclictest8416-21ssh22:40:092
2624999118,2cyclictest6266-21cut19:35:142
2624999118,2cyclictest4960-21ssh21:40:002
2624999118,2cyclictest31930-21expr22:15:142
2624999118,2cyclictest30426-21basename21:55:132
2624999118,2cyclictest29985-21df21:35:182
2624999118,2cyclictest28605-21taskset19:12:452
2624999118,2cyclictest28156-21tr20:20:162
2624999118,2cyclictest27521-21taskset22:31:272
2624999118,2cyclictest21758-21rm23:26:012
2624999118,2cyclictest17601-21expr21:05:142
2624999118,2cyclictest16264-21ssh23:42:262
2624999118,2cyclictest10966-21tr19:45:142
2624999118,1cyclictest0-21swapper/223:24:012
2624999117,3cyclictest14187-21grep20:55:272
2624999110,10cyclictest32034-21grep00:30:242
2624799118,2cyclictest6277-21ssh23:36:131
2624799118,2cyclictest29787-21head19:15:181
2624799118,2cyclictest25384-21cut22:50:131
2624799118,2cyclictest19403-21grep23:25:151
2624799118,2cyclictest18085-21sh23:24:391
2624799118,2cyclictest18025-21sh23:05:101
2624799118,2cyclictest16215-21ssh00:20:441
2624799118,2cyclictest1091-21ssh21:17:561
2624799118,2cyclictest10379-21tail22:40:251
2624799117,3cyclictest3712-21wc19:30:001
2624799115,2cyclictest225-21systemd-journal21:10:001
2624799114,5cyclictest487-21avahi-daemon19:10:221
2624799110,5cyclictest225-21systemd-journal22:06:201
2624799110,11cyclictest0-21swapper/123:00:401
2624799110,10cyclictest26236-21grep20:15:161
2624799110,10cyclictest11246-21cut20:50:191
2624699118,2cyclictest26090-21cut20:15:160
2624699118,2cyclictest16181-21tr22:25:130
2624699117,3cyclictest1139-21cron19:24:590
2624699117,2cyclictest1655-21tr19:25:160
2624699110,11cyclictest0-21swapper/023:40:120
2625399108,1cyclictest8070-21taskset19:37:353
2625399108,1cyclictest29208-21grep19:15:153
2625399108,1cyclictest10155-21cut23:00:143
2625399107,2cyclictest3000-21chrt20:35:123
2625399107,2cyclictest10393-21chrt19:41:393
2625399101,8cyclictest1037-21gdbus20:25:013
2624999108,2cyclictest6741-21basename20:40:232
2624999108,2cyclictest6741-21basename20:40:232
2624999108,2cyclictest20682-21taskset20:05:092
2624999108,1cyclictest30155-21sh00:10:142
2624999108,1cyclictest1762-21sh22:55:192
2624999108,1cyclictest16818-21sshd22:05:372
2624999107,2cyclictest28079-21sh23:10:252
2624999107,2cyclictest18873-21grep20:00:202
2624999105,4cyclictest2515-21cron20:35:002
2624999100,9cyclictest30438-21basename20:25:132
2624999100,9cyclictest19862-21rm23:45:052
2624999100,1cyclictest0-21swapper/200:01:382
2624799108,2cyclictest6297-21if_err_enp2s020:40:191
2624799108,2cyclictest6297-21if_err_enp2s020:40:191
2624799108,1cyclictest28210-21expr20:20:161
2624799108,1cyclictest16048-21cat19:55:161
2624799108,1cyclictest102992sleep119:40:281
2624799107,2cyclictest670-21tr20:30:151
2624799107,2cyclictest32157-21grep19:20:171
2624799107,2cyclictest12815-21tr20:55:151
2624799107,2cyclictest12085-21taskset19:45:201
2624699107,2cyclictest13-21ksoftirqd/020:35:160
2624699100,2cyclictest0-21swapper/020:25:250
262539997,1cyclictest20756-21grep20:05:143
262499998,1cyclictest25844-21expr20:15:142
262499998,1cyclictest1385-21chrt19:25:112
262499996,2cyclictest7769-21apache_processe20:45:132
262469996,2cyclictest10964-21grep19:45:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional