You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 22:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Thu Apr 23, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
484429465,25sleep00-21swapper/007:06:480
295128876,8sleep20-21swapper/207:05:012
500628169,8sleep30-21swapper/307:08:563
364127758,14sleep10-21swapper/107:05:161
42342640,0sleep30-21swapper/309:52:053
192832620,0sleep10-21swapper/111:19:141
36962480,0sleep30-21swapper/310:10:593
517999264,2cyclictest30400-21diskmemload10:48:573
517999253,4cyclictest0-21swapper/309:55:253
5177992523,1cyclictest0-21swapper/208:35:122
5169992521,3cyclictest13-21ksoftirqd/009:23:240
517999230,2cyclictest0-21swapper/309:40:133
517799230,23cyclictest0-21swapper/211:00:252
517999221,1cyclictest0-21swapper/309:11:373
517999220,1cyclictest0-21swapper/311:04:303
517799220,2cyclictest0-21swapper/210:54:172
516999225,17cyclictest13-21ksoftirqd/011:22:470
291242220,0sleep10-21swapper/110:26:201
133832220,0chrt13381-21sshd08:33:291
517999212,7cyclictest19675-21ntp_states10:40:203
517999212,3cyclictest0-21swapper/311:40:403
517999210,17cyclictest0-21swapper/307:25:123
517999210,0cyclictest0-21swapper/309:15:593
516999218,3cyclictest5163-21cyclictest10:56:490
517999202,7cyclictest20626-21fschecks_time07:40:163
517799200,1cyclictest0-21swapper/208:10:142
517799200,19cyclictest0-21swapper/210:47:212
517999197,1cyclictest30400-21diskmemload10:08:393
517999195,1cyclictest30400-21diskmemload09:36:153
516999190,1cyclictest0-21swapper/012:24:470
516999190,1cyclictest0-21swapper/009:31:320
517999184,5cyclictest225-21systemd-journal08:35:003
517999184,4cyclictest0-21swapper/310:26:523
517999182,9cyclictest0-21swapper/311:25:133
517999182,1cyclictest0-21swapper/310:58:183
517999181,16cyclictest28478-21sh11:05:153
517799180,1cyclictest20084-21sed10:40:252
517799180,18cyclictest0-21swapper/210:33:252
517799180,18cyclictest0-21swapper/210:09:112
517399181,1cyclictest30400-21diskmemload10:55:491
517399180,1cyclictest0-21swapper/109:00:121
516999187,1cyclictest0-21swapper/010:39:070
516999185,6cyclictest17332-21awk07:35:000
516999185,13cyclictest0-21swapper/009:36:500
5169991815,3cyclictest0-21swapper/010:11:430
516999180,18cyclictest0-21swapper/011:29:250
516999180,18cyclictest0-21swapper/009:53:330
517999179,7cyclictest18522-21cut10:20:163
517999172,2cyclictest12360-21apt-get07:25:003
517999170,16cyclictest0-21swapper/311:34:043
517999170,15cyclictest0-21swapper/312:32:293
5177991717,0cyclictest0-21swapper/210:16:502
517799170,16cyclictest0-21swapper/212:14:442
517399172,2cyclictest18016-21ssh12:15:521
516999178,3cyclictest13-21ksoftirqd/012:29:210
516999170,1cyclictest0-21swapper/010:27:430
517999168,7cyclictest912-21grep08:05:253
517999168,7cyclictest16174-21sort07:30:203
517999160,5cyclictest0-21swapper/312:20:213
517999160,16cyclictest0-21swapper/309:22:323
517799160,0cyclictest0-21swapper/212:07:522
5173991612,1cyclictest17614-21ssh11:56:171
517399161,1cyclictest30400-21diskmemload11:36:411
517399160,15cyclictest0-21swapper/111:02:201
516999169,6cyclictest161rcuc/009:55:300
516999164,6cyclictest27570-21cat11:05:000
516999162,12cyclictest13-21ksoftirqd/010:22:430
5169991615,1cyclictest13-21ksoftirqd/009:29:240
5169991615,1cyclictest13-21ksoftirqd/008:07:480
516999161,3cyclictest13-21ksoftirqd/010:51:540
516999160,0cyclictest0-21swapper/009:48:380
517999158,6cyclictest31728-21ssh11:45:233
517999158,6cyclictest29189-21taskset08:00:003
517999158,6cyclictest26328-21ssh09:26:063
517999158,6cyclictest25071-21wc08:55:243
517999158,6cyclictest10805-21wc10:15:223
517999158,6cyclictest10171-21ssh11:52:023
517999158,2cyclictest7707-21cut08:20:213
517999158,2cyclictest7707-21cut08:20:203
517999157,7cyclictest32728-21grep12:25:153
517999157,2cyclictest8664-21ssh10:53:323
517999157,2cyclictest4068-21cat08:15:013
517999157,2cyclictest20791-21ssh11:20:113
517999157,2cyclictest17840-21ssh10:39:593
517999152,3cyclictest225-21systemd-journal12:38:473
517999152,2cyclictest225-21systemd-journal11:20:003
517999151,4cyclictest0-21swapper/312:15:013
517999151,13cyclictest13565-21apache_volume11:35:133
517999150,14cyclictest3627-21ssh12:07:403
5177991512,3cyclictest32-21ksoftirqd/209:46:232
517799150,15cyclictest0-21swapper/212:31:372
5173991513,2cyclictest30400-21diskmemload10:48:091
517399150,14cyclictest0-21swapper/112:06:111
516999158,6cyclictest32361-21missed_timers12:05:200
516999153,6cyclictest19696-21munin-run07:40:000
516999152,12cyclictest161rcuc/011:50:490
5169991513,1cyclictest0-21swapper/012:16:290
5169991511,3cyclictest30400-21diskmemload11:03:590
5169991510,3cyclictest13-21ksoftirqd/012:03:060
517999148,5cyclictest29056-21ssh12:03:433
517999148,5cyclictest19472-21taskset07:37:223
517999148,2cyclictest1735-21ssh09:30:443
517999148,1cyclictest19391-21ssh10:00:513
517999147,2cyclictest29516-21ssh09:48:003
517999147,2cyclictest26034-21cut09:00:133
517999146,2cyclictest5450-21wc11:10:203
517999140,5cyclictest225-21systemd-journal07:55:003
517999140,5cyclictest22359-21cron12:00:003
517799149,3cyclictest23235-21cron08:55:002
5177991411,2cyclictest20874-21basename08:50:002
5177991410,3cyclictest5663-21awk11:50:002
5177991410,3cyclictest10107-21grep10:35:122
517799140,4cyclictest0-21swapper/210:59:012
517799140,12cyclictest0-21swapper/211:29:482
517399146,4cyclictest225-21systemd-journal08:45:001
516999149,3cyclictest18674-21tr10:40:140
516999148,4cyclictest0-21swapper/011:12:170
5169991411,2cyclictest10676-21df07:20:140
5169991410,2cyclictest0-21swapper/012:10:190
516999140,0cyclictest0-21swapper/009:11:110
199472140,0chrt0-21swapper/012:36:380
517999137,5cyclictest22244-21cut08:50:203
517999137,2cyclictest7368-21ssh10:33:103
517999133,4cyclictest16071-21sed12:15:173
517999132,2cyclictest225-21systemd-journal08:44:563
517799139,3cyclictest28262-21grep11:05:142
517799139,3cyclictest15503-21tr09:20:152
517799133,7cyclictest22021-21ssh11:20:192
517799133,1cyclictest141rcu_preempt09:12:582
517799132,3cyclictest30400-21diskmemload10:20:102
5177991312,1cyclictest0-21swapper/211:55:122
5177991310,3cyclictest138682sleep211:15:242
5177991310,2cyclictest9914-21wc09:55:212
5177991310,2cyclictest9913-21munin-plugin-st07:19:592
5177991310,2cyclictest531-21grep12:25:172
5177991310,2cyclictest31898-21grep08:05:132
5177991310,2cyclictest25121-21grep08:55:242
5177991310,2cyclictest17195-21tr09:40:202
5177991310,2cyclictest16796-21tr12:15:212
5177991310,2cyclictest1292-21grep09:30:262
517399139,3cyclictest5509-21grep07:10:121
517399139,3cyclictest23535-21ls07:45:181
517399138,4cyclictest17165-21taskset07:33:121
517399138,4cyclictest10580-21ssh11:14:011
517399132,7cyclictest18225-21wc12:35:251
5173991312,1cyclictest0-21swapper/110:00:131
5173991311,1cyclictest0-21swapper/111:27:141
5173991310,2cyclictest9618-21grep08:25:161
5173991310,2cyclictest27531-21rm11:24:101
5173991310,2cyclictest25343-21grep07:50:171
5173991310,2cyclictest10013-21grep10:15:171
517399130,9cyclictest0-21swapper/111:08:171
517399130,2cyclictest0-21swapper/110:35:531
517399130,12cyclictest30795-21cat12:05:001
516999139,1cyclictest30400-21diskmemload12:30:140
516999136,7cyclictest13-21ksoftirqd/008:50:130
516999135,6cyclictest161rcuc/010:30:110
516999135,6cyclictest13-21ksoftirqd/011:44:360
516999134,6cyclictest13-21ksoftirqd/011:35:310
516999132,11cyclictest30400-21diskmemload09:15:460
5169991313,0cyclictest0-21swapper/008:30:130
5169991311,1cyclictest0-21swapper/009:40:140
5169991310,2cyclictest9425-21ssh10:15:140
5169991310,2cyclictest5443-21tr07:10:120
5169991310,2cyclictest4116-21wc08:15:000
5169991310,2cyclictest26369-21sed07:50:260
516999130,7cyclictest225-21systemd-journal07:45:000
517999129,2cyclictest9571-21sort08:25:163
517999129,2cyclictest5696-21tr07:10:143
517999127,1cyclictest61592sleep308:18:113
517999127,1cyclictest29410-21chrt09:05:193
517999127,1cyclictest23712-21tr07:45:203
517999126,1cyclictest31068-21sed08:00:263
517999124,5cyclictest7789-21apache_volume07:15:133
517799129,2cyclictest9013-21aten2_r7power_e08:25:132
517799129,2cyclictest4949-21grep08:15:182
517799129,2cyclictest23978-21grep07:45:232
517799129,2cyclictest20677-21grep07:40:162
517799129,2cyclictest16565-21grep12:35:132
517799129,2cyclictest1500-21df10:10:142
517799129,2cyclictest11951-21grep08:30:152
517799128,3cyclictest16311-21grep07:30:212
517799122,8cyclictest0-21swapper/207:20:122
517799122,4cyclictest26682-21cut10:25:172
5177991210,1cyclictest28787-21grep07:55:262
5177991210,1cyclictest18347-21grep07:35:172
517399129,2cyclictest6867-21grep08:20:141
517399129,2cyclictest6867-21grep08:20:141
517399129,2cyclictest32104-21tr08:05:131
517399129,2cyclictest31646-21cron12:24:591
517399129,2cyclictest31468-21cat11:45:201
517399129,2cyclictest24215-21head08:55:161
517399129,2cyclictest23783-21cat09:45:011
517399129,2cyclictest22057-21grep11:40:141
517399129,2cyclictest21582-21grep07:40:251
517399129,2cyclictest19777-21grep10:40:211
517399129,2cyclictest1888-21tr08:10:141
517399129,2cyclictest1809-21cat10:30:101
517399129,2cyclictest14763-21cat08:35:181
517399129,2cyclictest14661-21munin-run09:20:001
517399128,3cyclictest14625-21ssh09:58:381
517399128,2cyclictest6421-21cat09:15:001
517399121,8cyclictest38150irq/131-snd_hda07:15:141
5173991210,1cyclictest0-21swapper/109:30:171
517399120,3cyclictest0-21swapper/110:14:041
517399120,11cyclictest8924-21head09:35:211
517399120,11cyclictest27788-21grep07:55:161
516999129,2cyclictest29031-21sed09:05:170
516999129,2cyclictest20294-21idleruntime-cro11:20:010
516999129,2cyclictest14929-21cron07:30:000
516999128,3cyclictest14737-21wc08:35:170
516999126,4cyclictest13-21ksoftirqd/011:33:420
516999126,4cyclictest13-21ksoftirqd/010:04:080
516999125,6cyclictest0-21swapper/010:05:200
516999123,4cyclictest161rcuc/010:45:140
516999121,6cyclictest29216-21sendmail08:00:000
5169991210,2cyclictest13-21ksoftirqd/011:58:430
517999114,2cyclictest18899-21grep08:45:143
517999110,3cyclictest225-21systemd-journal08:37:543
517799118,2cyclictest7667-21grep07:15:012
517799118,2cyclictest7511-21ssh09:35:122
517799118,2cyclictest7383-21sed11:50:202
517799118,2cyclictest29930-21cat08:00:142
517799118,2cyclictest27787-21ssh09:27:162
517799118,2cyclictest26858-21wc07:55:002
517799118,2cyclictest25522-21ssh12:00:562
517799118,2cyclictest23624-21ssh12:20:052
517799118,2cyclictest17092-21grep08:40:172
517799118,2cyclictest13151-21ssh09:18:492
517799118,2cyclictest10975-21ssh11:33:252
517799118,2cyclictest10115-21ssh11:13:382
517799117,3cyclictest29978-21sed09:05:252
517799116,2cyclictest18706-21sed10:00:262
517799111,8cyclictest0-21swapper/211:35:132
517799111,7cyclictest0-21swapper/211:40:122
517799110,1cyclictest0-21swapper/209:52:052
517399119,1cyclictest17968-21tr10:20:131
517399118,2cyclictest897-21cut09:50:171
517399118,2cyclictest7934-21rm12:30:091
517399118,2cyclictest6936-21grep11:50:161
517399118,2cyclictest5581-21ssh10:51:071
517399118,2cyclictest5488-21cut11:30:131
517399118,2cyclictest4463-21cut08:15:161
517399118,2cyclictest27540-21sh10:05:431
517399118,2cyclictest21398-21grep08:50:141
517399118,2cyclictest18901-21tr08:45:141
517399118,2cyclictest15204-21ssh09:20:131
517399118,2cyclictest1351-21ssh12:25:231
517399118,2cyclictest12935-21tr07:25:141
517399117,3cyclictest28765-21df09:05:151
517399117,3cyclictest24082-21ssh09:45:061
517399117,3cyclictest18687-21cut07:35:201
517399117,2cyclictest14963-21ssh12:15:001
517399116,4cyclictest12155-21taskset07:22:351
517399113,4cyclictest0-21swapper/109:25:131
516999118,2cyclictest29971-21ssh11:45:020
516999118,2cyclictest19496-21head08:45:170
516999118,2cyclictest17479-21tr08:40:190
516999117,2cyclictest13-21ksoftirqd/007:45:100
516999112,6cyclictest13-21ksoftirqd/007:15:130
517799108,1cyclictest12952-21cut07:25:132
517799107,2cyclictest6834-21cut08:20:152
517799107,2cyclictest6834-21cut08:20:142
517799107,2cyclictest27474-21cat09:00:242
517399107,2cyclictest29443-21taskset08:00:021
516999109,1cyclictest0-21swapper/009:00:120
516999108,1cyclictest8936-21taskset08:25:080
516999108,1cyclictest5406-21cut08:15:230
516999108,1cyclictest25453-21chrt08:58:040
516999107,2cyclictest8010-21wc08:20:260
516999107,2cyclictest8010-21wc08:20:250
516999100,9cyclictest31010-21cat08:00:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional