You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-17 - 02:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 17, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2290229178,9sleep20-21swapper/219:08:522
2264729078,8sleep00-21swapper/019:05:330
2298728667,15sleep30-21swapper/319:09:573
2272028359,19sleep10-21swapper/119:06:301
271692640,1sleep125-21ksoftirqd/100:09:101
2308099311,3cyclictest19855-21/usr/sbin/munin21:50:191
2308099305,24cyclictest10823-21cron00:00:001
2308099300,2cyclictest0-21swapper/123:22:221
2308399290,27cyclictest0-21swapper/221:28:462
2308099294,6cyclictest32640-21sendmail_mailqu22:55:231
23080992921,7cyclictest24340-21tr20:20:001
2308099274,3cyclictest31982-21munin-run23:14:591
2308399260,26cyclictest0-21swapper/223:40:512
23083992513,7cyclictest32-21ksoftirqd/219:40:002
2308399250,23cyclictest0-21swapper/222:21:242
2308099250,8cyclictest0-21swapper/123:28:171
2308099250,1cyclictest0-21swapper/123:15:261
2308399240,2cyclictest0-21swapper/223:04:552
2308399240,24cyclictest0-21swapper/221:53:132
2308399240,0cyclictest0-21swapper/200:03:242
2308399240,0cyclictest0-21swapper/200:03:242
2308099242,21cyclictest15694-21diskmemload23:35:021
2308099241,6cyclictest0-21swapper/122:50:141
2308099241,4cyclictest0-21swapper/123:30:191
2308099240,3cyclictest0-21swapper/121:10:361
23078992422,2cyclictest13-21ksoftirqd/022:15:390
2308399232,20cyclictest0-21swapper/221:17:512
2308099237,4cyclictest15694-21diskmemload23:06:241
23078992322,1cyclictest0-21swapper/022:41:200
2307899230,1cyclictest0-21swapper/021:52:230
2308399220,21cyclictest0-21swapper/223:39:062
2308399220,1cyclictest0-21swapper/221:49:302
2308399220,0cyclictest0-21swapper/223:14:192
2308099227,1cyclictest15694-21diskmemload21:37:481
2308099220,0cyclictest0-21swapper/121:31:081
2308399210,2cyclictest0-21swapper/222:05:342
2308399210,2cyclictest0-21swapper/221:56:082
2308399210,21cyclictest0-21swapper/223:46:542
2308399210,1cyclictest0-21swapper/200:12:092
2308399210,0cyclictest0-21swapper/223:23:292
2308099210,20cyclictest0-21swapper/121:05:191
23088992015,5cyclictest0-21swapper/320:55:133
2308399200,20cyclictest0-21swapper/223:08:422
2308399200,1cyclictest0-21swapper/222:41:302
23080992019,1cyclictest5195-21ssh22:00:251
2308099200,5cyclictest0-21swapper/122:38:141
23088991918,1cyclictest0-21swapper/300:03:403
23088991918,1cyclictest0-21swapper/300:03:403
2308899190,1cyclictest0-21swapper/322:00:223
2308399190,19cyclictest0-21swapper/222:58:482
2308399190,19cyclictest0-21swapper/200:19:012
2308399190,0cyclictest141rcu_preempt23:25:272
2308099195,2cyclictest0-21swapper/100:17:551
2308099192,3cyclictest0-21swapper/120:55:251
2308099190,18cyclictest0-21swapper/123:50:081
23078991919,0cyclictest0-21swapper/000:16:470
23078991916,1cyclictest0-21swapper/023:06:190
2308399180,18cyclictest0-21swapper/222:50:592
2308399180,18cyclictest0-21swapper/200:39:302
2308099182,6cyclictest141rcu_preempt00:14:071
2308099181,4cyclictest20917-21ssh00:24:561
2308899178,8cyclictest39-21ksoftirqd/300:06:273
23088991715,1cyclictest0-21swapper/323:11:053
2308399170,17cyclictest0-21swapper/222:11:132
2308399170,14cyclictest0-21swapper/222:16:462
2308099171,2cyclictest0-21swapper/121:45:151
2308099170,1cyclictest0-21swapper/123:46:541
2308899160,16cyclictest0-21swapper/321:45:563
23083991612,3cyclictest11439-21idleruntime-cro22:05:002
2308399160,14cyclictest0-21swapper/221:33:172
2308399160,14cyclictest0-21swapper/200:25:252
2308099169,6cyclictest3276-21head21:40:201
2308099167,2cyclictest11140-21sh22:42:581
2308099164,2cyclictest9761-21apt-get19:50:001
23080991615,1cyclictest0-21swapper/100:04:101
23080991615,1cyclictest0-21swapper/100:04:101
2308099161,2cyclictest0-21swapper/121:20:081
2308099160,6cyclictest15694-21diskmemload21:57:261
2308099160,15cyclictest22005-21cat00:25:161
23078991615,1cyclictest0-21swapper/021:15:270
2307899161,15cyclictest0-21swapper/021:28:460
2307899160,1cyclictest0-21swapper/000:32:290
2307899160,1cyclictest0-21swapper/000:02:490
2307899160,1cyclictest0-21swapper/000:02:480
23088991512,2cyclictest7946-21cron23:20:003
2308899150,14cyclictest0-21swapper/323:02:483
2308399152,1cyclictest141rcu_preempt00:08:052
23083991513,1cyclictest0-21swapper/221:00:002
23083991511,3cyclictest3719-21wc21:40:252
2308399151,13cyclictest0-21swapper/222:46:242
2308399150,12cyclictest7658-21apt-get19:45:142
2308099158,6cyclictest8316-21ssh00:35:421
2308099158,6cyclictest5776-21grep20:45:241
2308099157,1cyclictest13527-21sh23:03:541
2308099154,6cyclictest25096-1kworker/u9:222:10:131
2308099151,6cyclictest38150irq/131-snd_hda21:17:381
2308099151,13cyclictest23769-21apache222:08:111
2308099150,15cyclictest0-21swapper/122:24:541
2308099150,14cyclictest2332-21ssh00:32:351
2307899154,3cyclictest141rcu_preempt22:59:400
2307899152,7cyclictest14751-21perf20:00:000
23078991514,1cyclictest0-21swapper/023:23:150
2307899150,15cyclictest0-21swapper/000:27:440
2308899149,3cyclictest5927-21cut19:40:163
23088991411,2cyclictest7285-21sed22:40:213
23088991411,2cyclictest3802-21ls20:45:003
23088991410,2cyclictest1392-21cron20:40:003
23083991414,0cyclictest0-21swapper/221:39:522
2308399141,1cyclictest17414-21cat21:10:192
2308399141,12cyclictest8046-21sort23:20:002
23083991410,2cyclictest21199-21wc20:10:222
2308399140,1cyclictest0-21swapper/221:20:382
2308399140,1cyclictest0-21swapper/200:21:182
2308399140,13cyclictest27261-21wc23:30:232
2308099148,6cyclictest0-21swapper/123:41:431
2308099148,5cyclictest32061-21expr20:35:151
2308099148,5cyclictest28977-21chrt20:28:521
2308099148,5cyclictest15416-21ssh22:25:401
2308099147,6cyclictest17223-21cat21:30:001
2308099147,2cyclictest31207-21ssh22:15:481
2308099147,2cyclictest27218-21taskset22:33:181
2308099147,2cyclictest27218-21taskset22:33:171
2308099140,4cyclictest225-21systemd-journal20:30:231
2308099140,2cyclictest0-21swapper/122:45:121
2307899143,5cyclictest31995-21perf23:15:000
2307899143,0cyclictest0-21swapper/023:58:510
23078991413,1cyclictest0-21swapper/021:30:290
2307899141,13cyclictest0-21swapper/023:52:000
2307899140,8cyclictest19517-21perf20:10:000
2307899140,14cyclictest0-21swapper/022:33:260
2307899140,14cyclictest0-21swapper/022:33:260
2308899139,3cyclictest2137-21sed19:30:263
2308899139,3cyclictest19561-21cat20:10:003
2308899139,2cyclictest20812-21wc20:10:173
2308899132,6cyclictest0-21swapper/300:18:303
23088991310,2cyclictest9601-21turbostat.cron23:40:003
23088991310,2cyclictest129652sleep323:41:043
2308899130,13cyclictest0-21swapper/322:54:503
2308899130,12cyclictest15113-21grep22:25:283
2308899130,11cyclictest8677-21ls21:25:003
2308399139,3cyclictest5751-21grep20:45:242
2308399139,3cyclictest31854-21grep22:35:262
23083991312,1cyclictest0-21swapper/222:30:232
23083991312,1cyclictest0-21swapper/222:30:232
23083991310,2cyclictest30989-21grep20:30:262
23083991310,2cyclictest13828-21tr21:05:152
2308399130,13cyclictest0-21swapper/223:59:032
2308099138,4cyclictest6795-21expr20:50:151
2308099137,2cyclictest19191-21taskset20:06:181
2308099137,1cyclictest30991-21cat19:25:131
2308099136,2cyclictest12566-21date19:55:001
2308099136,1cyclictest1562-21tr19:30:191
2308099135,1cyclictest0-21swapper/119:40:221
2308099132,2cyclictest225-21systemd-journal20:00:171
2308099131,2cyclictest225-21systemd-journal19:50:151
23080991310,2cyclictest12880-21grep21:00:231
2308099130,4cyclictest225-21systemd-journal19:40:001
23078991310,2cyclictest8993-21grep23:00:250
23078991310,2cyclictest4957-21grep22:00:220
23078991310,2cyclictest10907-21cron21:00:000
2307899130,13cyclictest0-21swapper/000:05:100
2307899130,10cyclictest0-21swapper/021:21:280
2308899129,2cyclictest9715-21tr23:20:193
2308899129,2cyclictest8547-21munin-plugin-st20:55:003
2308899129,2cyclictest5707-21sed20:45:233
2308899129,2cyclictest29650-21taskset21:17:473
2308899129,2cyclictest29378-21ssh21:36:283
2308899129,2cyclictest27679-21grep20:25:163
2308899129,2cyclictest21969-21wc22:10:263
2308899129,2cyclictest16483-21grep21:10:143
2308899129,2cyclictest16468-21tail23:05:163
2308899129,2cyclictest15085-21head22:45:153
2308899129,2cyclictest13513-21chrt22:05:233
2308899129,2cyclictest12343-21sh19:55:003
2308899129,2cyclictest11301-21grep21:00:113
2308899128,3cyclictest28414-21sed21:55:173
2308899128,3cyclictest14259-21grep19:55:263
2308899128,3cyclictest12616-21apt22:25:003
2308899125,4cyclictest15701-21turbostat.cron21:10:013
2308899125,3cyclictest225-21systemd-journal21:55:003
2308899122,7cyclictest29372-21ssh22:15:153
2308899121,7cyclictest10828-21sh00:00:003
23088991211,1cyclictest2044-21ssh21:40:133
2308899120,11cyclictest27448-21head23:30:243
2308899120,0cyclictest0-21swapper/321:26:073
2308399129,2cyclictest31230-21grep19:25:142
2308399129,2cyclictest25582-21wc20:20:182
2308399129,2cyclictest14672-21wc22:25:242
2308399128,3cyclictest27226-21tr20:25:132
2308399120,12cyclictest0-21swapper/223:54:132
2308399120,12cyclictest0-21swapper/200:30:392
2308099126,1cyclictest26354-21chrt20:20:391
2308099126,1cyclictest2613-21uname20:40:191
2308099126,1cyclictest21538-21seq20:10:561
2308099125,2cyclictest24897-21cat19:10:241
2308099121,2cyclictest225-21systemd-journal19:15:081
2307899129,2cyclictest987-21grep20:35:260
2307899129,2cyclictest6018-21head19:40:160
2307899129,2cyclictest30058-21head20:30:170
2307899129,2cyclictest2053-21sed23:15:260
2307899129,2cyclictest16496-21tr21:10:140
2307899129,2cyclictest12225-21cat21:00:160
2307899128,3cyclictest27642-21grep20:25:150
2307899122,9cyclictest23634-21kworker/u8:122:25:120
23078991211,1cyclictest0-21swapper/023:40:510
2307899120,1cyclictest0-21swapper/021:41:120
2307899120,12cyclictest0-21swapper/023:46:510
2308899118,2cyclictest8757-21head19:45:203
2308899118,2cyclictest4237-21basename00:15:003
2308899118,2cyclictest31443-21cron20:35:003
2308899118,2cyclictest3082sleep319:30:003
2308899118,2cyclictest30648-21cut22:35:173
2308899118,2cyclictest30271-21tr00:30:153
2308899118,2cyclictest26655-21ssh00:27:483
2308899118,2cyclictest24075-21grep19:10:163
2308899118,2cyclictest23585-21wc20:15:243
2308899118,2cyclictest21205-21ssh21:31:433
2308899118,2cyclictest19219-21grep23:45:163
2308899118,2cyclictest17543-21sort23:25:173
2308899118,2cyclictest13834-21ssh00:20:163
2308899117,3cyclictest7755-21cat00:35:263
2308899117,3cyclictest21813-21tr22:30:153
2308899117,3cyclictest21813-21tr22:30:153
2308899114,7cyclictest26946-21/usr/sbin/munin23:50:283
23088991110,1cyclictest0-21swapper/322:57:063
2308399118,2cyclictest7884-21sort20:50:252
2308399118,2cyclictest32473-21cut20:35:172
2308399118,2cyclictest29802-21tail19:20:262
2308399118,2cyclictest26012-21tr19:15:142
2308399118,2cyclictest2537-21taskset19:34:362
2308399118,2cyclictest24094-21tail19:10:162
2308399118,2cyclictest17873-21grep20:05:142
2308399118,2cyclictest15774-21grep20:00:172
2308399117,2cyclictest13396-21sed19:55:172
2308399113,4cyclictest225-21systemd-journal20:15:192
2308099118,2cyclictest28806-21fschecks_time19:20:161
2307899119,1cyclictest5565-21head22:20:170
2307899119,1cyclictest28853-21grep19:20:170
2307899118,2cyclictest6734-21grep00:35:160
2307899118,2cyclictest6718-21grep20:50:150
2307899118,2cyclictest6174-21dump-pmu-power20:49:590
2307899118,2cyclictest4840-21chrt19:38:100
2307899118,2cyclictest3313-21ssh00:14:190
2307899118,2cyclictest3309-21grep20:40:260
2307899118,2cyclictest30672-21ssh21:37:290
2307899118,2cyclictest29818-21ssh22:54:240
2307899118,2cyclictest29659-21ssh22:35:010
2307899118,2cyclictest20596-21tr22:10:150
2307899118,2cyclictest13869-21grep21:05:160
2307899118,2cyclictest13405-21grep00:20:150
2307899118,2cyclictest12170-21tr22:05:130
2307899112,5cyclictest477-21rs:main0
23078991110,1cyclictest0-21swapper/023:31:130
2308899108,1cyclictest27711-21taskset19:18:153
2308899107,2cyclictest30246-21cron19:25:003
2308899107,2cyclictest15952-21cut20:00:193
2308399108,1cyclictest10784-21sed19:50:152
2308399102,3cyclictest225-21systemd-journal20:40:252
2307899108,2cyclictest3030-21ntp_states23:35:210
2307899108,2cyclictest20186-21expr20:10:120
2307899108,1cyclictest25015-21expr20:20:140
2307899108,1cyclictest23017-21if_err_enp0s31f20:15:170
2307899108,1cyclictest10481-21expr19:50:140
2307899107,2cyclictest26858-21cut19:15:190
2307899107,2cyclictest16799-21ssh23:25:100
2307899107,2cyclictest10210-21ssh21:45:020
2307899100,9cyclictest32534-21taskset19:28:580
2307899100,9cyclictest15780-21find22:45:220
230889996,2cyclictest4013-21ntpq19:35:203
230839991,2cyclictest0-21swapper/219:41:232
230839990,8cyclictest0-21swapper/221:00:092
230789998,1cyclictest27903-21expr21:55:140
230789997,1cyclictest23579-21tr19:10:130
230789994,2cyclictest225-21systemd-journal19:45:140
230889986,1cyclictest25229-21grep20:20:153
230789986,1cyclictest1079-21tune2fs19:30:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional