You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-28 - 11:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Mar 28, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
274129177,10sleep20-21swapper/219:05:042
484328767,16sleep10-21swapper/119:09:331
481028775,8sleep00-21swapper/019:09:060
455828471,9sleep30-21swapper/319:05:523
242482590,0sleep20-21swapper/222:41:502
495999250,1cyclictest0-21swapper/022:02:540
496899240,21cyclictest0-21swapper/200:37:502
495999240,23cyclictest0-21swapper/000:01:070
4968992323,0cyclictest0-21swapper/223:42:192
496499230,1cyclictest0-21swapper/123:02:301
496499227,1cyclictest141rcu_preempt00:28:181
496499220,18cyclictest0-21swapper/123:55:141
496499220,0cyclictest0-21swapper/121:16:191
4959992119,1cyclictest0-21swapper/023:36:340
495999210,1cyclictest0-21swapper/023:04:210
495999210,0cyclictest0-21swapper/022:12:110
496499201,2cyclictest31957-21kworker/1:021:54:421
4959992017,1cyclictest0-21swapper/023:59:200
4970991918,1cyclictest0-21swapper/321:16:433
497099190,1cyclictest0-21swapper/322:28:433
496499191,18cyclictest0-21swapper/100:10:541
495999190,1cyclictest0-21swapper/021:18:350
4964991813,4cyclictest7140-21ssh23:30:001
496499180,16cyclictest0-21swapper/100:16:351
4968991716,1cyclictest0-21swapper/220:00:112
4968991714,3cyclictest0-21swapper/222:02:172
496499173,6cyclictest225-21systemd-journal23:53:161
4964991716,1cyclictest0-21swapper/123:32:531
4964991712,5cyclictest38150irq/131-snd_hda22:33:281
4964991712,5cyclictest38150irq/131-snd_hda22:33:281
496499170,16cyclictest0-21swapper/121:24:301
496499170,16cyclictest0-21swapper/121:24:291
495999175,8cyclictest181irq_work/022:05:130
495999170,17cyclictest0-21swapper/022:58:190
496899161,1cyclictest141rcu_preempt21:37:252
496899160,9cyclictest0-21swapper/221:50:192
496899160,2cyclictest0-21swapper/221:55:082
496899160,16cyclictest0-21swapper/223:07:182
496499169,2cyclictest225-21systemd-journal21:41:161
496499162,13cyclictest0-21swapper/123:41:541
4964991611,4cyclictest25-21ksoftirqd/121:00:121
496499160,1cyclictest29956-21diskmemload21:34:331
496499160,1cyclictest0-21swapper/123:08:511
496499160,1cyclictest0-21swapper/122:42:511
4959991615,1cyclictest0-21swapper/023:29:470
497099156,7cyclictest39-21ksoftirqd/320:35:173
497099153,6cyclictest19790-21ls00:15:173
497099153,4cyclictest0-21swapper/320:20:003
497099151,7cyclictest0-21swapper/320:45:143
4970991514,1cyclictest0-21swapper/322:40:113
496899153,2cyclictest225-21systemd-journal00:01:222
496899151,13cyclictest0-21swapper/223:51:012
4968991510,4cyclictest16220-21wc21:20:182
4968991510,4cyclictest16220-21wc21:20:172
496899150,3cyclictest0-21swapper/222:08:392
496899150,1cyclictest0-21swapper/223:28:572
4964991514,1cyclictest26871-21kworker/1:222:52:151
4964991514,1cyclictest23184-21ssh22:21:561
496499150,1cyclictest0-21swapper/123:21:401
496499150,1cyclictest0-21swapper/100:09:401
496499150,13cyclictest0-21swapper/122:05:211
495999153,6cyclictest8298-21idleruntime-cro20:25:000
4959991510,3cyclictest7354-21apache200:27:260
495999150,15cyclictest0-21swapper/021:40:380
497099140,1cyclictest0-21swapper/323:55:403
497099140,1cyclictest0-21swapper/300:28:203
496899147,6cyclictest32-21ksoftirqd/200:15:042
496899144,6cyclictest487-21avahi-daemon23:55:212
4968991413,1cyclictest0-21swapper/223:13:562
4968991411,2cyclictest13020-21sort19:25:162
4968991411,2cyclictest13020-21sort19:25:152
496899140,13cyclictest0-21swapper/222:25:072
4964991412,1cyclictest0-21swapper/122:15:221
4964991411,2cyclictest0-21swapper/100:03:181
495999149,3cyclictest17128-21grep19:35:150
495999147,6cyclictest27593-21munin-run21:05:000
495999143,5cyclictest0-21swapper/023:25:010
495999142,4cyclictest225-21systemd-journal22:30:000
4959991410,3cyclictest12828-21cron22:54:590
3342141,11sleep0387-21apache_accesses23:45:110
497099139,3cyclictest13438-21wc22:35:203
497099134,4cyclictest0-21swapper/300:22:363
497099131,7cyclictest7563-21kworker/u8:122:59:183
497099131,7cyclictest11519-21sh22:35:013
497099131,7cyclictest11519-21sh22:35:003
4970991312,1cyclictest0-21swapper/323:30:143
4970991311,1cyclictest0-21swapper/321:52:433
4970991310,2cyclictest39-21ksoftirqd/323:39:473
4970991310,2cyclictest26587-21wc19:55:003
4970991310,2cyclictest13031-21tr00:30:213
497099130,5cyclictest0-21swapper/323:21:363
496899139,3cyclictest18782-21lockfile-create22:20:002
496899139,2cyclictest16506-21grep21:40:112
496899138,4cyclictest32-21ksoftirqd/220:40:122
496899136,7cyclictest17794-21apache222:11:192
496899135,3cyclictest0-21swapper/219:10:232
4968991310,2cyclictest7647-21grep19:15:152
4968991310,2cyclictest27892-21grep19:55:232
4968991310,2cyclictest11196-21grep19:20:212
4968991310,2cyclictest10148-21grep20:25:262
496899130,2cyclictest0-21swapper/200:10:142
496499139,2cyclictest25079-21head19:50:161
496499138,4cyclictest21510-21grep00:35:211
4964991310,2cyclictest7800-21tr19:15:151
4964991310,2cyclictest6435-21rm23:10:151
4964991310,2cyclictest4485-21head20:15:151
496499130,13cyclictest29956-21diskmemload22:01:551
496499130,12cyclictest9667-21tr20:25:201
496499130,12cyclictest31785-21ssh21:49:091
495999139,3cyclictest27104-21wc21:45:430
495999139,2cyclictest19811-21tr19:40:140
495999131,6cyclictest1112-21latency_hist20:09:590
4959991310,2cyclictest3741-21grep22:30:140
4959991310,2cyclictest3741-21grep22:30:130
4959991310,2cyclictest29471-21grep21:05:270
4959991310,2cyclictest1970-21grep00:05:150
4959991310,2cyclictest16031-21grep20:40:110
4959991310,2cyclictest13847-21grep22:35:250
495999130,2cyclictest0-21swapper/000:14:410
497099129,2cyclictest3047-21sed20:10:253
497099129,2cyclictest28072-21tr21:05:133
497099129,2cyclictest24025-21sh19:50:003
497099129,2cyclictest23901-21tr21:25:133
497099129,2cyclictest22804-21sh20:55:013
497099129,2cyclictest18262-21cat21:40:233
497099128,3cyclictest12590-21tail20:30:273
497099128,3cyclictest10248-21head21:55:173
497099128,2cyclictest4950-21cyclictest22:50:513
497099128,2cyclictest26182-21head21:00:183
497099128,2cyclictest12584-21grep19:25:143
497099128,2cyclictest12584-21grep19:25:133
497099124,6cyclictest19613-21cstates22:20:163
497099121,1cyclictest141rcu_preempt22:00:183
497099121,1cyclictest141rcu_preempt21:32:053
4970991211,1cyclictest55650irq/127-enp2s0-23:44:383
4970991210,1cyclictest28328-21grep22:05:273
4970991210,1cyclictest20288-21egrep00:35:143
497099120,3cyclictest0-21swapper/320:00:133
496899129,2cyclictest8442-21grep23:30:172
496899129,2cyclictest5901-21perf20:20:002
496899129,2cyclictest22136-21tr23:00:142
496899129,2cyclictest21404-21grep20:50:172
496899129,2cyclictest17830-21sort19:35:172
496899129,2cyclictest13574-21tr20:35:142
496899129,2cyclictest12535-21grep20:30:282
496899129,2cyclictest12119-21ssh22:35:132
496899128,3cyclictest26620-21ntpq21:00:202
496899128,3cyclictest22762-21cat23:20:012
496899128,3cyclictest14645-21munin-run21:20:002
496899128,1cyclictest13461-21expr00:30:262
496899124,2cyclictest225-21systemd-journal22:31:272
496899124,2cyclictest225-21systemd-journal22:31:262
496899121,5cyclictest455-21dbus-daemon00:06:582
496899120,12cyclictest0-21swapper/223:20:242
496499129,2cyclictest8074-21sshd21:35:131
496499129,2cyclictest2788-21ssh00:25:011
496499129,2cyclictest14068-21grep22:55:171
496499129,2cyclictest10737-21tail19:20:171
496499128,3cyclictest6184-21wc19:10:181
496499128,2cyclictest19835-21tr19:40:141
496499128,2cyclictest11992-21tr00:30:151
496499126,6cyclictest17794-21apache221:28:291
496499122,3cyclictest225-21systemd-journal20:55:121
495999129,2cyclictest479-21sed21:30:170
495999129,2cyclictest31586-21grep21:10:180
495999129,2cyclictest31566-21idleruntime-cro21:30:000
495999129,2cyclictest26373-21sed23:40:240
495999129,2cyclictest24943-21sort19:50:160
495999129,2cyclictest21180-21cut22:40:140
495999129,2cyclictest15322-21grep21:20:130
495999129,2cyclictest15322-21grep21:20:120
495999129,2cyclictest12487-21sort00:30:160
495999128,3cyclictest15397-21sort19:30:170
495999128,3cyclictest13415-21ssh21:38:020
495999128,2cyclictest10010-21tr19:20:140
495999124,2cyclictest29956-21diskmemload23:52:050
495999122,8cyclictest18333-21apache_volume20:45:140
495999120,11cyclictest0-21swapper/023:33:170
497099119,1cyclictest6456-21cat23:10:153
497099118,2cyclictest90492sleep323:50:153
497099118,2cyclictest8087-21sed19:15:153
497099118,2cyclictest8067-21ssh21:35:123
497099118,2cyclictest6508-21gzip19:10:213
497099118,2cyclictest32095-21grep21:10:233
497099118,2cyclictest31740-21ssh22:46:043
497099118,2cyclictest31460-21ssh00:03:163
497099118,2cyclictest30100-21ssh23:05:123
497099118,2cyclictest27397-21grep19:55:183
497099118,2cyclictest2643-21tail00:05:193
497099118,2cyclictest2509-21sh22:10:143
497099118,2cyclictest17586-21grep20:40:253
497099118,2cyclictest15148-21sh21:20:123
497099118,2cyclictest15148-21sh21:20:113
497099118,2cyclictest14274-21ssh00:11:483
497099118,2cyclictest1317-21ssh23:25:303
497099118,2cyclictest12185-21cat19:25:003
497099118,2cyclictest12131-21head22:15:193
497099118,2cyclictest10644-21sh20:30:003
497099117,3cyclictest29956-21diskmemload23:45:193
497099110,7cyclictest0-21swapper/320:20:133
497099110,1cyclictest0-21swapper/323:18:263
497099110,10cyclictest30447-21ssh21:48:063
497099110,10cyclictest0-21swapper/323:00:153
496899118,2cyclictest744-21tr23:45:142
496899118,2cyclictest7262-21tr20:20:192
496899118,2cyclictest5209-21cut22:50:162
496899118,2cyclictest3366-21cut00:25:142
496899118,2cyclictest28615-21tail21:05:152
496899118,2cyclictest27091-21ssh00:20:042
496899118,2cyclictest25068-21grep21:45:132
496899118,2cyclictest25058-21head19:50:162
496899118,2cyclictest23920-21tr21:25:132
496899118,2cyclictest23708-21cut20:55:152
496899118,2cyclictest19078-21ssh23:36:152
496899118,2cyclictest18992-21ssh22:20:022
496899118,2cyclictest16260-21rm22:56:082
496899118,2cyclictest1392-21ssh21:11:182
496899117,2cyclictest0-21swapper/222:46:542
496499118,2cyclictest7710-21cat20:20:261
496499118,2cyclictest3345-21chrt20:13:101
496499118,2cyclictest3264-21ssh23:46:031
496499118,2cyclictest32319-21ssh22:46:311
496499118,2cyclictest21889-21open_inodes20:50:211
496499118,2cyclictest17294-21grep19:35:151
496499118,2cyclictest16440-21expr23:35:141
496499118,2cyclictest15569-21cut23:15:171
496499118,2cyclictest14505-21ssh21:57:511
496499118,2cyclictest14307-21ssh22:35:391
496499118,2cyclictest13492-21cat20:35:141
496499118,2cyclictest12942-21grep19:25:161
496499118,2cyclictest12942-21grep19:25:151
496499118,2cyclictest12497-21cat20:30:281
496499117,3cyclictest4170-21wc22:10:271
496499110,10cyclictest0-21swapper/121:10:101
495999118,2cyclictest753-21ssh21:50:030
495999118,2cyclictest7272-21cat23:10:200
495999118,2cyclictest6038-21cat20:19:590
495999118,2cyclictest31605-21ssh23:05:230
495999118,2cyclictest3137-21ssh22:49:010
495999118,2cyclictest22933-21cat23:20:000
495999118,2cyclictest21214-21sed22:20:260
495999118,2cyclictest1468-21ssh00:24:000
495999118,2cyclictest13952-21grep19:25:260
495999118,2cyclictest13952-21grep19:25:250
495999118,2cyclictest12028-21uname20:30:220
495999118,2cyclictest11222-21wc21:55:250
495999117,2cyclictest12034-21ntp_offset22:15:180
495999110,11cyclictest0-21swapper/000:37:400
497099108,1cyclictest39-21ksoftirqd/319:35:173
497099108,1cyclictest331-21systemctl20:05:233
497099108,1cyclictest23134-21basename20:55:133
497099108,1cyclictest21601-21taskset19:44:283
497099108,1cyclictest15978-21ps19:30:243
496899108,1cyclictest31837-21ssh21:30:032
496899108,1cyclictest23364-21ls19:45:242
496899107,2cyclictest32248-21wc20:05:182
496899107,2cyclictest2482-21tr20:10:192
496899107,2cyclictest19780-21grep19:40:142
496899107,2cyclictest14679-21taskset19:30:062
496499108,1cyclictest28688-21ssh22:25:171
496499108,1cyclictest19217-21if_err_enp0s31f20:45:191
496499108,1cyclictest16767-21cat20:40:161
496499108,1cyclictest15548-21basename19:30:181
496499107,2cyclictest32522-21needreboot20:05:201
496499107,2cyclictest27588-21uname19:55:201
496499107,2cyclictest22106-21egrep19:45:141
496499106,3cyclictest309412sleep120:02:121
496499106,3cyclictest27876-21wget21:05:131
495999108,1cyclictest9123-21diskstats20:25:160
495999108,1cyclictest31672sleep020:10:530
495999108,1cyclictest310792sleep020:03:590
495999108,1cyclictest20466-21ssh00:15:210
495999108,1cyclictest13286-21apache_processe20:35:120
495999107,2cyclictest8325-21grep19:15:180
495999107,2cyclictest23857-21grep20:55:170
495999107,2cyclictest23062-21ntpq19:45:200
495999100,9cyclictest28671-21chrt19:58:510
49689996,2cyclictest18570-21cut20:45:162
49599997,1cyclictest6160-21sort19:10:180
49599997,1cyclictest21099-21cat20:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional