You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-11 - 07:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Jul 11, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
718229783,9sleep00-21swapper/019:06:540
528429466,9sleep30-21swapper/319:05:033
729628667,15sleep20-21swapper/219:08:232
733228265,13sleep10-21swapper/119:08:511
47832810,0sleep10-21swapper/123:08:091
287122630,0sleep10-21swapper/123:22:151
750799270,27cyclictest0-21swapper/222:59:102
750799270,27cyclictest0-21swapper/221:27:202
750299255,1cyclictest25-21ksoftirqd/121:10:131
750799240,24cyclictest0-21swapper/222:48:052
750799240,24cyclictest0-21swapper/222:41:112
750799230,1cyclictest0-21swapper/221:17:002
7497992323,0cyclictest0-21swapper/021:59:310
7511992220,1cyclictest0-21swapper/322:33:093
7507992221,1cyclictest0-21swapper/223:03:342
750799221,20cyclictest0-21swapper/220:35:182
750299220,3cyclictest0-21swapper/120:35:141
7511992119,2cyclictest4349-21kworker/3:100:20:233
751199210,1cyclictest0-21swapper/300:36:303
750799210,20cyclictest0-21swapper/222:39:152
750799210,20cyclictest0-21swapper/222:11:142
7511992018,2cyclictest2855-21kworker/3:222:35:333
7511992017,2cyclictest9919-21kworker/3:000:01:533
7497992012,2cyclictest2097-21basename00:05:000
7511991917,2cyclictest9919-21kworker/3:023:01:123
7507991917,1cyclictest32-21ksoftirqd/200:31:542
750799190,19cyclictest0-21swapper/223:11:322
750799190,18cyclictest0-21swapper/219:10:132
750299194,14cyclictest25752-21sh00:37:401
750299193,2cyclictest26833-21kworker/1:023:41:371
7502991916,2cyclictest25-21ksoftirqd/122:45:141
7502991915,1cyclictest25-21ksoftirqd/123:14:271
750299190,1cyclictest0-21swapper/119:35:171
7511991817,1cyclictest0-21swapper/321:44:163
751199180,1cyclictest0-21swapper/322:47:093
751199180,18cyclictest0-21swapper/323:43:193
750299183,6cyclictest0-21swapper/122:05:231
7502991816,1cyclictest0-21swapper/123:33:131
7502991816,1cyclictest0-21swapper/123:33:131
7502991815,2cyclictest25-21ksoftirqd/121:30:211
7502991815,2cyclictest241rcuc/100:20:121
7502991815,2cyclictest141rcu_preempt23:15:171
750299180,17cyclictest0-21swapper/120:30:131
750299180,17cyclictest0-21swapper/100:32:371
749799184,5cyclictest22825-21cron23:00:000
749799181,14cyclictest0-21swapper/021:00:140
7511991716,1cyclictest0-21swapper/323:12:583
750799179,1cyclictest141rcu_preempt21:37:462
750799172,1cyclictest141rcu_preempt22:25:152
750799170,16cyclictest0-21swapper/221:55:382
7502991714,2cyclictest25-21ksoftirqd/119:50:001
7502991712,4cyclictest25-21ksoftirqd/122:35:111
750299171,13cyclictest25-21ksoftirqd/121:45:191
750299170,17cyclictest0-21swapper/120:40:141
749799179,7cyclictest6095-21latency_hist22:50:000
7497991716,1cyclictest13-21ksoftirqd/000:26:090
751199162,13cyclictest0-21swapper/322:22:023
7511991615,1cyclictest0-21swapper/300:29:043
7511991614,2cyclictest9919-21kworker/3:023:59:433
7511991614,2cyclictest2855-21kworker/3:221:12:033
751199160,13cyclictest12027-21gltestperf20:25:163
7507991612,4cyclictest32-21ksoftirqd/223:25:292
750799160,16cyclictest0-21swapper/222:04:272
750799160,14cyclictest0-21swapper/223:46:212
750799160,0cyclictest0-21swapper/200:23:542
750299161,1cyclictest32452-21diskmemload00:01:081
750299160,1cyclictest0-21swapper/123:29:181
749799162,8cyclictest2426-21grep21:10:260
7497991614,2cyclictest13-21ksoftirqd/022:01:240
750799159,5cyclictest32-21ksoftirqd/221:54:122
7507991511,3cyclictest21367-21rm00:35:142
750799150,15cyclictest0-21swapper/200:03:592
7502991514,1cyclictest2728-21ssh23:45:181
750299150,1cyclictest0-21swapper/123:55:031
750299150,15cyclictest0-21swapper/122:33:231
749799154,5cyclictest31212-21perf23:05:000
749799150,8cyclictest13417-21awk22:35:000
749799150,2cyclictest0-21swapper/023:42:080
749799150,15cyclictest0-21swapper/021:39:210
749799150,14cyclictest0-21swapper/000:12:520
751199143,6cyclictest32452-21diskmemload21:15:063
7511991411,2cyclictest3229-21ssh23:45:233
7511991411,2cyclictest11033-21cron20:25:013
750799149,4cyclictest3500-21apache222:18:302
7507991410,3cyclictest2641-21basename00:05:142
7507991410,2cyclictest11861-21ssh00:10:142
750799140,6cyclictest141rcu_preempt21:21:102
750299148,3cyclictest225-21systemd-journal19:10:261
750299141,2cyclictest241rcuc/121:17:311
7502991410,3cyclictest4302-21ssh22:29:341
7502991410,3cyclictest25-21ksoftirqd/122:00:121
7502991410,3cyclictest11543-21grep20:25:141
750299140,4cyclictest22450-21ssh23:37:531
749799147,6cyclictest32240-21ssh23:25:000
7497991412,1cyclictest0-21swapper/000:39:330
749799140,9cyclictest20577-21perf20:45:000
751199139,3cyclictest26785-21sed20:55:213
751199139,3cyclictest2158-21basename21:30:163
751199139,3cyclictest19231-21readlink21:59:593
751199139,3cyclictest13759-21tr00:30:223
751199137,4cyclictest21282-21unixbench_multi19:35:273
751199131,3cyclictest0-21swapper/322:17:443
7511991311,2cyclictest55650irq/127-enp2s0-22:03:293
751199131,11cyclictest0-21swapper/323:15:313
7511991310,2cyclictest5574-21awk22:10:203
7511991310,2cyclictest28893-21grep21:45:243
7511991310,2cyclictest27444-21cat00:20:003
7511991310,2cyclictest23709-21wc22:40:233
7511991310,2cyclictest17655-21grep20:35:253
7511991310,2cyclictest14460-21cron22:55:003
7511991310,2cyclictest1033-21sed20:00:273
751199130,13cyclictest0-21swapper/300:12:553
750799139,2cyclictest20702-21cat20:45:012
750799136,3cyclictest141rcu_preempt23:15:142
750799132,2cyclictest577-21ntp_states23:05:212
7507991310,2cyclictest5806-21chrt00:25:282
7507991310,2cyclictest21403-21grep00:15:272
7507991310,2cyclictest1380-21grep21:10:162
750799130,7cyclictest28493-21/usr/sbin/munin22:05:152
750799130,12cyclictest16988-21cron19:30:002
7502991312,1cyclictest0-21swapper/121:35:131
7502991311,2cyclictest27310-21kworker/u8:219:53:391
7502991310,3cyclictest25-21ksoftirqd/100:05:561
7502991310,2cyclictest4915-21sed20:10:231
7502991310,2cyclictest4670-21cat21:50:231
7502991310,2cyclictest25-21ksoftirqd/122:14:381
7502991310,2cyclictest14460-21cron22:54:591
7502991310,2cyclictest12802-21basename22:15:151
7502991310,2cyclictest11829-21grep19:15:271
750299130,2cyclictest141rcu_preempt00:29:501
749799135,4cyclictest225-21systemd-journal23:55:000
749799135,4cyclictest225-21systemd-journal21:45:000
7497991310,2cyclictest8595-21grep22:50:260
7497991310,2cyclictest2510-21munin-run21:49:590
7497991310,2cyclictest22376-21grep22:40:150
7497991310,2cyclictest18138-21tr21:20:160
7497991310,2cyclictest13-21ksoftirqd/023:49:350
749799130,12cyclictest22892-21sed22:20:260
749799130,12cyclictest17607-21grep23:35:130
751199129,2cyclictest8832-21cat19:10:193
751199129,2cyclictest8639-21cron20:20:013
751199129,2cyclictest3441-21head20:05:263
751199129,2cyclictest25216-21head23:20:173
751199129,2cyclictest24482-21grep19:45:153
751199129,2cyclictest18227-21ls19:30:193
751199129,2cyclictest15933-21wc20:35:013
751199129,2cyclictest15094-21basename22:55:143
751199128,3cyclictest22440-21grep20:45:273
751199128,3cyclictest15490-21cut19:25:173
751199127,3cyclictest13655-21tr19:20:213
751199124,4cyclictest225-21systemd-journal20:00:003
751199121,10cyclictest0-21swapper/323:38:323
751199120,11cyclictest31077-21tail21:05:173
750799129,2cyclictest28163-21cat21:45:172
750799129,2cyclictest21741-21cron19:40:002
750799129,2cyclictest170572sleep223:35:012
750799129,2cyclictest16633-21http23:35:012
750799129,2cyclictest16633-21http23:35:012
750799129,2cyclictest1605-21wc20:05:002
750799129,2cyclictest13604-21grep19:20:202
750799128,3cyclictest25298-21sh20:54:592
750799127,1cyclictest32-21ksoftirqd/223:55:012
750799122,1cyclictest141rcu_preempt23:58:362
7507991212,0cyclictest0-21swapper/223:41:092
750799120,12cyclictest0-21swapper/221:32:152
750799120,11cyclictest0-21swapper/222:30:282
750799120,0cyclictest0-21swapper/219:45:142
750299129,2cyclictest21821-21grep22:20:171
750299129,2cyclictest19004-21cat00:14:591
750299129,2cyclictest17986-21head19:30:181
750299129,2cyclictest17067-21wc19:30:001
750299128,3cyclictest23968-21ssh22:40:261
750299128,3cyclictest11257-21tr23:50:201
750299126,5cyclictest18714-21apt21:40:121
750299125,6cyclictest12757-21tr19:20:141
750299124,2cyclictest25-21ksoftirqd/122:55:491
750299122,8cyclictest21368-21sed20:45:161
750299122,1cyclictest32452-21diskmemload21:55:261
7502991211,1cyclictest25-21ksoftirqd/123:03:411
749799129,2cyclictest93532sleep022:12:490
749799129,2cyclictest8499-21head23:10:170
749799129,2cyclictest3803-21ls00:05:200
749799129,2cyclictest32299-21df23:05:150
749799129,2cyclictest3101-21chrt22:28:400
749799129,2cyclictest30586-21grep19:55:220
749799129,2cyclictest19468-21cat19:35:000
749799129,2cyclictest16818-21sleep21:20:000
749799129,2cyclictest15667-21wc22:35:250
749799129,2cyclictest14731-21cat19:24:590
749799129,2cyclictest14166-21grep20:30:150
749799129,2cyclictest13421-21sleep20:30:010
749799128,2cyclictest9961-21cat19:15:000
749799125,3cyclictest28497-21aten2_r7power_v22:05:150
749799124,3cyclictest26511-21cron19:50:000
749799123,4cyclictest225-21systemd-journal20:05:000
749799123,4cyclictest225-21systemd-journal19:54:260
749799122,5cyclictest3282-21perf00:25:000
751199118,2cyclictest6142-21ssh00:06:263
751199118,2cyclictest5100-21ssh21:50:293
751199118,2cyclictest32693-21ssh22:07:173
751199118,2cyclictest29945-21ssh21:27:053
751199118,2cyclictest18699-21taskset20:40:143
751199118,2cyclictest18603-21ssh21:20:203
751199118,2cyclictest1535-21ssh23:05:373
751199118,2cyclictest10828-21head19:15:173
751199118,2cyclictest10090-21ssh23:30:213
751199118,2cyclictest10090-21ssh23:30:213
751199117,3cyclictest2139-21ls23:25:243
751199117,2cyclictest29942-21df_inode22:25:133
751199114,3cyclictest11023-21/usr/sbin/munin23:50:183
750799119,1cyclictest3998-21wc20:10:012
750799119,1cyclictest10587-21grep20:20:262
750799118,2cyclictest26395-21grep23:20:262
750799118,2cyclictest26036-21ssh21:44:372
750799118,2cyclictest25790-21tr20:55:142
750799118,2cyclictest24309-21cat19:45:002
750799118,2cyclictest23047-21cat20:50:002
750799118,2cyclictest19389-21perf19:35:002
750799118,2cyclictest10079-21chrt19:15:052
7507991110,1cyclictest291irq_work/221:03:132
750799110,2cyclictest0-21swapper/222:51:422
750799110,10cyclictest8764-21sed20:20:012
750299118,2cyclictest32113-21taskset21:05:491
750299118,2cyclictest26889-21ssh21:25:191
750299118,2cyclictest23186-21ntpq19:40:191
750299118,2cyclictest18623-21cat21:20:201
750299117,3cyclictest29629-21tr19:55:151
750299113,4cyclictest19514-21ssh00:15:141
749799118,2cyclictest8137-21unixbench-2d20:15:240
749799118,2cyclictest7607-21ssh21:33:140
749799118,2cyclictest5158-21ssh23:27:360
749799118,2cyclictest3806-21grep21:50:140
749799118,2cyclictest31938-21expr21:05:250
749799118,2cyclictest25754-21chrt21:25:090
749799118,2cyclictest25383-21cat20:55:000
749799118,2cyclictest22106-21ssh23:56:510
749799118,2cyclictest21316-21tail19:35:260
749799118,2cyclictest20803-21tr00:15:210
749799118,2cyclictest2020-21seq20:05:150
749799118,2cyclictest18220-21chrt22:18:150
749799118,2cyclictest17012-21ls19:30:000
749799118,2cyclictest16348-21expr23:15:140
749799118,2cyclictest15636-21ssh00:31:270
749799111,1cyclictest0-21swapper/023:30:080
749799111,1cyclictest0-21swapper/023:30:080
751199108,1cyclictest28058-21cut21:00:153
751199108,1cyclictest27627-21cut19:50:163
751199108,1cyclictest10295-21ssh21:35:103
751199107,2cyclictest4984-21sort20:10:223
751199107,2cyclictest22269-21expr19:40:133
751199100,9cyclictest23649-21expr20:50:133
750799108,1cyclictest6275-21munin-run20:15:002
750799108,1cyclictest29022-21taskset19:54:062
750799108,1cyclictest14997-21grep20:30:222
750799107,2cyclictest30595-21expr21:05:132
750799107,2cyclictest29604-21grep19:55:152
750799106,3cyclictest21056-21tr22:20:122
750799101,6cyclictest12024-21gltestperf20:25:162
750299108,1cyclictest6793-21expr20:15:121
750299108,1cyclictest35632sleep120:05:561
750299108,1cyclictest23151-21taskset20:50:021
750299107,2cyclictest32131-21expr20:00:141
750299107,2cyclictest26280-21grep20:55:161
750299107,2cyclictest25-21ksoftirqd/121:00:161
750299106,3cyclictest9160-21tr20:20:141
749799108,1cyclictest9572-21tune2fs20:20:160
749799108,1cyclictest25768-21cat20:55:140
749799107,2cyclictest5493-21run-parts20:10:280
749799107,2cyclictest16279-21tr20:35:140
749799107,2cyclictest10352-21grep19:15:140
74979986,2cyclictest23736-21awk19:40:250
74979986,1cyclictest20958-21wget20:45:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional