You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-18 - 21:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon May 18, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3048029279,9sleep00-21swapper/007:06:180
3042029177,9sleep20-21swapper/207:05:312
3070228666,16sleep10-21swapper/107:09:121
3071428169,8sleep30-21swapper/307:09:223
3085699270,25cyclictest0-21swapper/009:12:110
3086299250,24cyclictest0-21swapper/210:26:552
30862992423,1cyclictest0-21swapper/211:00:332
3085699242,19cyclictest0-21swapper/009:59:100
3086299230,23cyclictest0-21swapper/211:55:192
3086299230,22cyclictest0-21swapper/211:54:002
30859992318,4cyclictest477-21rs:main1
3086599225,2cyclictest24092-21diskmemload10:58:033
30862992221,1cyclictest0-21swapper/210:12:332
3086299220,21cyclictest0-21swapper/210:46:492
3085999220,2cyclictest0-21swapper/112:19:031
30856992221,1cyclictest0-21swapper/009:35:470
3086299211,1cyclictest141rcu_preempt11:28:172
3086299210,2cyclictest0-21swapper/212:39:102
3086299210,18cyclictest0-21swapper/209:40:502
30859992121,0cyclictest0-21swapper/110:07:251
3085999210,1cyclictest0-21swapper/110:55:411
3085699210,1cyclictest0-21swapper/012:05:440
3086599201,18cyclictest0-21swapper/308:55:143
3086599200,1cyclictest0-21swapper/309:45:023
3086299200,20cyclictest0-21swapper/210:39:142
3086299200,20cyclictest0-21swapper/209:12:112
3085999200,1cyclictest0-21swapper/110:35:391
30856992017,2cyclictest13-21ksoftirqd/010:08:470
3085699200,19cyclictest0-21swapper/011:34:210
30865991915,4cyclictest21113-21kworker/3:112:33:223
3086599190,2cyclictest0-21swapper/309:56:063
3086299192,17cyclictest0-21swapper/209:55:322
30862991917,1cyclictest0-21swapper/211:34:202
3086299190,1cyclictest0-21swapper/211:20:022
3086299190,19cyclictest0-21swapper/209:18:312
3085999190,1cyclictest2655-21smart_sda07:15:261
3085999190,1cyclictest0-21swapper/111:49:551
3085999190,18cyclictest0-21swapper/110:47:461
30865991815,2cyclictest21113-21kworker/3:112:25:413
3086299181,14cyclictest0-21swapper/212:05:132
3086299181,10cyclictest0-21swapper/208:40:122
3086299180,18cyclictest0-21swapper/211:36:142
30859991817,0cyclictest0-21swapper/111:14:591
3085999181,1cyclictest0-21swapper/110:01:401
3085999180,1cyclictest0-21swapper/112:23:481
3085999180,17cyclictest0-21swapper/112:08:111
30856991817,1cyclictest0-21swapper/009:45:370
3085699180,18cyclictest0-21swapper/011:46:290
3085699180,17cyclictest0-21swapper/010:20:530
3085699180,12cyclictest0-21swapper/010:39:420
30862991716,1cyclictest0-21swapper/211:46:292
3086299170,17cyclictest0-21swapper/210:52:552
3085999176,10cyclictest0-21swapper/111:30:141
30856991715,1cyclictest0-21swapper/011:18:210
3085699170,17cyclictest0-21swapper/011:24:470
30865991614,2cyclictest21113-21kworker/3:112:37:523
30865991614,1cyclictest12058-21perf10:24:543
30865991611,3cyclictest17074-21grep11:40:123
30865991611,3cyclictest17074-21grep11:40:113
30862991612,3cyclictest0-21swapper/212:32:482
3086299160,16cyclictest0-21swapper/211:19:122
3086299160,15cyclictest0-21swapper/209:31:372
3086299160,0cyclictest0-21swapper/211:05:152
3085699160,15cyclictest0-21swapper/010:45:320
3086599155,3cyclictest1549-21apache_volume09:35:133
3086299152,7cyclictest0-21swapper/209:22:372
3086299152,7cyclictest0-21swapper/209:22:372
3086299152,13cyclictest0-21swapper/211:42:182
3086299152,13cyclictest0-21swapper/211:42:172
3086299150,15cyclictest0-21swapper/209:35:462
3085999157,5cyclictest8455-21apache212:01:221
30859991511,3cyclictest4090-21wc09:55:181
3085699150,1cyclictest141rcu_preempt11:38:150
146002150,0sleep00-21swapper/011:00:110
3086599144,3cyclictest225-21systemd-journal08:37:493
30865991410,3cyclictest26418-21wc09:10:273
30865991410,3cyclictest11611-21grep11:55:273
3086599140,13cyclictest22254-21sed07:55:263
3086299149,4cyclictest18811-21grep12:00:172
3086299140,8cyclictest0-21swapper/207:15:152
3086299140,4cyclictest0-21swapper/210:24:452
3086299140,2cyclictest0-21swapper/212:26:282
3086299140,14cyclictest0-21swapper/211:12:372
30859991413,1cyclictest0-21swapper/110:32:221
30859991413,0cyclictest0-21swapper/110:44:041
30859991412,2cyclictest0-21swapper/109:25:511
30859991412,1cyclictest0-21swapper/110:24:441
30859991410,3cyclictest25139-21ssh11:25:161
3085999140,13cyclictest11507-21cat08:40:251
3085699146,7cyclictest24092-21diskmemload12:23:310
3085699140,3cyclictest0-21swapper/009:54:360
3085699140,13cyclictest0-21swapper/012:35:150
3086599139,3cyclictest31200-21cut08:15:163
3086599139,2cyclictest15308-21cat11:20:003
3086599132,9cyclictest537-21snmpd10:48:173
3086599132,4cyclictest0-21swapper/309:25:153
30865991312,1cyclictest21113-21kworker/3:112:20:423
30865991310,2cyclictest7049-21sed10:35:263
30865991310,2cyclictest18055-21rm12:00:143
30865991310,2cyclictest1608-21grep11:30:223
30865991310,2cyclictest12409-21wc07:35:263
30865991310,2cyclictest11836-21wc12:15:193
3086599130,1cyclictest0-21swapper/310:12:493
3086299139,3cyclictest29560-21cat10:30:152
3086299131,12cyclictest26052-21ssh10:08:442
30862991310,2cyclictest28581-21grep08:10:182
30862991310,2cyclictest25078-21cron09:30:002
3086299130,1cyclictest0-21swapper/209:45:302
3086299130,13cyclictest0-21swapper/210:03:072
3085999139,2cyclictest9096-21ssh09:20:131
3085999139,2cyclictest9096-21ssh09:20:131
3085999139,2cyclictest24184-21tr08:00:221
3085999138,4cyclictest26082-21ssh11:06:161
3085999138,1cyclictest7088-21ssh09:18:591
3085999133,6cyclictest3410-21wc12:10:201
3085999132,2cyclictest225-21systemd-journal07:55:001
30859991311,1cyclictest4063-21ssh10:15:101
30859991311,1cyclictest0-21swapper/110:14:581
30859991310,2cyclictest9138-21turbostat.cron11:55:011
30859991310,2cyclictest26373-21grep09:30:161
30859991310,2cyclictest16835-21grep07:45:221
3085999130,2cyclictest0-21swapper/109:40:561
3085699134,2cyclictest225-21systemd-journal11:51:550
30856991310,2cyclictest8467-21apache211:27:120
3085699130,6cyclictest10199-21sh12:15:010
3086599129,2cyclictest8705-21grep07:30:153
3086599129,2cyclictest3009-21grep11:50:253
3086599129,2cyclictest21532-21df10:25:153
3086599129,2cyclictest2011-21tail07:15:193
3086599129,2cyclictest16997-21wc11:20:183
3086599129,2cyclictest15754-21wc11:00:203
3086599129,2cyclictest10680-21sort08:40:163
3086599128,3cyclictest773-21turbostat07:15:003
3086599128,3cyclictest5950-21sort10:15:263
3086599128,3cyclictest5390-21tr08:30:153
3086599128,2cyclictest6300-21grep07:25:143
3086599124,2cyclictest225-21systemd-journal11:14:553
3086599123,2cyclictest11485-21apache_processe10:00:133
30865991210,2cyclictest39-21ksoftirqd/312:12:223
3086599120,8cyclictest13653-21ssh10:40:133
3086599120,1cyclictest0-21swapper/311:05:133
3086599120,1cyclictest0-21swapper/310:52:313
3086299129,2cyclictest9564-21grep07:30:202
3086299129,2cyclictest8676-21ssh10:17:232
3086299129,2cyclictest725-21grep08:20:152
3086299129,2cyclictest6486-21grep08:30:232
3086299129,2cyclictest26387-21chrt12:24:362
3086299129,2cyclictest19091-21cut07:50:202
3086299129,2cyclictest17177-21basename08:55:122
3086299129,2cyclictest13211-21head08:45:172
3086299129,2cyclictest11088-21tr07:35:152
3086299129,1cyclictest0-21swapper/210:41:292
3086299121,2cyclictest0-21swapper/212:15:122
30862991210,1cyclictest5130-21cat08:29:592
30862991210,1cyclictest19866-21grep09:00:142
30862991210,1cyclictest14491-21wc07:40:222
3086299120,11cyclictest21785-21ls07:55:212
3085999129,2cyclictest31117-21grep10:50:161
3085999129,2cyclictest23455-21wc09:05:251
3085999129,2cyclictest12276-21tr12:35:161
3085999129,2cyclictest11898-21wc07:35:201
3085999128,3cyclictest13103-21sort08:45:171
3085999125,7cyclictest25-21ksoftirqd/112:25:341
3085999121,4cyclictest225-21systemd-journal07:35:011
30859991210,2cyclictest38150irq/131-snd_hda11:57:091
3085999120,11cyclictest0-21swapper/111:02:581
3085699129,2cyclictest7168-21tr07:25:200
3085699129,2cyclictest32187-21grep07:10:210
3085699129,2cyclictest31144-21grep10:50:160
3085699129,2cyclictest24959-21grep11:05:280
3085699129,2cyclictest22848-21grep10:25:250
3085699129,2cyclictest17965-21sed08:55:170
3085699129,2cyclictest17357-21basename09:25:140
3085699129,2cyclictest10840-21grep12:15:140
3085699129,2cyclictest10729-21sort08:40:160
3085699128,3cyclictest29744-21run-parts10:10:230
3085699128,3cyclictest2485-21chrt10:33:140
3085699128,3cyclictest10660-21sh09:40:160
3085699126,6cyclictest8467-21apache210:55:290
3085699125,7cyclictest0-21swapper/011:10:280
3085699124,2cyclictest225-21systemd-journal09:16:540
3085699120,1cyclictest0-21swapper/009:22:370
3085699120,1cyclictest0-21swapper/009:22:360
3085699120,11cyclictest7484-21ssh12:32:070
3085699120,11cyclictest4707-21chrt07:20:200
3085699120,11cyclictest0-21swapper/011:43:000
3085699120,11cyclictest0-21swapper/011:43:000
3085699120,11cyclictest0-21swapper/010:02:570
3086599119,1cyclictest14561-21idleruntime-cro08:50:003
3086599118,2cyclictest8716-21ssh09:20:013
3086599118,2cyclictest8716-21ssh09:20:013
3086599118,2cyclictest28457-21sh11:27:093
3086599118,2cyclictest2822-21ssh09:15:383
3086599118,2cyclictest27588-21ssh12:05:223
3086599118,2cyclictest25628-21wget09:30:133
3086599118,2cyclictest25555-21tr11:45:143
3086599118,2cyclictest23655-21sh10:06:523
3086599118,2cyclictest1990-21ssh09:54:363
3086599118,2cyclictest17948-21chrt07:50:083
3086599118,2cyclictest16253-21sed08:50:213
3086599118,2cyclictest15211-21chrt07:43:313
3086599118,2cyclictest1265-21head08:20:173
3086599118,2cyclictest10686-21ssh09:40:173
3086599117,2cyclictest29596-21tr10:30:143
3086599117,2cyclictest22999-21cut08:00:133
3086599110,10cyclictest4754-21sed07:20:203
3086599110,10cyclictest10893-21ssh11:35:413
3086299119,1cyclictest7915-21grep08:35:162
3086299119,1cyclictest7214-21head10:55:212
3086299119,1cyclictest32547-21ssh09:53:092
3086299119,1cyclictest31858-21head07:10:182
3086299119,1cyclictest16385-21grep07:45:172
3086299118,2cyclictest3133-21grep12:10:172
3086299118,2cyclictest25070-21cron08:05:002
3086299118,2cyclictest15708-21if_err_enp0s31f08:50:152
3086299112,6cyclictest32-21ksoftirqd/207:25:122
3085999118,2cyclictest9804-21sendmail08:40:001
3085999118,2cyclictest9721-21cat11:35:191
3085999118,2cyclictest6324-21ssh12:31:131
3085999118,2cyclictest31717-21ssh09:52:291
3085999118,2cyclictest30778-21tr08:15:131
3085999118,2cyclictest20776-21ssh10:25:071
3085999118,2cyclictest16890-21ssh11:40:101
3085999118,2cyclictest16890-21ssh11:40:091
3085999118,2cyclictest14436-21cut07:40:211
3085999118,2cyclictest1389-21ssh09:35:091
3085999117,2cyclictest19535-21ssh11:21:421
3085999113,4cyclictest225-21systemd-journal07:30:011
3085999112,5cyclictest13325-21kworker/1:208:55:181
3085999110,10cyclictest23474-21taskset09:47:461
3085999110,10cyclictest0-21swapper/108:50:111
3085699118,2cyclictest9661-21chrt08:38:300
3085699118,2cyclictest5638-21ssh10:15:230
3085699118,2cyclictest407-21ssh09:34:360
3085699118,2cyclictest31173-21cut08:15:160
3085699118,2cyclictest29512-21ssh12:25:300
3085699118,2cyclictest25949-21cat08:05:140
3085699118,2cyclictest23100-21wget08:00:140
3085699118,2cyclictest20558-21bc07:55:110
3085699118,2cyclictest18214-21tr07:50:150
3085699118,2cyclictest17805-21ssh12:00:050
3085699118,2cyclictest16004-21sh10:40:430
3085699118,2cyclictest15882-21wc08:50:170
3085699118,2cyclictest15470-21cat07:45:000
3085699118,2cyclictest13219-21expr08:45:170
3085699118,2cyclictest1293-21hddtemp_smartct08:20:180
3085699118,2cyclictest12253-21ssh11:55:550
3085699118,2cyclictest1223-21chrt07:15:100
3085699110,10cyclictest15891-21grep07:45:150
3086599108,1cyclictest16004-21expr07:45:153
3086599107,2cyclictest3270-21cat08:25:143
3086599107,2cyclictest28076-21egrep08:10:163
3086599107,2cyclictest25463-21sort08:05:003
3086599107,2cyclictest22721-21sed09:05:183
3086299108,2cyclictest4146-21sed07:20:162
3086299108,1cyclictest30940-21cat08:15:132
3086299108,1cyclictest219352sleep209:05:072
3085999108,1cyclictest25584-21cut08:05:141
3085999108,1cyclictest1823-21grep08:20:231
3085999107,2cyclictest6797-21sort08:30:271
3085999107,2cyclictest3802-21uname08:25:181
3085999107,2cyclictest32734-21grep07:10:271
3085999107,2cyclictest28688-21grep08:10:191
3085999107,2cyclictest26023-21ssh09:10:231
3085999107,2cyclictest21314-21grep07:55:171
3085999107,2cyclictest19815-21tr09:00:141
3085999100,9cyclictest3659-21chrt07:20:131
3085699108,1cyclictest11542-21grep07:35:170
3085699107,2cyclictest9250-21tail07:30:180
3085699107,2cyclictest6281-21grep08:30:200
3085699107,2cyclictest2863-21cat08:25:110
3085699107,2cyclictest23291-21fgrep09:05:230
3085699100,9cyclictest0-21swapper/009:03:270
308659991,4cyclictest225-21systemd-journal09:04:553
308629992,4cyclictest25068-21cron08:05:002
308569997,1cyclictest28576-21fschecks_count08:10:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional