You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-22 - 16:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Jun 22, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27351210184,13sleep00-21swapper/007:07:270
2731228465,15sleep10-21swapper/107:06:561
2725028269,9sleep20-21swapper/207:06:082
2725128168,9sleep30-21swapper/307:06:083
2764699331,31cyclictest0-21swapper/210:28:512
2764299290,26cyclictest0-21swapper/110:54:471
2764699250,25cyclictest0-21swapper/211:00:562
27642992524,1cyclictest0-21swapper/112:08:221
2764299250,3cyclictest0-21swapper/110:24:011
2764299241,21cyclictest7604-21cut09:40:161
2763999240,2cyclictest0-21swapper/012:33:000
2764699231,22cyclictest0-21swapper/210:14:282
2764299236,7cyclictest11961-21latency_hist11:20:001
27642992216,5cyclictest25-21ksoftirqd/111:10:271
2763999222,11cyclictest0-21swapper/009:45:120
2764999218,9cyclictest39-21ksoftirqd/310:40:123
2764699215,13cyclictest21711-21/usr/sbin/munin11:45:202
2764699210,1cyclictest0-21swapper/212:32:592
2764299210,2cyclictest0-21swapper/111:00:131
2764299202,1cyclictest9805-21apt-get07:40:121
2763999200,1cyclictest0-21swapper/009:50:130
2763999200,17cyclictest0-21swapper/010:27:590
27646991918,1cyclictest0-21swapper/209:11:552
27646991916,2cyclictest29764-21ssh11:10:272
2763999190,2cyclictest0-21swapper/009:39:550
2763999190,18cyclictest0-21swapper/011:35:530
27639991817,1cyclictest0-21swapper/010:52:560
2763999180,18cyclictest0-21swapper/011:44:480
2763999180,18cyclictest0-21swapper/011:44:470
2764999174,12cyclictest0-21swapper/311:42:283
2764999174,12cyclictest0-21swapper/311:42:273
27649991716,1cyclictest0-21swapper/311:23:533
2764999170,17cyclictest0-21swapper/309:42:343
27642991712,2cyclictest20653-21diskmemload10:01:451
2764299170,0cyclictest0-21swapper/111:43:121
2764299170,0cyclictest0-21swapper/111:43:111
2763999176,3cyclictest1142-21runrttasks10:41:190
2763999172,5cyclictest0-21swapper/010:30:130
2764699162,2cyclictest0-21swapper/209:25:472
27646991615,1cyclictest0-21swapper/212:15:142
27642991613,2cyclictest25-21ksoftirqd/109:56:451
27642991610,5cyclictest516-21gdbus08:25:231
2764299160,4cyclictest0-21swapper/108:35:141
2764299160,1cyclictest0-21swapper/111:50:471
2763999163,11cyclictest17370-21timerwakeupswit12:20:250
27639991612,3cyclictest13-21ksoftirqd/012:35:250
27639991612,1cyclictest13-21ksoftirqd/010:13:240
2763999160,1cyclictest0-21swapper/012:16:460
2763999160,15cyclictest0-21swapper/012:04:210
2764999151,1cyclictest0-21swapper/311:39:173
2764999150,15cyclictest0-21swapper/309:46:193
2764999150,13cyclictest0-21swapper/307:30:123
2764699152,2cyclictest0-21swapper/208:25:142
2764699150,2cyclictest0-21swapper/211:31:562
2764699150,0cyclictest0-21swapper/209:50:122
2764299158,6cyclictest5749-21tail07:30:171
2764299158,5cyclictest23851-21expr09:50:141
27642991511,3cyclictest0-21swapper/110:06:041
2764299150,15cyclictest0-21swapper/109:29:341
27639991511,1cyclictest13-21ksoftirqd/010:39:560
2763999151,0cyclictest141rcu_preempt09:10:290
2763999150,13cyclictest0-21swapper/011:47:180
271232150,1sleep00-21swapper/011:28:530
2764999142,5cyclictest0-21swapper/309:21:313
27649991413,1cyclictest0-21swapper/309:25:383
27649991412,1cyclictest0-21swapper/311:20:003
27649991411,2cyclictest821-21df07:20:153
27649991410,3cyclictest25397-21tail10:10:173
27649991410,3cyclictest17818-21sed09:00:273
27649991410,3cyclictest15256-21sed08:55:273
2764999140,13cyclictest5268-21wc08:35:273
2764699142,9cyclictest32-21ksoftirqd/209:20:132
27646991411,2cyclictest23745-21head12:05:172
2764299148,4cyclictest15701-21http09:00:001
2764299147,6cyclictest16024-21/usr/sbin/munin12:00:211
27642991410,3cyclictest8602-21grep12:15:221
2764299140,14cyclictest0-21swapper/110:56:271
2763999142,1cyclictest141rcu_preempt09:30:380
27639991413,1cyclictest0-21swapper/008:25:140
27639991412,2cyclictest13-21ksoftirqd/009:23:300
27639991411,2cyclictest0-21swapper/012:13:490
2763999140,5cyclictest0-21swapper/011:15:050
2763999140,3cyclictest0-21swapper/012:25:350
2763999140,2cyclictest0-21swapper/011:23:240
2764999139,3cyclictest20094-21head08:00:183
2764999139,3cyclictest10395-21wc10:20:223
2764999139,2cyclictest1344-21grep10:15:163
2764999133,3cyclictest225-21systemd-journal07:54:593
27649991312,1cyclictest39-21ksoftirqd/310:28:573
27649991310,2cyclictest8688-21head10:00:163
27649991310,2cyclictest31312-21cat08:25:013
27649991310,2cyclictest29209-21grep11:10:213
27649991310,2cyclictest12232-21wc08:50:193
27649991310,2cyclictest1085-21ls10:35:013
2764999130,1cyclictest0-21swapper/311:49:563
2764699139,2cyclictest28567-21cron11:30:002
2764699136,5cyclictest16110-21apt-get09:00:032
2764699135,7cyclictest20653-21diskmemload10:48:402
2764699133,4cyclictest11915-21munin-run07:45:002
2764699131,8cyclictest14038-21df_inode11:40:152
2764699131,8cyclictest14038-21df_inode11:40:152
2764699131,1cyclictest0-21swapper/211:24:062
27646991310,2cyclictest8193-21ls10:00:142
27646991310,2cyclictest4224-21grep11:15:132
27646991310,2cyclictest25023-21ssh11:08:152
27646991310,2cyclictest23101-21ls08:05:222
27646991310,2cyclictest10975-21chrt12:35:592
2764299139,3cyclictest29101-21basename08:20:121
2764299139,3cyclictest25127-21ssh10:49:111
2764299139,2cyclictest7845-21wc08:40:251
27642991311,1cyclictest38150irq/131-snd_hda07:25:151
27642991310,2cyclictest6434-21rm11:35:221
27642991310,2cyclictest28583-21head07:10:161
27642991310,2cyclictest22955-21wc09:50:001
27642991310,2cyclictest20447-21tr08:00:211
27642991310,2cyclictest17182-21tr07:55:151
27642991310,2cyclictest13316-21tr07:45:211
2764299130,1cyclictest0-21swapper/109:21:511
27639991312,1cyclictest0-21swapper/011:34:390
27639991311,1cyclictest0-21swapper/007:40:120
27639991310,2cyclictest8890-21wc07:35:240
27639991310,2cyclictest492-21apache_processe10:15:110
27639991310,2cyclictest14642-21grep08:55:190
2763999130,1cyclictest141rcu_preempt09:25:570
2763999130,12cyclictest0-21swapper/011:55:480
2763999130,12cyclictest0-21swapper/009:43:530
2764999129,2cyclictest7643-21grep07:35:143
2764999129,2cyclictest31943-21grep07:15:263
2764999129,2cyclictest31812-21sh09:55:113
2764999129,2cyclictest30228-21grep11:50:133
2764999129,2cyclictest27529-21head10:50:163
2764999129,2cyclictest25887-21grep08:10:283
2764999129,2cyclictest24242-21cut12:05:223
2764999129,2cyclictest24052-21sh10:10:013
2764999129,2cyclictest23006-21grep08:05:203
2764999129,2cyclictest22256-21head09:10:183
2764999129,2cyclictest12189-21cat11:00:183
2764999128,3cyclictest9906-21ssh12:35:243
2764999124,4cyclictest225-21systemd-journal09:10:013
27649991212,0cyclictest0-21swapper/312:17:463
2764999120,12cyclictest0-21swapper/312:21:553
2764699129,2cyclictest9867-21ssh10:40:082
2764699129,2cyclictest30044-21cat08:20:182
2764699129,2cyclictest23166-21tail09:30:152
2764699129,2cyclictest2253-21wc10:15:242
2764699129,2cyclictest16957-21cut12:20:202
2764699129,2cyclictest16483-21sed12:00:262
2764699129,2cyclictest14573-21cat08:55:202
2764699129,2cyclictest12793-21grep08:50:272
2764699129,1cyclictest26403-21munin-run08:15:002
2764699128,3cyclictest0-21swapper/211:58:122
2764699128,2cyclictest5151-21rm09:58:202
2764699124,4cyclictest225-21systemd-journal08:05:002
2764699124,2cyclictest537-21snmpd07:27:352
2764699123,6cyclictest14550-21cron09:44:592
27646991211,1cyclictest0-21swapper/212:14:022
27646991211,1cyclictest0-21swapper/210:52:562
27646991211,1cyclictest0-21swapper/209:39:072
27646991210,1cyclictest28764-21cron08:20:002
27646991210,1cyclictest18099-21grep10:05:272
2764699120,12cyclictest0-21swapper/211:35:522
2764699120,11cyclictest0-21swapper/211:50:572
2764299129,2cyclictest2479-21ssh10:35:181
2764299129,2cyclictest19229-21grep10:25:261
2764299129,2cyclictest16869-21tail09:00:181
2764299129,2cyclictest1395-21uname07:20:191
2764299128,3cyclictest9511-21cron07:40:001
2764299128,1cyclictest32352-21taskset07:20:011
2764299127,4cyclictest15422-21ssh12:39:261
2764299123,6cyclictest20058-21df_inode11:05:141
2764299123,5cyclictest17859-21sshd12:20:391
27642991210,1cyclictest8836-21grep08:45:141
2764299120,1cyclictest0-21swapper/112:12:381
2764299120,1cyclictest0-21swapper/111:30:511
2763999129,2cyclictest32718-21cut07:20:130
2763999129,2cyclictest29184-21tr11:10:220
2763999129,2cyclictest21232-21ssh11:05:240
2763999129,2cyclictest19021-21cron07:59:590
2763999129,2cyclictest10797-21grep10:20:260
2763999128,3cyclictest4959-21sh11:54:290
2763999126,2cyclictest13-21ksoftirqd/012:07:060
2763999120,3cyclictest0-21swapper/010:01:180
2764999118,2cyclictest8136-21taskset10:38:593
2764999118,2cyclictest6973-21df08:40:143
2764999118,2cyclictest6565-21ssh10:56:543
2764999118,2cyclictest5909-21ssh11:55:023
2764999118,2cyclictest4403-21awk07:25:283
2764999118,2cyclictest4079-21ssh12:32:063
2764999118,2cyclictest3389-21ssh09:37:483
2764999118,2cyclictest31887-21ssh09:15:333
2764999118,2cyclictest31607-21ssh11:30:513
2764999118,2cyclictest28105-21expr07:10:153
2764999118,2cyclictest2488-21ssh12:11:413
2764999118,2cyclictest24747-21head12:25:173
2764999118,2cyclictest24136-21grep09:50:153
2764999118,2cyclictest23440-21sh10:47:523
2764999118,2cyclictest20563-21ssh11:25:063
2764999118,2cyclictest18675-21chrt07:55:383
2764999118,2cyclictest15422-21sh12:00:173
2764999118,2cyclictest12305-21cut07:45:163
2764999118,2cyclictest10240-21switchtime08:45:263
2764699119,1cyclictest3190-21expr10:55:152
2764699119,1cyclictest20813-21wc09:10:002
2764699119,1cyclictest1981-21grep07:20:252
2764699119,1cyclictest1731-21cut08:30:142
2764699119,1cyclictest15322-21grep09:45:142
2764699118,2cyclictest7050-21grep08:40:152
2764699118,2cyclictest4409-21forks08:35:172
2764699118,2cyclictest4031-21ssh10:35:472
2764699118,2cyclictest31849-21taskset09:15:322
2764699118,2cyclictest28077-21ssh10:30:492
2764699118,2cyclictest19023-21cron07:59:592
2764699118,2cyclictest11018-21sh10:20:322
2764699113,5cyclictest537-21snmpd07:39:572
2764699110,9cyclictest0-21swapper/212:26:492
2764699110,10cyclictest14772-21cut07:50:152
2764299118,2cyclictest9158-21ssh11:56:131
2764299118,2cyclictest38150irq/131-snd_hda09:30:191
2764299118,2cyclictest32356-21ssh09:35:251
2764299118,2cyclictest30172-21ssh10:32:261
2764299118,2cyclictest29789-21cut09:15:131
2764299118,2cyclictest27389-21grep08:15:161
2764299118,2cyclictest26945-21ssh10:10:451
2764299118,2cyclictest22722-21ssh11:45:171
2764299118,2cyclictest20826-21grep11:25:121
2764299118,2cyclictest16407-21ssh11:22:021
2764299118,2cyclictest15581-21cat07:50:201
2764299118,2cyclictest11209-21grep08:50:141
2764299117,3cyclictest3225-21taskset08:30:531
2764299116,4cyclictest23285-21cut08:05:231
2764299110,2cyclictest0-21swapper/112:27:301
2764299110,10cyclictest20653-21diskmemload12:30:071
2764299110,10cyclictest12661-21ssh10:40:571
2764299110,10cyclictest0-21swapper/110:15:121
2763999119,2cyclictest19829-21fgrep10:45:220
2763999119,1cyclictest6674-21tr08:40:140
2763999119,1cyclictest25953-21grep08:10:270
2763999119,1cyclictest11418-21tr11:00:140
2763999118,2cyclictest6558-21cat07:30:260
2763999118,2cyclictest4680-21chrt07:29:020
2763999118,2cyclictest4322-21sed08:35:160
2763999118,2cyclictest30574-21sed08:20:240
2763999118,2cyclictest29464-21sh09:15:030
2763999118,2cyclictest29071-21cut07:10:210
2763999118,2cyclictest28752-21chrt08:19:550
2763999118,2cyclictest2187-21sh09:56:000
2763999118,2cyclictest20779-21sed08:00:260
2763999118,2cyclictest18352-21ssh10:05:330
2763999116,2cyclictest2725-21ssh10:55:130
2763999115,2cyclictest225-21systemd-journal08:55:010
2764999108,1cyclictest1405-21taskset08:30:043
2764999108,1cyclictest1118-21taskset08:29:293
2764999107,2cyclictest27026-21expr08:15:133
2764999107,2cyclictest23370-21ssh09:30:173
2764999107,2cyclictest19892-21expr11:05:143
2764999107,2cyclictest11827-21taskset07:44:063
2764699108,2cyclictest32320-21sshd07:19:402
2764699108,1cyclictest5365-21cat07:30:142
2764699108,1cyclictest30155-21cut07:15:012
2764299108,1cyclictest260322sleep108:10:341
2764299108,1cyclictest20962-21touch09:10:011
2764299108,1cyclictest20667-21basename09:10:001
2763999108,2cyclictest20193-21tail09:05:260
2763999108,1cyclictest3552-21sh08:34:590
2763999108,1cyclictest165682sleep007:54:030
2763999108,1cyclictest10406-21grep08:45:250
2763999107,2cyclictest31617-21tr07:15:220
2763999107,2cyclictest16925-21cut09:00:180
276469997,1cyclictest12180-21apache_volume07:45:132
276469992,4cyclictest10545-21sshd08:45:282
276399997,1cyclictest13821-21grep07:45:270
276399994,2cyclictest225-21systemd-journal08:05:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional