You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 23:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Jun 27, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
356529177,8sleep00-21swapper/007:09:480
351428864,19sleep10-21swapper/107:09:091
336628572,9sleep30-21swapper/307:07:133
324228269,9sleep20-21swapper/207:05:352
366399220,5cyclictest0-21swapper/208:05:132
365999220,2cyclictest0-21swapper/109:34:231
365899220,3cyclictest0-21swapper/011:38:350
367099200,19cyclictest3797-21kworker/3:210:46:403
366399201,4cyclictest0-21swapper/211:37:072
366399201,1cyclictest0-21swapper/211:50:452
3659992019,1cyclictest0-21swapper/109:00:511
367099191,17cyclictest0-21swapper/308:10:143
366399190,19cyclictest0-21swapper/209:19:052
3658991919,0cyclictest0-21swapper/012:39:510
365899190,12cyclictest5921-21tr11:50:130
3670991817,1cyclictest39-21ksoftirqd/309:53:533
3670991817,1cyclictest0-21swapper/309:45:413
366399182,0cyclictest141rcu_preempt11:15:302
3663991817,1cyclictest31505-21kworker/u8:009:11:472
366399180,1cyclictest0-21swapper/212:28:542
366399180,17cyclictest0-21swapper/210:35:222
365999183,1cyclictest141rcu_preempt12:27:491
365999180,1cyclictest0-21swapper/111:02:551
365999180,18cyclictest0-21swapper/111:56:391
366399173,1cyclictest26891-21apt-get11:05:022
366399172,1cyclictest141rcu_preempt09:41:542
366399170,8cyclictest225-21systemd-journal12:24:592
366399170,1cyclictest0-21swapper/212:18:272
366399170,12cyclictest0-21swapper/211:00:192
365999173,13cyclictest12201-21grep07:25:211
3659991716,1cyclictest25-21ksoftirqd/111:16:341
365999170,1cyclictest0-21swapper/109:54:051
365899171,9cyclictest0-21swapper/011:00:130
367099160,0cyclictest0-21swapper/309:23:273
3663991614,1cyclictest32-21ksoftirqd/209:28:302
366399160,1cyclictest0-21swapper/212:33:482
365999164,1cyclictest141rcu_preempt12:00:181
365999160,15cyclictest0-21swapper/109:20:231
365899165,6cyclictest14575-21munin-run12:15:000
3658991616,0cyclictest0-21swapper/010:47:350
365899160,2cyclictest0-21swapper/012:20:260
367099159,5cyclictest9110-21awk10:35:013
367099159,5cyclictest9110-21awk10:35:003
367099154,11cyclictest0-21swapper/310:53:433
366399152,2cyclictest0-21swapper/207:30:132
3663991514,1cyclictest0-21swapper/211:47:312
3663991514,1cyclictest0-21swapper/211:47:302
366399151,1cyclictest141rcu_preempt09:38:532
3663991510,3cyclictest10819-21cat07:25:012
366399150,1cyclictest0-21swapper/212:12:372
3659991513,1cyclictest0-21swapper/112:33:501
365899152,6cyclictest28873-21idleruntime-cro09:10:010
3658991512,2cyclictest28882-21diskmemload11:11:570
365899151,10cyclictest3472-21ssh10:50:200
3658991510,4cyclictest13159-21cron07:30:000
367099147,4cyclictest8394-21sh07:20:003
367099147,4cyclictest8394-21sh07:20:003
367099141,12cyclictest11035-21/usr/sbin/munin10:35:223
3670991410,3cyclictest9467-21head10:15:173
366399144,5cyclictest16961-21apt-get10:20:132
3663991411,3cyclictest32-21ksoftirqd/210:12:252
3663991411,2cyclictest9084-21grep08:25:262
3663991411,2cyclictest29484-21head11:25:182
366399140,13cyclictest9622-21head09:55:262
366399140,13cyclictest830-21tr09:50:232
366399140,13cyclictest0-21swapper/211:57:422
366399140,13cyclictest0-21swapper/209:23:352
3659991412,1cyclictest3675-21sh10:30:451
3659991412,1cyclictest3675-21sh10:30:451
3659991411,2cyclictest13450-21sendmail-msp09:20:001
3659991410,3cyclictest16993-21sed12:15:261
365999140,1cyclictest0-21swapper/110:14:291
365899149,3cyclictest5300-21cat11:50:000
365899149,3cyclictest5300-21cat11:50:000
365899149,3cyclictest2614-21wc08:15:010
365899143,5cyclictest29435-21apt-get09:10:150
365899142,5cyclictest141rcu_preempt11:27:400
365899141,7cyclictest24829-21munin-run10:25:000
3658991410,2cyclictest0-21swapper/009:45:260
365899140,13cyclictest0-21swapper/011:55:140
367099138,3cyclictest27498-21head09:05:173
367099135,4cyclictest225-21systemd-journal10:25:003
3670991310,2cyclictest6045-21rm11:50:133
3670991310,2cyclictest4859-21basename08:20:003
3670991310,2cyclictest31371-21grep09:10:293
3670991310,2cyclictest30591-21tr08:05:153
367099130,1cyclictest0-21swapper/311:43:023
367099130,12cyclictest8969-21tr09:55:213
367099130,12cyclictest0-21swapper/311:49:433
367099130,12cyclictest0-21swapper/311:49:423
3663991313,0cyclictest0-21swapper/211:42:362
366399131,1cyclictest28882-21diskmemload11:13:532
3663991310,2cyclictest5212-21df11:30:172
3663991310,2cyclictest19885-21grep10:40:262
3663991310,2cyclictest18043-21wc07:40:002
3663991310,2cyclictest10952-21grep08:30:222
366399130,12cyclictest4194-21grep07:10:172
366399130,12cyclictest0-21swapper/210:30:512
366399130,12cyclictest0-21swapper/210:30:512
366399130,11cyclictest0-21swapper/210:55:142
365999139,3cyclictest17441-21wc10:00:211
3659991313,0cyclictest0-21swapper/111:50:121
3659991310,2cyclictest28131-21sh11:25:001
3659991310,2cyclictest22152-21ssh09:25:011
3659991310,2cyclictest16970-21grep09:40:261
365999130,12cyclictest0-21swapper/110:47:541
365999130,12cyclictest0-21swapper/110:25:151
365899139,3cyclictest4140-21grep07:10:170
365899139,3cyclictest21479-21ssh10:41:360
365899139,3cyclictest15442-21fgrep09:20:230
365899131,8cyclictest0-21swapper/010:15:140
365899131,6cyclictest22693-21munin-run07:50:000
365899131,6cyclictest19326-21munin-run08:50:000
3658991310,2cyclictest32115-21open_inodes12:05:230
3658991310,2cyclictest25721-21ssh10:05:210
3658991310,2cyclictest14325-21cron08:40:000
3658991310,2cyclictest12034-21grep11:15:130
365899130,12cyclictest32407-21sh08:09:590
367099129,2cyclictest8063-21grep12:30:153
367099129,2cyclictest4129-21tr07:10:173
367099129,2cyclictest29120-21ssh11:25:173
367099129,2cyclictest25122-21grep10:05:173
367099129,2cyclictest23493-21grep07:50:153
367099129,2cyclictest22636-21chrt10:42:303
367099129,2cyclictest15910-21grep07:35:153
367099129,2cyclictest14564-21cron09:40:003
367099129,2cyclictest1339-21grep12:25:273
367099129,2cyclictest12961-21head08:35:183
367099128,3cyclictest7191-21ssh09:15:243
367099128,2cyclictest20428-21tr11:00:233
367099126,3cyclictest20788-21systemctl11:40:003
367099120,12cyclictest0-21swapper/311:32:213
366399129,2cyclictest30842-21grep12:05:152
366399129,2cyclictest27142-21grep10:45:172
366399129,2cyclictest25194-21sed10:05:172
366399129,2cyclictest17576-21cut10:00:222
366399128,3cyclictest8905-21tr10:15:152
366399128,3cyclictest28637-21taskset09:07:062
366399128,3cyclictest25997-21grep09:00:262
366399128,3cyclictest1618-21sendmail_mailtr08:10:232
366399127,4cyclictest0-21swapper/209:45:222
366399127,1cyclictest55750irq/128-enp2s0-12:02:242
366399120,11cyclictest12152-21cat07:25:202
365999129,2cyclictest9430-21sed07:20:171
365999129,2cyclictest573-21chrt12:05:391
365999129,2cyclictest4848-21tr08:20:001
365999129,2cyclictest28447-21grep08:00:161
365999129,2cyclictest20795-21tr07:45:151
365999129,2cyclictest13181-21chrt12:13:571
365999128,2cyclictest6014-21sh07:15:001
365999127,4cyclictest5205-21ssh10:51:081
365999126,3cyclictest28882-21diskmemload11:41:251
365999122,9cyclictest28882-21diskmemload09:10:381
365999122,8cyclictest24809-21cron10:25:011
3659991211,1cyclictest0-21swapper/108:12:141
3659991210,1cyclictest25-21ksoftirqd/108:35:161
365999120,4cyclictest0-21swapper/112:23:501
365999120,10cyclictest27425-21sh08:00:001
365899129,2cyclictest9105-21ls10:35:010
365899129,2cyclictest9105-21ls10:35:000
365899129,2cyclictest7531-21apt-key12:30:000
365899129,2cyclictest7065-21grep09:35:150
365899129,2cyclictest26079-21sed07:55:170
365899129,2cyclictest21367-21ssh11:20:200
365899129,2cyclictest16456-21grep12:15:220
365899129,2cyclictest1214-21grep09:50:270
365899128,3cyclictest6693-21cat09:15:190
365899128,3cyclictest30305-21wc12:05:010
365899128,2cyclictest9876-21tr10:35:150
365899126,4cyclictest14285-21ssh10:56:420
365899122,6cyclictest8157-21ssh11:31:380
365899122,6cyclictest0-21swapper/009:40:200
365899121,6cyclictest8420-21munin-run07:20:000
365899121,6cyclictest8420-21munin-run07:20:000
365899120,7cyclictest30314-21basename09:30:000
367099119,1cyclictest15369-21grep08:40:193
367099118,2cyclictest974-21ssh09:30:563
367099118,2cyclictest9458-21sed07:20:183
367099118,2cyclictest9025-21ls08:25:263
367099118,2cyclictest7511-21head12:10:183
367099118,2cyclictest7202-21cron08:25:003
367099118,2cyclictest32602-21ssh10:10:113
367099118,2cyclictest31964-21cat12:05:213
367099118,2cyclictest27696-21cut11:05:153
367099118,2cyclictest27679-21chrt08:00:063
367099118,2cyclictest26642-21fgrep07:55:223
367099118,2cyclictest25198-21ssh09:26:003
367099118,2cyclictest24842-21expr09:00:143
367099118,2cyclictest24558-21ssh11:22:133
367099118,2cyclictest24303-21chrt12:20:163
367099118,2cyclictest24043-21cron12:40:003
367099118,2cyclictest23642-21sed08:55:263
367099118,2cyclictest17662-21ssh09:40:533
367099118,2cyclictest17496-21tr08:45:153
367099118,2cyclictest15918-21tail11:55:273
367099118,2cyclictest15899-21ssh10:00:093
367099118,2cyclictest10132-21tr08:30:173
367099117,3cyclictest15067-21grep07:30:263
367099114,5cyclictest15160-21ssh12:15:143
367099113,4cyclictest225-21systemd-journal11:00:003
367099112,6cyclictest22401-21ssh12:00:143
367099111,5cyclictest0-21swapper/311:15:133
3670991110,1cyclictest5889-21kworker/u8:011:11:103
367099110,6cyclictest0-21swapper/310:26:413
366399119,1cyclictest25870-21df07:55:172
366399118,2cyclictest2552-21ssh09:32:072
366399118,2cyclictest23793-21chrt08:56:102
366399118,2cyclictest2188-21ssh10:50:102
366399118,2cyclictest21047-21grep11:20:182
366399118,2cyclictest12303-21aten2_r7power_p08:35:162
366399114,6cyclictest26946-21sensors10:25:252
366399114,6cyclictest0-21swapper/207:45:132
366399110,10cyclictest5413-21cat08:20:142
365999119,1cyclictest18312-21grep08:45:201
365999118,2cyclictest8403-21awk07:20:011
365999118,2cyclictest8403-21awk07:20:001
365999118,2cyclictest5866-21sed08:20:181
365999118,2cyclictest28868-21cron09:10:001
365999118,2cyclictest27072-21ssh10:06:011
365999118,2cyclictest24220-21grep09:45:191
365999118,2cyclictest22244-21tr08:55:141
365999118,2cyclictest20514-21chrt12:37:131
365999118,2cyclictest18946-21ssh10:40:191
365999118,2cyclictest18263-21ssh11:38:021
365999118,2cyclictest14723-21grep08:40:151
365999118,2cyclictest1427-21sh11:08:461
365999118,2cyclictest1338-21sh11:47:061
365999118,2cyclictest1338-21sh11:47:061
365999118,2cyclictest12103-21wc08:35:001
365999118,2cyclictest11518-21cut10:55:171
365999118,2cyclictest11028-21ssh09:56:301
365999118,2cyclictest10412-21tail10:35:171
365999110,11cyclictest9388-21ssh09:35:561
365999110,11cyclictest0-21swapper/111:30:171
365899118,2cyclictest6567-21ssh12:29:290
365899118,2cyclictest30939-21grep09:30:130
365899118,2cyclictest27929-21tr11:05:160
365899118,2cyclictest25241-21ssh10:25:100
365899118,2cyclictest25157-21grep09:00:160
365899118,2cyclictest24142-21basename09:00:010
365899118,2cyclictest22926-21ssh11:40:260
365899118,2cyclictest19631-21ssh10:01:420
365899117,3cyclictest10578-21grep08:30:190
365899117,2cyclictest9088-21grep08:25:260
365899115,4cyclictest0-21swapper/010:10:160
365899110,10cyclictest13229-21ssh09:58:140
207232110,1sleep30-21swapper/308:50:243
367099108,1cyclictest225692sleep307:48:443
367099107,2cyclictest12723-21sed07:25:263
366399108,1cyclictest23492-21sed07:50:152
366399108,1cyclictest215542sleep208:53:262
366399108,1cyclictest18559-21expr07:40:152
366399108,1cyclictest14845-21expr08:40:162
366399107,2cyclictest17811-21grep08:45:172
366399106,3cyclictest16396-21cut12:35:162
365999108,1cyclictest8529-21chrt08:25:201
365999108,1cyclictest4027-21cat11:10:161
365999108,1cyclictest28584-21ssh11:25:111
365999108,1cyclictest20350-21tail08:50:181
365999108,1cyclictest201262sleep107:42:491
365999108,1cyclictest11437-21ssh10:16:051
365999107,2cyclictest32398-21chrt08:09:571
365999107,2cyclictest23321-21cat07:50:141
365999107,2cyclictest14073-21grep07:30:161
365899108,1cyclictest295962sleep008:02:110
365899108,1cyclictest21822-21date08:55:000
365899108,1cyclictest19502-21awk07:40:220
365899108,1cyclictest15016-21expr08:40:160
365899107,2cyclictest4399-21sed08:15:270
365899107,2cyclictest24106-21ntpq07:50:200
365899107,2cyclictest17200-21sensors_temp07:35:260
365899107,2cyclictest14175-21grep07:30:160
365899107,2cyclictest10468-21taskset07:21:000
36709996,2cyclictest18452-21grep07:40:153
36639997,1cyclictest3031-21expr08:15:152
36639996,2cyclictest7189-21cat07:15:192
36639996,2cyclictest7189-21cat07:15:182
36639996,2cyclictest27927-21cat08:00:142
36599996,2cyclictest17589-21taskset07:35:581
36589996,2cyclictest6361-21wc08:20:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional