You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-07 - 16:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu May 07, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6276210164,33sleep00-21swapper/007:07:500
644429873,20sleep10-21swapper/107:10:001
613329166,20sleep20-21swapper/207:05:592
621928471,9sleep30-21swapper/307:07:053
47972650,0sleep10-21swapper/109:31:481
652099390,38cyclictest0-21swapper/009:20:110
652099291,19cyclictest32515-21df11:05:150
653199270,27cyclictest0-21swapper/209:12:012
6527992726,1cyclictest0-21swapper/110:06:141
652099270,20cyclictest0-21swapper/009:28:520
652799260,25cyclictest0-21swapper/110:20:541
652099250,1cyclictest0-21swapper/011:16:110
652099240,23cyclictest0-21swapper/012:00:500
653199231,20cyclictest475-21in:imuxsock07:15:102
653199230,17cyclictest0-21swapper/207:35:132
652099230,2cyclictest0-21swapper/011:02:490
652099230,0cyclictest0-21swapper/012:32:100
6527992221,1cyclictest0-21swapper/109:40:281
652099220,17cyclictest0-21swapper/010:59:540
212822210,0sleep10-21swapper/110:58:471
6534992019,1cyclictest0-21swapper/310:50:343
6531992018,2cyclictest55750irq/128-enp2s0-10:36:592
6531992017,2cyclictest55750irq/128-enp2s0-10:43:542
653199200,20cyclictest0-21swapper/211:48:262
653199200,1cyclictest0-21swapper/209:59:522
652799203,15cyclictest31439-21kworker/1:211:10:121
652799202,16cyclictest0-21swapper/109:45:241
6531991917,2cyclictest55850irq/129-enp2s0-09:23:162
652799191,18cyclictest31583-21diskmemload11:05:481
652799190,0cyclictest0-21swapper/110:49:261
6520991911,5cyclictest225-21systemd-journal10:43:150
652099190,18cyclictest0-21swapper/012:38:240
6534991817,1cyclictest55850irq/129-enp2s0-10:01:293
653199181,16cyclictest0-21swapper/208:40:152
652099183,8cyclictest11919-21perf10:15:000
6534991716,1cyclictest10753-21smartctl11:30:243
6534991715,2cyclictest0-21swapper/309:58:413
653499170,17cyclictest0-21swapper/310:25:163
6531991715,2cyclictest55550irq/126-enp2s0-11:15:532
6531991714,2cyclictest55750irq/128-enp2s0-09:16:552
6527991716,1cyclictest0-21swapper/112:35:401
6527991714,2cyclictest25-21ksoftirqd/112:03:231
652799170,17cyclictest0-21swapper/110:34:431
652799170,16cyclictest0-21swapper/111:37:271
652099172,1cyclictest0-21swapper/009:37:400
652099170,17cyclictest0-21swapper/010:29:460
652099170,0cyclictest0-21swapper/011:51:350
6531991615,1cyclictest392-21hddtemp_smartct11:05:172
6531991614,1cyclictest55650irq/127-enp2s0-12:36:242
6531991614,1cyclictest55650irq/127-enp2s0-10:11:342
653199160,16cyclictest0-21swapper/210:50:132
652799160,1cyclictest0-21swapper/109:58:471
652799160,13cyclictest0-21swapper/111:18:211
652799160,12cyclictest0-21swapper/112:25:031
652099163,1cyclictest141rcu_preempt10:24:410
652099160,2cyclictest0-21swapper/012:15:410
653499152,9cyclictest29567-21aten2_r7power_p09:05:163
6534991513,2cyclictest0-21swapper/311:45:513
653499150,15cyclictest0-21swapper/309:50:043
653199156,3cyclictest32-21ksoftirqd/210:15:072
653199156,3cyclictest225-21systemd-journal08:45:002
6531991514,1cyclictest55750irq/128-enp2s0-09:44:192
653199150,4cyclictest0-21swapper/211:32:152
653199150,1cyclictest0-21swapper/211:29:342
653199150,15cyclictest0-21swapper/210:46:352
653199150,15cyclictest0-21swapper/209:46:212
653199150,15cyclictest0-21swapper/209:35:422
652799157,5cyclictest11321-21munin-run07:19:591
6527991513,1cyclictest55850irq/129-enp2s0-11:33:251
6527991510,4cyclictest38150irq/131-snd_hda09:15:311
652799150,14cyclictest0-21swapper/109:05:221
652099152,7cyclictest22058-21munin-run08:50:000
6520991515,0cyclictest0-21swapper/010:09:300
652099150,1cyclictest0-21swapper/010:39:000
652099150,0cyclictest0-21swapper/009:11:100
101382150,0chrt0-21swapper/010:51:370
653499148,5cyclictest30159-21apt-config10:45:023
6534991413,1cyclictest0-21swapper/312:13:153
6534991411,2cyclictest1277-21chrt11:25:143
653499140,11cyclictest0-21swapper/309:32:003
653199142,11cyclictest17753-21ssh10:56:012
6531991411,2cyclictest25356-21cron11:40:002
6531991410,2cyclictest32-21ksoftirqd/211:11:082
653199140,12cyclictest0-21swapper/210:25:142
652799149,4cyclictest25961-21tail11:20:211
652799140,3cyclictest0-21swapper/112:34:511
652799140,1cyclictest0-21swapper/111:50:131
652799140,13cyclictest0-21swapper/110:15:131
652099140,13cyclictest0-21swapper/012:23:300
6534991311,1cyclictest15619-21df10:55:173
6534991310,2cyclictest8469-21sed07:10:253
6534991310,2cyclictest26715-21grep09:45:133
6534991310,2cyclictest23083-21sed08:50:163
6534991310,2cyclictest14692-21grep07:25:163
653499130,0cyclictest0-21swapper/309:18:383
653199133,2cyclictest31583-21diskmemload12:20:192
6531991311,1cyclictest55850irq/129-enp2s0-12:02:332
6531991310,2cyclictest5741-21awk12:25:202
6531991310,2cyclictest2529-21grep08:05:272
653199130,1cyclictest0-21swapper/211:03:032
652799139,3cyclictest12089-21wc10:15:001
652799137,1cyclictest30614-21munin-run08:00:001
6527991310,2cyclictest4131-21cut12:05:171
6527991310,2cyclictest16820-21tail08:35:251
6527991310,2cyclictest16820-21tail08:35:251
6527991310,2cyclictest12538-21turbostat08:30:001
652799130,10cyclictest38150irq/131-snd_hda08:10:131
652799130,0cyclictest0-21swapper/108:00:121
652099139,2cyclictest22559-21grep08:50:140
6520991312,1cyclictest0-21swapper/012:11:570
6520991312,1cyclictest0-21swapper/011:57:490
6520991310,2cyclictest26668-21tr09:45:130
6520991310,2cyclictest13647-21strings10:15:190
652099130,3cyclictest0-21swapper/007:20:140
652099130,13cyclictest0-21swapper/011:22:300
652099130,12cyclictest0-21swapper/010:30:500
652099130,0cyclictest0-21swapper/011:48:240
653499129,2cyclictest9848-21grep07:15:153
653499129,2cyclictest6127-21seq12:06:063
653499129,2cyclictest31126-21tr08:00:133
653499129,2cyclictest30194-21cat12:20:243
653499129,2cyclictest27045-21sed09:25:253
653499129,2cyclictest24146-21expr07:45:133
653499129,2cyclictest21593-21tr12:35:153
653499129,2cyclictest19608-21tail11:55:193
653499129,2cyclictest19472-21sed11:35:273
653499129,2cyclictest19264-21grep08:40:263
653499129,2cyclictest13704-21ls07:24:593
653499129,2cyclictest10704-21grep08:25:143
653499124,6cyclictest31583-21diskmemload12:35:003
653499124,4cyclictest10296-21apache_accesses11:50:113
653499121,4cyclictest6058-21hwlatdetect10:30:173
6534991210,2cyclictest39-21ksoftirqd/311:02:423
653199129,2cyclictest31660-21sed08:00:162
653199129,2cyclictest30218-21sh11:42:142
653199129,2cyclictest28753-21tr07:55:152
653199129,2cyclictest28585-21turbostat.cron10:24:552
653199129,2cyclictest25960-21grep11:20:212
653199129,2cyclictest23053-21head08:50:162
653199129,2cyclictest21916-21grep12:15:242
653199129,2cyclictest12312-21head07:20:182
653199128,3cyclictest28199-21tr10:05:152
6531991211,1cyclictest0-21swapper/208:10:432
6531991210,1cyclictest55650irq/127-enp2s0-12:30:472
653199120,11cyclictest8925-21cron07:15:002
652799129,2cyclictest7989-21wc10:50:211
652799129,2cyclictest29310-21cut12:20:151
652799129,2cyclictest20672-21tail08:45:151
652799129,2cyclictest19676-21tr10:00:131
652799129,2cyclictest1622-21cut08:05:181
652799129,2cyclictest14227-21grep07:25:141
652799129,2cyclictest12516-21grep12:10:181
652799128,3cyclictest9830-21awk11:50:011
652799128,3cyclictest21942-21ssh11:56:241
652799127,4cyclictest9425-21tr08:20:211
652799127,4cyclictest2626-21sh09:50:141
652799127,4cyclictest25281-21ssh10:41:241
652799120,4cyclictest0-21swapper/111:25:031
652799120,0cyclictest0-21swapper/110:28:521
652099129,2cyclictest32537-21grep10:45:240
652099129,2cyclictest18757-21cut11:35:200
652099129,2cyclictest1364-21sort11:25:160
652099129,2cyclictest11323-21sleep07:19:590
652099128,3cyclictest30693-21grep09:05:230
652099128,3cyclictest26365-21tail08:55:280
652099128,3cyclictest18764-21sed08:40:210
652099128,3cyclictest10513-21grep09:55:000
652099124,4cyclictest225-21systemd-journal07:55:000
653499119,1cyclictest8024-21aten2_r7power_v11:10:133
653499119,1cyclictest26356-21grep07:50:133
653499119,1cyclictest25238-21grep08:55:153
653499119,1cyclictest20353-21lockfile-remove10:20:003
653499119,1cyclictest18791-21head09:40:163
653499119,1cyclictest0-21swapper/308:30:123
653499118,2cyclictest9175-21tr08:20:193
653499118,2cyclictest5621-21ssh10:10:253
653499118,2cyclictest27823-21grep11:40:253
653499118,2cyclictest27341-21ssh12:00:123
653499118,2cyclictest2289-21cat08:05:273
653499118,2cyclictest20668-21ssh10:39:213
653499118,2cyclictest16930-21grep09:20:133
653499118,2cyclictest15146-21basename08:35:123
653499118,2cyclictest15146-21basename08:35:113
653499118,2cyclictest12385-21cron12:30:003
653499114,5cyclictest455-21dbus-daemon07:40:003
653499114,3cyclictest20733-21runrttasks08:45:153
653499112,3cyclictest0-21swapper/309:00:123
653499110,11cyclictest0-21swapper/312:18:213
653499110,11cyclictest0-21swapper/310:06:143
653199118,2cyclictest7570-21ssh10:30:442
653199118,2cyclictest5021-21ssh09:50:462
653199118,2cyclictest29713-21cut09:05:162
653199118,2cyclictest28573-21cat09:00:252
653199118,2cyclictest26568-21gzip09:25:212
653199118,2cyclictest24421-21cut07:45:152
653199118,2cyclictest21375-21wc07:40:152
653199118,2cyclictest20834-21proc_pri10:00:192
653199118,2cyclictest18487-21cron07:35:002
653199118,2cyclictest14445-21sed08:30:252
653199118,2cyclictest13008-21rm11:50:572
653199118,2cyclictest12427-21cut12:10:172
653199118,2cyclictest12319-21chrt08:27:212
653199117,3cyclictest32-21ksoftirqd/211:55:202
653199113,7cyclictest0-21swapper/212:05:192
6531991110,1cyclictest0-21swapper/209:31:422
652799118,2cyclictest8939-21sleep07:15:011
652799118,2cyclictest30502-21ssh11:42:271
652799118,2cyclictest28742-21sed09:00:251
652799118,2cyclictest25394-21tail07:45:251
652799118,2cyclictest24832-21ssh11:00:201
652799118,2cyclictest22807-21sed08:50:141
652799118,2cyclictest22615-21ssh12:15:511
652799118,2cyclictest19257-21grep07:35:151
652799118,2cyclictest18362-21wc09:20:251
652799118,2cyclictest15541-21tail10:35:251
652799118,2cyclictest14436-21tail08:30:261
652799118,2cyclictest12838-21head07:20:221
652799118,2cyclictest11738-21sh09:35:371
652799117,3cyclictest25440-21cut08:55:171
652799115,5cyclictest26608-21grep07:50:131
6527991111,0cyclictest0-21swapper/109:27:041
652799110,0cyclictest0-21swapper/109:13:351
652099118,2cyclictest7698-21cron08:20:000
652099118,2cyclictest7685-21chrt07:10:180
652099118,2cyclictest7059-21ssh12:25:530
652099118,2cyclictest4496-21tail12:05:200
652099118,2cyclictest3762-21df08:10:150
652099118,2cyclictest3245-21ssh09:30:350
652099118,2cyclictest32118-21awk08:00:200
652099118,2cyclictest27646-21cat07:50:240
652099118,2cyclictest2679-21taskset08:06:390
652099118,2cyclictest26729-21head11:40:160
652099118,2cyclictest25105-21awk07:45:200
652099118,2cyclictest23243-21ssh09:42:590
652099118,2cyclictest19755-21cat10:00:120
652099118,2cyclictest16576-21cut07:30:130
652099118,2cyclictest15218-21basename08:35:130
652099118,2cyclictest15218-21basename08:35:130
652099118,2cyclictest11792-21head09:55:160
652099118,2cyclictest10725-21ssh11:11:010
652099118,2cyclictest10049-21grep09:15:250
652099117,2cyclictest8125-21taskset08:20:140
653499108,1cyclictest9322-21sshd09:35:053
653499108,1cyclictest31948-21ssh09:10:083
653499108,1cyclictest31686-21ssh11:05:003
653499107,2cyclictest5806-21tr08:15:133
653499107,2cyclictest25029-21tr11:20:153
653499107,2cyclictest21296-21sh10:20:143
653499107,2cyclictest16073-21sh11:15:003
653499104,5cyclictest0-21swapper/307:30:253
653499101,7cyclictest28498-21apt-get07:55:153
653499100,9cyclictest21977-21ssh10:40:103
653199108,1cyclictest8630-21df08:20:152
653199108,1cyclictest6651-21wc08:15:182
653199108,1cyclictest282802sleep207:55:002
653199107,2cyclictest25896-21cut08:55:232
653199107,2cyclictest15771-21grep08:35:162
653199107,2cyclictest15771-21grep08:35:162
653199107,2cyclictest14333-21expr07:25:142
652799107,2cyclictest6136-21grep08:15:141
652799107,2cyclictest21989-21cat07:40:181
652799107,2cyclictest16311-21taskset07:30:071
652099108,1cyclictest13768-21cat08:30:180
652099108,1cyclictest11694-21chrt11:30:520
652099107,2cyclictest27852-21head09:00:160
652099107,2cyclictest23281-21basename07:45:000
652099107,2cyclictest19514-21grep07:35:190
652099107,2cyclictest10628-21grep08:25:140
652099102,6cyclictest516-21gdbus07:25:220
65349998,1cyclictest21143-21cat07:40:123
65349997,1cyclictest3580-21expr08:10:143
65279982,5cyclictest17848-21cpuspeed_turbos08:40:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional