You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 14:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Mar 30, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1069729846,46sleep10-21swapper/107:05:251
1111729768,9sleep20-21swapper/207:06:242
1120828977,8sleep00-21swapper/007:07:330
1120128067,9sleep30-21swapper/307:07:293
1148599270,26cyclictest0-21swapper/210:58:292
11481992622,2cyclictest0-21swapper/109:30:121
1148599231,4cyclictest0-21swapper/207:10:192
1148599230,20cyclictest0-21swapper/211:32:552
1147699234,8cyclictest0-21swapper/011:55:150
217712220,0chrt0-21swapper/109:20:131
1148799220,0cyclictest0-21swapper/310:34:223
1148599220,22cyclictest0-21swapper/209:14:302
1148599220,20cyclictest0-21swapper/210:24:082
11476992221,1cyclictest0-21swapper/012:19:050
1148799210,21cyclictest0-21swapper/309:12:363
1148799210,20cyclictest0-21swapper/310:44:493
1148199211,19cyclictest25-21ksoftirqd/111:09:551
11487992018,1cyclictest24827-21kworker/3:009:45:123
11485992019,1cyclictest0-21swapper/210:45:212
1148599200,2cyclictest0-21swapper/211:08:212
1148599191,18cyclictest0-21swapper/212:02:452
1148599190,19cyclictest0-21swapper/210:14:132
11481991917,1cyclictest25-21ksoftirqd/109:44:401
1148199190,1cyclictest4537-21diskmemload09:28:521
1148199190,18cyclictest0-21swapper/112:05:181
1147699196,6cyclictest7367-21apt-get12:24:590
1148799180,18cyclictest0-21swapper/309:28:333
11481991815,1cyclictest25-21ksoftirqd/111:15:161
1148199180,16cyclictest0-21swapper/111:58:461
1148199180,0cyclictest0-21swapper/109:49:511
1148799170,17cyclictest0-21swapper/312:29:473
11481991715,2cyclictest0-21swapper/110:15:271
1148199170,16cyclictest0-21swapper/109:16:341
1147699176,1cyclictest141rcu_preempt11:14:420
1148599162,1cyclictest0-21swapper/211:18:302
11485991616,0cyclictest0-21swapper/212:34:202
11485991616,0cyclictest0-21swapper/210:09:012
11485991610,5cyclictest32712-21cat12:40:002
1148599160,16cyclictest0-21swapper/211:23:002
1148199164,1cyclictest11467-21cyclictest11:26:501
1148199164,1cyclictest11467-21cyclictest11:26:491
11481991614,2cyclictest25-21ksoftirqd/111:20:131
11481991613,2cyclictest0-21swapper/111:50:171
1148199160,0cyclictest0-21swapper/110:55:521
1148799154,2cyclictest537-21snmpd10:48:213
1148799153,9cyclictest5837-21sensors11:25:233
1148799153,9cyclictest5837-21sensors11:25:233
1148799152,12cyclictest0-21swapper/311:55:023
1148799150,4cyclictest0-21swapper/312:39:173
1148799150,14cyclictest1021-21cat10:05:203
1148599153,5cyclictest26667-21tail10:40:162
11485991514,1cyclictest0-21swapper/210:15:322
11485991511,3cyclictest28395-21cut08:50:172
1148599150,1cyclictest0-21swapper/212:27:192
1148599150,13cyclictest0-21swapper/211:25:422
1148599150,13cyclictest0-21swapper/211:25:412
1148199152,13cyclictest241rcuc/112:38:401
11481991513,2cyclictest0-21swapper/110:15:001
11481991512,2cyclictest2406-21python10:25:251
1148799144,5cyclictest0-21swapper/312:20:233
11487991411,2cyclictest3350-21apache212:11:373
11487991410,3cyclictest31555-21grep12:00:173
11487991410,3cyclictest30697-21cron07:50:003
11487991410,3cyclictest27703-21cat11:20:013
11487991410,3cyclictest16821-21sort10:15:163
11487991410,3cyclictest14794-21sed08:20:273
11487991410,3cyclictest13816-21cron07:15:003
11487991410,2cyclictest23540-21sed09:40:163
11485991411,2cyclictest25588-21wc10:00:242
1148599140,14cyclictest0-21swapper/212:15:142
11481991411,2cyclictest4537-21diskmemload11:40:121
11481991411,2cyclictest22557-21cron08:40:011
11481991411,2cyclictest0-21swapper/110:33:091
1148199140,2cyclictest0-21swapper/109:53:401
1148199140,0cyclictest141rcu_preempt12:04:251
1147699141,7cyclictest29544-21munin-run09:24:590
11476991410,3cyclictest28827-21df11:20:150
1148799139,3cyclictest32664-21ls10:24:593
1148799135,7cyclictest3338-21apache209:35:113
11487991311,1cyclictest0-21swapper/311:41:033
11487991310,2cyclictest30184-21ssh12:19:053
11487991310,2cyclictest28329-21tr11:20:123
11487991310,2cyclictest17188-21grep08:25:243
11487991310,2cyclictest14076-21wc11:30:203
1148799130,0cyclictest0-21swapper/309:59:013
1148599132,6cyclictest0-21swapper/209:20:132
11485991310,2cyclictest9825-21grep08:10:272
11485991310,2cyclictest9825-21grep08:10:262
11485991310,2cyclictest12629-21grep08:20:002
1148599130,8cyclictest0-21swapper/209:50:142
1148599130,1cyclictest0-21swapper/209:55:182
1148599130,12cyclictest2440-21sort11:04:592
1148599130,0cyclictest0-21swapper/211:57:222
1148199139,3cyclictest14207-21grep08:20:201
1148199138,4cyclictest32670-21head07:50:271
1148199131,2cyclictest141rcu_preempt12:31:291
11481991311,1cyclictest815-21irqstats10:05:191
11481991311,1cyclictest0-21swapper/112:29:521
11481991310,2cyclictest8847-21grep08:10:181
11481991310,2cyclictest8847-21grep08:10:171
11481991310,2cyclictest1323-21grep12:20:251
11481991310,2cyclictest0-21swapper/112:20:001
11481991310,1cyclictest12530-21ssh09:14:441
1148199130,12cyclictest0-21swapper/109:36:281
1147699139,3cyclictest21299-21wc07:30:000
1147699139,3cyclictest16789-21ssh12:30:140
1147699139,2cyclictest30637-21grep08:55:160
1147699133,5cyclictest16613-21aten2_r7power_c10:15:140
11476991310,2cyclictest9551-21sort08:10:260
11476991310,2cyclictest9551-21sort08:10:250
11476991310,2cyclictest4972-21ssh11:05:270
11476991310,2cyclictest3597-21wc10:45:250
11476991310,2cyclictest23880-21cut10:00:120
11476991310,2cyclictest23091-21cat12:14:590
11476991310,2cyclictest2148-21grep07:55:220
11476991310,2cyclictest21208-21cron09:20:000
11476991310,2cyclictest16712-21head08:25:190
11476991310,2cyclictest14628-21sed11:50:200
11476991310,2cyclictest13283-21grep11:30:150
1148799129,2cyclictest8495-21tr10:10:143
1148799129,2cyclictest8175-21tr12:05:233
1148799129,2cyclictest21455-21ssh11:54:563
1148799129,2cyclictest10599-21cut10:50:163
1148799128,3cyclictest23879-21grep10:00:133
1148799127,4cyclictest20401-21fgrep07:25:203
1148799127,3cyclictest22198-21ssh09:20:163
1148799126,5cyclictest537-21snmpd11:37:143
1148799123,5cyclictest225-21systemd-journal11:04:593
1148799120,9cyclictest0-21swapper/310:35:133
1148799120,11cyclictest3129-21grep09:05:163
1148599129,2cyclictest7049-21uname09:30:172
1148599129,2cyclictest5041-21chrt09:48:512
1148599129,2cyclictest30731-21grep11:40:222
1148599129,2cyclictest21607-21taskset11:35:162
1148599129,2cyclictest2024-21grep10:25:202
1148599129,2cyclictest14378-21wc08:20:222
1148599124,5cyclictest31998-21grep09:25:272
1148599124,4cyclictest225-21systemd-journal08:45:002
1148599122,5cyclictest12027-21threads10:50:272
1148199129,3cyclictest0-21swapper/111:33:301
1148199129,2cyclictest6402-21grep08:05:161
1148199129,2cyclictest23448-21grep08:40:161
1148199129,2cyclictest19518-21grep08:30:271
1148199129,2cyclictest1825-21ssh10:45:121
1148199129,2cyclictest16959-21grep07:20:161
1148199126,5cyclictest4537-21diskmemload10:52:041
11481991211,1cyclictest2916-21chrt07:58:391
1148199121,11cyclictest28322-21ssh11:39:381
1148199121,0cyclictest141rcu_preempt10:38:531
1148199120,1cyclictest141rcu_preempt10:22:371
1147699129,2cyclictest9848-21ssh12:25:270
1147699129,2cyclictest6495-21wc09:10:210
1147699129,2cyclictest30909-21cut09:25:170
1147699129,2cyclictest26094-21sed10:20:190
1147699129,2cyclictest259762sleep012:35:200
1147699129,2cyclictest25484-21cat10:40:000
1147699129,2cyclictest12810-21sort08:20:010
1147699129,2cyclictest10077-21sed10:10:270
1147699128,2cyclictest0-21swapper/010:59:000
1147699127,3cyclictest15135-21cat09:55:010
1147699125,4cyclictest4537-21diskmemload12:05:000
1147699125,3cyclictest4221-21apache_volume11:25:110
1147699125,3cyclictest4221-21apache_volume11:25:100
1147699123,5cyclictest31080-21ssh09:45:140
1147699120,12cyclictest0-21swapper/010:41:550
1148799118,2cyclictest9371-21ssh09:50:423
1148799118,2cyclictest5421-21grep11:45:173
1148799118,2cyclictest3828-21sed11:05:183
1148799118,2cyclictest32650-21grep09:00:143
1148799118,2cyclictest29562-21taskset08:52:473
1148799118,2cyclictest24551-21head07:35:173
1148799118,2cyclictest22682-21grep07:30:213
1148799118,2cyclictest22535-21ssh10:56:573
1148799118,2cyclictest19043-21ssh12:30:543
1148799118,2cyclictest15829-21chrt09:15:473
1148799118,2cyclictest1507-21cut10:25:173
1148799118,2cyclictest14510-21ssh11:11:213
1148799118,2cyclictest10570-21grep08:15:133
1148799118,2cyclictest1054-21wget07:55:143
1148799118,2cyclictest10443-21sshd09:32:203
1148799118,2cyclictest10099-21chrt08:13:243
1148799118,2cyclictest10099-21chrt08:13:243
1148799110,2cyclictest0-21swapper/307:20:143
1148799110,10cyclictest17864-21sshd08:30:133
1148599118,2cyclictest7600-21ssh11:45:392
1148599118,2cyclictest4342-21chrt09:07:432
1148599118,2cyclictest30711-21turbostat07:50:002
1148599118,2cyclictest28617-21ssh09:43:302
1148599118,2cyclictest2832-21ssh12:21:302
1148599118,2cyclictest26676-21cat07:40:142
1148599118,2cyclictest26503-21tr08:45:222
1148599118,2cyclictest24431-21sort07:35:172
1148599118,2cyclictest23557-21cron07:34:592
1148599118,2cyclictest2110-21munin-run09:05:002
1148599118,2cyclictest20256-21ssh09:38:302
1148599118,2cyclictest18494-21ssh09:17:522
1148599118,2cyclictest18246-21cut10:35:162
1148599118,2cyclictest17441-21ssh11:13:392
1148599118,2cyclictest16937-21tr07:20:162
1148599118,2cyclictest15960-21sed07:15:262
1148599118,2cyclictest15451-21grep12:10:152
1148599118,2cyclictest14040-21grep11:50:182
1148599118,2cyclictest13757-21chrt12:09:212
1148599118,2cyclictest1297-21expr07:55:152
1148599118,2cyclictest11528-21ssh10:30:422
1148599115,3cyclictest0-21swapper/208:55:222
1148199118,2cyclictest9855-21ssh11:47:241
1148199118,2cyclictest3297-21uname09:05:191
1148199118,2cyclictest31207-21rm11:02:101
1148199118,2cyclictest29134-21ssh10:02:591
1148199118,2cyclictest26260-21expr10:40:141
1148199118,2cyclictest25687-21expr08:45:141
1148199118,2cyclictest22424-21chrt07:30:181
1148199118,2cyclictest16953-21cat12:10:261
1148199118,2cyclictest16655-21uname09:55:201
1148199118,2cyclictest16165-21grep08:25:151
1148199118,2cyclictest15993-21sed07:15:261
1148199118,2cyclictest14256-21taskset11:11:101
1148199117,3cyclictest24063-21expr07:35:151
1148199115,2cyclictest225-21systemd-journal08:55:001
1147699118,2cyclictest9186-21ssh10:30:130
1147699118,2cyclictest3641-21grep08:00:140
1147699118,2cyclictest32342-21cut10:05:150
1147699118,2cyclictest29275-21grep07:45:160
1147699118,2cyclictest27982-21chrt07:40:350
1147699118,2cyclictest27450-21expr11:00:140
1147699118,2cyclictest25820-21taskset11:18:410
1147699118,2cyclictest25260-21apache_accesses08:45:110
1147699118,2cyclictest23628-21ssh11:35:560
1147699118,2cyclictest23552-21ssh09:40:160
1147699118,2cyclictest2266-21uname10:25:220
1147699118,2cyclictest22145-21cut07:30:160
1147699118,2cyclictest1698-21ssh12:01:300
1147699118,2cyclictest15596-21cut09:35:180
1147699118,2cyclictest15563-21cat09:55:130
1147699118,2cyclictest11965-21grep10:50:260
1147699118,2cyclictest11528-21ssh09:33:120
1147699117,2cyclictest11826-21grep07:10:200
1147699115,4cyclictest29900-21df_abs11:40:150
1147699113,4cyclictest225-21systemd-journal07:20:000
1147699110,10cyclictest5250-21ssh11:45:100
1147699110,10cyclictest28529-21cat08:50:190
1148799108,1cyclictest7294-21wc08:05:253
1148799108,1cyclictest31067-21basename07:50:153
1148799108,1cyclictest22359-21taskset08:37:313
1148799108,1cyclictest15689-21mailstats07:15:223
1148799107,2cyclictest3374-21apache_processe08:00:123
1148799107,2cyclictest31038-21wc08:55:193
1148799107,2cyclictest27204-21taskset08:47:593
1148799107,2cyclictest26627-21expr07:40:143
1148599108,1cyclictest5417-21chrt08:04:172
1148599108,1cyclictest17574-21chrt08:29:502
1148599107,2cyclictest6464-21head08:05:172
1148599107,2cyclictest22606-21basename08:40:012
1148599107,2cyclictest21197-21perf07:30:002
1148599100,7cyclictest0-21swapper/207:50:142
1148199108,1cyclictest30630-21tune2fs08:55:161
1148199108,1cyclictest28787-21egrep07:45:141
1148199108,1cyclictest12455-21tail07:10:231
1148199108,1cyclictest12281-21taskset08:15:491
1148199106,3cyclictest5494-21munin-run08:05:001
1148199100,9cyclictest26450-21cut07:40:151
1147699108,1cyclictest2977-21tune2fs09:05:160
1147699108,1cyclictest22931-21hostname08:40:130
1147699108,1cyclictest18903-21cat07:25:000
1147699108,1cyclictest18312-21expr08:30:150
1147699107,2cyclictest6152-21cat08:05:150
1147699107,2cyclictest1953-21sshd09:03:210
1147699107,2cyclictest13421-21cat08:20:140
114879996,2cyclictest22942-21basename08:40:123
114859997,1cyclictest19302-21sed08:30:262
114819997,1cyclictest19167-21grep07:25:121
114769997,1cyclictest31549-21df07:50:170
114769997,1cyclictest24661-21cat07:35:180
114769996,2cyclictest20676-21tr08:35:140
114819987,1cyclictest638-21tune2fs09:00:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional