You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 16:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Mar 23, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1844929878,16sleep00-21swapper/007:06:010
1841528661,9sleep20-21swapper/207:05:352
1857228168,9sleep30-21swapper/307:07:373
1873427857,17sleep10-21swapper/107:09:441
206602710,0sleep20-21swapper/211:30:452
18843992726,1cyclictest0-21swapper/007:20:110
18853992416,3cyclictest39-21ksoftirqd/312:05:143
1884899240,24cyclictest0-21swapper/210:19:242
1884899240,22cyclictest0-21swapper/210:32:322
1884899230,21cyclictest0-21swapper/209:14:072
1884899230,1cyclictest0-21swapper/211:38:142
1885399225,15cyclictest30221-21missed_timers09:40:183
1884899220,20cyclictest0-21swapper/210:59:592
1884899210,0cyclictest0-21swapper/210:47:002
18845992116,1cyclictest38150irq/131-snd_hda12:07:391
18843992118,2cyclictest537-21snmpd11:19:410
1885399200,20cyclictest0-21swapper/310:48:523
1884899200,20cyclictest0-21swapper/212:05:312
1884899200,19cyclictest0-21swapper/211:18:152
1885399196,10cyclictest20356-21ssh12:10:133
18853991916,2cyclictest22282-21sshd11:51:183
18848991919,0cyclictest0-21swapper/212:04:582
18848991916,2cyclictest24618-21apache211:45:132
1884899190,19cyclictest0-21swapper/210:14:032
18845991919,0cyclictest0-21swapper/108:47:581
18845991917,1cyclictest0-21swapper/111:05:281
18843991919,0cyclictest0-21swapper/011:42:130
1884399190,1cyclictest0-21swapper/012:29:350
1884899187,9cyclictest24615-21apache211:51:292
1884899180,1cyclictest0-21swapper/209:41:162
1884599180,1cyclictest11410-21diskmemload09:49:171
1884599180,1cyclictest11410-21diskmemload09:42:451
1884599180,18cyclictest0-21swapper/111:18:031
1884899170,17cyclictest0-21swapper/211:41:502
18845991715,1cyclictest28757-21apt-get12:35:011
1884399175,6cyclictest19795-21awk08:19:590
18843991717,0cyclictest0-21swapper/011:48:210
18843991714,2cyclictest24629-21apache209:15:140
1884399171,1cyclictest0-21swapper/012:08:190
1884399170,17cyclictest0-21swapper/011:31:160
18853991615,1cyclictest0-21swapper/311:26:383
1885399161,13cyclictest0-21swapper/312:15:133
18848991613,2cyclictest3832-21sh12:19:592
18845991616,0cyclictest0-21swapper/110:45:241
18845991610,2cyclictest225-21systemd-journal12:19:511
1884599160,1cyclictest11410-21diskmemload11:25:011
1884599160,16cyclictest0-21swapper/111:11:281
1885399153,2cyclictest141rcu_preempt09:55:443
18853991513,1cyclictest537-21snmpd11:15:033
18853991513,1cyclictest0-21swapper/311:03:583
1885399151,1cyclictest141rcu_preempt10:14:473
1885399150,2cyclictest0-21swapper/310:36:353
1884899150,14cyclictest0-21swapper/209:46:282
1884599159,4cyclictest11405-21cron09:10:011
1884599159,2cyclictest11410-21diskmemload09:50:261
1884599155,3cyclictest0-21swapper/108:15:111
1884599150,1cyclictest11410-21diskmemload10:05:481
1884399153,2cyclictest141rcu_preempt10:00:520
18843991514,1cyclictest0-21swapper/009:42:080
1884399150,13cyclictest0-21swapper/012:36:270
1885399149,3cyclictest5163-21tail12:20:163
1885399149,1cyclictest39-21ksoftirqd/309:19:543
18853991412,1cyclictest39-21ksoftirqd/311:21:253
18853991412,1cyclictest0-21swapper/311:09:273
1885399141,1cyclictest141rcu_preempt10:33:033
18853991411,2cyclictest537-21snmpd11:49:473
1885399140,1cyclictest0-21swapper/311:55:023
1885399140,1cyclictest0-21swapper/310:19:043
18848991411,2cyclictest30244-21cut12:35:182
18848991410,3cyclictest21019-21cut08:20:172
18848991410,2cyclictest17896-21grep08:15:132
1884899140,14cyclictest0-21swapper/210:37:242
1884899140,13cyclictest0-21swapper/209:15:532
1884899140,0cyclictest0-21swapper/211:22:512
1884599149,3cyclictest23561-21cron07:20:011
18845991413,1cyclictest11410-21diskmemload10:51:251
18845991410,3cyclictest29637-21cat08:40:011
1884599140,2cyclictest0-21swapper/110:35:161
1884599140,2cyclictest0-21swapper/109:23:341
1884599140,12cyclictest6998-21grep09:00:141
1884599140,0cyclictest0-21swapper/111:24:471
18843991411,2cyclictest23018-21turbostat10:35:010
18843991411,2cyclictest12209-21sh12:24:590
18843991410,3cyclictest16005-21tail08:10:160
18843991410,3cyclictest11514-21sort12:05:010
1884399140,1cyclictest0-21swapper/010:48:470
1884399140,13cyclictest0-21swapper/011:55:250
258742130,0sleep20-21swapper/212:32:452
1885399139,3cyclictest8063-21ls10:25:213
1885399139,3cyclictest29583-21tr08:39:593
1885399139,3cyclictest20396-21wc07:10:223
1885399139,3cyclictest18213-21sed10:50:263
1885399139,3cyclictest14027-21cat09:30:243
1885399139,2cyclictest11604-21wc09:10:003
1885399131,0cyclictest141rcu_preempt09:50:443
18853991310,2cyclictest31738-21grep07:35:193
18853991310,2cyclictest26469-21cat10:55:263
18853991310,2cyclictest24831-21cat08:29:593
18853991310,2cyclictest23165-21grep07:15:263
18853991310,2cyclictest14224-21cat10:10:003
1885399130,12cyclictest25193-21basename08:30:143
1884899139,3cyclictest9185-21cat07:55:192
1884899139,3cyclictest31936-21cron08:45:012
1884899139,3cyclictest26548-21wc08:30:252
1884899139,2cyclictest32100-21ls10:00:272
1884899139,1cyclictest141rcu_preempt12:29:362
1884899138,3cyclictest4443-21wc07:45:202
18848991312,1cyclictest0-21swapper/209:57:412
18848991310,2cyclictest32525-21wc10:20:262
18848991310,2cyclictest13676-21sed08:05:182
18848991310,2cyclictest12023-21cat09:30:002
1884899130,12cyclictest0-21swapper/211:13:542
1884899130,11cyclictest0-21swapper/209:32:052
1884599139,3cyclictest22756-21grep12:30:251
1884599139,3cyclictest12620-21cat11:45:251
1884599139,3cyclictest10323-21cron11:45:011
1884599133,2cyclictest11410-21diskmemload10:22:261
1884599132,10cyclictest14359-21vmstat09:30:251
1884599131,11cyclictest21984-21ntp_states09:35:201
18845991311,1cyclictest0-21swapper/109:25:151
18845991310,2cyclictest27952-21ls12:15:011
18845991310,2cyclictest25135-21grep08:30:131
1884399139,2cyclictest15189-21cat08:09:590
18843991313,0cyclictest0-21swapper/009:30:570
18843991310,2cyclictest30794-21cat07:35:000
18843991310,2cyclictest30310-21grep09:20:260
18843991310,2cyclictest22550-21sed09:35:250
18843991310,2cyclictest1978-21cron11:39:590
18843991310,2cyclictest10250-21munin-run07:59:590
1884399130,2cyclictest0-21swapper/009:55:160
1884399130,12cyclictest32702-21grep10:20:260
1884399130,10cyclictest7507-21diskstats09:00:150
1885399129,2cyclictest8267-21grep07:55:143
1885399129,2cyclictest5295-21tr09:45:153
1885399129,2cyclictest495-21head08:45:173
1885399129,2cyclictest3203-21grep11:40:163
1885399129,2cyclictest18671-21cat08:15:203
1885399129,2cyclictest12212-21cut09:10:143
1885399127,4cyclictest18909-21grep11:10:263
1885399126,2cyclictest225-21systemd-journal08:05:003
1885399121,5cyclictest0-21swapper/310:21:453
1885399120,11cyclictest7529-21cut09:00:173
1884899129,2cyclictest4464-21grep12:20:132
1884899129,2cyclictest24140-21munin-plugin-st10:55:002
1884899129,2cyclictest22775-21wc07:15:222
1884899129,2cyclictest21176-21cron07:15:002
1884899129,2cyclictest10633-21wc11:05:262
1884899129,2cyclictest10037-21head09:05:182
1884899128,3cyclictest8191-21sed10:05:282
1884899128,3cyclictest29539-21ssh11:55:242
1884899128,3cyclictest24939-21wc07:20:182
1884899128,3cyclictest1248-21tr07:40:142
1884899120,1cyclictest0-21swapper/210:43:342
1884599129,2cyclictest4312-21sort12:00:181
1884599129,2cyclictest3522-21grep07:45:141
1884599129,2cyclictest31059-21head08:40:211
1884599129,2cyclictest27528-21head11:35:181
1884599129,2cyclictest27275-21wc07:25:201
1884599129,2cyclictest24866-21cut10:55:131
1884599129,2cyclictest13609-21head08:05:181
1884599129,2cyclictest1343-21head10:40:261
1884599128,3cyclictest7362-21sh11:04:181
1884599128,3cyclictest30755-21cron07:35:001
1884599128,3cyclictest22066-21ssh11:31:511
1884599120,9cyclictest0-21swapper/109:58:391
1884599120,12cyclictest0-21swapper/109:12:011
1884399129,2cyclictest5798-21grep09:45:160
1884399129,2cyclictest5162-21cut08:55:180
1884399129,2cyclictest26539-21grep07:25:160
1884399129,2cyclictest25010-21tr10:55:150
1884399129,2cyclictest19974-21sort11:50:190
1884399129,2cyclictest1867-21tr11:20:140
1884399129,2cyclictest15301-21grep10:10:170
18843991212,0cyclictest0-21swapper/012:11:510
1884399120,0cyclictest0-21swapper/009:28:270
1885399119,2cyclictest26699-21kworker/u8:010:40:513
1885399118,2cyclictest6351-21taskset08:56:223
1885399118,2cyclictest5979-21tr07:50:143
1885399118,2cyclictest5148-21seq07:45:443
1885399118,2cyclictest39-21ksoftirqd/308:50:153
1885399118,2cyclictest39-21ksoftirqd/308:50:153
1885399118,2cyclictest32394-21ssh12:36:093
1885399118,2cyclictest30678-21grep10:00:153
1885399118,2cyclictest29430-21sed07:30:173
1885399118,2cyclictest28174-21ssh09:20:033
1885399118,2cyclictest26007-21ssh09:38:013
1885399118,2cyclictest21756-21cat08:20:253
1885399118,2cyclictest18270-21aten2_r7power_c11:30:113
1885399118,2cyclictest16376-21rm12:26:433
1885399118,2cyclictest15988-21head08:10:163
1885399118,2cyclictest1423-21sh11:39:353
1885399112,3cyclictest39-21ksoftirqd/309:25:173
1884899119,1cyclictest2734-21df_inode08:50:162
1884899119,1cyclictest2734-21df_inode08:50:152
1884899118,2cyclictest6682-21sleep09:00:002
1884899118,2cyclictest31819-21grep07:35:192
1884899118,2cyclictest28212-21ssh09:20:042
1884899118,2cyclictest24546-21taskset08:27:342
1884899118,2cyclictest20977-21ssh12:10:152
1884899118,2cyclictest20943-21tr09:35:142
1884899118,2cyclictest14572-21ssh10:29:532
1884899118,2cyclictest1455-21tail08:45:262
1884899118,2cyclictest14327-21fschecks_time09:50:172
1884899118,2cyclictest12361-21taskset08:01:422
1884899118,2cyclictest11236-21cat11:25:202
1884599118,2cyclictest6176-21tail08:55:271
1884599118,2cyclictest3744-21ssh10:03:261
1884599118,2cyclictest32525-21cat07:35:281
1884599118,2cyclictest24771-21ssh10:15:401
1884599118,2cyclictest21929-21grep09:15:271
1884599118,2cyclictest2126-21chrt08:50:081
1884599118,2cyclictest2126-21chrt08:50:071
1884599118,2cyclictest20811-21grep08:20:151
1884599118,2cyclictest19119-21ssh11:50:111
1884599118,2cyclictest15549-21tr10:30:131
1884599118,2cyclictest14352-21swap12:25:241
1884599118,2cyclictest12624-21cron08:05:001
1884599117,3cyclictest7016-21grep07:50:221
1884599117,3cyclictest5014-21sort12:20:151
1884599117,3cyclictest27722-21grep11:55:121
1884599113,7cyclictest23898-21aten2_r7power_e07:20:121
1884599110,1cyclictest0-21swapper/110:11:031
1884399119,1cyclictest779-21grep10:40:210
1884399119,1cyclictest21045-21grep08:20:170
1884399119,1cyclictest19782-21grep07:10:160
1884399118,2cyclictest4517-21ntpq07:45:200
1884399118,2cyclictest4161-21taskset08:53:440
1884399118,2cyclictest4161-21taskset08:53:430
1884399118,2cyclictest30663-21sh12:15:360
1884399118,2cyclictest30296-21cat08:40:140
1884399118,2cyclictest2833-21taskset07:41:290
1884399118,2cyclictest24484-21ssh10:15:290
1884399118,2cyclictest23984-21sh10:54:540
1884399118,2cyclictest23767-21tr08:25:200
1884399118,2cyclictest23554-21ssh10:35:140
1884399118,2cyclictest22844-21head12:30:250
1884399118,2cyclictest21678-21tr07:15:150
1884399118,2cyclictest17902-21tail11:10:160
1884399118,2cyclictest1715-21sort11:00:180
1884399118,2cyclictest15548-21head09:50:270
1884399118,2cyclictest14314-21ssh11:08:120
1884399118,2cyclictest11682-21taskset10:27:370
1884399118,2cyclictest11446-21wc11:25:220
1884399118,2cyclictest10513-21chrt10:07:120
1884399117,3cyclictest13817-21sed09:10:250
1884399114,4cyclictest722-21munin-plugin-st07:40:000
1885399108,1cyclictest25863-21taskset07:23:093
1885399108,1cyclictest23052-21ssh12:30:343
1885399108,1cyclictest13101-21grep08:05:163
1885399107,2cyclictest29819-21cat08:40:123
1885399107,2cyclictest27070-21grep07:25:183
1885399100,9cyclictest5794-21ssh12:00:433
1884899108,1cyclictest29453-21hddtemp_smartct07:30:172
1884899108,1cyclictest1567-21if_err_enp2s011:00:182
1884899107,2cyclictest8556-21sed09:00:262
1884899107,2cyclictest29061-21sed08:35:272
1884899107,2cyclictest26284-21apache_accesses07:25:132
1884899107,2cyclictest16596-21sendmail_mailst08:10:212
1884599108,1cyclictest7308-21cat10:25:161
1884599108,1cyclictest247602sleep108:29:361
1884599107,2cyclictest16105-21cat08:10:161
1884599106,2cyclictest19214-21cut07:10:131
1884599102,4cyclictest225-21systemd-journal07:59:591
1884599101,5cyclictest273-21systemd-udevd07:42:041
1884399108,1cyclictest269902sleep008:32:430
1884399107,2cyclictest877-21grep08:45:200
188539997,1cyclictest1447-21expr07:40:143
188489995,3cyclictest6977-21tr07:50:222
188439997,1cyclictest6041-21cut07:50:150
188439996,2cyclictest28702-21wc08:35:230
188439996,2cyclictest10943-21sed09:05:270
188439996,2cyclictest10601-21aten2_r7power_p08:00:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional