You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 21:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu May 21, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1415129684,8sleep00-21swapper/007:08:380
1392729364,9sleep30-21swapper/307:05:453
1422528875,9sleep20-21swapper/207:09:362
1391528665,6sleep10-21swapper/107:05:341
206302630,0sleep30-21swapper/310:53:523
262012480,0sleep20-21swapper/212:14:252
1434399335,27cyclictest0-21swapper/209:13:172
1433699270,27cyclictest0-21swapper/010:53:550
1433999260,26cyclictest0-21swapper/109:36:161
1434399250,0cyclictest0-21swapper/209:27:162
14339992524,1cyclictest0-21swapper/108:05:221
1434899241,0cyclictest0-21swapper/311:15:383
1434899231,1cyclictest55750irq/128-enp2s0-12:11:233
1433999230,22cyclictest0-21swapper/109:55:131
1433699230,21cyclictest0-21swapper/010:37:470
1434899222,15cyclictest1343-21ssh11:40:133
1434899220,1cyclictest55750irq/128-enp2s0-12:08:213
1434399211,4cyclictest0-21swapper/210:01:052
1434399211,4cyclictest0-21swapper/210:01:052
1434399210,2cyclictest0-21swapper/210:17:032
1433999210,20cyclictest0-21swapper/109:10:001
1434899204,5cyclictest141rcu_preempt12:00:003
1434399207,9cyclictest5717-21/usr/sbin/munin10:25:172
1434399200,17cyclictest0-21swapper/212:32:002
1433999200,20cyclictest0-21swapper/111:16:481
1434899190,1cyclictest0-21swapper/309:19:163
14343991917,1cyclictest0-21swapper/211:24:332
1434399190,19cyclictest0-21swapper/210:31:572
1434399190,18cyclictest0-21swapper/209:20:132
1434399190,0cyclictest0-21swapper/211:16:232
14339991917,2cyclictest25-21ksoftirqd/111:41:411
14336991918,1cyclictest0-21swapper/012:34:460
1434899180,1cyclictest0-21swapper/310:36:293
14339991815,2cyclictest0-21swapper/109:25:241
1433999180,4cyclictest0-21swapper/110:13:171
1433999180,18cyclictest0-21swapper/110:46:301
1433999180,16cyclictest0-21swapper/111:54:551
1433999180,16cyclictest0-21swapper/111:54:551
14336991818,0cyclictest0-21swapper/010:57:480
14336991816,1cyclictest13-21ksoftirqd/009:18:060
1434399177,10cyclictest0-21swapper/208:12:312
1434399174,9cyclictest28803-21sendmail07:40:002
1434399170,0cyclictest0-21swapper/211:34:032
14339991716,1cyclictest0-21swapper/111:28:571
1433999171,1cyclictest14645-21mailstats10:30:251
1433999170,1cyclictest0-21swapper/110:05:471
1433999170,1cyclictest0-21swapper/109:44:291
14336991716,0cyclictest13-21ksoftirqd/010:47:480
1433699170,0cyclictest0-21swapper/012:03:150
102502170,0sleep10-21swapper/112:24:101
1434399166,9cyclictest0-21swapper/208:45:272
1434399160,2cyclictest0-21swapper/212:02:422
1434399160,14cyclictest0-21swapper/212:08:382
1433999162,14cyclictest7146-21diskmemload09:51:591
1433999161,13cyclictest0-21swapper/109:22:091
1433999160,1cyclictest14329-21cyclictest12:39:371
1433699168,8cyclictest28914-21apache210:10:590
14336991614,1cyclictest13-21ksoftirqd/009:37:230
1433699161,1cyclictest141rcu_preempt09:52:030
1433699160,1cyclictest0-21swapper/012:20:380
1433699160,16cyclictest0-21swapper/010:34:550
1433699160,0cyclictest0-21swapper/010:29:010
1434899151,1cyclictest8575-21ssh10:45:523
1434899150,15cyclictest0-21swapper/310:33:153
1434899150,12cyclictest0-21swapper/311:47:303
14343991514,0cyclictest0-21swapper/209:56:062
1434399151,1cyclictest4721-21ssh10:05:212
1434399150,2cyclictest0-21swapper/210:39:442
14339991514,1cyclictest7146-21diskmemload10:23:111
14339991513,2cyclictest0-21swapper/111:09:101
1433999150,1cyclictest0-21swapper/112:05:121
1433999150,14cyclictest0-21swapper/110:52:141
1433999150,10cyclictest7146-21diskmemload12:25:211
1433699152,1cyclictest141rcu_preempt09:31:200
1433699151,8cyclictest26661-21sendmail-msp10:00:000
1433699151,0cyclictest141rcu_preempt10:00:300
1433699151,0cyclictest141rcu_preempt10:00:300
14348991411,2cyclictest30628-21wc07:40:253
14348991410,3cyclictest16262-21sed07:10:253
1434899140,2cyclictest0-21swapper/309:45:423
1434899140,14cyclictest0-21swapper/312:34:253
1434899140,13cyclictest0-21swapper/310:41:293
1434399143,10cyclictest9241-21kworker/u8:312:20:122
14343991413,1cyclictest0-21swapper/208:25:132
14343991411,2cyclictest3995-21turbostat12:40:002
14343991411,2cyclictest3767-21cut09:00:202
14343991410,3cyclictest18083-21cron08:25:002
1434399140,7cyclictest0-21swapper/209:05:132
1434399140,1cyclictest0-21swapper/211:50:452
1434399140,1cyclictest0-21swapper/211:50:452
1434399140,13cyclictest0-21swapper/208:00:122
14339991410,3cyclictest11457-21wc11:45:271
1433699148,2cyclictest13-21ksoftirqd/011:09:020
1433699147,6cyclictest9192-21munin-run11:45:000
1433699144,10cyclictest7902-21apt-get11:25:030
1433699141,1cyclictest141rcu_preempt10:41:580
14336991411,2cyclictest13-21ksoftirqd/012:10:130
1433699140,2cyclictest0-21swapper/011:54:180
1433699140,2cyclictest0-21swapper/011:54:170
1433699140,1cyclictest0-21swapper/010:05:500
1433699140,13cyclictest0-21swapper/012:39:580
1433699140,11cyclictest0-21swapper/009:20:010
1434899139,3cyclictest29610-21sed08:45:273
1434899139,3cyclictest12524-21grep12:25:163
1434899139,3cyclictest10399-21ssh11:25:263
1434899138,4cyclictest22184-21tr07:25:143
1434899134,5cyclictest225-21systemd-journal08:00:003
14348991310,2cyclictest3723-21grep10:05:163
14348991310,2cyclictest30479-21sh11:00:003
14348991310,2cyclictest29625-21grep10:20:183
14348991310,2cyclictest29309-21ls12:15:273
14348991310,2cyclictest20119-21grep09:55:203
14348991310,2cyclictest12403-21expr10:10:143
14348991310,2cyclictest10197-21cron12:05:003
1434899130,0cyclictest0-21swapper/311:14:223
1434399139,3cyclictest2710-21grep07:50:212
1434399137,5cyclictest4016-21apt-get07:55:032
14343991310,2cyclictest26558-21uname09:40:172
14343991310,2cyclictest2506-21ssh11:40:192
14343991310,2cyclictest25012-21sh09:40:012
14343991310,2cyclictest10992-21grep09:50:182
1434399130,3cyclictest0-21swapper/209:49:422
1434399130,1cyclictest0-21swapper/210:46:552
1434399130,12cyclictest398-21chrt11:00:242
1434399130,12cyclictest15192-21grep08:15:272
1434399130,12cyclictest15192-21grep08:15:262
1434399130,12cyclictest0-21swapper/210:24:352
1434399130,12cyclictest0-21swapper/208:35:252
1434399130,10cyclictest0-21swapper/210:43:412
1433999139,3cyclictest6297-21wc08:00:001
1433999139,2cyclictest7544-21grep08:00:191
1433999139,2cyclictest3104-21cat10:05:011
1433999139,2cyclictest3104-21cat10:05:001
14339991310,2cyclictest4059-21cat12:00:251
14339991310,2cyclictest31964-21grep08:50:271
14339991310,2cyclictest28525-21cut08:45:171
14339991310,2cyclictest23269-21cron11:15:001
14339991310,2cyclictest12678-21wc10:30:011
1433999130,4cyclictest0-21swapper/112:15:161
1433999130,12cyclictest0-21swapper/109:31:361
1433699139,3cyclictest15425-21apache_processe11:10:110
1433699137,5cyclictest1045-21sh09:45:000
14336991310,2cyclictest28524-21grep08:45:170
1433699130,12cyclictest31523-21ls08:50:210
1433699130,11cyclictest0-21swapper/008:10:140
1434899129,2cyclictest7980-21sed11:05:163
1434899129,2cyclictest7123-21grep08:00:163
1434899129,2cyclictest4795-21cat09:05:003
1434899129,2cyclictest32404-21cron08:54:593
1434899129,2cyclictest311-21wc07:45:213
1434899129,2cyclictest25637-21tr11:35:153
1434899129,2cyclictest18634-21tr08:25:143
1434899129,2cyclictest14264-21ssh09:52:053
1434899129,2cyclictest10050-21tr09:30:213
1434899129,2cyclictest0-21swapper/310:25:263
1434899128,3cyclictest537-21snmpd09:43:073
1434899120,1cyclictest0-21swapper/312:22:153
1434899120,1cyclictest0-21swapper/309:23:283
1434899120,11cyclictest25045-21head07:30:173
1434399129,2cyclictest9433-21tr11:25:172
1434399129,2cyclictest440-21tr08:55:132
1434399129,2cyclictest32402-21cron08:54:592
1434399129,2cyclictest29419-21tr07:40:142
1434399129,2cyclictest17731-21head07:15:182
1434399129,2cyclictest16230-21grep07:10:252
1434399129,2cyclictest15387-21cat10:50:192
1434399129,2cyclictest11752-21taskset12:25:102
1434399128,3cyclictest746-21chrt11:59:112
1434399121,8cyclictest16618-21apt09:35:002
1434399120,2cyclictest0-21swapper/207:30:192
1433999129,2cyclictest18919-21tail11:30:241
1433999129,2cyclictest17662-21tr07:15:171
1433999129,2cyclictest15291-21head07:10:161
1433999129,2cyclictest1156-21ls11:20:211
1433999127,4cyclictest28310-21ssh10:39:081
1433999127,2cyclictest24369-21cat10:55:251
1433999124,4cyclictest225-21systemd-journal08:45:001
1433999121,2cyclictest225-21systemd-journal09:09:551
14339991210,2cyclictest38150irq/131-snd_hda10:40:581
1433999120,12cyclictest0-21swapper/111:35:331
1433999120,0cyclictest0-21swapper/109:16:441
1433699129,3cyclictest13-21ksoftirqd/012:05:120
1433699129,3cyclictest13-21ksoftirqd/008:45:010
1433699129,2cyclictest9080-21tr08:05:140
1433699129,2cyclictest8380-21grep09:10:170
1433699129,2cyclictest7415-21cat08:00:190
1433699129,2cyclictest32121-21grep07:45:160
1433699129,2cyclictest23841-21grep08:35:160
1433699129,2cyclictest21629-21cut10:15:200
1433699129,2cyclictest17096-21cut11:30:120
1433699129,2cyclictest13026-21ssh12:25:190
1433699121,0cyclictest141rcu_preempt11:49:200
14336991210,1cyclictest5673-21grep07:55:270
14336991210,1cyclictest4269-21dump-pmu-power10:25:010
1433699120,11cyclictest26021-21head07:30:270
1433699120,11cyclictest0-21swapper/011:20:130
1433699120,0cyclictest0-21swapper/012:16:160
1434899119,1cyclictest310432sleep311:00:053
1434899119,1cyclictest3096-21grep07:50:263
1434899118,2cyclictest8137-21sed11:25:003
1434899118,2cyclictest7112-21taskset09:09:483
1434899118,2cyclictest4598-21sh09:27:083
1434899118,2cyclictest29356-21head12:35:163
1434899118,2cyclictest27652-21grep10:00:153
1434899118,2cyclictest27652-21grep10:00:153
1434899118,2cyclictest2281-21taskset08:59:023
1434899118,2cyclictest21501-21ssh09:37:173
1434899118,2cyclictest20965-21tr08:30:153
1434899118,2cyclictest19619-21ssh11:50:233
1434899118,2cyclictest19619-21ssh11:50:233
1434899118,2cyclictest17056-21wget11:30:123
1434899118,2cyclictest16216-21tr08:20:133
1434899118,2cyclictest12305-21grep08:10:193
1434899118,2cyclictest10453-21grep08:05:273
1434899117,3cyclictest8216-21ssh09:10:153
1434899117,3cyclictest26064-21taskset10:18:203
1434899110,10cyclictest27818-21ntp_kernel_pll_07:35:183
1434399118,2cyclictest28441-21cat12:15:192
1434399118,2cyclictest27837-21ssh10:57:582
1434399118,2cyclictest27783-21ssh11:35:492
1434399118,2cyclictest22029-21processes08:30:222
1434399118,2cyclictest16483-21uname11:10:182
1434399118,2cyclictest16426-21ssh11:49:092
1434399118,2cyclictest16006-21ssh09:15:112
1434399118,2cyclictest11693-21ssh10:10:012
1434399115,3cyclictest7160-21ssh11:05:122
1433999118,2cyclictest891-21cut08:55:151
1433999118,2cyclictest3699-21ssh09:45:311
1433999118,2cyclictest31345-21tr11:00:131
1433999118,2cyclictest30454-21grep07:40:211
1433999118,2cyclictest27754-21grep11:55:211
1433999118,2cyclictest27388-21cut07:35:141
1433999118,2cyclictest25962-21ssh12:33:241
1433999118,2cyclictest23209-21ssh10:16:061
1433999118,2cyclictest20467-21ssh12:10:221
1433999116,4cyclictest4669-21taskset09:03:531
1433699119,1cyclictest30771-21grep07:40:250
1433699119,1cyclictest22747-21hwlatdetect07:25:180
1433699118,2cyclictest3253-21grep09:00:150
1433699118,2cyclictest27292-21sort07:35:140
1433699118,2cyclictest27153-21head11:55:160
1433699118,2cyclictest25663-21grep11:15:260
1433699118,2cyclictest25444-21tr11:35:150
1433699118,2cyclictest2358-21cron08:59:590
1433699118,2cyclictest21692-21awk07:25:010
1433699118,2cyclictest19926-21cat08:25:260
1433699118,2cyclictest1673-21rm11:01:060
1433699118,2cyclictest14329-21cyclictest09:45:160
1433699113,7cyclictest13-21ksoftirqd/008:15:140
1433699113,7cyclictest13-21ksoftirqd/008:15:140
1434899108,1cyclictest25732-21expr08:40:133
1434899108,1cyclictest13940-21expr08:15:153
1434899108,1cyclictest13940-21expr08:15:153
1434899107,2cyclictest21252-21sed07:20:253
1434899107,2cyclictest16965-21dpkg07:15:133
1434399108,1cyclictest9039-21grep08:05:142
1434399108,1cyclictest26120-21expr08:40:152
1434399107,2cyclictest22419-21expr07:25:152
1434399100,6cyclictest1198-21sh07:50:002
1433999108,1cyclictest2871-21python07:50:251
1433999108,1cyclictest24644-21cat07:30:141
1433999108,1cyclictest19715-21which08:25:221
1433999108,1cyclictest17190-21cut08:20:201
1433999108,1cyclictest12232-21wc08:10:191
1433999107,2cyclictest852-21taskset07:45:441
1433999107,2cyclictest23760-21chrt07:26:251
1433999107,2cyclictest14544-21wc08:15:191
1433999107,2cyclictest14544-21wc08:15:191
1433999100,9cyclictest21887-21sh08:30:211
1433999100,1cyclictest0-21swapper/107:20:141
1433699108,1cyclictest5615-21entropy09:05:160
1433699108,1cyclictest15947-21taskset08:20:100
1433699107,2cyclictest765-21ssh09:25:140
1433699107,2cyclictest20857-21cut08:30:150
1433699107,2cyclictest17183-21tr07:15:150
1433699107,2cyclictest15664-21grep07:10:180
143489997,1cyclictest23347-21cut08:35:143
143439990,9cyclictest0-21swapper/207:20:132
143399997,1cyclictest23698-21forks08:35:161
143369997,1cyclictest1784-21cstates07:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional