You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-02 - 10:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jun 02, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
622729780,13sleep20-21swapper/219:06:322
633429380,9sleep00-21swapper/019:07:550
618628760,9sleep30-21swapper/319:06:003
636028463,6sleep10-21swapper/119:08:161
663599290,28cyclictest0-21swapper/022:17:330
664099250,15cyclictest0-21swapper/119:20:131
664699241,19cyclictest0-21swapper/323:25:133
6646992318,3cyclictest21418-21/usr/sbin/munin22:40:163
664099232,19cyclictest0-21swapper/121:05:201
664399220,20cyclictest0-21swapper/221:19:392
664399210,20cyclictest0-21swapper/223:25:512
664099210,21cyclictest0-21swapper/122:03:401
664099210,1cyclictest31582-21diskmemload22:18:441
6635992118,2cyclictest13-21ksoftirqd/022:00:130
664399200,19cyclictest0-21swapper/221:31:042
6646991917,2cyclictest0-21swapper/323:10:043
6646991917,2cyclictest0-21swapper/323:10:043
664399193,7cyclictest225-21systemd-journal20:14:252
664399190,19cyclictest0-21swapper/200:10:512
664099199,9cyclictest25-21ksoftirqd/123:46:111
6640991919,0cyclictest0-21swapper/121:36:531
664699181,2cyclictest0-21swapper/323:06:553
664699180,14cyclictest0-21swapper/322:19:263
6643991818,0cyclictest0-21swapper/222:41:182
664399180,16cyclictest0-21swapper/223:11:382
664399180,16cyclictest0-21swapper/223:11:382
6640991816,2cyclictest38150irq/131-snd_hda00:00:191
664399171,16cyclictest55550irq/126-enp2s0-22:27:102
664399170,16cyclictest0-21swapper/221:49:242
663599170,13cyclictest0-21swapper/023:54:510
664699160,16cyclictest0-21swapper/300:35:123
6643991615,1cyclictest0-21swapper/200:29:552
664099168,4cyclictest10626-21apache_volume00:10:141
664099166,8cyclictest38150irq/131-snd_hda20:50:151
664099164,11cyclictest1142-21runrttasks23:18:321
6640991616,0cyclictest0-21swapper/122:25:271
664099160,1cyclictest0-21swapper/121:27:491
663599162,2cyclictest225-21systemd-journal22:45:260
6635991611,4cyclictest12198-21wget00:30:140
6635991610,4cyclictest19922-21/usr/sbin/munin20:45:230
663599160,0cyclictest0-21swapper/023:24:510
6646991514,1cyclictest0-21swapper/323:24:083
6646991513,2cyclictest39-21ksoftirqd/323:15:153
664699151,1cyclictest0-21swapper/321:15:163
6646991511,3cyclictest1492-21sleep00:05:003
664699151,13cyclictest0-21swapper/322:03:423
664699150,15cyclictest0-21swapper/323:56:343
664399150,3cyclictest0-21swapper/223:51:182
664399150,15cyclictest0-21swapper/223:19:582
664399150,15cyclictest0-21swapper/221:13:392
664399150,15cyclictest0-21swapper/221:13:382
664399150,14cyclictest0-21swapper/222:32:402
664399150,13cyclictest0-21swapper/200:31:352
664099155,5cyclictest10406-21sh22:33:511
664099153,4cyclictest475-21in:imuxsock23:54:591
6640991512,2cyclictest16348-21ssh21:58:401
6640991511,3cyclictest25-21ksoftirqd/123:25:221
663599156,3cyclictest225-21systemd-journal19:25:250
663599154,5cyclictest28027-21turbostat19:55:000
663599151,7cyclictest5126-21munin-run20:15:010
663599151,2cyclictest0-21swapper/021:10:130
663599151,2cyclictest0-21swapper/021:10:130
663599150,8cyclictest18530-21perf00:15:000
663599150,1cyclictest0-21swapper/023:47:400
663599150,15cyclictest0-21swapper/000:24:550
6646991413,1cyclictest0-21swapper/322:28:033
6646991413,1cyclictest0-21swapper/322:23:013
6646991411,2cyclictest14777-21sed19:25:163
6646991410,3cyclictest15964-21ls23:35:003
664699140,1cyclictest0-21swapper/323:54:503
664699140,13cyclictest7601-21grep20:20:003
664399143,2cyclictest31582-21diskmemload00:10:002
6643991411,2cyclictest21665-21ssh22:01:162
664399140,2cyclictest0-21swapper/221:37:242
664399140,12cyclictest0-21swapper/221:44:202
664399140,12cyclictest0-21swapper/221:29:092
6640991412,1cyclictest0-21swapper/121:30:141
6640991411,2cyclictest0-21swapper/100:06:551
6640991411,1cyclictest0-21swapper/122:37:351
6640991410,3cyclictest10757-21wc22:15:001
664099140,14cyclictest0-21swapper/122:51:101
663599142,6cyclictest1296-21munin-run21:50:010
663599140,14cyclictest0-21swapper/022:38:210
663599140,13cyclictest0-21swapper/023:59:090
663599140,12cyclictest0-21swapper/021:24:310
6646991310,2cyclictest5109-21cron20:14:593
6646991310,2cyclictest29895-21grep19:55:263
6646991310,2cyclictest27732-21ssh22:05:163
6646991310,2cyclictest22051-21cron20:50:003
6646991310,2cyclictest21926-21grep19:40:173
6646991310,2cyclictest18513-21munin-run19:35:003
6646991310,2cyclictest17743-21sh23:35:223
6646991310,2cyclictest13225-21cut22:35:173
6646991310,2cyclictest13211-21cron22:55:003
664699130,10cyclictest0-21swapper/320:55:163
664399139,3cyclictest27610-21grep19:50:272
664399139,3cyclictest19264-21grep19:35:162
664399139,2cyclictest21393-21tr19:40:152
664399139,2cyclictest11753-21grep19:20:142
664399135,6cyclictest1-21systemd21:21:262
664399131,1cyclictest141rcu_preempt19:30:162
6643991311,1cyclictest32-21ksoftirqd/219:15:112
6643991311,1cyclictest32-21ksoftirqd/219:15:112
6643991310,2cyclictest3097-21sh00:25:002
6643991310,2cyclictest3087-21tr22:10:132
6643991310,2cyclictest28571-21tr19:55:142
6643991310,2cyclictest26074-21basename00:00:132
6643991310,2cyclictest22248-21wc00:35:262
664399130,12cyclictest0-21swapper/223:59:392
664099139,2cyclictest8981-21latency_hist19:15:001
6640991310,2cyclictest31582-21diskmemload00:18:091
6640991310,2cyclictest31389-21head20:00:171
6640991310,2cyclictest23869-21cut23:20:131
6640991310,2cyclictest18970-21sensors20:40:261
664099130,13cyclictest0-21swapper/122:09:571
664099130,12cyclictest0-21swapper/123:09:501
664099130,0cyclictest0-21swapper/100:30:421
663599139,3cyclictest460-21sh23:25:170
663599139,3cyclictest20189-21cat19:35:270
663599139,3cyclictest18044-21cron21:59:590
663599137,4cyclictest13-21ksoftirqd/021:25:190
663599135,6cyclictest10141-21if_err_enp2s019:15:180
663599135,6cyclictest10141-21if_err_enp2s019:15:170
663599135,4cyclictest225-21systemd-journal20:55:000
663599135,2cyclictest24385-21grep23:40:000
663599134,4cyclictest225-21systemd-journal21:05:000
663599131,6cyclictest13779-21awk19:25:000
6635991310,2cyclictest8400-21wc23:10:230
6635991310,2cyclictest8400-21wc23:10:230
6635991310,2cyclictest3610-21sed22:10:160
6635991310,2cyclictest18343-21grep20:40:200
6635991310,2cyclictest16536-21cat23:15:220
663599130,2cyclictest0-21swapper/000:35:130
663599130,12cyclictest5919-21grep22:30:270
664699129,2cyclictest3874-21grep00:05:253
664699129,2cyclictest25556-21cut23:40:163
664699129,2cyclictest23645-21wc20:50:213
664699129,2cyclictest18342-21head21:40:183
664699129,2cyclictest18069-21run-parts22:00:003
664699129,2cyclictest17080-21sh20:40:003
664699129,2cyclictest15504-21grep22:55:273
664699129,2cyclictest12339-21grep00:30:143
664699127,3cyclictest39-21ksoftirqd/323:02:143
664699125,6cyclictest23544-21apt-get19:45:153
664699120,2cyclictest0-21swapper/321:12:393
664699120,2cyclictest0-21swapper/321:12:393
664699120,1cyclictest0-21swapper/322:13:203
664399129,2cyclictest27013-21cat21:00:002
664399129,2cyclictest20199-21grep20:45:162
664399129,2cyclictest19107-21basename22:20:002
664399129,2cyclictest14713-21grep20:35:002
664399129,2cyclictest13798-21tr22:55:152
664399128,3cyclictest26396-21which23:40:222
664399128,2cyclictest850-21grep20:05:132
664399121,9cyclictest0-21swapper/219:10:132
6643991211,1cyclictest32-21ksoftirqd/222:21:092
664399120,8cyclictest0-21swapper/222:05:132
664399120,2cyclictest0-21swapper/221:58:482
664399120,10cyclictest0-21swapper/222:35:472
664099129,2cyclictest5932-21grep20:15:151
664099129,2cyclictest4440-21grep00:25:181
664099129,2cyclictest26205-21tr21:45:151
664099129,2cyclictest20786-21wc22:20:201
664099129,2cyclictest20576-21grep20:45:181
664099129,2cyclictest18398-21sort23:55:171
664099129,2cyclictest18062-21tail19:30:251
664099129,2cyclictest1334-21wc21:10:261
664099129,2cyclictest1334-21wc21:10:251
664099128,3cyclictest29718-21ssh22:45:161
664099125,5cyclictest21772-21dpkg23:00:001
664099124,4cyclictest225-21systemd-journal20:30:001
663599129,2cyclictest29957-21grep19:55:260
663599129,2cyclictest28863-21head22:25:180
663599129,2cyclictest2811-21sed00:05:160
663599129,2cyclictest1489-21basename00:05:010
663599127,2cyclictest452-21systemd-logind21:44:040
663599127,2cyclictest225-21systemd-journal22:42:040
6635991211,1cyclictest0-21swapper/021:05:160
663599120,11cyclictest0-21swapper/022:24:500
6892110,0sleep00-21swapper/021:30:100
664699118,2cyclictest9492-21tr19:15:133
664699118,2cyclictest9492-21tr19:15:133
664699118,2cyclictest6282-21ssh00:25:543
664699118,2cyclictest571-21sh21:30:073
664699118,2cyclictest5073-21taskset21:51:263
664699118,2cyclictest4454-21expr22:30:163
664699118,2cyclictest3490-21ssh22:48:593
664699118,2cyclictest28502-21ssh21:26:463
664699118,2cyclictest28282-21ssh00:20:163
664699118,2cyclictest2821-21grep23:45:283
664699118,2cyclictest26907-21ssh21:45:193
664699118,2cyclictest24441-21ssh00:18:063
664699118,2cyclictest19152-21cat19:35:153
664699118,2cyclictest1891-21wc20:05:203
664699118,2cyclictest17388-21rm20:40:003
664699118,2cyclictest13311-21head20:30:173
664699118,2cyclictest12421-21cut19:20:173
664699118,2cyclictest11363-21ssh21:35:353
664699118,2cyclictest10932-21tr00:10:163
664699116,2cyclictest225-21systemd-journal20:24:043
664699115,2cyclictest225-21systemd-journal19:55:003
664399119,1cyclictest31574-21sed23:05:182
664399118,2cyclictest9683-21ssh23:30:232
664399118,2cyclictest8497-21sort20:20:172
664399118,2cyclictest4752-21ssh21:51:112
664399118,2cyclictest4106-21ssh23:46:262
664399118,2cyclictest31451-21tail20:00:172
664399118,2cyclictest30707-21ntpq22:45:222
664399118,2cyclictest27408-21ssh23:21:442
664399118,2cyclictest25386-21chrt19:46:472
664399118,2cyclictest20515-21ssh00:15:232
664399118,2cyclictest18037-21systemctl23:35:232
664399118,2cyclictest11315-21sed20:25:172
664399117,3cyclictest6020-21tune2fs22:50:172
664399117,3cyclictest28795-21grep21:00:262
664399117,3cyclictest14254-21cut19:25:132
6643991110,1cyclictest0-21swapper/221:05:152
664399110,10cyclictest24937-21ssh23:00:542
664099118,2cyclictest8006-21tr23:10:201
664099118,2cyclictest8006-21tr23:10:201
664099118,2cyclictest6866-21sh21:52:471
664099118,2cyclictest30384-21cron19:59:591
664099118,2cyclictest29147-21ssh00:20:221
664099118,2cyclictest28640-21sort21:00:261
664099118,2cyclictest2665-21taskset20:09:101
664099118,2cyclictest23891-21ssh23:00:221
664099118,2cyclictest23278-21ssh21:43:231
664099118,2cyclictest20798-21ssh22:40:131
664099118,2cyclictest20164-21ssh00:35:011
664099118,2cyclictest19325-21sh23:36:061
664099118,2cyclictest16760-21cut21:20:141
664099117,3cyclictest9885-21cat21:15:291
664099117,1cyclictest22003-21if_enp0s31f619:40:181
664099110,8cyclictest13626-21kworker/1:219:25:121
664099110,11cyclictest0-21swapper/123:33:091
663599119,1cyclictest8136-21grep20:20:150
663599118,2cyclictest9948-21head21:35:170
663599118,2cyclictest8739-21ssh23:30:160
663599118,2cyclictest8634-21ssh22:51:250
663599118,2cyclictest8086-21ssh21:15:140
663599118,2cyclictest5354-21ssh23:09:240
663599118,2cyclictest398-21wc20:05:000
663599118,2cyclictest28441-21grep22:05:210
663599118,2cyclictest26808-21sendmail21:00:000
663599118,2cyclictest24313-21tail19:45:180
663599118,2cyclictest16945-21chrt20:38:250
663599118,2cyclictest1538-21ssh21:50:020
663599117,3cyclictest20032-21if_err_enp0s31f00:15:190
663599116,3cyclictest11033-21chrt20:25:150
663599115,3cyclictest225-21systemd-journal00:25:250
664699108,1cyclictest32123-21wc20:00:263
664699108,1cyclictest29757-21expr21:05:163
664699108,1cyclictest290892sleep321:03:413
664699108,1cyclictest12121-21taskset20:27:283
664699107,2cyclictest6840-21seq19:10:103
664699107,2cyclictest16858-21expr21:20:153
664399108,1cyclictest5678-21cat20:15:152
664399108,1cyclictest22349-21cut20:50:132
664399108,1cyclictest18411-21cat20:40:192
664399108,1cyclictest15160-21egrep20:35:152
664099108,1cyclictest27869-21chrt19:53:101
664099108,1cyclictest24671-21ssh23:40:041
664099108,1cyclictest20499-21uptime19:35:281
664099108,1cyclictest13956-21cat20:30:231
664099107,2cyclictest8085-21cut20:20:141
664099107,2cyclictest4807-21taskset20:11:091
664099107,2cyclictest25802-21grep20:55:221
664099107,2cyclictest16595-21sed20:35:271
664099107,2cyclictest11273-21taskset19:19:001
664099107,2cyclictest11273-21taskset19:18:591
664099100,9cyclictest24160-21expr19:45:181
663599108,1cyclictest16361-21chrt19:30:050
663599108,1cyclictest13732-21ssh22:55:120
663599108,1cyclictest12730-21cut20:30:160
663599107,2cyclictest7026-21ls20:15:260
663599107,2cyclictest26258-21cut23:40:220
663599107,2cyclictest23171-21tail23:00:160
663599107,2cyclictest21647-21chrt19:40:150
66359997,1cyclictest8200-21taskset19:10:220
66359997,1cyclictest26382sleep020:08:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional