You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-16 - 14:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2s.osadl.org (updated Wed Jul 16, 2025 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2621327069,0sleep30-21swapper/306:42:403
2596723430,0sleep20-21swapper/206:39:512
2625923324,0sleep10-21swapper/106:43:141
2622023221,0sleep00-21swapper/006:42:450
28415070,0irq/27-eth0-tx-10541-21munin-node08:09:460
28415070,0irq/27-eth0-tx-0-21swapper/008:34:350
4160,0ktimersoftd/031192-21ssh10:07:110
339960,0migration/36443-21sh11:09:513
339960,0migration/34744-21kthreadcore07:14:443
339960,0migration/331143-21sh10:26:473
339960,0migration/323982-21timerwakeupswit08:39:483
28415060,0irq/27-eth0-tx-1831-21munin-node07:14:490
28415060,0irq/27-eth0-tx-0-21swapper/212:01:332
28415060,0irq/27-eth0-tx-0-21swapper/211:49:022
28415060,0irq/27-eth0-tx-0-21swapper/211:40:062
28415060,0irq/27-eth0-tx-0-21swapper/211:35:192
28415060,0irq/27-eth0-tx-0-21swapper/211:30:512
28415060,0irq/27-eth0-tx-0-21swapper/211:11:292
28415060,0irq/27-eth0-tx-0-21swapper/210:58:022
28415060,0irq/27-eth0-tx-0-21swapper/210:52:482
28415060,0irq/27-eth0-tx-0-21swapper/210:43:022
28415060,0irq/27-eth0-tx-0-21swapper/210:36:092
28415060,0irq/27-eth0-tx-0-21swapper/210:18:562
28415060,0irq/27-eth0-tx-0-21swapper/209:55:112
28415060,0irq/27-eth0-tx-0-21swapper/209:40:052
28415060,0irq/27-eth0-tx-0-21swapper/209:34:312
28415060,0irq/27-eth0-tx-0-21swapper/209:34:312
28415060,0irq/27-eth0-tx-0-21swapper/209:28:082
28415060,0irq/27-eth0-tx-0-21swapper/209:13:452
28415060,0irq/27-eth0-tx-0-21swapper/008:24:440
28415060,0irq/27-eth0-tx-0-21swapper/006:57:450
28415060,0irq/27-eth0-tx-0-21swapper/006:44:420
28405060,0irq/26-eth0-rx-6840-21ssh10:31:333
28405060,0irq/26-eth0-rx-4213-21ssh09:29:443
28405060,0irq/26-eth0-rx-25504-21if_err_eth1.211:24:393
264349960,0cyclictest0-21swapper/111:29:361
264349960,0cyclictest0-21swapper/110:19:461
264349960,0cyclictest0-21swapper/109:59:261
264339960,0cyclictest0-21swapper/012:09:560
264339960,0cyclictest0-21swapper/011:05:080
249960,0watchdog/2441-21munin-node12:09:382
249960,0watchdog/20-21swapper/211:57:422
249960,0watchdog/20-21swapper/211:04:422
249960,0watchdog/20-21swapper/209:59:462
249960,0watchdog/20-21swapper/207:59:432
149960,0migration/032261-21latency_hist09:49:240
149960,0migration/031288-21sh12:05:430
149960,0migration/011662-21kthreadcore09:14:420
8225050,0irq/24-0000:00:16223-21crond11:19:232
4150,0ktimersoftd/00-21swapper/008:39:470
339950,0migration/36730-21switchtime06:49:483
339950,0migration/3378-21latency_hist08:49:243
339950,0migration/33384-21taskset08:01:553
339950,0migration/332560-21diskmemload10:49:373
339950,0migration/330931-21taskset09:46:043
339950,0migration/329662-21kthreadcore10:44:443
339950,0migration/322832-21taskset11:39:593
339950,0migration/321916-21taskset07:29:243
339950,0migration/321427-21kthreadcore08:14:413
339950,0migration/319697-21kthreadcore09:59:413
339950,0migration/319418-21kthreadcore08:59:443
339950,0migration/31907-21kthreadcore11:29:403
339950,0migration/318317-21kthreadcore09:19:423
339950,0migration/318317-21kthreadcore09:19:423
339950,0migration/317964-21kthreadcore10:59:413
339950,0migration/317856-21taskset08:38:333
339950,0migration/317734-21kernelversion09:39:433
339950,0migration/317451-21kthreadcore06:59:433
339950,0migration/317188-21fschecks_time11:19:403
339950,0migration/31706-21kthreadcore08:24:453
339950,0migration/315557-21cpuspeed_turbos07:24:373
339950,0migration/314935-21kthreadcore08:09:433
339950,0migration/310365-21hwlatdetect07:44:393
28415050,0irq/27-eth0-tx-6680-21ssh11:49:522
28415050,0irq/27-eth0-tx-6372-21iceccd07:03:520
28415050,0irq/27-eth0-tx-6372-21iceccd06:49:250
28415050,0irq/27-eth0-tx-30352-21rt-features11:24:462
28415050,0irq/27-eth0-tx-29552-21munin-node07:59:410
28415050,0irq/27-eth0-tx-25126-21cpuspeed_turbos08:44:380
28415050,0irq/27-eth0-tx-1577-21iostat08:49:382
28415050,0irq/27-eth0-tx-0-21swapper/212:07:132
28415050,0irq/27-eth0-tx-0-21swapper/211:15:172
28415050,0irq/27-eth0-tx-0-21swapper/210:59:312
28415050,0irq/27-eth0-tx-0-21swapper/210:29:252
28415050,0irq/27-eth0-tx-0-21swapper/210:21:012
28415050,0irq/27-eth0-tx-0-21swapper/210:12:502
28415050,0irq/27-eth0-tx-0-21swapper/008:22:370
28415050,0irq/27-eth0-tx-0-21swapper/007:49:380
28415050,0irq/27-eth0-tx-0-21swapper/007:19:480
28405050,0irq/26-eth0-rx-8236-21fschecks_time06:54:383
28405050,0irq/26-eth0-rx-14172-21sendmail_mailtr11:14:483
28405050,0irq/26-eth0-rx-12587-21kthreadcore09:14:423
28405050,0irq/26-eth0-rx-12029-21kthreadcore09:34:413
28405050,0irq/26-eth0-rx-12029-21kthreadcore09:34:413
28405050,0irq/26-eth0-rx-0-21swapper/312:09:433
28405050,0irq/26-eth0-rx-0-21swapper/312:09:083
28405050,0irq/26-eth0-rx-0-21swapper/311:59:563
28405050,0irq/26-eth0-rx-0-21swapper/311:54:413
28405050,0irq/26-eth0-rx-0-21swapper/311:34:493
28405050,0irq/26-eth0-rx-0-21swapper/311:05:273
28405050,0irq/26-eth0-rx-0-21swapper/310:41:213
28405050,0irq/26-eth0-rx-0-21swapper/310:21:473
28405050,0irq/26-eth0-rx-0-21swapper/310:09:403
28405050,0irq/26-eth0-rx-0-21swapper/310:07:053
28405050,0irq/26-eth0-rx-0-21swapper/309:55:553
28405050,0irq/26-eth0-rx-0-21swapper/309:49:413
28405050,0irq/26-eth0-rx-0-21swapper/308:44:433
264359955,0cyclictest0-21swapper/208:24:272
264349955,0cyclictest0-21swapper/112:12:291
264349955,0cyclictest0-21swapper/112:06:581
264349955,0cyclictest0-21swapper/111:51:311
264349955,0cyclictest0-21swapper/111:27:151
264349955,0cyclictest0-21swapper/111:06:161
264349955,0cyclictest0-21swapper/110:12:591
264349955,0cyclictest0-21swapper/109:03:281
264349953,0cyclictest655-21kworker/1:210:27:411
264349950,0cyclictest0-21swapper/111:43:571
264349950,0cyclictest0-21swapper/110:59:111
264349950,0cyclictest0-21swapper/109:48:141
264339955,0cyclictest0-21swapper/011:58:160
264339954,0cyclictest16162-21idleruntime-cro09:39:240
264339954,0cyclictest16162-21idleruntime-cro09:39:240
264339950,0cyclictest0-21swapper/012:04:060
264339950,0cyclictest0-21swapper/010:40:330
259950,0migration/227407-21kthreadcore08:19:412
259950,0migration/227205-21kthreadcore10:44:432
259950,0migration/224890-21gltestperf10:04:372
249950,0watchdog/28631-21cut07:19:352
249950,0watchdog/272652sleep206:52:552
249950,0watchdog/223637-21sh07:54:392
249950,0watchdog/20-21swapper/210:25:142
249950,0watchdog/20-21swapper/209:49:542
249950,0watchdog/20-21swapper/209:46:342
249950,0watchdog/20-21swapper/209:30:022
249950,0watchdog/20-21swapper/209:24:182
249950,0watchdog/20-21swapper/209:24:182
249950,0watchdog/20-21swapper/209:16:302
249950,0watchdog/20-21swapper/209:04:582
249950,0watchdog/20-21swapper/208:59:382
249950,0watchdog/20-21swapper/208:54:462
249950,0watchdog/20-21swapper/208:44:342
249950,0watchdog/20-21swapper/208:09:392
249950,0watchdog/20-21swapper/208:08:232
179950,0migration/12645-21latency_hist07:39:241
169950,0watchdog/127725-21tr09:04:461
149950,0migration/08583-21cpuspeed_turbos10:54:360
149950,0migration/06624-21runrttasks11:10:180
149950,0migration/06492-21ksmtuned10:50:200
149950,0migration/05689-21rt-features10:09:480
149950,0migration/05291-21kthreadcore08:04:440
149950,0migration/029832-21kthreadcore07:09:430
149950,0migration/028834-21kthreadcore10:24:420
149950,0migration/025270-21fschecks_count09:24:370
149950,0migration/023367-21sh11:21:480
149950,0migration/022779-21latency_hist07:54:240
149950,0migration/022415-21unixbench_multi09:39:500
149950,0migration/021933-21ntp_kernel_pll_10:59:440
149950,0migration/017696-21munin-node09:19:410
149950,0migration/017696-21munin-node09:19:410
149950,0migration/017006-21idleruntime10:19:390
149950,0migration/015314-21taskset11:36:340
149950,0migration/013865-21kthreadcore08:54:460
149950,0migration/012426-21kthreadcore10:14:420
339940,0migration/34828-21taskset08:29:243
339940,0migration/332355-21kthreadcore07:09:443
339940,0migration/330054-21taskset08:22:163
339940,0migration/326671-21kthreadcore09:04:463
28415040,0irq/27-eth0-tx-0-21swapper/007:08:570
28405040,0irq/26-eth0-rx-6445-21snmpd10:55:263
28405040,0irq/26-eth0-rx-30516-21sendmail_mailst09:24:453
28405040,0irq/26-eth0-rx-0-21swapper/311:44:373
28405040,0irq/26-eth0-rx-0-21swapper/310:34:453
28405040,0irq/26-eth0-rx-0-21swapper/310:14:363
28405040,0irq/26-eth0-rx-0-21swapper/309:09:423
28405040,0irq/26-eth0-rx-0-21swapper/308:54:453
28405040,0irq/26-eth0-rx-0-21swapper/307:49:433
28405040,0irq/26-eth0-rx-0-21swapper/307:39:423
28405040,0irq/26-eth0-rx-0-21swapper/307:34:423
264359944,0cyclictest3173-21kthreadcore07:14:432
264349944,0cyclictest0-21swapper/110:06:121
264349943,0cyclictest9602-21awk07:19:391
264349943,0cyclictest720-21awk09:49:381
264349943,0cyclictest655-21kworker/1:211:35:051
264349943,0cyclictest655-21kworker/1:211:09:271
264349943,0cyclictest655-21kworker/1:210:31:111
264349943,0cyclictest6445-21snmpd07:10:551
264349943,0cyclictest6445-21snmpd06:45:401
264349943,0cyclictest495-21kworker/1:009:35:201
264349943,0cyclictest495-21kworker/1:009:35:201
264349943,0cyclictest495-21kworker/1:009:10:041
264349943,0cyclictest495-21kworker/1:008:52:061
264349943,0cyclictest495-21kworker/1:008:29:361
264349943,0cyclictest495-21kworker/1:007:14:521
264349943,0cyclictest3425-21lldpd10:59:371
264349943,0cyclictest3425-21lldpd09:39:331
264349943,0cyclictest3425-21lldpd08:25:181
264349943,0cyclictest3425-21lldpd08:04:461
264349943,0cyclictest3425-21lldpd07:28:451
264349943,0cyclictest25309-21awk10:44:411
264349943,0cyclictest24776-21awk08:19:391
264349943,0cyclictest19052-21awk08:39:401
264349943,0cyclictest18171-21awk08:14:391
264349943,0cyclictest17721-21awk09:19:411
264349943,0cyclictest17721-21awk09:19:411
264349943,0cyclictest17211-21awk07:49:421
264349943,0cyclictest15201-21awk06:59:411
264339944,0cyclictest0-21swapper/009:47:060
264339943,0cyclictest9390-21awk11:14:420
264339943,0cyclictest936-21awk10:29:400
264339943,0cyclictest8875-21awk09:54:410
264339943,0cyclictest6445-21snmpd11:27:120
264339943,0cyclictest5693-21awk08:29:390
264339943,0cyclictest408-21cut11:49:370
264339943,0cyclictest3425-21lldpd11:29:400
264339943,0cyclictest2855-21munin-node07:39:510
264339943,0cyclictest25831-21awk09:04:440
264339943,0cyclictest25604-21awk11:44:390
264339943,0cyclictest17499-21awk11:39:410
264339943,0cyclictest16126-21awk07:24:400
264339943,0cyclictest1531-21awk08:49:380
264339943,0cyclictest10533-21awk07:44:400
259940,0migration/231602-21kthreadcore07:09:442
259940,0migration/230338-21kthreadcore07:34:412
259940,0migration/220255-21kthreadcore08:39:422
259940,0migration/219373-21kthreadcore07:24:422
259940,0migration/218328-21irqcore08:14:402
259940,0migration/214334-21kthreadcore07:44:422
249940,0watchdog/20-21swapper/208:34:422
249940,0watchdog/20-21swapper/207:49:472
249940,0watchdog/20-21swapper/207:40:072
249940,0watchdog/20-21swapper/207:29:352
249940,0watchdog/20-21swapper/207:05:432
249940,0watchdog/20-21swapper/206:59:432
249940,0watchdog/20-21swapper/206:54:352
249940,0watchdog/20-21swapper/206:44:272
179940,0migration/19702-21kthreadcore06:54:421
179940,0migration/18318-21cpuspeed_turbos09:54:391
179940,0migration/18260-21latency_hist11:14:231
179940,0migration/16821-21sh09:30:371
179940,0migration/127105-21kthreadcore07:54:421
179940,0migration/125037-21cpuspeed_turbos11:44:371
179940,0migration/121971-21missed_timers11:19:451
179940,0migration/120008-21kthreadcore10:39:451
179940,0migration/116597-21latency_hist11:59:231
179940,0migration/113524-21kthreadcore08:34:421
179940,0migration/113309-21kthreadcore11:54:411
169940,0watchdog/18642-21ssh09:14:261
169940,0watchdog/18598-21python07:39:511
169940,0watchdog/132360-21grep07:59:431
169940,0watchdog/131639-21ssh09:26:581
169940,0watchdog/126064-21kthreadcore07:29:431
169940,0watchdog/112393-21cut08:09:431
169940,0watchdog/10-21swapper/110:34:421
149940,0migration/02867-21kthreadcore09:09:430
149940,0migration/022406-21netstat08:59:460
149940,0migration/022325-21kthreadcore08:14:420
149940,0migration/018178-21kthreadcore09:59:410
28405030,0irq/26-eth0-rx-0-21swapper/311:49:283
28405030,0irq/26-eth0-rx-0-21swapper/308:04:383
28405030,0irq/26-eth0-rx-0-21swapper/307:54:573
28405030,0irq/26-eth0-rx-0-21swapper/307:19:353
28405030,0irq/26-eth0-rx-0-21swapper/307:04:363
28405030,0irq/26-eth0-rx-0-21swapper/306:47:573
264349933,0cyclictest655-21kworker/1:210:49:271
264349933,0cyclictest655-21kworker/1:210:14:271
264349933,0cyclictest495-21kworker/1:008:54:321
264349933,0cyclictest495-21kworker/1:008:44:281
264349933,0cyclictest495-21kworker/1:007:44:261
264349933,0cyclictest3425-21lldpd07:05:151
264339932,0cyclictest28571-21munin-node07:34:400
264339932,0cyclictest1083-21gltestperf09:29:410
264339932,0cyclictest0-21swapper/007:29:450
264339931,0cyclictest8515-21cut10:34:370
264339931,0cyclictest0-21swapper/010:44:430
249930,0watchdog/20-21swapper/208:30:582
169930,0watchdog/10-21swapper/106:49:431
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional