You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-05 - 15:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack8slot2s.osadl.org (updated Thu Nov 05, 2026 12:43:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
312116399722,42cyclictest3199747-21kworker/u16:11+xprtiod10:25:090
3121164996217,24cyclictest3276435-21ls12:14:571
312116499611,35cyclictest3266978-21sendmail-msp12:01:011
312116499611,35cyclictest3266978-21sendmail-msp12:01:011
312116499611,2cyclictest0-21swapper/111:50:161
312116499597,49cyclictest0-21swapper/111:30:281
312116499581,54cyclictest3229427-21/usr/sbin/munin11:00:231
312116499581,2cyclictest0-21swapper/108:20:111
312116499572,50cyclictest3167337-21ssh09:21:241
3121164995722,25cyclictest3193054-21kworker/u16:12+xprtiod10:20:131
312116499562,49cyclictest3191075-21sendmail-msp10:00:471
312116499562,49cyclictest3191075-21sendmail-msp10:00:471
312116499562,32cyclictest0-21swapper/109:39:571
312116499561,51cyclictest3208084-21ssh10:25:491
312116499561,51cyclictest3185863-21sh09:50:391
312116499561,50cyclictest3201662-21ntpq10:15:371
312116499561,50cyclictest3201662-21ntpq10:15:371
312116499561,49cyclictest3241657-21sendmail-msp11:20:461
3121164995525,27cyclictest331ktimers/107:25:171
312116499552,50cyclictest3290405-21ntpq12:35:401
312116499552,45cyclictest3178747-21ssh09:40:161
312116499551,50cyclictest3255676-21ssh11:41:051
312116399553,26cyclictest3255536-21/usr/sbin/munin11:40:520
312116499543,31cyclictest3227041-21ntpq10:55:421
312116499542,49cyclictest3214421-21ntpq10:35:431
312116499542,49cyclictest3214421-21ntpq10:35:431
3121164995415,35cyclictest3239673-21ntpq11:15:431
312116499541,41cyclictest3157637-21kworker/u16:10+xprtiod09:10:091
3121164995410,39cyclictest0-21swapper/107:59:571
3121165995322,27cyclictest0-21swapper/207:55:212
3121165995317,20cyclictest3191216-21latency_hist09:59:582
312116499533,45cyclictest3223755-21ssh10:50:361
3121164995331,19cyclictest0-21swapper/112:30:081
312116499532,46cyclictest3220965-21/usr/sbin/munin10:45:551
312116499532,46cyclictest3220965-21/usr/sbin/munin10:45:551
312116499532,46cyclictest0-21swapper/108:45:081
312116499532,45cyclictest0-21swapper/107:30:501
312116499532,37cyclictest3277531-21ntpq12:15:381
312116499532,31cyclictest3131275-21fschecks_time07:40:311
312116499531,5cyclictest0-21swapper/109:33:101
312116499531,49cyclictest3265068-21sh11:55:461
312116499531,49cyclictest3245129-21aten_r8power_en11:25:241
312116499531,29cyclictest3200114-21systemd-userwor10:13:531
312116399533,24cyclictest3167159-21ssh09:21:150
3121165995228,19cyclictest41-21rcuc/211:25:082
312116499524,43cyclictest3142797-21ntpq08:15:411
312116499524,43cyclictest3142797-21ntpq08:15:411
312116499522,47cyclictest3159256-21apt-get09:09:561
312116499522,44cyclictest3164768-21ssh09:17:081
312116499522,44cyclictest3164768-21ssh09:17:081
312116499522,31cyclictest3195280-21rm10:05:371
312116499522,31cyclictest3195280-21rm10:05:371
312116499521,47cyclictest3254498-21latency_hist11:39:581
312116499521,47cyclictest3254498-21latency_hist11:39:581
312116499521,35cyclictest3124353-21sendmail-msp07:20:511
312116399522,39cyclictest3216653-21apt-get10:40:130
3121163995220,26cyclictest3143388-21sendmail-msp08:20:410
3121163995215,31cyclictest3203724-21sendmail-msp10:20:170
312116399521,40cyclictest3188468-21fschecks_count09:55:270
312116599512,24cyclictest3232899-21gltestperf11:05:282
312116599512,24cyclictest3232899-21gltestperf11:05:282
312116499512,45cyclictest3188841-21memory09:55:321
312116499512,44cyclictest3236476-21ntpq11:10:401
312116499512,35cyclictest3211204-21ssh10:30:431
312116499512,35cyclictest3211204-21ssh10:30:431
312116499511,46cyclictest3182934-21cat09:45:541
312116499511,46cyclictest3182934-21cat09:45:541
312116499511,45cyclictest3280746-21ssh12:20:431
312116499511,31cyclictest3216984-21ssh10:40:281
312116499502,43cyclictest3130711-21latency_hist07:39:571
3121164995023,24cyclictest3122721-21sshd-session07:13:131
3121164995023,24cyclictest3122721-21sshd-session07:13:131
312116499502,27cyclictest3283980-21ntpq12:25:491
312116499502,27cyclictest1163-21rs:main1
312116499501,45cyclictest3149669-21vmstat08:35:591
312116499501,45cyclictest3149669-21vmstat08:35:591
312116499501,44cyclictest3123640-21needreboot07:15:311
312116499501,2cyclictest0-21swapper/112:05:431
3121165994923,22cyclictest3237191-21kworker/u16:7+rpciod11:20:212
312116499492,44cyclictest3139370-21/usr/sbin/munin08:05:321
312116499492,44cyclictest0-21swapper/108:01:551
312116499492,44cyclictest0-21swapper/108:01:551
312116499492,40cyclictest3156204-21grep08:59:581
3121164994919,26cyclictest0-21swapper/108:10:131
312116499491,45cyclictest649-21CodeMeterLin09:00:491
312116499491,45cyclictest0-21swapper/109:25:351
312116499491,43cyclictest3147412-21missed_timers08:30:291
312116399492,23cyclictest3213415-21/usr/sbin/munin10:35:200
312116399492,23cyclictest3213415-21/usr/sbin/munin10:35:200
3121163994922,21cyclictest3285917-21apt-get12:30:080
3121165994821,21cyclictest3216388-21sendmail-msp10:42:182
312116499482,41cyclictest3232346-21/usr/sbin/munin11:05:461
312116499482,41cyclictest3232346-21/usr/sbin/munin11:05:461
3121164994821,24cyclictest331ktimers/111:45:191
312116499481,43cyclictest3133298-21ntpq07:45:411
312116399483,19cyclictest3238922-21aten_r8power_po11:15:230
312116399482,40cyclictest3280385-21/usr/sbin/munin12:20:330
3121163994817,21cyclictest3241882-21ssh11:20:010
3121165994723,17cyclictest3137041-21sendmail-msp08:01:042
3121165994723,17cyclictest3137041-21sendmail-msp08:01:042
312116599472,24cyclictest3273207-21ssh12:09:562
312116599472,23cyclictest3179453-21ssh09:40:442
3121165994721,6cyclictest3264950-21ntpq11:55:372
3121165994721,22cyclictest3236459-21ssh11:10:392
312116499471,39cyclictest3133852-21apt-get07:50:071
3121163994726,15cyclictest3243283-21kworker/u16:8+rpciod12:01:100
3121163994726,15cyclictest3243283-21kworker/u16:8+rpciod12:01:100
312116399471,38cyclictest3175553-21ssh09:35:140
312116599462,40cyclictest3181745-21ssh09:45:082
312116599462,40cyclictest3181745-21ssh09:45:082
3121164994618,22cyclictest0-21swapper/108:25:341
312116499461,41cyclictest3153926-21ntpq08:50:431
312116399463,37cyclictest3230930-21sshd-auth11:01:140
3121163994626,16cyclictest3277690-21ssh12:15:500
312116399462,22cyclictest3160674-21/usr/sbin/munin09:10:560
3121163994622,19cyclictest3163768-21ssh09:15:450
3121163994622,19cyclictest3163768-21ssh09:15:450
3121163994621,18cyclictest3173374-21ssh09:30:490
3121163994613,18cyclictest3266029-21kworker/u16:11+xprtiod12:35:080
312116399461,21cyclictest3130891-21/usr/sbin/munin07:41:020
3121166994525,15cyclictest0-21swapper/311:40:303
3121165994519,20cyclictest3130690-21sendmail-msp07:40:422
3121165994515,14cyclictest3204049-21ssh10:20:232
312116399452,22cyclictest3181847-21/usr/sbin/munin09:45:200
312116399452,22cyclictest3181847-21/usr/sbin/munin09:45:200
3121163994519,21cyclictest3227333-21sh10:55:560
3121163994518,20cyclictest3248175-21ssh11:30:120
3121163994516,23cyclictest3149729-21sendmail-msp08:42:220
312116399451,36cyclictest3147112-21fschecks_time08:30:240
3121163994512,28cyclictest3274477-21ntpq12:10:450
3121163994511,13cyclictest3264849-21needreboot11:55:310
3121166994428,12cyclictest0-21swapper/309:00:513
3121165994432,9cyclictest0-21swapper/210:00:372
3121165994432,9cyclictest0-21swapper/210:00:372
3121165994427,12cyclictest0-21swapper/209:10:082
312116599442,36cyclictest3163839-21ssh09:15:512
312116599442,36cyclictest3163839-21ssh09:15:512
3121165994419,21cyclictest0-21swapper/210:10:072
3121165994419,19cyclictest3261409-21ssh11:50:272
3121165994417,24cyclictest3200902-21aten_r8power_en10:15:202
3121165994417,24cyclictest3200902-21aten_r8power_en10:15:202
312116399444,32cyclictest3159421-21cat09:09:580
312116399442,18cyclictest3157034-21ntpq09:00:420
312116399441,36cyclictest3271320-21ssh12:05:490
312116399441,23cyclictest3257783-21ssh11:45:220
3121166994320,18cyclictest0-21swapper/311:00:303
3121166994318,20cyclictest0-21swapper/310:20:173
3121165994328,11cyclictest0-21swapper/209:30:542
3121165994327,13cyclictest0-21swapper/211:30:532
3121165994321,18cyclictest3239934-21ssh11:15:562
3121165994320,19cyclictest3290116-21ssh12:35:302
3121165994320,19cyclictest3251420-21ssh11:35:182
3121165994320,19cyclictest3251420-21ssh11:35:182
3121165994316,22cyclictest3266978-21sendmail-msp12:02:112
3121165994316,22cyclictest3266978-21sendmail-msp12:02:112
312116499431,35cyclictest3121308-21latency_hist07:09:581
312116499431,35cyclictest3121308-21latency_hist07:09:581
312116399433,30cyclictest3134061-21/usr/sbin/munin07:50:050
312116399432,25cyclictest3225058-21kworker/u16:4+rpciod11:05:110
312116399432,25cyclictest3225058-21kworker/u16:4+rpciod11:05:110
312116399432,18cyclictest3148565-21cpuspeed08:35:230
312116399432,18cyclictest3148565-21cpuspeed08:35:230
312116399432,17cyclictest3245698-21latency11:25:340
312116399432,16cyclictest3170006-21ntpq09:25:390
3121163994319,18cyclictest3260874-21ssh11:50:140
3121163994314,23cyclictest3236629-21ssh11:10:510
312116399431,20cyclictest3137249-21apt-get08:00:190
312116399431,20cyclictest3137249-21apt-get08:00:190
3121166994223,15cyclictest0-21swapper/309:42:543
3121166994219,17cyclictest0-21swapper/310:45:383
3121166994219,17cyclictest0-21swapper/310:45:383
3121165994226,12cyclictest0-21swapper/210:56:002
3121165994226,10cyclictest0-21swapper/210:50:472
312116599422,20cyclictest3123311-21uniq07:15:262
3121165994219,19cyclictest3208095-21ssh10:25:502
3121165994217,21cyclictest0-21swapper/209:25:072
3121165994216,22cyclictest3285917-21apt-get12:29:572
3121165994214,23cyclictest3175548-21ssh09:35:142
312116599421,17cyclictest3280335-21fschecks_time12:20:322
312116399422,35cyclictest3210983-21ssh10:30:330
312116399422,35cyclictest3210983-21ssh10:30:330
312116399422,31cyclictest3196106-21ssh10:06:180
312116399422,31cyclictest3196106-21ssh10:06:180
312116399422,16cyclictest3125299-21ntpq07:20:390
312116399421,35cyclictest3179152-21tune2fs09:40:360
312116399421,18cyclictest3251367-21apt-get11:35:110
312116399421,18cyclictest3251367-21apt-get11:35:110
3121166994123,14cyclictest0-21swapper/310:40:583
3121166994123,13cyclictest0-21swapper/309:49:583
3121166994123,13cyclictest0-21swapper/309:49:583
3121166994121,15cyclictest0-21swapper/311:21:083
3121166994121,15cyclictest0-21swapper/310:01:103
3121166994121,15cyclictest0-21swapper/310:01:103
3121166994120,16cyclictest0-21swapper/308:25:553
3121165994127,11cyclictest0-21swapper/209:09:572
3121165994125,13cyclictest0-21swapper/209:21:262
3121165994120,18cyclictest11050irq/206-8000000.ethernet-rx011:00:142
3121165994120,17cyclictest3258735-21ssh11:45:532
3121165994119,19cyclictest0-21swapper/211:40:082
3121165994117,20cyclictest3195277-21ssh10:05:372
3121165994117,20cyclictest3195277-21ssh10:05:372
3121165994116,16cyclictest3274276-21ssh12:10:312
312116399413,30cyclictest3197598-21ssh10:10:100
312116399412,32cyclictest3222900-21ssh10:50:100
312116399412,32cyclictest3193389-21ssh10:02:440
312116399412,32cyclictest3193389-21ssh10:02:440
3121163994119,18cyclictest3220678-21ssh10:45:420
3121163994119,18cyclictest3220678-21ssh10:45:420
3121163994119,18cyclictest3153839-21ntpq08:50:340
3121163994117,19cyclictest3185254-21ssh09:50:230
3121163994116,21cyclictest3283824-21ssh12:25:380
3121166994022,15cyclictest0-21swapper/310:30:563
3121166994022,15cyclictest0-21swapper/310:30:563
3121166994022,14cyclictest0-21swapper/312:05:143
3121166994022,14cyclictest0-21swapper/308:00:133
3121166994022,14cyclictest0-21swapper/308:00:133
3121166994021,15cyclictest0-21swapper/311:30:123
3121166994020,15cyclictest0-21swapper/310:10:243
3121166994019,17cyclictest0-21swapper/307:54:573
3121166994018,18cyclictest0-21swapper/310:55:083
3121166994018,16cyclictest0-21swapper/308:10:273
3121166994017,19cyclictest0-21swapper/307:20:133
3121166994017,18cyclictest0-21swapper/309:20:133
3121166994015,20cyclictest0-21swapper/310:25:093
3121166994015,19cyclictest0-21swapper/312:30:093
312116599403,27cyclictest3125906-21kworker/u16:18+rpciod07:50:052
3121165994026,10cyclictest0-21swapper/210:45:402
3121165994026,10cyclictest0-21swapper/210:45:402
3121165994023,13cyclictest11050irq/206-8000000.ethernet-rx009:00:182
312116599402,18cyclictest3144367-21ntpq08:20:452
312116599402,15cyclictest3211180-21ntpq10:30:422
312116599402,15cyclictest3211180-21ntpq10:30:422
312116399405,27cyclictest3156061-21munin-run08:59:560
312116399402,27cyclictest3200771-21ssh10:15:090
312116399402,27cyclictest3200771-21ssh10:15:090
312116399401,17cyclictest3128077-21fschecks_time07:30:240
3121166993921,14cyclictest0-21swapper/308:20:503
3121166993919,15cyclictest0-21swapper/311:54:563
3121166993918,16cyclictest0-21swapper/311:19:593
3121166993918,16cyclictest0-21swapper/309:15:083
3121166993918,16cyclictest0-21swapper/309:15:083
3121166993917,17cyclictest0-21swapper/308:41:003
3121165993928,7cyclictest3214491-21ssh10:35:482
3121165993928,7cyclictest3214491-21ssh10:35:482
3121165993926,9cyclictest0-21swapper/207:25:552
3121165993923,13cyclictest421ktimers/208:55:102
3121165993923,11cyclictest0-21swapper/212:30:432
312116599392,17cyclictest3138946-21head08:05:232
3121165993918,17cyclictest3130145-21ntpq07:35:452
3121165993916,10cyclictest3184922-21apt09:50:062
312116399393,21cyclictest3126499-21fschecks_time07:25:260
312116399391,34cyclictest3133685-21/usr/sbin/munin07:45:520
3121166993821,13cyclictest0-21swapper/311:25:093
3121166993821,13cyclictest0-21swapper/309:14:573
3121166993821,12cyclictest0-21swapper/310:50:533
3121166993820,14cyclictest0-21swapper/312:20:303
3121166993820,14cyclictest0-21swapper/308:59:583
3121166993819,15cyclictest0-21swapper/312:14:583
3121166993819,14cyclictest0-21swapper/307:30:193
3121166993818,16cyclictest0-21swapper/309:35:563
3121166993818,13cyclictest0-21swapper/307:10:223
3121166993818,13cyclictest0-21swapper/307:10:223
3121166993817,16cyclictest0-21swapper/312:00:563
3121166993817,16cyclictest0-21swapper/312:00:563
3121166993817,16cyclictest0-21swapper/311:35:343
3121166993817,16cyclictest0-21swapper/311:35:343
3121165993826,9cyclictest0-21swapper/208:45:092
3121165993824,10cyclictest0-21swapper/208:40:392
312116599381,9cyclictest311971450kworker/u16:16+xprtiod07:20:122
3121165993819,14cyclictest0-21swapper/207:09:582
3121165993819,14cyclictest0-21swapper/207:09:582
312116599381,30cyclictest3148765-21tail08:35:282
312116599381,30cyclictest3148765-21tail08:35:282
312116599381,17cyclictest3142422-21grep08:15:272
312116599381,17cyclictest3142422-21grep08:15:272
3121163993823,10cyclictest3151276-21kworker/u16:7+xprtiod08:45:070
312116399382,26cyclictest3140417-21apt-get08:10:130
3121163993814,18cyclictest3129297-21/usr/sbin/munin07:35:070
3121166993721,13cyclictest0-21swapper/307:55:413
3121166993719,13cyclictest0-21swapper/311:55:293
3121166993719,13cyclictest0-21swapper/310:19:563
3121166993719,13cyclictest0-21swapper/310:19:563
3121166993719,13cyclictest0-21swapper/310:05:433
3121166993719,13cyclictest0-21swapper/310:05:433
3121166993717,15cyclictest0-21swapper/312:25:213
3121166993716,16cyclictest0-21swapper/307:39:573
3121166993716,14cyclictest0-21swapper/309:05:383
3121166993715,17cyclictest0-21swapper/307:40:123
3121165993723,11cyclictest421ktimers/208:25:122
312116599372,30cyclictest3278832-21ssh12:17:452
3121165993718,13cyclictest0-21swapper/208:30:532
3121165993714,19cyclictest3140795-21fw_packets08:10:282
312116399373,25cyclictest3135660-21modprobe07:55:080
3121163993719,14cyclictest3141998-21sshd-session08:15:120
3121163993719,14cyclictest3141998-21sshd-session08:15:120
312116399371,31cyclictest3121662-21grep07:10:240
312116399371,31cyclictest3121662-21grep07:10:240
3121166993619,13cyclictest0-21swapper/311:10:553
3121166993617,15cyclictest0-21swapper/312:35:293
3121166993616,15cyclictest0-21swapper/310:35:103
3121166993616,15cyclictest0-21swapper/310:35:103
312116599362,29cyclictest3132860-21grep07:45:252
3121165993622,10cyclictest0-21swapper/207:30:202
3121166993519,12cyclictest0-21swapper/309:55:213
3121166993518,13cyclictest0-21swapper/312:15:373
3121166993518,13cyclictest0-21swapper/307:45:503
3121166993517,14cyclictest0-21swapper/309:25:283
3121166993516,14cyclictest0-21swapper/311:45:583
312116599351,13cyclictest3153277-21aten_r8power_vo08:50:222
3121163993516,15cyclictest3123812-21ntpq07:15:490
3121163993510,22cyclictest181ktimers/008:25:480
3121166993417,13cyclictest0-21swapper/308:30:063
3121166993417,10cyclictest0-21swapper/308:15:423
3121166993417,10cyclictest0-21swapper/308:15:423
3121165993421,10cyclictest0-21swapper/207:10:342
3121165993421,10cyclictest0-21swapper/207:10:342
3121166993317,11cyclictest0-21swapper/308:45:323
3121163993320,9cyclictest181ktimers/008:05:130
3121166993226,3cyclictest0-21swapper/308:35:213
3121166993226,3cyclictest0-21swapper/308:35:213
3121166993215,13cyclictest0-21swapper/307:09:583
3121166993215,13cyclictest0-21swapper/307:09:583
3121166993118,9cyclictest0-21swapper/308:50:363
3121166993117,10cyclictest0-21swapper/308:05:243
3121166993020,6cyclictest0-21swapper/309:50:503
3121166993016,12cyclictest0-21swapper/311:05:243
3121166993016,12cyclictest0-21swapper/311:05:243
3121166992822,3cyclictest0-21swapper/309:31:183
312116399282,19cyclictest3121187-21latency_hist07:09:570
312116399282,19cyclictest3121187-21latency_hist07:09:570
3121166992719,4cyclictest0-21swapper/307:29:563
3121166992620,3cyclictest0-21swapper/307:15:313
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional