You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-16 - 00:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackfslot7s.osadl.org (updated Tue Jul 15, 2025 00:44:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
346578899362018,3596cyclictest3524761-21kworker/3:0+events22:20:153
346578899333415,3313cyclictest3568389-21kworker/3:2+events00:35:203
34657809930896,3078cyclictest3522005-21kworker/1:2+events22:10:181
346578099289113,2875cyclictest3553909-21kworker/1:2+events00:05:001
346578099279613,2779cyclictest3560577-21kworker/1:1+events00:15:201
346578399278013,2761cyclictest3526647-21kworker/2:2+events22:30:132
346578899277811,2764cyclictest3533078-21kworker/3:1+events22:47:323
346577599270314,2687cyclictest3526462-21kworker/0:2+events22:45:130
346578099266810,2653cyclictest3531406-21kworker/1:2+events22:40:121
346577599266413,2648cyclictest3504319-21kworker/0:0+events21:20:140
346578099265213,2636cyclictest3512269-21kworker/1:0+events21:50:241
346577599264613,2629cyclictest3555116-21kworker/0:2+events23:50:140
346577599255412,2539cyclictest3526462-21kworker/0:2+events22:52:290
346578099253713,2519cyclictest3564713-21kworker/1:0+events00:20:131
34657809925030,2499cyclictest3505598-21kworker/1:2+events21:23:381
346577599249412,2479cyclictest3547254-21kworker/0:2+events23:24:400
346578099249312,2478cyclictest3560577-21kworker/1:1+events00:10:211
346578099249212,2476cyclictest3481057-21kworker/1:2+events19:56:361
346577599249212,2476cyclictest3504319-21kworker/0:0+events21:11:320
346577599249112,2475cyclictest3529982-21kworker/0:1+events22:30:580
346577599248610,2472cyclictest3488446-21kworker/0:2+events20:20:530
34657889924779,2464cyclictest3520639-21kworker/3:2+events22:05:173
346577599247710,2463cyclictest3511970-21kworker/0:2+events21:33:560
34657889924677,2458cyclictest3563201-21kworker/3:0+events00:30:063
346578099246212,2446cyclictest3531406-21kworker/1:2+events22:51:191
346577599246111,2446cyclictest3491400-21kworker/0:1+events20:44:290
346578399244410,2430cyclictest3520202-21kworker/2:0+events22:04:552
346578099243812,2423cyclictest3478701-21kworker/1:0+events20:01:131
346578399243312,2417cyclictest3520202-21kworker/2:0+events22:16:042
346578399243010,2416cyclictest3496165-21kworker/2:0+events21:18:452
346578399242812,2412cyclictest3534561-21kworker/2:0+events22:49:022
346578399242712,2411cyclictest3473779-21kworker/2:0+events19:32:582
346578399242212,2407cyclictest3473779-21kworker/2:0+events19:47:022
346578099241212,2396cyclictest3485035-21kworker/1:2+events20:59:251
346578399241011,2396cyclictest3496165-21kworker/2:0+events20:55:562
346578399239912,2383cyclictest3549056-21kworker/2:2+events00:14:102
346578399239713,2381cyclictest3486508-21kworker/2:1+events20:31:512
346578899238812,2373cyclictest3540844-21kworker/3:0+events23:05:333
346578099238812,2372cyclictest3537791-21kworker/1:0+events23:00:561
346578899238312,2368cyclictest3493064-21kworker/3:1+events20:37:373
346578899236412,2349cyclictest3553782-21kworker/3:2+events23:56:393
346578899236211,2347cyclictest3506326-21kworker/3:2+events21:16:063
346578899236111,2347cyclictest3473811-21kworker/3:2+events19:49:433
346578899235711,2342cyclictest3480065-21kworker/3:1+events20:06:283
346578099235612,2341cyclictest3568179-21kworker/1:1+events00:31:341
346578899235312,2338cyclictest3488171-21kworker/3:1+events20:19:213
346578899234812,2333cyclictest3513628-21kworker/3:0+events21:38:513
346578399233610,2323cyclictest3492941-21kworker/2:0+events20:36:302
346577599233412,2319cyclictest3540905-21kworker/0:0+events23:10:280
346578099232612,2312cyclictest3539489-21kworker/1:2+events23:20:031
346578099232612,2311cyclictest3522005-21kworker/1:2+events22:15:411
346578099232611,2312cyclictest3515292-21kworker/1:2+events21:49:371
346578099232112,2307cyclictest3556688-21kworker/1:1+events23:55:091
346578899231411,2297cyclictest3518747-21kworker/3:0+events21:59:123
346578899231311,2298cyclictest3510394-21kworker/3:2+events21:34:183
346578099230811,2295cyclictest3470751-21kworker/1:0+events19:40:551
346578099230311,2289cyclictest3539489-21kworker/1:2+events23:25:471
34657839923009,2287cyclictest3518566-21kworker/2:1+events21:55:242
346578099230011,2286cyclictest3502745-21kworker/1:0+events21:13:251
346577599229010,2276cyclictest3469154-21kworker/0:1+events19:27:350
346578899228911,2274cyclictest3463957-21kworker/3:0+events19:18:253
34657759922859,2273cyclictest3502664-21kworker/0:2+events21:07:220
34657759922838,2272cyclictest3510278-21kworker/0:1+events21:28:350
346578399227711,2262cyclictest3549056-21kworker/2:2+events23:45:142
346577599227611,2262cyclictest3507083-21kworker/0:2+events21:16:290
346577599227511,2262cyclictest3473692-21kworker/0:2+events19:37:320
346577599227511,2260cyclictest3534923-21kworker/0:1+events22:57:530
346578399227410,2261cyclictest3549056-21kworker/2:2+events23:31:292
346577599227211,2258cyclictest3526462-21kworker/0:2+events22:40:340
346578899227110,2257cyclictest3493064-21kworker/3:1+events20:43:213
346577599227111,2257cyclictest3555116-21kworker/0:2+events23:46:460
346577599227110,2257cyclictest3568300-21kworker/0:1+events00:29:420
346577599227011,2256cyclictest3476945-21kworker/0:1+events20:00:490
346578399226912,2254cyclictest3558707-21kworker/2:3+events00:01:342
346578099226910,2255cyclictest3485035-21kworker/1:2+events20:26:081
34657759922697,2258cyclictest3469154-21kworker/0:1+events19:31:490
346577599226911,2255cyclictest3513858-21kworker/0:3+events21:59:570
346577599226911,2254cyclictest3486839-21kworker/0:1+events20:18:560
346577599226910,2255cyclictest3548916-21kworker/0:0+events23:42:110
346577599226910,2255cyclictest3520348-21kworker/0:0+events22:04:070
34657759922689,2256cyclictest3555116-21kworker/0:2+events00:08:470
34657759922689,2256cyclictest3498032-21kworker/0:2+events20:54:480
346577599226811,2254cyclictest3480127-21kworker/0:0+events19:54:420
34657759922677,2256cyclictest3526462-21kworker/0:2+events22:29:490
346577599226711,2253cyclictest3520348-21kworker/0:0+events22:14:080
346577599226611,2252cyclictest3483586-21kworker/0:2+events20:11:190
346577599226610,2253cyclictest3564622-21kworker/0:2+events00:16:500
346577599226511,2251cyclictest3571228-21kworker/0:0+events00:39:340
346577599226411,2250cyclictest3485239-21kworker/0:0+events20:27:180
346577599226311,2249cyclictest3473692-21kworker/0:2+events19:42:490
34657759922626,2253cyclictest3491400-21kworker/0:1+events20:36:530
346578399226011,2246cyclictest3545805-21kworker/2:0+events23:23:312
346577599226011,2245cyclictest3526462-21kworker/0:2+events22:36:460
346577599225910,2246cyclictest3516928-21kworker/0:0+events21:47:410
346578899225813,2242cyclictest3470450-21kworker/3:2+events19:22:373
346577599225510,2241cyclictest3540905-21kworker/0:0+events23:02:050
34657759922509,2238cyclictest3473692-21kworker/0:2+events19:49:220
346578399224412,2228cyclictest3510257-21kworker/2:0+events21:30:322
346578399224210,2228cyclictest3496165-21kworker/2:0+events21:11:082
34657759922408,2229cyclictest3511970-21kworker/0:2+events21:36:330
346578399223910,2226cyclictest3475385-21kworker/2:1+events19:42:022
34657759922394,2232cyclictest3496324-21kworker/0:0+events20:46:470
346578399223711,2223cyclictest3480225-21kworker/2:2+events20:08:452
34657809922378,2226cyclictest3508910-21kworker/1:0+events21:26:421
34657889922359,2222cyclictest3555384-21kworker/3:1+events23:51:213
346578399223511,2220cyclictest3480225-21kworker/2:2+events20:04:572
346578399223510,2222cyclictest3552258-21kworker/2:1+events23:44:292
346578399223411,2220cyclictest3497776-21kworker/2:2+events20:51:472
346578399223410,2221cyclictest3500967-21kworker/2:1+events21:01:412
346578399223410,2220cyclictest3473779-21kworker/2:0+events19:36:462
34657889922339,2221cyclictest3472084-21kworker/3:1+events19:28:213
346578399223310,2220cyclictest3473779-21kworker/2:0+events19:57:002
34657839922319,2219cyclictest3516940-21kworker/2:2+events21:48:502
346578399223011,2215cyclictest3496165-21kworker/2:0+events21:07:472
34657839922289,2215cyclictest3554112-21kworker/2:0+events23:53:582
346578399222811,2213cyclictest3486508-21kworker/2:1+events20:47:102
346578399222810,2215cyclictest3564718-21kworker/2:0+events00:19:042
346578399222810,2215cyclictest3528539-21kworker/2:0+events22:28:402
346578399222810,2215cyclictest3494551-21kworker/2:2+events20:42:132
346578399222810,2214cyclictest3544178-21kworker/2:2+events23:16:562
346578399222510,2212cyclictest3536164-21kworker/2:2+events22:56:212
346577599222511,2212cyclictest3540905-21kworker/0:0+events23:15:040
346578399222311,2209cyclictest3513453-21kworker/2:0+events21:39:372
346578399222310,2210cyclictest3568255-21kworker/2:2+events00:33:292
346577599222111,2207cyclictest3555116-21kworker/0:2+events00:04:370
346577599222011,2206cyclictest3518627-21kworker/0:1+events21:54:130
346578899221910,2205cyclictest3550344-21kworker/3:1+events23:39:043
346578899221910,2205cyclictest3550344-21kworker/3:1+events23:39:043
34657839922175,2209cyclictest3526647-21kworker/2:2+events22:38:432
346578399221710,2204cyclictest3537770-21kworker/2:1+events23:01:432
34657759922169,2204cyclictest3526462-21kworker/0:2+events22:22:570
34657839922157,2204cyclictest3462077-21kworker/2:2+events19:19:582
346578399221410,2201cyclictest3547418-21kworker/2:1+events23:26:342
346578399221211,2198cyclictest3489745-21kworker/2:0+events20:25:232
346578399221010,2196cyclictest3549056-21kworker/2:2+events00:07:152
346578399220411,2190cyclictest3505847-21kworker/2:2+events21:29:462
346578399220410,2190cyclictest3566366-21kworker/2:1+events00:26:412
346578399220311,2189cyclictest3558262-21kworker/2:1+events23:58:572
346578899220212,2186cyclictest3544110-21kworker/3:0+events23:15:483
346578399220211,2189cyclictest3520202-21kworker/2:0+events22:11:032
346578399220011,2185cyclictest3473779-21kworker/2:0+events19:51:182
346578399219910,2186cyclictest3542570-21kworker/2:1+events23:11:352
34657809921997,2189cyclictest3501044-21kworker/1:1+events21:09:351
346577599219911,2185cyclictest3491400-21kworker/0:1+events20:58:150
346577599219411,2180cyclictest3513858-21kworker/0:3+events22:09:310
34657839921919,2179cyclictest3550668-21kworker/2:0+events23:37:092
34657839921919,2179cyclictest3550668-21kworker/2:0+events23:37:092
346578899219012,2175cyclictest3529805-21kworker/3:2+events22:37:333
346578899218810,2174cyclictest3555384-21kworker/3:1+events00:05:453
346578899218111,2167cyclictest3563201-21kworker/3:0+events00:18:203
346578399217911,2165cyclictest3534561-21kworker/2:0+events22:50:332
346578899217111,2157cyclictest3515535-21kworker/3:2+events21:53:513
34657889921685,2159cyclictest3564803-21kworker/3:1+events00:23:153
34657889921678,2154cyclictest3524761-21kworker/3:0+events22:16:523
346578899216711,2153cyclictest3553782-21kworker/3:2+events23:49:273
346578899216611,2152cyclictest3529805-21kworker/3:2+events22:29:043
346578899216611,2151cyclictest3504261-21kworker/3:0+events21:13:503
346578899216611,2151cyclictest3494672-21kworker/3:0+events20:52:533
34657889921659,2153cyclictest3478010-21kworker/3:0+events19:52:263
346578899216510,2152cyclictest3520639-21kworker/3:2+events22:01:053
346578899216411,2150cyclictest3529805-21kworker/3:2+events22:54:243
34657889921639,2150cyclictest3483972-21kworker/3:0+events20:10:353
34657889921638,2153cyclictest3475471-21kworker/3:1+events19:40:103
346578899216311,2149cyclictest3555384-21kworker/3:1+events00:02:433
346578899216311,2149cyclictest3533078-21kworker/3:1+events22:44:043
346578899216311,2148cyclictest3544110-21kworker/3:0+events23:44:063
346578899216311,2148cyclictest3500830-21kworker/3:1+events21:26:203
346578899216310,2149cyclictest3481851-21kworker/3:2+events20:02:213
346578899216211,2148cyclictest3500830-21kworker/3:1+events21:21:453
346578899216211,2147cyclictest3512002-21kworker/3:1+events21:43:043
346578899216111,2148cyclictest3547579-21kworker/3:2+events23:28:503
346578899216111,2146cyclictest3529805-21kworker/3:2+events22:57:303
34657889921599,2147cyclictest3517139-21kworker/3:1+events22:14:553
34657889921598,2147cyclictest3463957-21kworker/3:0+events19:37:563
34657889921578,2146cyclictest3544110-21kworker/3:0+events23:30:463
346578899215211,2137cyclictest3491461-21kworker/3:0+events20:29:113
34657759921468,2135cyclictest3513858-21kworker/0:3+events21:42:190
346578099214513,2128cyclictest3470751-21kworker/1:0+events19:25:431
346578899213911,2125cyclictest3564803-21kworker/3:1+events00:25:353
346578899213711,2122cyclictest3531469-21kworker/3:0+events22:33:413
346578099213411,2119cyclictest3515292-21kworker/1:2+events21:44:591
346578099213310,2119cyclictest3495049-21kworker/1:0+events20:41:031
346577599213010,2117cyclictest3548916-21kworker/0:0+events23:36:020
346577599213010,2117cyclictest3548916-21kworker/0:0+events23:36:020
346578899212911,2116cyclictest3500830-21kworker/3:1+events21:04:183
346578899212910,2115cyclictest3478010-21kworker/3:0+events19:57:233
346578899212811,2113cyclictest3542509-21kworker/3:2+events23:13:303
34657839921225,2114cyclictest3539376-21kworker/2:0+events23:09:422
346578099211911,2102cyclictest3552147-21kworker/1:0+events23:39:281
346578099211911,2102cyclictest3552147-21kworker/1:0+events23:39:271
346577599211810,2104cyclictest3476945-21kworker/0:1+events19:58:100
346577599211410,2100cyclictest3548916-21kworker/0:0+events23:31:520
34657839921084,2101cyclictest3516940-21kworker/2:2+events21:52:192
34657809921086,2099cyclictest3558348-21kworker/1:0+events00:05:231
346578099210811,2094cyclictest3478701-21kworker/1:0+events20:06:521
346578099210711,2092cyclictest3552147-21kworker/1:0+events23:41:011
346578099210711,2092cyclictest3478701-21kworker/1:0+events19:50:541
346578099210710,2094cyclictest3518474-21kworker/1:1+events22:05:411
34657889921068,2094cyclictest3488171-21kworker/3:1+events20:21:583
346578099210611,2092cyclictest3469036-21kworker/1:2+events19:19:131
346578099210511,2091cyclictest3550522-21kworker/1:1+events23:34:341
34657759921059,2093cyclictest3483586-21kworker/0:2+events20:07:140
34657759921047,2094cyclictest3564622-21kworker/0:2+events00:24:250
34657809921008,2089cyclictest3510514-21kworker/1:1+events21:35:471
346578099210011,2086cyclictest3497747-21kworker/1:1+events20:51:241
34657759921006,2091cyclictest3491400-21kworker/0:1+events20:33:020
346578099209811,2084cyclictest3553909-21kworker/1:2+events23:50:581
34657759920989,2086cyclictest3467085-21kworker/0:2+events19:19:340
346578099209711,2083cyclictest3475205-21kworker/1:2+events19:38:161
34657759920969,2084cyclictest3556893-21kworker/0:1+events23:57:240
346577599209510,2082cyclictest3467085-21kworker/0:2+events19:23:470
34657839920946,2085cyclictest3513453-21kworker/2:0+events21:44:142
346578099209411,2080cyclictest3539489-21kworker/1:2+events23:05:551
34657759920939,2081cyclictest3500803-21kworker/0:0+events21:00:560
34657809920917,2081cyclictest3526339-21kworker/1:1+events22:31:211
346578399208710,2073cyclictest3486508-21kworker/2:1+events20:24:392
34657839920856,2073cyclictest3483703-21kworker/2:0+events20:10:572
34657839920847,2073cyclictest3564718-21kworker/2:0+events00:22:522
346578099208410,2071cyclictest3505598-21kworker/1:2+events21:18:221
34657759920828,2071cyclictest3524853-21kworker/0:1+events22:17:120
34657839920786,2069cyclictest3525017-21kworker/2:1+events22:23:432
346578399207710,2064cyclictest3486508-21kworker/2:1+events20:16:182
346578399207610,2063cyclictest3505847-21kworker/2:2+events21:21:002
34657839920754,2068cyclictest3467058-21kworker/2:1+events19:24:112
34657889920719,2058cyclictest3502455-21kworker/3:2+events21:07:003
34657839920703,2064cyclictest3470539-21kworker/2:0+events19:26:062
346578399206910,2056cyclictest3462077-21kworker/2:2+events19:13:102
346578899205710,2044cyclictest3491461-21kworker/3:0+events20:30:413
346578399205610,2043cyclictest3521828-21kworker/2:1+events22:07:372
34657889920518,2039cyclictest3494672-21kworker/3:0+events20:49:513
34657809920439,2031cyclictest3485035-21kworker/1:2+events20:47:321
34657839920417,2030cyclictest3568255-21kworker/2:2+events00:38:492
346578099204111,2027cyclictest3466996-21kworker/1:1+events19:22:141
34657809920393,2033cyclictest3505598-21kworker/1:2+events21:32:021
346578899203110,2018cyclictest3513628-21kworker/3:0+events21:48:273
34657759920269,2014cyclictest3548916-21kworker/0:0+events23:29:590
346578099202411,2009cyclictest3472624-21kworker/1:1+events19:32:341
346578099201811,2004cyclictest3528653-21kworker/1:0+events22:27:541
34657889920137,2003cyclictest3494672-21kworker/3:0+events20:56:403
34657809920115,2003cyclictest3501044-21kworker/1:1+events21:02:481
346578899200810,1995cyclictest3561447-21kworker/3:2+events00:13:023
346578899200610,1993cyclictest3540844-21kworker/3:0+events23:02:293
34657889920057,1995cyclictest3465730-21kworker/3:1+events19:14:413
346578099199411,1980cyclictest3485035-21kworker/1:2+events20:23:301
34657759919899,1976cyclictest3462160-21kworker/0:0+events19:13:330
34657809919809,1968cyclictest3537791-21kworker/1:0+events22:58:391
346578099196710,1954cyclictest3562925-21kworker/1:2+events00:27:281
346578099195910,1945cyclictest3544290-21kworker/1:0+events23:18:051
346578099195210,1940cyclictest3478701-21kworker/1:0+events20:17:281
346578099195010,1937cyclictest3539489-21kworker/1:2+events23:14:171
346578099194710,1934cyclictest3536158-21kworker/1:1+events22:49:261
34657889919469,1935cyclictest3545836-21kworker/3:1+events23:21:583
346578099194310,1930cyclictest3520166-21kworker/1:0+events22:01:511
346578099194310,1930cyclictest3518474-21kworker/1:1+events21:57:391
34657809919427,1932cyclictest3568179-21kworker/1:1+events00:39:121
34657759919426,1933cyclictest3561877-21kworker/0:0+events00:12:400
34657889919329,1920cyclictest3472084-21kworker/3:1+events19:30:403
34657809918927,1882cyclictest3460438-21kworker/1:0+events19:13:541
34657809918688,1857cyclictest3552147-21kworker/1:0+events23:46:231
34657809918679,1855cyclictest3526339-21kworker/1:1+events22:36:231
34653222160117,24sleep30-21swapper/319:07:023
34654292155127,23sleep10-21swapper/119:08:231
34652032143112,26sleep20-21swapper/219:05:332
34652492141109,26sleep00-21swapper/019:06:060
346578099103101,1cyclictest3476914-21kworker/1:1+pm19:49:571
3465780992016,2cyclictest331ktimers/122:20:181
346578099181,15cyclictest3494664-21sshd20:38:331
3465783991712,3cyclictest3534244-21pmu-power22:40:192
346578099178,6cyclictest3492457-21ntpq20:30:191
346578099161,8cyclictest3486893-21perf20:15:011
346577599160,14cyclictest3568894-21/usr/sbin/munin00:30:160
346577599150,2cyclictest3542432-21sshd23:05:320
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional