You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 18:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Thu Nov 20, 2025 12:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2895999720,60cyclictest28498-21unixbench-2d09:30:330
2895999626,50cyclictest13324-21fschecks_count09:55:240
2895999430,47cyclictest5356-21aten_rbpower_en12:00:190
2895999020,36cyclictest0-21swapper/010:00:180
2895998825,52cyclictest2125-21ssh12:37:120
2895998626,47cyclictest21893-21ssh10:06:240
2895998623,50cyclictest26183-21idleruntime-cro11:45:060
289599861,60cyclictest0-21swapper/008:40:140
2895998521,48cyclictest0-21swapper/011:20:170
2895998520,55cyclictest1268-21taskset09:38:520
2895998429,39cyclictest20619-21diskstats12:20:210
2895998328,33cyclictest0-21swapper/010:40:170
2895998326,45cyclictest4086-21uptime07:10:340
2895998324,42cyclictest12916-21rm09:55:060
2895998322,48cyclictest0-21swapper/011:00:190
2895998322,41cyclictest32028-21grep10:20:320
2895998320,50cyclictest0-21swapper/009:25:050
2895998228,38cyclictest25911-21ssh10:12:280
2895998224,47cyclictest12224-21idleruntime-cro10:40:050
2895998219,55cyclictest0-21swapper/010:18:050
2895998216,49cyclictest0-21swapper/010:57:280
2895998123,43cyclictest27388-21ls08:15:340
2895998021,44cyclictest24708-21basename08:10:190
2895998018,53cyclictest11786-21sh12:08:560
289599797,52cyclictest0-21swapper/010:50:200
2895997926,38cyclictest5014-21taskset10:29:210
2895997925,12cyclictest31198-21grep08:25:500
2895997923,42cyclictest31846-21ssh12:34:460
2895997920,46cyclictest0-21swapper/009:25:240
2895997920,44cyclictest0-21swapper/012:25:560
2895997919,10cyclictest11391-21grep09:00:330
2895997918,46cyclictest13715-21rm11:26:010
2895997825,32cyclictest17207-21cron07:50:050
2895997823,32cyclictest29105-21cat08:20:320
2895997721,38cyclictest7303-21ssh10:31:470
2895997720,43cyclictest0-21swapper/011:57:150
2895997719,47cyclictest0-21swapper/011:37:270
2895997719,44cyclictest0-21swapper/011:48:520
2895997715,54cyclictest0-21swapper/012:17:480
2895997620,44cyclictest0-21swapper/007:41:200
218902760,2sleep10-21swapper/110:52:321
2895997523,40cyclictest3407-21ssh11:11:410
2895997520,43cyclictest0-21swapper/011:32:070
2895997520,43cyclictest0-21swapper/008:50:060
62122740,4sleep2289799cyclictest09:45:262
289899741,62cyclictest27288-21apt-get10:15:193
2895997427,26cyclictest22629-21apt08:05:050
2895997427,26cyclictest22629-21apt08:05:050
2895997425,11cyclictest8911-21ssh09:49:480
2895997421,40cyclictest19149-21ssh09:16:540
2895997417,43cyclictest0-21swapper/011:17:490
2895997322,38cyclictest646-21diskmemload11:08:510
2895997322,37cyclictest2554-21runrttasks08:36:490
2895997318,43cyclictest17685-21sh10:46:010
2895997317,41cyclictest19280-21aten_rbpower_cu07:55:170
2895997223,39cyclictest14339-21sh12:11:200
2895997223,39cyclictest14339-21sh12:11:200
2895997220,37cyclictest0-21swapper/007:36:090
2895997219,38cyclictest0-21swapper/009:41:570
2895997216,46cyclictest0-21swapper/011:54:590
2895997119,33cyclictest6157-21taskset07:18:320
2895997117,45cyclictest13534-21runrttasks09:08:410
179202710,1sleep30-21swapper/311:32:153
2895996918,42cyclictest1120-21runrttasks08:10:030
2895996917,42cyclictest0-21swapper/007:31:020
2895996819,34cyclictest15567-21taskset09:11:260
289899671,40cyclictest776-21ssh11:54:383
2895996619,33cyclictest0-21swapper/007:23:200
2895996619,30cyclictest0-21swapper/008:57:520
289899651,55cyclictest12763-21df10:40:223
289899651,46cyclictest27489-21/usr/sbin/munin08:15:353
289899651,44cyclictest7950irq/56-dwc_otg_09:22:303
2897996526,13cyclictest29739-21ssh10:18:082
289899641,51cyclictest25525-21ssh11:43:343
289899641,42cyclictest30094-21ssh12:31:333
2897996427,17cyclictest26685-21/usr/sbin/munin11:00:252
2895996425,25cyclictest9559-21runrttasks07:25:570
289899631,45cyclictest9053-21apt-get09:50:113
289899631,43cyclictest8631-21apt-get11:20:183
2897996329,13cyclictest24942-21fschecks_count08:10:222
2895996322,28cyclictest8044-21seq08:52:470
2895996322,28cyclictest8044-21seq08:52:470
289899622,38cyclictest7850irq/56-dwc_otg_10:15:063
289899621,51cyclictest22002-21ssh12:21:133
289899621,40cyclictest18932-21ssh10:48:553
2895996223,21cyclictest679-21runrttasks08:31:110
2895996212,41cyclictest0-21swapper/007:51:400
289899611,52cyclictest11947-21ssh12:09:223
289899611,50cyclictest19224-21ssh10:03:393
289899611,47cyclictest15183-21sh11:29:203
289899611,45cyclictest24229-21sshd08:07:413
2897996130,12cyclictest28582-21awk09:30:342
2897996127,17cyclictest24961-21/usr/sbin/munin09:25:312
2897996126,16cyclictest32059-21sh11:52:072
2897996124,12cyclictest3912-21wc12:40:062
289899603,10cyclictest7950irq/56-dwc_otg_07:50:173
289899601,46cyclictest26944-21ssh11:00:243
289899601,44cyclictest31816-21ssh11:06:013
289899601,43cyclictest1273-21ssh09:38:533
289899601,41cyclictest3796-21ssh11:57:533
2897996028,15cyclictest27388-21rm12:29:072
2897996026,8cyclictest8883-21diskstats11:20:242
2897996026,20cyclictest23089-21/usr/sbin/munin11:40:212
2897996026,12cyclictest20601-21ssh12:20:202
2897996025,19cyclictest25237-21ssh10:57:312
289899591,46cyclictest13537-21sshd09:08:433
2897995927,14cyclictest12676-21ssh10:40:212
2897995924,15cyclictest3789-21ssh11:57:522
289899582,36cyclictest7950irq/56-dwc_otg_09:55:283
289899581,49cyclictest7761-21ssh09:47:103
289899581,47cyclictest19169-21ssh12:18:163
289899581,47cyclictest16765-21ssh09:14:133
2897995827,20cyclictest32600-21ssh09:36:122
2897995826,11cyclictest5507-21/usr/sbin/munin12:00:212
2897995825,12cyclictest7313-21ssh10:31:472
2897995825,12cyclictest22670-21cut08:05:062
2897995825,12cyclictest22670-21cut08:05:062
2897995819,13cyclictest11929-21taskset12:09:192
289699582,8cyclictest7750irq/56-dwc_otg11:40:181
289899571,43cyclictest3906-21ssh09:41:363
289899571,36cyclictest3816-21ssh10:26:323
289799575,9cyclictest7850irq/56-dwc_otg_11:35:182
2897995733,18cyclictest0-21swapper/207:10:082
2897995725,13cyclictest19053-21sh11:34:392
2897995725,12cyclictest28242-21/usr/sbin/munin12:30:332
2897995724,17cyclictest4669-21ssh11:14:372
2897995723,20cyclictest3937-21chrt09:41:402
2897995723,18cyclictest19347-21ssh10:03:562
2897995723,15cyclictest21935-21ssh10:06:292
289899561,45cyclictest7791-21ssh12:03:403
289899561,41cyclictest24716-21missed_timers09:25:273
289899561,39cyclictest1012-21ssh10:23:243
289899561,38cyclictest21562-21ssh11:37:523
2897995625,11cyclictest19142-21ssh09:16:532
2897995620,11cyclictest21743-21taskset10:52:122
23502562,13sleep20-21swapper/207:06:422
23502562,13sleep20-21swapper/207:06:422
289899553,8cyclictest0-21swapper/310:30:173
289899551,43cyclictest3454-21rm11:11:473
289899551,42cyclictest25917-21ssh12:26:183
289899551,41cyclictest7950irq/56-dwc_otg_08:10:283
289899551,41cyclictest3530-21ssh12:39:463
289899551,34cyclictest29921-21ssh09:33:253
2897995523,15cyclictest16221-21ssh09:12:592
2897995522,20cyclictest15477-21ssh09:58:052
2897995515,15cyclictest14187-21ssh11:27:042
289899548,4cyclictest7950irq/56-dwc_otg_07:35:183
289899541,34cyclictest24412-21sshd10:55:443
2897995424,12cyclictest11743-21ssh09:53:002
2897995424,10cyclictest0-21swapper/209:25:062
2897995423,11cyclictest11205-21ssh10:37:492
2897995418,22cyclictest7950irq/56-dwc_otg_12:15:062
2897995418,22cyclictest7950irq/56-dwc_otg_12:15:062
289899531,41cyclictest23155-21taskset10:09:193
2897995326,17cyclictest17885-21ssh10:46:262
2897995325,12cyclictest29338-21grep08:20:352
2897995324,18cyclictest3822-21ssh10:26:332
92422520,4sleep20-21swapper/207:25:302
81342520,2sleep10-21swapper/107:25:051
289899524,6cyclictest5164-21ssh11:15:173
289899521,39cyclictest31313-21taskset08:25:583
2898995213,8cyclictest7850irq/56-dwc_otg_07:15:193
289899521,37cyclictest140-21systemd-journal10:55:053
289899521,36cyclictest0-21swapper/311:48:523
2897995225,17cyclictest31553-21chrt08:28:582
2897995224,17cyclictest7225-21ssh11:17:292
2897995220,21cyclictest15038-21chrt07:41:452
289899511,47cyclictest10004-21sshd08:59:423
289899511,46cyclictest11335-21ls09:00:323
289899511,45cyclictest21763-21ntp_states08:00:283
289899511,45cyclictest21763-21ntp_states08:00:283
289899511,39cyclictest16117-21ssh12:14:593
289899511,39cyclictest16117-21ssh12:14:593
2897995122,19cyclictest6203-21seq08:47:122
2897995122,18cyclictest20788-21chrt07:59:522
2897995118,19cyclictest7950irq/56-dwc_otg_12:17:572
289899503,41cyclictest0-21swapper/307:10:083
2898995022,26cyclictest10987-21switchtime07:30:313
289899501,46cyclictest8342-21apt-get07:25:173
289899501,45cyclictest3292-21/usr/sbin/munin08:40:193
289899501,41cyclictest0-21swapper/309:20:013
2897995030,7cyclictest18022-21memory07:50:262
2897995024,17cyclictest27893-21taskset08:18:192
2897995022,10cyclictest7850irq/56-dwc_otg_08:55:172
2897995016,10cyclictest7950irq/56-dwc_otg_11:48:592
289899491,45cyclictest7884-21sshd08:50:423
289899491,45cyclictest7884-21sshd08:50:423
289899491,45cyclictest5430-21load08:45:243
289899491,45cyclictest13760-21apt-get07:40:173
2897994924,10cyclictest662-21ssh11:08:532
2897994921,15cyclictest7777-21taskset07:21:082
289699490,11cyclictest7750irq/56-dwc_otg12:30:161
289899481,44cyclictest7741-21sshd07:20:403
289899481,44cyclictest28288-21apt-get08:20:203
289899481,44cyclictest2655-21seq08:38:063
289899481,44cyclictest19961-21ntp_states07:55:273
289899481,44cyclictest11484-21ssh10:38:283
289899481,43cyclictest31880-21/usr/sbin/munin08:30:343
2897994821,16cyclictest634-21ssh10:22:312
2897994819,19cyclictest25952-21taskset10:12:342
2897994819,12cyclictest13320-21seq09:05:592
2897994818,12cyclictest11602-21chrt07:33:562
289699487,26cyclictest0-21swapper/109:00:191
2897994716,10cyclictest7950irq/56-dwc_otg_08:50:172
2897994716,10cyclictest7950irq/56-dwc_otg_08:50:172
289699475,17cyclictest0-21swapper/109:35:151
289899461,41cyclictest15634-21taskset07:45:123
2897994620,11cyclictest24620-21cut08:10:052
2897994620,11cyclictest11467-21who09:00:342
2897994519,15cyclictest5920-21seq07:15:462
25032453,34sleep00-21swapper/007:08:420
25032453,34sleep00-21swapper/007:08:420
2897994431,11cyclictest15312-21apt-get07:45:122
2897994417,10cyclictest27732sleep208:39:372
289699443,18cyclictest0-21swapper/107:10:081
2896994413,9cyclictest12555-21ssh10:40:191
289699436,5cyclictest23-21ksoftirqd/108:15:051
289699432,4cyclictest32059-21ls08:30:191
112692430,2sleep10-21swapper/109:51:531
2897994228,11cyclictest3829-21ps08:40:272
2897994227,12cyclictest32576-21/usr/sbin/munin08:30:262
289699427,24cyclictest0-21swapper/108:15:211
289699426,23cyclictest0-21swapper/110:30:191
289699424,31cyclictest0-21swapper/111:50:191
289699421,6cyclictest7750irq/56-dwc_otg11:25:211
2896994214,21cyclictest23-21ksoftirqd/112:00:191
2896994210,5cyclictest7750irq/56-dwc_otg09:15:051
2897994127,11cyclictest12336-21/usr/sbin/munin07:35:232
289699416,22cyclictest0-21swapper/110:20:191
289699413,23cyclictest7750irq/56-dwc_otg10:00:321
289699409,4cyclictest0-21swapper/112:39:061
2896994010,8cyclictest7750irq/56-dwc_otg12:15:341
62052390,5sleep10-21swapper/108:47:141
34642390,2sleep10-21swapper/111:11:481
157372390,1sleep10-21swapper/109:58:421
289699387,6cyclictest0-21swapper/109:20:331
289699387,22cyclictest0-21swapper/107:45:121
289699381,23cyclictest8688-21ls07:25:231
2896993621,5cyclictest20826-21if_eth012:20:231
2896993617,9cyclictest0-21swapper/111:05:341
2896993617,5cyclictest1378950wget07:40:201
2896993615,3cyclictest7750irq/56-dwc_otg10:45:231
2896993610,9cyclictest0-21swapper/107:35:171
289699354,4cyclictest7750irq/56-dwc_otg08:25:481
289699354,23cyclictest5158-21ssh11:15:161
289699354,21cyclictest0-21swapper/110:11:561
289699352,18cyclictest23345-21fschecks_count10:55:231
289699352,17cyclictest0-21swapper/110:06:111
267632351,4sleep126770-21sh09:29:281
267423510,11sleep30-21swapper/307:10:073
267423510,11sleep30-21swapper/307:10:073
289699347,7cyclictest7750irq/56-dwc_otg11:36:091
289699347,5cyclictest0-21swapper/109:05:171
289699345,21cyclictest27672-21ls11:00:341
289699344,8cyclictest0-21swapper/109:45:251
2896993426,5cyclictest221ktimersoftd/108:50:171
2896993426,5cyclictest221ktimersoftd/108:50:171
289699342,3cyclictest7950irq/56-dwc_otg_07:50:181
289699341,24cyclictest3591-21latency08:40:231
289699341,22cyclictest14997-1kworker/u9:212:25:191
289699338,5cyclictest0-21swapper/110:35:071
289699336,6cyclictest7750irq/56-dwc_otg11:30:271
289699335,20cyclictest12663-21ssh12:10:191
289699335,20cyclictest12663-21ssh12:10:191
289699333,6cyclictest0-21swapper/108:00:301
289699333,6cyclictest0-21swapper/108:00:301
289699331,7cyclictest7750irq/56-dwc_otg11:25:051
289699331,25cyclictest0-21swapper/110:29:431
289699331,24cyclictest2254-21timerwakeupswit08:35:361
289699331,23cyclictest8755-21uniq08:55:441
289699331,22cyclictest2445-21ssh11:55:311
2896993312,12cyclictest23-21ksoftirqd/111:45:201
2896993310,4cyclictest7750irq/56-dwc_otg07:30:221
23422333,21sleep10-21swapper/107:06:371
23422333,21sleep10-21swapper/107:06:371
289699329,6cyclictest7750irq/56-dwc_otg09:40:211
289699327,5cyclictest28477-21grep09:30:331
289699322,6cyclictest7750irq/56-dwc_otg09:15:391
289699322,15cyclictest0-21swapper/112:05:311
289699311,6cyclictest28542-21grep08:20:241
289699311,17cyclictest0-21swapper/108:00:061
289699307,4cyclictest0-21swapper/110:16:001
289699283,9cyclictest23324-21irqstats08:05:231
289699277,9cyclictest7850irq/56-dwc_otg_07:15:311
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional