You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 14:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Sat Nov 22, 2025 12:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8534997149,9cyclictest29566irq/58-eth010:35:153
852699593,48cyclictest26346irq/40-dwc2_hso11:20:152
851499563,45cyclictest26923-21ssh10:15:151
852699531,48cyclictest25615-21ssh12:00:142
8514995316,4cyclictest131rcu_preempt12:25:151
850699530,44cyclictest6691-21ssh11:05:150
852699521,45cyclictest16059-21ssh10:50:152
852699521,45cyclictest0-21swapper/210:00:142
851499524,5cyclictest131rcu_preempt11:10:121
852699511,3cyclictest0-21swapper/207:40:162
851499513,5cyclictest0-21swapper/109:35:161
8526995030,10cyclictest26943-21diskmemload12:20:142
852699502,42cyclictest0-21swapper/207:45:162
853499496,25cyclictest29566irq/58-eth009:45:183
852699491,41cyclictest0-21swapper/207:10:132
852699470,26cyclictest25191-21memory11:15:252
852699463,38cyclictest0-21swapper/208:35:152
852699461,4cyclictest0-21swapper/207:35:142
850699460,4cyclictest131rcu_preempt08:00:150
852699451,39cyclictest24537-21apt-get08:00:142
852699451,27cyclictest365-21apt-get12:25:152
852699441,39cyclictest27896-21apt-get07:30:132
852699441,30cyclictest26911-21apt-get10:15:152
8514994419,3cyclictest131rcu_preempt11:30:131
853499435,23cyclictest29566irq/58-eth010:10:133
852699431,38cyclictest32681-21ssh10:40:142
852699431,38cyclictest13993-21apt-get11:10:132
852699430,29cyclictest4909-21irqrtprio11:25:202
850699431,34cyclictest377-21ssh12:25:130
852699422,32cyclictest0-21swapper/211:50:142
852699421,36cyclictest0-21swapper/209:35:152
852699420,27cyclictest23431-21ssh09:28:412
852699420,27cyclictest23431-21ssh09:28:402
852699411,28cyclictest6069-21usage_Anybus10:20:292
852699401,28cyclictest1585166ssh10:28:062
852699401,23cyclictest0-21swapper/211:05:302
852699401,21cyclictest0-21swapper/211:42:132
852699392,32cyclictest26346irq/40-dwc2_hso12:30:142
852699391,33cyclictest12004-21apt-get11:30:142
852699391,24cyclictest2644-21/usr/sbin/munin09:15:202
852699391,24cyclictest0-21swapper/210:08:142
853499385,16cyclictest29566irq/58-eth011:55:143
853499384,18cyclictest0-21swapper/312:25:153
853499383,22cyclictest29566irq/58-eth009:40:143
852699381,2cyclictest0-21swapper/211:00:142
852699372,32cyclictest0-21swapper/212:35:152
852699371,32cyclictest0-21swapper/208:50:132
852699371,26cyclictest0-21swapper/208:10:212
852699370,22cyclictest0-21swapper/208:55:132
850699371,25cyclictest16735-21ssh12:35:150
853499363,17cyclictest29566irq/58-eth009:15:163
853499363,15cyclictest29566irq/58-eth012:35:153
852699361,2cyclictest0-21swapper/209:20:122
852699361,2cyclictest0-21swapper/208:05:142
852699361,26cyclictest19621-21ntp_192.168.11408:25:252
852699361,26cyclictest14031-21sed07:15:192
852699361,22cyclictest0-21swapper/208:15:212
852699360,28cyclictest23152-21sshd10:10:432
852699360,25cyclictest30283-21cut09:10:222
853499353,18cyclictest29566irq/58-eth011:15:173
852699354,19cyclictest0-21swapper/211:46:472
852699352,29cyclictest21272-21apt-get09:50:162
852699351,30cyclictest25840-21latency_hist08:35:022
852699351,2cyclictest0-21swapper/212:15:142
852699351,25cyclictest23586-21head07:55:292
852699350,21cyclictest2118-21ssh09:57:052
8514993516,15cyclictest21-21ksoftirqd/110:25:121
8514993514,3cyclictest131rcu_preempt08:45:131
852699342,27cyclictest0-21swapper/211:38:432
852699341,28cyclictest26346irq/40-dwc2_hso10:35:152
852699341,25cyclictest10311-21/usr/sbin/munin09:40:262
852699341,24cyclictest12106-21kthreadcore08:20:212
852699341,22cyclictest6905-21ssh12:08:492
852699340,2cyclictest7936-21apt-get12:10:152
8514993418,9cyclictest21-21ksoftirqd/111:56:031
852699334,26cyclictest26943-21diskmemload10:32:372
852699331,29cyclictest14853-21/usr/sbin/munin07:50:292
852699330,29cyclictest28883-21ssh10:56:072
853499324,18cyclictest29566irq/58-eth010:45:153
853499324,16cyclictest29566irq/58-eth009:10:113
853499324,15cyclictest29566irq/58-eth011:35:133
8534993217,8cyclictest0-21swapper/307:50:163
852699321,22cyclictest31286-21apt09:35:012
852699320,29cyclictest3055-21munin-node08:45:132
852699320,28cyclictest21893-21grep07:20:312
852699312,26cyclictest22148-21idleruntime-cro09:05:012
852699311,27cyclictest8123-21ssh10:45:022
852699311,22cyclictest14891-21ssh09:45:222
8514993110,3cyclictest131rcu_preempt07:25:151
850699318,3cyclictest131rcu_preempt12:00:140
853499303,19cyclictest29566irq/58-eth012:20:133
853499303,14cyclictest29566irq/58-eth008:00:143
852699301,26cyclictest2834-21sshd08:44:412
852699301,26cyclictest19824-21kthreadcore11:55:202
852699301,25cyclictest27835-21latency_hist07:30:012
852699300,27cyclictest26932-21sshd09:08:422
851499302,14cyclictest23752-21ssh10:55:151
8514993016,8cyclictest21-21ksoftirqd/110:35:441
850699302,22cyclictest29611-21ssh10:35:440
853499296,17cyclictest29566irq/58-eth011:30:133
853499292,17cyclictest29566irq/58-eth009:35:163
853499291,18cyclictest29566irq/58-eth009:30:133
8514992917,7cyclictest530-21usb-storage09:50:151
850699292,5cyclictest131rcu_preempt09:15:150
853499285,12cyclictest0-21swapper/310:40:143
853499281,16cyclictest29566irq/58-eth012:15:143
852699281,23cyclictest8560-21latency_hist07:10:022
851499282,9cyclictest21-21ksoftirqd/110:45:151
853499273,17cyclictest0-21swapper/311:05:583
853499272,18cyclictest0-21swapper/311:26:003
851499276,14cyclictest21-21ksoftirqd/109:00:141
851499270,22cyclictest4240-21ssh11:25:151
8506992716,3cyclictest91-21pr/ttyS012:20:010
853499269,11cyclictest0-21swapper/311:41:033
853499262,18cyclictest0-21swapper/312:31:073
851499265,5cyclictest131rcu_preempt07:30:121
851499262,19cyclictest0-21swapper/112:36:421
850699264,13cyclictest12-21ksoftirqd/009:00:130
853499258,11cyclictest0-21swapper/311:21:003
853499255,12cyclictest0-21swapper/312:05:143
853499254,11cyclictest29566irq/58-eth012:10:153
851499250,19cyclictest13798-21ssh12:31:081
850699257,15cyclictest12-21ksoftirqd/010:25:120
850699253,7cyclictest131rcu_preempt11:33:440
850699251,18cyclictest8429-21ssh10:23:410
853499242,10cyclictest0-21swapper/310:23:413
8534992411,10cyclictest0-21swapper/311:00:313
851499247,8cyclictest21-21ksoftirqd/109:05:221
851499246,3cyclictest131rcu_preempt08:00:131
850699246,9cyclictest17305-21unixbench_multi07:15:280
850699245,12cyclictest23733-21ssh10:55:120
850699242,16cyclictest11543-21ntp_192.168.11412:10:260
850699241,17cyclictest24441-21ssh12:20:130
8506992411,5cyclictest12127-21kthreadcore09:20:200
853499231,12cyclictest29566irq/58-eth009:55:163
851499237,2cyclictest131rcu_preempt10:52:441
851499235,7cyclictest21-21ksoftirqd/112:10:211
8514992318,3cyclictest0-21swapper/110:20:141
851499231,2cyclictest131rcu_preempt12:15:181
850699239,6cyclictest26943-21diskmemload09:11:050
850699237,12cyclictest12-21ksoftirqd/010:16:420
850699236,4cyclictest131rcu_preempt11:46:420
8534992213,6cyclictest29566irq/58-eth011:10:253
853499221,14cyclictest0-21swapper/311:50:153
851499229,9cyclictest21-21ksoftirqd/110:10:151
851499222,6cyclictest21-21ksoftirqd/107:35:131
8514992211,3cyclictest21-21ksoftirqd/112:05:231
851499221,12cyclictest131rcu_preempt11:02:191
8514992210,3cyclictest21-21ksoftirqd/110:05:281
850699228,6cyclictest12-21ksoftirqd/009:35:160
850699225,6cyclictest12-21ksoftirqd/007:20:170
850699223,14cyclictest131rcu_preempt11:15:170
850699223,14cyclictest0-21swapper/009:30:120
850699221,5cyclictest17286-21kthreadcore09:45:270
850699221,17cyclictest12726-21apt-get08:55:130
850699220,10cyclictest5216-21cpuspeed07:40:170
853499215,10cyclictest0-21swapper/310:50:153
853499212,14cyclictest29566irq/58-eth011:46:433
853499212,14cyclictest0-21swapper/310:05:163
851499219,9cyclictest21-21ksoftirqd/108:35:181
851499217,6cyclictest131rcu_preempt09:59:561
851499217,6cyclictest131rcu_preempt09:29:351
851499217,6cyclictest131rcu_preempt09:29:341
851499217,11cyclictest21-21ksoftirqd/110:00:151
851499215,13cyclictest21-21ksoftirqd/107:15:181
8514992113,5cyclictest21-21ksoftirqd/110:40:221
8514992111,3cyclictest21-21ksoftirqd/107:10:021
851499211,13cyclictest131rcu_preempt11:43:581
851499211,13cyclictest131rcu_preempt11:39:471
8514992110,3cyclictest21-21ksoftirqd/109:40:221
850699218,4cyclictest8397-21kworker/0:207:10:170
850699214,3cyclictest131rcu_preempt08:40:210
850699212,3cyclictest131rcu_preempt11:55:200
850699211,7cyclictest1263966ssh12:30:300
850699211,5cyclictest0-21swapper/011:44:340
850699211,5cyclictest0-21swapper/009:05:350
850699211,4cyclictest0-21swapper/007:45:240
850699211,2cyclictest131rcu_preempt11:50:190
850699211,16cyclictest1763-21apt-get08:10:130
8506992111,5cyclictest131rcu_preempt10:45:150
850699211,13cyclictest0-21swapper/011:35:140
850699211,10cyclictest16953-21/usr/sbin/munin10:30:150
853499205,9cyclictest29566irq/58-eth007:55:143
851499205,8cyclictest0-21swapper/112:23:111
851499205,12cyclictest24541-21ssh09:30:151
851499201,8cyclictest21901-21/usr/sbin/munin11:15:201
8514992017,2cyclictest0-21swapper/109:10:141
8514992013,4cyclictest21-21ksoftirqd/107:45:011
851499201,2cyclictest131rcu_preempt10:34:261
851499201,2cyclictest131rcu_preempt09:49:041
851499201,14cyclictest29735-21kworker/u8:111:05:161
8514992010,2cyclictest21-21ksoftirqd/111:46:041
8514992010,2cyclictest21-21ksoftirqd/108:50:211
850699201,9cyclictest31378-21/usr/sbin/munin11:00:170
850699201,9cyclictest21008-21/usr/sbin/munin08:30:230
850699201,7cyclictest131rcu_preempt10:00:170
850699201,6cyclictest8475-21cat11:25:290
850699201,5cyclictest18036-21tr07:50:260
850699201,5cyclictest0-21swapper/009:50:240
850699201,4cyclictest0-21swapper/007:35:170
850699201,15cyclictest6699-21apt-get09:40:140
850699201,13cyclictest0-21swapper/011:20:240
850699200,11cyclictest131rcu_preempt09:29:470
850699200,11cyclictest131rcu_preempt09:29:470
853499193,11cyclictest0-21swapper/310:32:433
853499192,10cyclictest0-21swapper/308:10:143
853499191,12cyclictest0-21swapper/307:30:233
853499191,12cyclictest0-21swapper/307:20:263
8534991910,6cyclictest0-21swapper/310:56:383
851499192,9cyclictest21-21ksoftirqd/109:16:371
8514991913,3cyclictest21-21ksoftirqd/109:20:121
8514991913,3cyclictest21-21ksoftirqd/108:55:171
851499191,2cyclictest131rcu_preempt12:00:051
851499190,2cyclictest131rcu_preempt08:30:201
850699195,9cyclictest8496-21cyclictest11:10:130
850699195,3cyclictest31118-21kworker/0:012:05:190
850699193,4cyclictest19157-21kthreadcore08:25:230
850699193,10cyclictest29181-21cut09:55:200
850699192,3cyclictest131rcu_preempt08:35:480
850699192,2cyclictest131rcu_preempt08:20:140
850699191,9cyclictest131rcu_preempt08:15:250
850699191,5cyclictest0-21swapper/010:10:230
850699191,3cyclictest0-21swapper/010:44:250
850699191,2cyclictest131rcu_preempt10:50:130
850699190,7cyclictest131rcu_preempt07:25:150
853499185,9cyclictest0-21swapper/312:00:133
853499183,10cyclictest0-21swapper/310:25:433
853499182,9cyclictest29566irq/58-eth010:15:163
853499182,11cyclictest0-21swapper/308:25:153
853499182,10cyclictest0-21swapper/309:50:083
853499182,10cyclictest0-21swapper/307:15:143
853499181,13cyclictest0-21swapper/309:20:293
853499181,13cyclictest0-21swapper/308:05:303
851499184,4cyclictest18128-21tr08:25:211
851499184,3cyclictest131rcu_preempt08:15:211
851499181,8cyclictest10936-21irqcore11:50:181
851499181,2cyclictest131rcu_preempt07:10:181
8514991812,3cyclictest29026-21/usr/sbin/munin11:20:201
8514991812,3cyclictest21-21ksoftirqd/108:20:191
8514991811,4cyclictest21-21ksoftirqd/108:05:271
8514991811,4cyclictest21-21ksoftirqd/107:45:231
850699187,3cyclictest2825-21kworker/0:008:45:150
850699186,10cyclictest24282-21cron08:00:010
850699183,5cyclictest131rcu_preempt08:05:200
850699181,4cyclictest131rcu_preempt10:05:240
8506991812,4cyclictest0-21swapper/007:30:130
853499179,6cyclictest0-21swapper/310:02:163
853499179,6cyclictest0-21swapper/309:26:483
853499179,6cyclictest0-21swapper/309:26:473
853499172,10cyclictest0-21swapper/309:05:143
853499172,10cyclictest0-21swapper/308:40:163
853499172,10cyclictest0-21swapper/308:35:173
853499172,10cyclictest0-21swapper/307:40:163
853499172,10cyclictest0-21swapper/307:35:153
853499172,10cyclictest0-21swapper/307:25:153
853499171,11cyclictest0-21swapper/307:10:153
851499172,2cyclictest131rcu_preempt08:10:461
8514991711,3cyclictest21-21ksoftirqd/107:55:211
850699173,4cyclictest8561-21latency_hist07:10:010
850699171,3cyclictest131rcu_preempt08:50:120
853499162,10cyclictest0-21swapper/308:15:153
853499161,9cyclictest0-21swapper/307:45:333
851499162,11cyclictest614-21systemd-logind07:50:171
851499161,3cyclictest18968-21kthreadcore07:20:221
851499161,10cyclictest31185-21irqrtprio08:40:201
853499151,8cyclictest0-21swapper/308:30:173
853499145,5cyclictest0-21swapper/308:55:123
853499141,10cyclictest0-21swapper/309:00:163
853499133,7cyclictest0-21swapper/308:45:193
853499131,10cyclictest0-21swapper/308:20:213
853499122,7cyclictest0-21swapper/308:50:203
853499101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional