You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 19:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Sun Dec 07, 2025 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
15328996625,6cyclictest131rcu_preempt12:15:142
1532899553,9cyclictest0-21swapper/210:05:162
1532899553,9cyclictest0-21swapper/210:05:162
1531999531,43cyclictest26104-21ssh11:10:141
1532899504,23cyclictest20295-21sshd07:15:142
15342994937,9cyclictest29566irq/58-eth012:05:133
1532899471,28cyclictest3657-21ssh12:35:122
1531999463,29cyclictest9786-21sshd12:20:141
1530899453,37cyclictest9612-21ssh09:15:130
1530899442,7cyclictest131rcu_preempt12:20:120
1532899416,25cyclictest27-21ksoftirqd/211:00:142
1534299391,19cyclictest29566irq/58-eth011:35:143
1532899372,18cyclictest27304-21sh12:30:142
15328993715,6cyclictest4382-21ssh10:35:162
15319993722,8cyclictest0-21swapper/109:15:141
15319993720,11cyclictest31328-21sshd08:00:141
1530899375,19cyclictest1829-21ssh09:10:140
1530899375,19cyclictest1829-21ssh09:10:140
1534299364,19cyclictest29566irq/58-eth010:00:143
1534299362,22cyclictest29566irq/58-eth009:25:143
1534299353,22cyclictest29566irq/58-eth010:45:133
1532899349,19cyclictest27-21ksoftirqd/207:35:142
15328993410,14cyclictest31329-21sshd08:00:142
1534299336,21cyclictest29566irq/58-eth008:00:143
1534299331,18cyclictest29566irq/58-eth010:05:163
1534299331,18cyclictest29566irq/58-eth010:05:163
15342993311,16cyclictest29566irq/58-eth011:15:143
15328993315,13cyclictest27-21ksoftirqd/212:00:152
1531999332,14cyclictest1470-21diskmemload12:25:141
1532899323,22cyclictest0-21swapper/209:45:162
1531999322,23cyclictest506-21sshd11:35:141
1531999322,23cyclictest13476-21ssh11:40:481
1530899311,27cyclictest1479-21ssh12:15:140
1532899303,19cyclictest26346irq/40-dwc2_hso08:55:132
15328993013,14cyclictest15286-21cron09:40:012
1531999303,20cyclictest12016-21sshd07:40:141
1530899302,23cyclictest10308-21sshd11:00:140
1530899302,20cyclictest29944-21ssh10:10:130
1534299292,17cyclictest29566irq/58-eth012:30:143
1534299292,16cyclictest29566irq/58-eth009:40:113
1534299292,14cyclictest29566irq/58-eth007:20:143
1532899294,16cyclictest26346irq/40-dwc2_hso10:50:142
1532899292,17cyclictest26346irq/40-dwc2_hso11:05:142
1532899292,15cyclictest26346irq/40-dwc2_hso07:20:162
1532899292,11cyclictest6924-21ssh11:16:152
15328992914,7cyclictest1124-21sshd11:55:142
1531999292,20cyclictest19233-21ntp_192.168.11409:40:221
1531999291,20cyclictest25761-21ssh12:10:131
1531999291,18cyclictest22113-21ssh10:05:161
1531999291,18cyclictest22113-21ssh10:05:161
1530899292,25cyclictest3671-21ssh12:35:130
1530899291,25cyclictest24908-21ssh09:25:140
1534299283,16cyclictest0-21swapper/310:10:133
1534299281,17cyclictest29566irq/58-eth009:30:153
1532899283,18cyclictest707-21rs:main2
1532899282,11cyclictest0-21swapper/211:28:142
1532899281,13cyclictest73-21kswapd010:20:162
1532899281,13cyclictest73-21kswapd010:20:162
1531999280,21cyclictest5401-21ssh10:15:121
1530899281,16cyclictest7811-21ssh09:35:130
1532899275,15cyclictest26346irq/40-dwc2_hso11:20:122
1532899275,15cyclictest26346irq/40-dwc2_hso11:20:122
1532899275,12cyclictest26346irq/40-dwc2_hso10:00:132
1532899273,20cyclictest27-21ksoftirqd/211:50:162
1532899272,20cyclictest5509-21ssh10:15:152
1532899271,7cyclictest24906-21sshd09:25:142
15328992714,6cyclictest1168-21ssh11:14:472
15328992711,10cyclictest0-21swapper/210:25:172
1530899272,19cyclictest8554-21diskstats11:40:140
1530899272,18cyclictest30868-21sshd09:50:140
1534299263,15cyclictest0-21swapper/307:15:143
1534299263,12cyclictest0-21swapper/312:35:123
1532899264,13cyclictest12092-21ssh10:40:132
1532899261,14cyclictest18405-21sshd12:25:142
1532899261,14cyclictest1-21systemd07:40:142
1530899262,20cyclictest25194-21rm11:50:160
1530899261,19cyclictest4383-21ssh10:35:150
1534299253,16cyclictest29566irq/58-eth009:50:133
1534299253,14cyclictest0-21swapper/311:40:483
1532899251,9cyclictest0-21swapper/211:40:132
15328992514,8cyclictest15781-21grep09:40:152
1532899251,15cyclictest9797-21grep09:15:162
1531999253,15cyclictest21-21ksoftirqd/107:25:131
1531999251,19cyclictest8520-21apt-get08:10:141
1531999251,19cyclictest8520-21apt-get08:10:141
1530899258,11cyclictest12-21ksoftirqd/012:25:120
1534299249,8cyclictest29566irq/58-eth012:15:263
15328992414,4cyclictest27-21ksoftirqd/208:43:362
15328992412,9cyclictest8877-21ssh09:14:242
15328992412,9cyclictest8877-21ssh09:14:242
15328992410,11cyclictest0-21swapper/210:45:252
1531999243,15cyclictest25032-21ssh11:30:141
1531999242,17cyclictest1470-21diskmemload12:32:141
1530899242,17cyclictest27358-21ssh10:50:150
1530899241,18cyclictest12102-21ssh10:40:140
15308992411,11cyclictest12-21ksoftirqd/009:30:160
1532899236,6cyclictest17469-21ssh12:05:132
1532899234,15cyclictest261rcuc/209:55:142
1532899232,7cyclictest23353-21kworker/u8:109:30:152
1532899232,6cyclictest0-21swapper/211:39:142
1532899232,16cyclictest26346irq/40-dwc2_hso08:05:132
1532899232,13cyclictest0-21swapper/209:05:142
1532899232,13cyclictest0-21swapper/209:05:142
15328992314,4cyclictest27-21ksoftirqd/211:33:102
1531999231,19cyclictest4349-21find10:35:141
1531999231,17cyclictest1470-21diskmemload10:00:151
1531999231,15cyclictest15478-21sshd07:10:141
1531999231,12cyclictest1648066ssh11:24:121
1531999231,12cyclictest1648066ssh11:24:121
1530899232,15cyclictest24576-21kworker/u8:111:32:140
1530899231,18cyclictest12333-21grep07:40:180
1534299229,8cyclictest0-21swapper/311:00:153
1534299222,12cyclictest29566irq/58-eth008:30:133
1532899229,6cyclictest2112-21cron07:30:002
1532899225,12cyclictest26346irq/40-dwc2_hso09:20:152
1532899222,6cyclictest9772-21ssh12:20:132
1532899221,9cyclictest26536-21grep08:25:302
1532899221,12cyclictest131rcu_preempt08:50:302
1531999221,8cyclictest9950-21munin-run11:00:011
1531999221,18cyclictest20049-21uniq10:45:161
1531999221,13cyclictest17484-21ssh12:05:141
1530899227,3cyclictest131rcu_preempt08:45:010
1530899221,18cyclictest19303-21kthreadcore11:25:230
1530899221,18cyclictest18328-21uniq08:20:200
1530899221,17cyclictest13526-21sh10:20:160
1530899221,17cyclictest13526-21sh10:20:160
1530899220,18cyclictest0-21swapper/012:10:130
1534299213,12cyclictest0-21swapper/307:55:153
1534299213,11cyclictest29566irq/58-eth012:10:133
1534299211,16cyclictest29566irq/58-eth012:20:133
1532899219,3cyclictest0-21swapper/208:30:132
1532899217,5cyclictest29790-21grep12:10:302
1532899215,12cyclictest26346irq/40-dwc2_hso08:45:122
1532899213,15cyclictest18356-21fschecks_time08:20:202
1532899212,6cyclictest0-21swapper/208:10:142
1532899212,6cyclictest0-21swapper/208:10:142
1532899212,16cyclictest0-21swapper/211:45:042
1532899212,11cyclictest6842-21grep10:55:322
1532899211,14cyclictest0-21swapper/207:45:132
1532899211,10cyclictest26390-21latency_hist07:55:012
1532899211,10cyclictest1641-21head10:10:302
1532899210,5cyclictest27761-21ls09:00:212
15319992111,7cyclictest167-21systemd-network11:47:441
1530899214,14cyclictest4213-21sed08:05:190
1530899212,14cyclictest629-21Async14:00:480
1530899212,14cyclictest505-21ssh11:35:140
1530899212,14cyclictest17466-21apt-get12:05:130
1530899212,12cyclictest842-21munin-node11:20:140
1530899212,12cyclictest842-21munin-node11:20:140
1530899211,18cyclictest5668-21idleruntime10:15:160
1530899211,17cyclictest7235-21ls07:35:150
1530899211,14cyclictest0-21swapper/009:00:120
1534299202,12cyclictest29566irq/58-eth010:20:163
1534299202,12cyclictest29566irq/58-eth010:20:153
15342992012,6cyclictest0-21swapper/310:55:173
1532899204,8cyclictest3963-21kthreadcore07:30:212
1532899201,5cyclictest0-21swapper/208:15:132
1532899201,10cyclictest31321-21kthreadcore10:30:202
1532899200,17cyclictest27904-21kthreadcore07:55:202
1531999202,14cyclictest26629-21ssh09:45:271
1531999201,5cyclictest9146-21munin-run11:20:011
15319992012,6cyclictest24552-21cron09:25:011
1531999201,15cyclictest13913-21ssh11:00:251
1531999201,13cyclictest28822-21sh11:50:291
1531999200,16cyclictest19804-21ls08:55:161
1530899206,10cyclictest12-21ksoftirqd/009:55:140
1530899202,13cyclictest4745-21rm11:55:260
15308992011,6cyclictest31351-21grep12:30:310
1530899201,15cyclictest18116-21sshd11:05:130
1530899201,15cyclictest16905-21sshd11:45:140
1530899201,14cyclictest1-21systemd07:10:140
1530899200,18cyclictest26875-21ls09:45:310
1534299193,11cyclictest0-21swapper/312:29:143
1534299193,11cyclictest0-21swapper/311:25:443
1534299192,11cyclictest0-21swapper/309:45:153
1534299192,11cyclictest0-21swapper/309:15:053
1534299192,11cyclictest0-21swapper/308:45:143
1534299192,11cyclictest0-21swapper/308:40:173
1534299192,10cyclictest0-21swapper/309:00:063
1534299191,14cyclictest0-21swapper/311:45:163
1534299191,13cyclictest0-21swapper/311:50:163
1534299191,13cyclictest0-21swapper/308:05:153
1534299191,12cyclictest0-21swapper/309:05:323
1534299191,12cyclictest0-21swapper/309:05:323
1534299191,12cyclictest0-21swapper/307:45:013
1534299191,11cyclictest0-21swapper/310:55:023
1534299191,11cyclictest0-21swapper/307:25:283
1534299191,10cyclictest29566irq/58-eth009:55:153
1532899191,9cyclictest32712-21/usr/sbin/munin08:35:162
1532899191,9cyclictest15480-21cpu07:10:142
1531999199,7cyclictest22801-21cat11:05:321
1531999196,3cyclictest131rcu_preempt12:20:021
1531999191,9cyclictest10197-21wc09:55:301
1531999191,8cyclictest7459-21awk09:35:011
15319991911,6cyclictest0-21swapper/108:25:311
15319991911,5cyclictest0-21swapper/108:50:271
1531999191,13cyclictest0-21swapper/112:35:581
1530899192,11cyclictest19240-21ssh09:40:220
1530899190,13cyclictest6567-21switchtime10:55:310
1530899190,12cyclictest19326-21latency_hist08:55:020
1530899190,10cyclictest4053-21tr07:30:200
1534299183,11cyclictest0-21swapper/307:35:143
1534299183,10cyclictest0-21swapper/311:20:153
1534299183,10cyclictest0-21swapper/311:20:143
1534299182,11cyclictest0-21swapper/310:30:123
1534299182,10cyclictest0-21swapper/312:00:163
1534299182,10cyclictest0-21swapper/308:55:143
1534299182,10cyclictest0-21swapper/308:15:143
1534299182,10cyclictest0-21swapper/307:30:143
15342991812,4cyclictest0-21swapper/311:30:253
1534299181,11cyclictest0-21swapper/309:10:153
1534299181,11cyclictest0-21swapper/309:10:153
15342991810,5cyclictest0-21swapper/311:55:253
15342991810,5cyclictest0-21swapper/309:21:243
1531999184,11cyclictest8830-21cut07:35:201
1531999183,11cyclictest0-21swapper/110:50:151
1531999182,6cyclictest9344-21df_inode12:00:171
1531999182,10cyclictest0-21swapper/107:55:141
15319991812,3cyclictest8685-21mailstats08:40:421
15319991811,5cyclictest29942-21sshd10:10:131
1531999181,15cyclictest14809-21kthreadcore08:15:201
1531999181,13cyclictest20959-21apt-get10:25:051
1531999181,11cyclictest13488-21find10:20:141
1531999181,11cyclictest13488-21find10:20:141
15319991810,5cyclictest0-21swapper/111:25:191
1531999180,9cyclictest15765-21ntp_states10:40:271
1531999180,13cyclictest29300-21uniq10:30:141
1530899189,6cyclictest2725-21unixbench_multi08:00:290
1530899188,8cyclictest26445-21cat10:05:350
1530899188,8cyclictest26445-21cat10:05:340
1530899188,6cyclictest14680-21cat08:50:010
1530899185,8cyclictest12-21ksoftirqd/007:15:150
1530899185,11cyclictest25344-21timerwakeupswit10:25:330
1530899183,5cyclictest14251-21diskstats10:00:160
1530899182,12cyclictest26090-21apt-get11:10:150
1530899181,13cyclictest17525-21fschecks_count09:20:180
15308991810,5cyclictest19988-21df10:45:140
1534299179,6cyclictest0-21swapper/311:11:213
1534299179,6cyclictest0-21swapper/311:08:433
1534299179,6cyclictest0-21swapper/310:35:293
1534299179,5cyclictest0-21swapper/310:25:353
1534299178,6cyclictest0-21swapper/309:37:403
1534299172,9cyclictest0-21swapper/308:10:163
1534299172,9cyclictest0-21swapper/308:10:153
1534299172,6cyclictest0-21swapper/310:40:143
1534299172,10cyclictest0-21swapper/308:35:153
1534299172,10cyclictest0-21swapper/307:10:143
1532899171,8cyclictest15291-21cyclictest07:10:022
1531999176,9cyclictest5120-21cut09:10:241
1531999176,9cyclictest5120-21cut09:10:231
1531999172,12cyclictest26159-21kthreadcore07:20:211
1531999172,11cyclictest30844-21apt-get09:50:141
1531999171,7cyclictest986-21/usr/sbin/munin11:55:241
1531999171,6cyclictest11370-21ntp_192.168.11409:35:261
1531999171,13cyclictest24970-21latency_hist07:20:021
15319991710,5cyclictest10017-21sshd08:45:141
15319991710,4cyclictest1672-21latency_hist09:10:011
15319991710,3cyclictest0-21swapper/108:20:171
1530899179,5cyclictest0-21swapper/007:20:180
1530899178,6cyclictest1798-21diskstats11:15:160
1530899177,7cyclictest2750-21kthreadcore08:35:220
1530899172,12cyclictest22994-21ntpq08:55:240
1530899172,11cyclictest0-21swapper/009:05:140
1530899172,11cyclictest0-21swapper/009:05:140
1530899172,11cyclictest0-21swapper/008:30:130
1530899172,11cyclictest0-21swapper/008:15:140
1530899171,8cyclictest25493-21unixbench_multi07:50:280
1530899171,12cyclictest8523-21sshd08:10:140
1530899171,12cyclictest8523-21sshd08:10:140
1530899171,12cyclictest4001-21cron10:35:000
1530899171,11cyclictest23819-21kthreadcore08:25:220
1534299169,5cyclictest0-21swapper/310:16:563
1531999168,6cyclictest6905-21sshd07:32:141
1531999165,9cyclictest25342-21kthreadcore09:00:181
1531999163,9cyclictest818-21irqrtprio08:35:191
1531999162,10cyclictest0-21swapper/108:05:141
1531999162,10cyclictest0-21swapper/107:50:141
1531999161,7cyclictest31582-21kthreadcore09:05:211
1531999161,12cyclictest27599-21apt-get08:30:031
1531999161,12cyclictest19649-21kthreadcore07:45:211
1531999161,11cyclictest25078-21fschecks_count09:25:181
1530899164,9cyclictest17480-21kernelversion07:45:180
1530899161,12cyclictest26483-21cpu07:55:150
1530899160,14cyclictest30334-21irqcore07:25:180
15342991510,3cyclictest33-21ksoftirqd/308:25:303
1534299141,11cyclictest0-21swapper/307:55:003
1534299141,10cyclictest0-21swapper/308:50:153
1534299141,10cyclictest0-21swapper/308:22:143
1531999141,10cyclictest15351-21latency_hist07:10:021
1530899142,10cyclictest15234-21latency_hist07:10:020
1534299131,10cyclictest0-21swapper/307:45:233
1534299111,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional