You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-21 - 09:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu May 21, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
55099734,44cyclictest6652-21rm00:20:112
55099673,32cyclictest4953-21ssh21:20:132
55099673,32cyclictest4953-21ssh21:20:132
542996520,8cyclictest131rcu_preempt23:15:141
54299645,47cyclictest15030-21ssh22:55:111
55099625,45cyclictest7476-21ssh22:15:122
562995834,10cyclictest35966irq/58-eth022:30:123
55099551,36cyclictest930-21snmpd23:35:122
562995436,13cyclictest35966irq/58-eth021:30:103
55099542,32cyclictest27957-21ssh23:20:112
542994912,24cyclictest21-21ksoftirqd/123:30:131
562994812,27cyclictest35966irq/58-eth022:20:123
54299484,7cyclictest131rcu_preempt20:40:121
550994621,17cyclictest131rcu_preempt20:20:132
55099460,38cyclictest9828-21ssh23:10:132
54299454,30cyclictest131rcu_preempt19:45:111
550994323,11cyclictest27-21ksoftirqd/219:15:112
550994323,11cyclictest27-21ksoftirqd/219:15:112
55099431,34cyclictest2159-21ssh22:30:122
53199411,37cyclictest20315-21sh22:40:130
55099403,31cyclictest19249-21ssh21:10:142
54299402,10cyclictest21-21ksoftirqd/121:15:121
53199402,33cyclictest29427-21ssh22:45:130
56299385,15cyclictest35966irq/58-eth020:00:133
550993821,12cyclictest271ksoftirqd/223:25:122
55099353,26cyclictest12702-21ssh22:00:132
55099351,22cyclictest3601-21ssh21:55:122
531993527,3cyclictest30832-21apt-get20:15:130
56299341,19cyclictest35966irq/58-eth019:10:083
55099347,18cyclictest27346irq/40-dwc2_hso20:25:122
54299344,10cyclictest21-21ksoftirqd/123:40:121
53199349,18cyclictest0-21swapper/022:20:110
531993426,3cyclictest26022-21apt-get20:10:120
531993422,5cyclictest13959-21apt-get21:05:030
531993421,8cyclictest30035-21apt-get00:15:120
56299331,18cyclictest35966irq/58-eth000:05:133
55099333,23cyclictest31669-21ssh23:40:112
550993315,10cyclictest521-21cyclictest23:55:122
53199336,12cyclictest9447-21ssh21:56:390
53199335,10cyclictest29400-21kthreadcore20:45:200
53199333,25cyclictest8312-21ssh23:45:130
53199333,25cyclictest8312-21ssh23:45:120
55099324,5cyclictest131rcu_preempt20:05:152
55099324,5cyclictest131rcu_preempt20:05:152
550993217,9cyclictest27-21ksoftirqd/220:55:122
550993217,9cyclictest27-21ksoftirqd/220:55:122
531993220,8cyclictest722-21apt-get19:10:070
531993215,12cyclictest31679-21ssh23:40:120
562993113,12cyclictest35966irq/58-eth021:45:143
55099316,16cyclictest27346irq/40-dwc2_hso21:15:142
55099315,15cyclictest27346irq/40-dwc2_hso23:05:112
55099314,16cyclictest27-21ksoftirqd/219:25:132
55099312,22cyclictest17395-21ssh23:50:142
54299314,23cyclictest21-21ksoftirqd/121:35:121
54299313,26cyclictest16512-21ssh22:20:121
54299312,24cyclictest21019-21ssh00:10:111
54299312,22cyclictest8830-21sh21:40:121
53199316,10cyclictest14945-21ssh23:10:510
53199315,9cyclictest13018-21switchtime21:00:290
53199315,16cyclictest30864-21ssh23:02:500
53199315,16cyclictest30864-21ssh23:02:490
53199315,10cyclictest21111-21ssh23:33:400
53199314,10cyclictest7588-21ssh23:07:550
53199311,11cyclictest24932-21ssh23:53:440
55099303,17cyclictest27346irq/40-dwc2_hso19:45:132
54299303,9cyclictest21-21ksoftirqd/123:10:131
53199305,9cyclictest10928-21ls22:15:220
53199304,8cyclictest26536-21ssh00:11:160
531993022,3cyclictest4552-21apt-get23:25:130
53199302,11cyclictest16686-21cat00:25:180
531993018,9cyclictest30892-21ssh22:10:120
53199301,20cyclictest12062-21ssh00:05:150
56299291,17cyclictest35966irq/58-eth022:55:133
56299291,14cyclictest35966irq/58-eth000:00:133
55099295,13cyclictest27346irq/40-dwc2_hso20:40:132
54299292,5cyclictest131rcu_preempt21:00:131
54299292,19cyclictest7492-21sh22:15:121
542992916,8cyclictest14002-21ssh21:25:121
53199298,17cyclictest12-21ksoftirqd/019:50:100
53199295,10cyclictest12416-21ssh22:52:220
53199294,16cyclictest25572-21ssh22:25:130
53199294,10cyclictest12766-21ssh21:24:010
53199294,10cyclictest12766-21ssh21:24:000
53199293,14cyclictest12-21ksoftirqd/023:35:120
53199292,9cyclictest16307-21ntp_192.168.11422:00:230
53199292,21cyclictest11221-21ssh22:35:140
531992921,4cyclictest3334-21apt-get20:20:130
531992920,4cyclictest521-21cyclictest20:50:110
56299289,11cyclictest35966irq/58-eth000:20:113
56299281,21cyclictest35966irq/58-eth022:10:133
55099282,21cyclictest6601-21ssh00:36:102
55099282,20cyclictest0-21swapper/221:05:142
55099282,16cyclictest27346irq/40-dwc2_hso19:30:122
55099282,11cyclictest5529-21ssh00:19:102
550992811,13cyclictest22367-21cron20:40:002
54299283,18cyclictest673-21ssh23:05:131
53199285,7cyclictest6724-21ssh00:00:250
53199285,14cyclictest4780-21netstat00:35:220
53199280,11cyclictest29212-21usage_EthcTask19:35:290
56299271,12cyclictest35966irq/58-eth021:10:143
562992710,11cyclictest0-21swapper/300:27:103
55099275,14cyclictest27346irq/40-dwc2_hso19:40:132
550992713,11cyclictest29426-21ssh22:45:132
54299272,19cyclictest32055-21ssh00:33:381
53199277,14cyclictest17855-21ssh21:45:120
53199274,9cyclictest29581-21grep19:40:140
53199274,10cyclictest2377-21ssh23:22:520
53199273,9cyclictest29301-21ssh22:08:370
53199273,8cyclictest8823-21apt21:40:110
53199273,8cyclictest27922-21ssh21:15:010
53199272,10cyclictest27212-21/usr/sbin/munin21:50:150
531992719,3cyclictest17824-21apt-get20:35:120
53199271,8cyclictest0-21swapper/000:31:470
53199271,12cyclictest21226-21rm21:28:160
53199270,11cyclictest29495-21ssh21:32:210
56299268,12cyclictest0-21swapper/323:25:133
55099262,16cyclictest27346irq/40-dwc2_hso19:35:132
55099261,15cyclictest167-21systemd-network19:55:142
55099260,19cyclictest13959-21find21:25:092
54299262,19cyclictest21872-21ssh00:27:111
53199264,6cyclictest15919-21usage_EthcTask19:55:300
53199262,9cyclictest944-21ssh21:15:480
531992619,4cyclictest1969-21apt-get19:45:110
531992619,3cyclictest26412-21apt-get23:55:130
531992613,11cyclictest0-21swapper/020:00:130
531992612,11cyclictest15021-21apt-get22:55:120
53199260,10cyclictest8685-21usage_Anybus20:55:290
53199260,10cyclictest8685-21usage_Anybus20:55:280
562992516,7cyclictest0-21swapper/322:25:153
562992513,8cyclictest35966irq/58-eth023:45:133
562992513,8cyclictest35966irq/58-eth023:45:133
55099255,14cyclictest27346irq/40-dwc2_hso20:50:122
55099254,13cyclictest27346irq/40-dwc2_hso22:10:122
55099253,17cyclictest0-21swapper/200:33:372
55099252,17cyclictest20322-21ssh22:40:122
55099251,12cyclictest15193-21ls22:35:292
53199259,11cyclictest21169-21apt-get20:05:140
53199259,11cyclictest21169-21apt-get20:05:130
53199255,4cyclictest12-21ksoftirqd/020:30:120
53199255,4cyclictest12-21ksoftirqd/020:30:120
53199250,10cyclictest10588-21ssh22:34:450
55099243,15cyclictest27346irq/40-dwc2_hso23:30:132
55099242,16cyclictest0-21swapper/222:05:052
550992415,7cyclictest29745-21ls20:10:282
55099241,18cyclictest15035-21ssh22:55:122
54299241,5cyclictest3086-21ssh00:00:131
54299241,18cyclictest4563-21ssh23:25:121
542992411,11cyclictest0-21swapper/121:32:381
53199243,8cyclictest6439-21kthreadcore19:15:180
53199243,8cyclictest6439-21kthreadcore19:15:170
53199241,7cyclictest12-21ksoftirqd/021:35:160
53199240,10cyclictest22635-21cpu20:40:130
53199240,10cyclictest18914-21ssh23:15:130
56299233,14cyclictest0-21swapper/300:33:383
562992311,9cyclictest0-21swapper/300:15:343
55099232,16cyclictest0-21swapper/200:26:102
55099232,16cyclictest0-21swapper/200:14:102
550992311,9cyclictest548-21ssh22:28:372
55099230,7cyclictest3210-21latency_hist21:55:012
54299232,10cyclictest19830-21apt-get22:40:031
54299231,7cyclictest27892-21munin-run21:15:011
542992314,6cyclictest17390-21ssh23:50:121
542992311,10cyclictest0-21swapper/100:18:411
542992310,4cyclictest21-21ksoftirqd/119:30:021
53199232,8cyclictest6612-21/usr/sbin/munin00:20:260
53199230,8cyclictest13824-21ntpq19:20:250
562992215,5cyclictest0-21swapper/322:15:113
562992214,5cyclictest0-21swapper/322:40:243
562992212,8cyclictest0-21swapper/322:37:553
55099227,6cyclictest367-21latency_hist23:05:012
55099227,6cyclictest367-21latency_hist23:05:012
55099221,8cyclictest8762-21irqcore23:45:182
55099221,8cyclictest8762-21irqcore23:45:182
550992211,9cyclictest6831-21ssh21:38:082
550992211,9cyclictest0-21swapper/200:00:112
55099221,11cyclictest8022-21kthreadcore22:50:202
55099221,10cyclictest599-21latency_hist19:10:022
55099221,10cyclictest21713-21swap21:45:282
550992210,9cyclictest15214-1kworker/u9:022:20:192
54299228,3cyclictest21-21ksoftirqd/100:24:001
542992216,4cyclictest0-21swapper/120:35:251
542992213,7cyclictest21-21ksoftirqd/122:05:111
54299221,15cyclictest12059-21sh00:05:131
542992211,3cyclictest21-21ksoftirqd/119:50:161
542992210,3cyclictest21-21ksoftirqd/121:55:121
53199227,11cyclictest0-21swapper/019:25:120
531992215,4cyclictest0-21swapper/020:30:020
56299216,11cyclictest35966irq/58-eth023:05:133
56299214,13cyclictest35966irq/58-eth019:50:113
562992115,4cyclictest0-21swapper/323:56:193
562992115,4cyclictest0-21swapper/321:35:303
562992115,4cyclictest0-21swapper/321:15:263
562992114,5cyclictest0-21swapper/323:50:053
562992114,5cyclictest0-21swapper/321:20:233
562992114,5cyclictest0-21swapper/321:20:223
562992113,5cyclictest0-21swapper/323:36:393
562992112,6cyclictest0-21swapper/322:45:133
56299211,17cyclictest35966irq/58-eth022:50:123
56299211,15cyclictest35966irq/58-eth023:40:103
562992110,8cyclictest0-21swapper/321:52:053
55099217,11cyclictest18278-21kthreadcore20:00:222
55099212,9cyclictest131rcu_preempt19:20:172
55099211,9cyclictest14740-21cut00:05:232
55099211,6cyclictest23872-21irqstats21:30:232
55099211,6cyclictest17481-21awk21:45:012
55099211,10cyclictest19337-21irqcore23:15:182
54299219,9cyclictest21-21ksoftirqd/120:20:141
54299217,4cyclictest131rcu_preempt21:05:181
54299216,3cyclictest131rcu_preempt23:25:001
54299215,2cyclictest131rcu_preempt23:40:021
54299212,8cyclictest30981-21grep21:50:281
54299212,15cyclictest16315-21ssh22:00:241
54299211,7cyclictest21-21ksoftirqd/120:50:111
542992112,6cyclictest27660-21ssh23:00:251
542992112,6cyclictest27660-21ssh23:00:241
54299211,15cyclictest30893-21ssh22:10:131
542992111,2cyclictest21-21ksoftirqd/121:45:121
542992110,3cyclictest21-21ksoftirqd/122:48:341
542992110,3cyclictest21-21ksoftirqd/122:31:521
542992110,3cyclictest21-21ksoftirqd/122:27:141
542992110,2cyclictest21-21ksoftirqd/122:50:141
562992014,4cyclictest0-21swapper/323:20:183
562992014,4cyclictest0-21swapper/323:15:163
562992014,4cyclictest0-21swapper/323:02:393
562992014,4cyclictest0-21swapper/323:02:393
562992014,4cyclictest0-21swapper/322:05:253
562992014,4cyclictest0-21swapper/300:10:173
562992013,5cyclictest0-21swapper/323:10:183
562992013,5cyclictest0-21swapper/321:00:183
562992012,6cyclictest0-21swapper/322:00:103
562992012,6cyclictest0-21swapper/321:56:043
562992012,5cyclictest0-21swapper/321:25:103
562992011,6cyclictest0-21swapper/319:30:243
55099208,9cyclictest0-21swapper/220:30:162
55099208,9cyclictest0-21swapper/220:30:152
55099205,9cyclictest27-21ksoftirqd/220:15:122
54299209,3cyclictest21-21ksoftirqd/123:55:261
54299208,3cyclictest21-21ksoftirqd/120:15:111
54299205,4cyclictest21-21ksoftirqd/120:30:141
54299205,4cyclictest21-21ksoftirqd/120:30:141
54299204,7cyclictest11896-21irqcore19:55:181
542992013,4cyclictest21-21ksoftirqd/119:15:191
542992013,4cyclictest21-21ksoftirqd/119:15:181
54299201,2cyclictest131rcu_preempt00:39:031
54299201,17cyclictest4019-21ls19:10:181
542992011,6cyclictest12961-21ssh23:45:311
542992011,6cyclictest12961-21ssh23:45:311
54299201,15cyclictest8126-21find20:25:101
54299200,17cyclictest14983-21ls19:25:001
53199201,6cyclictest21059-21kthreadcore19:30:180
562991913,4cyclictest0-21swapper/323:30:183
562991912,5cyclictest0-21swapper/321:40:163
56299191,12cyclictest0-21swapper/320:15:163
56299191,11cyclictest35966irq/58-eth020:30:123
56299191,11cyclictest35966irq/58-eth020:30:123
562991910,7cyclictest0-21swapper/300:35:103
55099191,9cyclictest11262-21sed21:00:212
55099191,7cyclictest777-21cpuspeed19:10:102
55099191,14cyclictest8789-21kthreadcore19:50:182
55099191,10cyclictest29435-21kthreadcore20:45:192
54299199,3cyclictest21-21ksoftirqd/121:20:031
54299199,3cyclictest21-21ksoftirqd/121:20:031
54299197,3cyclictest21-21ksoftirqd/119:35:241
53199197,9cyclictest533-21latency_hist19:10:020
562991813,3cyclictest0-21swapper/319:55:203
562991812,4cyclictest0-21swapper/320:45:153
562991812,4cyclictest0-21swapper/320:35:143
562991812,4cyclictest0-21swapper/320:25:203
562991812,4cyclictest0-21swapper/320:20:193
562991812,4cyclictest0-21swapper/320:05:233
562991812,4cyclictest0-21swapper/320:05:233
562991812,4cyclictest0-21swapper/319:45:153
562991812,4cyclictest0-21swapper/319:40:283
562991811,5cyclictest0-21swapper/319:35:113
54299186,9cyclictest28023-21irqrtprio20:45:171
54299181,9cyclictest16341-21/usr/sbin/munin20:00:171
542991814,2cyclictest21-21ksoftirqd/120:55:211
542991814,2cyclictest21-21ksoftirqd/120:55:211
542991813,3cyclictest0-21swapper/122:35:151
542991810,4cyclictest21-21ksoftirqd/119:40:161
562991712,3cyclictest0-21swapper/321:05:193
562991712,3cyclictest0-21swapper/320:10:143
562991711,4cyclictest0-21swapper/320:40:223
562991711,4cyclictest0-21swapper/319:25:263
562991711,4cyclictest0-21swapper/319:20:243
562991711,4cyclictest0-21swapper/319:15:223
562991711,4cyclictest0-21swapper/319:15:223
56299171,10cyclictest0-21swapper/320:55:133
56299171,10cyclictest0-21swapper/320:55:123
54299177,2cyclictest21-21ksoftirqd/120:05:231
54299177,2cyclictest21-21ksoftirqd/120:05:221
542991713,2cyclictest0-21swapper/119:30:141
562991610,4cyclictest0-21swapper/320:50:163
54299162,9cyclictest0-21swapper/120:10:041
54299151,11cyclictest655-21latency_hist19:10:011
56299101,3cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional