You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-06 - 03:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Sep 06, 2026 00:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
336899725,18cyclictest12-21ksoftirqd/000:25:150
338899644,45cyclictest16658-21ssh23:30:122
336899563,43cyclictest31415-21ssh21:15:120
3396995531,16cyclictest35966irq/58-eth000:15:133
337499531,37cyclictest17110-21ssh21:25:131
336899511,40cyclictest22388-21ssh21:10:130
339699485,26cyclictest35966irq/58-eth021:45:133
337499483,36cyclictest24337-21ssh00:10:131
336899483,34cyclictest17143-21ssh21:25:140
336899482,35cyclictest6667-21ssh21:55:130
336899471,33cyclictest3840-21ssh23:05:150
339699463,27cyclictest35966irq/58-eth000:05:143
3374994527,14cyclictest0-21swapper/123:10:141
3374994527,14cyclictest0-21swapper/123:10:131
337499451,34cyclictest9211-21ssh22:50:181
3374994513,21cyclictest21-21ksoftirqd/119:50:141
336899452,30cyclictest19811-21ssh22:20:130
3374994415,20cyclictest21-21ksoftirqd/100:05:141
3368994316,19cyclictest0-21swapper/022:25:140
338899411,23cyclictest26296-21ssh21:30:172
3368994115,3cyclictest131rcu_preempt22:05:120
3388993911,22cyclictest27-21ksoftirqd/221:10:132
3368993718,14cyclictest12-21ksoftirqd/023:25:120
3368993711,21cyclictest12-21ksoftirqd/019:10:140
3396993622,10cyclictest35966irq/58-eth000:10:143
3368993613,20cyclictest12-21ksoftirqd/023:00:130
3368993419,8cyclictest12-21ksoftirqd/023:53:110
339699339,11cyclictest35966irq/58-eth022:25:133
339699335,15cyclictest35966irq/58-eth000:30:183
338899332,19cyclictest9230-21ssh22:50:192
337499331,29cyclictest5533-21ssh22:30:161
3388993227,3cyclictest27-21ksoftirqd/220:25:142
338899322,20cyclictest24342-21ssh00:10:142
338899313,20cyclictest14555-21ssh22:35:152
336899314,18cyclictest0-21swapper/022:35:160
336899313,24cyclictest12-21ksoftirqd/019:20:150
336899311,21cyclictest32612-21ssh22:45:140
3368993110,4cyclictest131rcu_preempt20:45:130
338899306,20cyclictest12852-21ssh23:10:132
338899306,20cyclictest12852-21ssh23:10:132
338899303,17cyclictest27346irq/40-dwc2_hso22:10:162
338899303,17cyclictest27346irq/40-dwc2_hso21:05:152
338899303,17cyclictest27346irq/40-dwc2_hso21:05:142
337499300,16cyclictest11983-21ssh21:40:141
336899304,14cyclictest12-21ksoftirqd/019:30:120
3368992914,8cyclictest0-21swapper/021:40:160
338899282,20cyclictest0-21swapper/219:50:152
3388992815,10cyclictest0-21swapper/223:45:162
3374992821,3cyclictest0-21swapper/120:40:131
3374992815,10cyclictest0-21swapper/123:55:141
336899285,8cyclictest12-21ksoftirqd/000:10:140
336899282,17cyclictest11395-21ssh23:45:110
336899281,22cyclictest25738-21ssh23:35:130
336899281,22cyclictest25738-21ssh23:35:120
338899275,14cyclictest27346irq/40-dwc2_hso20:40:142
338899272,9cyclictest9427-21ssh23:43:092
3388992720,5cyclictest27-21ksoftirqd/220:15:122
338899271,18cyclictest27346irq/40-dwc2_hso00:15:122
336899273,21cyclictest30964-21ssh23:20:150
336899273,20cyclictest10027-21ssh00:20:150
336899272,18cyclictest2405-21ssh23:40:130
336899272,17cyclictest29533-21ssh23:55:120
339699262,16cyclictest35966irq/58-eth022:10:133
3396992618,5cyclictest35966irq/58-eth021:15:133
338899262,20cyclictest4798-21ssh23:22:092
338899262,15cyclictest27-21ksoftirqd/223:25:132
3388992611,12cyclictest0-21swapper/222:55:222
337499268,4cyclictest131rcu_preempt00:20:021
3368992615,6cyclictest23576-21ssh22:40:130
3368992613,7cyclictest0-21swapper/020:40:150
339699258,10cyclictest35966irq/58-eth022:30:163
3396992516,6cyclictest35966irq/58-eth021:50:163
338899256,16cyclictest29534-21ssh23:55:122
337499258,3cyclictest131rcu_preempt22:20:281
336899256,11cyclictest10697-21ssh22:15:130
336899252,6cyclictest12-21ksoftirqd/000:05:140
336899251,17cyclictest21896-21ssh23:15:130
336899251,15cyclictest12-21ksoftirqd/019:25:140
3396992415,4cyclictest35966irq/58-eth022:35:163
338899245,10cyclictest17024-21kthreadcore20:30:202
338899243,17cyclictest0-21swapper/223:19:092
338899242,13cyclictest930-21snmpd21:15:132
338899241,19cyclictest14114-21ssh21:40:232
338899241,14cyclictest10703-21kthreadcore21:20:242
338899241,14cyclictest1038-21kthreadcore22:45:202
338899241,13cyclictest6684-21rm21:55:142
337499242,19cyclictest1714-21ssh22:10:151
336899241,17cyclictest2866-21apt-get20:50:140
339699232,13cyclictest0-21swapper/322:50:183
338899239,11cyclictest22655-21kthreadcore21:45:192
338899238,7cyclictest5971-21ssh00:35:232
338899238,7cyclictest5971-21ssh00:35:222
338899232,19cyclictest23988-21ssh00:10:002
338899232,11cyclictest21056-21kthreadcore19:25:212
338899231,15cyclictest18640-21latency_hist00:25:022
338899231,14cyclictest9543-21cron19:50:002
338899231,13cyclictest9489-21kthreadcore19:15:192
338899231,12cyclictest27413-21kthreadcore22:05:202
338899231,12cyclictest20169-21kthreadcore00:25:202
338899231,11cyclictest1882-21grep20:45:272
3388992310,6cyclictest26290-21ssh22:22:292
337499232,12cyclictest25851-21df_inode23:35:151
337499232,12cyclictest25851-21df_inode23:35:141
336899234,11cyclictest12-21ksoftirqd/019:15:130
336899232,16cyclictest30112-21ssh21:50:170
3368992315,4cyclictest930-21snmpd21:20:150
336899231,17cyclictest28113-21ssh00:30:180
336899231,16cyclictest2876-21sh21:35:140
339699223,13cyclictest0-21swapper/323:10:153
339699223,13cyclictest0-21swapper/323:10:143
338899227,12cyclictest24731-21sed20:05:182
338899224,11cyclictest0-21swapper/220:00:112
338899222,17cyclictest5243-21kthreadcore20:50:242
338899222,17cyclictest16697-21cut22:00:212
338899222,15cyclictest0-21swapper/223:54:092
338899222,12cyclictest30101-21kthreadcore19:35:212
338899222,11cyclictest16522-21grep19:20:242
338899221,9cyclictest28568-21irqcore00:30:232
3388992218,2cyclictest1481-1kworker/u9:223:05:142
338899221,18cyclictest916-21kthreadcore19:40:192
338899221,17cyclictest23540-21/usr/sbin/munin22:40:262
338899221,13cyclictest9192-21kthreadcore20:20:222
338899221,12cyclictest13210-21ps21:00:192
338899221,11cyclictest2830-21find21:35:122
338899221,11cyclictest25863-21sh23:35:152
338899221,11cyclictest25863-21sh23:35:142
338899221,11cyclictest21208-21irqcore20:35:172
338899221,11cyclictest18098-21ps19:55:272
338899221,11cyclictest17586-21irqcore21:25:182
338899221,10cyclictest5721-21grep22:30:202
3388992210,9cyclictest0-21swapper/219:30:182
337499224,13cyclictest21-21ksoftirqd/122:55:131
337499221,9cyclictest131rcu_preempt23:25:201
337499221,17cyclictest32631-21ssh22:45:161
337499221,14cyclictest19015-21find00:25:131
336899228,7cyclictest0-21swapper/022:33:060
336899221,18cyclictest15842-21ls21:00:220
3368992211,3cyclictest0-21swapper/000:35:030
3368992211,3cyclictest0-21swapper/000:35:030
339699217,9cyclictest35966irq/58-eth000:25:153
339699213,12cyclictest0-21swapper/323:30:073
3396992115,4cyclictest35966irq/58-eth023:25:133
338899219,10cyclictest30088-21ssh21:50:162
338899213,10cyclictest29386-21irqcore22:25:182
338899212,10cyclictest30578-21kthreadcore20:10:212
338899212,10cyclictest30578-21kthreadcore20:10:202
338899211,8cyclictest29214-21kthreadcore23:00:222
338899211,16cyclictest3495-21/usr/sbin/munin19:10:152
338899211,10cyclictest7795-21fschecks_count20:55:142
338899211,10cyclictest6442-21fschecks_count00:00:172
337499214,13cyclictest28083-21ssh00:30:171
337499212,15cyclictest11074-21apt-get20:25:151
337499212,14cyclictest9030-1kworker/u9:223:48:101
337499211,17cyclictest11706-21ssh21:20:261
337499211,16cyclictest15856-21ssh22:00:151
337499211,13cyclictest24358-21cut20:05:151
336899219,3cyclictest12-21ksoftirqd/022:03:090
336899219,3cyclictest121ksoftirqd/000:00:230
336899217,3cyclictest0-21swapper/022:51:150
3368992115,4cyclictest0-21swapper/022:10:250
3368992113,5cyclictest12-21ksoftirqd/020:05:010
3368992110,3cyclictest12-21ksoftirqd/023:11:140
3368992110,3cyclictest12-21ksoftirqd/023:11:130
339699203,11cyclictest0-21swapper/323:20:093
339699202,15cyclictest0-21swapper/323:05:143
339699202,11cyclictest0-21swapper/300:20:083
3396992010,7cyclictest35966irq/58-eth021:55:143
3396992010,6cyclictest35966irq/58-eth021:10:143
338899201,10cyclictest11443-21ssh22:15:192
3374992013,4cyclictest0-21swapper/121:10:141
337499201,16cyclictest4733-21ssh00:35:171
337499201,16cyclictest4733-21ssh00:35:161
337499201,15cyclictest8006-21grep20:55:171
337499201,14cyclictest21897-21sh23:15:141
337499201,13cyclictest23810-21cut22:40:171
337499200,17cyclictest26608-21ls19:30:271
336899208,2cyclictest12-21ksoftirqd/019:35:250
336899206,9cyclictest16628-21acpi23:30:110
3368992014,3cyclictest12-21ksoftirqd/019:55:320
336899201,16cyclictest1472-21ls20:15:150
339699198,7cyclictest35966irq/58-eth023:15:133
339699194,12cyclictest35966irq/58-eth021:35:133
339699193,11cyclictest0-21swapper/323:54:093
339699193,11cyclictest0-21swapper/323:45:083
339699193,10cyclictest0-21swapper/323:57:093
339699192,13cyclictest35966irq/58-eth023:40:133
339699192,10cyclictest0-21swapper/320:50:153
3396991912,4cyclictest0-21swapper/300:35:253
3396991912,4cyclictest0-21swapper/300:35:243
339699191,16cyclictest0-21swapper/320:30:133
339699191,13cyclictest0-21swapper/319:20:173
339699191,12cyclictest0-21swapper/322:50:023
339699191,12cyclictest0-21swapper/321:40:263
339699191,10cyclictest0-21swapper/319:35:293
3396991910,4cyclictest0-21swapper/321:30:183
337499199,7cyclictest31006-21cat19:35:241
337499199,7cyclictest19415-21apt-get20:00:121
337499198,3cyclictest131rcu_preempt20:35:021
337499192,14cyclictest844-21munin-node21:30:171
337499191,9cyclictest0-21swapper/122:16:511
337499191,16cyclictest14075-21ls00:20:291
337499191,14cyclictest18197-21ssh22:35:271
337499191,13cyclictest2318-21ssh23:40:081
336899199,2cyclictest12-21ksoftirqd/020:55:270
336899198,3cyclictest12-21ksoftirqd/022:57:190
3368991914,3cyclictest0-21swapper/021:30:360
3368991913,3cyclictest12-21ksoftirqd/021:50:010
3368991912,4cyclictest12-21ksoftirqd/020:30:010
3368991912,4cyclictest12-21ksoftirqd/019:45:270
3368991911,6cyclictest1152-21sshd20:13:050
3368991911,6cyclictest1152-21sshd20:13:040
336899191,10cyclictest131rcu_preempt00:19:160
339699189,6cyclictest0-21swapper/322:21:433
339699187,3cyclictest0-21swapper/322:44:403
339699182,10cyclictest0-21swapper/321:00:133
339699182,10cyclictest0-21swapper/320:40:133
3396991811,4cyclictest0-21swapper/322:08:273
339699181,12cyclictest0-21swapper/320:20:273
337499186,9cyclictest20403-21ps23:30:241
337499186,9cyclictest13037-21cron19:20:011
337499183,12cyclictest6978-21sed21:35:301
337499182,12cyclictest2190-21ssh23:20:251
3374991811,4cyclictest7034-21kthreadcore00:00:211
337499181,13cyclictest21982-21diskmemload22:05:251
337499181,11cyclictest1688-21ssh21:50:321
3374991810,5cyclictest0-21swapper/120:25:031
336899186,8cyclictest12-21ksoftirqd/020:05:120
3368991813,3cyclictest12-21ksoftirqd/019:40:280
3368991811,4cyclictest12-21ksoftirqd/020:35:280
336899181,11cyclictest6221-21find20:20:110
339699179,6cyclictest0-21swapper/323:35:093
339699179,6cyclictest0-21swapper/323:35:083
339699179,6cyclictest0-21swapper/321:26:373
339699179,6cyclictest0-21swapper/321:23:363
339699179,5cyclictest35966irq/58-eth022:00:143
339699179,3cyclictest0-21swapper/300:01:233
339699178,6cyclictest35966irq/58-eth022:55:143
339699172,9cyclictest0-21swapper/320:10:153
339699172,9cyclictest0-21swapper/320:10:143
339699172,10cyclictest35966irq/58-eth019:30:123
339699171,12cyclictest0-21swapper/323:00:123
337499179,5cyclictest24558-21grep21:45:241
337499178,7cyclictest3513-21apt-get19:10:121
337499178,7cyclictest3091-21grep20:50:191
337499177,8cyclictest17091-21apt-get21:05:021
337499174,3cyclictest131rcu_preempt21:15:131
337499173,10cyclictest0-21swapper/119:55:141
337499172,12cyclictest31509-21tr20:45:191
337499172,12cyclictest28879-21ssh22:25:141
337499172,11cyclictest0-21swapper/121:05:161
337499172,11cyclictest0-21swapper/121:05:151
337499171,4cyclictest21-21ksoftirqd/123:00:181
3374991711,4cyclictest0-21swapper/121:55:151
337499171,13cyclictest3934-21df23:05:151
3374991710,5cyclictest0-21swapper/120:35:171
336899171,5cyclictest53646usb-storage21:05:110
336899171,5cyclictest53646usb-storage21:05:100
3368991710,4cyclictest12-21ksoftirqd/020:30:290
3368991710,4cyclictest12-21ksoftirqd/019:50:300
339699163,10cyclictest0-21swapper/322:15:143
339699162,9cyclictest0-21swapper/319:50:143
338899166,6cyclictest3426-21latency_hist19:10:022
337499163,10cyclictest29233-21tune2fs20:10:171
337499163,10cyclictest29233-21tune2fs20:10:171
337499161,12cyclictest32604-21apt-get19:40:141
337499161,12cyclictest20663-21diskstats23:50:151
337499161,11cyclictest21543-21ntp_states19:25:271
337499160,13cyclictest6181-21latency_hist20:20:021
339699154,8cyclictest35966irq/58-eth019:40:113
339699151,6cyclictest0-21swapper/319:15:023
339699151,11cyclictest0-21swapper/320:35:133
337499151,11cyclictest4971-21apt-get19:45:151
337499151,11cyclictest13345-21cpuspeed19:20:151
336899151,10cyclictest3421-21latency_hist19:10:010
339699149,3cyclictest0-21swapper/320:15:263
339699149,3cyclictest0-21swapper/320:05:263
339699141,8cyclictest0-21swapper/320:00:133
339699141,7cyclictest0-21swapper/319:30:023
339699141,11cyclictest0-21swapper/321:05:313
339699141,11cyclictest0-21swapper/321:05:303
339699141,10cyclictest0-21swapper/319:45:223
337499141,10cyclictest3442-21latency_hist19:10:021
339699138,3cyclictest0-21swapper/320:45:153
339699138,3cyclictest0-21swapper/319:15:293
339699131,10cyclictest0-21swapper/320:55:223
339699123,7cyclictest0-21swapper/320:25:253
339699121,9cyclictest0-21swapper/319:55:153
339699101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional