You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 13:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Mon May 11, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2697799761,9cyclictest12-21ksoftirqd/022:10:130
2697799668,43cyclictest20556-21ssh21:50:130
2699999653,41cyclictest8789-21ssh22:55:122
2699999641,38cyclictest0-21swapper/221:50:132
2699999626,6cyclictest131rcu_preempt22:45:122
26999996111,41cyclictest131rcu_preempt22:05:142
2699999585,13cyclictest131rcu_preempt00:15:132
2697799552,41cyclictest25846-21ssh21:35:140
2699999514,30cyclictest930-21snmpd21:40:132
2698799516,40cyclictest9597-21ssh00:25:131
26999994916,25cyclictest27-21ksoftirqd/200:35:122
2698799493,7cyclictest131rcu_preempt00:35:121
2698799483,30cyclictest11448-21ssh23:50:151
2699999464,21cyclictest20331-21ssh23:55:122
2698799455,34cyclictest6276-21ssh22:00:131
26987994529,13cyclictest23074-21apt-get22:45:121
26987994528,11cyclictest13978-21apt-get22:40:111
2698799445,30cyclictest25794-21ssh23:40:141
26987994232,4cyclictest11678-21apt-get20:35:131
26987994214,17cyclictest24650-21grep21:15:231
26987994214,17cyclictest24650-21grep21:15:231
2698799414,32cyclictest15338-21ssh22:05:131
26987994124,12cyclictest24305-21apt-get22:10:131
26987994110,12cyclictest14137-21ssh21:45:211
2697799413,28cyclictest460-21ssh00:20:120
26977994113,18cyclictest12-21ksoftirqd/022:40:110
26987994028,9cyclictest0-21swapper/121:10:111
26987994028,9cyclictest0-21swapper/121:10:111
26977994014,3cyclictest131rcu_preempt22:35:120
2701199397,19cyclictest35966irq/58-eth000:05:123
2699999394,4cyclictest131rcu_preempt20:05:142
26999993912,8cyclictest131rcu_preempt21:35:132
2699999391,22cyclictest10049-21ssh22:20:142
2699999390,19cyclictest5921-21ssh00:05:132
26987993934,3cyclictest29739-21apt-get20:20:121
26987993923,11cyclictest19092-21ssh22:25:121
26987993920,14cyclictest0-21swapper/121:55:131
26987993829,4cyclictest4387-21apt-get19:20:111
26987993825,10cyclictest8778-21apt-get22:55:131
26987993812,10cyclictest14466-21ssh00:09:551
26987993811,12cyclictest23537-21grep23:20:201
26987993811,12cyclictest23537-21grep23:20:201
2697799381,6cyclictest131rcu_preempt20:20:120
27011993730,3cyclictest35966irq/58-eth023:15:153
26999993711,10cyclictest27-21ksoftirqd/222:10:132
2698799379,9cyclictest7445-21cron21:25:001
26987993728,4cyclictest25819-21apt-get21:35:131
26987993726,6cyclictest449-21apt-get00:20:131
26987993713,16cyclictest3312-21ssh22:33:361
26987993712,9cyclictest18376-21cat19:30:301
26987993712,11cyclictest16029-21ssh23:34:451
26987993712,10cyclictest27777-21/usr/sbin/munin19:10:181
26987993710,12cyclictest32187-21ssh22:50:121
26987993710,10cyclictest9583-21cat22:35:301
26977993710,23cyclictest12-21ksoftirqd/022:25:130
2699999362,28cyclictest11021-21ssh23:13:272
26999993618,8cyclictest27-21ksoftirqd/200:20:122
2698799369,13cyclictest5384-21cut21:40:221
2698799361,32cyclictest3649-21sh23:10:131
26987993613,13cyclictest32496-21kthreadcore20:55:191
26987993610,12cyclictest20084-21kthreadcore23:35:231
26987993610,11cyclictest8801-21sh23:47:231
26987993610,10cyclictest2560-21cut19:50:181
2697799363,15cyclictest131rcu_preempt23:17:130
27011993520,12cyclictest35966irq/58-eth023:50:153
2698799359,10cyclictest14841-21usage_EthcTask22:20:311
2698799356,11cyclictest31347-21cat23:05:311
26987993529,3cyclictest2115-21apt-get20:25:131
2698799352,24cyclictest17888-21ssh23:00:141
26987993520,10cyclictest28517-21apt-get19:45:131
26987993511,7cyclictest0-21swapper/100:33:461
2698799349,10cyclictest3020-21ssh00:01:571
26987993411,9cyclictest23681-21/usr/sbin/munin19:40:171
2697799340,31cyclictest0-21swapper/023:30:140
27011993318,12cyclictest35966irq/58-eth022:40:133
2701199331,20cyclictest35966irq/58-eth022:05:143
2699999333,14cyclictest17867-21ssh00:29:372
26987993318,9cyclictest30985-21ssh23:25:101
26987993310,8cyclictest27587-21ntp_192.168.11400:15:251
26987993310,8cyclictest18617-21ssh00:10:261
26987993310,8cyclictest16976-21diskstats21:30:181
26987993310,10cyclictest29257-21cat20:15:301
26987993310,10cyclictest20483-21cut20:10:171
26977993310,15cyclictest12-21ksoftirqd/000:00:130
2701199321,17cyclictest35966irq/58-eth022:55:133
2698799329,7cyclictest26304-21perl20:50:161
26987993210,7cyclictest28989-21ssh21:54:471
26987993210,7cyclictest1672-21irqstats22:15:191
2697799323,25cyclictest5912-21ssh00:05:120
2699999316,15cyclictest27-21ksoftirqd/223:35:142
2699999314,17cyclictest27346irq/40-dwc2_hso19:25:142
2699999312,19cyclictest27346irq/40-dwc2_hso23:45:142
2698799319,8cyclictest25092-21cat20:45:291
2698799319,7cyclictest10803-21if_eth120:00:181
26987993125,3cyclictest16505-21apt-get20:40:131
26987993111,9cyclictest10551-21threads20:30:281
26987993110,8cyclictest11813-21sed21:25:291
2697799312,22cyclictest6096-21ntp_states23:45:270
2699999302,22cyclictest25918-21ssh21:15:282
2699999302,22cyclictest25918-21ssh21:15:282
2698799309,9cyclictest25841-21sh23:56:161
2698799309,8cyclictest15055-21cut23:15:221
26987993015,12cyclictest0-21swapper/121:05:141
2699999293,21cyclictest0-21swapper/200:14:272
2699999293,18cyclictest27346irq/40-dwc2_hso21:10:112
2699999293,18cyclictest27346irq/40-dwc2_hso21:10:112
2699999292,22cyclictest0-21swapper/223:24:272
2699999292,22cyclictest0-21swapper/223:24:272
2699999292,11cyclictest22808-21ntp_states22:25:262
2699999291,16cyclictest27-21ksoftirqd/222:35:142
2699999290,7cyclictest131rcu_preempt21:25:122
2698799298,7cyclictest0-21swapper/119:55:171
2698799295,8cyclictest0-21swapper/121:00:291
2698799292,21cyclictest844-21munin-node19:25:151
26987992915,10cyclictest441-21tr19:15:191
2697799291,22cyclictest21959-21ssh23:20:140
2697799291,22cyclictest21959-21ssh23:20:140
2697799291,14cyclictest12-21ksoftirqd/019:20:120
2699999282,19cyclictest16585-21ssh23:15:272
26999992811,12cyclictest13987-21ssh22:40:112
2697799283,15cyclictest31235-21ssh21:20:120
26977992824,2cyclictest12-21ksoftirqd/022:47:060
2701199273,16cyclictest36066irq/59-eth023:00:143
2701199272,17cyclictest35966irq/58-eth023:35:143
2699999273,17cyclictest27346irq/40-dwc2_hso19:20:112
2699999272,5cyclictest131rcu_preempt23:40:152
2699999272,20cyclictest18608-21ssh00:30:142
2699999272,17cyclictest0-21swapper/220:40:132
2699999272,11cyclictest0-21swapper/221:00:132
2699999271,18cyclictest30991-21ls23:05:292
26987992715,9cyclictest19131-21idleruntime19:35:171
26987992715,8cyclictest15431-21entropy20:05:161
2697799279,2cyclictest131rcu_preempt23:55:130
2697799274,16cyclictest12-21ksoftirqd/020:05:140
26977992711,5cyclictest0-21swapper/021:00:280
2701199269,11cyclictest0-21swapper/322:32:263
2699999264,15cyclictest27346irq/40-dwc2_hso19:15:132
2699999263,17cyclictest27346irq/40-dwc2_hso20:10:132
2699999262,8cyclictest17829-21ssh23:52:272
2699999262,17cyclictest27346irq/40-dwc2_hso19:45:122
2699999262,17cyclictest0-21swapper/221:30:142
2699999262,16cyclictest27346irq/40-dwc2_hso23:00:132
2697799266,4cyclictest131rcu_preempt22:50:130
26977992615,3cyclictest12-21ksoftirqd/022:30:230
2701199258,11cyclictest0-21swapper/322:28:263
27011992519,4cyclictest321rcuc/323:47:173
2699999252,13cyclictest0-21swapper/220:00:142
2699999251,12cyclictest25035-21cut20:15:162
2697799252,19cyclictest29481-21ssh23:40:270
26977992512,3cyclictest12-21ksoftirqd/000:10:140
2697799251,20cyclictest6292-21ssh22:00:140
2701199243,13cyclictest0-21swapper/323:25:273
2701199242,18cyclictest35966irq/58-eth000:10:133
2701199242,17cyclictest35966irq/58-eth022:00:133
2701199242,13cyclictest0-21swapper/300:25:143
27011992419,3cyclictest321rcuc/319:30:123
2701199241,15cyclictest35966irq/58-eth021:20:123
2699999243,15cyclictest27346irq/40-dwc2_hso21:05:152
2699999242,17cyclictest844-21munin-node20:20:132
26999992417,5cyclictest930-21snmpd22:30:232
2697799242,19cyclictest5091-21ssh23:27:270
2697799241,20cyclictest11580-21sh21:45:120
2701199233,16cyclictest35966irq/58-eth021:35:133
27011992317,4cyclictest321rcuc/323:11:503
27011992317,4cyclictest321rcuc/323:06:233
27011992317,3cyclictest321rcuc/300:35:163
2701199231,16cyclictest35966irq/58-eth020:30:123
2699999233,7cyclictest32221-21ssh22:50:132
2699999233,15cyclictest27346irq/40-dwc2_hso20:35:122
2699999232,6cyclictest0-21swapper/223:26:272
2699999232,16cyclictest0-21swapper/223:34:272
2697799235,10cyclictest628-21tcf-agent21:25:120
2697799234,3cyclictest131rcu_preempt21:20:000
2697799234,3cyclictest131rcu_preempt21:20:000
2697799232,15cyclictest13279-21ntp_states00:25:270
2697799231,18cyclictest2353-21ls20:25:180
2697799231,17cyclictest10064-21ssh22:20:150
26977992310,10cyclictest12529-21ssh23:15:020
2701199224,13cyclictest0-21swapper/321:10:113
2701199224,13cyclictest0-21swapper/321:10:113
2701199223,13cyclictest0-21swapper/322:47:063
27011992216,4cyclictest321rcuc/323:33:593
27011992216,4cyclictest321rcuc/321:32:213
27011992211,8cyclictest0-21swapper/322:35:143
27011992210,8cyclictest35966irq/58-eth022:50:153
2699999225,4cyclictest19940-21kthreadcore19:35:202
2699999225,3cyclictest131rcu_preempt20:30:302
2699999223,13cyclictest27346irq/40-dwc2_hso19:50:112
2699999221,10cyclictest20154-21cron21:50:012
2697799227,3cyclictest131rcu_preempt21:30:150
2697799222,14cyclictest2009-21kworker/u8:023:07:270
26977992215,4cyclictest19167-21irqrtprio00:30:180
2697799221,14cyclictest12-21ksoftirqd/020:00:130
2701199219,9cyclictest0-21swapper/322:14:283
2701199213,11cyclictest0-21swapper/320:15:143
27011992116,3cyclictest321rcuc/323:57:003
27011992116,3cyclictest321rcuc/321:15:403
27011992116,3cyclictest321rcuc/321:15:403
27011992115,4cyclictest321rcuc/300:18:393
27011992112,4cyclictest0-21swapper/321:48:493
27011992110,8cyclictest0-21swapper/321:52:323
27011992110,7cyclictest35966irq/58-eth021:25:133
2699999214,8cyclictest2609-21grep21:20:272
2699999212,8cyclictest261rcuc/221:55:142
2699999212,16cyclictest5390-21cat20:25:232
2699999212,16cyclictest30119-21kthreadcore19:10:212
2699999211,17cyclictest1115-21unixbench_singl00:00:312
2699999211,11cyclictest5832-21fschecks_count19:55:172
26999992110,5cyclictest14114-21ssh22:04:032
26987992110,8cyclictest27002-21latency_hist19:10:031
2697799219,3cyclictest12-21ksoftirqd/022:05:200
2697799212,3cyclictest131rcu_preempt20:45:150
2697799211,14cyclictest20191-21ls20:10:130
2697799211,10cyclictest131rcu_preempt00:17:260
2701199208,8cyclictest35966irq/58-eth023:40:143
27011992014,4cyclictest321rcuc/321:55:123
27011992014,4cyclictest321rcuc/320:35:133
27011992013,4cyclictest0-21swapper/322:21:273
27011992010,8cyclictest0-21swapper/300:03:503
2699999202,4cyclictest0-21swapper/220:45:132
2699999201,9cyclictest3287-21kthreadcore22:15:212
2699999201,9cyclictest27553-21grep19:40:282
2699999201,10cyclictest32702-21kthreadcore20:55:192
2699999201,10cyclictest14263-21fschecks_time19:30:162
2697799209,2cyclictest12-21ksoftirqd/021:56:140
2697799202,15cyclictest18530-21kthreadcore20:40:200
26977992013,4cyclictest12-21ksoftirqd/023:50:280
26977992013,3cyclictest12-21ksoftirqd/022:55:130
2697799201,10cyclictest4552-21kthreadcore21:40:210
26977992010,3cyclictest12-21ksoftirqd/022:19:240
2697799200,16cyclictest24373-21sed19:40:180
2701199193,11cyclictest0-21swapper/300:21:273
27011991914,3cyclictest321rcuc/320:41:453
27011991914,3cyclictest321rcuc/319:40:113
27011991913,4cyclictest321rcuc/319:25:153
27011991913,4cyclictest0-21swapper/300:32:023
27011991911,6cyclictest0-21swapper/321:44:133
2701199191,12cyclictest35966irq/58-eth019:55:143
2699999191,10cyclictest28147-21kthreadcore20:50:202
2697799199,2cyclictest12-21ksoftirqd/023:39:520
2697799196,3cyclictest12-21ksoftirqd/000:35:220
2697799191,9cyclictest9971-21kthreadcore21:05:220
2697799191,3cyclictest131rcu_preempt23:00:130
26977991913,3cyclictest12-21ksoftirqd/019:55:200
2697799191,10cyclictest29648-21tr19:45:190
2701199182,10cyclictest0-21swapper/320:50:133
2701199182,10cyclictest0-21swapper/320:10:133
2701199182,10cyclictest0-21swapper/319:50:123
27011991813,3cyclictest321rcuc/322:15:163
27011991812,4cyclictest0-21swapper/323:20:073
27011991812,4cyclictest0-21swapper/323:20:073
2699999181,15cyclictest26959-21cyclictest19:10:022
2697799182,2cyclictest131rcu_preempt20:55:190
2697799181,5cyclictest13571-21tail21:10:160
2697799181,5cyclictest13571-21tail21:10:160
26977991812,3cyclictest12-21ksoftirqd/019:40:010
2697799181,14cyclictest9530-21kthreadcore20:30:210
26977991811,4cyclictest0-21swapper/019:10:140
2701199172,10cyclictest0-21swapper/320:20:133
2701199172,10cyclictest0-21swapper/320:00:143
27011991712,3cyclictest321rcuc/321:05:203
2701199171,10cyclictest0-21swapper/319:45:173
2697799175,7cyclictest0-21swapper/020:39:540
2697799172,2cyclictest131rcu_preempt20:50:210
2697799171,3cyclictest131rcu_preempt20:15:230
26977991710,4cyclictest151rcuc/019:50:110
2697799170,9cyclictest16645-21kthreadcore19:30:210
2701199162,10cyclictest0-21swapper/320:55:043
2701199162,10cyclictest0-21swapper/320:25:143
27011991611,3cyclictest321rcuc/320:45:213
27011991611,3cyclictest0-21swapper/321:00:193
27011991610,4cyclictest321rcuc/319:35:143
27011991610,4cyclictest321rcuc/319:20:123
27011991610,4cyclictest0-21swapper/320:05:133
2697799164,9cyclictest11852-21kthreadcore19:25:220
2697799163,8cyclictest0-21swapper/019:15:130
2701199149,3cyclictest321rcuc/319:15:123
2701199141,7cyclictest0-21swapper/319:10:173
2697799141,11cyclictest26905-21latency_hist19:10:020
2701199101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional