You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-22 - 03:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Aug 22, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2123099572,9cyclictest131rcu_preempt22:20:141
21230995618,31cyclictest0-21swapper/119:50:121
21230995039,4cyclictest5896-21apt-get20:35:101
21230995039,4cyclictest5896-21apt-get20:35:091
21253994931,11cyclictest35966irq/58-eth023:15:133
21241994828,11cyclictest20965-21apt-get00:00:122
2125399465,25cyclictest35966irq/58-eth023:50:123
21253994634,9cyclictest35966irq/58-eth023:05:133
2125399463,28cyclictest35966irq/58-eth022:40:123
2123099462,33cyclictest5467-21ssh21:45:121
21241994531,3cyclictest14455-21apt-get21:50:122
2124199449,22cyclictest671-21ssh00:25:022
21241994423,17cyclictest3102-21ssh23:50:132
2123099444,28cyclictest26544-21ssh23:45:121
21241994324,14cyclictest0-21swapper/220:10:102
21241994313,20cyclictest19576-21grep00:15:272
21241994312,19cyclictest17738-21ssh23:20:252
21230994335,4cyclictest30449-21apt-get22:35:131
21230994313,21cyclictest21-21ksoftirqd/123:30:121
2121999434,34cyclictest1056-21ssh00:25:120
2121999432,30cyclictest25249-21rm21:20:140
2124199427,14cyclictest4510-21munin-run20:00:022
21241994213,11cyclictest0-21swapper/222:30:272
21230994229,10cyclictest9052-21apt-get22:05:131
2121999422,30cyclictest19022-21ssh00:35:130
21219994214,23cyclictest12-21ksoftirqd/000:20:130
2121999421,32cyclictest28862-21ssh21:40:120
2121999421,32cyclictest28862-21ssh21:40:110
21241994130,7cyclictest10862-21apt-get21:30:162
21241994128,9cyclictest17639-21apt-get23:40:122
21241994110,15cyclictest9982-21ssh22:22:152
21230994136,3cyclictest23950-21apt-get20:20:131
21230994127,10cyclictest23122-21apt-get23:25:131
21230994115,9cyclictest14082-21timerwakeupswit00:30:281
21230994114,10cyclictest26783-21ssh23:08:221
21230994114,10cyclictest23023-21ssh21:17:501
21230994114,10cyclictest16792-21ssh21:31:531
21230994113,8cyclictest17442-21users22:25:311
21253994020,13cyclictest35966irq/58-eth023:25:113
2124199407,18cyclictest10551-21latency_hist21:30:022
21241994020,13cyclictest0-21swapper/222:05:142
21241994017,20cyclictest0-21swapper/220:20:122
2123099408,15cyclictest22507-21latency00:35:241
21230994033,4cyclictest2295-21apt-get21:05:111
21230994033,4cyclictest2295-21apt-get21:05:111
21230994032,5cyclictest15423-21apt-get00:15:121
21230994028,7cyclictest24461-21apt-get00:20:121
21230994015,8cyclictest9369-21ssh21:28:451
21230994015,8cyclictest6913-21ssh22:02:561
21230994015,11cyclictest10670-21ssh23:16:211
21230994015,10cyclictest10973-21sh23:54:141
21230994014,9cyclictest22781-21latency_hist23:25:001
21230994014,10cyclictest30178-21ssh21:20:391
21230994012,11cyclictest28768-21ssh23:10:171
21230994011,11cyclictest30245-21grep22:35:021
2124199399,19cyclictest0-21swapper/219:25:152
2124199399,14cyclictest32233-21latency_hist22:00:032
21241993926,9cyclictest16210-21ssh21:15:142
2124199391,22cyclictest8427-21ssh22:57:572
21241993910,15cyclictest32042-21kthreadcore21:40:222
21241993910,15cyclictest32042-21kthreadcore21:40:212
21230993929,5cyclictest28867-21ssh21:40:121
21230993929,5cyclictest28867-21ssh21:40:111
21230993927,9cyclictest28733-21apt-get20:25:121
21230993924,11cyclictest18063-21apt-get22:10:141
21230993914,8cyclictest7147-21fschecks_count22:40:141
21230993914,10cyclictest16114-21ssh23:38:381
2125399383,26cyclictest35966irq/58-eth022:25:113
2124199389,7cyclictest27374-21ps22:15:192
21241993831,4cyclictest5127-21apt-get23:15:152
21241993812,11cyclictest302-21ssh23:30:192
21241993812,11cyclictest28332-21wc20:55:222
21230993819,15cyclictest0-21swapper/121:50:131
21230993815,10cyclictest6784-21ssh22:55:571
21230993815,10cyclictest32586-21ssh22:16:241
21230993814,8cyclictest6851-21diskmemload22:49:031
21230993811,12cyclictest26393-21sed19:45:261
2124199379,18cyclictest0-21swapper/223:04:072
2124199378,12cyclictest12187-21ssh22:40:532
2124199377,8cyclictest17478-21ssh23:56:132
2124199377,12cyclictest0-21swapper/223:05:162
21241993712,16cyclictest32150-21netstat23:10:232
21241993711,12cyclictest930-21snmpd00:35:112
21241993710,13cyclictest13917-21usage_EthcTask20:05:312
21230993727,7cyclictest6505-21ssh00:10:131
21230993726,7cyclictest29939-21apt-get00:05:141
21230993719,14cyclictest0-21swapper/123:55:121
21230993713,11cyclictest7017-21ssh00:26:501
21230993712,8cyclictest11682-21ntp_192.168.11419:30:241
21230993711,10cyclictest15223-21usage_EthcTask23:00:301
2121999377,17cyclictest12-21ksoftirqd/022:00:130
2124199364,16cyclictest10910-21grep20:40:162
21241993612,8cyclictest24838-21ssh21:35:452
21241993611,8cyclictest18199-21diskstats22:10:152
2123099369,14cyclictest5783-21kthreadcore20:00:191
21230993630,3cyclictest29906-21apt-get21:00:121
21230993626,6cyclictest19769-21apt-get21:35:141
21230993624,7cyclictest7246-21ssh21:10:131
21230993617,15cyclictest0-21swapper/120:55:111
21230993612,9cyclictest31000-21ssh22:51:571
21219993614,16cyclictest131rcu_preempt23:40:100
2121999361,32cyclictest5158-21ssh23:15:140
21219993613,20cyclictest12-21ksoftirqd/019:55:140
2124199359,8cyclictest15748-21ssh00:31:312
2124199359,16cyclictest0-21swapper/221:00:152
2124199359,12cyclictest30068-21df_inode00:05:162
21241993526,4cyclictest15986-21apt-get22:45:122
21241993522,10cyclictest27346irq/40-dwc2_hso20:25:132
21230993524,7cyclictest0-21swapper/121:55:121
21230993518,15cyclictest0-21swapper/120:50:131
21230993512,8cyclictest31047-21apt-get19:20:111
21230993512,10cyclictest21575-21irqrtprio00:00:191
2124199349,11cyclictest0-21swapper/222:35:092
21230993427,4cyclictest13162-21apt-get19:35:121
21230993415,15cyclictest0-21swapper/120:10:111
21230993412,8cyclictest29604-21ntp_states19:15:251
2125399331,22cyclictest35966irq/58-eth022:15:133
2124199339,8cyclictest30216-21ssh22:50:592
2124199338,8cyclictest3663-21ssh22:00:242
2124199338,16cyclictest30985-21/usr/sbin/munin19:50:242
2124199337,7cyclictest6851-21diskmemload00:11:472
2124199336,11cyclictest30792-21ssh23:45:282
2124199332,13cyclictest0-21swapper/223:37:572
21241993312,9cyclictest13162-21apt-get19:35:112
2124199331,15cyclictest0-21swapper/221:45:132
21241993310,9cyclictest31815-21ssh21:22:352
21230993312,10cyclictest22802-21kthreadcore19:10:191
21219993313,12cyclictest12-21ksoftirqd/021:25:120
2125399321,19cyclictest35966irq/58-eth023:00:133
2124199329,7cyclictest12585-21apt-get22:25:122
2124199328,13cyclictest26526-21irqcore19:15:182
2124199328,11cyclictest31563-21irqrtprio19:20:162
2124199327,11cyclictest0-21swapper/221:10:262
2124199323,12cyclictest0-21swapper/220:35:252
2124199323,12cyclictest0-21swapper/220:35:242
21241993214,14cyclictest0-21swapper/219:30:112
2123099328,9cyclictest1873-21grep19:55:191
21230993223,4cyclictest21-21ksoftirqd/120:40:131
21230993211,8cyclictest0-21swapper/123:42:441
2121999324,21cyclictest21579-21kworker/u8:322:55:110
2121999322,24cyclictest30551-21ssh22:51:240
2121999321,23cyclictest12615-21ssh22:25:140
2125399315,14cyclictest35966irq/58-eth022:35:123
2124199319,8cyclictest5571-21cat20:30:312
2124199315,9cyclictest0-21swapper/221:05:122
2124199315,9cyclictest0-21swapper/221:05:112
21241993111,8cyclictest16948-21kthreadcore20:45:192
21241993110,6cyclictest0-21swapper/223:28:492
21219993120,6cyclictest0-21swapper/019:35:130
2121999311,19cyclictest12-21ksoftirqd/021:15:130
2124199308,7cyclictest24738-21ntp_192.168.11419:10:232
2124199305,10cyclictest4636-21cat00:25:232
2124199303,13cyclictest0-21swapper/219:40:112
21241993016,10cyclictest27346irq/40-dwc2_hso19:45:122
21241993010,7cyclictest0-21swapper/220:50:282
2123099309,10cyclictest5478-21cat20:30:321
21230993015,12cyclictest23703-21idleruntime-cro20:20:021
21219993014,5cyclictest0-21swapper/023:35:290
2124199292,21cyclictest0-21swapper/220:15:162
21230992922,4cyclictest0-21swapper/119:25:081
21230992918,8cyclictest15487-21munin-node20:45:101
2121999296,17cyclictest131rcu_preempt22:40:110
2121999291,20cyclictest12-21ksoftirqd/000:30:130
2123099289,9cyclictest21864-21sed19:40:281
21230992815,10cyclictest13273-21timerwakeupswit20:05:281
2121999281,23cyclictest27003-21ssh22:15:150
2125399278,9cyclictest0-21swapper/321:20:143
2125399273,16cyclictest35966irq/58-eth023:10:143
2121999272,21cyclictest19796-21ssh21:35:130
2125399262,14cyclictest35966irq/58-eth023:35:123
2125399261,15cyclictest35966irq/58-eth022:55:103
21241992620,2cyclictest0-21swapper/220:00:112
21230992615,9cyclictest21248-21date19:10:011
2121999263,14cyclictest6851-21diskmemload21:30:160
2121999262,18cyclictest21578-21sshd22:46:220
2121999261,3cyclictest131rcu_preempt20:35:100
2121999261,3cyclictest131rcu_preempt20:35:090
2125399253,10cyclictest0-21swapper/322:51:243
2125399251,13cyclictest35966irq/58-eth022:30:123
2121999254,16cyclictest12-21ksoftirqd/023:50:120
2121999252,19cyclictest21001-21ssh00:00:140
21219992415,4cyclictest0-21swapper/020:55:140
2121999241,18cyclictest28002-21grep19:15:210
2121999239,9cyclictest6509-21ssh22:39:560
2121999239,12cyclictest12-21ksoftirqd/000:05:130
2121999238,3cyclictest131rcu_preempt21:05:120
2121999238,3cyclictest131rcu_preempt21:05:110
2125399223,12cyclictest0-21swapper/300:25:133
2125399222,11cyclictest0-21swapper/321:30:183
2121999223,15cyclictest19559-21ssh23:05:130
2121999221,15cyclictest844-21munin-node21:00:110
2121999215,3cyclictest0-21swapper/023:26:080
21219992115,4cyclictest26340-21mailstats19:45:270
21219992115,4cyclictest0-21swapper/022:10:130
21219992113,5cyclictest0-21swapper/020:50:250
2121999210,3cyclictest131rcu_preempt22:30:210
2125399206,8cyclictest35966irq/58-eth000:05:123
21253992016,2cyclictest0-21swapper/319:55:123
21253992013,5cyclictest0-21swapper/300:00:193
21253992011,5cyclictest35966irq/58-eth000:10:133
2125399201,14cyclictest0-21swapper/320:05:283
21241992011,7cyclictest21206-21cyclictest19:10:022
2121999208,3cyclictest131rcu_preempt20:20:170
2121999206,2cyclictest131rcu_preempt21:55:290
2121999205,3cyclictest131rcu_preempt19:10:190
2121999204,8cyclictest131rcu_preempt22:05:140
2121999203,5cyclictest17438-21kthreadcore21:50:220
2121999203,11cyclictest12-21ksoftirqd/020:25:130
2121999202,3cyclictest131rcu_preempt22:20:220
2121999202,3cyclictest131rcu_preempt20:05:330
2121999201,3cyclictest131rcu_preempt23:45:220
2121999201,2cyclictest131rcu_preempt23:02:170
21219992012,6cyclictest0-21swapper/023:56:370
2121999201,16cyclictest6667-21fschecks_count00:10:160
21219992011,6cyclictest0-21swapper/023:10:140
2121999201,15cyclictest6851-21diskmemload21:10:280
2121999201,11cyclictest17639-21missed_timers20:10:210
2121999200,4cyclictest131rcu_preempt23:23:060
2125399199,8cyclictest0-21swapper/321:58:223
21253991913,4cyclictest0-21swapper/323:20:203
21253991910,6cyclictest35966irq/58-eth000:20:123
21253991910,6cyclictest0-21swapper/321:53:463
2121999196,11cyclictest6254-21ssh21:45:190
2121999193,12cyclictest151rcuc/020:40:120
2121999192,14cyclictest17153-21ssh00:15:180
2121999191,7cyclictest131rcu_preempt19:20:040
2121999190,5cyclictest53646usb-storage19:30:040
2125399189,6cyclictest35966irq/58-eth000:35:133
2125399189,6cyclictest0-21swapper/323:57:043
2125399188,3cyclictest0-21swapper/300:33:453
2125399183,10cyclictest0-21swapper/322:46:563
2125399182,10cyclictest0-21swapper/323:40:123
2125399182,10cyclictest0-21swapper/323:30:043
2125399182,10cyclictest0-21swapper/319:35:043
2125399182,10cyclictest0-21swapper/319:20:053
21253991810,6cyclictest35966irq/58-eth021:35:123
2121999186,3cyclictest131rcu_preempt23:30:180
2121999183,12cyclictest20601-21kthreadcore20:15:210
2121999183,12cyclictest19345-21grep20:45:270
2121999182,13cyclictest18160-21fschecks_time19:40:140
2125399179,6cyclictest0-21swapper/322:12:203
2125399179,5cyclictest35966irq/58-eth023:48:573
2125399179,5cyclictest35966irq/58-eth021:10:133
2125399178,6cyclictest0-21swapper/321:40:273
2125399178,6cyclictest0-21swapper/321:40:263
2125399173,11cyclictest35966irq/58-eth021:15:133
2125399172,9cyclictest35966irq/58-eth020:10:123
2125399172,10cyclictest0-21swapper/321:25:123
2125399172,10cyclictest0-21swapper/320:45:133
21253991711,4cyclictest0-21swapper/300:15:173
2125399171,13cyclictest35966irq/58-eth022:00:133
2125399171,12cyclictest0-21swapper/319:10:283
21253991710,4cyclictest35966irq/58-eth021:46:083
2121999172,3cyclictest131rcu_preempt20:00:130
2121999172,12cyclictest3584-21fschecks_time19:25:130
2121999171,12cyclictest32087-21vmstat19:50:300
2125399162,9cyclictest0-21swapper/320:35:123
2125399162,9cyclictest0-21swapper/320:35:113
2125399161,12cyclictest0-21swapper/320:20:283
2125399161,10cyclictest0-21swapper/320:00:233
21253991610,4cyclictest0-21swapper/322:06:053
2121999161,11cyclictest1443-21grep20:30:190
2125399159,4cyclictest0-21swapper/320:15:193
2125399152,10cyclictest0-21swapper/320:25:123
21253991510,3cyclictest33-21ksoftirqd/321:05:173
21253991510,3cyclictest33-21ksoftirqd/321:05:163
21253991510,3cyclictest33-21ksoftirqd/320:30:263
21253991510,3cyclictest0-21swapper/322:20:103
2121999157,6cyclictest21225-21wc19:10:010
2125399141,9cyclictest0-21swapper/320:55:263
2125399141,7cyclictest0-21swapper/319:25:143
2125399141,6cyclictest0-21swapper/320:50:193
2125399141,11cyclictest0-21swapper/319:40:273
2125399138,3cyclictest33-21ksoftirqd/319:30:253
2125399137,4cyclictest33-21ksoftirqd/319:50:293
2125399131,6cyclictest0-21swapper/320:40:183
2125399129,2cyclictest33-21ksoftirqd/319:45:263
2125399121,8cyclictest0-21swapper/319:20:003
2125399113,6cyclictest0-21swapper/321:00:133
2125399101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional